Skip to content

Allow saving non-ascii strings to netCDF with no '_Encoding' attribute #7195

Description

@pp-mo

While writing the detail account in #7194, it has struck me that one usage that you can't get with the current proposal is to "automatically" encode data with utf-8 in the absence of an specific _Encoding attribute, and so to save without an _Encoding attribute.

So, the proposal is to change that :

  • make "utf8" the default encoding for write, as it is for load
  • (therefore) don't require an encoding attribute to save non-ascii strings

Update 2026-07-15: I think we have now discussed this + agreed it would be better

So, I believe that is correct + ok according to the latest CF Conventions (>=v1.12).
But with the current proposal, Iris will insist on having an encoding attribute, and saving with one.

  • From the latest CF (post-v1.12), such data is certainly "ok"
  • .. and Iris will load it without question
  • .. but Iris refuses to save like this without also adding the _Encoding attribute
  • so, such data won't directly read-write roundtrip (will fail on re-save),
    and requires the user to add an encoding attribute to make it save

Technically, with the code as it is, if we were simply to change iris.fileformats.netcdf.DEFAULT_WRITE_ENCODING to "utf-8", then I think all the above problems would be solved :

  • unicode data with no encoding attribute will automatically save as utf8, with no encoding attribute created
  • both ascii and utf8 data don't need an "_Encoding" attribute, for either read or write
    but can have one if wanted.
  • changes to Chardata docs #7194 will be needed -- but I think it all gets simpler

We held off including this change in #7194
because

  • (a) it is out of scope, and
  • (b) it is non-trivial because
    • tests need fixing / extending
    • docs need adjusting accordingly (N.B. overall its a slightly simpler message)

NOTE: #7092 may well affect the same areas.
I suspect that we could use attributes['_Encoding'] = 'NC_STRING' or similar, to represent that data was-loaded-as/should-be-saved-as variable-length strings (but not write that in files).
However I think this is now unlikely to make it into the v3.16 release.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Status
    No status
    Status
    Backlog
    Status
    📚 Backlog

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions