core/datatype-1.0.0 byteorder underspecification

[embray]

This is a companion to #470. In that one I managed to understand that if the ndarray source is a binary block then specifying the datatype byteorder is required (unless the datatype is nested, in which case the byteorder is only required for individual fields).

I noticed however that at some point there was also a core/datatype-1.0.0 tag added. This tag never seems to be used inside the ndarray objects themselves, where it's just implicit (and that same datatype schema is already a sub-schema of the ndarray schema).

If I had to guess pulling out a separate datatype tag has two (I think good) motivations:

• Factor it out from the ndarray schema which is already fairly complex
• Make it usable in other custom tags where one might want to describe a datatype (e.g. a custom sparse array type)

That's all fine--however, the datatype schema allows the datatype to be a simple string like "int64". Without a byteorder specified it's ambiguous and at best implementation-defined. The schema documentation reads simply:

The byteorder for the field. If not provided, the byteorder of the datatype as a whole will be used.

But this provision is only meaningful for compound datatypes. It doesn't say what should be assumed if there is just a scalar datatype given.

Some options from which we could take a combination:

• Patch datatype-1.0.0 to specify a default byte order to assume if not specified (should never just assume host datatype which is useless for an interchange format like ASDF)--this should be done I think regardless to eliminate the ambiguity
• Update -> datetype-1.1.0 to add scalar datatype names like ">int32" and "<int32" to specify the byteorder where applicable (still would need to specify which is default without the prefix)
• Update -> datatype-2.0.0 (breaking change) to disallow a standalone scalar datatype without a byteorder specified: This is the most drastic where it might be nicer just to specify a default

[braingram]

Thanks for opening this issue.

I noticed however that at some point there was also a core/datatype-1.0.0 tag added. This tag never seems to be used inside the ndarray objects themselves, where it's just implicit (and that same datatype schema is already a sub-schema of the ndarray schema).

One thing to clarify is that there is no core/datatype-* tag, only a core/datatype-* schema. The schema is not yet used in any "stable" schema. It's only used in ndarray-1.2.0 (unstable):

asdf-standard/resources/unstable/schemas/stsci.edu/asdf/core/ndarray-1.2.0.yaml

Line 258 in 4bd3e06

$ref: "datatype-1.0.0"

More details about this can be found in: #479
There are more "unstable" changes coming before the 1.7.0 manifest is marked as stable (which will require downstream changes).

The main motivation for the datatype schema was to decouple the asdf metaschema from the ndarray schema (which has not been completed).

asdf-standard/resources/stable/schemas/stsci.edu/asdf/asdf-schema-1.1.0.yaml

Line 39 in 4bd3e06

- $ref: "http://stsci.edu/schemas/asdf/core/ndarray-1.1.0#/definitions/datatype"

Since the metaschema references ndarray, a ndarray schema update (like that to fix for example #452) will trigger a metaschema update which will then trigger an update to every schema.

If I had to guess pulling out a separate datatype tag has two (I think good) motivations:

* Factor it out from the `ndarray` schema which is already fairly complex

* Make it usable in other custom tags where one might want to describe a datatype (e.g. a custom sparse array type)

That's all fine--however, the datatype schema allows the datatype to be a simple string like "int64". Without a byteorder specified it's ambiguous and at best implementation-defined. The schema documentation reads simply:

The byteorder for the field. If not provided, the byteorder of the datatype as a whole will be used.

But this provision is only meaningful for compound datatypes. It doesn't say what should be assumed if there is just a scalar datatype given.

Some options from which we could take a combination:

* Patch datatype-1.0.0 to specify a default byte order to assume if not specified (should never just assume host datatype which is useless for an interchange format like ASDF)--this should be done I think regardless to eliminate the ambiguity

* Update -> datetype-1.1.0 to add scalar datatype names like `">int32"` and `"<int32"` to specify the byteorder where applicable (still would need to specify which is default without the prefix)

* Update -> datatype-2.0.0 (breaking change) to disallow a standalone scalar datatype without a byteorder specified: This is the most drastic where it might be nicer just to specify a default

The datatype schema here isn't intended to be (necessarily) a complete description of a datatype. The main target is the metaschema usage where things like int32 are acceptable. I don't think the byteorder description is helpful for the metaschema usage (but is for the unstable ndarray usage).

[embray]

🤯

…

On Wed, Jan 21, 2026, 14:35 Brett Graham ***@***.***> wrote: *braingram* left a comment (asdf-format/asdf-standard#501) <#501 (comment)> Thanks for opening this issue. I noticed however that at some point there was also a core/datatype-1.0.0 <https://www.asdf-format.org/projects/asdf-standard/en/latest/generated/stsci.edu/asdf/core/datatype-1.0.0.html> tag added. This tag never seems to be used inside the ndarray objects themselves, where it's just implicit (and that same datatype schema is already a sub-schema of the ndarray schema). One thing to clarify is that there is no core/datatype-* tag, only a core/datatype-* schema. The schema is not yet used in any "stable" schema. It's only used in ndarray-1.2.0 (unstable): https://github.com/asdf-format/asdf-standard/blob/4bd3e0603ee2be23629f9687c3a7ecb8fc195f95/resources/unstable/schemas/stsci.edu/asdf/core/ndarray-1.2.0.yaml#L258 More details about this can be found in: #479 <#479> There are more "unstable" changes coming before the 1.7.0 manifest is marked as stable (which will require downstream changes). The main motivation for the datatype schema was to decouple the asdf metaschema from the ndarray schema (which has not been completed). https://github.com/asdf-format/asdf-standard/blob/4bd3e0603ee2be23629f9687c3a7ecb8fc195f95/resources/stable/schemas/stsci.edu/asdf/asdf-schema-1.1.0.yaml#L39 Since the metaschema references ndarray, a ndarray schema update (like that to fix for example #452 <#452>) will trigger a metaschema update which will then trigger an update to every schema. If I had to guess pulling out a separate datatype tag has two (I think good) motivations: * Factor it out from the `ndarray` schema which is already fairly complex * Make it usable in other custom tags where one might want to describe a datatype (e.g. a custom sparse array type) That's all fine--however, the datatype schema allows the datatype to be a simple string like "int64". Without a byteorder specified it's ambiguous and at best implementation-defined. The schema documentation reads simply: The byteorder for the field. If not provided, the byteorder of the datatype as a whole will be used. But this provision is only meaningful for compound datatypes. It doesn't say what should be assumed if there is just a scalar datatype given. Some options from which we could take a combination: * Patch datatype-1.0.0 to specify a default byte order to assume if not specified (should never just assume host datatype which is useless for an interchange format like ASDF)--this should be done I think regardless to eliminate the ambiguity * Update -> datetype-1.1.0 to add scalar datatype names like `">int32"` and `"<int32"` to specify the byteorder where applicable (still would need to specify which is default without the prefix) * Update -> datatype-2.0.0 (breaking change) to disallow a standalone scalar datatype without a byteorder specified: This is the most drastic where it might be nicer just to specify a default The datatype schema here isn't intended to be (necessarily) a complete description of a datatype. The main target is the metaschema usage where things like int32 are acceptable. I don't think the byteorder description is helpful for the metaschema usage (but is for the unstable ndarray usage). — Reply to this email directly, view it on GitHub <#501 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAFFCNJDDMPKLEFK3EWOEAT4H56BLAVCNFSM6AAAAACSMVMA56VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTONZYGE3TANJYGA> . You are receiving this because you authored the thread.Message ID: ***@***.***>