FITSIO Replacement Interest

[meawoppl]

Heyo astronomer friends!

I have been working with some friends to open source a bunch of rust astronomy utilities from my previous work place. I was curious if y'all would accept a PR to switch over to a pure rust fits library with feature parity to the current fitsio crate.

We use it already at @CosmicFrontierLabs for our space telescope simulation and guidance tooling as well as in the @OrbitalCommons , and I would be happy to whip up a quick PR to get y'all building against it. We have a compatibility layer that exports the same symbols, so the diff would be minimal, and it make things much easier from a dev and deployment standpoint (I see y'all have a lot of workaround to wedge cfitsio in carefully)

Let me know if interested! Thanks for doing what y'all do.

[d3v-null]

That sounds great @meawoppl ! Would you mind writing it so that it's a feature flag that swaps between old and new libraries to allow for easier comparison with the cfitsio version? that's fine if that's too much added complexity.

Heads up to @gsleap who is currently about to make an update to a dependency involving cfitsio MWATelescope/mwalib@main...bf_metafits_changes_vcs_read_size

[meawoppl]

I did a bit of profiling last night, and it is generally within a factor of two for performance wise.

I see outside of testing, y'all mostly use the low level file handles. (Maybe transliterated from some earlier c/c++ code?) Not a problem, but I was curious if I might be able to clean that up a bit in parallel. Using the low level handles can lead to somewhat unpredictable behaviors in my experience with that library...

What are the approximate sizes of the data that flows through those?

I think those calls could be simpler (in practice the filesystem is almost always the limiter), but I want to make sure I don't cause a performance regressions etc.

[d3v-null]

Thanks Matty,
Yes that part of Marlu probably evolved from something like this https://github.com/MWATelescope/Conv2UVFITS/blob/main/uvfits.c

I believe at the time the high level fitsio API didn't support all the features we needed for group params, which are definitely needed for AIPS117, the uvfits standard. https://www.aips.nrao.edu/TEXT/PUBL/AIPSMEM117.PDF

I'm interested in seeing if there's a performance benefit in swapping fits libraries, but for full adoption, we would need to be sure that fitsio-pure is compatible with the whole MWA stack. As you've already mapped this out, you're probably aware that there are a few gaps that would need to be addressed.

mwalib fitsio-pure/adopters/mwalib.md at main · OrbitalCommons/fitsio-pure - needs long strings and caller owned buffers
marlu fitsio-pure/adopters/marlu.md at main · OrbitalCommons/fitsio-pure - needs support for random groups
hyperdrive fitsio-pure/adopters/mwa_hyperdrive.md at main · OrbitalCommons/fitsio-pure - needs, array-in-cell columns, column range reads.

MWALib is used (by Birli ) to read large "correlator" fits files that can be tens of GB per file, in batches totaling over 1TB. Then we typically average before writing to CASA measurement set (most common in radio astronomy), or uvfits (much faster), so these files can be hundreds of GB too. MWALib also uses fits files to store metadata about telescope settings for an observation (metafits), and hyperdrive also writes calibration solutions to another fits format, but these are usually much smaller.

These repositories all contain (small) test datasets if you want to get a better feel for the formats. I can also provide larger files for testing, but I'm happy to run the benchmarks myself (these use large datasets that aren't included in the repo)

I'll also be testing this on arm64

I'd imagine there's a lot of opportunity to improve the readability of the all the parts of the code that call the low level fitsio api, the unit test coverage should be pretty good.

[meawoppl]

Thanks for the detailed notes.

A few notes from my end:

• The core functionality is well exercised in arm64, we use it as an interchange format from our arm64 hardware presently.
• We added the random groups support last week
• The long string feature also was poked into place
• There is a bit more work to get to full spec parity, mainly in testing, but I think that could land before next week depending on a few other thread I am working on.

These new features are a bit less exercised than ideal, but I would be happy to pull your test cases into the regression suite I have been building up with your blessing: https://github.com/OrbitalCommons/fits-test-cases From there I can squish any lingering bugs.

There are a few small changes that will close the performance gap (mainly mmap low level stuff) which is pretty easy to do when you have a strong battery of correctness checks to backup the work. I don't expect the performance of this library will exceed the cfitsio except maybe in very small files where the c version is a little naive in how it does buffers. In practice this format is almost always disk limited, so its a bit moot IMHO.

Longer term, I want to lock in some benchmarks, less as optimization goals, and more as making sure that maintenance work doesn't tank performance by accident. Around then, I may solicit your larger file tests. To date, most files wrangled by this library are in the few hundred MB range and the speed difference with cfitsio is like 400ms vs 300ms to load/save which is pretty immaterial for our use cases which do many seconds of processing afterward per frame.

[d3v-null]

That's great to hear you're filling out some of those features!

Another factor that comes into play is that this software mostly runs on a lustre HPC filesystem, making it very sensitive to io syscall response times, so a change that might not be noticeable when benchmarking on a fast SSD on a dev machine would blow out when we test on the HPC filesystem, so that's one layer of testing we'll need to do on our end.

Once you're done with fits, maybe you could work your magic on casa measurement sets? (JK)

@gsleap can I get your permission to provide (non-proprietary) MWA data to OrbitalCommons for benchmarking / regression testing purposes?

[gsleap]

Yeah absolutely - any non-embargoed visibilities are all public! Get Outlook for Android<https://aka.ms/AAb9ysg>

…

________________________________ From: Dev Null ***@***.***> Sent: Thursday, February 19, 2026 10:21:42 AM To: MWATelescope/Marlu ***@***.***> Cc: Greg Sleap ***@***.***>; Mention ***@***.***> Subject: Re: [MWATelescope/Marlu] FITSIO Replacement Interest (Issue #39) [https://avatars.githubusercontent.com/u/2578076?s=20&v=4]d3v-null left a comment (MWATelescope/Marlu#39)<#39 (comment)> That's great to hear you're filling out some of those features! Another factor that comes into play is that this software mostly runs on a lustre HPC filesystem, making it very sensitive to io syscall response times, so a change that might not be noticeable when benchmarking on a fast SSD on a dev machine would blow out when we test on the HPC filesystem, so that's one layer of testing we'll need to do on our end. Once you're done with fits, maybe you could work your magic on casa measurement sets<#38>? (JK) @gsleap<https://github.com/gsleap> can I get your permission to provide (non-proprietary) MWA data to OrbitalCommons for benchmarking / regression testing purposes? — Reply to this email directly, view it on GitHub<#39 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AE2L5FT5MLULDERUBFV6HQ34MTXYNAVCNFSM6AAAAACVHMZKGCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTSMRTG4ZDSNBRHA>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[meawoppl]

I left the HPC world behind some years ago, so I will leave the esoterica of lustre for y'all to ponder.

cftsio is well made and again I don't expect to be faster, this was originally spawned out of a desire to vastly decrease maintenance efforts on some code that a number of people have to run on at least 2 operating systems and 3 arch'es, so that is a thing.

[meawoppl]

I'll ping back here when the other features land. I'll import those files as test cases obviously leaving a breadcrumb trail back to ya'll

[meawoppl]

Didn't mean to close the issue, can y'all reopen?