Spool Visualisation

[andreas-wuestefeld]

A spool is a great source of information. Notably for data-availability and acquisition parameters.

Spool.viz... returns currently a "not implemented" type of error

I suggest to implement basic plotting of data evolution. Two parameters should be straight forward

1. Show data-gaps (i.e. instrument downtime); This assumes that all manufacturers have their own characteristic file duration. Not sure if Sintella ProtoBuf is possible, but that can be seen later

2. Sampling rate: See if and when acquisition parameters were changed.

Furthermore, adding gaugelength to the spool would allow to also plot that parameter evolution.

Here is a vizualisation of a recent dataset (boring for sampling rate, but you should get the idea)

Questions:
Is this considered useful by others?
Where would be a logical place to put such code?

[d-chambers]

Hi @andreas-wuestefeld,

I think these could be quite useful. All of the visualizations would be based on the dataframe of the spool contents (created with Spool.get_contents()), so it should be quite straightforward to make a single function that plots how a selected attr (or multiple attrs) varies with time.

For plotting gaps, I have been meaning to add a get_gap_df (or something similar) method to the spool for awhile. There is a similar function in ObsPlus so it should be straight-forward to adapt. However, I would prefer the spool method remain light-weight, and the dataframe wrangling bits go in uitls/pd.py, it's the place for misc pandas code. In pd.py you might define get_gaps_from_df, then Spool.get_gap_df can just call get_contents and pass that to `get_gaps_df`` and return.

The tests for these functions would go in the corresponding test files, (tests/test_utils/test_pd.py and tests/test_core/test_spool). In each case, make a new test class named after the function name.

For the visualizations, I think making a viz/spool.py module would be a good option.
The Patch.viz name space is just a patch namespace. To make a spool namespace you can just define it like the PatchVizNamespace and add a similar entry point in pyproject.toml.

[andreas-wuestefeld]

@d-chambers I finally got around to work more on this. But I need some expert input and opinion:

The main concern is about the future data base. If this is happening soon-ish, I'd rather wait with this until the new mechanism is in place.
I would love to add more statisics plots (like evolution of channel spacing, file size, gauge length,...) to utilize the full power of the database.

Then there is a question on how to handle documentation and tests. my test files are tens-of-megabyte, and should not be part of the git-package. Can we somehow auto-generate an example spool on the fly?

Then is a question how to handle short spools (less than 24, or just a few days). the current calendar-view is geared towards long (multi-month) spools. We could then show the calendar as hour-long cells instead of the day-cells. But that could also be a future feature.

Note:

1. I kept so far the gap detection logic inside the scripts. they can be moved outside once the exact mechanisms are agreed
2. currently (for testing purposes) it runs as a standalone script if the user alters the script to a valid spool file. I would be open for better suggestions.

See the branch here: https://github.com/DASDAE/dascore/tree/feature/spool-viz

[andreas-wuestefeld]

[d-chambers]

Hey @andreas-wuestefeld,

Thanks for putting this together; the screenshots make the use case much clearer.

On the spool database concern, I don't think the backend work needs to block this. The Spool on dev still exposes get_contents(), and that should be the boundary used by these visualizations. If the plotting code accepts a spool, calls spool.get_contents(), and then operates only on the resulting DataFrame, changes to the index/catalog backend should not matter. The new spool index has already landed on dev, so you could rebase to dev and continue, but honestly there is so many new things in dev I hope to have a decently long beta release.

The one backend-specific part to remove is the standalone example using HDFPatchIndexManager; normal usage should start from a spool or DataFrame instead. So rather than this df = HDFPatchIndexManager(tmpfile).get_index() you can just do Spool.get_contents() to get the same thing. The only difference on dev is that all coords/attrs will be included in the dataframe, whereas now only some attrs and time coord are included, but extra columns shouldn't hurt your functions.

For the same reason, the tests do not need large real data files or even generated patches. I suggest allowing each plotting function to accept either a spool or a DataFrame, with a small helper that converts a spool using get_contents() and otherwise validates the DataFrame. Nearly all behavior can then be tested using small, deterministic synthetic DataFrames containing time_min, time_max, time_step, and any optional metadata needed by a particular plot. One small integration test using an existing example spool should be enough to confirm that the spool input follows the same path as spool.get_contents().

A few suggestions on the current implementation:

• It would be useful to keep the calculations separate from the Matplotlib rendering. For example, the calendar-matrix and timing-statistics helpers can return ordinary arrays/data structures that are easy to test directly, while the plotting functions only turn those results into axes.

• DASCore already has the SpoolNameSpace mechanism, so these could ultimately be exposed as something like spool.viz.calendar() and spool.viz.stats(). Keeping the underlying functions callable with a DataFrame would still be useful for testing and standalone use.

• The gap and availability calculations should define how multiple independent streams are handled. Sorting every row together can create false gaps between stations/acquisitions, while summing overlapping rows can report more than 100% daily availability.(EG For multi-acquisition spools). Grouping by the appropriate spool metadata and unioning time intervals would make the behavior predictable. - I suggest adding a group_columns=None argument to the gap and availability calculations. On master, None can resolve to the same metadata columns the spool currently passes to ChunkManager (network, station, dims, data_type, and tag), while an explicit sequence overrides them and an empty sequence deliberately treats the input as one stream. Only columns present in the contents DataFrame need to be used. Calculations should then be performed independently within each group, merging overlapping or adjacent time intervals before computing gaps or daily coverage so availability cannot exceed 100%. This keeps the public API stable while allowing dev to resolve None through its newer spool/configured grouping behavior later.

• The default calendar range currently treats the final date as an exclusive endpoint, so a one-day spool can produce no populated cells. I think a daily calendar is a reasonable first implementation for short spools once that boundary is fixed; an hourly layout can be a later enhancement.

• method="mode" is currently advertised but not implemented, so it should either be implemented or omitted for now. Also, using a_min/a_max with np.clip would retain compatibility with DASCore's supported NumPy versions. (I think some

I would not block an initial PR on every possible metadata statistic or the hourly calendar. A focused first version with a stable DataFrame-based input, correct gap/overlap semantics, tests, and the spool namespace wiring would already be useful.