np_datasamples forces a full-size copy that dominates parse time in cgroup-limited containers — proposal for a safe owned-array accessor

[mdenolle]

Summary

MS3TraceSeg.np_datasamples returns a borrowed view over the C-owned sample
buffer (np.frombuffer over an ffi.buffer, mstracelist.py:390 in 0.8.1)
whose lifetime is tied to the MS3TraceList. The docstring is explicit that
"if the data are needed beyond the lifetime of this instance, a copy must be
made" (mstracelist.py:360-364), so every consumer that keeps the arrays
does seg.np_datasamples.copy().

That copy is nearly free on bare metal but is the single dominant cost in
memory-cgroup-limited containers (Fargate/Lambda-class): it is a second
full-size allocation touched once (fresh-page memcpy), and under a cgroup
plus VM it costs 25–75 ms per 11 MB channel-day — more than the entire
libmseed parse+decode. Meanwhile the library already contains the efficient
alternative (record_list=True + create_numpy_array_from_recordlist(),
mstracelist.py:392-431), but it is marked "for advanced use only"
(add_buffer docstring, mstracelist.py:1155-1156) and is easy to miss.

Why this matters downstream

We hit this while moving an ambient-noise cross-correlation pipeline
(NoisePy fed by seisfetch, which
uses pymseed as its only miniSEED reader) onto AWS Fargate/Lambda, where
S3 archives are read as whole channel-day objects and every container is
cgroup-limited. Profiling showed parse throughput 1.3–1.6x worse than bare
metal, and the entire regression was this one .copy() — not libmseed, not
Python overhead. Switching to create_numpy_array_from_recordlist() made
parse time flat across every environment (native, 2cpu/4g, 0.5cpu/512m) and
cut parse peak RSS from ~92 MB to ~37 MB per day file, which is the
difference between fitting a 512 MB Lambda and not. Serverless/container
deployment is increasingly the default for cloud-archive seismology, so the
"obvious" API steering users to the one pattern that degrades there seems
worth fixing.

Micro-benchmark

11.3 MB SCEDC Steim2 channel-day (CI.PASC.00.BHZ, 21,984 records, merging to
1 trace / 1 segment, 6,912,000 int32 samples). time.perf_counter, min of 7,
pymseed 0.8.1, numpy 2.5, CPython 3.12, arm64. Container = python:3.12-slim
(linux/arm64) under Docker with --cpus=2 --memory=4g.

case	native macOS (ms)	container 2cpu/4g (ms)	container 0.5cpu/512m (ms)
from_buffer(unpack_data=True)	24.1	21.5	21.4
from_buffer(unpack_data=True, record_list=True)	28.6	24.0	24.6
from_buffer(record_list=True) (headers only)	9.1	5.9	6.0
record_list=True → create_numpy_array_from_recordlist()	21.2	20.1	20.2
unpack_data=True → np_datasamples.copy()	25.4	48.3	95.2
per-record MS3Record.from_buffer(unpack_data=True) loop	31.8	32.6	32.8
obspy 1.4 read() (reference)	33.7	31.0	32.6

Observations:

• The C parse/decode itself (unpack_data=True, first row) does not degrade
  under cgroup limits at all.
• The only component that degrades is the numpy copy of the C buffer:
  +1.3 ms native, +27 ms at 2 cpu/4 GB, +74 ms at 0.5 cpu/512 MB. Two
  full-size buffers (libmseed internal + numpy copy) and a memcpy over fresh
  pages is the difference; peak RSS for the copy path is ~92 MB vs ~37 MB
  for the decode-into-numpy path.
• create_numpy_array_from_recordlist() (decode directly into a numpy-owned
  array via mstl3_unpack_recordlist) is the fastest correct path in every
  environment — but it requires record_list=True (+~4.5 ms and ~6 MB of
  per-record header duplicates for a 22k-record day file) and lives behind
  an "advanced use only" warning.
• validate_crc=False is worth ~0.5–0.7 ms on v2 data (no CRCs present).

Proposal

Any of these would let downstream users hit the fast path without private
API or "advanced" workflows:

1. Safe owned accessor — e.g. MS3TraceSeg.to_numpy() (or an
   np_datasamples keepalive) where the returned array holds a reference to
   the parent MS3TraceList (arr.base chain or a small owner capsule), so
   the zero-copy view is safe to use after the trace list goes out of scope.
   Zero cost, removes the reason the copy idiom exists.

2. Bulk decode-to-numpy without a record list — a mode such as
   MS3TraceList.from_buffer(buffer, unpack='numpy') that decodes each
   segment directly into a numpy-owned allocation during the read (what
   create_numpy_array_from_recordlist() does, minus the record-list
   requirement), and retains the first-record encoding on the segment so
   callers don't need record_list=True just to learn "STEIM2".

3. Docs — promote record_list=True +
   create_numpy_array_from_recordlist() as the recommended
   high-performance buffer workflow; the current "advanced use only" note
   steers users toward unpack_data=True + copy, which is the pathological
   pattern in serverless/container deployments.

Happy to share the benchmark script (self-contained, reads one channel-day
file) or PR any of the above.

Environment

• pymseed 0.8.1 (same API verified present and working in 0.6.0/0.7.0 and in the current release 0.9.3 — the benchmark table numbers are from 0.8.1)
• libmseed (bundled), CPython 3.12.x, numpy 2.5.1
• native: macOS 15 / Apple Silicon; container: Docker python:3.12-slim
  linux/arm64 with --cpus/--memory cgroup limits

[chad-earthscope]

Hey, I think proposal #1 was done here:
3e06380

Bottom line: update to the latest pymseed (currently v0.9.4) and try again.

As for proposal #2, the record list is needed to know how big of an array is needed, otherwise you have to guess or continuously extend/resize. An alternative might be to decode and concatenate when done reading, it's more data movement in memory and more memory overall though so needs benchmarking and thought. Worth considering for ObsPy too: obspy/obspy#3604

[mdenolle]

Confirmed — the 0.9.4 keepalive ("Hold the trace list from its data sample views", 3e06380) fully resolves the headline problem. Re-ran the micro-benchmark on the same 11.3 MB Steim2 channel-day (min of 7, numpy 2.4, CPython 3.12 in the container):

case	native macOS (ms)	container 0.5cpu/512m (ms)
unpack_data=True → np_datasamples, no copy (0.9.4 safe view)	23.5	21.8
unpack_data=True → np_datasamples.copy() (old idiom)	26.5	94.4
record_list=True → create_numpy_array_from_recordlist()	21.5	21.7

The view now survives trace-list GC (verified: .base chain pins the memoryview → trace list), so the copy idiom — and its 4× cgroup penalty — is simply no longer needed. The safe view matches the record-list decode path everywhere.

On proposal #2 — understood on the sizing constraint, and agreed the decode-and-concatenate alternative is a memory-traffic trade that needs real benchmarking; thanks for the obspy/#3604 pointer. From our side there's no remaining need: seisfetch stays on the record-list path (we want the per-record encoding metadata anyway) and its test suite passes clean against 0.9.4. As far as we're concerned, proposal #1 closes the issue — the only thing left is the docs nudge (proposal #3) if you think it's worth steering buffer users toward either safe path. Thanks for the fast turnaround!

[chad-earthscope]

The just released pymseed v0.9.5 contains this:

Add MS3TraceSeg.take_np_datasamples(), transferring a segment's decoded sample buffer to a numpy array with no copy, so the array has no dependency on the trace list and can outlive it.

Unless you wish to do something like read data and then selectively unpack samples only from certain segments, this is a huge win over creating record list. The common case of reading a file and getting arrays of data in numpy (which can still be selected using codes), should always use this method over record_list=True → create_numpy_array_from_recordlist(). Any help testing that would be appreciated.