Add an I/O observation facility by madsbk · Pull Request #1033 · rapidsai/kvikio

madsbk · 2026-08-14T08:51:32Z

This PR introduces a hook into monitoring KvikIO operations, with the goal of building statistics, Quent timelines, and whatever else wants to know what the I/O layer is doing. This PR is the base. A follow-up introduces a concrete Monitor that makes statistics easy to get.

The hook reports whole KvikIO operations (the logical level) so a pread() is a single observation however many reads the thread pool issued underneath. Physical operations can be added along the same path later, which could be the basis of #1016.

Each call produces a kvikio::Observation: its span, the offset and size etc. To receive them, derive from kvikio::Monitor and register it. A monitor is told when an operation starts as well as when it finishes.

// Example of a Monitor that tracks how many KvikIO operations are in flight at any moment.
class QueueDepth : public kvikio::Monitor {
  void on_start(kvikio::Observation const&) noexcept override { ++_in_flight; }
  void on_finish(kvikio::Observation const&) noexcept override { --_in_flight; }
  std::atomic<int> _in_flight{0};
};

QueueDepth gauge;
auto id = kvikio::register_monitor(&gauge);

Overhead

Measured on my local workstation:

~3 ns per call when nobody is observing, which is a gate check and a branch.
~60 ns per call when somebody is, or 1 % of a 64 KiB pread(), and nothing detectable at a megabyte.

Confirmed against a real workload: cudf-polars PDS-H query 1 at scale 10, with and without a monitor attached, showed no difference outside noise.

What is not observed

The cuFile asynchronous API on a GDS system, and the batch API, complete without KvikIO seeing it, so they emit nothing. Handling those needs a stream-completion callback, which is future work.

Follow-up: statistics

The next PR adds kvikio::statistics::SummaryMonitor, which is a Monitor and nothing more:

monitor = kvikio.SummaryMonitor()   # statistics are now on
...
print(monitor.get())

KvikIO I/O summary
  wall time            1.876 s
  busy time            366.592 ms (19.54 % of the wall time)
  busy bandwidth       8.95 GB/s
  operations           7970
  bytes requested      3.06 GiB
  bytes transferred    3.06 GiB
  errors               0

Those are real numbers, from a cudf-polars run, and they show a very useful busy bandwidth. 8.95 GB/s is the rate while KvikIO actually had work in flight, where dividing the same bytes by the wall clock would have said 1.75 GB/s and described the query rather than the storage.

A TimelineMonitor, for when things happened rather than how much, is planned after that.

`nbytes()` caches the file size and `write()` invalidates it, but two `pwrite()` paths never go through `write()`: the host path and the sub-threshold device shortcut. So `nbytes()` could report a stale size after a write. Both now invalidate the cache once the write has completed, so a `nbytes()` call racing with an in-flight `pwrite()` cannot leave a stale size cached either. Unrelated to the rest of this branch.

madsbk · 2026-08-14T11:55:49Z

-  if (is_read_out_of_bounds(file_offset, size, _nbytes)) {
-    std::stringstream ss;
-    ss << "cannot read " << file_offset << "+" << size << " bytes into a " << _nbytes
-       << " bytes file (" << _endpoint->str() << ")";
-    KVIKIO_FAIL(ss.str(), std::invalid_argument);
-  }


Not removed, just moved to the top of read, before the buffer is classified.

madsbk · 2026-08-14T11:59:14Z

-  if (is_read_out_of_bounds(file_offset, size, _nbytes)) {
-    std::stringstream ss;
-    ss << "cannot read " << file_offset << "+" << size << " bytes into a " << _nbytes
-       << " bytes file (" << _endpoint->str() << ")";
-    KVIKIO_FAIL(ss.str(), std::invalid_argument);
-  }


Not removed, just moved to the top of read, before the buffer is classified.

wence-

Broadly looks good, I think.

wence- · 2026-08-14T13:08:05Z

+  ObservationKind kind{ObservationKind::Logical};
+
+  /// When the operation started.
+  TimePoint start{};
+  /// When the operation finished.
+  TimePoint end{};
+  /// Byte offset into the file or remote object.
+  std::size_t offset{};
+  /// Number of bytes requested.
+  std::size_t size{};
+  /// Number of bytes actually transferred. Differs from `size` on a short read, and is zero for an
+  /// operation that failed.
+  std::size_t bytes_transferred{};
+
+  /// The backend that carried out the operation.
+  IoBackend backend{IoBackend::Posix};
+  /// The direction of the operation.
+  TransferDirection direction{TransferDirection::Read};
+  /// The kind of memory the caller's buffer lives in.
+  MemoryKind memory_kind{MemoryKind::Host};
+  /// False if the operation failed.
+  bool ok{true};
+  /// Identifies this operation, uniquely within the process.
+  std::uint64_t id{};
+
+  /// HTTP method, e.g. `"GET"`. Null for local I/O.
+  char const* http_method{nullptr};


nit: Check the layout of this struct for potentially more padding than necessary

Good catch, moved kind down to join the other small enums and id up with the other 8-byte fields. Now
64 bytes with a single 3-byte tail hole.

wence- · 2026-08-14T13:15:15Z

  KVIKIO_NVTX_FUNC_RANGE(size);
-  if (get_compat_mode_manager().is_compat_mode_preferred()) {
-    return detail::posix_device_read(
+  detail::expect_not_in_monitor();


question: Should we make this a no-op unless in debug mode?

I would rather keep it in release. It is one thread-local bool load per user-facing call, and the whole disabled observation path measured at about 3 ns.

What it buys is a clear error instead of a hang.

wence- · 2026-08-14T13:24:08Z

+  // A monitor registered after this operation began never saw its start and ignores the finish,
+  // which is why no flag is latched here.
+  notify_finished(_observation);
+}


I don't understand this comment

Fair, it was too compressed. It was trying to say why the recorder does not track which monitors it notified. Rewritten as:

// Delivered to whoever is registered now, which need not be who was registered when the // operation began. A monitor that missed the start ignores this, as `Monitor::on_finish()` // documents, so the recorder does not have to remember which monitors it told.

wence- · 2026-08-14T13:36:01Z

+      if (o.size == 0 || o.id == 0 || o.start == kvikio::TimePoint{}) { ++_malformed; }
+      if (o.end != kvikio::TimePoint{} || o.bytes_transferred != 0) { ++_malformed; }


I don't understand this checking of the start against the timepoint. These could match/mismatch validly, no?

You are right TimePoint{} is the clock epoch, so it is a stand-in for "unset" that a real timestamp could in principle collide with.

Replaced with something that has content: the map now holds the whole Observation, and on_finish() checks the completion carries the same size and the same start as the record seen at submission.

madsbk self-assigned this Aug 14, 2026

madsbk added improvement Improves an existing functionality non-breaking Introduces a non-breaking change labels Aug 14, 2026

madsbk force-pushed the logical-observations branch 3 times, most recently from d42da15 to 0dece5b Compare August 14, 2026 10:50

madsbk force-pushed the logical-observations branch 2 times, most recently from e1bf9bf to 7360ca8 Compare August 14, 2026 11:52

madsbk commented Aug 14, 2026

View reviewed changes

madsbk force-pushed the logical-observations branch from 7360ca8 to 0846f46 Compare August 14, 2026 12:10

Add an I/O observation facility

9a4fe49

madsbk force-pushed the logical-observations branch from 0846f46 to 9a4fe49 Compare August 14, 2026 12:12

madsbk marked this pull request as ready for review August 14, 2026 12:55

madsbk requested review from a team as code owners August 14, 2026 12:55

rapidsai deleted a comment from copy-pr-bot Bot Aug 14, 2026

madsbk requested a review from kingcrimsontianyu August 14, 2026 12:56

wence- reviewed Aug 14, 2026

View reviewed changes

based on reviews

11f3c68

madsbk requested a review from wence- August 14, 2026 14:20

kingcrimsontianyu reviewed Aug 14, 2026

View reviewed changes

Comment thread cpp/include/kvikio/observation.hpp

Use ALL_CAPS enumerators

36bc839

madsbk requested a review from kingcrimsontianyu August 14, 2026 14:39

Navigation Menu

Uh oh!

Add an I/O observation facility - #1033

Add an I/O observation facility#1033
madsbk wants to merge 4 commits into
rapidsai:mainfrom
madsbk:logical-observations

madsbk commented Aug 14, 2026 •

edited

Loading

Uh oh!

madsbk Aug 14, 2026 •

edited

Loading

Uh oh!

madsbk Aug 14, 2026

Uh oh!

wence- left a comment

Uh oh!

wence- Aug 14, 2026

Uh oh!

madsbk Aug 14, 2026

Uh oh!

wence- Aug 14, 2026

Uh oh!

madsbk Aug 14, 2026 •

edited

Loading

Uh oh!

wence- Aug 14, 2026

Uh oh!

madsbk Aug 14, 2026

Uh oh!

Uh oh!

wence- Aug 14, 2026

Uh oh!

madsbk Aug 14, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		if (o.size == 0 \|\| o.id == 0 \|\| o.start == kvikio::TimePoint{}) { ++_malformed; }
		if (o.end != kvikio::TimePoint{} \|\| o.bytes_transferred != 0) { ++_malformed; }

Uh oh!

Conversation

madsbk commented Aug 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overhead

What is not observed

Follow-up: statistics

Uh oh!

madsbk Aug 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wence- left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

madsbk Aug 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

madsbk commented Aug 14, 2026 •

edited

Loading

madsbk Aug 14, 2026 •

edited

Loading

madsbk Aug 14, 2026 •

edited

Loading