Skip to content

mds watch silently loses edits made during startup #317

Description

@dean0x

crates/mds-cli/src/watch.rs publishes compiled output before arming change detection, in both watch modes:

Step File mode (run_watch_file) Dir mode (dir_watch_startup)
1. compile + write output :934 :1711
2. create watcher :973 :1730
3. watcher.watch() (arm inotify) :991 :1757, :1780
4. snapshot_state (mtime, size baseline) :1025 :1826

An edit landing between steps 1 and 4 is invisible to both detectors, permanently:

  • inotify was not yet armed, so no event is ever generated — not delayed, never created.
  • The liveness probe (state_differs, :718) compares against a baseline snapshotted at step 4, which captured the already-edited state, so it reports "no change" indefinitely.

first_tick (:637, :713) was added to close this window but fires once and is consumed; the poisoned baseline defeats every later tick.

User impact

Someone who saves a file during mds watch startup silently loses that save. No error, no warning. The process stays alive and healthy; the edit is simply never picked up until the file is saved again.

Evidence

Measured on Linux aarch64 in Docker; reproduces on both overlayfs and tmpfs, so not a container artifact:

  • Lost updates did not recover at 10 s, 20 s, or 60 s — the loss is permanent, not slow.
  • The child process was alive in 100% of lost cases and recovered on the next edit (4/120 direct probe runs lost an update).
  • Reproduced at load average 0.06 with a single child process — CPU contention is an amplifier, not the cause.
  • Delaying the edit by 2 s eliminated it entirely (0/40).
  • Platform-specific: Linux full-suite 9/15 runs failed; macOS 0/5 (consistent with inotify vs FSEvents semantics).

How it surfaced

As chronic flakiness in crates/mds-cli/tests/cli_watch.rs, most visibly watch_clear_non_tty_no_ansi_escape failing at :520 — the wall-clock wait, not an ESC assertion. It failed 2 of 3 CI attempts on PR #308's head. Because Rust — fmt, clippy, test is a required status context, this coin-flips release merges. The flaky test was reporting a real defect.

Rejected fixes, on measured evidence

  • Raising the timeout — lost updates don't recover at 60 s; predicted effect zero, and it makes each failure 6× more expensive.
  • Serializing the test suite — 8× wall-clock cost (3.49 s → 27.9 s), and still 2/5 failures.
  • nextest retry / flaky annotations — would hide a real user-facing bug.

Fix direction

Reorder both startups so the watcher is armed and the baseline captured before output is published, establishing the invariant: for every file of interest, its (mtime, size) baseline is captured no later than the moment it is first read, and its directory watch is armed no later than that.

A fix is in progress.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingcliCLI commands and optionsrustPull requests that update rust code

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions