Skip to content

fix: e2e cleanup sweeps all active mounts before rm#636

Open
qiffang wants to merge 2 commits into
mainfrom
dat9-dev1/fix-e2e-teardown-cleanup
Open

fix: e2e cleanup sweeps all active mounts before rm#636
qiffang wants to merge 2 commits into
mainfrom
dat9-dev1/fix-e2e-teardown-cleanup

Conversation

@qiffang

@qiffang qiffang commented Jun 27, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Root cause: cleanup() only unmounted the last tracked $MOUNT_POINT, leaving earlier mounts from multi-mount scenarios (sandbox_restore, fast_worktree) still active
  • When set -e triggered exit mid-scenario, rm -rf $RUN_ROOT hit "Device or resource busy" and in-flight HTTP requests got context-canceled
  • Now cleanup() sweeps $RUN_ROOT for any remaining FUSE mounts and unmounts each before removing the directory
  • Also dumps all mount logs on failure, not just the last one

Context

From e2e failure:

rm: cannot remove '.../drive9-sandbox_restore/mount': Device or resource busy
git workspace refresh failed: context canceled

Test plan

  • Run git-workspace e2e on CI
  • Verify multi-mount scenarios (sandbox_restore, fast_worktree) clean up properly on early exit

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Bug Fixes
    • Improved smoke-test log collection to include additional related log files, not only the most recent mount log.
    • Enhanced cleanup to better detect and stop any remaining mounted filesystem(s) when failures occur between mount operations, reducing leftover mount issues.

The cleanup() function only unmounted the last tracked $MOUNT_POINT,
leaving earlier mounts from multi-mount scenarios (sandbox_restore,
fast_worktree) still active. When set -e triggered exit mid-scenario,
rm -rf $RUN_ROOT hit "Device or resource busy" and in-flight HTTP
requests got context-canceled.

Now cleanup() sweeps RUN_ROOT for any remaining FUSE mounts and
unmounts each before removing the directory. Also dumps all mount
logs on failure, not just the last one.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Jun 27, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 38394ab9-97b8-4172-bf81-b62fe12a2437

📥 Commits

Reviewing files that changed from the base of the PR and between 129a53c and 14efeb5.

📒 Files selected for processing (1)
  • e2e/git-workspace-smoke-test.sh
🚧 Files skipped from review as they are similar to previous changes (1)
  • e2e/git-workspace-smoke-test.sh

📝 Walkthrough

Walkthrough

The smoke-test script now dumps all mount-related logs under RUN_ROOT except the active one, and cleanup now scans RUN_ROOT for remaining mounted FUSE directories and stops each of them after the tracked mount.

Changes

Smoke-test teardown and log collection

Layer / File(s) Summary
Mount log collection
e2e/git-workspace-smoke-test.sh
dump_mount_log now iterates over RUN_ROOT *.log files and prints each one except the current MOUNT_LOG.
Mount cleanup sweep
e2e/git-workspace-smoke-test.sh
cleanup now stops the tracked mount, then finds mounted directories under RUN_ROOT and calls stop_mount for each remaining mount path.

Estimated code review effort: 2 (Simple) | ~10 minutes

Suggested reviewers: srstack

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately captures the main change: e2e cleanup now sweeps and unmounts all active mounts before removing the run directory.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch dat9-dev1/fix-e2e-teardown-cleanup

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@e2e/git-workspace-smoke-test.sh`:
- Around line 316-320: The log sweep in dump_mount_log() is traversing
still-mounted FUSE trees, which can hang and print unrelated workspace logs.
Update the find/read loop to avoid live mountpoints by pruning mounted
directories explicitly or restricting the search to the dedicated mount-log
location, and keep the existing MOUNT_LOG exclusion in place.
- Around line 666-674: The mount cleanup loop in the shell script can still
descend into active FUSE mounts before stop_mount runs, which can block cleanup
and leave rm -rf exposed to resource-busy failures. Update the find pipeline
used in the mount enumeration to avoid traversing inside mount directories (for
example by pruning candidate mount points) or switch to iterating the known
parent mount directory directly. Keep the fix localized around the while/read
loop and the is_mounted check so stop_mount only operates on discovered mount
roots.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 0e9fa746-363f-4b19-acd9-6328a33fb226

📥 Commits

Reviewing files that changed from the base of the PR and between 669f9c7 and 129a53c.

📒 Files selected for processing (1)
  • e2e/git-workspace-smoke-test.sh

Comment thread e2e/git-workspace-smoke-test.sh
Comment thread e2e/git-workspace-smoke-test.sh

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 129a53c1c0

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread e2e/git-workspace-smoke-test.sh Outdated
fi
# Also dump any other mount logs under RUN_ROOT that weren't the last one.
if [ -n "${RUN_ROOT:-}" ] && [ -d "$RUN_ROOT" ]; then
find "$RUN_ROOT" -name '*.log' -type f 2>/dev/null | while read -r logf; do

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Prune active mounts from log discovery

When a scenario fails while its FUSE mount is still active, on_exit calls this before cleanup, so this recursive find can walk into $case_root/mount and traverse the mounted repo instead of only the case-root log files. I checked GNU find locally: --help lists -prune/-xdev as traversal controls, and find -D search confirms a normal recursive search visits children under a matched mount directory; in the failure mode this can block on the unhealthy FUSE mount or dump unrelated repo .log files before cleanup runs.

Useful? React with 👍 / 👎.

Comment thread e2e/git-workspace-smoke-test.sh Outdated
[ -z "$mp" ] && continue
MOUNT_POINT="$mp"
stop_mount
done < <(find "$RUN_ROOT" -maxdepth 3 -name mount -type d 2>/dev/null | while read -r d; do

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Prune mount dirs during stale-mount sweep

This sweep is meant to find mounted $RUN_ROOT/*/mount directories so it can unmount them, but find -maxdepth 3 still descends through each depth-2 mount directory to inspect depth-3 repo entries before the loop gets a chance to call stop_mount. In the exact stale/wedged FUSE case this cleanup handles, that traversal can hang or do remote I/O; use a shallower search or print-and-prune matching mount directories so discovery does not enter the mounts.

Useful? React with 👍 / 👎.

- dump_mount_log(): add -xdev to avoid crossing into mounted FUSE
  filesystems that may be wedged (D-state), preventing hangs and
  stray log file collection from mounted workspaces
- cleanup(): reduce -maxdepth 3 to -maxdepth 2 with -prune so find
  discovers */mount dirs without descending into them, avoiding
  hangs on stuck FUSE mounts before stop_mount can run

Addresses review comments from coderabbitai and codex on PR #636.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant