Skip to content

Lazily load the file tree per directory instead of scanning it whole - #1

Open
yazydzhi wants to merge 1 commit into
Diffuzmetall:mainfrom
yazydzhi:feat/lazy-directory-tree
Open

Lazily load the file tree per directory instead of scanning it whole#1
yazydzhi wants to merge 1 commit into
Diffuzmetall:mainfrom
yazydzhi:feat/lazy-directory-tree

Conversation

@yazydzhi

Copy link
Copy Markdown

Problem

`listTree` fetches the entire workspace recursively (`host.list_paths`, capped at `TREE_LIMIT = 10_000`) on every mount of the Files panel, and again every 10 seconds via the background poll (`useFilesWorkspace`'s `setInterval`). Expanding/collapsing a folder in `TreePane` only filters that already-fully-loaded flat list client-side (`filterVisibleEntries`) — collapsing never actually frees anything, because everything was already in memory.

On a large workspace (tens of thousands of files — e.g. a big Obsidian vault, a big monorepo) this means:

  • Every panel open/poll does a full recursive filesystem walk, even though the user only ever looks at a few expanded folders at a time.
  • The 10k-entry cap (`TREE_LIMIT`) gets hit constantly, showing "results truncated" even though the user only wanted to see one small subfolder.
  • (Separately, until fix(host-daemon): avoid stack overflow listing large directory trees get-bb/bb#2363 lands/ships, a workspace with a single subtree over ~100k files could even crash the recursive host-side walk with a stack overflow — this PR isn't about that, but it means the plugin no longer needs to lean on the biggest, slowest possible call for routine browsing.)

Fix

Add a new `listDirectory` RPC — a single-level directory read via `bb.sdk.hosts.directory` (`host.browse_directory`), which BB already exposes for exactly this "interactive path browser" use case. Unlike `host.list_paths`, it reads exactly one directory, so its cost is independent of workspace size.

`useFilesWorkspace` now keeps loaded children keyed by directory path (`Map<path, FileTreeEntry[]>`) instead of one big flat array:

  • Mount fetches only the root directory (`listDirectory({ path: "" })`).
  • Expanding a folder fetches its children with one shallow call and merges them in.
  • Collapsing a folder deletes its subtree from state (not just from a "visible" filter), so re-expanding fetches fresh data instead of stale cached data sitting around forever.
  • Opening a path (e.g. right after creating a file) expands every ancestor folder to reveal it, fetching each one lazily — same UX as before, just fetched on demand.
  • The 10s background refresh and the post-mutation refresh (`refreshTree`) now re-fetch only the root plus whatever directories are currently expanded, instead of walking the whole tree every time.

`listTree` is unchanged and still does a full recursive host-side search when the search box has a query — that's still exactly the right tool for search, since search has to look at everything regardless of what's expanded.

Testing

  • `npx tsc --noEmit` — clean.
  • `npx vitest run` — 40/40 passing.
  • Updated `app.test.tsx`'s initial-mount mocks from `listTree` to `listDirectory` (they were exercising the eager root load, which is now served by the new endpoint — the mock shape needed to change, the test intent didn't).
  • Added a new test, "lazily loads a directory's children on expand and drops them on collapse", covering the actual behavior change: expanding fetches and merges a subdirectory's children, and collapsing removes them from state (verified by re-collapsing and checking `entries` no longer contains the child path).

Test plan

  • `npm run typecheck`
  • `npm test` (40/40)
  • Manual: open a large workspace, confirm the tree shows only the root initially, expanding folders fetches their contents on demand, and collapsing then re-expanding shows fresh data

…whole

listTree fetched the entire workspace recursively (host.list_paths,
capped at TREE_LIMIT=10,000) on every mount, and again every 10s on
the background poll. Expand/collapse only filtered that in-memory
flat list — collapsing a folder never freed anything. On a large
workspace (tens of thousands of files) this was slow, wasteful, and
after a host-daemon bugfix landed, still routinely hit the plugin's
10k display cap on every load.

Add a new listDirectory RPC (single-level, via
bb.sdk.hosts.directory / host.browse_directory) alongside the
existing listTree. useFilesWorkspace now keeps loaded children keyed
by directory path:

- Mount fetches only the root directory.
- Expanding a folder fetches its children with one shallow call.
- Collapsing a folder drops its subtree from state so it's re-fetched
  fresh next time, instead of just being hidden.
- Opening a path (e.g. after creating a file) expands every ancestor
  folder to reveal it, fetching each lazily.
- The 10s background refresh and post-mutation refresh now re-fetch
  only the root plus whatever's currently expanded, not the whole
  tree.

listTree is unchanged and still does a full recursive host-side
search when the search box has a query — full-depth search is still
exactly what search should do.

Updated app.test.tsx's initial-mount mocks from listTree to
listDirectory (they were exercising the eager root load, now served
by the new endpoint), and added a test covering expand fetching
children and collapse dropping them from state.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant