Skip to content

[V2:05] Add Source Materialization Layer - #19

Open
ritorhymes wants to merge 2 commits into
eips-wg:masterfrom
ritovision:v2/05-git-source-materialization
Open

[V2:05] Add Source Materialization Layer#19
ritorhymes wants to merge 2 commits into
eips-wg:masterfrom
ritovision:v2/05-git-source-materialization

Conversation

@ritorhymes

@ritorhymes ritorhymes commented May 5, 2026

Copy link
Copy Markdown
Contributor

Part of V2: Multi-repo local build system.

Summary

This PR adds the source-materialization boundary that later workspace, runtime, and serve/watch PRs use to choose between clean cloned sources and dirty local working trees. It introduces SourceMaterialization::{Clean, Dirty} plus git helpers for availability, clone/fetch, dirty working-tree copying, tracked-path sync, and sibling merge behavior. Existing build and changed-file flows still use Clean, so review should focus on the git.rs source-state contract, not on a switch to dirty as the default.

  • Add SourceMaterialization::{Clean, Dirty} and use manifest-derived RepositoryUse values to carry resolved repository endpoints into git::Fresh::new.
  • Add repository availability, clone, fetch, dirty status, and working-tree materialization helpers in git.rs.
  • Add tracked working-tree path discovery and sync helpers for dirty local source materialization.
  • Update sibling merge behavior so local file sibling sources follow checked-out HEAD, while remote sibling sources continue fetching master.
  • Route existing build and changed-file flows through clean source materialization.

Review Notes

This is the git/source foundation for later workspace and runtime behavior. It does not make dirty local workspaces the default yet; existing build and changed flows continue using clean cloned sources.

Workspace init uses the clone/repo availability pieces in V2:06. Execution policy starts selecting clean versus dirty source modes in V2:08. The prepared runtime pipeline consumes materialized active and sibling sources in V2:10, and serve/watch behavior consumes the dirty sync helpers in V2:11.

Review this as the source-materialization boundary: git.rs owns repository availability, cloned source setup, dirty working-tree copying, tracked path sync, and sibling merge behavior. Later PRs decide when those paths are used.

Dirty materialization rejects conflicted index entries before copying and fails explicitly on unsupported dirty paths, so later runtime PRs can fail with an explicit source state error instead of silently building from an ambiguous tree.

RepositoryUse carries the active Build.toml location and declared sibling repository endpoints into source materialization, keeping repository resolution explicit at the git boundary.

Verification

  • Review src/git.rs for SourceMaterialization, RepositoryUse, clone/fetch setup, dirty materialization, tracked path sync, dirty rejection errors, and sibling merge behavior.
  • Review src/changed.rs and src/main.rs for the clean materialization wiring used by existing behavior.
  • Review src/config.rs for manifest-derived repository locations and RepositoryUse construction used by source materialization.
  • Review git materialization tests for tracked file copying, staged file sync, symlink handling, and non-repository errors.

@ritorhymes
ritorhymes force-pushed the v2/05-git-source-materialization branch from 9d809a2 to 34531df Compare May 5, 2026 07:53
@ritorhymes ritorhymes changed the title Add Source Materialization Layer [V2:05] Add Source Materialization Layer May 5, 2026
@ritorhymes
ritorhymes force-pushed the v2/05-git-source-materialization branch 2 times, most recently from 5d204fb to 65a9b3c Compare May 9, 2026 20:10
@ritorhymes
ritorhymes force-pushed the v2/05-git-source-materialization branch from 65a9b3c to 4b52595 Compare May 26, 2026 01:02
@ritorhymes
ritorhymes force-pushed the v2/05-git-source-materialization branch from 4b52595 to 75f58c8 Compare June 21, 2026 22:41

@SamWilsn SamWilsn left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This'll be an incredibly useful feature!

It'll have to wait until after we get this renderer running in CI though. No point in improving DevEX for a tool that can't be used 😅

Add workspace-local `.build-eips.toml` loading and starter configuration. Introduce `config::ActiveRepo` to load the selected checkout’s `Build.toml`, validate explicit `-C` roots, and expose active repository context to later commands. Keep source materialization, initialization, and diagnostics in their owning later branches.
Add clean and dirty source materialization for the active repository and sibling content. Use `config::RepositoryUse` throughout the runtime path, preserve tracked working-tree materialization and `index_path`, and reject dirty active manifests in clean modes. Build, check, and serve prepare sources without fetching the active upstream; `changed` retains upstream fetching for comparison.
@ritorhymes
ritorhymes force-pushed the v2/05-git-source-materialization branch from 75f58c8 to c66646b Compare July 18, 2026 06:44
@ritorhymes

Copy link
Copy Markdown
Contributor Author

This'll be an incredibly useful feature!

It'll have to wait until after we get this renderer running in CI though. No point in improving DevEX for a tool that can't be used 😅

That sequencing logic collapses under its own premise. The WG EIPs system is not currently in use either, so by the same reasoning there would be no point improving the renderer, the preprocessor, or any other part of V2 until the completed system were already deployed. But the completed system cannot become usable until its constituent parts are designed and implemented.

Whether a particular capability belongs in the final architecture is a legitimate question. “The tool is not in use yet” is not. This source-materialization layer belongs in the intended end state and is already integrated into the downstream build, check, and serve paths.

This is also not merely a DevEX improvement. Source materialization defines what source the system actually builds:

  • clean committed content versus tracked working-tree changes;
  • active-repository versus sibling-repository sources;
  • how those sources are prepared in an isolated build tree;
  • explicit rejection of a dirty working tree in clean mode, with the offending tracked paths and untracked-file count reported;
  • and dedicated handling for conflicted or unsupported paths during dirty materialization.

Those semantics apply to CI as well as local development. A renderer running in CI still needs a deterministic definition of which repository state it is rendering and how that state is prepared. This branch establishes that layer; the later runtime branches consume it.

I also want to avoid unnecessary churn across this stack. There are fourteen downstream preprocessor branches after this one, in addition to the coordinated work in the other repositories. When you overhauled the manifest branch, I spent an entire weekend reworking downstream consumption to fit the new design. That was substantial churn, but it produced a meaningful architectural improvement, so it was worth doing.

I am willing to absorb that kind of churn for changes that materially improve correctness, architecture, or the final system. I do not think we should create comparable churn by removing or postponing valid foundational work merely because a later CI integration slice has not landed yet.

At the current review rate, this process is likely to take another six to eight months. We do not need to add avoidable rework to that timeline when the implementation is valid, belongs in the final system, and is already consumed downstream.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants