Pin and shallow-fetch every upstream source; build toolshed in parallel - #75
Merged
Merged
Conversation
Each tool stage cloned its upstream repo in full and then checked out a pinned SHA, downloading the entire history to use a single snapshot. Replace that with `git init` + `git remote add` + `git fetch --depth=1 <sha>` + `git checkout FETCH_HEAD`, which fetches only the pinned commit and skips the throwaway default-branch download entirely. Measured on nitros9: 39MB of history down to 18MB, 3669 commits down to 1. Note that plain `git fetch <sha>` is NOT depth-limited -- fetching into a shallow clone deepens it back to full history -- so the `--depth=1` on the fetch is load-bearing, not decorative. Toolshed is a tag rather than a SHA, so it uses `git clone --depth=1 --branch v2_5` instead. `git fetch origin <tag>` would not work here: it populates only FETCH_HEAD without creating refs/tags/<tag>, so a subsequent `git checkout v2_5` fails with "pathspec did not match". ZX0 was the one unpinned source in the file; pin it at ecde3a2, which is the current default-branch HEAD, so this locks in what the build already produced rather than moving it. Also drop the `yes | rm -r` guard on the preprocessor tree, which existed to auto-confirm write-protected files and is unnecessary now that the directory is one we created. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Upstream release bump. Isolated in its own commit so a miscompile in the 6809 C compiler bisects here rather than to a build-plumbing change. 0.1.98 is still published, so this is a deliberate upgrade rather than a forced one. Verified that `cmoc --os9` still builds a working OS-9 module via the smoke test. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Move the pin from 0e60e91 to 1fd2f46. The version bump and the fetch
rewrite land together because they are genuinely coupled: the archive
filenames embed the release, so the sparse-checkout path cannot be
written without choosing a version.
Three things change as a consequence of the new pin:
- The in-tree archives are now v5.33, so the v5.28 filenames would no
longer resolve.
- v5.33 ships `BasTo6809.2.Compile` under its final name, so the
rename of "BasTo6809.2.Compile copy" is gone. Confirmed against both
the arm64 and x86_64 archives.
- The arch branches collapse into one unzip/mv pair driven by an ARCH
variable, since only the filename differed.
Upstream keeps every historical release binary in the tree, so even a
--depth=1 checkout pulls ~1.9GB and takes ~2.5 minutes. Fetching blobs
lazily and sparse-checking out only the manual plus this architecture's
zip brings that to ~43MB in ~5 seconds, against the same immutable SHA.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The smoke suite exercised five tools and left five others with no
runtime coverage at all -- a stage could build and install a broken
binary without any test noticing.
Add five steps:
- tasm6801 assembles a minimal MC-10 source to .c10/.obj
- zx0 -> dzx0 round trip, asserting the output actually shrank
- salvador compressing and dzx0 decompressing, which cross-checks the
two compressors against each other rather than trusting either alone
- decbpp minifying a line of DECB BASIC (it reads stdin, not a file
argument -- passing a filename silently produces nothing)
- the nitros9 defs being present for lwasm
The nitros9 step is only a presence check. Asserting on it properly
means assembling a source that `use`s os9.d through lwasm, which is
worth doing later.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`make lint` covered the two shell scripts but nothing checked the
Dockerfile itself, which is where most of this image's complexity lives.
Wire hadolint into the lint target alongside shellcheck, pinned to
v2.15.1 rather than a floating tag.
Fix the one finding worth fixing: pip was baking its wheel cache into
the image layer, so `--no-cache-dir` (DL3042).
Ignore three rules in .hadolint.yaml, each with its reasoning recorded
there rather than as inline pragmas:
- DL3003 (use WORKDIR, not cd) works against the multi-stage design.
Each stage is deliberately one RUN that clones, builds, stages and
deletes its tree; WORKDIR would leak those directories into the
image and split each stage across layers.
- DL3008 (pin apt versions) is impractical for a devcontainer tracking
Ubuntu 24.04. The reproducibility that matters here -- the 6809
toolchain -- is pinned by SHA and tarball version instead.
- DL3015 (--no-install-recommends) is unmeasured. default-jdk and the
SDL/Mesa dev packages may pull needed runtime libraries in via
recommends, so adopting it needs a build plus `make test` on both
architectures.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Toolshed sat in `foundation`, which every tool stage derives from. That made it a serialization point -- all eleven tool stages waited on it -- and a cache liability, since bumping toolshed invalidated every one of them. Nothing actually builds against it. Checking each stage's build files, the only build-time dependency on the foundational toolchain is CMOC's configure, which aborts without `lwasm` (>= 4.11). Toolshed is needed only at run time, because basto6809todsk shells out to `decb`. So move it to `FROM foundation AS toolshed` alongside the other tools and pull it into `final` with one more COPY. lwtools stays where it is. Its Makefile maps DESTDIR to $(DESTDIR)/usr/bin rather than /usr/local/bin, so a plain `make install DESTDIR=/staging` would relocate the binaries. Install normally and copy the paths into /staging, the same way the jgrinder stage handles naken_asm. Verified against a full build: all ten binaries land at their original paths reporting Toolshed 2.5, `make test` passes (step 2 exercises `decb` end to end), and building --target cmoc no longer references toolshed at all. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Six focused commits covering how the image fetches its upstream sources, two
toolchain upgrades, test coverage for the tools that had none, and Dockerfile
linting.
The largest single change is mechanical: every stage cloned its upstream repo in
full and then checked out a pinned SHA, downloading an entire history to use one
snapshot. These now fetch only the pinned commit.
Commits
a8e2e1cec2e8741af788f024b752299a05fmake lint;pip --no-cache-dirc6c9c2aThe two toolchain upgrades are deliberately isolated so that a miscompile
bisects to the compiler change rather than to build plumbing.
Things worth a second opinion
git fetch <sha>is not depth-limited. Fetching into a shallow clonedeepens it back to full history, so
--depth=1on the fetch is load-bearing.On nitros9 the difference is 39MB/3669 commits versus 18MB/1 commit.
Toolshed cannot use
git fetch origin <tag>. That populates onlyFETCH_HEADwithout creatingrefs/tags/<tag>, so a followinggit checkout v2_5fails with "pathspec did not match". It usesclone --depth=1 --branch v2_5instead.BASIC-To-6809 keeps every historical release binary in-tree, so even a
depth-1 checkout pulls ~1.9GB and takes ~2.5 minutes. A blob-filtered sparse
checkout of just the manual and one architecture's zip brings that to ~43MB in
~5 seconds against the same immutable SHA. Its version bump and fetch rewrite
share a commit because the archive filenames embed the release — the
sparse-checkout path cannot be written without picking a version.
Nothing built against toolshed. The only build-time dependency on the
foundational toolchain is CMOC's configure requiring
lwasm. Moving toolshedout of
foundationremoves a serialization point for all eleven tool stages andstops a toolshed bump from invalidating their caches. Its Makefile maps DESTDIR
to
$(DESTDIR)/usr/binrather than/usr/local/bin, so the stage installsnormally and copies the paths into
/staging.Verification
make build,make test(10/10) andmake lintall pass at the branch tip.The toolshed move was checked further: all ten binaries land at their original
paths reporting Toolshed 2.5, and
--target cmocno longer references toolshedat all.
This was all verified on arm64 only. The x86_64 leg — including the other
BASIC-To-6809 v5.33 archive, whose contents I inspected but never executed — is
unproven until CI's matrix runs.
The nitros9 test step is only a presence check; asserting on it properly means
assembling a source that
usesos9.dthrough lwasm.No version bump here —
bump-version.ymlownspackage.json,coco-dev:10and
docker-compose.build.yml:4.🤖 Generated with Claude Code