Skip to content

Pin and shallow-fetch every upstream source; build toolshed in parallel #75

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jamieleecho merged 6 commits into from
Aug 6, 2026
Merged

Pin and shallow-fetch every upstream source; build toolshed in parallel #75

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
a8e2e1c
build: fetch upstream sources shallowly and pin every revision
jamieleecho Aug 6, 2026
ec2e874
build: upgrade CMOC 0.1.98 -> 0.1.99
jamieleecho Aug 6, 2026
1af788f
build: upgrade BASIC-To-6809 to v5.33 and fetch it sparsely
jamieleecho Aug 6, 2026
024b752
test: cover tasm6801, ZX0, salvador, decbpp and the nitros9 defs
jamieleecho Aug 6, 2026
299a05f
build: lint the Dockerfile with hadolint
jamieleecho Aug 6, 2026
c6c9c2a
refactor: build toolshed as its own parallel stage
jamieleecho Aug 6, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .dockerignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
.git
.github
.gitignore
.hadolint.yaml
Makefile
README.md
bump-info.json
Expand Down
28 changes: 28 additions & 0 deletions .hadolint.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
# hadolint configuration for the coco-dev image.
#
# Every ignore below is deliberate; please justify any addition here rather
# than sprinkling inline `# hadolint ignore=` pragmas through the Dockerfile.
ignored:
# "Use WORKDIR to switch to a directory".
#
# Each tool stage is a single RUN that clones, builds, installs into
# /staging, and deletes its source tree. The `cd`s are scoped to that one
# layer on purpose -- switching to WORKDIR would leak the directories into
# the image as persistent state and split each stage across several layers,
# which is exactly what the multi-stage refactor set out to avoid.
- DL3003

# "Pin versions in apt-get install".
#
# This is a devcontainer that intentionally tracks whatever Ubuntu 24.04
# ships. Pinning ~40 apt versions would break on every upstream point
# release. The reproducibility that matters here -- the 6809 toolchain -- is
# pinned by git SHA and tarball version instead.
- DL3008

# "Avoid additional packages by specifying --no-install-recommends".
#
# Not yet measured. Several of the packages below (default-jdk, the SDL and
# Mesa dev packages) can pull needed runtime libraries in via recommends, so
# this needs a build plus `make test` on both arches to adopt safely.
- DL3015
136 changes: 88 additions & 48 deletions Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,12 +3,15 @@
# Multi-stage build.
#
# `foundation` holds everything shared by every later build: the apt packages,
# the Python venv, and the two foundational toolchain pieces (lwtools and
# toolshed) that other builds may rely on. Every remaining tool is built in its
# own stage `FROM foundation`, so BuildKit compiles them concurrently and a
# change to one tool no longer invalidates the others' cache. Each tool stage
# installs into an isolated `/staging` prefix; the `final` stage assembles the
# image with one `COPY --from=<tool> /staging/ /` per tool.
# the Python venv, and lwtools. lwtools is the one genuine build-time
# dependency -- CMOC's configure aborts without `lwasm` (>= 4.11). Toolshed is
# only needed at run time, so it builds as its own parallel stage below.
#
# Every remaining tool is built in its own stage `FROM foundation`, so BuildKit
# compiles them concurrently and a change to one tool no longer invalidates the
# others' cache. Each tool stage installs into an isolated `/staging` prefix;
# the `final` stage assembles the image with one `COPY --from=<tool> /staging/ /`
# per tool.
#
# Compile-heavy stages (mame, cmoc, java_grinder, lwtools, toolshed, zx0) use a
# ccache BuildKit cache mount, so recompiling the same source (e.g. after a
Expand Down Expand Up @@ -66,7 +69,7 @@ WORKDIR /root
RUN python -m venv venv
ENV VIRTUAL_ENV=/root/venv
ENV PATH="$VIRTUAL_ENV/bin:$PATH"
RUN pip install \
RUN pip install --no-cache-dir \
coco-tools==0.27 \
milliluk-tools==0.1 \
mc10-tools==0.10 \
Expand All @@ -86,7 +89,7 @@ RUN pip install \
ENV CCACHE_DIR=/root/.ccache \
CCACHE_MAXSIZE=2G

# --- Foundational toolchain (other build stages may depend on these) ---
# --- Foundational toolchain (other build stages depend on this) ---

# Install lwtools
RUN --mount=type=cache,target=/root/.ccache,sharing=shared \
Expand All @@ -97,45 +100,65 @@ RUN --mount=type=cache,target=/root/.ccache,sharing=shared \
make install && \
cd /root && rm -rf lwtools-4.24 lwtools-4.24.tar.gz


# ===========================================================================
# Parallel tool stages. Each is independent and FROM foundation, and installs
# into /staging (mirroring the final layout) so `final` can COPY it in.
# ===========================================================================

# Install Toolshed
#
# Nothing builds against this -- it is a run-time dependency of the image
# (basto6809todsk shells out to `decb`), so it belongs here rather than in
# foundation. Its Makefile maps DESTDIR to $(DESTDIR)/usr/bin instead of
# /usr/local/bin, so install normally and copy the paths into /staging, the
# same way the jgrinder stage handles naken_asm.
FROM foundation AS toolshed
RUN --mount=type=cache,target=/root/.ccache,sharing=shared \
git clone https://github.com/nitros9project/toolshed.git && \
git clone --depth=1 --branch v2_5 \
https://github.com/nitros9project/toolshed.git && \
cd toolshed && \
git checkout v2_5 && \
make -j -C build/unix CC="ccache gcc" && \
make -C build/unix install && \
mkdir -p /staging/usr/local/bin /staging/usr/local/share && \
for t in ar2 os9 mamou cecb decb tocgen makewav dis68 lst2cmt cocofuse; do \
cp "/usr/local/bin/$t" /staging/usr/local/bin/; \
done && \
cp -R /usr/local/share/toolshed /staging/usr/local/share/ && \
cd /root && rm -rf toolshed


# ===========================================================================
# Parallel tool stages. Each is independent and FROM foundation, and installs
# into /staging (mirroring the final layout) so `final` can COPY it in.
# ===========================================================================

# Install preprocessor
FROM foundation AS preproc
RUN git clone https://github.com/yggdrasilradio/preprocessor.git && \
RUN git init -q preprocessor && \
(cd preprocessor && \
git checkout 62c4ace79eeffa48817f429363816d79abea77c3 && \
git remote add origin https://github.com/yggdrasilradio/preprocessor.git && \
git fetch --depth=1 origin 62c4ace79eeffa48817f429363816d79abea77c3 && \
git checkout -q FETCH_HEAD && \
mkdir -p /staging/usr/local/bin && \
cp decbpp /staging/usr/local/bin/) && \
(yes | rm -r preprocessor)
rm -rf preprocessor

# Install ZX0 data compressor
FROM foundation AS zx0
RUN --mount=type=cache,target=/root/.ccache,sharing=shared \
git clone https://github.com/einar-saukas/ZX0 && \
cd "ZX0/src" && \
git init -q ZX0 && \
cd ZX0 && \
git remote add origin https://github.com/einar-saukas/ZX0 && \
git fetch --depth=1 origin ecde3a2ae05061fe06469ed46df81a33b7de7d86 && \
git checkout -q FETCH_HEAD && \
cd src && \
make -j CC="ccache gcc" CFLAGS=-O3 EXTENSION= && \
mkdir -p /staging/usr/local/bin && \
cp zx0 dzx0 /staging/usr/local/bin && \
cd /root && rm -rf ZX0

# Install salvador (fast near-optimal ZX0 compressor)
FROM foundation AS salvador
RUN git clone https://github.com/emmanuel-marty/salvador && \
RUN git init -q salvador && \
cd salvador && \
git checkout 1662b625a8dcd6f3f7e3491c88840611776533f5 && \
git remote add origin https://github.com/emmanuel-marty/salvador && \
git fetch --depth=1 origin 1662b625a8dcd6f3f7e3491c88840611776533f5 && \
git checkout -q FETCH_HEAD && \
mkdir clang-hack && \
ln -s /usr/bin/cc clang-hack/clang && \
(PATH=./clang-hack:$PATH make -j) && \
Expand All @@ -146,25 +169,33 @@ RUN git clone https://github.com/emmanuel-marty/salvador && \

# Install key OS-9 defs from nitros-9
FROM foundation AS nitros9
RUN git clone https://github.com/nitros9project/nitros9.git && \
RUN git init -q nitros9 && \
cd nitros9 && \
git checkout 27c67d5c445db631abfd5b45d49870364d9eacb6 && \
git remote add origin https://github.com/nitros9project/nitros9.git && \
git fetch --depth=1 origin 27c67d5c445db631abfd5b45d49870364d9eacb6 && \
git checkout -q FETCH_HEAD && \
mkdir -p /staging/usr/local/share/lwasm && \
cp -R defs/* /staging/usr/local/share/lwasm/ && \
cd /root && rm -rf nitros9

# Install java grinder (and naken_asm, which builds it and runs the tests)
FROM foundation AS jgrinder
RUN --mount=type=cache,target=/root/.ccache,sharing=shared \
git clone https://github.com/mikeakohn/naken_asm.git && \
git clone https://github.com/mikeakohn/java_grinder && \
git init -q naken_asm && \
(cd naken_asm && \
git remote add origin https://github.com/mikeakohn/naken_asm.git && \
git fetch --depth=1 origin b6e83f1976a5fa0b1a371bd4d6db935a386b95ef && \
git checkout -q FETCH_HEAD) && \
git init -q java_grinder && \
(cd java_grinder && \
git remote add origin https://github.com/mikeakohn/java_grinder && \
git fetch --depth=1 origin 4dca222bae458766c320f045c015754aa6c17376 && \
git checkout -q FETCH_HEAD) && \
cd naken_asm && \
git checkout b6e83f1976a5fa0b1a371bd4d6db935a386b95ef && \
./configure && \
make CC="ccache gcc" && \
make install && \
cd ../java_grinder && \
git checkout 4dca222bae458766c320f045c015754aa6c17376 && \
make -j CC="ccache gcc" CXX="ccache g++" && \
make java && \
(cd samples/trs80_coco && make -j grind) && \
Expand All @@ -177,9 +208,11 @@ RUN --mount=type=cache,target=/root/.ccache,sharing=shared \

# Install tasm6801
FROM foundation AS tasm
RUN git clone https://github.com/gregdionne/tasm6801.git && \
RUN git init -q tasm6801 && \
cd tasm6801 && \
git checkout 0820625bf8e78053ced348a3d747191d54e5e24f && \
git remote add origin https://github.com/gregdionne/tasm6801.git && \
git fetch --depth=1 origin 0820625bf8e78053ced348a3d747191d54e5e24f && \
git checkout -q FETCH_HEAD && \
cd src && \
make -j && \
mkdir -p /staging/usr/local/bin && \
Expand All @@ -188,9 +221,11 @@ RUN git clone https://github.com/gregdionne/tasm6801.git && \

# Install mcbasic
FROM foundation AS mcbasic
RUN git clone https://github.com/gregdionne/mcbasic.git && \
RUN git init -q mcbasic && \
cd mcbasic && \
git checkout 1030ec4413df400e07709a9aabffcaaf4772eb82 && \
git remote add origin https://github.com/gregdionne/mcbasic.git && \
git fetch --depth=1 origin 1030ec4413df400e07709a9aabffcaaf4772eb82 && \
git checkout -q FETCH_HEAD && \
make -j && \
mkdir -p /staging/usr/local/bin && \
cp mcbasic /staging/usr/local/bin && \
Expand All @@ -199,30 +234,34 @@ RUN git clone https://github.com/gregdionne/mcbasic.git && \
# Install CMOC
FROM foundation AS cmoc
RUN --mount=type=cache,target=/root/.ccache,sharing=shared \
curl -LO http://sarrazip.com/dev/cmoc-0.1.98.tar.gz && \
tar -zxpvf cmoc-0.1.98.tar.gz && \
cd cmoc-0.1.98 && \
curl -LO http://sarrazip.com/dev/cmoc-0.1.99.tar.gz && \
tar -zxpvf cmoc-0.1.99.tar.gz && \
cd cmoc-0.1.99 && \
./configure CC="ccache gcc" CXX="ccache g++" && \
make && \
make install DESTDIR=/staging && \
cd /root && rm -rf cmoc-0.1.98 cmoc-0.1.98.tar.gz
cd /root && rm -rf cmoc-0.1.99 cmoc-0.1.99.tar.gz

# Build and install BASIC-To-6809
#
# Upstream keeps every historical release binary in the tree, so even a depth-1
# checkout is ~1.9GB. Fetch blobs lazily (--filter=blob:none) and sparse-check
# out just the manual and this architecture's zip, which pulls ~43MB instead.
FROM foundation AS basto
RUN git clone https://github.com/nowhereman999/BASIC-To-6809.git && \
RUN if [ "$(uname -m)" = "aarch64" ]; then ARCH=arm64; else ARCH=x86_64; fi && \
ZIP="BASIC-To-6809_v5.33_Linux_$ARCH.zip" && \
git init -q BASIC-To-6809 && \
cd BASIC-To-6809 && \
git checkout 0e60e91fae063324fb9608f0117f6e9ac0582125 && \
git remote add origin https://github.com/nowhereman999/BASIC-To-6809.git && \
git config remote.origin.promisor true && \
git config remote.origin.partialclonefilter blob:none && \
git sparse-checkout set --no-cone /Manual.pdf "/Binary_Versions/$ZIP" && \
git fetch --depth=1 --filter=blob:none origin 1fd2f46923e61b60f54a16e5b317a633eaa43c80 && \
git checkout -q FETCH_HEAD && \
mkdir -p /staging/usr/local/share/doc && \
cp Manual.pdf /staging/usr/local/share/doc/basto6809.pdf && \
cd Binary_Versions && \
if [ "$(uname -m)" = "aarch64" ]; then \
unzip BASIC-To-6809_v5.28_Linux_arm64.zip -d /tmp/basto6809 && \
mv /tmp/basto6809/BASIC-To-6809_Linux_arm64 /staging/usr/local/share/basto6809; \
else \
unzip BASIC-To-6809_v5.28_Linux_x86_64.zip -d /tmp/basto6809 && \
mv /tmp/basto6809/BASIC-To-6809_Linux_x86_64 /staging/usr/local/share/basto6809; \
fi && \
mv "/staging/usr/local/share/basto6809/BasTo6809.2.Compile copy" "/staging/usr/local/share/basto6809/BasTo6809.2.Compile" && \
unzip -q "Binary_Versions/$ZIP" -d /tmp/basto6809 && \
mv "/tmp/basto6809/BASIC-To-6809_Linux_$ARCH" /staging/usr/local/share/basto6809 && \
chmod -R o+rx /staging/usr/local/share/basto6809 && \
cd /root && rm -rf BASIC-To-6809 /tmp/basto6809

Expand Down Expand Up @@ -270,6 +309,7 @@ RUN --mount=type=cache,target=/root/.ccache,sharing=shared \
# ===========================================================================
FROM foundation AS final

COPY --from=toolshed /staging/ /
COPY --from=preproc /staging/ /
COPY --from=zx0 /staging/ /
COPY --from=salvador /staging/ /
Expand Down
36 changes: 30 additions & 6 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,14 +5,15 @@ IMAGE := jamieleecho/coco-dev
VERSION := $(shell sed -nE 's/.*"version" *: *"([^"]+)".*/\1/p' package.json)
TAG ?= $(IMAGE):$(VERSION)
SHELLCHECK_IMAGE := koalaman/shellcheck-alpine:stable
HADOLINT_IMAGE := hadolint/hadolint:v2.15.1-alpine

help:
@echo "coco-dev Makefile targets:"
@echo ""
@echo " make build Build the docker image and tag as $(TAG)"
@echo " make test Run smoke tests inside the built image"
@echo " make shell Drop into a one-off bash shell in the image"
@echo " make lint Run shellcheck on the shell scripts"
@echo " make lint Run shellcheck on the shell scripts + hadolint on the Dockerfile"
@echo " make size Print the size of the built image"
@echo " make push Push the image to Docker Hub"
@echo " make clean Remove the local image"
Expand All @@ -26,25 +27,46 @@ build:

test:
docker run --rm -v "$(CURDIR)/tests:/sources:ro" $(TAG) bash -euc '\
echo "[1/5] java_grinder -> .bin"; \
echo "[1/10] java_grinder -> .bin"; \
work=$$(mktemp -d); cp /sources/java_grinder/* $$work/; cd $$work; \
javac Hello.java; \
java_grinder Hello.class Hello.asm trs80_coco; \
naken_asm -l -type bin -o Hello.bin Hello.asm; \
test -s Hello.bin; \
echo "[2/5] basto6809todsk -> .DSK"; \
echo "[2/10] basto6809todsk -> .DSK"; \
work=$$(mktemp -d); cp /sources/basto6809/* $$work/; cd $$work; \
basto6809todsk HELLO.BAS; \
test -s HELLO.DSK; \
echo "[3/5] mcbasic -> .c10"; \
echo "[3/10] mcbasic -> .c10"; \
work=$$(mktemp -d); cp /sources/mcbasic/* $$work/; cd $$work; \
mcbasic MC10HELLO.BAS; \
test -s MC10HELLO.c10; \
echo "[4/5] cmoc --os9 -> OS-9 module"; \
echo "[4/10] cmoc --os9 -> OS-9 module"; \
work=$$(mktemp -d); cp /sources/cmoc-os9/* $$work/; cd $$work; \
cmoc --os9 hello.c; \
test -s hello; \
echo "[5/5] mame coco3 driver present + headless"; \
echo "[5/10] tasm6801 -> .c10"; \
work=$$(mktemp -d); cp /sources/tasm6801/* $$work/; cd $$work; \
tasm6801 -obj -c10 HELLO.asm; \
test -s HELLO.c10; \
test -s HELLO.obj; \
echo "[6/10] zx0 -> dzx0 round trip"; \
work=$$(mktemp -d); cd $$work; \
i=0; while [ $$i -lt 200 ]; do printf "HELLO COCO "; i=$$((i+1)); done > orig.bin; \
zx0 orig.bin packed.zx0 >/dev/null; \
dzx0 packed.zx0 unpacked.bin >/dev/null; \
cmp orig.bin unpacked.bin; \
test "$$(wc -c <packed.zx0)" -lt "$$(wc -c <orig.bin)"; \
echo "[7/10] salvador -> dzx0 cross-check"; \
salvador -c orig.bin packed.sal >/dev/null; \
dzx0 packed.sal unpacked2.bin >/dev/null; \
cmp orig.bin unpacked2.bin; \
echo "[8/10] decbpp minifies DECB BASIC"; \
printf "print \"HELLO COCO\"\nend\n" | decbpp 2>/dev/null | grep -q "^1PRINT.*:END$$"; \
echo "[9/10] nitros9 defs installed for lwasm"; \
test -s /usr/local/share/lwasm/os9.d; \
test -s /usr/local/share/lwasm/rbf.d; \
echo "[10/10] mame coco3 driver present + headless"; \
mame -listfull coco3 | grep -qw coco3; \
mame -validate coco3 >/dev/null; \
echo "All smoke tests passed."'
Expand All @@ -55,6 +77,8 @@ shell:
lint:
docker run --rm -v "$(CURDIR):/work" -w /work $(SHELLCHECK_IMAGE) \
shellcheck coco-dev utils/basto6809todsk
docker run --rm -v "$(CURDIR):/work" -w /work $(HADOLINT_IMAGE) \
hadolint Dockerfile

size:
@bytes=$$(docker image inspect --format='{{.Size}}' $(TAG)); \
Expand Down
16 changes: 10 additions & 6 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,8 @@ Color Computer](https://en.wikipedia.org/wiki/TRS-80_Color_Computer)
applications. It implements a Docker image that includes the following tools:

* CoCo Languages and Libraries
* [BasTo6809 V5.28](https://github.com/nowhereman999/BASIC-To-6809)
* [CMOC 0.1.98](http://sarrazip.com/dev/cmoc.html)
* [BasTo6809 V5.33](https://github.com/nowhereman999/BASIC-To-6809)
* [CMOC 0.1.99](http://sarrazip.com/dev/cmoc.html)
* [Java Grinder](http://www.mikekohn.net/micro/java_grinder.php)
* [LWTOOLS 4.24](http://lwtools.projects.l-w.ca)
* [naken](http://www.mikekohn.net/micro/naken_asm.php)
Expand Down Expand Up @@ -110,12 +110,16 @@ make build

Run `make help` to see the available targets. After building, `make test`
runs a quick smoke test that exercises CMOC, BasTo6809, mcbasic, Java
Grinder, and the CoCo 3 MAME build against the built image.
Grinder, tasm6801, ZX0/salvador, decbpp, the nitros9 defs, and the CoCo 3
MAME build against the built image. `make lint` runs shellcheck over the
shell scripts and hadolint over the Dockerfile.

The image is a multi-stage build: a shared `foundation` stage (apt packages,
the Python venv, lwtools and toolshed) followed by one stage per tool, which
BuildKit compiles in parallel. Compile-heavy stages use a `ccache` cache mount,
so rebuilding unchanged sources locally is fast.
the Python venv and lwtools) followed by one stage per tool, which BuildKit
compiles in parallel. lwtools lives in `foundation` because CMOC's configure
requires `lwasm`; every other tool, toolshed included, is only needed at run
time and so builds as its own parallel stage. Compile-heavy stages use a
`ccache` cache mount, so rebuilding unchanged sources locally is fast.

Building MAME from source is the slow part of the image build. A few of MAME's
core source files need ~2 GB of RAM each in the compiler, so the job count is
Expand Down
Loading
Loading