Skip to content

Use docker.io everywhere, and restore two commits lost from #199 - #201

Merged
wu-sheng merged 3 commits into
masterfrom
restore-and-align-dockerhub
Aug 31, 2026
Merged

Use docker.io everywhere, and restore two commits lost from #199#201
wu-sheng merged 3 commits into
masterfrom
restore-and-align-dockerhub

Conversation

@wu-sheng

Copy link
Copy Markdown
Member

The 401 is explained, and it was ours

The swck repository hit the identical bug and tested it there. The mechanism:

docker login <host> stores the credential under the host you name. helm push then normalises back to Docker Hub's canonical host and looks there, finding nothing.

Both hostnames pull anonymously without complaint — I confirmed the chart's own BanyanDB dependency resolves through oci://docker.io/apache at the identical digest sha256:0955a793…. So the mismatch is invisible until a push with credentials, which is the one operation this repository had never exercised: every release through 4.8.0 was pushed by hand, and CI's Docker Hub path only exists from 5.0.0.

apache/skywalking's publish-docker has always used docker.io with a bare apache/<name> target, and publishes without trouble. INFRA's "configured correctly" was right — this was never a permissions problem.

I should flag that my earlier diagnosis pointed the other way. From "anonymous returns 200" I concluded the 401 meant credentials were sent and refused. That was wrong: the anonymous 200 carries granted=pull only, a reduced scope, so it doesn't discriminate the way I claimed. I opened #200 to settle it with a diagnostic; that's now closed as unnecessary.

Swept all 46 occurrences, not just the workflow

Across 25 files, so the hostname a user copies out of the docs is the one we publish to:

  • .github/workflows/publish-helm.yaml — login and push
  • Makefile — the publish target, the manual fallback
  • chart/skywalking/Chart.yaml — the chart's own dependency repository
  • tools/releasing/release-passed.sh — the ANNOUNCE mail's install command
  • README.md, chart/skywalking/README.md, and every install/upgrade example under docs/

The release-passed.sh one was nearly missed — my first sweep's --include list omitted *.sh, so the mail to announce@apache.org would have advertised a hostname we no longer publish to.

Two commits lost from #199's squash

#199 merged from an earlier head — its stat shows release-passed.sh | 6 +-, but the notes-builder change was ~50 lines. Restored here:

commit state on master before this PR
announce URL uses ${TAG} landed
fetch voted chart from dist/release landed
workflow_dispatch landed
release-notes builder missing
docker.io login missing

So master's publish would still have failed exactly as before.

Verified after the change

check result
helm dep up via oci://docker.io/apache pulls banyandb-helm 0.7.0, identical digest
make package succeeds
packaged chart renders 33 resources, subchart bundled
doc links / menu.yml all resolve
registry-1.docker.io remaining only the comment saying not to use it

Once merged, publishing 5.0.0 is publish-helm → Run workflow → tag: v5.0.0. It fetches the voted artifact from dist/release and pushes it.

The release body was one line: "See .../docs/changes/changes.md". Nobody follows
a link to find out what changed in the release they are already looking at, and
every release before this one carried its notes inline -- 4.7.0 through 4.9.0 all
have the generated "What's Changed" list on the page.

github_release now assembles the body before publishing anything: the curated
"## $VERSION" section from the changelog, followed by GitHub's own generated list
of merged PRs and new contributors. Assembled here rather than passed as
--generate-notes so the exact text is known before it is published.

Two details that only show up by looking at the result. The changelog's links are
relative -- ../upgrade/upgrading.md resolves inside docs/ and 404s on a release
page -- so they are rewritten to the hosted docs for the tag, and all six were
checked to return 200. And the section is read from changes-$VERSION.md when the
next-version PR has already merged, falling back to changes.md when it has not,
because release-passed.sh can run either side of that merge.

A version with no matching section logs a warning and falls back to the old link
rather than failing: this runs after the artifacts are already promoted, so it is
the wrong place to abort.

Verified by running the function itself, not a copy: for 5.0.0 it reproduces the
notes now published on the release page, and for a version with no section it
takes the fallback.
apache/skywalking's publish-docker sets DOCKER_REGISTRY=docker.io and pushes the
bare reference $(HUB)/$(NAME):$(TAG) -- apache/skywalking-oap-server -- and that
repository publishes to Docker Hub from CI without trouble. This one used the
explicit endpoint form on both halves: login to registry-1.docker.io, push to
oci://registry-1.docker.io/apache.

Both hosts serve the same registry, and `helm show chart` reads our own published
4.8.0 through either. What differs is where the credential lands: `docker login
docker.io` stores it under Docker Hub's canonical key, "https://index.docker.io/v1/",
while logging into registry-1.docker.io stores it under that literal host. Only
the first is the key every other apache repository authenticates with.

This is not proven to be the cause of the 401 -- an anonymous token request for
repository:apache/skywalking-helm:pull,push returns 200, so a credential the
client failed to find would have produced an anonymous token rather than a
rejection, which points at authorization on the Docker Hub side. But it is a real
difference from a known-good setup, it costs nothing to remove, and leaving the
org on two different forms is worse than picking the one that is known to work.
This is what broke the 5.0.0 chart publish, and the swck repository hit the same
bug independently.

`docker login <host>` stores the credential under the host you name. `helm push`
then normalises back to Docker Hub's canonical host and looks there, finding
nothing. Both hostnames PULL anonymously without complaint -- the chart's own
BanyanDB dependency resolves through either, verified, same digest -- so the
mismatch stays invisible until a push with credentials. That is the one operation
this repository had never exercised: every release through 4.8.0 was pushed by
hand, and CI's Docker Hub path only existed from 5.0.0.

apache/skywalking's publish-docker has always used docker.io with a bare
apache/<name> push target, and publishes without trouble.

Swept all 46 occurrences across 25 files, not just the workflow, so the hostname
a user copies out of the docs is the one we publish to: the Makefile's `publish`
target, the chart's own dependency repository, README, chart README, and every
install and upgrade example.

Verified after the change: `helm dep up` pulls skywalking-banyandb-helm 0.7.0
through oci://docker.io/apache at the identical digest, `make package` succeeds,
the packaged chart renders 33 resources and bundles the subchart, and the docs
links and menu still resolve.

An earlier comment in this workflow justified registry-1.docker.io on the grounds
that it was simply the explicit endpoint form. That was not a verified claim, and
it read as deliberate when it was not.
@wu-sheng wu-sheng added this to the 5.1.0 milestone Aug 31, 2026
@wu-sheng wu-sheng added the bug Something isn't working label Aug 31, 2026
@wu-sheng
wu-sheng merged commit 74f9c6a into master Aug 31, 2026
5 checks passed
@wu-sheng
wu-sheng deleted the restore-and-align-dockerhub branch August 31, 2026 11:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants