feat(targethasher): collapse bzlmod external targets to repo-level hash

[yushan8]

Summary

For legacy WORKSPACE repos, tango collapses external source/generated files to a single //external:repo hash so HashRecursively doesn't visit each file individually. Bzlmod repos (@@repo+...//...) don't have //external: rules, so this collapse was skipped — every external source file got hashed one by one during the DFS. On a python bzlmod repo (2.49M targets, 82% external pip wheels), this made HashRecursively take ~15 minutes single-threaded.

This PR adds the equivalent collapse for bzlmod using Bazel's marker file content hashes from $(output_base)/external/@repo.marker. Each marker file contains a hash of the repository rule's inputs (URL, sha256, patches, etc.) that changes on any version bump — even for repos like protobuf+ whose canonical name stays the same across versions.

How it works

1. After bazel query, read marker files from $(output_base)/external/
2. For each bzlmod external source/generated file, look up its repo's marker hash
3. If found, pre-hash the target with sha1(marker_content_hash) before the DFS
4. If no marker exists, skip collapsing — let HashRecursively hash the actual file content

Why marker files instead of repo name hashing

Bzlmod canonical repo names fall into two categories:

• With content hash suffix (e.g. rules_python++pip+..._a6ebbe51): 10,585 repos, 1.87M files — repo name changes on upgrade
• Without content hash suffix (e.g. protobuf+, grpc+, zlib+): 455 repos, 104K files — repo name stays the same on upgrade

Hashing just the repo name string would miss version bumps for the second category. Marker files solve this because they always reflect the repository rule's inputs.

Benchmark (2.49M targets)

Metric	Before	After
HashRecursively	899s	~120s
Total compute	969s (16.2 min)	~182s (3 min)

Test plan

• Unit tests for bzlmodRepoName, collapseBzlmodExternalTargets (marker lookup, no-marker fallback, excluded regex, fullHashRepos skip)
• End-to-end test: fromProto with excluded bzlmod targets gets empty hash, not marker hash
• Benchmarked on uber-one with --bypass-cache
• Verified target set is identical (2,493,050 targets) between original and collapsed
• Repos without markers fall through to HashRecursively for file content hashing

[xytan0056]

Suggested change

	if err != nil {
	if err != nil || len(repoMarkerHashes) <= 0{

when bzlmodEnable==true, do you think we must require repo hashes to be non-empty? or fallback to hashing external files if repo hash is not found?

[yushan8]

Actually I think there should be a .marker file for every external repo as long as it's in the dependency graph of the repo.
bazel query 'deps(//...all-targets)' should download the external dependencies + their marker files.
Updated HashExternalTargetsBzlmod to return an error if a marker file doesn't exist. I tested it against the python repo and it finished running successfully. This means every external target that is in the target graph will have a marker file associated with it.

[xytan0056]

next PR, let's make these params a struct

[xytan0056]

Suggested change

	func collapseBzlmodExternalTargets(targets map[string]*Target, fullHashRepos set.Set[string], excludedRegex []*regexp.Regexp, repoMarkerHashes map[string][]byte) {
	// TODO: hash dependency/attributes of repo too
	func collapseBzlmodExternalTargets(targets map[string]*Target, fullHashRepos set.Set[string], excludedRegex []*regexp.Regexp, repoMarkerHashes map[string][]byte) {

I just realized that in HashExternalTargets, we needed to call hashRecursively for external repos too because they have have patches. If patches change, the hash of the external repo should change too.
This is done via single_version_override in bzlmod, could you verify bazel query/marker file include that info and we hash that too?

[yushan8]

Good point. Synced offline, the marker file provides the patch file and the hash of the contents.

0550009ebfa63439c1ddfc210c82aded99c7acfa211fa75808937f24ef3ba690
ENV:BAZEL_HTTP_RULES_URLS_AS_DEFAULT_CANONICAL_ID \0
FILE:@@//third_party/patches/rules_proto_grpc_legacy_python_grpc.patch 149f920174b2d81db398ec9d0f8d3a2031b0be577c41c240fabba2538b69e28b

Updated the PR to hash the whole marker file instead of just the first line.