Move from regex/path-based package resolution to hash-based package identification

[absol27]

Background

sbomit currently resolves files in witness attestations to packages using regex patterns on file paths:

• Python: extracts name/version from site-packages/foo-1.2.3.dist-info
• Go: parses /pkg/mod/<module>@<version>/ from the module cache path
• Rust: splits crate-name-1.0.0 on -, which is ambiguous since crate names can contain hyphens

Problems

• Fragile by design - path structure is a convention, but package managers change install layouts across versions
• Ambiguous splits - Rust's hyphen-delimited name-version format has no reliable delimiter
• System packages - files from apt/rpm/apk have no version in their paths at all
• Hash is ignored - witness records SHA256 for every file, but we never use it for lookup, it's our most reliable signal

Goal

Use a file's content hash (already in the attestation) as the primary key for package lookup, falling back to path as a hint. Given sha256:<hash> → return pkg:<ecosystem>/<name>@<version>. This does not touch the network-trace resolver, which already handles HTTP exchange data well. Focus is purely on the file-trace side.

Potential Directions

• Software Heritage - accepts SHA256 lookups, but maps to a blob rather than a package directly. Getting from "file exists in archive" to "file belongs to package@version" requires additional traversal
• deps.dev - dont think it works with individual file hashes, but works for jar, wheels for example
• Registry metadata - do some package manager have a lookup option we can utilize
• We build our own index - I built a apt hash lookup database, which we can extend potentially to other ecosystems

@yongjae354

[yongjae354]

Looks like a solid start; thanks @absol27! I'll work on this.

[yongjae354]

We build our own index - I built a apt hash lookup database, which we can extend potentially to other ecosystems

@absol27 is there somewhere I can take a look at this?

[absol27]

https://github.com/absol27/apt-file2pkg this is basically offline equivalent of the apt-file command

[absol27]

After thinking through the hash lookup approach, here's where it actually helps vs. doesn't

Problems with hash lookup for (Python, Go, Rust)

1. Unchanged files across versions. Most source files don't change between releases. A hash of requests/adapters.py might match 12 different versions of requests. Narrowing to a specific version requires intersecting hashes across multiple files which is expensive to query.

2. Identical files across packages. Vendored code, or some small utility modules can appear verbatim in multiple unrelated packages. A single hash could map to a long list of (package, version) candidates, and disambiguation by path or context isn't always possible.

3. Vendored dependencies. Many packages bundle copies of other libraries. A hash might correctly identify the vendored library but attribute it to the outer package instead.

4. Index maintenance is continuous and expensive. You'd need to pull and hash every file of every version of every package, and keep doing so as new releases land. For PyPI alone (~500K projects) will be a significant ongoing infrastructure burden.

Where hash lookup works imo: system packages (apt/rpm/apk)

• Every file shipped by a package is listed in a published content manifest (Contents-*.gz for apt, filelists.xml.gz for RPM, APKINDEX.tar.gz for Alpine)
• System package files are compiled binaries and hashes are specific to version, arch, and distro
• There's no regex-based fallback for these in SBOMit today: /usr/lib/x86_64-linux-gnu/libssl.so.3, these files currently go unresolved

I already built an offline apt-file2pkg index that proves this works for Debian/Ubuntu. Porting it to BigQuery and extending to RPM-based distros and Alpine would give us attribution for files that currently don't resolve to any package. The index needs to account for different hashes across architectures (amd64 vs arm64), distros (Ubuntu 22.04 vs Debian 12), and compiler flags.

Recommendation

Hash lookup fills the system package attribution gap that regex genuinely can't address, rather than replacing regex in places where it already works well. The highest-ROI starting point is porting apt-file2pkg to BigQuery and extending coverage to RPM and Alpine. Happy to start scoping the schema and ingestion pipeline if this direction makes sense.