fix: record Gemara source provenance in pack artifacts

[trevor-vaughan]

Summary

Resolves declared gemara.sources during complypack pack and writes per-policy provenance (imported catalogs and guidance references) into the OCI config blob. Packs with no declared sources are unaffected.

Key changes:

• Config model redesign — Config.Source is now []Provenance, each carrying a PolicyID and []GemaraRef (ReferenceID, URI, Version). This is a breaking change to the config blob JSON schema.
• Provenance pipeline — new BuildProvenance function in internal/pipeline maps resolved policies to deterministically ordered provenance records (sorted by PolicyID, then ReferenceID) for byte-identical config blobs.
• URI sanitization (CWE-200) — credentials, query strings, and fragments are stripped from URIs before recording into the published blob. Local file:// paths are omitted entirely.
• Credential redaction in errors (CWE-209) — RedactCredentials in internal/registry strips userinfo from source references in error messages. loadFileArtifacts redacts both the wrapper message and the wrapped os.PathError.
• Batch error reporting — LoadAndResolve collects all load/merge failures into one joined error naming every offending source, instead of failing on the first.
• 5-minute context timeout (CWE-400) — bounds aggregate resolution of all declared Gemara sources so a slow or hostile source cannot hang the command.
• --cache-dir flag — for headless or restricted environments where HOME is unset.
• README — documents gemara.sources config, pack flags, and source provenance blob format.

Related Issues

• Closes bug: complypack.Config.Source provenance is never populated #221

Review Hints

• Start with pkg/complypack/config.go to see the new Provenance / GemaraRef types and validation, then internal/pipeline/provenance.go for BuildProvenance and sanitizeURI.

• internal/registry/client.go has RedactCredentials and splitScheme — the credential stripping heuristic uses last-@-before-first-/ to handle passwords containing @ while preserving digest @sha256: references. Worth checking that the edge cases in TestRedactCredentials cover your mental model.

• cmd/complypack/cli/pack_provenance_test.go has the end-to-end credential-leak test: it pushes a pack with a credentialed mapping-reference URL and asserts the published config blob contains neither the password nor the query string.

• The batch-error rework in internal/pipeline/pipeline.go changed the error format from "failed to load artifacts from ..." to "source ...: ...", which required updating assertions in internal/mcp/server_test.go and acceptance/mcp_server_test.go.

• Review all commits together — they build on each other as a single logical change.

[github-actions]

✅ CRAP Load Analysis: PASS (no baseline)

No baseline file found at .gaze/baseline.json. Showing current scores without regression detection.

How to Enable Regression Detection

Generate and commit a baseline file to track CRAP score changes over time:

# 1. Install gaze
go install github.com/unbound-force/gaze/cmd/gaze@latest

# 2. Run tests and generate baseline
go test -coverprofile=coverage.out ./...
mkdir -p .gaze
gaze crap --format=json --coverprofile=coverage.out ./... > .gaze/baseline.json

# 3. Commit the baseline
git add .gaze/baseline.json
git commit -m "chore: add CRAP baseline for regression detection"

For more information:

• Gaze README
• CI Integration Guide

Summary

Metric	Value
Functions analysed	134
Avg complexity	4.6
Avg line coverage	79.4%
Avg CRAP score	14.7
CRAPload (>= 15)	14
Avg contract coverage	81.8%
Avg GazeCRAP score	6.1
GazeCRAPload (>= 15)	2

View full analysis logs

[jpower432]

🤖 LLM-assisted

issue (blocking): BuildProvenance doesn't distinguish bundled OCI sources from other source types.

In go-gemara, looking at bundle/assemble.go: when Gemara assembles a bundle, it fetches transitive dependencies from MappingReference.Url at assembly time and packs them as layers.

By the time complypack resolves a pre-assembled OCI bundle, those URLs were already consumed -- the authoritative provenance is the bundle's OCI reference + digest, not the mapping reference URLs which describe a past assembly step.

For file sources and unbundled/live resolution, recording MappingReference.Url is reasonable.

For pre-assembled OCI bundles specifically, the provenance should record the bundle's OCI reference.

thought: For reference, this intersects with an open Gemara question about whether bundles should be resolved at pack time or fetch time.

[yvonnedevlinrh]

Review Summary

Overall this is a well-structured PR — the architecture follows the thin-transport/domain-logic split, security defense is layered at three levels (URI sanitization, credential redaction, error redaction), and the test suite is comprehensive with 20+ cases covering determinism, cardinality, and edge cases. CI and local pre-flight checks pass.

Requesting changes on one functional bug and one test gap that masks it. Two additional low-severity items noted inline.

Findings

#	Severity	Location	Description
1	High	provenance.go:39	BuildProvenance returns non-nil empty slice, causing "source":[] in config blob instead of omitting the key (contradicts README)
2	Medium	pack_provenance_test.go:166	assert.Empty masks #1 — should be assert.Nil; missing test for sources-declared-but-zero-policies → JSON serialization path
3	Low	pipeline.go:53-55	policySources map silently overwrites on duplicate policy ID across sources; deterministic but undocumented
4	Low	client.go:89-100	RedactCredentials doc claims generic scheme handling but splitScheme rejects digit-leading prefixes, passing credentials through; limitation is tested but not documented

What looks good

• Deterministic output: sort-by-PolicyID + sort-by-ReferenceID, verified with 20-iteration fuzz test
• CWE-200/CWE-209 defense in depth across sanitizeURI, RedactCredentials, and redactPathError
• CWE-400 timeout bound on source resolution (packResolveTimeout)
• Backward-compatible LoadResult.PolicySources addition — existing callers unaffected
• Config.Validate() loop correctly no-ops on nil/empty Source
• Addressed prior review feedback (slog migration, SanitizeSourceID removal)

[yvonnedevlinrh]

LGTM - No blockers. All change requests from both me and Jenn have been addressed

Only one medium finding that currently has zero impact today

Findings

1. policySources populated before Merge — latent stale-entry risk

Severity: Medium | Category: Correctness
File: internal/pipeline/pipeline.go:56-62

policySources[id] = src is populated inside the LoadArtifacts loop before loaded.Merge() runs. If Merge fails, policySources contains entries from the failing source, but since LoadAndResolve returns nil on any error (line 72), the stale map is never consumed. No runtime bug today, but if the error handling is ever relaxed to partial results, this ordering would silently include policies from a failed merge.

[trevor-vaughan]

@jpower432 Rebased and your items have been addressed when you get a chance.

@yvonnedevlinrh Addressed your comment since I had to rebase anyway, thanks

[yvonnedevlinrh]

LGTM

[jpower432]

LGTM