Registry cache provenance

[kaathewisegit]

Is there some kind of provenance mechanism for the cached artifacts from the registry?

brioche install --registry downloads files from the registry, but I wasn't able to figure out how they are validated. I think that Brioche needs some kind of provenance mechanism for them, if there isn't one yet.

From what I've seen, most classical package managers use signatures. Alpine, for example, ships with public keys of the core team in /etc/apk/key/. And it seems that the NixOS binary caches also uses signing keys. The other approach is reproducible builds, with a hash of the outputs in brioche.lock. But that'd be much more challenging, as it would have to be done on package by package basis

[kylewlacy]

This is a great question! Would definitely love to hear ideas on the right direction to go here!

(Brief thing about reproducible builds: I think they are valuable, but I feel they're insufficient to solve the problem of provenance. Even if a build is reproducible, the only way for the client to validate that the cache's output matches a reproducible build's output is to just run the build again. So, as long as build caching is a thing, we still need to consider the cache a trust boundary, even for reproducible builds. Reproducibility makes it possible to audit if the cache is corrupted or compromised-- but only reactively, not proactively)

I think a signature-based scheme would mesh better with Brioche's design, for a lot of the same reasons I think it's the right choice for Nix. Basically, each baked recipe in the remote cache should be signed, and the client should validate this signature before trusting it.

...but beyond that, I've had a lot of ideas but nothing concrete 😅 A few random thoughts:

• The most naive option would be to just use a single, fixed signing key. Each build machine would be given access to this key to sign artifacts.
  • Personally, I don't see this as much of an improvement over the guarantees we get with HTTPS/TLS-- it's better, but only marginally. This defends against domain hijacking at least, but still has a single credential, which is either trusted or untrusted as a whole.
• Even better would be to use a certificate chain. There'd be one root CA key, and each build machine would be given its own machine key signed by the CA. We could even get more fine-grained and use the machine key to sign separate temporary build keys.
  • I like this model a lot better because it means we could limit the blast radius of a compromise. Even if all build machines were found to be compromised, we could just revoke the impacted keys and provision new ones with the CA key.
  • Separate build keys also means we could have some strong guarantees about the build metadata itself, like when it was built and which machine did the build. This is potentially very valuable for auditability!
  • ...but this is also a lot more work to manage! I've only scraped the surface of managing CA key-signing stuff before, but the tooling seems pretty baroque.
• I'm also very interested in Sigstore, which feels like it's designed for exactly the problem.
  • Sigstore is used for GitHub's artifact attestation features. We could even piggy-back off that, but honestly I'm doing everything in my power to make it so GitHub isn't a hard dependency on Brioche's infrastructure (which is also why the install script pulls releases from https://releases.brioche.dev instead of https://github.com)
  • I'm still pretty fuzzy on how Sigstore would actually fit-in overall though. My understanding is that Sigstore's idea is roughly "translate OIDC login token into a short-term signing key". I can't actually tell if that's any better for us than just managing our own root CA certificate though... (well, we would at least get a certificate transparency log for free, at least)
• I also vaguely know about The Update Framework. I've tried to wrap my head around it a few times, but I can't say it's ever clicked for me... just wanted to mention it anyway though!
• Our infrastructure is pretty scrappy. Luckily, TPMs are ubiquitous nowadays, so that might be a nice way to deal with signing keys. I assume more "enterprise-y" systems have an HSM or something? I wonder if there are any resources for how different projects securely manage signing keys...?

[kaathewisegit]

Yeah, it's a complex topic. I myself am an amateur when it comes to this, so I don't think I can weight in on any of the options :D

I also haven't heard about TUF or Sigstore before, so I'll look into them. And I think it'd make sense to look into classical Linux distributions such as Debian or Alpine, because they had to solve a very similar problem.

On the note of reproducible builds, I agree that they are not enough, but I feel that it'd be cool to have first-class support for them. Maybe something like this:

type Hashes = Record<string, std.Sha256Hash>

async function reproducible(recipe: std.Recipe, hashes: Hashes): std.Recipe {
	for (const [path, hash] of Object.entries(hashes)) {
		const file = std.castToFile(recipe.get(path))
		const data = await file.readBytes()
		const dataHash = calculateSha256(data)

		std.assert(hash.hex === dataHash.hex)
	}

	return recipe
}

export default async function build(): std.Recipe<std.File> {
	let source = Brioche.glob("Cargo.toml", "Cargo.lock", "src/main.rs")
	const cargo = cargoBuild({ source })

	return reproducible(
		cargo,
		{
			"bin/b": std.sha256Hash("dab8ba18b17ff6b049553768bf1e876fe0f74ca6db2971a867386a38d44db379"),
			// ... other files
		}
	)
}

So that the author of the package can specify the hashes, moving the trust to the registry. An implementation such as this can be put into std (except for the sha256 calculator, opened #382 for that). Or maybe the hashes can be stored in brioche.lock with a brioche cli subcommand which can scan the output and automatically populate it. Though, having configuration options in packages would complicate that, as there'd have to be a hash set for every option combination

[kylewlacy]

@kaathewisegit definitely like the idea of having a reproducible function! Although to me, I think this might work better as a built-in recipe rather than handling it in userspace, for a few reasons:

1. Calling e.g. file.readBytes() a bunch is going to be a bit of a performance challenge. Each call will need to jump from JS -> Rust -> JS, and unfortunately that can get costly. Probably not an issue as a one-off, but becomes more challenging the more recipes we have that use this pattern over time!
2. Pushing stuff from userspace down into recipes gives us more flexibility. Example: we've talked about a brioche plan command before, with the goal that it could show you everything that would need to be (re)built. With this function, for example, we couldn't plan anything "after" a userspace reproducible recipe, since the JS can't make progress until after the calculateSha256 promise resolves. More context here: Condition recipes #359 (comment)

I think a good analogy is the std.castTo* family of recipes. reproducible feels similar, since it just passes through an inner recipe, but just makes an assertion on the result

[kylewlacy]

...actually, I realized we could do this in userspace but without requiring an assert today already! Something like this should work (not tested):

async function reproducible(recipe: std.Recipe, hashes: Hashes): std.Recipe {
  return std.runBash`
    # If the files in the output match the expected hashes, return success and
    # leave the recipe as-is. Otherwise, fail with an error
    sha256sum -c "$hashes"
  `
    .dependencies(jq)
    .outputScaffold(recipe)
    .env({ hashes: std.file(buildSha256HashesFile(hashes)) })
}

function buildSha256HashesFile(hashes: Hashes): string {
  // TODO: Take `hashes`, and convert it to a string that `sha256 -c` expects, e.g.:
  return "bin/a  a1b2c3d4...\nbin/b  deadbeef...."
}

(here I used bash, but this seems like we could use a std.process directly, or maybe using a custom small binary to do the heavy lifting to give better error messages or something?)

There are some downsides with this approach but I think we have some ways to mitigate them:

1. The recipe is different per-platform. I think this ties into Cross-compilation support #302 (comment), i.e. we'll generally need some way to mark a recipe as platform-agnostic
2. Both the "inner" and "outer" recipes will be synced to the cache. That's because we implicitly treat each process recipe as if it's expensive. I think it'd be nice if we had some way to mark a process as "cheap", meaning that it shouldn't be synced to the cache because it can be cheaply re-run based only on its inputs. Then again, this isn't a huge deal, since the data from both recipes would be de-duplicated

[kylewlacy]

Also, wanted to split this off as a separate thread...

Mostly I've thought about signatures for the cache-- this is the most important trust boundary IMO, since it'd be a juicy target for an attacker to poison cache outputs, and it'd be extremely hard to validate if the cache is fully truthful or not (and I'm not sure it's possible to tell a "truthful-but-corrupted cache"-- say a build machine with failing RAM-- from an intentionally malicious cache).

But there's a second trust boundary too: which is the registry's mapping from a package name to the project source (e.g. in brioche install -r curl, the registry maps curl to a set of .bri files that are then evaluated by the client). The client trusts that the registry is actually returning .bri files for a specific curl build.

I feel the registry is just a smaller degree of trust compared to the cache. .bri files are human-readable source, and it's easy and cheap (in theory at least!) to compare the project files returned from the registry to the upstream source from the brioche-packages repo. If an attacker compromised the registry, say, then you could just read the code for what the malicious curl build is doing. Even if the code were subtle and obfuscated, we could still spot the difference because Brioche represents a project with all its dependencies too, and we can compare the hashes of the malicious curl to the upstream one.

So my gut instinct is that TLS/HTTPS is probably sufficient for the level of trust we need for the registry.

[kaathewisegit]

I saw disagreement about this online, but I myself fully agree here. A lot of package managers were created before HTTPS because widespread, so they needed signatures. Alpine, for example, ships with http:// links to the apk repository mirrors by default. But if Brioche enforces HTTPS for all connections, that becomes redundant