Official support for Spegel

[awoimbee]

Related to #1963, #3314

What I'd like:

I'd like bottlerocket to completely, officially, support Spegel.

Why

Spegel is currently kinda supported

See comment from Bottlerocket maintainer:

Indeed, it's the compatibility concern with Spegel that's kept this default in place.

Spegel should be supported

Bottlerocket goes hand in hand with karpenter for fast cluster auto-scaling, the exact use-case that Spegel is tailored to.

Currently, using Spegel on bottelrocket is painful

See: spegel-org/spegel#47 (comment)
To make Spegel (and alternatives) work on Bottlerocket, we have to resort to workarounds that can break when we upgrade.

The best workaround (writing to /.bottlerocket/rootfs/etc/containerd/certs.d/_default/hosts.toml) doesn't work on Bottlerocket for EKS 1.34 1.51. Now we can set container-registry.mirrors but we can't configure capabilities anymore.

Alternatives

Personally I use Trow, which works just like Spegel from containerd's perspective, but it can resolve images, so I can do without capabilities.
Still, the config required is messy:

const setupContainerdMirrorScript = outdent`
  #!/bin/sh
  set -xeuo pipefail
  IMDS_TOKEN="$(curl -s -X PUT -H "X-aws-ec2-metadata-token-ttl-seconds: 360" "http://[fd00:ec2::254]/latest/api/token")"
  IP="$(curl -H "X-aws-ec2-metadata-token: $IMDS_TOKEN" http://[fd00:ec2::254]/latest/meta-data/${isIpv6Cluster ? "ipv6" : "local-ipv4"})"
  apiclient set --json "{\\"container-registry\\":{\\"mirrors\\":{\\"*\\":[\\"http://\${IP}:${TROW_PORT}\\"]}}}"
`;
userDataSections.push(outdent`
  [settings.bootstrap-containers.containerd-trow-proxy]
  mode = "always"
  essential = true
  user-data = "${Buffer.from(setupContainerdMirrorScript).toString("base64")}"
`);

(the IP thing is necessary on IPv6 clusters as [::1]:${TROW_PORT} -same for 127.0.0.1- isn't routed to the Trow service)