Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .derived/codebase-index/by-spec/006-fleet.json
Original file line number Diff line number Diff line change
Expand Up @@ -28,5 +28,5 @@
"specStatus": "approved"
},
"schemaVersion": "1.1.0",
"shardHash": "f047fde050b7ef249853c6c497c7a1bbabe267ca679b36528ecf4e47e1301020"
"shardHash": "6fb00aa7d68b4675557a231d91033bd26c1f35cc227736ebc02dcb79f1d4d5b8"
}
Original file line number Diff line number Diff line change
Expand Up @@ -115,5 +115,5 @@
"specStatus": "approved"
},
"schemaVersion": "1.1.0",
"shardHash": "5478dde659fdaf2418cf2ac372aaef462b2d145b85ac9b7282afb0a186afdf35"
"shardHash": "921b56b33fbbc47ab67416fec2c146adf2bc1cf1813d9ee48b153b21f6812354"
}
Original file line number Diff line number Diff line change
Expand Up @@ -28,5 +28,5 @@
"specStatus": "approved"
},
"schemaVersion": "1.1.0",
"shardHash": "488a8a9034c90945a6281eb07cc89fc3b831ce55b761e530da918bdca5a62eb7"
"shardHash": "e66285f43ed7e810579a06a20c094791e6ac1434c56c61be67c173478be45481"
}
5 changes: 3 additions & 2 deletions .derived/spec-registry/by-spec/006-fleet.json
Original file line number Diff line number Diff line change
Expand Up @@ -25,13 +25,14 @@
"5. Out of scope",
"2026-07-20: the addon transferred out; this spec narrowed, not retired",
"Amendment (2026-07-21): spec 011 tenant lifecycle",
"Amendment (2026-07-22): remove-gate confirmation attributes"
"Amendment (2026-07-22): remove-gate confirmation attributes",
"Amendment (2026-07-23): deploy-chosen container port"
],
"specPath": "specs/006-fleet/spec.md",
"status": "approved",
"summary": "Milestone M3: operate stamped apps. The unit of placement is \"one EnRaHiTu container + one volume + one ingress\" on the existing hetzner-k3s cluster. deployd-api-rs (the OAP-era Rust K8s orchestrator, axum + hiqlite) donates its orchestration core as a napi-rs addon (the hiqlite-native pattern): the axum HTTP layer disappears, the K8s knowledge stays. A fleet/ Encore service exposes deploy / status / update / backup over the addon. Done-when is a fleet of ten stamped apps on one box with update and backup exercised.\n",
"title": "Fleet: deployd's core as an in-process addon, placing EnRaHiTu apps"
},
"shardHash": "2519cf1793b565a51af9968b30333967442bc1e027055e6a5fa352659724fad7",
"shardHash": "67db401155879258649c91fd1e408de84c840ddaa8223f2d4e31692f4ebc4539",
"specVersion": "1.1.0"
}
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,6 @@
"summary": "Stand the control plane up on the spec 010 cluster as a platform-grade K8s deployment at app.statecraft.ing. Rewritten ground-up 2026-07-19 to the two-plane thesis (001 section 3), which unpauses it. The rewrite inverts the deploy's job: the published image is embedded-rauthy and self-seeds its own identity on first boot, so the shared-rauthy secret set is void, five of the eleven Encore secrets are discarded by the entrypoint if injected, and the /data volume becomes the identity anchor of the platform. What the deploy still owes the container: nine real secrets, the non-secret env, a Postgres URL, an ingress, and one seeder pass for the only thing first boot cannot seed (the upstream GitHub provider, which rauthy has no declarative bootstrap for). Operator surfaces gate on the custom statecraft_operator role, seeded here; rauthy_admin stays break-glass. Live bring-up is a human checkpoint.\n",
"title": "The control-plane deploy: one governed container on the statecraft cluster"
},
"shardHash": "8a779d39ed43b1e0ef827c010b36a9786c59a82a334a8e914a505c072178b1dd",
"shardHash": "5becd8699f22edf419ce219bc450f8db5626707821e7073ea8785a2af538c78d",
"specVersion": "1.1.0"
}
5 changes: 3 additions & 2 deletions .derived/spec-registry/by-spec/010-statecraft-cluster.json
Original file line number Diff line number Diff line change
Expand Up @@ -31,13 +31,14 @@
"6. Human checkpoints",
"7. Out of scope",
"Amendment (2026-07-22): the RAUTHY_API_KEY catalog delta",
"Amendment (2026-07-22): spec 012 frontend-admin adoption, the scrape lands"
"Amendment (2026-07-22): spec 012 frontend-admin adoption, the scrape lands",
"Amendment (2026-07-23): FLEET_IMAGE_PULL_SECRET is a name, not a credential"
],
"specPath": "specs/010-statecraft-cluster/spec.md",
"status": "approved",
"summary": "The statecraft-owned hetzner-k3s cluster, reconciled by Flux from an in-repo GitOps tree, with one documented secret source that generates the operator `.env.example` and the SOPS-encrypted secrets Flux decrypts in-cluster. Rewritten ground-up 2026-07-19 to the two-plane thesis (001 §3): identity and observability moved inside the control-plane container, so the cluster keeps only what a container cannot do for itself. The standalone cluster rauthy is retired (embedded rauthy is THE platform IdP) and cluster Grafana is dropped with its OIDC client (platform observability is the in-substrate flag-gated admin dashboard); Prometheus is kept, demoted to an unexposed in-cluster metrics sink. What stands: the Flux tree, SOPS, cert-manager, ingress-nginx, reflector, Postgres, NSQ, and Hetzner Object Storage. The cluster is live (PRs #27-#29); this rewrite is the first change that prunes services from it. Amended 2026-07-20 on explicit operator authorization, closing spec 009 checkpoint 1: the claim that every `RAUTHY_*` key survives the move into the container is corrected to the verified image behavior, which self-seeds its own identity. The catalog goes from 47 keys to 33, `auth.<DOMAIN>` is settled as not returning, and the keys a deploy genuinely owes the container are named.\n",
"title": "The statecraft cluster: the substrate beneath the control-plane container"
},
"shardHash": "5f1d8fc39f38a8d5ef07773e6405fb69f6d72b6c5e70a22e38d0e7e5c0a238ac",
"shardHash": "c169aeed543fd75420f2d4e95ee6272b4b04142778bf8e0b4654f272c25c0ba2",
"specVersion": "1.1.0"
}
31 changes: 27 additions & 4 deletions backend/fleet/api.ts
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ import { backupTarget, fleetBaseDomain, fleetImagePullSecret } from "./config";
import type { FleetApp, FleetAppStatus } from "./entities";
import { gateOrDeny } from "./gate";
import * as native from "./native";
import { isValidAppName } from "./ops";
import { FLEET_DEFAULT_PORT, isValidAppName, isValidPort } from "./ops";
import {
createApp,
finishOp,
Expand All @@ -38,6 +38,7 @@ export interface FleetAppView {
namespace: string;
image: string;
volumeSize: number;
port: number;
host: string;
status: string;
createdAt: string;
Expand All @@ -53,6 +54,7 @@ function toView(a: FleetApp): FleetAppView {
namespace: a.namespace,
image: a.image,
volumeSize: a.volumeSize,
port: a.port,
host: a.host,
status: a.status,
createdAt: a.createdAt.toISOString(),
Expand Down Expand Up @@ -99,6 +101,15 @@ interface DeployRequest {
name: string;
image: string;
volumeSize?: number;
/**
* Container port the image serves (default 4000, the addon's default;
* integer 1024-65535, privileged ports rejected because placed pods run
* non-root). enrahitu chassis images are fixed on 8080, so placing one
* requires passing it here; the probes, Service, and Ingress all key off
* it. Immutable after deploy: update forwards the persisted value, so
* changing it means remove + redeploy.
*/
port?: number;
stampJobId?: string;
}

Expand All @@ -109,7 +120,7 @@ interface DeployRequest {
*/
export const deploy = api(
{ expose: true, auth: true, method: "POST", path: "/api/v1/tenants/:id/fleet" },
async ({ id, name, image, volumeSize, stampJobId }: DeployRequest): Promise<FleetAppView> => {
async ({ id, name, image, volumeSize, port, stampJobId }: DeployRequest): Promise<FleetAppView> => {
const auth = getAuthData()!;
const tenant = await authorizeTenant(id, principalFrom(auth), "write");
if (!tenant) throw APIError.notFound("tenant not found");
Expand All @@ -128,6 +139,12 @@ export const deploy = api(
const namespace = namespaceFor(id);
const host = `${appName}.${domain}`;
const size = volumeSize && volumeSize > 0 ? volumeSize : 1;
const appPort = port ?? FLEET_DEFAULT_PORT;
if (!isValidPort(appPort)) {
throw APIError.invalidArgument(
"port must be an integer between 1024 and 65535 (placed pods run non-root and cannot bind privileged ports)",
);
}
const pullSecret = fleetImagePullSecret();

const gated = await gateOrDeny("deploy", { tenantId: id, app: appName, image: img }, "soft");
Expand All @@ -138,6 +155,7 @@ export const deploy = api(
namespace,
image: img,
volumeSize: size,
port: appPort,
host,
});
const op = await startOp(app.id, "deploy");
Expand All @@ -149,6 +167,7 @@ export const deploy = api(
image: img,
host,
volumeSizeGi: size,
port: appPort,
...(pullSecret ? { imagePullSecret: pullSecret } : {}),
});
const ok = status.status === "running";
Expand All @@ -157,7 +176,7 @@ export const deploy = api(
host: status.host || host,
});
await finishOp(op.id, ok ? "succeeded" : "failed", status.message ?? null);
await record("deploy", app, auth.userID, { tenantId: id, app: appName, namespace, image: img, host }, gated.configHash);
await record("deploy", app, auth.userID, { tenantId: id, app: appName, namespace, image: img, host, port: appPort }, gated.configHash);
logInfo("fleet.deployed", { app: app.id, namespace, host, ok });
return toView((await getApp(app.id))!);
} catch (err) {
Expand Down Expand Up @@ -226,12 +245,16 @@ export const update = api(
image: img,
host: app.host,
volumeSizeGi: app.volumeSize,
// The deploy-chosen port, persisted on the row: update rebuilds the
// Deployment spec, and omitting it would revert probes to the addon
// default (4000) on every image change.
port: app.port,
...(pullSecret ? { imagePullSecret: pullSecret } : {}),
});
const ok = live.status === "running";
await setAppStatus(app.id, ok ? "running" : "failed", { image: img, host: live.host || app.host });
await finishOp(op.id, ok ? "succeeded" : "failed", live.message ?? null);
await record("update", app, auth.userID, { tenantId: app.tenantId, app: app.name, image: img }, gated.configHash);
await record("update", app, auth.userID, { tenantId: app.tenantId, app: app.name, image: img, port: app.port }, gated.configHash);
logInfo("fleet.updated", { app: app.id, image: img, ok });
return toView((await getApp(app.id))!);
} catch (err) {
Expand Down
2 changes: 2 additions & 0 deletions backend/fleet/entities.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,7 @@ for (const arm of arms) {
namespace: `t-${randomUUID()}`,
image: "ghcr.io/acme/app:v1",
volumeSize: 3,
port: 8080,
host: "acme.deployd.xyz",
status: "placing" as const,
});
Expand All @@ -65,6 +66,7 @@ for (const arm of arms) {
const back = await repo.findById(app.id);
expect(back?.image).toBe("ghcr.io/acme/app:v1");
expect(back?.volumeSize).toBe(3);
expect(back?.port).toBe(8080);
expect(back?.status).toBe("placing");
expect(back?.stampJobId).toBeNull();

Expand Down
7 changes: 7 additions & 0 deletions backend/fleet/entities.ts
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,13 @@ export class FleetApp {
@Column() image = "";
/** PVC size in GiB (default 1). */
@Column({ type: "integer" }) volumeSize = 1;
/**
* Container port the app serves; the addon keys PORT env, probes, Service,
* and Ingress off it. Persisted so update rebuilds the Deployment with the
* same port the deploy chose (enrahitu chassis images are fixed on 8080;
* the addon default is 4000).
*/
@Column({ type: "integer" }) port = 4000;
@Column() host = "";
@Column({ index: true }) status: FleetAppStatus = "placing";
@Column({ type: "timestamp" }) createdAt = new Date();
Expand Down
20 changes: 20 additions & 0 deletions backend/fleet/ops.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,12 @@ import { describe, expect, it } from "vitest";

import type { FleetAppStatus } from "./entities";
import {
FLEET_DEFAULT_PORT,
canTransitionApp,
canTransitionOp,
type FleetOpStatus,
isValidAppName,
isValidPort,
} from "./ops";

describe("fleet-op state machine", () => {
Expand Down Expand Up @@ -65,3 +67,21 @@ describe("app name validation (DNS-1123 label)", () => {
}
});
});

describe("container port validation", () => {
it("accepts the unprivileged range boundaries and the defaults in use", () => {
for (const p of [1024, FLEET_DEFAULT_PORT, 8080, 65535]) {
expect(isValidPort(p)).toBe(true);
}
});

it("rejects privileged, out-of-range, and non-integer ports", () => {
for (const p of [0, 1, 80, 443, 1023, 65536, -8080, 8080.5, Number.NaN, Infinity]) {
expect(isValidPort(p)).toBe(false);
}
});

it("treats float-typed integers as integers (JS number semantics)", () => {
expect(isValidPort(8080.0)).toBe(true);
});
});
13 changes: 13 additions & 0 deletions backend/fleet/ops.ts
Original file line number Diff line number Diff line change
Expand Up @@ -60,3 +60,16 @@ export class InvalidAppTransitionError extends Error {
export function isValidAppName(name: string): boolean {
return name.length <= 63 && /^[a-z0-9]([-a-z0-9]*[a-z0-9])?$/.test(name);
}

/** The addon's default container port, applied when deploy names none. */
export const FLEET_DEFAULT_PORT = 4000;

/**
* The deploy-chosen container port. Privileged ports are rejected up front:
* placed pods run as a non-root UID with no NET_BIND_SERVICE, so a port
* below 1024 cannot be bound and would hang the rollout wait on a probe
* aimed at a port nothing can listen on.
*/
export function isValidPort(port: number): boolean {
return Number.isInteger(port) && port >= 1024 && port <= 65535;
}
2 changes: 2 additions & 0 deletions backend/fleet/store.ts
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,7 @@ export interface CreateAppInput {
namespace: string;
image: string;
volumeSize: number;
port: number;
host: string;
}

Expand All @@ -88,6 +89,7 @@ export async function createApp(input: CreateAppInput): Promise<FleetApp> {
namespace: input.namespace,
image: input.image,
volumeSize: input.volumeSize,
port: input.port,
host: input.host,
status: "placing" as FleetAppStatus,
createdAt: now,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -181,6 +181,15 @@ spec:
value: "https://nbg1.your-objectstorage.com"
- name: FLEET_RESTIC_IMAGE
value: "restic/restic:0.17.3"
# The NAME of the dockerconfigjson Secret fleet references as
# imagePullSecrets on placed pods; a resource name, not a
# credential, hence plain env and not the Secret. The Secret
# itself must be operator-provisioned in each tenant namespace:
# fleet's RBAC (rbac.yaml) deliberately grants nothing on
# secrets, so fleet can reference but never create it. Empty
# would mean public images only.
- name: FLEET_IMAGE_PULL_SECRET
value: "ghcr-pull"
- name: FACTORY_TEMPLATE_REPO
value: "statecrafting/enrahitu"
- name: FACTORY_TEMPLATE_REF
Expand Down
12 changes: 4 additions & 8 deletions infra/hetzner/.env.example
Original file line number Diff line number Diff line change
Expand Up @@ -110,6 +110,10 @@ GOOGLE_UPSTREAM_CLIENT_ID=
# required: yes | consumer: seed-rauthy Job, once it converges a second provider; spec 009 section 4.5 specifies only the GitHub one | group: google_upstream (all-or-nothing)
GOOGLE_UPSTREAM_CLIENT_SECRET=

# NAME of the kubernetes.io/dockerconfigjson Secret fleet references as imagePullSecrets on placed pods (ghcr-pull in production). A resource name, not a credential: the Secret itself is operator-provisioned in each tenant namespace, because fleet's RBAC deliberately grants nothing on secrets. Empty means public images only. Until 2026-07-23 this entry claimed to be the dockerconfigjson value itself, which no code ever read that way.
# required: yes | consumer: fleet (imagePullSecrets name on placed pods)
FLEET_IMAGE_PULL_SECRET=

# ==========================================================================
# Generated: minted with crypto/rand or `npm run generate-keys`.
# ==========================================================================
Expand Down Expand Up @@ -165,11 +169,3 @@ RAUTHY_CLIENT_SECRET=
# Rauthy admin API key in the form <name>$<secret>, used by the seeder to create the GitHub upstream provider. Only the provider needs the API: the OIDC client is seeded declaratively at first boot and scopes are not converged at all. Not required yet, because first-boot.mjs does not compose api_keys.json (spec 009 section 4.8 item 3), so rauthy has not handed this value back.
# required: no | consumer: seed-rauthy Job (spec 009 section 4.5); its own Secret, never the app pod
RAUTHY_ADMIN_TOKEN=

# ==========================================================================
# Derived: computed from another value (must equal the formula).
# ==========================================================================

# Base64 dockerconfigjson (derived from GHCR_PAT) that fleet attaches to placed pods so they can pull private GHCR images.
# required: yes | consumer: fleet (image pull)
FLEET_IMAGE_PULL_SECRET=
28 changes: 16 additions & 12 deletions infra/secrets/catalog.toml
Original file line number Diff line number Diff line change
Expand Up @@ -23,9 +23,11 @@
# user-supplied the operator provides it (tokens, domain, upstream creds)
# generated minted with crypto/rand or `npm run generate-keys`
# provider-produced a provider hands it back (GitHub App, rauthy OIDC client)
# derived computed from another value (the sole survivor is the
# dockerconfigjson built from GHCR_PAT; the DOMAIN-derived
# service URLs were dropped, see the amendment below)
# derived computed from another value (no members today: the
# dockerconfigjson built from GHCR_PAT became the NAME of
# an operator-provisioned Secret, see
# FLEET_IMAGE_PULL_SECRET; the DOMAIN-derived service URLs
# were dropped, see the amendment below)
#
# `group` ties optional keys into an all-or-nothing set: if any member is set,
# every member must be set; if none are set, the group is skipped as optional.
Expand Down Expand Up @@ -442,17 +444,19 @@ consumer = "seed-rauthy Job (spec 009 section 4.5); its own Secret, never the ap
description = "Rauthy admin API key in the form <name>$<secret>, used by the seeder to create the GitHub upstream provider. Only the provider needs the API: the OIDC client is seeded declaratively at first boot and scopes are not converged at all. Not required yet, because first-boot.mjs does not compose api_keys.json (spec 009 section 4.8 item 3), so rauthy has not handed this value back."

# ---------------------------------------------------------------------------
# Derived: computed from another value. The operator .env lists them
# explicitly. The two URL formulas that lived here (APP_BASE_URL and
# RAUTHY_URL) are gone with their consumers: no code reads either, and the
# issuer is now same-origin and derived by the entrypoint from
# ENRAHITU_PUBLIC_URL (spec 009 section 2.4).
# Fleet placement config. FLEET_IMAGE_PULL_SECRET sat here as "derived"
# (a dockerconfigjson computed from GHCR_PAT) until 2026-07-23; it is
# actually the NAME of the pre-provisioned pull Secret, per fleet/config.ts.
# The two URL formulas that lived here (APP_BASE_URL and RAUTHY_URL) are
# gone with their consumers: no code reads either, and the issuer is now
# same-origin and derived by the entrypoint from ENRAHITU_PUBLIC_URL
# (spec 009 section 2.4).
# ---------------------------------------------------------------------------

[[key]]
name = "FLEET_IMAGE_PULL_SECRET"
provenance = "derived"
secret = true
provenance = "user-supplied"
secret = false
required = true
consumer = "fleet (image pull)"
description = "Base64 dockerconfigjson (derived from GHCR_PAT) that fleet attaches to placed pods so they can pull private GHCR images."
consumer = "fleet (imagePullSecrets name on placed pods)"
description = "NAME of the kubernetes.io/dockerconfigjson Secret fleet references as imagePullSecrets on placed pods (ghcr-pull in production). A resource name, not a credential: the Secret itself is operator-provisioned in each tenant namespace, because fleet's RBAC deliberately grants nothing on secrets. Empty means public images only. Until 2026-07-23 this entry claimed to be the dockerconfigjson value itself, which no code ever read that way."
Loading
Loading