feat(schema): add the generator forma kind - #706
Merged
Conversation
Introduces Generator as a new top-level forma kind alongside Stacks, Targets, Resources, and Policies: a source of a generated value that secrets will later reference. It belongs to a stack (like a resource) but has no target, NativeID, provider, or Read, and is referenced by other nodes rather than attached to a stack (like a policy). This slice only declares and round-trips the kind through all five ingestion surfaces (the forma union, FormaRender, the JSON schema plugin, and the extraction generator) plus the Go model. Nothing schedules, resolves, or rotates a generated value yet. Adds the first concrete generator, PasswordGenerator, with eval-time validation so an unsatisfiable spec (every character class disabled, or excludeCharacters emptying an enabled class) fails at PKL eval rather than during unattended rejection sampling later.
Add the generators table (sqlite, postgres, mssql migrations), the Datastore CreateGenerator/UpdateGenerator/DeleteGenerator/GetGenerator/ LoadGeneratorsByStack methods, and a SQLite implementation. A generator is always owned by exactly one stack: unlike a policy it has no standalone form, so the table carries stack_id NOT NULL and there is no junction table or attach/detach. Identity is the row's KSUID, stable across an update found by (label, stack), so a rename does not read as delete-then-create. The behavioural test suite lives in internal/datastore/dstest as suite_generators.go, following the existing RunAll pattern, so Postgres, Aurora and MSSQL can be held to the same six behaviours once implemented. Those three backends and the hand-rolled test mocks get stub/panic methods for now to keep the Datastore interface satisfied.
… label policies.stack_id holds a resolved stack KSUID; generators.stack_id was storing the stack's label instead, a schema-level mismatch between two sibling tables sharing a column name with different semantics. Add StackID (with GetStackID/SetStackID) to the Generator model alongside the existing label-carrying GetStack/SetStack, mirroring how Policy carries both a label field and a resolved StackID. CreateGenerator and UpdateGenerator now persist gen.GetStackID() into stack_id. GetGenerator/DeleteGenerator/LoadGeneratorsByStack keep their public, label-scoped signatures but resolve the label to its current stack row internally before querying, the same way the datastore already resolves a stack by label elsewhere. The dstest generator suite sets StackID directly from the stack it creates, the same way the policy suite sets StackID on a TTLPolicy -- resolving a label to an ID during a real apply is deferred to the generator_update lifecycle, not part of this change.
…and aurora Replaces the "not yet implemented" generator stubs in the three remaining Datastore backends with real implementations that mirror each backend's existing policy methods: Postgres and MSSQL follow their own CreatePolicy/ UpdatePolicy/DeletePolicy idiom directly, and Aurora gets its own rdsdata-based implementation (it does not delegate to Postgres). Write methods trust the caller's resolved stack KSUID; the read methods resolve a stack label to its KSUID internally, matching how the policy methods already behave. Wires GeneratorIDForTest into each backend's dstest harness so the shared KSUID-stability test runs instead of skipping.
SplitByStack routed a generator-only stack into the split even though the stack had no resources, which the reconcile generator reads as "desired state has zero resources" and deletes every existing resource on that stack. Drop the generator branch from SplitByStack entirely: nothing reads split.Generators, so this is removing speculative surface. Also finishes off other loose ends from the generator-kind slice: require a generator's stack (with actual eval-time enforcement, since a StackResolvable's own fields are all optional and silently default-construct otherwise), cut the unused rotation/RotationSpec/ EverySeconds knob (it evaluated and persisted but nothing ever read it), stop PasswordGenerator.Type from being settable out of band with GetType(), make the generator round-trip test actually evaluate its output, escape excludeCharacters when emitting PKL, and align two datastore error-handling spots with their sibling backends.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a new top-level forma kind,
Generator, alongside Stacks, Targets, Resources and Policies. A generator is a source of generated values that resources will later reference.This branch declares, round-trips and persists. Nothing rotates, schedules or resolves. No scheduler, no reference envelope, no consumer.
A generator is deliberately neither a resource nor a policy. Not a resource: it has no target, no NativeID, no provider and no
Read, so it is never discovered and never drifts — the "targetless resource" shortcut would mean threading a must-not-dispatch exception through the executor, resource updater, discovery and NativeID handling. Not a policy: a policy is attached to a stack, while a generator is referenced by other nodes, which inverts the reference direction. So it reuses policy's storage and lifecycle patterns without being either.It belongs to exactly one stack and may be referenced across stacks, the same way a secret is today.
What is here
Generatorbase andPasswordGenerator, with the character-class vocabulary (length,uppercase/lowercase/digits/symbols,excludeCharacters,requireEachIncludedType) rather than the two knobsrandom.password(length, useSpecial)offers. An unsatisfiable spec — every class flag false, orexcludeCharactersemptying an enabled class — fails at PKL eval, not at runtime.formaunion and theFormaRenderconstruction inschema/forma.pkl,FormaRenderinschema/formae.pkl, the anonymous struct ininternal/schema/json/json.go, andpklGenerator.pklextraction.Generatorinterface andParseGenerator, mirroringParsePolicy's discriminated decode.generatorstable across all four backends (SQLite, Postgres, Aurora, MSSQL), held to one shared behavioural suite ininternal/datastore/dstest.generators.stack_idholds the stack KSUID, matchingpolicies.stack_id. An earlier revision stored the label; two sibling tables sharing a column name with different semantics is a trap, and00009_convert_stack_id_to_ksuid.sqlshows that conversion was a deliberate past decision. Write and read are asymmetric by design and match the Policy idiom exactly:Create/UpdatetrustGetStackID()verbatim, while the read methods take a stack label and resolve it.Two things deliberately cut
rotation/RotationSpec. An earlier revision let an author writerotation { every = 30.d }and have it evaluate, render, persist and round-trip while nothing read it. A rotation knob that silently enforces nothing is a trap: an author sets it and believes rotation is on. It ships with the scheduler that consumes it.Generator routing through
SplitByStack. This one was a live defect, caught in review and fixed here.generateResourceUpdatesForReconcileiterates the split, loads each stack's existing resources and destroys every one not matched bystack.Resources. Before this branch a stack could only enter the split via a resource, so a zero-resource entry was impossible; routing generators created one, and a generator-only forma made reconcile delete that stack's resources — reachable viaformae apply, and the intended cross-stack shape. Nothing readsplit.Generators, so the routing was speculative as well as destructive, and it is removed.TestGenerateResourceUpdatesForReconcile_GeneratorOnlyStackKeepsExistingResourcespins it; the test was confirmed to fail without the fix rather than assumed to.Verification
GOTOOLCHAIN=go1.26.0 go test ./internal/... ./pkg/... -tags unit -count=1, plus all four datastore backends against the shared suite and the PKL integration tests.Two failures in the full run, both confirmed not caused by this branch:
TestAutoReconciler_RetriesFailureAndRevertsOOBDriftSimultaneously— timing-sensitive under full-suite parallel load (11.4s vs 5.5s); passes in isolation.TestLoadResourcesByStack_ExcludesDeletedResourceWhenVersionsMixCase— fails identically on the merge-base with no generator code present, and its own message names it a test precondition:default collation must treat 'U' as greater than 'f'. A property of the local Postgres server, not of this change.Local environment notes for reviewers:
go build ./...fails insidego-json-experiment/jsonunder Go 1.27 because the pinned commit conflicts with 1.27's stdlib jsonv2 aliases —go.modsaysgo 1.26.0and CI uses the go.mod toolchain, soGOTOOLCHAIN=go1.26.0is the local fix.internal/schema/pkltests are behind//go:build integrationand needmake version-semverin the working tree.