feat(gcp): add 24 types across eight services, seven of them new - #198
Merged
Conversation
A mesh, the four route kinds that attach to it, an endpoint policy and a
service load balancing policy. This is the direct consumer of the Network
Security policies added earlier on this branch: an endpoint policy's
clientTlsPolicy, serverTlsPolicy and authorizationPolicy all point at them.
Every collection answers 200 at both locations/global and locations/{region},
and they are separate namespaces - a mesh created globally is absent from the
regional list and 404s on a regional GET. A wrong guess creates unfindable
resources rather than failing, so the scope is pinned in a map with that
reasoning rather than inferred from the target.
Reference fields are full paths on the wire and short names in a forma, and
the short form is refused, so both halves of the translation exist with the
identity pinned in unit tests.
A PATCH here validates the resource from the request body alone rather than
merging over stored state, so an update omitting type, hostnames or rules is
refused as though the stored value did not exist. Those fields are
non-optional and always sent. This API also clears labels on any patch whose
mask does not name them, and omits false booleans the proto3 way so a declared
false reads back absent; both are documented at the fields.
Each route kind avoids a billable prerequisite differently: HTTP redirects,
gRPC injects a fault, TCP uses its original destination, and TLS - the one
kind with no destination-free action - points at a backend service with no
backends. gateways is absent from all four because a gateway allocates Envoy
proxies and bills, so the field could not be verified. ServiceBinding is
absent because its only functional field is deprecated in the API's own
discovery document.
Target, DeliveryPipeline, CustomTargetType, DeployPolicy and Automation - the declarative half of Cloud Deploy. Releases, rollouts and job runs are deliberately absent: a release renders through Cloud Build and a rollout actually deploys, so both cost money, and neither is a desired state rather than a record of an action already taken. This is where polling earns its keep. A create accepted with HTTP 200 came back as a done operation carrying an error - a stage naming its target by full path is refused with "is not a valid resource ID for resource type stage.targetId" - and the pipeline never existed. The API also serialises operations per resource and answers a second concurrent mutation with 409 ABORTED. The reference forms are mixed, which is the trap: stage.targetId must be a short id and rejects a full path, while customTarget.customTargetType is a full path. So a pipeline stage passes its resolvable straight through while Target expands and shortens. executionConfigs is not declarable. Cloud Deploy fills it in whether or not it was sent and rewrites what was sent, adding artifactStorage, executionTimeout and a defaultPool mirror - nested too deep for a schema hint to tolerate and too meaningful to strip. An automation's rules come back carrying an output-only condition two levels down, which is stripped instead. Discovery needs no walking provisioner: Cloud Deploy accepts "-" as a wildcard delivery pipeline.
Workflow invocations and compilation results are absent: both execute against BigQuery. Verified rather than assumed that the shipped types do not - a workflow config with no cronSchedule left both collections empty, and this project cannot schedule automatic releases at all. The whole API is synchronous; no operations collection exists in it. Left on the async path a create would report in-progress with an empty request id and then poll the bare base URL forever. Three fields are immutable in a way nothing in the discovery document says, and immutable by field mask rather than by value: a PATCH whose mask names invocationConfig, codeCompilationConfig or kmsKeyName is refused with "update_mask contains immutable fields" even when the object sent is byte-identical to the stored one. Since the mask is built from the body, all three leave update bodies - that is the difference between a working timeZone change and a rejected one. internalMetadata, a bookkeeping blob that changes on every write, is stripped from all four types, along with containingFolder, which can only be changed through a custom verb and so could never be reconciled. A workspace has no patch method at all, so it replaces. The schema module is repository_dataform.pkl: verify-schema rejects duplicate basenames across services and artifactregistry already ships repository.pkl.
Every mutating call in the Firestore Admin API answers with a long-running operation and two of the three cannot be polled. The operation a PATCH names answers 404 "Operation does not exist" on a patch that plainly applied, and the one a DELETE names carries the deleted database under "response" but never sets "done" - not false, absent - so an async poller waits forever on a database already gone. Create needs no polling either: it returns the finished database inline in about a second. So the type is registered synchronous and reads the resource back after an update. Three fields the API volunteers are dropped rather than defaulted. etag is recomputed on every read of an untouched database - three consecutive GETs gave three etags with an identical updateTime. earliestVersionTime moves continuously. realtimeUpdatesMode is reported for every database and refused on create for Standard-edition ones, so declaring it would put a value in state that fails the create it came from. enhancedTextSearchQueryMode is dropped for a different reason: the API returns it on every database and the v1 discovery document does not mention it. deleteProtectionState is declared explicitly in the fixture because a database created with protection enabled cannot be deleted at all, and every run would leak one. A deleted id is held for about five minutes, so the case exercises update rather than replace. Index is absent: its name is server-assigned and the API ignores a caller-supplied one, so a declared name could never round-trip, and an index build on an empty collection did not complete in five minutes of polling. BackupSchedule is absent because retained backup data bills. The schema module is database_firestore.pkl: verify-schema rejects duplicate basenames and sql already ships database.pkl.
Global only, and that is a limit of the API rather than a simplification. Parameter Manager serves each region from its own host, parametermanager.<region>.rep.googleapis.com, and serves locations/global only from the plain host. Cross the two and it answers 403 PERMISSION_DENIED, "Read access to project was denied" - a wrong-host error wearing a missing-IAM- grant's clothes, which is worth knowing before anyone responds to it by granting a role. base.APIConfig.BaseURL is one constant string, so the location segment is pinned to global and a target configured for a region cannot walk into that 403. What base would need to support the regional host is written down where the next person will look. A version's payload is stripped from every response, and that is a security control rather than a drift fix. The payload is user data and may be a secret, and unlike Secret Manager - whose API withholds secret material - Parameter Manager hands it straight back, both on a GET defaulting to view=FULL and in the create response. So data is declared write-only, the payload is removed on read, and versions:render, which would additionally resolve Secret Manager references inside it, is never called from a read path. The whole API is synchronous. disabled is ignored on create - a create sending true answers 200 and the version is enabled anyway - so it carries a provider default and takes effect only through a patch. A parameter with versions refuses to delete and its delete takes no force flag, so the declared reference is also what orders the teardown.
The attestor's Grafeas note lives in the Container Analysis API, which this
plugin does not ship, and that turns out not to matter: Binary Authorization
does not resolve noteReference at create time, so an attestor pointing at a
note that does not exist - in a project where Container Analysis is not even
enabled - is accepted. That is what makes the type declarable with no
prerequisite in someone else's API.
A public key sent without an id comes back with one the API computed from the
key's DER digest, so id is required rather than defaulted.
asciiArmoredPgpPublicKey is absent from the schema for the harder version of
the same problem: with a PGP key the API overwrites id with the key's
fingerprint, so declared and stored could never agree. Update is a PUT - this
API has no patch, no update mask and no long-running operation anywhere - so
omitting an optional field really does clear it.
PlatformPolicy is why the project-wide projects/{project}/policy singleton is
not modelled: it offers the same expressive power as a real, deletable
collection, where the singleton has only a get and a PUT and a "create" would
mean mutating live admission policy. The platform is a URL segment rather than
a field: v1 exposes only gke, and a field with exactly one legal value the API
never echoes back is an input-only field that reads as drift. ListItemsKey is
set because the response keys its array platformPolicies while the collection
segment is policies.
An API key is the first type here whose creation mints a secret, and the
create call is the only call that returns it: keyString arrives inline in the
completed operation and nowhere else. It is dropped on every read path rather
than merely expected to be absent, the same treatment
compute.vpnTunnels.sharedSecret gets, and getKeyString is never called from a
read. There is no field to declare opaque - the value has no authored
counterpart, only something to refuse to store.
Every key response reports name in project-number form while the target
declares a project id, so the native ID is rebuilt from context or the same
key would be managed under one identity and discovered under another. And
delete is a soft delete: the key stops working and drops out of listings, but
a read keeps answering 200 for thirty days with deleteTime set, so
ReadTreatAsMissing turns such a read into NotFound. Without it every sync
inside that month would put a deleted key back into inventory. That window
also reserves the id, so the case exercises update rather than replace.
Creating a trigger starts nothing, which was checked rather than assumed: the
project's builds collection was empty before the first probe trigger and still
empty after nine creations and seven patches. The fixture is doubly inert -
disabled, and with a sourceToBuild URL as its only source, so no webhook, no
Pub/Sub and no repo connection exists that could fire it.
Three behaviours compensated for. The trigger methods are synchronous: create
and patch answer with the trigger and delete with an empty body, unlike this
API's own builds.create which answers with an Operation. It authorizes a PATCH
against the request body's resourceName rather than the URL's, so a replayed
one turns an in-project update into a 403 on another project's path, and that
field is stripped from requests. And it silently drops false booleans, so
disabled:false comes back absent and an approvalConfig of
{approvalRequired:false} comes back as {}; both are restored on read.
A project with no legacy Cloud Build service account - every project created
in recent years - rejects a trigger that names none with a bare HTTP 400
naming no field. Five different source forms and gcloud's own "builds triggers
create manual" all fail identically, and the cause is the missing
serviceAccount. That opaque 400 is the biggest trap in this API and is
documented in the schema.
WorkerPool is absent: a private pool holds provisioned machines billed per
hour. The v2 Connection and Repository types need an external OAuth account.
…esolvable
Row-level access control on a single table: a SQL boolean predicate deciding
which rows a principal may see, so one shared table can serve each tenant only
its own rows instead of a view per tenant. This was dropped from the previous
batch on a missing bigquery.datasets.create grant, which has since landed.
Its identity is not a name but the composite rowAccessPolicyReference
{projectId,datasetId,tableId,policyId}, which every request body must carry
and which the API checks against the URL segment by segment - "Project ID in
URL path and content do not match." So the four flat properties a forma
declares are assembled on the way out and flattened back on the way in.
Update is a PUT with no field mask, and sending one is not ignored: the API
answers `Unknown name "updateMask": Cannot bind query parameter`. A delete
that would leave the table with no policy is refused outright unless
force=true is sent, so the type always sends it - without that flag every
teardown of a single-policy table fails.
grantees is accepted on insert and returned by neither get nor list, so it is
not declared: an input-only field reads as drift on every sync. It is readable
only through a separate getIamPolicy call, and a PUT that omits it clears the
roles/bigquery.filteredDataViewer binding it created. Both facts are recorded
at the field, because together they mean a formae-managed policy grants row
access to nobody until someone binds that role out of band.
Policies hang off a table and BigQuery accepts no wildcard for the dataset or
table segment - datasets/-/tables/- answers 404 blaming the dataset - so
discovery walks datasets, then tables, then policies rather than reporting an
empty inventory.
Table.datasetId was typed plain String, so dataset.res.datasetId failed
evaluation and a table could only name a dataset some other forma had already
created. The sibling Routine already had this right.
…wrong Three defects that a missing conformance case had been hiding, plus the case. cloudkms keyRings.delete did not exist when GCP::KMS::KeyRing was written, and the plugin said so in a comment, a doc comment and a unit test - so a forma could create a key ring and never reclaim it, and the fixture was dropped for exactly that reason. The method exists now: it answers with an already- finished Operation, the ring 404s on the very next GET, and the deleted id can be re-used. The case is back with a -replace companion, since there is still no keyRings.patch and the id is the only declarable field. Nine key rings leaked by the nightly before the case was dropped have been reclaimed. regionTargetHttpsProxies.patch enforces the fingerprint rather than treating it as advisory - a PATCH without one is refused with "Required field 'resource.fingerprint' not specified" - so optimistic locking here is not an optimisation, it is the only way an update lands at all, and it was off. Every update of GCP::Compute::RegionTargetHttpsProxy failed with a 400. tlsEarlyData also gains hasProviderDefault: the API reports DISABLED back on every read whether or not it was sent, exactly as the global sibling already documents. regionTargetHttpProxies has no patch method at all - delete, get, insert, list and setUrlMap only - but GCP::Compute::RegionTargetHttpProxy claimed SupportsUpdate, so a PATCH landed on a URL the API does not serve and came back as Google's HTML 404 page rather than an API error. Its global sibling does have patch, which is how the regional one came to claim it. A change now replaces. The region-http-lb case, which uses this type and has no -update companion, is what let it hide; it still passes. EssentialContacts::Contact.email is marked createOnly: the API refuses a masked change to it and the field was annotated as though it were mutable. Inert today because the type does not support update, but it stops the trap.
… creates Blank imports for apikeys, binaryauthorization, cloudbuild, clouddeploy, dataform, firestore, networkservices and parametermanager; README rows and CHANGELOG entries for the 24 new types; a disposition row for each new hasProviderDefault annotation. Eight new sweep sections, all REST because none of these collections has a usable gcloud surface, and all eight sharing one helper rather than eight near-identical copies - the copies are exactly how the earlier sweeps drifted apart. Every list URL was dry-run against the live API before being wired in. Ordering is encoded in the order of the lists, not left to chance: a Cloud Deploy pipeline refuses to delete while it has an automation, a Dataform repository while it has workspaces or configs, a Parameter Manager parameter while it has versions and its delete takes no force flag, and a Network Services mesh goes after the routes attached to it. Nested collections are reached through each API's own "-" wildcard parent. API keys get half a sweep, and it says so. A key delete is a soft delete: the key stops working and drops out of listings, but a read keeps answering 200 for thirty days and the v2 API has no purge, only undelete. So a tombstoned key cannot be collected by anything and every conformance run leaves one behind until it self-purges - free and non-functional, but recorded so nobody goes looking for the sweep that is missing. What is collected is a live key a run created and failed to delete, which is the case that would otherwise leave a working credential lying around.
verify-schema rejects duplicate basenames across services, so this batch's dataform/repository.pkl and firestore/database.pkl were renamed to repository_dataform.pkl and database_firestore.pkl - the convention instance_spanner.pkl and table_bigquery.pkl already follow. Pkl derives a module's identifier from its filename when the import carries no alias, so renaming the file silently renamed the identifier and every `repository.Repository` / `database.Database` reference stopped resolving: "Cannot find module import `database`". Both now carry an explicit `as` alias, exactly as testdata/spanner-database-replace.pkl already does. Neither `pkl eval formae-plugin.pkl` nor verify-schema evaluates a fixture, so both gates passed while all ten of these cases failed at conformance step 1. Evaluating every fixture directly is what catches this.
… create Two defects that only a conformance run surfaces, plus the base hook one of them needed. databases.list takes no pagination parameters at all. It returns every database in the project in one response and refuses the ones it does not know: "?pageSize=100" answers 400 "page_size is not supported." and "?maxResults=100" answers 400 Unknown name "maxResults". The whole request is refused rather than the parameter ignored, so the registration's PageSizeParam turned every List into an error instead of a list - the type was undiscoverable while create, read, update and delete all worked. Pagination is now disabled for this API. Firestore also answers a patch issued while a create is still settling with 409 ABORTED, "There are concurrent database changes, please try again." That is deterministic rather than occasional - a create followed immediately by a patch reproduces it every time - and it is the exact shape of a reconcile, so the conformance update step failed on a database that was perfectly healthy. base already mapped a retryable error to NotStabilized so formae core re-runs the operation, but only on the delete path, where Cloud SQL needed it. The same check now guards both update paths, and Firestore classifies the 409. This is not Firestore-specific: Cloud Deploy answers a concurrent mutation with 409 ABORTED too, and any API that asks for a retry was previously reported as a terminal failure.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #196 — targets
naxty/gcpDepthBatch, so the diff here is this batch only.Merge #196 first.
24 new resource types across eight services, seven of them entirely new to the plugin, plus
four defect fixes on types that already shipped. Every candidate was probed against the live API
before any code was written, by nine agents working in parallel; everything that survived was then
run through conformance serially.
Conformance: 25 cases, 50 runs, all green
Run locally one case at a time after
make install,TIMEOUT=12. "CRUD" is the full lifecycle:apply, verify, extract, force-reconcile, update, destroy, re-apply, out-of-band delete, sync.
region-http-lb, for the RegionTargetHttpProxy changeNothing in this PR is unverified.
Four defects on already-shipping types
Each was found by writing the missing conformance case — which is the point of writing them.
GCP::KMS::KeyRingcould be created and never destroyed.keyRings.deletedid not exist whenthe type was written; the plugin said so in a comment, a doc comment and a unit test, and the
fixture had been dropped for exactly that reason. The method exists now and works. Nine key
rings leaked by the nightly before the case was dropped have been reclaimed.
GCP::Compute::RegionTargetHttpsProxycould never be updated. That collection enforces thepatch fingerprint rather than treating it as advisory —
Required field 'resource.fingerprint' not specified— and optimistic locking was off, so every update failed with a 400.GCP::Compute::RegionTargetHttpProxyclaimed an update it does not have. Nopatchmethodexists; a PATCH landed on a URL the API does not serve and came back as Google's HTML 404 page.
GCP::BigQuery::Table.datasetIdrejected a resolvable, so a table could only name a datasetsome other forma had already created.
Two
basechanges, both driven by a live failureResourceConfig.ReadTreatAsMissing(added in feat(gcp): add 37 resource types across thirteen services #196) — for an API that keeps serving atombstone after a 200 delete.
OperationConfig.RetryableErrornow guards the update path, not only delete. Firestoreanswers a patch issued while a create is still settling with 409
ABORTED, "There are concurrentdatabase changes, please try again" — deterministic, and the exact shape of a reconcile. It was
reported as a terminal failure. Cloud Deploy returns 409
ABORTEDon concurrent mutations too,so this is a class, not one API.
Behaviours the probes caught that would otherwise have shipped broken
databases.listtakes no pagination parameters and refusesthe ones it does not know —
?pageSize=100→ 400page_size is not supported.The whole requestis refused, so every List returned an error instead of a list while create, read, update and
delete all worked. Only a discovery run finds that.
answers 403
PERMISSION_DENIED, "Read access to project was denied" — a wrong-host errorwearing a missing-IAM-grant's clothes, worth knowing before anyone responds by granting a role.
The types are registered global-only, and what
basewould need for the regional host is writtendown.
invocationConfig,codeCompilationConfigorkmsKeyNameeven when the object sent isbyte-identical, and its own discovery document calls all three plain "Optional".
operation carrying an error, and the pipeline never existed.
builds triggers create manualall fail identically with a 400 naming no field; the cause is a missing
serviceAccount, whichevery project created in recent years needs.
grantees(and a PUT that omits it clears the IAM binding it created), Cloud Build'sincludedFiles, Firestore'stags.Secrets
Two types mint or carry credentials, and neither puts one in state.
ApiKeys::Key—keyStringarrives inline in the create response and nowhere else. It isdropped on every read path rather than merely expected to be absent, and
getKeyStringis nevercalled from a read.
ParameterManager::ParameterVersion— unlike Secret Manager, this API hands the payloadstraight back on a GET.
datais write-only, the payload is stripped on read, andversions:renderis never called from a read path.Sweeps
Eight new sections, all REST because none of these collections has a usable gcloud surface, sharing
one helper rather than eight near-identical copies — the copies are how the earlier sweeps drifted
apart. Every list URL was dry-run against the live API, and the sweep logic was then run live and
observed collecting the real leftovers a conformance run leaves (prerequisites
Destroyspares):Cloud Deploy pipelines and targets, Dataform repositories, a Parameter Manager parameter, four
Network Services meshes. Child-before-parent ordering is encoded in the list order and was observed
working.
API keys get half a sweep, and it says so. A key delete is a soft delete: the key stops
working and drops out of listings, but a read keeps answering 200 for thirty days, and the v2 API
has no purge — only undelete. A tombstoned key cannot be collected by anything, so every run leaves
one behind until it self-purges. Free and non-functional, but recorded so nobody goes looking for
the sweep that is missing. Live keys are collected.
Dropped after probing
ServiceBinding(its only functional field is deprecated in the API's own discovery document) ·Cloud Deploy
releases/rollouts/jobRuns(a release renders through Cloud Build, a rolloutdeploys; neither is a desired state) · Dataform
workflowInvocations/compilationResults(executeagainst BigQuery) · Binary Authorization's
projects/{p}/policysingleton (no create, no delete;PlatformPolicygives the same power as a real collection) · FirestoreIndex(server-assignedname that ignores a caller-supplied one; a build on an empty collection did not finish in five
minutes) and
BackupSchedule(retained backup data bills) · Cloud BuildWorkerPool(provisionedmachines, billed hourly) and the v2
Connection/Repository(need an external OAuth account) ·Network Services
gateways, the four*Extensions,wasmPlugins,agentGateways,multicast*and
edgeCache*.Of the 15 shipped types that had no conformance case, one is now covered (
KMS::KeyRing) and therest were each confirmed blocked with a reason: not independently creatable, needs a billable
cluster or provisioned capacity, or mutates project-wide state. Two need only a decision:
EssentialContacts::Contactneeds an allowed contact domain, andGKEHub::Featureneedsgkehub.googleapis.comenabled plusroles/gkehub.admin.One thing I got wrong, and how it was caught
verify-schemarejects duplicate schema basenames across services, sodataform/repository.pkland
firestore/database.pklhad to be renamed. Pkl derives a module's identifier from its filenamewhen the import has no alias, so the rename silently broke every
repository.Repositoryreferenceand all ten of those cases failed at conformance step 1 — while
pkl eval formae-plugin.pklandverify-schemaboth passed, because neither evaluates a fixture. Fixed with explicitasaliases,and evaluating every fixture directly is now part of the gate run.
Gates
go build ./...,golangci-lint run ./...(0 issues),make test-unit(0 failures, 50 packages),make verify-schema(243 types, 0 duplicates),pkl eval formae-plugin.pkl,bash -non everytracked
.sh,make lint-reuse(930/930), and every new or changed fixture evaluates.Project verified clean afterwards across every collection this batch touches.