Handle stuck provisioning forks in us-west-2 smoke cleanup

[mashenjun]

Background

A live smoke-all run against https://aws-us-west-2.drive9.ai reproduced a new CLI fork provisioning failure. The ordinary live tenants for the same endpoint became active and most suites passed, but the CLI ctx fork tenant stayed in provisioning long enough for the smoke script to fail. Cleanup then could not delete the fork because the server rejects DELETE /v1/fork while the fork tenant is still provisioning, and the parent tenant cannot be deleted while the non-deleted fork exists.

This is separate from the known layer-fs mode failure tracked in #684.

Reproduction

From branch smk/issue-676-e2e-tidbcloud-native commit cd8cdae:

DRIVE9_BASE=https://aws-us-west-2.drive9.ai \
DRIVE9_E2E_TMPDIR=$HOME/.cache/drive9-smoke/tmp \
DRIVE9_E2E_CLEANUP=always \
CLI_SOURCE=build \
bash e2e/smoke-all.sh

The scheduled wrapper run was:

bash scripts/run-drive9-smoke-scheduled.sh \
  20260707T125402Z \
  https://aws-us-west-2.drive9.ai \
  aws-us-west-2

Evidence

Log path on the smoke runner:

/home/ubuntu/.cache/drive9-smoke/scheduled/aws-us-west-2/20260707T125402Z/smoke-all.log

Suite summary:

api: 107/107 passed
cli: failed
journal: 21/21 passed
layer-fs: 98/100, known #684 mode failures
fuse: 103/103 passed
posix-permission: 32/32 passed
smoke-all: PASS=4 FAIL=2
exit_code=1

CLI fork failure excerpt:

registered cleanup: suite=cli kind=fork tenant_id=601ce771-344f-4385-8239-7c2dee866a47
PASS ctx fork returns api_key
PASS ctx fork returns tenant_id
PASS ctx fork initial status is provisioning (got=provisioning)
fork-status=200:provisioning
... repeated ...
FAIL fork tenant becomes active (want=active got=provisioning)
fs cp: HTTP 503:
FAIL [cli] (rc=1)

Sanitized status check after the run still showed:

fork tenant_id=601ce771-344f-4385-8239-7c2dee866a47 http=200 status_or_error=provisioning
parent tenant_id=1115159c-b014-4adc-be39-2d3389763fb3 http=200 status_or_error=active

Cleanup failure excerpt:

cleanup delete failed: suite=cli kind=fork tenant_id=601ce771-344f-4385-8239-7c2dee866a47 http=503 reason=tenant is provisioning
cleanup delete failed: suite=cli kind=live tenant_id=1115159c-b014-4adc-be39-2d3389763fb3 http=409 reason=tenant has non-deleted forks
cleanup: created_live=6 created_fork=1 delete_failed=2 net_remaining=2
cleanup incomplete; registry remains pending: /home/ubuntu/.cache/drive9-smoke/cleanup-pending/drive9-e2e-20260707T125415Z-145943.jsonl

An immediate cleanup-pending.sh retry returned the same 503 tenant is provisioning for the fork and 409 tenant has non-deleted forks for the parent.

Impact

• ctx fork can leave a fork tenant stuck in provisioning, making the fork API key unusable (fs cp returned HTTP 503).
• The cleanup path cannot delete that provisioning fork, so scheduled smoke leaves both the fork and parent CLI tenant pending deletion.
• Parent tenant cleanup is blocked by the non-deleted fork, increasing leak risk for scheduled production smoke runs.

Root-cause analysis / suspicious code path

There appear to be two related problems:

1. Fork provisioning can stay in provisioning for at least the CLI smoke polling window on aws-us-west-2, while other live tenant provisioning on the same endpoint succeeds. The relevant server path is pkg/server/fork.go:

   • createForkTenant creates a fork tenant in meta.TenantProvisioning.
   • provisionForkTenantAsyncWithProvision retries for forkProvisionRetryWindow (30 minutes) before marking the fork failed.
   • ensureForkBranchConnection / WaitForBranchUserWithCredentials or schema initialization are likely places to inspect for the stuck state.

2. The delete path rejects cleanup while the fork is still provisioning. In pkg/server/auth.go, handleDeleteNoAcquireAuth special-cases /v1/fork but still returns 503 tenant is provisioning for meta.TenantProvisioning. As a result pkg/server/fork.go:handleForkDelete never gets a chance to mark the fork deleting or clean up branch metadata. Then parent tenant deletion fails with tenant has non-deleted forks.

Suggested fix direction

• Investigate server logs for fork_provision_retry, fork_provision_failed, fork_provision_ok, and TiDBCloud branch/user wait errors for tenant 601ce771-344f-4385-8239-7c2dee866a47 on the aws-us-west-2 deployment.
• Make fork deletion safe for stuck provisioning forks. Options:
  • allow DELETE /v1/fork through auth for TenantProvisioning, then let handleForkDelete mark the fork deleting and clean up/persist branch deletion as appropriate;
  • or add an explicit cancel/delete path for provisioning forks that marks the fork failed/deleting and unblocks parent tenant deletion.
• Add a regression test around deleting a TenantKindFork in TenantProvisioning via /v1/fork, including the parent-deletion unblock behavior.
• Consider surfacing provisioning retry/failure reason in /v1/status for fork tenants so smoke reports can distinguish TiDBCloud branch delay, branch-user delay, and schema init failure.

Validation plan

• Unit test: DELETE /v1/fork with a provisioning fork should return 202 and move toward deleting/deleted or a retryable cleanup state instead of being blocked by auth.
• Unit/integration test: parent tenant delete succeeds after the stuck fork cleanup path completes.
• Live validation: rerun smoke-all against https://aws-us-west-2.drive9.ai; expected CLI fork smoke reaches active or, on failure, cleanup returns delete_failed=0 net_remaining=0.

[mashenjun]

Targeted fork-only rerun on us-west-2 reproduced the same behavior.

Run:

• endpoint: https://aws-us-west-2.drive9.ai
• run_ts: 20260707T131117Z
• log: /home/ubuntu/.cache/drive9-smoke/scheduled/aws-us-west-2/fork-20260707T131117Z/fork-smoke.log
• branch/commit: smk/issue-676-e2e-tidbcloud-native / cd8cdae

Result:

• parent tenant became active
• ctx fork returned api_key and tenant_id
• fork tenant stayed provisioning for the full 120s polling window
• fork fs cp failed with HTTP 503
• fork fs cat failed with tenant is provisioning
• drive9 delete fork failed with HTTP 503: tenant is provisioning
• cleanup could not delete the fork or parent live tenant

Relevant ids from this rerun:

• parent live tenant: 3d066c93-f384-4b5e-b385-9f6150e64735
• fork tenant: 52594e17-d8d7-4ae5-9377-1c032a0d90c5
• pending cleanup registry: /home/ubuntu/.cache/drive9-smoke/cleanup-pending/drive9-e2e-20260707T131117Z-153678.jsonl

This narrows the failure to the fork path itself rather than a later broad cli-smoke side effect.

[mashenjun]

Additional smoke evidence from ad hoc 4-region batch 20260708T162548Z.

us-west-2 result:

• Endpoint: https://aws-us-west-2.drive9.ai
• Run log: /home/ubuntu/.cache/drive9-smoke/scheduled/aws-us-west-2/20260708T162548Z/smoke-all.log
• Cleanup summary: /home/ubuntu/.cache/drive9-smoke/cleanup-completed/drive9-e2e-20260708T165731Z-240996.summary.json
• CLI parent tenant: 6743952b-bdde-43ee-9c88-a6dbd1a251bd
• CLI fork tenant: c263d6ae-7e08-4c31-a045-09a35c9254c9

Local symptom:

• ctx fork returned a fork tenant and API key.
• The fork stayed provisioning through the CLI wait loop.
• Fork-context fs cp failed with HTTP 503.

Different from the original retained-resource case: cleanup later succeeded in this run. The fork and parent deletes were accepted and verified, with delete_failed=0 and net_remaining=0.

This points at a late fork-readiness path that can outlive the current smoke wait window; cleanup handling was sufficient once the fork became deletable.