Avoid write failures when TiDB auto-embedding provider is unavailable

[mashenjun]

Summary

The scheduled 4-region smoke batch 20260709T000000Z hit a write-path failure in aws-ap-southeast-1: a normal drive9 fs cp in layer-fs-smoke-test.sh failed because TiDB Cloud Inference could not generate a semantic embedding through Bedrock.

This makes a basic file write depend on the availability of the embedding provider. The local CLI surfaces the error as storage/backend unavailable even though the failing backend operation is semantic insertion.

Reproduction / evidence

Run context:

• Region: aws-ap-southeast-1
• Endpoint: https://aws-ap-southeast-1.drive9.ai
• Run id: 20260709T000000Z
• Branch/commit used by smoke runner: smk/issue-676-e2e-tidbcloud-native / ca7e0796046110eb17d98590ce9a4c75206c4dd0
• Local log: /home/ubuntu/.cache/drive9-smoke/scheduled/aws-ap-southeast-1/20260709T000000Z/smoke-all.log
• Failed suite: layer-fs
• Tenant: 5497522f-36d5-4529-bf1e-2d8362d0aa7b
• Failed path: /layer-smoke-1783555796/rename-src.txt

Local smoke excerpt:

registered cleanup: suite=layer-fs kind=live tenant_id=5497522f-36d5-4529-bf1e-2d8362d0aa7b
PASS POST /v1/provision returns 202 (got=202)
PASS provision response contains api_key
PASS tenant eventually becomes active (got=active)
PASS drive9 binary ready
fs cp: storage backend unavailable; contact support
FAIL [layer-fs] (rc=1)

Clinic app-log evidence for the same tenant/request:

2026-07-09T00:10:07Z error server_event write_failed tenant_id=5497522f-36d5-4529-bf1e-2d8362d0aa7b path=/layer-smoke-1783555796/rename-src.txt error="insert semantic: Error 1105 (HY000): TiDB Cloud Inference: failed to generate embedding: Bedrock is unable to process your request."

Cleanup completed for the run: delete_failed=0, net_remaining=0; no retained resources.

Impact

When the TiDB auto-embedding provider is transiently unavailable, ordinary filesystem writes can fail with HTTP 500. This can break write-heavy workflows and makes smoke health depend on Bedrock/TiDB Inference availability rather than only Drive9 storage/metadata correctness.

Suspected code path

The write path inserts semantic content for the file revision. In database-managed auto-embedding mode, the semantic.embedding generated column invokes TiDB Cloud Inference. If that generated embedding call fails, the insert fails and propagates back through handleWrite as a user-visible write failure.

Relevant areas to inspect:

• pkg/backend/dat9.go write/upsert paths that populate semantic content
• tenant schema/profile handling for database-managed auto-embedding
• the async embedding docs/code path already moving embedding generation into durable app-managed tasks

Suggested direction

Make write success independent from embedding-provider availability. Possible directions:

• use app-managed async embedding for hosted smoke/prod tenants and keep embedding failures in semantic_tasks retry/dead-letter state;
• or disable database-managed auto-embedding for tenants where writes must remain available during provider outages;
• ensure grep/search degrades when embeddings are missing or delayed instead of making write APIs fail.

Validation

1. Re-run e2e/layer-fs-smoke-test.sh against aws-ap-southeast-1 with the same TiDB Cloud native provisioning mode.
2. Inject or simulate embedding-provider failure and verify ordinary fs cp / PUT /v1/fs/* still returns success while semantic indexing is retried asynchronously or skipped according to config.
3. Confirm cleanup still reports delete_failed=0 and net_remaining=0.