test: minimal opt-in behaviour coverage for Grok on Vertex (xai-vertex), non-blocking

[waynesun09]

What

Add a minimal behaviour scenario that exercises Grok on Vertex through the xai-vertex provider,
modelled on the existing e2e/behaviour/features/runtime/pi.feature, and gated on its own capability
so it never blocks CI.

Why unit tests are not enough here

internal/runtime covers the model-spec normalisation and the rendered command line well, and
#6572 added tests for both. What no unit test can reach is the handoff:

1. the extension actually loading from /usr/local/share/pi-extensions/xai-vertex in the built
   image — pi drops a failed extension silently, so this fails as a generic "model not found";
2. ambient ADC resolving inside the sandbox, against the rewritten external_account config
   from prepare-sandbox-credentials.sh rather than a developer's gcloud login;
3. the three-segment spec surviving runner → pi → Vertex intact.

All three broke at least once while landing #6572, and none produced a failure that named its cause.
(2) in particular shipped as a bug in the plugin's v0.1.0 and was only caught by running it in a real
sandbox — every developer machine passed, because they had a stale credential that a fresh sandbox
does not.

Why it must not gate CI

Grok availability is outside fullsend's control: Model Garden enablement is per-project, and xAI
quota on Vertex is separate from the Anthropic quota the suite already depends on. A quota or
availability hiccup must not red the whole behaviour suite. Gate it the way runtime-pi already is:

@requires:capability:runtime-pi-xai

Skipped unless the runner declares BEHAVIOUR_CAPABILITIES=runtime-pi-xai, so it is opt-in, run
deliberately, and cannot become a blocker.

What it should cost and reuse

Almost nothing new. It reuses the same leased per-repo install, the same E2E_GCP_PROJECT_ID, and
the same WIF credential path the Claude-on-Vertex runs already use — the xai-vertex extension
authenticates through the same ADC, only the project variable differs, and it defaults to the
fleet's Vertex project.

One small Grok call. Grok 4.6 is $2/$6 per million tokens, so a smoke-sized run is cents — comparable
to the haiku run runtime-pi already costs.

Prerequisite — confirmed. A single GCP project can serve all three providers: Claude, Grok and
Gemini were each run through the pi runtime against one project that has all three enabled in Vertex
Model Garden, on the same ADC credentials, with only the model spec changing. So this scenario needs
no separate project or extra credential plumbing — it needs the E2E project to have xai/grok-4.6
enabled in Model Garden, which is a Model Garden setting rather than a fullsend change.

If for any reason the E2E project cannot have Grok enabled, XAI_VERTEX_PROJECT_ID can point just
this scenario at one that does — #6572 deliberately leaves an explicitly-set value alone rather than
overriding it. Note that if the model is not enabled, the run fails 403 PERMISSION_DENIED and the
extension does not warn, because it did resolve a project — just one without the model.

Suggested assertions

Reuse the pi.feature shape — minimal agent, deliberate tool use — and assert:

• metrics.json has runtime: pi and model xai/grok-4.6 (the publisher segment is retained
  on purpose; a bare grok-4.6 would mean piBareModelID regressed)
• the agent completed with a real response and non-zero token usage, i.e. the provider resolved and
  ADC worked in-sandbox
• a toolCall in the pi session transcript, so the hook adapter is exercised on this provider too
• XAI_API_KEY is unset in the run, so the built-in xai provider cannot shadow the Vertex one

Worth covering both spec forms, since they take different paths through translatePiModel:
harness model: grok-4.6 plus FULLSEND_PI_PROVIDER=xai-vertex (the only form a harness can express,
since validModelName forbids /), and the fully-qualified --model xai-vertex/xai/grok-4.6.

Out of scope

Not proposing xai coverage in the default CI path, nor a second provider in the existing
runtime-pi scenario — that one should keep testing the Claude path it tests today.

[fullsend-ai-triage]

🤖 Finished Triage · ✅ Success · Started 12:48 AM UTC · Completed 12:52 AM UTC

Commit: 6c60f60 · View workflow run →

Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $0.83

[fullsend-ai-triage]

Blocked by open prerequisite PR 6572, which implements the xai-vertex provider and vendors the extension into the sandbox image. The test exercises extension loading, ADC resolution, and model spec routing that only exist once that PR lands.

Once 6572 merges, this issue is ready for implementation. The existing pi.feature provides a clear pattern, the assertions are well-specified, and the capability gating keeps it off the default CI path.

Before enabling the capability on the e2e runner, confirm xai/grok-4.6 is enabled in the E2E GCP project Vertex Model Garden.

Blocked by:

• feat(#6571): add xai-vertex provider to the pi runtime #6572

---

Labels: E2e behaviour test issue for the xai-vertex provider, explicitly opt-in and non-blocking.

[fullsend-ai-triage]

🤖 Finished Triage · ✅ Success · Started 1:37 AM UTC · Completed 1:41 AM UTC

Commit: 84c8bbb · View workflow run →

Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $0.58

[fullsend-ai-triage]

Prerequisite resolved: PR #6572 (xai-vertex provider implementation) has been merged. The blocked label is no longer applicable.

Pre-implementation note: Before enabling BEHAVIOUR_CAPABILITIES=runtime-pi-xai on the e2e runner, confirm xai/grok-4.6 is enabled in the E2E GCP project's Vertex Model Garden — without it, runs fail with 403 PERMISSION_DENIED and the extension gives no warning.

Proposed test outline:

@requires:capability:runtime-pi-xai
Scenario: xai-vertex provider routes through pi with correct model spec
  Given the enrolled test repository
  And the repository runtime is "pi"
  And a custom harness "pi-xai-smoke" with:
    # model: grok-4.6 + FULLSEND_PI_PROVIDER=xai-vertex
  And a pi agent "pi-xai-smoke" defined as:
    # minimal agent with Bash(ls) and Write
  And an issue
  When the issue is labeled "ready-for-pi-xai-smoke"
  Then the harness "pi-xai-smoke" workflow completes successfully
  And the run selected the "pi" runtime
  And the pi session transcript records at least one tool call
  And the run metrics report tokens
  # Additional: verify model field retains publisher segment xai/grok-4.6

Consider a second scenario variant with --model xai-vertex/xai/grok-4.6 to cover both translatePiModel paths.

---

Next steps:

• /fs-code — agent creates a PR to implement this issue
• /fs-code <your instruction> — agent implements with your specific guidance