Make default tests non-interactive and gate keystore prompts

[acoliver]

Summary

Make the default test workflow non-interactive so routine local and CI verification does not prompt for the OS keystore password repeatedly. Keystore-specific coverage should still exist, but it should be opt-in instead of running during unrelated changes.

Problem

Some test paths currently exercise keychain or secure-store behavior in a way that can prompt the developer for the macOS Keychain password multiple times during a normal verification run. That is painful when running the full test suite, and it blocks unattended runs when the developer is not present.

Most test runs do not need to verify the OS keystore integration. For unrelated documentation, UI, or business-logic changes, tests should be able to use a non-interactive API key source instead.

A local key file such as:

~/.keys/.zai2_key

can be used for normal non-interactive profile/API-key setup, while true keystore tests can be gated behind an explicit opt-in flag.

Proposed approach

1. Define a non-interactive test auth mode that is the default for normal test runs.

   • Prefer an environment variable API key when present, such as PA_E2E_API_KEY.
   • Otherwise allow an environment variable key-file path, such as PA_E2E_API_KEY_FILE.
   • Provide a sensible local default for developer machines, such as ~/.keys/.zai2_key, when running explicit E2E/profile tests that need a real provider key.

2. Keep keystore integration tests, but make them opt-in.

   • Add an explicit flag such as PA_TEST_KEYSTORE=1, PA_E2E_USE_KEYSTORE=1, or a similarly named project convention.
   • Tests that can trigger OS credential prompts should skip unless that flag is set.
   • The skip message should tell developers exactly which flag to set when they intentionally want to test keystore behavior.

3. Centralize test profile/key setup.

   • Add or update a test support helper that builds E2E model profiles from environment/file-backed keys without touching the OS keystore by default.
   • Existing tests that need a model profile should use this helper instead of independently reaching into keychain-backed auth.
   • Keystore-specific tests should be clearly named and isolated from the default path.

4. Document the two workflows.

   • Normal non-interactive verification: no keystore prompts; reads from PA_E2E_API_KEY, PA_E2E_API_KEY_FILE, or the configured local key file.
   • Explicit keystore verification: set the keystore opt-in flag and expect OS credential prompts if the platform requires them.

Acceptance criteria

• Running the normal verification suite does not prompt for the macOS Keychain password.
• Tests that require a real provider key can use a non-interactive key source, including a key file such as ~/.keys/.zai2_key.
• Keystore-specific tests still exist and can be run intentionally with an explicit opt-in flag.
• Tests that would otherwise prompt for the keystore skip with a clear message when the opt-in flag is not set.
• The project documentation or test README explains the default non-interactive mode and the explicit keystore-test mode.
• CI remains non-interactive and does not depend on desktop keystore prompts.

Notes

This should reduce friction for routine development while preserving dedicated coverage for the secure-store integration. The key distinction is that default tests should validate application behavior without OS credential UI, and only the focused keystore tests should validate the OS keystore path.