Skip to content

docs: AI agent operations, prompts/models, testing guide, and contract error reference - #601

Closed
0xDeon wants to merge 1 commit into
codebestia:mainfrom
0xDeon:docs/ai-agent-ops-and-contract-errors
Closed

docs: AI agent operations, prompts/models, testing guide, and contract error reference#601
0xDeon wants to merge 1 commit into
codebestia:mainfrom
0xDeon:docs/ai-agent-ops-and-contract-errors

Conversation

@0xDeon

@0xDeon 0xDeon commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

Adds four reference documents. Everything here was written against the source and checked against it, rather than from the issue descriptions alone.

Closes #586
Closes #585
Closes #584
Closes #580


apps/ai_agent/docs/operations.md#586

Production uvicorn invocation and worker sizing, and why the python main.py path in the __main__ guard is development-only.

/health is documented as a liveness probe, not a readiness probe. It returns 200 with no API key set and with Weaviate down, because the handler performs no dependency checks at all. That is correct behaviour for liveness — an orchestrator should not restart a process because a third-party dependency is degraded — but it means a healthy /health is not evidence the service can do useful work, and it must not gate a deploy.

The Weaviate dependency is covered as a per-request connect/close with a full behaviour matrix for the outage cases, including two asymmetries worth knowing: /search returns 200 {"results": []} when the collection is missing, so an unindexed corpus is indistinguishable from a genuine zero-hit query; and /index/message creates the collection on demand.

On cost controls, the honest answer is that there are none, so the doc records their absence as a known operational risk rather than describing an intended design. There is no rate limiting, no authentication, no max_tokens, no input size cap, no caching, and no retry/backoff — so any party with network reach to the service can spend against the OpenAI account. The deployment-level mitigations required until controls exist in the application are listed, along with a numbered risk table.

apps/ai_agent/docs/concepts-prompts-and-models.md#585

Each prompt is reproduced verbatim from main.py — byte-identical, including indentation, so a diff against source is meaningful — alongside the behaviour it is designed to produce.

Models are documented per endpoint with guidance on changing one safely. The important distinction is that swapping a chat model is a reversible code edit, while swapping the embedding model is a corpus migration: text-embedding-3-small appears on both the write and read paths, vectors from different models are not comparable, and changing one side produces confident nonsense with no error.

Output parsing is documented per endpoint, with the transfer analyser's .get() defaults as the worked example. The two defaults deliberately fail in opposite directions — a missing flagged defaults to False (do not flag) while a missing confidence defaults to 0.0 (no confidence) — so a caller that gates on confidence catches both malformed cases, and a caller that reads flagged alone silently treats model failure as a clean transfer.

The privacy boundary is stated explicitly. /index/message sends the complete plaintext body of every indexed message to OpenAI, while its messageId, conversationId, and senderId never leave the system — content goes, metadata stays. The conversationId filter on /search is enforced inside Weaviate, not by the model.

apps/ai_agent/docs/testing.md#584

Each conftest.py fixture: what it patches, what it returns by default, and why mock_openai targets main.OpenAI rather than openai.OpenAI (the name must be patched where it is used). Includes a worked example adding an endpoint test using the fixtures, and notes that the existing modules are not uniform — only test_chat.py uses them today, while the others patch inline.

Coverage configuration and how to read the report, using the real measured output rather than an invented sample.

The /index/message gap is recorded with a case table so a contributor can pick it up directly.

contracts/docs/contracts-errors.md#580

Every panic and error path across the three contracts, grouped by function, separating authorization failures, validation failures, and state-machine violations (voting twice, approving after expiry, executing an unfinalised proposal).

Since none of the contracts define a #[contracterror] enum, there are no stable numeric codes and panic strings are not reliable at the ABI boundary. The doc therefore tells lib/soroban.ts callers to branch on the invocation stage — simulation failure, signature failure, submission failure, post-submission revert, timeout — rather than parsing messages, and to do pre-flight state reads so most failures become disabled controls instead of failed transactions. Notably, simulation catches nearly every deterministic panic before the user signs or pays a fee, and a confirmation timeout is not a failure and must never invite a retry. Cross-links apps/web/docs/api-soroban-client.md.

A few behaviours surfaced while tracing the code that are documented as findings, not fixed here — they are contract behaviour changes and belong in their own PR:

  • group_treasury::list_proposals and get_pending_proposals iterate 1..=count, but ids are assigned from 0, so the first proposal ever created is never returned by either.
  • proposals::finalize_expired_proposal can close an Active past-expiry proposal that would otherwise have Passed, discarding the tally and permanently blocking execution.
  • proposals::execute_withdraw calls the treasury's admin-gated withdraw, so the proposals contract address must itself be the treasury admin or execution fails at the final step after every other check has passed.
  • group_treasury::withdraw is admin-only and bypasses the proposal/threshold flow entirely.
  • Removing a member does not clean up their votes, and shrinks the blocking_minority denominator for open proposals.

Verification

The ai_agent suite was run to check the documented behaviour against reality: 35 passed, 82% coverage. The coverage report and uncovered line range in testing.md are the measured values, and 183-228 maps exactly to the body of index_message, confirming the documented gap.

I also wrote the /index/message cases from the gap table and ran them to confirm the guidance actually works. That surfaced a genuine error in my first draft: a missing OPENAI_API_KEY returns 503, not 500, on both Weaviate endpoints, because _openai_client() raises HTTPException(500, ...) inside a try whose except Exception as e re-raises everything as 503. All three affected docs record the actual behaviour, and operations.md notes the operational consequence — a 503 from these endpoints does not reliably mean Weaviate is down, so triage has to read the detail string. The probe tests were scaffolding and are not included; this PR is documentation only, and the suite is unchanged at its 35-passing baseline.

Every relative link in the four documents was checked to resolve against files present on main.

Adds four reference documents covering the AI agent service and the
Soroban contract failure surface.

apps/ai_agent/docs/operations.md (codebestia#586)
- Production uvicorn command and worker sizing; why `python main.py`
  is the development path only.
- /health semantics for orchestrators: liveness, not readiness. It
  stays 200 with no API key and with Weaviate down, so it must not
  gate a deploy.
- Weaviate dependency: per-request connect/close, and the full
  behaviour matrix when it is unreachable.
- Records the absence of rate limiting, auth, spend caps and
  max_tokens as a known operational risk, with the deployment-level
  mitigations required until controls exist in the application.

apps/ai_agent/docs/concepts-prompts-and-models.md (codebestia#585)
- Each prompt reproduced verbatim from main.py with the behaviour it
  is designed to produce.
- Model per endpoint, and how to change one safely — including why
  changing the embedding model is a corpus migration, not a redeploy.
- Output parsing per endpoint, with the transfer analyser's missing
  confidence/flagged defaults as the worked example, and why the two
  defaults deliberately fail in opposite directions.
- States the privacy boundary: what is sent to OpenAI and what never
  leaves the system.

apps/ai_agent/docs/testing.md (codebestia#584)
- Each conftest fixture, what it patches and what it returns by
  default, including why mock_openai patches main.OpenAI.
- Worked example adding an endpoint test using the fixtures.
- Coverage configuration and how to read the report.
- Notes the /index/message coverage gap with a case table so a
  contributor can pick it up.

contracts/docs/contracts-errors.md (codebestia#580)
- Every panic and error path across token_transfer, group_treasury
  and proposals, grouped by function.
- Separates authorization, validation and state-machine violations
  (voting twice, approving after expiry, executing an unfinalised
  proposal).
- Documents what a failure looks like from the frontend and how
  lib/soroban.ts callers should map it, branching on invocation
  stage rather than parsing panic strings.
- Cross-links the frontend Soroban client doc.

Verified against the source: the ai_agent suite was run to confirm the
documented behaviour (35 passed, 82% coverage), and the coverage report
and uncovered line range are the measured values. Testing the documented
/index/message cases surfaced that a missing OPENAI_API_KEY returns 503
rather than 500 on both Weaviate endpoints, because the broad
`except Exception` swallows the helper's HTTPException; all three
affected docs record the actual behaviour.
@drips-wave

drips-wave Bot commented Aug 31, 2026

Copy link
Copy Markdown

@0xDeon Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits

@github-actions

Copy link
Copy Markdown

👋 Hi @0xDeon, thanks for your contribution!

Pull requests from contributors must target the dev branch — only the repo maintainer merges into main.

This PR is being closed automatically. Please open a new PR (or retarget this one by reopening it after editing the base branch) against dev.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

AI agent deployment and operations AI agent prompt and model configuration AI agent testing guide Contract error and panic reference

1 participant