docs: AI agent operations, prompts/models, testing guide, and contract error reference - #602
Merged
codebestia merged 1 commit intoAug 31, 2026
Conversation
Adds four reference documents covering the AI agent service and the Soroban contract failure surface. apps/ai_agent/docs/operations.md (codebestia#586) - Production uvicorn command and worker sizing; why `python main.py` is the development path only. - /health semantics for orchestrators: liveness, not readiness. It stays 200 with no API key and with Weaviate down, so it must not gate a deploy. - Weaviate dependency: per-request connect/close, and the full behaviour matrix when it is unreachable. - Records the absence of rate limiting, auth, spend caps and max_tokens as a known operational risk, with the deployment-level mitigations required until controls exist in the application. apps/ai_agent/docs/concepts-prompts-and-models.md (codebestia#585) - Each prompt reproduced verbatim from main.py with the behaviour it is designed to produce. - Model per endpoint, and how to change one safely — including why changing the embedding model is a corpus migration, not a redeploy. - Output parsing per endpoint, with the transfer analyser's missing confidence/flagged defaults as the worked example, and why the two defaults deliberately fail in opposite directions. - States the privacy boundary: what is sent to OpenAI and what never leaves the system. apps/ai_agent/docs/testing.md (codebestia#584) - Each conftest fixture, what it patches and what it returns by default, including why mock_openai patches main.OpenAI. - Worked example adding an endpoint test using the fixtures. - Coverage configuration and how to read the report. - Notes the /index/message coverage gap with a case table so a contributor can pick it up. contracts/docs/contracts-errors.md (codebestia#580) - Every panic and error path across token_transfer, group_treasury and proposals, grouped by function. - Separates authorization, validation and state-machine violations (voting twice, approving after expiry, executing an unfinalised proposal). - Documents what a failure looks like from the frontend and how lib/soroban.ts callers should map it, branching on invocation stage rather than parsing panic strings. - Cross-links the frontend Soroban client doc. Verified against the source: the ai_agent suite was run to confirm the documented behaviour (35 passed, 82% coverage), and the coverage report and uncovered line range are the measured values. Testing the documented /index/message cases surfaced that a missing OPENAI_API_KEY returns 503 rather than 500 on both Weaviate endpoints, because the broad `except Exception` swallows the helper's HTTPException; all three affected docs record the actual behaviour.
This was referenced Aug 31, 2026
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds four reference documents. Everything here was written against the source and checked against it, rather than from the issue descriptions alone.
Retargeted onto
dev— my first attempt (#601) went tomainand was auto-closed by the branch-policy bot. Same commit, rebased ontodev.Closes #586
Closes #585
Closes #584
Closes #580
apps/ai_agent/docs/operations.md— #586Production
uvicorninvocation and worker sizing, and why thepython main.pypath in the__main__guard is development-only./healthis documented as a liveness probe, not a readiness probe. It returns200with no API key set and with Weaviate down, because the handler performs no dependency checks at all. That is correct behaviour for liveness — an orchestrator should not restart a process because a third-party dependency is degraded — but it means a healthy/healthis not evidence the service can do useful work, and it must not gate a deploy.The Weaviate dependency is covered as a per-request connect/close with a full behaviour matrix for the outage cases, including two asymmetries worth knowing:
/searchreturns200 {"results": []}when the collection is missing, so an unindexed corpus is indistinguishable from a genuine zero-hit query; and/index/messagecreates the collection on demand.On cost controls, the honest answer is that there are none, so the doc records their absence as a known operational risk rather than describing an intended design. There is no rate limiting, no authentication, no
max_tokens, no input size cap, no caching, and no retry/backoff — so any party with network reach to the service can spend against the OpenAI account. The deployment-level mitigations required until controls exist in the application are listed, along with a numbered risk table.apps/ai_agent/docs/concepts-prompts-and-models.md— #585Each prompt is reproduced verbatim from
main.py— byte-identical, including indentation, so a diff against source is meaningful — alongside the behaviour it is designed to produce.Models are documented per endpoint with guidance on changing one safely. The important distinction is that swapping a chat model is a reversible code edit, while swapping the embedding model is a corpus migration:
text-embedding-3-smallappears on both the write and read paths, vectors from different models are not comparable, and changing one side produces confident nonsense with no error.Output parsing is documented per endpoint, with the transfer analyser's
.get()defaults as the worked example. The two defaults deliberately fail in opposite directions — a missingflaggeddefaults toFalse(do not flag) while a missingconfidencedefaults to0.0(no confidence) — so a caller that gates onconfidencecatches both malformed cases, and a caller that readsflaggedalone silently treats model failure as a clean transfer.The privacy boundary is stated explicitly.
/index/messagesends the complete plaintext body of every indexed message to OpenAI, while itsmessageId,conversationId, andsenderIdnever leave the system — content goes, metadata stays. TheconversationIdfilter on/searchis enforced inside Weaviate, not by the model.apps/ai_agent/docs/testing.md— #584Each
conftest.pyfixture: what it patches, what it returns by default, and whymock_openaitargetsmain.OpenAIrather thanopenai.OpenAI(the name must be patched where it is used). Includes a worked example adding an endpoint test using the fixtures, and notes that the existing modules are not uniform — onlytest_chat.pyuses them today, while the others patch inline.Coverage configuration and how to read the report, using the real measured output rather than an invented sample.
The
/index/messagegap is recorded with a case table so a contributor can pick it up directly.contracts/docs/contracts-errors.md— #580Every panic and error path across the three contracts, grouped by function, separating authorization failures, validation failures, and state-machine violations (voting twice, approving after expiry, executing an unfinalised proposal).
Since none of the contracts define a
#[contracterror]enum, there are no stable numeric codes and panic strings are not reliable at the ABI boundary. The doc therefore tellslib/soroban.tscallers to branch on the invocation stage — simulation failure, signature failure, submission failure, post-submission revert, timeout — rather than parsing messages, and to do pre-flight state reads so most failures become disabled controls instead of failed transactions. Notably, simulation catches nearly every deterministic panic before the user signs or pays a fee, and a confirmation timeout is not a failure and must never invite a retry. Cross-linksapps/web/docs/api-soroban-client.md, and the events/testing/upgrades/resource-budget docs now ondev.A few behaviours surfaced while tracing the code that are documented as findings, not fixed here — they are contract behaviour changes and belong in their own PR:
group_treasury::list_proposalsandget_pending_proposalsiterate1..=count, but ids are assigned from0, so the first proposal ever created is never returned by either.proposals::finalize_expired_proposalcan close anActivepast-expiry proposal that would otherwise havePassed, discarding the tally and permanently blocking execution.proposals::execute_withdrawcalls the treasury's admin-gatedwithdraw, so theproposalscontract address must itself be the treasury admin or execution fails at the final step after every other check has passed.group_treasury::withdrawis admin-only and bypasses the proposal/threshold flow entirely.blocking_minoritydenominator for open proposals.Verification
The
ai_agentsuite was run to check the documented behaviour against reality: 35 passed, 82% coverage, unchanged before and after the rebase ontodev. The coverage report and uncovered line range quoted intesting.mdare the measured values, and183-228maps exactly to the body ofindex_message, confirming the documented gap.I also wrote the
/index/messagecases from the gap table and ran them to confirm the guidance actually works. That surfaced a genuine error in my first draft: a missingOPENAI_API_KEYreturns503, not500, on both Weaviate endpoints, because_openai_client()raisesHTTPException(500, ...)inside atrywhoseexcept Exception as ecatches it and re-raises as503. The observed response is503 {"detail": "500: OPENAI_API_KEY is not configured"}.One thing worth a maintainer's eye: this contradicts
apps/ai_agent/docs/configuration.mdalready ondev, which states these endpoints return500when Weaviate is reachable and503only when it is unreachable. I re-checked with a mocked-healthy Weaviate and the key removed — both/index/messageand/searchreturn503in that case too.operations.mddocuments the observed behaviour and carries a short note correcting the sibling doc; happy to instead patchconfiguration.mddirectly if you would prefer the correction to live there.The probe tests were scaffolding and are not included — this PR is documentation only, and the test suite is untouched at its 35-passing baseline. Every relative link in the four documents was checked to resolve against files present on
dev.