docs: documentation index, architecture overview, contract events reference, and resource budget - #593
Merged
Merged
Conversation
…stia#481) Document the RAG pipeline used by /index/message and /search endpoints: - Architecture diagrams for indexing and retrieval pipelines - Step-by-step flow for both endpoints - OpenAI usage: text-embedding-3-small for search vs gpt-4o-mini for chat (independent systems) - Data scope and conversation isolation via Weaviate filters - Error handling table for all failure scenarios (503, 422, etc.) - Privacy considerations: conversation isolation at DB query level - Dependencies reference with version requirements - Current integration status: /search is standalone, not yet integrated into /chat prompts OpenAI usage is documented separately from the LLM chat pipeline so future contributors understand the two distinct API roles.
…rchitecture docs: add comprehensive RAG search architecture documentation
Adds docs/README.md as the single entry point to the repository's documentation. Every .md file in the repo (excluding generated output and dependency directories) is listed exactly once with a one-line description. Documents are grouped by reader intent rather than by directory: new contributor, backend developer, frontend developer, contract developer, operator, security reviewer, and AI/data developer. A document needed by two roles is listed under both, since each section is meant to stand alone. Includes a "Start here" path for first-time contributors that ends at the contribution guidelines, and a maintenance section stating that the index must be updated whenever a document is added, moved, or removed. The root README now links to the index and to the architecture overview. Note: the issue refers to a top-level CONTRIBUTING.md and a docs/adr/ directory. Neither exists in this repository, so the "Start here" path ends at the Contributing section of the root README and calls out that it should be repointed if a dedicated CONTRIBUTING.md is added.
Adds docs/architecture-overview.md as the orientation entry point for the system: one Mermaid diagram showing all four apps, the stateful infrastructure (Postgres, Redis, object storage, Weaviate), the Stellar network, and the external LLM provider, plus a table naming the protocol on every edge. Each component gets a short paragraph covering its responsibility and, explicitly, what it must never do — the invariants that are easy to break accidentally (the gateway must not acquire a decryption path, Redis must not be treated as durable, the AI agent must not gate a transfer). Two end-to-end paths are traced concretely against the current code: sending an encrypted message (per-device envelope encryption through deliveryPipeline fan-out to delivery receipts) and executing a treasury withdrawal (on-chain propose/vote/execute with the listener maintaining an off-chain mirror). The document is explicitly scoped as orientation only and defers to the per-app documents, which it links throughout. Notes a real defect found while tracing path 2: the backend listener subscribes to a `proposal_executed` topic that no contract publishes — `proposals` emits the truncated symbols `executed` and `execut`. The per-event detail belongs in the contract events reference (codebestia#581); this document links to it. The issue refers to IMPLEMENTATION_DOCS.md as the existing long-form design spec. No such file exists in this repository, so this document links to the per-app docs as the deeper reference instead.
Adds contracts/docs/contracts-events.md documenting all sixteen env.events().publish(...) call sites across token_transfer, group_treasury and proposals: topic tuple, data payload field by field, the state change each event signals, and whether the publish happens before or after the corresponding storage write. Emission order is called out per event because two of them — group_treasury's proposal_approved and proposal_rejected — publish before the proposal is persisted, setting the status on the in-memory struct first. Not observable to an off-chain consumer given transaction atomicity, but relevant when reading the contract. A summary table marks each event as consumed or unconsumed by the backend listener, and a dedicated section collects the five consumption gaps found while cross-referencing the contracts against stellarListener.ts: - the listener subscribes to `proposal_executed`, a topic no contract publishes. proposals emits `executed` and `execut` instead, so the status map entry is unreachable and a proposal never reaches the executed status in the off-chain mirror via the listener - group_treasury has no execution event tying a fund movement back to the proposal that authorised it - deposit and withdraw, the two value-movement events, have no consumer - only one contract id is watched for treasury topics, and both contracts publish a `proposal_created` with different payloads and different id widths (u32 vs u64) - vote-level events are unconsumed; on-chain votes are invisible to the backend until a transition event fires Also documents two payload gaps (DepositEvent and WithdrawEvent omit the token address despite per-token balance tracking) and notes that the proposer's auto-approval emits no withdraw_vote, so counting those events undercounts approvals by one. The `executed` / `execut` split is flagged as a naming defect rather than a constraint: symbol_short! permits 9 characters and "executed" is 8, so the truncation is unforced. Cross-links the backend chain listener source, the treasury API doc, and the deployment doc.
Adds contracts/docs/concepts-resource-budget.md covering the deployed size constraint and the wider resource picture around it. Documents the gate as CI actually implements it: 102,400 bytes per contract, enforced in the "Report WASM binary sizes" step of contracts-ci.yml, run per-contract across a fail-fast:false matrix, measured against the raw cargo build output with no wasm-opt step, and posted to the PR as an update-in-place comment keyed on a marker comment. Notes that the gate is per-contract rather than cumulative and that the step reports the full table before failing. Records current sizes measured from a local release build, with headroom: group_treasury 31,843 B (31.1% of gate), proposals 25,931 B (25.3%), token_transfer 10,539 B (10.3%). Includes the command to reproduce them, since sizes drift with soroban-sdk upgrades. Documents each [profile.release] setting and the cost of changing it, including that overflow-checks = true costs size deliberately and should not be traded away in contracts that move funds. Gives concrete size-reduction techniques in effort order: profile with twiggy first, run stellar contract optimize, replace panic string literals with a #[contracterror] enum, keep formatting machinery out of the shipped path, deduplicate generic instantiations, split along a real boundary, and only then raise the threshold. Closes with the resources beyond raw size — CPU instructions, memory, ledger entry reads/writes, events, rent/TTL, and transaction size — and identifies the constraint that actually binds first in this codebase: all three contracts keep per-proposal and per-voter data in instance storage, which grows unbounded and is read and written whole on every call, and several functions do linear scans over it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds four documentation entry points that were missing. Docs only — no code changed.
Closes #589
Closes #587
Closes #581
Closes #582
Supersedes #590, which was auto-closed for targeting
main; this one targetsdevand could not be retargeted in place.docs/README.mddocs/architecture-overview.mdcontracts/docs/contracts-events.mdcontracts/docs/concepts-resource-budget.mdEach was developed on its own branch (
docs/589-*,docs/587-*,docs/581-*,docs/582-*) and merged here so the cross-links between them resolve. Per-issue detail is in the individual commit messages.Verified, not assumed
.mdfiles in the repo are indexed exactly oncemermaid-cliwith no parse errorscargo build --release --target wasm32-unknown-unknown, arithmetic re-checked against the measured bytesNote on the diff against
devThe branch was cut from
main, which is currently ahead ofdev. The diff therefore also showsapps/ai_agent/docs/concepts-rag-search-architecture.md(+186), which is not part of this work — it is an already-merged doc from #536 thatdevhas not yet received. Our own contribution is the four files in the table above plus a small README cross-link.Defect found while writing the events reference
The backend chain listener subscribes to a
proposal_executedtopic that no contract publishes.proposalsemitssymbol_short!("executed")andsymbol_short!("execut")instead;group_treasuryhas no execution event at all. The listener'sstatusMaphas aproposal_executed: 'executed'entry, so the code path exists but is unreachable — a proposal never reachesexecutedstatus in the off-chain mirror via the listener.Pre-existing, and not introduced in any single commit:
"executed"(dc4169e, 2026-05-29),"execut"(9929bf9, 2026-06-25), and the listener's'proposal_executed'(78f8bb9, 2026-06-26) were written by three different authors across four weeks, with no test crossing the boundary to catch the mismatch.symbol_short!permits 9 characters and"executed"is 8, so the truncation inexecutlooks unforced.Documented here, not fixed — changing an emitted topic is a behavioural change to the chain interface and wants its own issue and review. Worth noting for whoever takes it: fixing the topic requires a redeploy, and events already emitted on-chain keep the old name permanently, so the listener likely needs to match all three names through a transition rather than only the corrected one.
Four further consumption gaps are documented in the same file:
group_treasuryhas no execution event tying a fund movement back to its proposal;depositandwithdraw(the two value-movement events) have no consumer; only one contract id is watched for treasury topics, while both contracts publish aproposal_createdwith different payloads and different id widths (u32 vs u64); and vote-level events are unconsumed.Notes on the issue text
Three files referenced by the issues do not exist in this repository, and never have (checked full history):
CONTRIBUTING.mdand adocs/adr/directory. The "start here" path therefore ends at the Contributing section of the root README, with a note to repoint it if a dedicatedCONTRIBUTING.mdis added.IMPLEMENTATION_DOCS.mdas the existing long-form design spec. The overview links to the per-app docs as the deeper reference instead.#582 implies the contracts are near the size limit. They are not —
group_treasury31,843 B (31.1% of the 100 KB gate),proposals25,931 B (25.3%),token_transfer10,539 B (10.3%). The doc records the real headroom and identifies the constraint that actually binds first: all three contracts keep per-proposal and per-voter data in instance storage, which grows unbounded and is read and written whole on every call, with several functions doing linear scans over it.Issue linking
Each issue is closed with its own
Closeskeyword above — the keyword does not chain across a comma, soCloses #589, #587would only close the first.Note that all four issues are currently unassigned on GitHub. Flagging in case the assignment was expected to be in place before merge.