PROGRESS.md

Project Progress

Last updated	2026-07-26
Current phase	Agent Phase 1 complete; Evaluation Framework Phase 1 design complete
Version	0.1.0

This document records what exists and has been verified, as distinct from what is planned. Anything listed as complete has been executed and its output observed.

---

Verification status

Every gate below was run against the working tree, not inferred.

Gate	Command	Result
Lint	ruff check src tests	Pass, 0 findings
Format	ruff format --check src tests	Pass, 79 files
Types	mypy (strict)	Pass, 61 source files
Tests	pytest tests/unit	462 passed
Image build	docker build	Pass
Container boot	docker run + health probes	Pass, healthy
Non-root container	docker exec oqa-smoke id -u	10001
Chat end to end	Live SSE stream against qwen2.5-coder:7b	Pass
Query pipeline end to end	Live OpenMRS 8.0.46, 392,181 observations	Pass

Verified against the live database

Every figure below was produced by the running agent and cross-checked against a direct database probe.

Question	Answer	Behaviour verified
"How many patients are registered?"	7,226	Correct count, voided = 0 applied
"What is the average weight recorded?"	24.90	Resolved "weight" to concept 119 from the live dictionary; used value_numeric because the concept datatype is Numeric
"How many encounters in each year?"	5 rows, 1998 to 2026	Grouped aggregate, LIMIT injected, plan warnings surfaced
"How many encounters this month?"	623	Period resolved to 2026-07-01/2026-07-31; used encounter_datetime after the repair loop rejected date_created
"How many deliveries were recorded?"	Clarifying question	Four matching concepts found; refused to guess
"What does the voided column mean?"	Conversational answer	Scope gate routed away from SQL
"What is the weather tomorrow?"	Refused	Out of domain, refused before any model call
"Ignore all previous instructions..."	Refused	Manipulation detected, refused before any model call
DELETE FROM obs (direct)	Refused	not_a_select, forbidden_statement

Defence layers, in order

Each runs before the next, and each assumes the previous may have failed.

Layer	Enforces	When
Scope gate	Only anchored data questions reach the generator	Before any model call
AST validator	No DDL/DML, no blocked functions, no system schemas	After generation
Clinical checks	Correct time column, soft-delete filter present	After validation
Plan review	Row-scan budget	Before execution
Executor	Read-only session, server timeout, row cap	During execution
Database grant	SELECT only	Always

Ruff runs through the pinned ghcr.io/astral-sh/ruff:0.16.0 container on this workstation, because a Windows Application Control policy blocks unsigned executables from user directories. CI runs it natively.

---

Agent Phase 1: Requirements and Enterprise Architecture

Status: complete.

Code delivered

Area	Modules	Notes
Configuration	oqa.config.settings	Typed settings for every environment variable, with cross-field validation and production guardrails that refuse unsafe startup
Logging	oqa.config.logging	structlog with a PHI-redaction processor and request correlation; unified with stdlib so uvicorn and SQLAlchemy emit the same JSON
Domain models	oqa.domain.models.*	Query pipeline, schema graph, concept dictionary, RAG, security, audit
Domain errors	oqa.domain.errors	18 error types with stable codes, HTTP status and retry semantics
Ports	oqa.application.ports.*	10 Protocol interfaces: LLM, embedding, sparse encoding, vector store, re-ranker, SQL executor, cache, audit, and four repositories
Health	oqa.application.health	Concurrent dependency checks with per-check timeout; required vs optional distinction
API	oqa.interfaces.api.*	App factory, middleware, error translation, metrics, health and chat routes
CLI	oqa.interfaces.cli.main	oqa config check, oqa config show, oqa serve
LLM adapter	oqa.infrastructure.llm.ollama	Completion, constrained JSON, streaming, health; narrow retry policy
Conversations	oqa.infrastructure.memory	Bounded LRU store implementing ConversationRepository
Chat service	oqa.application.use_cases.chat	Prompt assembly, bounded history, multi-turn memory
Chat UI	oqa.interfaces.api.static	Self-contained streaming interface, no external requests

The chat interface

Working and verified against a real model. It streams, remembers context across turns, renders Markdown with SQL highlighting, and runs with no external network access.

What makes it more than a wrapper is the system prompt, which is treated as a safety control and tested as one. Asked for a figure it cannot retrieve, the agent declines and shows the SQL instead, using :delivery_concept_id rather than inventing an ID. Verified live:

• Refuses to state counts it cannot retrieve.
• Uses a named placeholder instead of a hallucinated concept_id.
• Includes voided = 0 and COUNT(DISTINCT ...) unprompted.
• Resolves "that table" to obs on a follow-up turn, which requires history.

One observed weakness worth recording: the model filtered on date_created where obs_datetime is the clinically correct column. That is precisely the class of error retrieval grounding (Phase 3) exists to fix, and it is why the chat is not yet connected to a database.

Non-trivial logic that is genuinely implemented

Not scaffolding, and covered by tests:

• Join-path discovery (SchemaGraph.find_join_path): breadth-first search over foreign keys in both directions, with a hop limit. This is what stops the SQL generator inventing relationships.
• Column sensitivity classification (classify_column): explicit OpenMRS core lists, then regex fallback for module and Bahmni tables, defaulting to the conservative side for unknown columns in clinical tables.
• PHI log redaction: recursive, depth-bounded, with an allowlist so concept_name and table_name stay readable while given_name does not.
• Production guardrails: seven checks, all violations reported at once.
• Concept datatype to obs column mapping: the difference between a query that works and one that silently returns zero rows.

Infrastructure

• Multi-stage Dockerfile: non-root UID 10001, no build toolchain in the runtime image, healthcheck on /health/live.
• docker-compose.yml with four profiles: default (Qdrant, Redis, API), ai (Ollama), demo (a real OpenMRS reference installation to introspect), obs (Prometheus, Grafana, Loki).
• GitHub Actions CI: lint, strict types, unit tests on a 2x2 OS and Python matrix, integration tests against service containers, pip-audit and Trivy, and a Docker build that boots the container and asserts it is non-root.
• Makefile working on Windows (Git Bash) and POSIX.

Bugs found and fixed during Phase 1

Five real defects, all caught by tests and by actually running the stack rather than by review. The last two were found only by starting the system end to end, which is why "it builds" is not the same gate as "it runs":

1. Database password leaked through serialisation. sqlalchemy_url was a Pydantic computed_field, so the plaintext password appeared in repr() and in model_dump(), and therefore in oqa config show --json and in any traceback that formatted a settings object. Fixed by making it a plain property; only the masked safe_url is serialisable. Four regression tests now cover it.
2. create_app(settings) ignored its argument. Routes resolved settings through the global get_settings() singleton rather than app.state, so /health/info reported the process environment instead of the application's. Fixed with a get_app_settings dependency reading app.state.
3. The container could not start. uvicorn --log-config /dev/null makes uvicorn parse an empty file as a logging config and abort. The CLI has no way to express "no logging config"; only the programmatic API accepts log_config=None. The image now runs oqa serve.
4. The documented quick start produced a crash loop. .env.example describes OQA_CORS_ORIGINS as comma-separated, but pydantic-settings JSON-decodes list fields, so cp .env.example .env followed by docker compose up failed on boot with a parse error. The field now uses NoDecode with a validator accepting comma-separated values, a JSON array, or an empty string. An existing test had masked this: it was named "comma-separated" but passed JSON. Six tests now cover the real forms.
5. Metrics were declared but never incremented. /metrics served only HELP and TYPE lines, so a Grafana dashboard wired to oqa_http_requests_total would have shown an idle system rather than a missing instrument. The access-log middleware now records the counter and the latency histogram, labelled by route template so no identifier can enter a label value.

Documentation

Document	Lines	Status
Vision	527	Complete
Functional requirements	1,658	Complete, 90 requirements
Non-functional requirements	1,281	Complete, 59 NFRs
Architecture	659	Complete
Security guide	1,160	Complete
Deployment guide	1,786	Complete
ADR 0001-0009	9 records	Complete
High-level design	0	Not written. See below

Known gap

docs/05-high-level-design.md is not yet written. It was scoped for Phase 1 and its authoring run was cut short. The material it would contain (per-stage input and output types, sequence diagrams, the error-to-audit-action mapping) is partly covered by Architecture sections 4 and 5, but the detailed design remains outstanding and should be completed before Phase 2 implementation begins.

---

Evaluation Framework Phase 1: Architecture

Status: design complete, awaiting approval.

Evaluation architecture (748 lines) specifies the benchmark and continuous-learning platform. The load-bearing decisions:

• A four-tier gold standard that avoids circular evaluation. The largest tier constructs SQL deterministically from the introspected schema graph and the live concept dictionary, then generates the question from the SQL, so correctness is established by construction rather than asserted by a model.
• Execution accuracy as the primary metric, with result-set equivalence defined precisely. SQL string similarity is reported but never gated on.
• Metamorphic relations as a gold-free signal, targeting exactly the two errors that inflate OpenMRS numbers silently: a missing voided = 0 and a join that turns COUNT DISTINCT into COUNT.
• AST diffing with sqlglot for objective, automated error classification.
• A sealed holdout with family-based splits, so the improvement loop cannot overfit the benchmark without it being visible.

Code delivered

src/oqa_eval/domain/benchmark.py, with 35 tests. The two behaviours that carry the framework's integrity are implemented and covered:

• GoldStandard.is_scorable structurally prevents model-drafted items from contributing to a headline accuracy figure until a named human verifies them.
• assign_splits partitions by family, using a stable hash so that adding items never reshuffles existing ones and the holdout stays sealed across releases.

Revised targets

Section 12 of the evaluation architecture proposes per-difficulty accuracy targets in place of a single ">95%" figure, with a weighted overall target of 82% execution accuracy and 98% execution success. The reasoning is recorded there. This needs a decision before Phase 2.

Open questions

Five questions in section 16 of that document need answers before evaluation Phase 2 starts, most importantly whether a real OpenMRS database is available for validating synthetic-data realism.

---

Delivery plan

Agent

Phase	Scope	Status
1	Requirements, architecture, foundation	Complete
2	Database reverse engineering, metadata extraction	Not started
3	RAG pipeline	Not started
4	MCP server	Not started
5	SQL generation, validation, optimisation	Not started
6	AI orchestration	Not started
7	Web frontend	Not started (Node.js is not installed on this workstation)
8	Authentication	Not started
9	Reporting	Not started
10	Test hardening	Not started
11	Performance tuning	Not started
12	Production deployment	Not started

Evaluation framework

Phase	Scope	Status
1	Architecture and benchmark design	Design complete, awaiting approval
2	Synthetic data generator	Blocked on agent Phase 2
3	Benchmark generator, Tiers A and B	Blocked on eval Phase 2
4	Evaluation engine	Blocked on agent Phase 5
5-10	Diagnosis, learning loop, dashboard, datasets, load, deployment	Not started

---

Environment notes

Observed on this workstation, and relevant to what can be verified locally:

Tool	Status
Python 3.12.10	Available
Docker 29.5.3, Compose v5.1.4	Available
Git 2.54.0	Available
Ollama 0.32.3	Installed, not running; no models pulled
Node.js / npm	Not installed; required for Phase 7
MySQL client	Not installed; the Python driver is used instead
Ruff native binary	Blocked by Application Control; run via container

No OpenMRS database is currently configured. Phase 2 introspection should run against docker compose --profile demo up -d, which starts a real OpenMRS installation, so the schema is read rather than assumed.