Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 0 additions & 7 deletions docs/AGENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -2500,13 +2500,6 @@ the role's own grants are the whole boundary (A3).
and never what it said, so an empty completion reaches it too and a model's recorded
`retryEmptyTurn: false` decides nothing. Pinned as it behaves rather than narrowed, because the
narrowing would move behaviour five passing runs were measured under.
- **B76** — the aggregated capture that fixed the wide-catalog refusal now ADMITS the image's own
extension views, so a grounded run on TimescaleDB, Cloudberry or AlloyDB Omni reasons over an
inventory that is mostly internal objects (measured: AlloyDB's least-privilege role sees 67 extension
views beside its 2 user tables). Not a privilege leak — the role genuinely sees them — but grounding
noise; the object set was the same under the flat projection, which refused before any of it reached
a run.

**Settled as limits rather than as work.** The seven below have no entry in `docs/BACKLOG.md`, and

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The B76 removal took the blank line before this paragraph with it, so "Settled as limits rather than as work." now renders as a continuation of the B65 bullet instead of a new paragraph. One blank line above this line fixes it.

that is the point: each is how the product behaves, stated where a reader of this document will meet
it, rather than a queue item nobody was going to pick up. A limitation needs a record; it does not
Expand Down
30 changes: 1 addition & 29 deletions docs/BACKLOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ None of it is a GitHub issue.
- [Security Phase 2 deferrals](#security-phase-2-deferrals) — C3–C11 · 8
- [Security Phase 3 deferrals](#security-phase-3-deferrals) — K4
- [Agent M1 deferrals (#328)](#agent-m1-deferrals-328) — A1–A5 · 4
- [Agent M2 deferrals (#329)](#agent-m2-deferrals-329) — B2–B76 · 22
- [Agent M2 deferrals (#329)](#agent-m2-deferrals-329) — B2–B75 · 21

---

Expand Down Expand Up @@ -1849,31 +1849,3 @@ so), and none of them has been measured.

**Done when:** a resume onto a repointed connection does one stated thing, and the run's own record
says which.

### B76. The aggregated grounding capture admits the image's own extension views

The aggregation that fixed the row-cap refusal made a stock image's wide catalog capture SUCCEED instead
of refusing, and the price
is that it succeeds by admitting the image's own objects. Measured live on 2026-09-01 against the
`compat` profile images (`database-compose.yml`), with two user tables seeded:

- TimescaleDB (`timescale/timescaledb:latest-pg17`): 385 column rows aggregate to 46 tables.
- Cloudberry (`woblerr/cloudberry:2.1.0-incubating`): 479 to 67 as gpadmin, 53 as a least-privilege role.
- AlloyDB Omni (`google/alloydbomni:17.9.0`): 542 to 70 as postgres, 69 as a least-privilege role.

AlloyDB is the sharp case: the least-privilege role's 69 tables are 2 user tables plus 67 extension views,
and where they SIT is the part that makes them noise rather than a schema to exclude — 49 of them are
installed into `public` itself (`google_db_advisor_*`, `hypopg_list_indexes`), beside 14 in `google_ml` and
4 in `ai`. Not a privilege leak —
`information_schema.columns` applies its own visibility rules and the role genuinely sees those views —
but grounding noise: the inventory a run reasons over is mostly objects the user did not create, and the
model-facing pack spends `AGENT_CONTEXT_PACK_MAX_CHARS` ranking them against the objective. The old flat
projection had the same object set; it just refused before any of it reached a run.

The candidate fixes are the ones the row-cap fix recorded and set aside: filter by `table_type`, or
exclude the objects the object browser already treats as internal. That fix refuted only the
schema-exclusion variant AS A FIX FOR THE ROW CAP; as a fix for grounding noise the question is open and
unmeasured.

**Done when:** a run grounded on one of these images reasons over the user's objects — or the internal
ones are excluded — with a test per shape (TimescaleDB, Cloudberry, AlloyDB Omni).
19 changes: 13 additions & 6 deletions docs/providers/postgres.md
Original file line number Diff line number Diff line change
Expand Up @@ -1081,12 +1081,19 @@ path is reachable from server code and not from a request.
Four things about the PostgreSQL side of that layer are worth knowing here:

- **The catalog read is a composed bounded read**, not a new operation. `inspect_schema` takes a
schema/table selector and the server writes
`SELECT … FROM information_schema.columns WHERE table_schema NOT IN ('pg_catalog', …)`, executed as
`sql.query.read` like any other statement. The model never supplies that SQL. Selectors are quoted
with `quoteLiteral` because `queryReadOnly` binds no parameters, and a selector carrying a
backslash is refused outright rather than quoted — the dialect-less span reader treats it as an
escape, so `'a\'` would read as an unterminated literal.
schema/table selector and the server writes the `columns` statement itself, executed as
`sql.query.read` like any other statement. The model never supplies that SQL. The statement's
`WHERE` excludes the engine's own objects three ways, all copied from this provider's object

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"all copied from this provider's object browser rather than invented on the agent path" is not accurate for the third way. The browser has no pg_class extension-ownership test; that one is new on the agent path. As written the sentence also hides the divergence: the relation test covers the columns read and not indexes / statistics or the browser.

browser rather than invented on the agent path: the full engine-builtin schema list (not only
`pg_catalog` / `information_schema`), every schema an extension created (`pg_depend` on
`pg_namespace`, `deptype = 'e'`), and every relation an extension created (`pg_depend` on
`pg_class`, `deptype = 'e'`). The relation test is the one that reaches AlloyDB Omni's extension
views, 49 of which sit in `public` itself where no schema filter can reach them; a user's own views
are never extension-owned, so they stay in the inventory. Measured live on 2026-09-07 with two user
tables seeded: 46 → 2 object rows on TimescaleDB, 67 → 2 on Cloudberry and 70 → 2 on AlloyDB Omni.
Selectors are quoted with `quoteLiteral` because `queryReadOnly` binds no parameters, and a selector
carrying a backslash is refused outright rather than quoted — the dialect-less span reader treats it
as an escape, so `'a\'` would read as an unterminated literal.
- **A run reads three catalog inventories at its start (#329 T8), not one.** `inspect_schema` takes
a `kind` — `columns` (the default), `relations` (foreign keys, from `pg_constraint` with
`unnest(conkey, confkey) WITH ORDINALITY` pairing the two sides) and `indexes` (from `pg_index`
Expand Down
121 changes: 117 additions & 4 deletions src/lib/agent/composed-sql.ts
Original file line number Diff line number Diff line change
Expand Up @@ -154,6 +154,89 @@ function equalsClause(column: string, value: string | undefined, field: string,
return ` AND ${column} = ${quoteLiteral(assertSelector(value, field), dialect)}`;
}

// ─── PostgreSQL internal-object exclusion ─────────────────────────────────

/**
* The engine-builtin schemas a PostgreSQL grounding read must not treat as user
* data. This is the SAME set the provider's object browser uses
* (`SYSTEM_SCHEMAS` in `src/lib/db/providers/sql/postgres.ts`), copied rather
* than imported because the agent side must not depend on the provider module.
* `tests/unit/lib/agent/composed-sql.test.ts` pins the two lists together so
* they cannot drift. Each group is sorted by engine and checked against that
* engine's own documentation or, where the doc is silent, a live instance.
*/
const POSTGRES_SYSTEM_SCHEMAS = [
// PostgreSQL itself.
"pg_catalog",
"information_schema",
"pg_toast",
// Materialize - materialize.com/docs/sql/system-catalog/
"mz_catalog",
"mz_internal",
"mz_introspection",
// CockroachDB - cockroachlabs.com/docs/stable/system-catalogs enumerates exactly
// four schemas; the two below are the ones stock PostgreSQL does not also have.
"crdb_internal",
"pg_extension",
// TimescaleDB - the extension's own sql/pre_install/schemas.sql creates all seven.
// `_timescaledb_internal` is the one that floods: it holds every hypertable chunk.
"_timescaledb_catalog",
"_timescaledb_config",
"_timescaledb_functions",
"_timescaledb_internal",
"_timescaledb_cache",
"timescaledb_experimental",
"timescaledb_information",
// Apache Cloudberry - cloudberry.apache.org create-and-manage-schemas documents the
// first three. `pg_ext_aux` is not in that page but holds the PAX auxiliary tables
// (pg_pax_tables, pg_pax_fastsequence) on a live 2.1.0 instance, so it is here on
// measurement rather than on the doc's authority.
"gp_toolkit",
"pg_aoseg",
"pg_bitmapindex",
"pg_ext_aux",
] as const;

// Rendered once. Callers interpolate this into a `NOT IN (...)` clause.
const POSTGRES_SYSTEM_SCHEMA_LIST = POSTGRES_SYSTEM_SCHEMAS.map((schema) => `'${schema}'`).join(", ");

/**
* Schemas an EXTENSION created. AlloyDB Omni is deliberately absent from the
* name list above: its `google_ml` and `ai` schemas are extension-created, and
* those are names a user could plausibly choose for schemas of their own, so
* hiding them by name would make the user's tables vanish. Ownership is the
* question the name was standing in for, and `pg_depend` answers it directly.
* Same query the provider composes (`EXTENSION_OWNED_SCHEMAS_SQL`).
*/
const POSTGRES_EXTENSION_OWNED_SCHEMAS_SQL =

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The provider does not read pg_depend / pg_extension unguarded. It carries a fallback: isMissingExtensionCatalogError -> withoutExtensionOwnershipTest (src/lib/db/providers/sql/postgres.ts:458 and :484), because Materialize raises on those catalogs.

The agent path has no equivalent. composeCatalogRead returns one composed statement and there is no rewrite chain around it, so on a connection typed postgres that is actually Materialize (and postgres is in AGENT_EXECUTION_ENGINES) all four catalog reads fail: the context capture goes silently unavailable and inspect_schema returns a DB failure.

The live verification covered TimescaleDB, Cloudberry and AlloyDB Omni, all of which do have pg_depend, so it could not see this. Either mirror the provider's fallback or state the Materialize exclusion explicitly.

"SELECT n.nspname FROM pg_namespace n " +
"JOIN pg_depend d ON d.objid = n.oid AND d.classid = 'pg_namespace'::regclass AND d.deptype = 'e' " +
"JOIN pg_extension e ON e.oid = d.refobjid";

/**
* RELATIONS (tables, views, materialized views, sequences) an extension created
* — the B76 sharp case. AlloyDB Omni installs 49 of its 67 extension views into
* `public` itself, where no schema filter can reach them, so the OBJECT must be
* excluded by ownership rather than by where it sits. This is the same
* `pg_depend` ownership test with `classid = 'pg_class'::regclass`: it returns
* the extension's own relations and nothing a user created, so a user's own
* views survive.
*/
const POSTGRES_EXTENSION_OWNED_RELATIONS_SQL =
"SELECT n.nspname, c.relname FROM pg_class c " +
"JOIN pg_namespace n ON n.oid = c.relnamespace " +
"JOIN pg_depend d ON d.objid = c.oid AND d.classid = 'pg_class'::regclass AND d.deptype = 'e' " +
"JOIN pg_extension e ON e.oid = d.refobjid";

/**
* The full "this schema is not the engine's own" test for one column: a fixed
* list of engine-builtin schemas, plus anything an extension created. Mirrors
* the provider's `schemaExclusion(column)`.
*/
function postgresSchemaExclusion(column: string): string {
return `${column} NOT IN (${POSTGRES_SYSTEM_SCHEMA_LIST}) AND ${column} NOT IN (${POSTGRES_EXTENSION_OWNED_SCHEMAS_SQL})`;
}

/**
* The column inventory, one row per TABLE with its columns aggregated (B52).
*
Expand Down Expand Up @@ -189,14 +272,42 @@ function equalsClause(column: string, value: string | undefined, field: string,
* bounds the prompt independently (`MAX_COLUMNS_PER_TABLE`,
* `AGENT_CONTEXT_PACK_MAX_CHARS`), so a wide table does not spend the context
* window on itself.
*
* This change closes B76: the aggregation that fixed the wide-catalog refusal
* then admitted the image's own objects. The WHERE clause below now excludes
* them two ways, both taken from the provider's object browser rather than
* invented here — the full engine-builtin schema list plus every schema an
* extension created, AND every RELATION an extension created (`pg_depend` with
* `classid = 'pg_class'::regclass`). The relation test is the load-bearing half
* on AlloyDB Omni, whose 67 extension views mostly sit in `public` itself: no
* schema filter can reach them, an ownership test can, and a user's own views
* are never extension-owned so they survive.
*
* Only the COLUMN read carries all three tests. The relation, index and
* statistics reads filter by schema alone, and that is complete rather than a
* gap: `buildPostgresTables` attaches a relation or index row to a table that
* is already in the column inventory, so a row for an extension-owned object
* the column read excluded has nothing to attach to and is dropped by the
* fold. The column read is the single source of object identity.
*
* Measured live on 2026-09-07 against the three `compat` images with two user
* tables seeded: the column read answers 46 → 2 object rows on TimescaleDB,
* 67 → 2 on Cloudberry and 70 → 2 on AlloyDB Omni. The sharp case is proven on
* AlloyDB, not assumed: the schema half alone leaves 51 objects (the 2 user
* tables plus the 49 extension views installed into `public`), `pg_depend`
* reports 68 extension-owned relations there, and the relation ownership test
* is what removes them. The committed tests pin the SQL SHAPE (the composed
* text carries each filter); the row counts above are the live BEHAVIOUR, and
* the two are asserted at different layers for that reason.
*/
function composePostgresCatalog(selector: AgentCatalogSelector): string {
return (
"SELECT table_schema, table_name, json_agg(json_build_object(" +
"'name', column_name, 'type', data_type, 'nullable', is_nullable) " +
"ORDER BY ordinal_position) AS columns " +
"FROM information_schema.columns " +
"WHERE table_schema NOT IN ('pg_catalog', 'information_schema')" +
`WHERE ${postgresSchemaExclusion("table_schema")}` +
` AND (table_schema, table_name) NOT IN (${POSTGRES_EXTENSION_OWNED_RELATIONS_SQL})` +

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The relation-ownership test is applied on the columns read only, which leaves the exclusion inconsistent across the reads a run makes:

  • kind=indexes and kind=statistics filter by schema alone and go to the model through readCatalog, not through buildPostgresTables
  • the object browser has no pg_class ownership test at all

Concretely, with PostGIS installed, public.spatial_ref_sys stays visible in the browser and in the index and statistics reads, but disappears from the column inventory. Same object, three different answers in one run. Worth either extending the test to the other kinds or recording why columns is the only place it belongs.

equalsClause("table_schema", selector.schema, "schema", "postgres") +
equalsClause("table_name", selector.table, "table", "postgres") +
" GROUP BY table_schema, table_name ORDER BY table_schema, table_name"
Expand Down Expand Up @@ -250,7 +361,8 @@ function composePostgresRelations(selector: AgentCatalogSelector): string {
"JOIN LATERAL unnest(c.conkey, c.confkey) WITH ORDINALITY AS k(attnum, fattnum, ord) ON true " +
"JOIN pg_attribute att ON att.attrelid = c.conrelid AND att.attnum = k.attnum " +
"JOIN pg_attribute fatt ON fatt.attrelid = c.confrelid AND fatt.attnum = k.fattnum " +
"WHERE c.contype = 'f' AND rn.nspname NOT IN ('pg_catalog', 'information_schema')" +
"WHERE c.contype = 'f' AND " +
postgresSchemaExclusion("rn.nspname") +
equalsClause("rn.nspname", selector.schema, "schema", "postgres") +
equalsClause("rel.relname", selector.table, "table", "postgres") +
" ORDER BY rn.nspname, rel.relname, k.ord"
Expand Down Expand Up @@ -302,7 +414,7 @@ function composePostgresIndexes(selector: AgentCatalogSelector): string {
"JOIN pg_namespace n ON n.oid = t.relnamespace " +
"JOIN LATERAL unnest(ix.indkey) WITH ORDINALITY AS k(attnum, ord) ON true " +
"LEFT JOIN pg_attribute att ON att.attrelid = t.oid AND att.attnum = k.attnum " +
"WHERE n.nspname NOT IN ('pg_catalog', 'information_schema', 'pg_toast')" +
`WHERE ${postgresSchemaExclusion("n.nspname")}` +
equalsClause("n.nspname", selector.schema, "schema", "postgres") +
equalsClause("t.relname", selector.table, "table", "postgres") +
" ORDER BY n.nspname, t.relname, i.relname, k.ord"
Expand Down Expand Up @@ -355,7 +467,8 @@ function composePostgresStatistics(selector: AgentCatalogSelector): string {
"JOIN pg_namespace n ON n.oid = c.relnamespace " +
"LEFT JOIN pg_stats s ON s.schemaname = n.nspname AND s.tablename = c.relname " +
"WHERE c.relkind IN ('r', 'p') " +
"AND n.nspname NOT IN ('pg_catalog', 'information_schema', 'pg_toast')" +
"AND " +
postgresSchemaExclusion("n.nspname") +
equalsClause("n.nspname", selector.schema, "schema", "postgres") +
equalsClause("c.relname", selector.table, "table", "postgres") +
" ORDER BY n.nspname, c.relname, s.attname"
Expand Down
80 changes: 79 additions & 1 deletion tests/unit/lib/agent/composed-sql.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ import {
MAX_CATALOG_SELECTOR_LENGTH,
} from "@/lib/agent/composed-sql";
import { agentReadSqlInput, inspectAgentStatement } from "@/lib/db/operations/statement-guard";
import { mkdtempSync, rmSync } from "node:fs";
import { mkdtempSync, readFileSync, rmSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { quoteLiteral } from "@/lib/sql/values";
Expand Down Expand Up @@ -82,6 +82,49 @@ describe("composeCatalogRead — PostgreSQL", () => {
// object, never as top-level select items.
expect(sql).not.toContain("column_name, data_type, is_nullable");
});

test("excludes the engine's own schemas the provider's browser already excludes (B76)", () => {
const sql = composeCatalogRead("postgres", {});

// The full engine-builtin list, not just pg_catalog/information_schema: a
// TimescaleDB chunk schema and Cloudberry's gp_toolkit are grounding noise too.
for (const schema of [
"pg_toast",
"mz_catalog",
"crdb_internal",
"pg_extension",
"_timescaledb_internal",
"gp_toolkit",
"pg_ext_aux",
]) {
expect(sql, schema).toContain(`'${schema}'`);
}
});

test("excludes schemas an extension created, by ownership and not by name (B76)", () => {
const sql = composeCatalogRead("postgres", {});

expect(sql).toContain("pg_depend");
expect(sql).toContain("'pg_namespace'::regclass");
expect(sql).toContain("deptype = 'e'");
});

test("excludes relations an extension created, so AlloyDB's public extension views drop out (B76)", () => {
const sql = composeCatalogRead("postgres", {});

// The sharp case: no schema filter reaches an object installed into `public`.
expect(sql).toContain("(table_schema, table_name) NOT IN");
expect(sql).toContain("'pg_class'::regclass");
expect(sql).toContain("deptype = 'e'");
});

test("does not use a blanket table_type filter, so a user's own views survive (B76)", () => {
const sql = composeCatalogRead("postgres", {});

// A table_type filter would also drop the user's views. The ownership test
// drops only what an extension created.
expect(sql).not.toContain("table_type");
});
});

describe("composeCatalogRead — SQLite", () => {
Expand Down Expand Up @@ -1015,3 +1058,38 @@ describe("the composers are reachable, which is the defect that put this file he
expect(() => composeEstimatingExplain("duckdb", "SELECT 1")).not.toThrow(AgentComposedSqlError);
});
});

describe("composeCatalogRead — the agent's exclusion set cannot drift from the provider's (B76)", () => {
const quotedNames = (block: string | undefined): string[] =>
block === undefined ? [] : [...block.matchAll(/"([^"]+)"/g)].map((match) => match[1]);

const readSource = (relativePath: string): string => readFileSync(join(process.cwd(), relativePath), "utf8");

test("POSTGRES_SYSTEM_SCHEMAS is the same list the provider ships", () => {
const providerSchemas = quotedNames(
/const SYSTEM_SCHEMAS = \[([\s\S]*?)\] as const;/.exec(readSource("src/lib/db/providers/sql/postgres.ts"))?.[1],
);
const agentSchemas = quotedNames(
/const POSTGRES_SYSTEM_SCHEMAS = \[([\s\S]*?)\] as const;/.exec(readSource("src/lib/agent/composed-sql.ts"))?.[1],
);

// Non-vacuity first: a regex that stops matching silently turns both lists
// empty and the assertion below into a tautology.
expect(providerSchemas.length).toBeGreaterThan(0);
expect(agentSchemas).toEqual(providerSchemas);
});

test("the extension-owned schema query is the provider's, verbatim", () => {
const providerQuery = /const EXTENSION_OWNED_SCHEMAS_SQL =([\s\S]*?);/.exec(
readSource("src/lib/db/providers/sql/postgres.ts"),
)?.[1];
const agentQuery = /const POSTGRES_EXTENSION_OWNED_SCHEMAS_SQL =([\s\S]*?);/.exec(
readSource("src/lib/agent/composed-sql.ts"),
)?.[1];

const normalize = (query: string | undefined): string => (query ?? "").replace(/\s+/g, " ").trim();

expect(normalize(providerQuery).length).toBeGreaterThan(0);
expect(normalize(agentQuery)).toBe(normalize(providerQuery));
});
});
Loading
Loading