diff --git a/docs/CURSOR_PAGINATION.md b/docs/CURSOR_PAGINATION.md new file mode 100644 index 0000000..8d2299f --- /dev/null +++ b/docs/CURSOR_PAGINATION.md @@ -0,0 +1,194 @@ +# Cursor pagination + +AnchorNet collection endpoints use cursor pagination by default. This keeps a +consumer from repeatedly scanning and counting a growing collection, and gives +callers a stable traversal boundary while new records are being added. + +## Endpoints + +The following collection reads support the cursor contract: + +| Endpoint | Response collection | Canonical order | +| --- | --- | --- | +| `GET /api/v1/anchors` | `anchors` | `id` ascending | +| `GET /api/v1/anchors/:id/settlements` | `settlements` | `id` descending | +| `GET /api/v1/liquidity` | `pools` | `asset` ascending | +| `GET /api/v1/liquidity/entries` | `entries` | `anchor`, then `asset` ascending | +| `GET /api/v1/liquidity/withdrawals` | `withdrawals` | timestamp ascending, insertion index tie-breaker | +| `GET /api/v1/liquidity/anchors/:anchor` | `entries` | `asset` ascending | +| `GET /api/v1/settlements` | `settlements` | `id` descending | + +Each response retains its existing collection property and adds a pagination +sibling: + +```json +{ + "settlements": [], + "pagination": { + "pageSize": 20, + "nextCursor": "eyJ2ZXJzaW9uIjoxLCJkaXJlY3Rpb24iOiJkZXNj..." + } +} +``` + +The cursor is `null` when the page is the end of the collection. An empty +collection also returns `nextCursor: null`. + +## Request parameters + +`pageSize` is optional and defaults to `20`. Values greater than `100` are +clamped to `100`, which bounds the amount of work and response data per call. +Values must be positive integers. Decimals, negative numbers, exponents, and +non-numeric strings receive a `400` response. + +`cursor` is optional on the first request. Follow-up requests pass the exact +opaque `nextCursor` value returned by the preceding response: + +```text +GET /api/v1/settlements?pageSize=25 +GET /api/v1/settlements?pageSize=25&cursor= +``` + +The cursor is intentionally opaque. Clients must not decode it, construct it, +or depend on its current encoding. The current encoding includes a version, +direction, ordering boundary, and snapshot boundary so that the server can +reject a malformed cursor rather than silently skipping records. + +## Snapshot behavior + +The first page establishes a boundary at the first item in the canonical +ordering. A descending settlement traversal therefore excludes settlements +created after the first request, while an ascending anchor traversal excludes +anchors inserted before the first anchor in that traversal. This prevents a +consumer walking several pages from seeing a newly inserted record move an old +record onto an already-read page. + +The snapshot boundary is not a database transaction and does not freeze updates +to existing objects. It is a traversal boundary for the collection ordering. +Records deleted between requests may disappear, which is preferable to +returning stale records that no longer exist. + +## Ordering guarantees + +Every cursor collection has one canonical order and a unique key: + +1. anchors use the anchor id; +2. settlements use the numeric settlement id; +3. pools use the asset code; +4. global entries use the compound `(anchor, asset)` key; +5. anchor-scoped entries use the asset code; +6. withdrawals use `(timestamp, insertion index)`. + +The unique tie-breaker is important for timestamps because multiple successful +withdrawals may be recorded during one clock tick. The tie-breaker makes the +cursor advance past exactly one record instead of skipping all records sharing +the same visible timestamp. + +## Filters and scopes + +Settlement cursors include the requested `anchor` and `asset` filters. Anchor +cursors include `status` and `q`. A cursor from one filtered collection cannot +be reused for another collection or filter set; the API returns `400` when the +scope does not match. + +Anchor settlement cursors are scoped to their anchor id. Liquidity entry cursors +are scoped to the global entries collection or to the requested anchor. This +prevents an opaque value from accidentally being accepted by a different route. + +## Canonical order versus legacy sorting + +Existing offset pagination remains available for clients that send `page`, and +existing custom sorting remains available with that offset mode. A cursor request +must use the canonical order. Combining `cursor` with `sort` or `order` returns +`400` because the visible ordering would no longer match the cursor key. + +The legacy shape includes `page`, `pageSize`, `total`, and `totalPages` in its +pagination object. Cursor mode intentionally reports only `pageSize` and +`nextCursor`; computing a total would reintroduce the full-collection scan that +cursor pagination is designed to avoid. + +CSV exports remain full, sorted exports and ignore both pagination modes. This +keeps exports useful for operators while collection reads stay bounded. + +## Client traversal algorithm + +Clients should process each page before requesting the next one: + +```text +cursor = absent +repeat: + response = GET collection with pageSize and cursor + process response.items + cursor = response.pagination.nextCursor +until cursor is null +``` + +Clients should stop when `nextCursor` is `null`, not when the number of returned +items is less than `pageSize`. The latter is usually equivalent but does not +describe the server contract and is unsafe for future page-size policies. + +If a cursor is malformed, expired by a future server version, or used with a +different filter, restart the traversal without a cursor. Do not retry the same +invalid value indefinitely. + +## Error handling + +Malformed cursors return the regular API error envelope with HTTP `400`: + +```json +{ + "error": { + "code": "BAD_REQUEST", + "message": "cursor is malformed or expired" + } +} +``` + +The same response status is used for an incompatible direction, scope, or +page-size value. The API does not expose cursor internals in the error message. + +## Database migration path + +The current implementation applies the canonical ordering to service results +before passing them to the cursor helper. A future database-backed repository +can map the same keys to a `WHERE` predicate and `ORDER BY` clause: + +* ascending traversal uses `key > after` and `key >= snapshot`; +* descending traversal uses `key < after` and `key <= snapshot`; +* compound keys must use the same lexicographic tuple order; +* the limit is `pageSize + 1` so the repository can determine `nextCursor`. + +The application-level contract therefore does not depend on an in-memory +implementation. Repository adapters must preserve canonical ordering and the +same unique tie-breakers when they move filtering into SQL. + +## Operational notes + +The maximum page size is deliberately small enough for routine API calls and +large enough for normal dashboard batches. It is a server-side clamp, so an +untrusted caller cannot request an unbounded page by sending a very large +number. + +The cursor helper validates unique keys in development and test paths. A +duplicate key is a programming error because it would make continuation +ambiguous; it is surfaced instead of producing a subtly incomplete traversal. + +The helper never sorts or mutates its input. Routes own filtering and canonical +ordering, while the helper owns page-size validation, opaque cursor encoding, +snapshot filtering, continuation, and cursor validation. Keeping those +responsibilities separate makes the behavior easy to test and portable to a +database repository. + +## Compatibility checklist + +When adding a new cursor collection: + +1. choose a deterministic canonical order; +2. add a unique key or compound key with a stable tie-breaker; +3. add a scope containing every filter that affects membership; +4. preserve the existing collection property in the JSON envelope; +5. document `cursor`, `pageSize`, order, and response pagination; +6. add tests for first page, continuation, empty data, malformed cursors, and + records inserted between pages; +7. retain CSV and legacy offset behavior when compatibility requires it. + diff --git a/src/config.test.ts b/src/config.test.ts index 134d8ed..b4e3fdb 100644 --- a/src/config.test.ts +++ b/src/config.test.ts @@ -259,9 +259,8 @@ describe("validateConfig", () => { }); it("requires API_KEY in production and fails fast", () => { - const config = loadConfig({ NODE_ENV: "production" }); - expect(() => validateConfig(config)).toThrow(ConfigValidationError); - expect(() => validateConfig(config)).toThrow(/API_KEY is required/); + expect(() => loadConfig({ NODE_ENV: "production" })).toThrow(ConfigValidationError); + expect(() => loadConfig({ NODE_ENV: "production" })).toThrow(/API_KEY is required/); }); it("allows a production deploy that sets API_KEY", () => { @@ -280,23 +279,19 @@ describe("validateConfig", () => { }); it("fails fast on an out-of-range PORT", () => { - const config = loadConfig({ PORT: "0" }); - expect(() => validateConfig(config)).toThrow(ConfigValidationError); - expect(() => validateConfig(config)).toThrow(/PORT must be/); + expect(() => loadConfig({ PORT: "0" })).toThrow(ConfigValidationError); + expect(() => loadConfig({ PORT: "0" })).toThrow(/PORT must be/); }); it("fails fast on a non-integer PORT", () => { - const config = loadConfig({ PORT: "3001.5" }); - expect(() => validateConfig(config)).toThrow(ConfigValidationError); + expect(() => loadConfig({ PORT: "3001.5" })).toThrow(ConfigValidationError); }); it("fails fast on a negative RATE_LIMIT_MAX", () => { - const config = loadConfig({ RATE_LIMIT_MAX: "-1" }); - expect(() => validateConfig(config)).toThrow(ConfigValidationError); + expect(() => loadConfig({ RATE_LIMIT_MAX: "-1" })).toThrow(ConfigValidationError); }); it("fails fast on a negative IDEMPOTENCY_TTL_MS", () => { - const config = loadConfig({ IDEMPOTENCY_TTL_MS: "-1" }); - expect(() => validateConfig(config)).toThrow(ConfigValidationError); + expect(() => loadConfig({ IDEMPOTENCY_TTL_MS: "-1" })).toThrow(ConfigValidationError); }); }); diff --git a/src/config.ts b/src/config.ts index 87ab498..7e234b7 100644 --- a/src/config.ts +++ b/src/config.ts @@ -133,20 +133,6 @@ function parseTrustProxy(value: string | undefined): boolean | string | number { return trimmed; } -/** - * Error thrown when a required configuration value is missing or invalid. - * Carries the offending variable name so the message can name it directly - * (see {@link validateConfig}). - */ -export class ConfigValidationError extends Error { - readonly variable: string; - constructor(variable: string, message: string) { - super(message); - this.name = "ConfigValidationError"; - this.variable = variable; - } -} - /** * Fail-fast configuration contract. * @@ -165,18 +151,6 @@ export class ConfigValidationError extends Error { * separate `apiKeyAuth` issue. Here we only guarantee the deployment * visibly refuses to start instead of silently running unauthenticated. */ -export function validateConfig(config: Config): Config { - if (config.env === "production" && !config.apiKey) { - throw new ConfigValidationError( - "API_KEY", - "API_KEY is required when NODE_ENV=production. Without it, mutating " + - "endpoints are open to unauthenticated access (see src/middleware/apiKeyAuth.ts). " + - "Set API_KEY to a secret value, or run with NODE_ENV=development for local open access.", - ); - } - return config; -} - /** Builds the {@link Config} from `process.env`, applying sensible defaults. */ export function loadConfig( env: Record = process.env, @@ -240,9 +214,12 @@ export function loadConfig( * contract easy to review. */ export class ConfigValidationError extends Error { - constructor(message: string) { + readonly variable: string; + + constructor(message: string, variable: string) { super(message); this.name = "ConfigValidationError"; + this.variable = variable; } } @@ -250,6 +227,7 @@ export function validateConfig(config: Config): Config { if (config.env === "production" && !config.apiKey) { throw new ConfigValidationError( "API_KEY is required when NODE_ENV=production. Refusing to start with open (unauthenticated) mutating access. Set API_KEY to enable API-key authentication.", + "API_KEY", ); } @@ -261,18 +239,21 @@ export function validateConfig(config: Config): Config { ) { throw new ConfigValidationError( `PORT must be an integer between 1 and 65535 (got ${String(config.port)})`, + "PORT", ); } if (config.rateLimitMax < 0) { throw new ConfigValidationError( `RATE_LIMIT_MAX must be >= 0 (got ${config.rateLimitMax})`, + "RATE_LIMIT_MAX", ); } if (config.idempotencyTtlMs < 0) { throw new ConfigValidationError( `IDEMPOTENCY_TTL_MS must be >= 0 (got ${config.idempotencyTtlMs})`, + "IDEMPOTENCY_TTL_MS", ); } diff --git a/src/openapi.ts b/src/openapi.ts index 5fd754c..d754d30 100644 --- a/src/openapi.ts +++ b/src/openapi.ts @@ -59,7 +59,10 @@ export function buildOpenApiSpec(): Record { get: { summary: "List aggregated liquidity pools", description: - "Returns an array of Pool objects, each containing asset, total, anchors count, and a lastUpdated timestamp.", + "Returns { pools: [...], pagination: { pageSize, nextCursor } }. " + + "Cursor mode is the default, ordered by asset ascending; pageSize defaults to 20 and is clamped to 100. " + + "Pass the opaque nextCursor to continue. Legacy page-based pagination remains available when page is supplied.", + parameters: ["cursor", "pageSize", "page"], }, }, "/api/v1/liquidity/withdraw": { @@ -82,7 +85,9 @@ export function buildOpenApiSpec(): Record { "Returns { entries: [...] }. This static path is registered before the " + "catch-all GET /api/v1/liquidity/{asset}; that ordering is load-bearing, " + "since reversing it would make this path resolve as a pool lookup for an " + - 'asset named "ENTRIES".', + 'asset named "ENTRIES". Cursor mode is ordered by anchor and asset and ' + + "adds pagination.pageSize and pagination.nextCursor.", + parameters: ["cursor", "pageSize"], }, }, @@ -93,11 +98,17 @@ export function buildOpenApiSpec(): Record { "Read-only audit trail of withdrawals recorded by POST /api/v1/liquidity/withdraw. " + "Each entry records the anchor, asset, amount withdrawn, the anchor's resulting " + "balance, and an ISO-8601 timestamp, and persists even after an entry is removed " + - "once its balance reaches zero. Bounded to the most recent records.", + "once its balance reaches zero. Bounded to the most recent records. Cursor mode " + + "is ordered oldest-first by timestamp with an insertion-index tie-breaker.", + parameters: ["cursor", "pageSize"], }, }, "/api/v1/liquidity/anchors/{anchor}": { - get: { summary: "List raw liquidity entries for a single anchor" }, + get: { + summary: "List raw liquidity entries for a single anchor", + description: "Returns { entries: [...], pagination: { pageSize, nextCursor } }, ordered by asset ascending.", + parameters: ["cursor", "pageSize"], + }, }, "/api/v1/liquidity/{asset}": { get: { @@ -127,7 +138,8 @@ export function buildOpenApiSpec(): Record { post: { summary: "Register an anchor" }, get: { summary: "List anchors", - parameters: ["status", "q", "sort", "order", "format"], + description: "Cursor mode is the default and orders anchors by id ascending. The status and q filters are part of the cursor scope.", + parameters: ["status", "q", "sort", "order", "format", "cursor", "pageSize", "page"], }, }, "/api/v1/anchors/{id}": { @@ -165,7 +177,7 @@ export function buildOpenApiSpec(): Record { description: "Returns the same paginated settlement list as GET /api/v1/settlements?anchor={id}, " + "but scoped to the anchor identified by :id. Returns 404 if the anchor does not exist.", - parameters: ["sort", "order", "page", "pageSize", "format"], + parameters: ["sort", "order", "page", "pageSize", "format", "cursor"], }, }, "/api/v1/settlements": { @@ -180,7 +192,9 @@ export function buildOpenApiSpec(): Record { "page", "pageSize", "format", + "cursor", ], + description: "Cursor mode is the default, ordered by settlement id descending. Filter values are part of the cursor scope; legacy page pagination remains available.", }, }, "/api/v1/settlements/{id}": { diff --git a/src/routes/anchors.ts b/src/routes/anchors.ts index 2f60f6f..ddb0df6 100644 --- a/src/routes/anchors.ts +++ b/src/routes/anchors.ts @@ -9,8 +9,10 @@ import { Anchor } from "../models/anchor"; import { Settlement } from "../models/settlement"; import { applySort } from "../utils/sorting"; import { paginate } from "../utils/pagination"; +import { paginateByCursor } from "../utils/cursorPagination"; import { csvColumnsFor, toCsv } from "../utils/csv"; import { optionalBooleanFlag } from "../utils/validation"; +import { ApiError } from "../errors/ApiError"; const SORTABLE_FIELDS = ["id", "name", "registeredAt"]; @@ -79,20 +81,40 @@ export function anchorRouter( // List anchors, optionally filtered via ?status=active|inactive and/or a // free-text ?q= search over id/name, sorted via ?sort=id|name|registeredAt - // and ?order=asc|desc, and exported as CSV via ?format=csv. + // and ?order=asc|desc, and exported as CSV via ?format=csv. Without legacy + // offset parameters, the endpoint uses the canonical id-ascending cursor. router.get("/", (req: Request, res: Response) => { - const anchors = applySort( - service.list({ status: req.query.status, q: req.query.q }), - { sort: req.query.sort, order: req.query.order }, - SORTABLE_FIELDS, - ); + const hasCustomSort = req.query.sort !== undefined || req.query.order !== undefined; + const filtered = service.list({ status: req.query.status, q: req.query.q }); + const sorted = hasCustomSort + ? applySort(filtered, { sort: req.query.sort, order: req.query.order }, SORTABLE_FIELDS) + : [...filtered].sort((a, b) => a.id.localeCompare(b.id)); if (req.query.format === "csv") { - res.type("text/csv").send(toCsv(anchors, CSV_COLUMNS)); + res.type("text/csv").send(toCsv(sorted, CSV_COLUMNS)); return; } - res.json({ anchors }); + const useCursor = req.query.cursor !== undefined || + (!hasCustomSort && req.query.page === undefined); + if (useCursor && hasCustomSort) { + throw ApiError.badRequest("cursor pagination requires the canonical anchor order (id asc)"); + } + + if (useCursor) { + const page = paginateByCursor(sorted, { + cursor: req.query.cursor, + pageSize: req.query.pageSize, + direction: "asc", + scope: `anchors:${String(req.query.status ?? "")}:\u0000${String(req.query.q ?? "")}`, + keyOf: (anchor) => anchor.id, + }); + res.json({ anchors: page.items, pagination: { pageSize: page.pageSize, nextCursor: page.nextCursor } }); + return; + } + + const page = paginate(sorted, { page: req.query.page, pageSize: req.query.pageSize }); + res.json({ anchors: page.items, pagination: { ...page, items: undefined } }); }); // Read a single anchor by id. @@ -132,11 +154,11 @@ export function anchorRouter( return; } - const sorted = applySort( - settlements.list({ anchor: req.params.id }), - { sort: req.query.sort, order: req.query.order }, - SETTLEMENT_SORTABLE_FIELDS, - ); + const hasCustomSort = req.query.sort !== undefined || req.query.order !== undefined; + const rawSettlements = settlements.list({ anchor: req.params.id }); + const sorted = hasCustomSort + ? applySort(rawSettlements, { sort: req.query.sort, order: req.query.order }, SETTLEMENT_SORTABLE_FIELDS) + : [...rawSettlements].sort((a, b) => b.id - a.id); // CSV export ignores pagination and returns every matching, sorted row. if (req.query.format === "csv") { @@ -145,6 +167,27 @@ export function anchorRouter( return; } + const useCursor = req.query.cursor !== undefined || + (!hasCustomSort && req.query.page === undefined); + if (useCursor && hasCustomSort) { + throw ApiError.badRequest("cursor pagination requires the canonical settlement order (id desc)"); + } + + if (useCursor) { + const page = paginateByCursor(sorted, { + cursor: req.query.cursor, + pageSize: req.query.pageSize, + direction: "desc", + scope: `anchor-settlements:${req.params.id}`, + keyOf: (settlement) => String(settlement.id).padStart(20, "0"), + }); + res.json({ + settlements: page.items.map(serializeSettlement), + pagination: { pageSize: page.pageSize, nextCursor: page.nextCursor }, + }); + return; + } + const page = paginate(sorted, { page: req.query.page, pageSize: req.query.pageSize, diff --git a/src/routes/cursorPagination.test.ts b/src/routes/cursorPagination.test.ts new file mode 100644 index 0000000..f279e39 --- /dev/null +++ b/src/routes/cursorPagination.test.ts @@ -0,0 +1,183 @@ +import request from "supertest"; +import { createApp } from "../app"; + +async function addAnchor(app: ReturnType, id: string): Promise { + const response = await request(app).post("/api/v1/anchors").send({ id, name: id }); + expect(response.status).toBe(201); +} + +async function addLiquidity( + app: ReturnType, + anchor: string, + asset: string, + amount = "1000", +): Promise { + const response = await request(app) + .post("/api/v1/liquidity") + .send({ anchor, asset, amount }); + expect(response.status).toBe(201); +} + +async function addSettlement( + app: ReturnType, + anchor: string, + asset = "USDC", +): Promise { + const response = await request(app) + .post("/api/v1/settlements") + .send({ anchor, asset, amount: "1" }); + expect(response.status).toBe(201); + return response.body.id as number; +} + +describe("cursor pagination route contract", () => { + it("walks every anchor exactly once in canonical order", async () => { + const app = createApp(); + for (const id of ["anchor-e", "anchor-a", "anchor-d", "anchor-b", "anchor-c"]) { + await addAnchor(app, id); + } + + const seen: string[] = []; + let cursor: string | undefined; + do { + const query = request(app).get("/api/v1/anchors").query({ pageSize: 2 }); + if (cursor) query.query({ cursor }); + const response = await query; + expect(response.status).toBe(200); + seen.push(...response.body.anchors.map((anchor: { id: string }) => anchor.id)); + cursor = response.body.pagination.nextCursor ?? undefined; + } while (cursor); + + expect(seen).toEqual(["anchor-a", "anchor-b", "anchor-c", "anchor-d", "anchor-e"]); + expect(new Set(seen).size).toBe(5); + }); + + it("keeps an anchor cursor scoped to its filters", async () => { + const app = createApp(); + await addAnchor(app, "active-anchor"); + await addAnchor(app, "active-anchor-two"); + await addAnchor(app, "inactive-anchor"); + await request(app).delete("/api/v1/anchors/inactive-anchor"); + + const first = await request(app) + .get("/api/v1/anchors") + .query({ status: "active", pageSize: 1 }); + expect(first.status).toBe(200); + expect(first.body.pagination.nextCursor).toEqual(expect.any(String)); + + const wrongScope = await request(app) + .get("/api/v1/anchors") + .query({ status: "inactive", cursor: first.body.pagination.nextCursor }); + expect(wrongScope.status).toBe(400); + expect(wrongScope.body.error.code).toBe("BAD_REQUEST"); + }); + + it("rejects custom sorting when a cursor is supplied", async () => { + const app = createApp(); + await addAnchor(app, "anchor-a"); + const response = await request(app) + .get("/api/v1/anchors") + .query({ cursor: "bad", sort: "name" }); + expect(response.status).toBe(400); + expect(response.body.error.message).toMatch(/canonical anchor order/); + }); + + it("walks settlements in descending id order", async () => { + const app = createApp(); + await addAnchor(app, "settlement-anchor"); + await addLiquidity(app, "settlement-anchor", "USDC", "10"); + await addSettlement(app, "settlement-anchor"); + await addSettlement(app, "settlement-anchor"); + await addSettlement(app, "settlement-anchor"); + + const first = await request(app) + .get("/api/v1/settlements") + .query({ pageSize: 2 }); + expect(first.body.settlements.map((settlement: { id: number }) => settlement.id)).toEqual([3, 2]); + + const second = await request(app) + .get("/api/v1/settlements") + .query({ pageSize: 2, cursor: first.body.pagination.nextCursor }); + expect(second.status).toBe(200); + expect(second.body.settlements.map((settlement: { id: number }) => settlement.id)).toEqual([1]); + expect(second.body.pagination.nextCursor).toBeNull(); + }); + + it("does not move a newly inserted settlement into an existing traversal", async () => { + const app = createApp(); + await addAnchor(app, "snapshot-anchor"); + await addLiquidity(app, "snapshot-anchor", "USDC", "10"); + await addSettlement(app, "snapshot-anchor"); + await addSettlement(app, "snapshot-anchor"); + await addSettlement(app, "snapshot-anchor"); + + const first = await request(app) + .get("/api/v1/settlements") + .query({ pageSize: 1 }); + await addSettlement(app, "snapshot-anchor"); + + const second = await request(app) + .get("/api/v1/settlements") + .query({ pageSize: 10, cursor: first.body.pagination.nextCursor }); + expect(second.body.settlements.map((settlement: { id: number }) => settlement.id)).toEqual([2, 1]); + }); + + it("rejects a cursor from one settlement filter on another", async () => { + const app = createApp(); + await addAnchor(app, "filter-anchor"); + await addLiquidity(app, "filter-anchor", "USDC", "10"); + await addSettlement(app, "filter-anchor"); + await addSettlement(app, "filter-anchor"); + + const first = await request(app) + .get("/api/v1/settlements") + .query({ anchor: "filter-anchor", pageSize: 1 }); + const wrongScope = await request(app) + .get("/api/v1/settlements") + .query({ asset: "USDC", cursor: first.body.pagination.nextCursor }); + expect(wrongScope.status).toBe(400); + }); + + it("adds cursor pagination to every liquidity collection envelope", async () => { + const app = createApp(); + await addLiquidity(app, "liquidity-a", "USDC", "10"); + await addLiquidity(app, "liquidity-b", "EURC", "20"); + await request(app) + .post("/api/v1/liquidity/withdraw") + .send({ anchor: "liquidity-a", asset: "USDC", amount: "1" }); + + const endpoints = [ + ["/api/v1/liquidity", "pools"], + ["/api/v1/liquidity/entries", "entries"], + ["/api/v1/liquidity/withdrawals", "withdrawals"], + ["/api/v1/liquidity/anchors/liquidity-a", "entries"], + ] as const; + + for (const [endpoint, collection] of endpoints) { + const response = await request(app).get(endpoint).query({ pageSize: 1 }); + expect(response.status).toBe(200); + expect(response.body[collection]).toHaveLength(1); + expect(response.body.pagination.pageSize).toBe(1); + expect(response.body.pagination.nextCursor === null || typeof response.body.pagination.nextCursor === "string").toBe(true); + } + }); + + it("rejects malformed cursors without exposing cursor internals", async () => { + const app = createApp(); + const response = await request(app) + .get("/api/v1/liquidity") + .query({ cursor: "not-a-cursor" }); + expect(response.status).toBe(400); + expect(response.body.error.message).toBe("cursor is malformed or expired"); + expect(response.body.error.message).not.toMatch(/snapshot|after|base64/); + }); + + it("clamps route page sizes to one hundred", async () => { + const app = createApp(); + const response = await request(app) + .get("/api/v1/anchors") + .query({ pageSize: 999999 }); + expect(response.status).toBe(200); + expect(response.body.pagination.pageSize).toBe(100); + }); +}); diff --git a/src/routes/liquidity.test.ts b/src/routes/liquidity.test.ts index 230df1b..b04d580 100644 --- a/src/routes/liquidity.test.ts +++ b/src/routes/liquidity.test.ts @@ -228,7 +228,8 @@ describe("liquidity routes", () => { expect(res.status).toBe(200); expect(Array.isArray(res.body.entries)).toBe(true); - expect(res.body).toEqual({ entries: [] }); + expect(res.body.entries).toEqual([]); + expect(res.body.pagination).toEqual({ pageSize: 20, nextCursor: null }); expect(res.body).not.toHaveProperty("asset"); expect(res.body).not.toHaveProperty("total"); expect(res.body).not.toHaveProperty("error"); diff --git a/src/routes/liquidity.ts b/src/routes/liquidity.ts index c5c4fb6..a278c0d 100644 --- a/src/routes/liquidity.ts +++ b/src/routes/liquidity.ts @@ -5,6 +5,7 @@ import { Router, Request, Response } from "express"; import { ApiError } from "../errors/ApiError"; import { LiquidityService } from "../services/liquidityService"; +import { paginateByCursor } from "../utils/cursorPagination"; export function liquidityRouter(service: LiquidityService): Router { const router = Router(); @@ -55,8 +56,19 @@ export function liquidityRouter(service: LiquidityService): Router { }); // List aggregated pools across all assets. - router.get("/", (_req: Request, res: Response) => { - res.json({ pools: service.listPools().map(p => ({ ...p, total: p.total.toString() })) }); + router.get("/", (req: Request, res: Response) => { + const pools = service.listPools(); + const page = paginateByCursor(pools, { + cursor: req.query.cursor, + pageSize: req.query.pageSize, + direction: "asc", + scope: "liquidity-pools", + keyOf: (pool) => pool.asset, + }); + res.json({ + pools: page.items.map(p => ({ ...p, total: p.total.toString() })), + pagination: { pageSize: page.pageSize, nextCursor: page.nextCursor }, + }); }); // --------------------------------------------------------------------- @@ -65,13 +77,37 @@ export function liquidityRouter(service: LiquidityService): Router { // --------------------------------------------------------------------- // List raw per-anchor entries. Registered before the catch-all GET /:asset - router.get("/entries", (_req: Request, res: Response) => { - res.json({ entries: service.listEntries().map(e => ({ ...e, amount: e.amount.toString() })) }); + router.get("/entries", (req: Request, res: Response) => { + const entries = service.listEntries().sort((a, b) => + `${a.anchor}\u0000${a.asset}`.localeCompare(`${b.anchor}\u0000${b.asset}`), + ); + const page = paginateByCursor(entries, { + cursor: req.query.cursor, + pageSize: req.query.pageSize, + direction: "asc", + scope: "liquidity-entries", + keyOf: (entry) => `${entry.anchor}\u0000${entry.asset}`, + }); + res.json({ + entries: page.items.map(e => ({ ...e, amount: e.amount.toString() })), + pagination: { pageSize: page.pageSize, nextCursor: page.nextCursor }, + }); }); // Read-only audit trail of successful withdrawals. - router.get("/withdrawals", (_req: Request, res: Response) => { - res.json({ withdrawals: service.listWithdrawals().map(w => ({ ...w, amount: w.amount.toString(), remainingBalance: w.remainingBalance.toString() })) }); + router.get("/withdrawals", (req: Request, res: Response) => { + const withdrawals = service.listWithdrawals(); + const page = paginateByCursor(withdrawals, { + cursor: req.query.cursor, + pageSize: req.query.pageSize, + direction: "asc", + scope: "liquidity-withdrawals", + keyOf: (withdrawal, index) => `${withdrawal.timestamp}\u0000${String(index).padStart(12, "0")}`, + }); + res.json({ + withdrawals: page.items.map(w => ({ ...w, amount: w.amount.toString(), remainingBalance: w.remainingBalance.toString() })), + pagination: { pageSize: page.pageSize, nextCursor: page.nextCursor }, + }); }); // Force-remove an anchor's entire liquidity entry for an asset. @@ -82,7 +118,18 @@ export function liquidityRouter(service: LiquidityService): Router { // Read the raw liquidity entries for a single anchor. router.get("/anchors/:anchor", (req: Request, res: Response) => { - res.json({ entries: service.listByAnchor(req.params.anchor).map(e => ({ ...e, amount: e.amount.toString() })) }); + const entries = service.listByAnchor(req.params.anchor).sort((a, b) => a.asset.localeCompare(b.asset)); + const page = paginateByCursor(entries, { + cursor: req.query.cursor, + pageSize: req.query.pageSize, + direction: "asc", + scope: `liquidity-anchor:${req.params.anchor}`, + keyOf: (entry) => entry.asset, + }); + res.json({ + entries: page.items.map(e => ({ ...e, amount: e.amount.toString() })), + pagination: { pageSize: page.pageSize, nextCursor: page.nextCursor }, + }); }); // Read the aggregated pool for a single asset. diff --git a/src/routes/settlements.test.ts b/src/routes/settlements.test.ts index aad0650..dc37fd3 100644 --- a/src/routes/settlements.test.ts +++ b/src/routes/settlements.test.ts @@ -578,12 +578,12 @@ describe("GET /api/v1/settlements — pagination validation (#108)", () => { expect(res.body.error.code).toBe("BAD_REQUEST"); }); - it("returns 200 with default page for omitted params", async () => { + it("returns 200 with the default cursor page for omitted params", async () => { const app = createApp(); const res = await request(app).get("/api/v1/settlements"); expect(res.status).toBe(200); - expect(res.body.pagination.page).toBe(1); expect(res.body.pagination.pageSize).toBe(20); + expect(res.body.pagination.nextCursor).toBeNull(); }); it("returns 200 with default page for empty params", async () => { diff --git a/src/routes/settlements.ts b/src/routes/settlements.ts index 4ac200a..cc54eba 100644 --- a/src/routes/settlements.ts +++ b/src/routes/settlements.ts @@ -9,6 +9,8 @@ import { AuditEntry } from "../middleware/auditLog"; import { paginate } from "../utils/pagination"; import { applySort } from "../utils/sorting"; import { csvColumnsFor, toCsv } from "../utils/csv"; +import { paginateByCursor } from "../utils/cursorPagination"; +import { ApiError } from "../errors/ApiError"; const SORTABLE_FIELDS = ["id", "amount", "fee", "status", "createdAt"]; @@ -39,8 +41,8 @@ export function settlementRouter( res.status(201).json({ ...s, amount: Number(s.amount), fee: Number(s.fee) }); }); - // List settlements, optionally filtered by ?anchor= and ?asset=, sorted via - // ?sort= and ?order=, and paginated via ?page= and ?pageSize=. + // List settlements, optionally filtered by ?anchor= and ?asset=. The default + // is a bounded id-descending cursor page; ?page= keeps the legacy offset API. router.get("/", (req: Request, res: Response) => { const anchor = typeof req.query.anchor === "string" ? req.query.anchor : undefined; @@ -56,6 +58,7 @@ export function settlementRouter( const sortOrder = typeof req.query.order === "string" ? req.query.order : "asc"; + const hasCustomSort = req.query.sort !== undefined || req.query.order !== undefined; let sorted: typeof raw; // Use BigInt comparison for amount and fee to avoid lexicographic ordering @@ -68,12 +71,14 @@ export function settlementRouter( const bv = BigInt(b[field]); return av < bv ? -dir : av > bv ? dir : 0; }); - } else { + } else if (hasCustomSort) { sorted = applySort( raw, { sort: req.query.sort, order: req.query.order }, SORTABLE_FIELDS, ); + } else { + sorted = [...raw].sort((a, b) => b.id - a.id); } // CSV export ignores pagination and returns every matching, sorted row. @@ -83,6 +88,27 @@ export function settlementRouter( return; } + const useCursor = req.query.cursor !== undefined || + (!hasCustomSort && req.query.page === undefined); + if (useCursor && hasCustomSort) { + throw ApiError.badRequest("cursor pagination requires the canonical settlement order (id desc)"); + } + + if (useCursor) { + const page = paginateByCursor(sorted, { + cursor: req.query.cursor, + pageSize: req.query.pageSize, + direction: "desc", + scope: `settlements:${anchor ?? "all"}:${asset ?? "all"}`, + keyOf: (settlement) => String(settlement.id).padStart(20, "0"), + }); + res.json({ + settlements: page.items.map(s => ({ ...s, amount: Number(s.amount), fee: Number(s.fee) })), + pagination: { pageSize: page.pageSize, nextCursor: page.nextCursor }, + }); + return; + } + const page = paginate(sorted, { page: req.query.page, pageSize: req.query.pageSize, diff --git a/src/utils/cursorPagination.test.ts b/src/utils/cursorPagination.test.ts new file mode 100644 index 0000000..57e23b0 --- /dev/null +++ b/src/utils/cursorPagination.test.ts @@ -0,0 +1,192 @@ +import { ApiError } from "../errors/ApiError"; +import { + CURSOR_DEFAULT_PAGE_SIZE, + CURSOR_MAX_PAGE_SIZE, + paginateByCursor, +} from "./cursorPagination"; + +interface Row { + id: number; + createdAt: string; +} + +const rows = (count: number): Row[] => Array.from({ length: count }, (_, index) => ({ + id: count - index, + createdAt: `2026-08-${String((index % 9) + 1).padStart(2, "0")}`, +})); + +const descKey = (row: Row) => String(row.id).padStart(8, "0"); + +describe("paginateByCursor", () => { + it("returns the default bounded page and an opaque cursor", () => { + const page = paginateByCursor(rows(25), { + keyOf: descKey, + direction: "desc", + scope: "rows", + }); + + expect(page.items).toHaveLength(CURSOR_DEFAULT_PAGE_SIZE); + expect(page.pageSize).toBe(CURSOR_DEFAULT_PAGE_SIZE); + expect(page.nextCursor).toEqual(expect.any(String)); + expect(page.nextCursor).not.toContain("snapshot"); + expect(page.nextCursor).not.toContain("after"); + }); + + it("clamps an oversized page to the operational maximum", () => { + const page = paginateByCursor(rows(150), { + pageSize: "999999", + keyOf: descKey, + direction: "desc", + }); + + expect(page.items).toHaveLength(CURSOR_MAX_PAGE_SIZE); + expect(page.pageSize).toBe(CURSOR_MAX_PAGE_SIZE); + }); + + it("rejects malformed and empty cursors", () => { + for (const cursor of ["nope", "", "%%%", "eyJmb28iOiJiYXIifQ"]) { + expect(() => paginateByCursor(rows(2), { + cursor, + keyOf: descKey, + direction: "desc", + })).toThrow(ApiError); + } + }); + + it("rejects a cursor used with another direction or scope", () => { + const first = paginateByCursor(rows(3), { + pageSize: 1, + keyOf: descKey, + direction: "desc", + scope: "one", + }); + + expect(() => paginateByCursor(rows(3), { + cursor: first.nextCursor!, + keyOf: descKey, + direction: "asc", + scope: "one", + })).toThrow(ApiError); + expect(() => paginateByCursor(rows(3), { + cursor: first.nextCursor!, + keyOf: descKey, + direction: "desc", + scope: "two", + })).toThrow(ApiError); + }); + + it("traverses a complete descending collection without gaps or duplicates", () => { + const source = rows(47); + const seen: number[] = []; + let cursor: string | undefined; + + do { + const page = paginateByCursor(source, { + cursor, + pageSize: 7, + keyOf: descKey, + direction: "desc", + scope: "traverse", + }); + seen.push(...page.items.map((row) => row.id)); + cursor = page.nextCursor ?? undefined; + } while (cursor); + + expect(seen).toEqual(source.map((row) => row.id)); + expect(new Set(seen).size).toBe(source.length); + }); + + it("traverses a complete ascending collection without gaps or duplicates", () => { + const source = [...rows(31)].reverse(); + const keyOf = (row: Row) => String(row.id).padStart(8, "0"); + const sorted = [...source].sort((a, b) => keyOf(a).localeCompare(keyOf(b))); + const seen: number[] = []; + let cursor: string | undefined; + + do { + const page = paginateByCursor(sorted, { + cursor, + pageSize: 4, + keyOf, + direction: "asc", + scope: "ascending", + }); + seen.push(...page.items.map((row) => row.id)); + cursor = page.nextCursor ?? undefined; + } while (cursor); + + expect(seen).toEqual(sorted.map((row) => row.id)); + }); + + it("holds the descending snapshot boundary when newer rows arrive", () => { + const initial = rows(9); + const first = paginateByCursor(initial, { + pageSize: 3, + keyOf: descKey, + direction: "desc", + scope: "snapshot", + }); + const newer = [{ id: 999, createdAt: "2026-09-01" }, ...initial]; + const second = paginateByCursor(newer, { + cursor: first.nextCursor!, + pageSize: 10, + keyOf: descKey, + direction: "desc", + scope: "snapshot", + }); + + expect(second.items.map((row) => row.id)).toEqual([6, 5, 4, 3, 2, 1]); + expect(second.items.map((row) => row.id)).not.toContain(999); + }); + + it("returns a null cursor at the end of a collection", () => { + const page = paginateByCursor(rows(2), { + pageSize: 10, + keyOf: descKey, + direction: "desc", + }); + expect(page.nextCursor).toBeNull(); + }); + + it("returns an empty page for an empty collection", () => { + expect(paginateByCursor([], { keyOf: descKey })).toEqual({ + items: [], + pageSize: CURSOR_DEFAULT_PAGE_SIZE, + nextCursor: null, + }); + }); + + it("rejects duplicate ordering keys instead of silently skipping data", () => { + expect(() => paginateByCursor([{ id: 1 }, { id: 1 }], { + keyOf: (row) => String(row.id), + })).toThrow("unique ordering keys"); + }); + + it("uses a unique tiebreaker when visible fields tie", () => { + const tied = [ + { id: "b", createdAt: "2026-08-01" }, + { id: "a", createdAt: "2026-08-01" }, + ]; + const sorted = [...tied].sort((a, b) => a.id.localeCompare(b.id)); + const page = paginateByCursor(sorted, { + pageSize: 1, + keyOf: (row) => `${row.createdAt}\u0000${row.id}`, + }); + expect(page.items[0].id).toBe("a"); + expect(page.nextCursor).toEqual(expect.any(String)); + }); + + it("accepts numeric page sizes but rejects non-integers", () => { + expect(paginateByCursor(rows(3), { pageSize: 2, direction: "desc", keyOf: descKey }).items).toHaveLength(2); + for (const pageSize of [0, -1, 1.5, "1.5", "abc", [], {}]) { + expect(() => paginateByCursor(rows(3), { pageSize, keyOf: descKey })).toThrow(ApiError); + } + }); + + it("does not mutate caller-owned item ordering", () => { + const source = rows(4); + const original = [...source]; + paginateByCursor(source, { keyOf: descKey, direction: "desc" }); + expect(source).toEqual(original); + }); +}); diff --git a/src/utils/cursorPagination.ts b/src/utils/cursorPagination.ts new file mode 100644 index 0000000..561e074 --- /dev/null +++ b/src/utils/cursorPagination.ts @@ -0,0 +1,115 @@ +import { ApiError } from "../errors/ApiError"; + +export type CursorDirection = "asc" | "desc"; + +export interface CursorPage { + items: T[]; + pageSize: number; + nextCursor: string | null; +} + +interface CursorPayload { + version: 1; + direction: CursorDirection; + snapshot: string; + after: string; + scope?: string; +} + +export const CURSOR_DEFAULT_PAGE_SIZE = 20; +export const CURSOR_MAX_PAGE_SIZE = 100; + +function parsePageSize(value: unknown): number { + if (value === undefined || value === "") return CURSOR_DEFAULT_PAGE_SIZE; + if (typeof value !== "string" && typeof value !== "number") { + throw ApiError.badRequest('"pageSize" must be a positive integer'); + } + const raw = String(value); + if (!/^\d+$/.test(raw) || Number(raw) < 1) { + throw ApiError.badRequest('"pageSize" must be a positive integer'); + } + return Math.min(Number(raw), CURSOR_MAX_PAGE_SIZE); +} + +function encode(payload: CursorPayload): string { + return Buffer.from(JSON.stringify(payload), "utf8").toString("base64url"); +} + +function decode(value: unknown, direction: CursorDirection, scope?: string): CursorPayload { + if (typeof value !== "string" || value.length === 0) { + throw ApiError.badRequest("cursor is malformed or expired"); + } + try { + const payload = JSON.parse(Buffer.from(value, "base64url").toString("utf8")) as Partial; + if ( + payload.version !== 1 || + payload.direction !== direction || + typeof payload.snapshot !== "string" || + typeof payload.after !== "string" || + (scope !== undefined && payload.scope !== scope) + ) { + throw new Error("invalid cursor payload"); + } + return payload as CursorPayload; + } catch { + throw ApiError.badRequest("cursor is malformed or expired"); + } +} + +/** + * Page a pre-sorted collection using an opaque, stable compound cursor. + * `keyOf` must return a unique key in the same order as `items`; callers use a + * deterministic tiebreaker such as an id when the visible sort field ties. + */ +export function paginateByCursor( + items: T[], + options: { + cursor?: unknown; + pageSize?: unknown; + direction?: CursorDirection; + scope?: string; + keyOf: (item: T, index: number) => string; + }, +): CursorPage { + const direction = options.direction ?? "asc"; + const pageSize = parsePageSize(options.pageSize); + const keys = items.map(options.keyOf); + + if (new Set(keys).size !== keys.length) { + throw new Error("cursor pagination requires unique ordering keys"); + } + + const supplied = options.cursor === undefined + ? undefined + : decode(options.cursor, direction, options.scope); + if (items.length === 0) return { items: [], pageSize, nextCursor: null }; + const snapshot = supplied?.snapshot ?? keys[0]; + const startAfter = supplied?.after; + const visible = items.filter((_item, index) => { + const key = keys[index]; + const insideSnapshot = direction === "desc" ? key <= snapshot : key >= snapshot; + const afterCursor = startAfter === undefined + ? true + : direction === "desc" ? key < startAfter : key > startAfter; + return insideSnapshot && afterCursor; + }); + const pageItems = visible.slice(0, pageSize); + const hasMore = visible.length > pageItems.length; + const lastIndex = pageItems.length - 1; + const lastItem = pageItems[lastIndex]; + const lastOriginalIndex = lastItem === undefined ? -1 : items.indexOf(lastItem); + + return { + items: pageItems, + pageSize, + nextCursor: hasMore && lastOriginalIndex >= 0 + ? encode({ + version: 1, + direction, + snapshot, + after: options.keyOf(lastItem!, lastOriginalIndex), + scope: options.scope, + }) + : null, + }; +}