Reconcile vaults: checkpoint store, drift anomaly logging, Soroban client integration - #1
Open
akinerin wants to merge 465 commits into
Open
Reconcile vaults: checkpoint store, drift anomaly logging, Soroban client integration#1akinerin wants to merge 465 commits into
akinerin wants to merge 465 commits into
Conversation
- Add pause/resume flags and progress tracking to BackfillCursorStore - Expose GET /api/admin/backfills, POST /api/admin/backfills/:name/pause|resume - Admin-only (requireAdmin middleware), audit-logged (backfill.pause/resume) - ETA derived from observed throughput samples - Consume isPaused in evidenceReindex.ts; stops claiming work when paused, resumes from saved cursor without gaps or duplicates - 20 tests covering progress list, pause/resume, audit log, RBAC, error paths Closes Disciplr-Org#842
…s + fix mime mock for ESM
…P-allowlist enforcement tests
…aint violation (Disciplr-Org#1364) - Wrap check-and-insert in db.transaction() so the existence check and INSERT are atomic, preventing TOCTOU race conditions where two concurrent approval requests from the same verifier for the same milestone could both pass the SELECT-before-INSERT check - Catch unique constraint violation errors (PostgreSQL 23505 / SQLITE_CONSTRAINT) and convert them to DuplicateVerifierVoteError as a backstop; the database unique constraint on (milestone_id, verifier_user_id) already exists in 20260429000000_add_multi_verifier_support.cjs:36 - Update test mocks to simulate constraint violations on duplicate inserts, so the transaction + constraint backstop path is exercised in tests Close Disciplr-Org#1111
…r-Org#1363) - Wrap the three unguarded deletes (refreshToken, vault, users) in a single Prisma interactive so that a crash between any two steps no longer leaves partially-deleted, inconsistent state. - Use tx. for the users table delete (the User model is managed via Knex, not Prisma) and throw on zero affected rows to trigger a rollback. - Add an optional actorUserId parameter and call createAuditLog with action 'user.hard_delete' when provided, matching the pattern used by changeRole in membership.ts. - Update the admin route to pass req.user!.userId and remove the duplicate route-level audit log for hard deletes (the service now owns it). Soft-delete audit logging remains in the route. Close Disciplr-Org#1083
…rg#1100) (Disciplr-Org#1362) Co-authored-by: Sakariyah Abdulhazeem <150973162+zeemscript@users.noreply.github.com>
Disciplr-Org#1360) resolveKeys() used Array.map() over all FIELD_ENCRYPTION_KEYS entries, so a single malformed or mis-sized retired rotation key (index > 0) threw EncryptionKeyError and aborted field encryption/decryption for the whole application — including the currently-active key that was otherwise perfectly valid. Fix: replace the .map() with an explicit loop that applies different failure policies per entry: - index 0 (active key): any shape or key-material error is still fatal (EncryptionKeyError), because encrypting new data would be impossible without a valid active key. - index > 0 (retired/rotation keys): a bad entry emits console.warn and is skipped. The skipped entry's kid is never added to the duplicate-id set, so a bad retired entry cannot also trigger a spurious duplicate- key error. The overall fail-closed guarantee is preserved: - Missing or all-invalid config still throws. - A bad active key still throws. - The duplicate-id check still runs over all successfully loaded keys. Tests added (4 new cases in 'key configuration validation'): - Skips a retired key missing the 'key' field, warns, returns the active key. - Skips a retired key with too-short base64 material, warns. - Still throws fatally when the active key (index 0) is missing fields. - Still throws fatally when the active key (index 0) has invalid material. Also adds 'jest' to the @jest/globals import (needed for spyOn). Closes Disciplr-Org#1300
…#1357) v2 subscribers serving multiple orgs through one endpoint had no way to determine which organization a delivery belonged to from the signed body alone: the v1 payload included organizationId but the v2 compact envelope only contained { schema_version, event_type, data }. While eventId/timestamp are redundantly available via the x-disciplr-event-id / x-disciplr-delivery-timestamp headers, there is no corresponding x-disciplr-organization-id header in either dispatcher (webhooks.ts deliverOnce or BoundedWebhookDispatcher.deliverOnce). Fix: add organization_id (snake_case, consistent with event_type and schema_version) to the v2 body so multi-tenant subscribers can reliably route and verify deliveries without inspecting a separate lookup. Updated JSDoc for buildVersionedPayload and both the unit test ('v2 returns compact envelope') and integration test ('delivers v2 envelope to version-2 subscribers') in webhooks.schemaVersion.test.ts. Closes Disciplr-Org#1298
…Org#1356) Replace four parallel db('verifications').count() calls with a single db.raw() query using COUNT(*) FILTER (WHERE ...) conditional aggregation, matching the pattern already used in src/services/team.ts (ROLLUP_SQL). This reduces the 4 × N database round-trips caused by calling getVerifierStats once per verifier in the GET /api/admin/verifiers Promise.all loop down to 1 × N (plus the one query to list verifiers). Closes Disciplr-Org#1296
…rogress and remove unused functions (Disciplr-Org#1351)
…isciplr-Org#1348) filter(Boolean) treated '' the same as null, silently dropping explicitly-empty descriptions from the embedding text just like genuinely absent ones. Filter on nullishness instead so empty strings still contribute a (blank) line. Closes Disciplr-Org#1295
…plr-Org#1403) - Add composite indexes for audit_logs, webhook_subscribers, vaults, notifications - Add EXPLAIN regression test verifying no sequential scans - Document audit findings in performance-testing.md Fixes Disciplr-Org#754
…privilege escalation (Disciplr-Org#975) (Disciplr-Org#1457) The invitation creation endpoint (POST /:orgId/invitations) was not storing the intended role in the org_invitations table. The accept endpoint (POST /:orgId/invitations/accept) then read the role directly from the untrusted request body, allowing anyone with a valid invitation token to claim any role including 'owner'. Changes: - Add DB migration adding a role column to org_invitations - Store the inviter-specified role at invitation creation time - Use the stored role on accept (ignore any role in request body) - Update OrgInvitation type and service return columns - Update all tests to reflect the new flow
…ingDrift (Disciplr-Org#1455) * Fix Disciplr-Org#1280: Distinguish decode failures from burn addresses in isUnsafeAddress - Refactored isUnsafeAddress to checkAddressSafety function that returns granular result - Added UnsafeAddressResult interface to distinguish between burn addresses and decode errors - Updated validation to provide specific error messages for each case: - Burn addresses: 'cannot be a zero or burn address (all-zero or all-ones bytes)' - Decode errors: 'Failed to decode destination address - unexpected SDK error' - Kept isUnsafeAddress as a deprecated legacy function for backward compatibility - This prevents legitimate addresses that trigger decode exceptions from being incorrectly rejected with misleading burn address messages * Fix Disciplr-Org#1266: Use EmbeddingDriftDb interface in detectEmbeddingDrift - Changed db parameter type from loose '{ (table: string): any }' to EmbeddingDriftDb - This applies the intended type safety that was defined but unused - The EmbeddingDriftDb interface was carefully designed to provide compile-time guarantees about the query builder shape (with .select()/.count()/.groupBy()) - Now tests can confidently inject simple fakes that satisfy the interface - Eliminates dead code by actually using the defined interface
…s in isUnsafeAddress (Disciplr-Org#1454) - Refactored isUnsafeAddress to checkAddressSafety function that returns granular result - Added UnsafeAddressResult interface to distinguish between burn addresses and decode errors - Updated validation to provide specific error messages for each case: - Burn addresses: 'cannot be a zero or burn address (all-zero or all-ones bytes)' - Decode errors: 'Failed to decode destination address - unexpected SDK error' - Kept isUnsafeAddress as a deprecated legacy function for backward compatibility - This prevents legitimate addresses that trigger decode exceptions from being incorrectly rejected with misleading burn address messages
…tion' fallbacks in the OAuth signing and verification paths with the centralized, Zod-validated getEnv().JWT_SECRET, so the existing insecure-default detection actually covers this path and the app fails closed if the secret is unset in production (Disciplr-Org#1451) Two production files (src/routes/oauth.ts and src/middleware/oauthBearer.ts) were reading JWT_SECRET directly from process.env with a fallback literal ('change-me-in-production') that differed from the Zod schema's default ('change-me-in-production-long-secret'). This meant the config.insecure_default warning in initEnv() never triggered for OAuth token issuance, and in an unset deployment every access token would be signed with a publicly-known string. Both files now import getEnv() from ../config/index.js and read the secret through the validated schema. The test suite (src/tests/oauth.clientCredentials.test.ts) was updated to mock getEnv via jest.unstable_mockModule (the project's standard ESM pattern) so the tests work without a real database — the mock omits DATABASE_URL, keeping the API key repository on its in-memory fallback as before. Closes Disciplr-Org#1020
…lient integration
…iation, add autoRepairVault and compareVaultStates, fix early-return driftedVaults
- Accept upstream's package-lock.json - Adopt upstream's batched Soroban-based reconcileVaults with options, autoRepairVault, compareVaultStates - Keep driftedVaults: [] fix in early-return branch - Accept upstream comments in reconcileVaults
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
There was a problem hiding this comment.
CodeQL found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.
- src/db/index.ts: remove injected Disciplr-Org#1037-FIX line, deduplicate pool setup - src/routes/milestones.ts: remove artifact lines, duplicate handlers, and dead in-memory completeVault call - src/services/auth.service.ts: remove trailing </absolute_path> tags - src/services/idempotency.ts: add missing closing brace on getIdempotentResponse - src/jobs/handlers.ts: fold escaped handlers into function body, remove duplicate definitions outside createDefaultJobHandlers scope
The merge of upstream/main removed local definitions of JobHandlerRegistry, logJob, and sleep that existed in the pre-merge reconcileVaults branch. Also switched from object literal to individual property assignments with a cast to avoid TS2740 (missing required JobType keys), since some handlers are conditionally registered. Removed payload.batchSize reference from deadline.check handler (not in DeadlineCheckJobPayload type).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
18 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Includes driftedVaults fix, ETL improvements, and lockfile regeneration