Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
4cbe79b
feat(review): integrate Claw & Order detection pipeline with Krill GP…
jason-allen-oneal Sep 17, 2026
891ef4c
test(review): add offline benchmark comparing Rowan bot and human pro…
jason-allen-oneal Sep 17, 2026
b134933
fix(review): enforce guild boundary, repair button dispatch, and add …
jason-allen-oneal Sep 17, 2026
6f53de2
feat(review): support discrawl export backend and harden escalation d…
jason-allen-oneal Sep 17, 2026
014fcfa
feat(review): harden receipt verification, export worker bridge, and …
jason-allen-oneal Sep 17, 2026
c6f4fad
fix(review): preserve uncertain delivery across claims, reject stale …
jason-allen-oneal Sep 17, 2026
f232cd5
fix(review): address all 6 ClawSweeper Rev 5 findings
jason-allen-oneal Sep 17, 2026
29cbfde
fix(review): preserve card recovery and correct migration proof
jason-allen-oneal Sep 18, 2026
adf1cfb
fix(review): harden decisions and recovery ownership
morrow-bluedot Sep 20, 2026
70ab90f
fix(review): persist ambiguous writes and legacy delivery state
jason-allen-oneal Sep 20, 2026
41c46ab
fix(review): retain in-flight card repair obligations
jason-allen-oneal Sep 20, 2026
f54a3ad
fix(review): retain unresolved writes until proven obsolete
jason-allen-oneal Sep 20, 2026
db72cce
test(review): add opt-in real Discord recovery proof runner
morrow-bluedot Sep 20, 2026
207b77d
test(review): compare Discord omitted button defaults faithfully
morrow-bluedot Sep 20, 2026
ad2fa35
fix(review): retire acknowledged stale writes after durable repair
morrow-bluedot Sep 20, 2026
a8e3815
test(review): use operator-managed proof role without bot privilege c…
morrow-bluedot Sep 20, 2026
ae6caa0
test(review): accept Carbon numeric role permissions in ingress proof
morrow-bluedot Sep 20, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
132 changes: 132 additions & 0 deletions docs/review-live-proof.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,132 @@
# Review live-transport proof

This is a runnable proof procedure, **not execution evidence**. A successful
offline preflight or unit test is not live proof. Do not remove those labels
until the corresponding run has produced an inspected artifact.

## Scope

`scripts/proof-review-live.ts` invokes unchanged production delivery,
synchronization, and outstanding-write recovery functions using Carbon's real
Discord REST client. It creates one synthetic card in an empty test text
channel, verifies every payload with real GETs, and leaves the card intact.
The database is a fresh **local Wrangler D1 binding**, not deployed D1. The
synthetic archive goes through the actual authenticated forwarder HTTP bridge.
The bridge has a random per-run bearer secret and is stopped afterward.

The test account being assessed is the test bot itself. Its input is generated
fixture text, never channel history or real member telemetry. Only the
configured channel is accessed. Production review guild/channel IDs are refused.
Card comparisons ignore server-assigned component IDs and treat an omitted
button `disabled` flag as false; true and every rendered field remain checked.
The runner overrides review coordinates in its own process; production files,
bot endpoint, command registrations, and deployed services remain unchanged.

## Required configuration

Provide these to the process through the authorized runtime credential facility.
Do not paste tokens into chat, command lines, artifacts, or source files.

- `HERMIT_PROOF_GUILD_ID`: authorized nonproduction guild.
- `HERMIT_PROOF_CHANNEL_ID`: empty dedicated text channel in that guild.
- `DISCORD_CLIENT_ID` and `DISCORD_BOT_TOKEN`: existing authorized test bot.
- `OPENAI_API_KEY`: only if the optional paid provider proof is authorized.

Automatic screening must remain disabled. Do not change any existing bot's
interaction endpoint or redeploy its commands for this runner. If OpenClaw
requires Discord-scoped execution, launch from that actual conversation; do
not extract file-backed credentials to bypass the scope restriction.

Run from a **clean committed tree** (Bun on PATH):

```sh
bun --no-env-file scripts/proof-review-live.ts --preflight
bun --no-env-file scripts/proof-review-live.ts --live
```

To additionally exercise the unchanged evaluator and its real provider request,
use `--live --provider` instead of `--live`. This can make the configured primary
request and its fallback, and spends API credit. Both HTTP results and the
provider's returned model name are recorded; do not label fallback output as the
primary model. The runner never installs or claims a separate Krill service.

## Decisive sequence

1. Verify the token's bot identity, the channel's guild, and that the channel is
empty. Apply migrations to fresh local D1. Fetch 24 synthetic observations
through the authenticated production Discrawl bridge and analyze them.
2. Optionally request a real provider assessment. Create the case and deliver a
card through `postReviewEscalationCard`. Fetch and compare the exact card.
This is the normal-delivery preservation control.
3. Enter production `syncSharedReviewCard` for an older assessment. At the
transport seam, hold its PATCH after production persisted the write ledger.
Leave its caller unresolved: neither catch nor acknowledgment may run.
4. Save and synchronize a newer assessment. GET verifies the newer payload.
Honor the actual 120-second due time, run outstanding-write recovery, GET
verifies the card, and assert the original ledger is still present.
5. Forward the held PATCH through the original Carbon REST client. A real GET
must now show the stale payload. Keep the original caller unresolved. Honor
the next actual backoff, run recovery again, and GET must show latest state.
6. Assert desired/synced revisions match, the original unresolved ledger remains,
and the test channel contains only the original card—not a replacement.

The barrier controls the ordering; waits only honor production retry eligibility.
No fake Discord responses, fabricated acknowledgment, clock rewrite, or fabricated
due timestamp is used. This injects a delayed/unacknowledged transport outcome;
it is **not** an actual Worker termination or proof about Discord latency bounds.

## Artifacts and limits

The private run directory is printed at exit under
`$XDG_STATE_HOME/hermit-live-proof/<run-id>` (or `~/.local/state/...`). It retains
`evidence.jsonl`, the synthetic fixture, generated local-only Wrangler config,
and local D1 state. A failure is recorded as failure. Proxies are disposed before
exit. No cleanup deletes remote cards or receipts automatically; inspect the
record before retrying, and use another empty test channel if needed.

Inspect and redact artifacts before PR publication. Evidence records HEAD,
ordering, actual GET assertions, revisions, retained obligations, provider
response model, and hashes. Never publish credentials, private endpoints, or
unrelated logs. Do not publish private Discord IDs without owner permission.

Still separate, even after this runner passes:

- Real `/review` and button ingress through the production Worker/Carbon router.
Capture an authorized staff test action, stale-button rejection, the D1
decision, and shared-card GET readback. Do not fabricate signed Discord events.
- Deployed staging Worker/D1 evidence, if required by the current review.
- An executed red baseline with the same transport harness; this script does
not assert one ran or silently disable production checks to simulate it.
- Pilot policy sponsorship and upstream CI approval.

Update `## Real Behavior Proof` with only the observed scope, exact tested HEAD,
command, relevant output, inspected artifact, and these explicit limits.

## Actual Gateway interaction proof

`bun --no-env-file scripts/proof-review-interactions.ts --live` is a separate
opt-in ingress runner, not a claim of a completed run. It requires the same test
configuration plus `HERMIT_PROOF_ACTOR_IDS` (comma-separated authorized owners) and
`HERMIT_PROOF_STAFF_ROLE_ID` (operator-created zero-permission test role).
It refuses an existing guild `/review` command or an application with an HTTP
interaction endpoint. It adds only its own temporary guild command and maps the existing test role
as staff inside the test process. The actual actor starts without that role.
After a genuine rejected command, the owner assigns the role through Discord
for the authorized control. It deletes its own command in `finally`, including
failure/timeout cleanup. The owner must remove the temporary role afterward;
the bot is not granted role-management privileges.

It uses a separate zero-intent Carbon Gateway connection, listens only for test
channel/actor interactions, and passes their actual payloads unchanged to the
same Carbon interaction router used by the Worker. The production command and
button handlers are unchanged. Existing bot configuration, global commands,
and interaction endpoint are not modified. No message history is ingested.

In the authorized browser, run `/review user:<test bot>` twice (first denied,
then allowed), click Dismiss on the now-stale ephemeral command card, then click
Dismiss on the current shared card. The runner asserts real webhook responses,
D1 denial/preservation/decision, and exact real shared-card GET readback. Never
fabricate a Discord event or interaction token. Artifacts remain private under
`~/.local/state/hermit-interaction-proof/`. This exercises real Gateway ingress,
not deployed Worker HTTP signature validation. Provider execution is excluded
and `OPENAI_API_KEY` must be absent to keep that proof independent.
52 changes: 52 additions & 0 deletions drizzle/0013_reflective_rictor.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
CREATE TABLE `review_cases` (
`id` integer PRIMARY KEY AUTOINCREMENT NOT NULL,
`case_id` text NOT NULL,
`guild_id` text NOT NULL,
`target_user_id` text NOT NULL,
`status` text DEFAULT 'open' NOT NULL,
`heuristic_score` integer NOT NULL,
`concordance` text NOT NULL,
`behavioral_families` text NOT NULL,
`evidence_message_id` text,
`krill_probability` text,
`krill_brief` text,
`krill_model` text,
`review_message_id` text,
`review_channel_id` text,
`delivery_status` text DEFAULT 'pending' NOT NULL,
`previous_delivery_status` text DEFAULT 'pending' NOT NULL,
`card_revision` integer DEFAULT 1 NOT NULL,
`synced_card_revision` integer DEFAULT 1 NOT NULL,
`expires_at` text,
`decided_by_id` text,
`decision_reason` text,
`created_at` text DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ', 'now')) NOT NULL,
`updated_at` text DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ', 'now')) NOT NULL
);
--> statement-breakpoint
CREATE UNIQUE INDEX `review_cases_case_id_unique` ON `review_cases` (`case_id`);--> statement-breakpoint
CREATE INDEX `idx_review_cases_guild_target` ON `review_cases` (`guild_id`,`target_user_id`);--> statement-breakpoint
CREATE INDEX `idx_review_cases_status` ON `review_cases` (`status`);--> statement-breakpoint
CREATE INDEX `idx_review_cases_review_msg` ON `review_cases` (`review_message_id`);--> statement-breakpoint
CREATE TABLE `review_observations` (
`id` integer PRIMARY KEY AUTOINCREMENT NOT NULL,
`message_id` text NOT NULL,
`guild_id` text NOT NULL,
`channel_id` text NOT NULL,
`author_id` text NOT NULL,
`created_at` text NOT NULL,
`reply_to_id` text,
`content_length` integer NOT NULL,
`line_count` integer NOT NULL,
`fingerprint` text NOT NULL,
`artifacts` text NOT NULL,
`similarity` text,
`semantic_score` integer,
`received_at` text DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ', 'now')) NOT NULL
);
--> statement-breakpoint
CREATE UNIQUE INDEX `review_observations_message_id_unique` ON `review_observations` (`message_id`);--> statement-breakpoint
CREATE INDEX `idx_review_obs_guild_author` ON `review_observations` (`guild_id`,`author_id`,`created_at`);--> statement-breakpoint
CREATE INDEX `idx_review_obs_author` ON `review_observations` (`author_id`,`created_at`);--> statement-breakpoint
CREATE INDEX `idx_review_obs_channel` ON `review_observations` (`channel_id`,`created_at`);--> statement-breakpoint
CREATE INDEX `idx_review_obs_message` ON `review_observations` (`message_id`);
11 changes: 11 additions & 0 deletions drizzle/0014_review_recovery_ownership.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
ALTER TABLE `review_cases` ADD `delivery_nonce` text;--> statement-breakpoint
ALTER TABLE `review_cases` ADD `delivery_claim_token` text;--> statement-breakpoint
ALTER TABLE `review_cases` ADD `delivery_claim_expires_at` text;--> statement-breakpoint
ALTER TABLE `review_cases` ADD `receipt_claim_token` text;--> statement-breakpoint
ALTER TABLE `review_cases` ADD `receipt_claim_expires_at` text;--> statement-breakpoint
ALTER TABLE `review_cases` ADD `receipt_next_attempt_at` text;--> statement-breakpoint
ALTER TABLE `review_cases` ADD `receipt_history_before` text;--> statement-breakpoint
ALTER TABLE `review_cases` ADD `card_sync_next_attempt_at` text;--> statement-breakpoint
ALTER TABLE `review_cases` ADD `card_sync_failure_count` integer DEFAULT 0 NOT NULL;--> statement-breakpoint
CREATE INDEX `idx_review_cases_receipt_recovery` ON `review_cases` (`guild_id`,`delivery_status`,`receipt_next_attempt_at`,`receipt_claim_expires_at`);--> statement-breakpoint
CREATE INDEX `idx_review_cases_card_sync_due` ON `review_cases` (`card_sync_next_attempt_at`,`card_revision`,`synced_card_revision`);
2 changes: 2 additions & 0 deletions drizzle/0015_review_delivery_attempt_state.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
ALTER TABLE `review_cases` ADD `delivery_preflight_completed_at` text;--> statement-breakpoint
ALTER TABLE `review_cases` ADD `delivery_post_attempted_at` text;
1 change: 1 addition & 0 deletions drizzle/0016_chilly_loners.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
ALTER TABLE `review_cases` ADD `key_signals` text DEFAULT '[]' NOT NULL;
17 changes: 17 additions & 0 deletions drizzle/0017_optimal_leopardon.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
CREATE TABLE `review_card_write_attempts` (
`attempt_token` text PRIMARY KEY NOT NULL,
`case_id` text NOT NULL,
`guild_id` text NOT NULL,
`channel_id` text NOT NULL,
`message_id` text NOT NULL,
`rendered_revision` integer NOT NULL,
`claim_token` text,
`claim_expires_at` text,
`next_attempt_at` text,
`failure_count` integer DEFAULT 0 NOT NULL,
`created_at` text DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ', 'now')) NOT NULL
);
--> statement-breakpoint
CREATE INDEX `idx_review_card_write_attempts_due` ON `review_card_write_attempts` (`guild_id`,`next_attempt_at`,`claim_expires_at`,`created_at`);--> statement-breakpoint
CREATE INDEX `idx_review_card_write_attempts_case` ON `review_card_write_attempts` (`case_id`);--> statement-breakpoint
ALTER TABLE `review_cases` ADD `delivery_attempt_state` text DEFAULT 'legacy_unknown' NOT NULL;
Loading
Loading