AO-native command layer for Blackcat Darkmesh. This repository hosts the write-side AO processes that enforce idempotent, authorized, and auditable changes to the canonical state maintained in blackcat-darkmesh-ao. No separate server-side authority exists; any bridge or admin client is only a transport adapter.
This repo is infrastructure/back-end only: no UI assets, no public read path, no long-lived secrets.
- Scope
- Architecture Snapshot
- Interfaces at a glance
- Links hub
- AO deploy (current runbook)
- Release gate / deep test
- Development
- Env toggles (write process)
- CLI helpers
- Prod hardening checklist
- Monitoring
- Bridge (stub)
- Security Guard Rails
- License
- CI notes
- Docker quickstart
- In scope: AO command processes, handlers, idempotency registry, audit/event emission, publish workflow (draft → review → publish → rollback), validators and schemas, minimal adapters, deploy/verify scripts, fixtures, CI workflows.
- Out of scope: public read/state model (lives in
blackcat-darkmesh-ao), gateway rendering, frontend assets, mailbox payload storage, SMTP/OTP/PSP integrations, template/catalog UI, or long-term secret storage (only public keys here).
- Role: command-first AO process set that owns write semantics, conflict detection, and append-only audit; delegates state materialization to
blackcat-darkmesh-ao. - Pipeline: command envelope → validation (schema + policy) → idempotency / anti-replay → handler → audit + event → downstream AO state update.
- Identity & auth: signed commands or capability tokens; gateway is never an implicit authority.
- Idempotence:
requestIdregistry and optimisticexpectedVersionguards to prevent duplicate writes. - Audit: append-only log with correlation to requestId and actor; deterministic status codes.
- Gateway compatibility:
CreateOrdersupports both cart-driven payloads (cartId) and direct template payloads (siteId + items) with safeorderId/currencyfallback generation.
flowchart LR
A[Client / Gateway] -->|Action envelope| V[Validation
- schema
- policy
- signature/JWT
- nonce/replay]
V --> I[Idempotency check
requestId + expectedVersion]
I --> H[Handler]
H --> O[Outbox event
+ HMAC]
H --> W[WAL/audit]
O --> D[Downstream AO / bridge]
style O fill:#0b7285,stroke:#0b7285,color:#fff
style W fill:#495057,stroke:#495057,color:#fff
flowchart LR
WB[PSP/Webhook] -->|POST| R[Webhook handler]
R -->|verify sig| S{Signature OK?}
S -->|no| B[Breaker + 4xx]
S -->|yes| DQ[dedupe window]
DQ -->|seen?| Drop[409 Conflict]
DQ -->|new| Q[enqueue retry queue]
Q -->|fetch PSP state| P[Process payment/update]
P --> EV[emit event + HMAC]
style Q fill:#0b7285,stroke:#0b7285,color:#fff
Legend: teal = queues/events, gray = WAL/audit paths.
| Area | Inputs | Outputs | Interfaces / Paths | Health | CI gates |
|---|---|---|---|---|---|
| Commands | Signed envelopes (Action, Request-Id, Actor, Tenant, Expected-Version, Nonce, Signature-Ref, Timestamp) |
WAL entries, outbox events (HMACed), audit | ao/write/process.lua, schemas under schemas/ |
scripts/verify/health.lua |
preflight, contracts, conflicts, batch fixtures |
| Webhooks (PSP) | Stripe/PayPal/GoPay webhook POSTs + signatures | PSP events → outbox, breaker state, retry queue | ao/shared/psp_webhooks.lua, retry queue |
metrics webhook_retry_*, breaker_open |
webhook_psp_spec, gopay_webhook_spec |
| Metrics | Prom text file (METRICS_PROM_PATH) |
scrape/alerts | /metrics (if exposed) |
scripts/verify/health.lua |
docs/ALERTS.md |
| Arweave gate | Artifact hash vs TX | pass/fail | scripts/verify/verify_arweave_hash.sh |
n/a | ENFORCE_ARWEAVE_HASH stage |
- Alerts & thresholds:
docs/ALERTS.md - Deploy runbook:
docs/runbooks/deploy.md - Rollback runbook (incl. hash-gate handling):
docs/runbooks/rollback.md - Env template:
ops/env.prod.example - Schemas:
schemas/ - PSP/webhook specs:
scripts/verify/webhook_psp_spec.lua,gopay_webhook_spec.lua - Gateway security manifest snapshot (internalized):
../blackcat-darkmesh-gateway/security/crypto-manifests/
- Canonical endpoint:
https://push.forward.computer - Scheduler:
n_XZJhUnmldNFo4dhajoPZWhBXuJk-OcQr5JQ49c4Zo - Runtime variant/tag family:
ao.TN.1(module + process + message path in this repo) - Source of truth for live module/PID history:
AO_DEPLOY_NOTES.md(do not treat this README as the TX ledger).
node scripts/build-write-bundle.js
ao-dev build
node scripts/publish-wasm.jsscripts/publish-wasm.js publishes dist/write/process.wasm with the expected tags:
Type=ModuleContent-Type=application/wasmVariant=ao.TN.1signing-format=ans104accept-bundle=trueaccept-codec=httpsig@1.0
AO_MODULE=<module_tx> \
HB_URL=https://push.forward.computer \
HB_SCHEDULER=n_XZJhUnmldNFo4dhajoPZWhBXuJk-OcQr5JQ49c4Zo \
WRITE_SIG_TYPE=ed25519 \
WRITE_SIG_PUBLIC=<pubkey_or_hex> \
node scripts/cli/spawn_wasm_tn.js- Expect temporary
404onhttps://arweave.net/raw/<tx>shortly after publish/spawn. - Do not trust runtime behavior before both module and PID are indexed/finalized.
- Typical observed delay in this project: initial 404 window + extended finalization (often tens of minutes).
HB_URL=https://push.forward.computer \
HB_SCHEDULER=n_XZJhUnmldNFo4dhajoPZWhBXuJk-OcQr5JQ49c4Zo \
AO_PID=<pid> \
node scripts/cli/diagnose_message.js
HB_URL=https://push.forward.computer \
HB_SCHEDULER=n_XZJhUnmldNFo4dhajoPZWhBXuJk-OcQr5JQ49c4Zo \
AO_PID=<pid> \
node scripts/cli/send_write_command.jsFor worker-signed end-to-end tests, set WORKER_SIGN_URL + WORKER_AUTH_TOKEN (test values are kept locally in tmp/test-secrets.json).
Gateway template contract expects write endpoints:
POST /api/checkout/orderPOST /api/checkout/payment-intent
This repo now ships a lightweight adapter:
WRITE_PROCESS_ID=<write_pid> \
WRITE_WALLET_PATH=wallet.json \
WRITE_HB_URL=https://push.forward.computer \
WRITE_HB_SCHEDULER=n_XZJhUnmldNFo4dhajoPZWhBXuJk-OcQr5JQ49c4Zo \
WRITE_SIGNER_URL=https://<worker-host>/sign \
WRITE_SIGNER_TOKEN=<worker_bearer_token> \
node scripts/http/checkout_api_server.mjsNotes:
- Adapter accepts already signed envelopes, or can call worker
/signwhen signature fields are missing. - It forwards envelope as
Write-CommandAO message and returns normalized write result. - Default listen address is
0.0.0.0:8789(PORTcan override). - Optional multi-site PID override is disabled by default. To enable safely, set both
WRITE_API_ALLOW_PID_OVERRIDE=1andWRITE_API_TOKEN, then passX-Write-Process-Id(or bodywriteProcessId) per request.
Run the v1.2.0 readiness gate in one command:
AO_PID=<pid> \
HB_URLS=https://push.forward.computer,https://push-1.forward.computer \
AO_SECRETS_PATH=tmp/test-secrets.json \
scripts/verify/release_gate_v120.sh --strictFlag form:
scripts/verify/release_gate_v120.sh \
--pid <pid> \
--urls https://push.forward.computer,https://push-1.forward.computer \
--secrets tmp/test-secrets.json \
--strict- The gate runs preflight, luacheck, stylua, the core write smokes, and AO deep checks in a fixed order.
- Without
--strict, it still requires the send and primary readback probes to pass. --strictextends CU readback assertions to every URL you pass.
docs/ # command contracts, flows, failure modes, ADRs, runbooks
ao/ # AO command process and shared libs
write/ # command handlers, routing
shared/ # auth, idempotency, validation, audit
schemas/ # JSON schemas for command envelopes and actions
scripts/ # deploy | verify
fixtures/ # sample command envelopes and expected outcomes
tests/ # contract, conflict, and security tests
scripts/bridge/ # stub forwarder from write outbox to -ao
scripts/cli/ # local helpers (run command)
.github/workflows/ # CI entrypoint
- Required tags:
Action,Request-Id,Actor,Tenant,Expected-Version,Nonce,Signature-Ref,Timestamp. - Core handlers (initial set):
SaveDraftPage,PublishPageVersion,UpsertRoute,UpsertProduct,UpsertProfile,AssignRole,GrantEntitlement,CreateOrder,CreatePaymentIntent,ConfirmPayment. - Conflict strategy: reject on missing/expired nonce, replayed
Request-Id, or mismatchedExpected-Version; return prior result when replayed.
- Prereqs:
lua5.4(orluac) andpython3. - Static checks:
scripts/verify/preflight.sh(JSON schema validation + Lua syntax). - Contract smoke tests:
LUA_PATH="?.lua;?/init.lua;ao/?.lua;ao/?/init.lua" lua5.4 scripts/verify/contracts.lua(or setRUN_CONTRACTS=1to run during preflight). - Conflict/security smoke tests:
LUA_PATH="?.lua;?/init.lua;ao/?.lua;ao/?/init.lua" lua5.4 scripts/verify/conflicts.lua(orRUN_CONFLICTS=1). - Batch fixtures:
LUA_PATH="?.lua;?/init.lua;ao/?.lua;ao/?/init.lua" lua5.4 scripts/cli/batch_run.lua(orRUN_BATCH=1in preflight) – compares fixtures to*.expected.json. - Branches:
main(releasable),develop(integration),feature/*,adr/*,release/*. - Message contracts and schemas are public API; prefer additive changes over breaking ones.
- Install deps:
sudo apt-get install lua5.4 lua5.4-dev luarocks libsodium-dev
then install rocks from the lockfile:while read -r name ver; do case "$name" in \#*|"") continue ;; esac luarocks --lua-version=5.4 install --local "$name" "$ver" done < ops/rocks.lock
- Copy env template:
cp ops/env.prod.example ops/.env.localand fill secrets:OUTBOX_HMAC_SECRET(required)- signature verifier (
WRITE_SIG_PUBLICorWRITE_SIG_SECRETwhenWRITE_SIG_TYPE=hmac) - optional
WRITE_JWT_HS_SECRETif you turn onWRITE_REQUIRE_JWT=1.
- Run checks:
RUN_DEPS_CHECK=1 LUA_PATH="?.lua;?/init.lua;ao/?.lua;ao/?/init.lua" LUA_CPATH="$HOME/.luarocks/lib/lua/5.4/?.so" scripts/verify/preflight.sh(ormake preflight RUN_DEPS_CHECK=1). - Fixtures:
RUN_BATCH=1 LUA_PATH="?.lua;?/init.lua;ao/?.lua;ao/?/init.lua" lua scripts/cli/batch_run.lua(uses the env from step 2; hashes/nonce/signature checks can be relaxed viaWRITE_REQUIRE_*env). - Outbox/queue paths in the template default to
/var/lib/ao/...; for dev you can override todev/*paths next to the repo. - Optional specs:
- JWT:
RUN_JWT_SPEC=1 lua5.4 scripts/verify/jwt_actor_spec.lua+scripts/verify/jwt_expiry_spec.lua - Rate/nonce:
RUN_RATE_SPEC=1 WRITE_RATE_STORE_PATH=dev/write-rate-store.json lua5.4 scripts/verify/rate_store_spec.lua;RUN_RATE_SPEC=1 lua5.4 scripts/verify/rate_tenant_scope_spec.lua - Outbox HMAC:
RUN_OUTBOX_SPEC=1 lua5.4 scripts/verify/outbox_hmac_spec.lua
- JWT:
WRITE_REQUIRE_SIGNATURE=1— reject commands withoutsignatureRef.WRITE_REQUIRE_NONCE=1— reject commands without nonce and block replay.WRITE_NONCE_TTL_SECONDS(default 300) andWRITE_NONCE_MAX(default 2048) — nonce cache sizing.WRITE_ALLOW_ANON=1— allow missing actor/tenant (off by default).WRITE_SIG_TYPE=ed25519|ecdsa|hmac(prod default:ed25519).- Signature verification inputs:
WRITE_SIG_PUBLIC(single key; PEM path orhex:<pubkey>form),WRITE_SIG_PUBLICS(rotation/keyring map keyed bysignatureRef, supportsdefault),WRITE_SIG_SECRET(forhmacmode).
- SignatureRef policy gate:
WRITE_SIGNATURE_POLICY_JSONorWRITE_SIGNATURE_POLICY_PATH(JSON map keyed bysignatureRefwith per-keyactionsand optionalroles; unknown or missing refs fail closed with deterministicsignature_policy_*errors),- policy is enforced after signature verification and before the existing role checks.
WRITE_ROLE_POLICY_STRICT=1— fail closed when an action has no role-policy entry (role_policy_missing_action).- Optional JWT gate: set
WRITE_JWT_HS_SECRET(HS256) and optionallyWRITE_REQUIRE_JWT=1to fail-closed; claimssub/tenant/role/noncepopulateactor/tenant/role/noncewhen missing. WRITE_WAL_PATH=/var/lib/ao/write-wal.ndjson— append-only WAL with request/response hashes.WRITE_IDEM_PATH=/var/lib/ao/write-idem.json— persist idempotent responses across restarts (optional).WRITE_OUTBOX_PATH=/var/lib/ao/write-outbox.json— persist outbox events (used by forwarders/export).- Checksum watchdog:
ops/systemd/write-checksum.service+scripts/verify/checksum_daemon.sh(setWRITE_WAL_PATH,WRITE_OUTBOX_PATH,CHECKSUM_INTERVAL_SEC). - Resolver flags:
WRITE_FLAGS_PATH=/etc/ao/resolver-flags.ndjsonto block/readonly resolvers (shared with registry/AO); enforced before policy. - Shipping/Tax export for AO: persist rates with
WRITE_RATE_STORE_PATHand runscripts/export/rates.lua [rate_store] [shipping.ndjson] [tax.ndjson]; point AO to the outputs viaAO_SHIPPING_RATES_PATH/AO_TAX_RATES_PATH. - Dispute evidence payload:
AddDisputeEvidenceacceptsevidence.url|hash|hashAlgo|type|note|fileNameto carry provider links/hashes; stored inpayment_disputesand can be sent via provider webhooks. WRITE_RL_WINDOW_SECONDS/WRITE_RL_MAX_REQUESTS— rate-limit per tenant+actor (default 60s / 200 reqs).WRITE_RL_BUCKET_TTL_SECONDS/WRITE_RL_MAX_BUCKETS— trim idle buckets (default 4× window, 4096 buckets).WRITE_RATE_STORE_PATH— persist rate-limit buckets across restarts (optional; JSON file written atomically).WRITE_NONCE_STORE_PATH— persist nonce cache (tenant+actor namespaced) to survive restarts.- Bridge/env for queue/HTTP:
AO_ENDPOINT=https://...(optional);AO_API_KEY;DRY_RUN=1orAO_BRIDGE_MODE=mock|off|http;AO_BRIDGE_RETRIES/AO_BRIDGE_BACKOFF_MS;AO_QUEUE_PATH(persisted queue),AO_QUEUE_LOG_PATH=/var/lib/ao/queue-log.ndjson,AO_QUEUE_MAX_RETRIES=5,AO_EXPECT_RESPONSE_HASHto enforce downstream body hash. - PSP webhook hardening: set
STRIPE_WEBHOOK_SECRET(32-byte secret from Stripe),PAYPAL_WEBHOOK_STRICT=1to require PayPal signatures, tune replay cache withWRITE_WEBHOOK_REPLAY_WINDOW/WRITE_WEBHOOK_SEEN_TTL(and optionalWRITE_WEBHOOK_SEEN_MAX), and cap backlog withWRITE_WEBHOOK_RETRY_MAX_QUEUEalongside the existingWRITE_WEBHOOK_RETRY_*knobs. - Outbox HMAC enforcement:
WRITE_STRICT_OUTBOX_HMAC=1rejects outbox events withouthmacwhenOUTBOX_HMAC_SECRETis set (default off; forwarder still checks mismatches whenhmacis present). HMAC input defaults to full canonical JSON of the event; setWRITE_OUTBOX_HMAC_MODE=legacyto use the older limited field hash. - Trust manifest signing (resolvers): set
TRUST_MANIFEST_HMACand runlua scripts/cli/trust_manifest_sign.lua manifest.json > manifest.signed.json; optionally setTRUST_MANIFEST_SIGNER. - Key management: keep public keys under
/etc/ao/keys, record theirsha256sumin ops docs, rotate on a schedule; never store private keys in repos, artifacts, or CI logs. - OTP/passwordless flows and payment/PSP callbacks have been removed from this repo; keep secrets and such logic in upstream gateway/web layers.
docker compose build
docker compose run --rm write # runs full preflight (schemas, contracts, conflicts, batch)
docker compose run --rm write bash # drop into shell with deps preinstalledNotes: the image ships with Lua openssl + luasodium so PSP/webhook specs run with crypto enabled. outbox replay smoke will print hmac failures: missing=1 if OUTBOX_HMAC_SECRET is unset; set it to silence that warning.
lua scripts/cli/run_command.lua ./fixtures/sample-save-draft.json— route a JSON command locally and print the response (uses in-memory state). A publish sample is atfixtures/sample-publish.json.RUN_BATCH=1 LUA_PATH="?.lua;?/init.lua;ao/?.lua;ao/?/init.lua" lua scripts/cli/batch_run.lua— run all fixtures and enforce matches to*.expected.json(CI uses this).- Queue forwarder (persisted outbox → HTTP):
AO_QUEUE_PATH=dev/outbox-queue.ndjson AO_QUEUE_LOG_PATH=dev/queue-log.ndjson AO_QUEUE_MAX_RETRIES=5 LUA_PATH="?.lua;?/init.lua;ao/?.lua;ao/?/init.lua" lua scripts/bridge/queue_forward.lua - Outbox replay into a fresh queue (with HMAC verify):
OUTBOX_HMAC_SECRET=dev-secret WRITE_OUTBOX_PATH=dev/outbox.json AO_QUEUE_PATH=dev/outbox-queue.ndjson lua scripts/worker/outbox_replay.lua - Health snapshot (write-side files & deps):
WRITE_WAL_PATH=... WRITE_OUTBOX_PATH=... AO_QUEUE_PATH=... LUA_PATH="?.lua;?/init.lua;ao/?.lua;ao/?/init.lua" lua scripts/verify/health.lua - Export verifier (PII scrub check):
WRITE_OUTBOX_EXPORT_PATH=dev/outbox.ndjson lua scripts/verify/export_verify.lua
🚦 Production readiness — keep these on before serving real traffic.
- Set
WRITE_STRICT_OUTBOX_HMAC=1and ensure every emitted event includeshmac. - Keep signature/JWT verification on (WRITE_REQUIRE_SIGNATURE/WRITE_REQUIRE_JWT) and rotate keys regularly.
- Persist idempotency/rate buckets/outbox where applicable (
WRITE_IDEM_PATH,WRITE_RATE_STORE_PATH,WRITE_OUTBOX_PATH) and back them up. - Monitor
/metrics(bearer from METRICS_BEARER_TOKEN) forrate_limited,replay_nonce, and outbox HMAC counters; alert on sustained errors. - If Arweave verification is mandatory on all branches, set
ENFORCE_ARWEAVE_HASH=1and provideARWEAVE_VERIFY_FILE/ARWEAVE_VERIFY_TX; CI will fail-closed otherwise. - Licensing quick links: BFNL-1.0 and Founder Fee Policy are authoritative — keep deployments within their terms.
Common pitfalls
OUTBOX_HMAC_SECRETmust be 64 hex chars (32B) or HMAC drifts andsecrets_lintwill fail.WRITE_SIG_TYPEmust match the key you provide (public key for ed25519/ecdsa vs shared secret for hmac).WRITE_NONCE_TTL_SECONDS/ replay window: too low → false rejects; too high → oversized seen-cache.- Arweave hash gate: with
ENFORCE_ARWEAVE_HASH=1you need the correctARWEAVE_VERIFY_TX+ artifact, otherwise CI fails closed.
- Expose Prom-style
/metricsviaao.shared.metrics(seeMETRICS_PROM_PATH,METRICS_LOG,METRICS_BEARER_TOKEN). - Key counters:
outbox_queue_depth,write.outbox.queue_sizewrite_auth_signature_failed_total,write_auth_signature_missing_totalwrite_auth_jwt_invalid_total,write_auth_jwt_expired_total,write_auth_jwt_not_before_total,write_auth_jwt_skew_total,write_auth_jwt_*_mismatch_totalwrite_auth_nonce_replay_totalwrite_auth_rate_limited_totalwrite.idempotency.collisions,idempotency_collisions_totalwrite.webhook.retry_queue,webhook_retry_queue,write.webhook.retry_lag_seconds,webhook_retry_lag_seconds,webhook_retry_overduewrite.psp.breaker_open,breaker_openwrite.wal.apply_duration_seconds,wal_apply_duration_secondswrite_outbox_hmac_missing_total,write_outbox_hmac_mismatch_total
- Sample alerts (PromQL):
increase(write_auth_rate_limited_total[5m]) > 50increase(write_auth_jwt_invalid_total[5m]) > 5 or increase(write_auth_jwt_expired_total[5m]) > 20increase(write_auth_nonce_replay_total[5m]) > 0increase(write_outbox_hmac_mismatch_total[5m]) > 0 or increase(write_outbox_hmac_missing_total[5m]) > 5max_over_time(outbox_queue_depth[5m]) > 100 or increase(webhook_retry_queue[5m]) > 20increase(breaker_open[5m]) > 0
- Add alerts on rising trends; log/Prom output controlled by
METRICS_*envs inao/shared/metrics.lua. - More alert examples:
docs/ALERTS.md.
scripts/bridge/forward_outbox.luareads the in-memory outbox (write._storage_outbox()) and logs events you would forward toblackcat-darkmesh-ao. Replaceforward_eventwith signed POST to AO endpoint (registry/site process) in production.scripts/bridge/export_outbox.lua [outfile]dumps outbox to NDJSON (defaultdev/outbox.ndjson) for offline inspection or manual upload.scripts/bridge/forward_outbox_http.luaposts outbox events toAO_ENDPOINT(setDRY_RUN=1to log only; optionalAO_API_KEY,AO_SITE_IDtag).
- No secrets or raw keys in AO state, manifests, or adapters.
- Gateways act only as clients; write process re-validates auth and policy.
- All comments and docs remain in English.
Blackcat Darkmesh Write is licensed under BFNL-1.0 (see LICENSE). Contribution and relicensing rules are governed by the companion documents in blackcat-darkmesh-ao/docs/. This repository is an official component of the Blackcat Covered System; the repo split inside BLACKCAT_MESH_NEXUS is only for maintenance/safety/auditability and does not trigger a separate fee event for the same deployment.
Current active covered-system repos are: blackcat-darkmesh-ao, blackcat-darkmesh-gateway, blackcat-darkmesh-web, blackcat-darkmesh-write, and blackcat-templates. Legacy backend/crypto modules are tracked as internalized packages inside blackcat-darkmesh-gateway (for example libs/legacy/, kernel-migration/, and security/crypto-manifests/).
Canonical licensing bundle:
- BFNL 1.0: https://github.com/Vito416/blackcat-darkmesh-ao/blob/main/docs/BFNL-1.0.md
- Founder Fee Policy: https://github.com/Vito416/blackcat-darkmesh-ao/blob/main/docs/FEE_POLICY.md
- Covered-System Notice: https://github.com/Vito416/blackcat-darkmesh-ao/blob/main/docs/LICENSING_SYSTEM_NOTICE.md
- CI currently runs schema/lua/stylua checks, ingest/envelope/action validation, publish/idempotency/conflict/hmac smokes, and Arweave/hash gates (based on env flags).
- If
scripts/sign-write.jsis used in verify scripts, Node dependencies must be installed (npm cistep is required in CI).
