Surface sync and import failures with row-level context

[joshka]

Parent: #100

Related Gitcrawl context: openclaw/gitcrawl#81

Observed Workflow

During initial Discrawl use, sparse database errors made it hard to tell whether a sync failure was a data-shape problem, an idempotency problem, or a local write-path bug.

One concrete case involved duplicate attachment insert errors. The useful first fix was not a broad behavior change; it was adding row-level context around the failing insert so the attempted attachment id, message id, guild id, channel id, author id, filename, content type, and size were visible.

Current Workaround

Without row context, failures have to be diagnosed by reading code, reproducing locally, or adding temporary logging.

That makes coverage and retry decisions harder because missing or partial data may reflect:

• not attempted yet
• failed during sync
• failed during local write
• fixed after retry
• intentionally skipped

Request

Surface sync/import/write failures with durable, queryable context.

Example shape:

discrawl failures --json
discrawl doctor --recent-failures --json

Suggested tracked fields:

• operation
• source, such as discord, wiretap, snapshot, media, embeddings
• guild id
• channel id
• message id, when applicable
• attachment id or related row id, when applicable
• error class
• error message
• first seen time
• last seen time
• retry count
• resolved time

Why This Matters

Codex-assisted archive use depends on knowing whether local data is missing because it has not been fetched yet, because a previous fetch failed, or because a local data-shape bug needs a targeted report. Without a ledger, that context disappears into shell logs.

Acceptance Criteria

• Recent sync/import/write failures are queryable in JSON.
• Failure context includes relevant Discord row identifiers when available.
• Successful retry or later ingestion can mark a failure resolved.
• Coverage/progress output can distinguish known failures from missing/unattempted rows.
• Existing raw SQLite errors are wrapped with enough row context before behavior changes are made.

---

Prepared with Codex, confirmed as accurate by human.

[clawsweeper]

Codex review: keeping this open for maintainer follow-up; there is still a little grit to resolve. Reviewed June 24, 2026, 11:37 AM ET / 15:37 UTC.

Summary
Keep open: current main partially addresses the duplicate attachment insert symptom, but it still lacks the requested durable, queryable sync/import/write failure ledger, failures or doctor --recent-failures JSON surface, and row-context wrapping for general write failures.

Reproducibility: yes. for the missing requested surface: current-main source inspection shows no failure ledger, no failures command, and no doctor --recent-failures output. The original duplicate attachment incident is described but not packaged as a runnable fixture or log.

Next step
Maintainers should choose the ledger schema, command shape, retention, and resolution semantics before any repair PR is queued.

Review details

Best possible solution:

Design a narrow failure ledger and JSON read surface, then instrument sync, wiretap/import, media/embedding, and store write paths so successful later ingestion can resolve row-scoped failures.

Do we have a high-confidence way to reproduce the issue?

Yes for the missing requested surface: current-main source inspection shows no failure ledger, no failures command, and no doctor --recent-failures output. The original duplicate attachment incident is described but not packaged as a runnable fixture or log.

Is this the best way to solve the issue?

Unclear until maintainers choose the persistent schema and lifecycle semantics. The likely best path is a durable ledger plus row-context wrappers before broader ingestion behavior changes.

AGENTS.md: not found in the target repository.

Remaining risk / open question:

• The durable schema, retention policy, and resolved/retry lifecycle need maintainer design before implementation.
• The duplicate attachment symptom is partly handled, but coverage output still cannot distinguish failed, missing, skipped, retried, and resolved rows across sync/import/write operations.

Codex review notes: model internal, reasoning high; reviewed against 670994a45f61.

Label changes

Label justifications:

• P2: This is a normal-priority diagnostic/product improvement for sync/import/write failures with limited blast radius.
• impact:data-loss: The issue concerns distinguishing missing local archive data from failed sync/import/write attempts and preserving that context for diagnosis.

Evidence reviewed

What I checked:

• Target policy check: No AGENTS.md or VISION.md exists under the target repository root, so no target-specific policy changed this review. (670994a45f61)
• CLI surface lacks failures command: The command dispatch includes sync, wiretap, attachments, sql, status, doctor, and related commands, but no failures command. (internal/cli/cli.go:204, 670994a45f61)
• Doctor has no recent-failures option: runDoctor parses only --json and rejects positional arguments, so it cannot query recent failure rows. (internal/cli/admin_commands.go:498, 670994a45f61)
• Schema has only narrow error fields: The schema has attachment fetch status/error and embedding job last_error fields, but no general operation/source/message/attachment failure ledger with first/last/resolved timestamps. (internal/store/store.go:366, 670994a45f61)
• Write path still returns raw attachment write errors: replaceAttachmentsTx returns InsertMessageAttachment errors directly, so non-duplicate attachment write failures are still not wrapped with attachment/message/guild/channel row context. (internal/store/write.go:420, 670994a45f61)
• Existing tracking is partial and source-specific: Media fetch failures are stored on attachment rows, and sync unavailable markers are channel-scoped in sync_state; neither is the requested row-level sync/import/write failure ledger. (internal/media/cache.go:93, 670994a45f61)

Likely related people:

• steipete: History shows substantial recent work in the store, attachment indexing/media, sqlc write refactor, sync retry/unavailable handling, and CLI diagnostics paths this feature would touch. (role: recent area contributor; confidence: high; commits: 408fb67b15b2, 724adb0088ae, bcd2929b8ac6; files: internal/store/write.go, internal/store/store.go, internal/store/attachments.go)
• vincentkoc: History ties this contributor to the shipped duplicate attachment snapshot fix, the v0.11.3 snapshot, and adjacent desktop import/cloud paths involved in sync/import diagnostics. (role: recent related implementation contributor; confidence: high; commits: 1abd76e437c9, 6a91d9cc67bb, 0a247939e841; files: internal/store/sqlc/queries.sql, internal/store/store_write_test.go, internal/discorddesktop/import.go)
• hannesrudolph: Recent syncer checkpoint and publisher-role work touched sync and store write behavior adjacent to retry/progress/failure resolution semantics. (role: adjacent sync/write contributor; confidence: medium; commits: 265a76cb90e3, f77e560318fa; files: internal/syncer/message_sync.go, internal/store/write.go)

How this review workflow works

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.