State-machine writing arm: adopting ConStory-Checker + a million-word corpus offer
Hi ConStory-Bench team,
Your finding that multi-step agent systems score worse on consistency than strong base models changed how we design our project — we wrote a kill criterion into our own benchmark (if a structured state approach doesn't beat naive-truncation at small scale, we stop and fix the architecture instead of scaling up). That criterion came directly from reading your paper.
We're the Benxiang Protocol (本象) team — an open, Apache-2.0 persistent object layer for AI (github.com/dongsheng123132/2origin). We run a long-form writing benchmark (ShadowBench-W) that tracks hidden state across 95k+ word continuations: a state-machine arm (a3) maintains characters/locations/plot objects in a verifiable store, and on "state that is never stated in any sentence" it scores 6/6 where bare-model and vector-RAG arms score 0/6. We would like to put that arm through ConStory-Bench.
Three asks:
-
Leaderboard submissions — what is the process? Run our arms on the 2000 prompts, judge with ConStory-Checker, then PR the evaluations/*.csv? Or is there a contact/submission form?
-
The precise CED definition (normalization window, how the 19 subtypes are weighted, how inter-annotator disagreement is resolved) — the abstract page doesn't show the full spec, and we'd rather adopt your exact metric than publish a near-miss variant that fragments comparison.
-
Corpus offer — we construct long-form corpora by authoring a structured world specification first (characters, locations, rules, timeline, planted plot threads, every state change bound to an originating event), then generate prose from it. Ground truth exists by construction, and deliberately planted contradictions give computable precision/recall for bug detection. We're scaling our corpus from 30k to 1M words under CC0 — if a million-word extension of ConStory-Bench interests you, we'd rather build it to your specification than beside it.
Full disclosure: our current results are n=6, single model, self-authored corpus. We want the external check.
Thanks for building the benchmark — and for the finding that cost us an assumption.
State-machine writing arm: adopting ConStory-Checker + a million-word corpus offer
Hi ConStory-Bench team,
Your finding that multi-step agent systems score worse on consistency than strong base models changed how we design our project — we wrote a kill criterion into our own benchmark (if a structured state approach doesn't beat naive-truncation at small scale, we stop and fix the architecture instead of scaling up). That criterion came directly from reading your paper.
We're the Benxiang Protocol (本象) team — an open, Apache-2.0 persistent object layer for AI (github.com/dongsheng123132/2origin). We run a long-form writing benchmark (ShadowBench-W) that tracks hidden state across 95k+ word continuations: a state-machine arm (a3) maintains characters/locations/plot objects in a verifiable store, and on "state that is never stated in any sentence" it scores 6/6 where bare-model and vector-RAG arms score 0/6. We would like to put that arm through ConStory-Bench.
Three asks:
Leaderboard submissions — what is the process? Run our arms on the 2000 prompts, judge with ConStory-Checker, then PR the evaluations/*.csv? Or is there a contact/submission form?
The precise CED definition (normalization window, how the 19 subtypes are weighted, how inter-annotator disagreement is resolved) — the abstract page doesn't show the full spec, and we'd rather adopt your exact metric than publish a near-miss variant that fragments comparison.
Corpus offer — we construct long-form corpora by authoring a structured world specification first (characters, locations, rules, timeline, planted plot threads, every state change bound to an originating event), then generate prose from it. Ground truth exists by construction, and deliberately planted contradictions give computable precision/recall for bug detection. We're scaling our corpus from 30k to 1M words under CC0 — if a million-word extension of ConStory-Bench interests you, we'd rather build it to your specification than beside it.
Full disclosure: our current results are n=6, single model, self-authored corpus. We want the external check.
Thanks for building the benchmark — and for the finding that cost us an assumption.