feat: add statistics + illustrative traces

[MrtinoRG]

Summary by Sourcery

Add analysis assets for the paper, including illustrative epistemic reasoning traces and a script to compute corpus-level statistics from experiment reports.

New Features:

• Introduce a LaTeX appendix section showcasing representative reasoning breakdown traces with annotated epistemic graphs for each major failure category.
• Add a Python analysis script that aggregates token usage, configuration–environment coverage, estimated API costs, and malformed-response rates from reports.jsonl.

Enhancements:

• Provide LaTeX-ready summary metrics and structured logging output to streamline transferring analysis statistics into the paper.

[sourcery-ai]

Reviewer's Guide

Adds an appendix-style LaTeX file with illustrative reasoning traces for four breakdown categories, and introduces a Python script that computes paper statistics (token usage, config–environment counts, estimated API cost, and scaffold error rates) from the experiment reports JSONL file.

Class diagram for compute_paper_stats module structure

classDiagram
  class compute_paper_stats_module {
    +dict PRICING
    +Path DATA_PATH
    +set config_env_pairs
    +defaultdict tokens
    +defaultdict scaffold_errors
    +defaultdict trials_affected_count
    +int total_trials
    +enc
    +count_tokens(text str) int
    +fmt_tokens(n int) str
    +agg_model(model str) (int input_tokens, int output_tokens)
    +agg_all() (int input_tokens, int output_tokens)
    +agg_verbosity(verb str) (int input_tokens, int output_tokens)
  }

  class tiktoken_encoder {
    +encode(text str) list~int~
  }

  class loguru_logger {
    +info(message str)
  }

  class reports_source {
    +path str
    +open()
  }

  compute_paper_stats_module --> tiktoken_encoder : uses
  compute_paper_stats_module --> loguru_logger : logs_via
  compute_paper_stats_module --> reports_source : reads_from

Loading

Flow diagram for compute_paper_stats statistics pipeline

flowchart TD
  A_Start([Start compute_paper_stats.py]) --> B_ReadFile
  B_ReadFile["Open results/data/reports.jsonl"] --> C_LoopLines

  subgraph S_LineProcessing[Per JSONL record]
    C_LoopLines --> D_Parse["Parse JSON line to rec"]
    D_Parse --> E_ExtractMeta["Extract model, agent_type, environment, verbosity"]
    E_ExtractMeta --> F_AddConfigEnv["Add (model, agent_type, verbosity, env) to config_env_pairs"]
    F_AddConfigEnv --> G_LoopTasks["Loop over Task Results"]

    subgraph S_TaskTrials[Per task and trial]
      G_LoopTasks --> H_LoopTrials["Loop over trials"]
      H_LoopTrials --> I_IncTotalTrials["Increment total_trials"]
      I_IncTotalTrials --> J_GetMessages["Get trial messages"]
      J_GetMessages --> K_TokenizeMsgs["Compute msg_tok via count_tokens(content)"]

      K_TokenizeMsgs --> L_AssistantTurns["For each assistant message"]
      L_AssistantTurns --> M_SumInput["Input tokens += sum(prior msg_tok)"]
      M_SumInput --> N_AddOutput["Output tokens += this assistant msg_tok"]

      K_TokenizeMsgs --> O_ScaffoldInit["Initialize trial_has_error=False"]
      O_ScaffoldInit --> P_ScanScaffold["Scan messages for scaffold errors"]
      P_ScanScaffold --> Q_UpdateScaffold["Update scaffold_errors[(model, agent_type)]"]
      Q_UpdateScaffold --> R_UpdateTrialsAffected{agent_type == react?}
      R_UpdateTrialsAffected -- Yes --> S_UpdateReact["Update trials_affected_count[model]"]
      R_UpdateTrialsAffected -- No --> T_NextTrial["Next trial"]
      S_UpdateReact --> T_NextTrial
    end
  end

  T_NextTrial --> U_NextLine{More lines?}
  U_NextLine -- Yes --> C_LoopLines
  U_NextLine -- No --> V_Aggregation

  subgraph S_Aggregation[Aggregation helpers]
    V_Aggregation["Compute aggregates via agg_model, agg_all, agg_verbosity"]
  end

  V_Aggregation --> W_LoggingSections

  subgraph S_Logging[Reporting via loguru]
    W_LoggingSections["Log configuration counts, token tables, costs, error rates, LaTeX-ready values"]
  end

  W_LoggingSections --> X_End([End])

Loading

File-Level Changes

Change	Details	Files
Introduce LaTeX appendix with illustrative epistemic traces for four reasoning-breakdown categories, including custom node badges, color scheme, and annotated examples with diagrams and annotator quotes.	Define color palette and a reusable \nodebadge macro for epistemic node types (H, T, E, J, C, F). Add a new subsection describing how to read the trace visualizations and linking to the online browser. For each breakdown category (evidence non-uptake, untested claim, fixed belief trace, contradiction without repair), add a tcolorbox with model/context metadata, excerpted messages, node/edge annotations, a small TikZ diagram, and an annotator quote explaining the pattern.	analysis/illustrative_traces.tex
Add an analysis script to compute aggregate experiment statistics (token counts, configuration–environment pairs, estimated proprietary API cost, and ReAct scaffold error rates) from reports.jsonl and log them in human- and LaTeX-ready formats.	Read analysis/results/data/reports.jsonl and iterate over all records, aggregating by model, agent type, environment, and verbosity. Compute per-assistant-turn token input/output using tiktoken with the o200k_base encoding, mirroring API usage by counting all prior messages as input and the current assistant message as output. Aggregate token statistics by model and verbosity, including grand totals, and pretty-print values using a compact formatter (K/M/B). Estimate API cost for proprietary models using a per-million-token pricing table, including a breakdown by verbosity. Detect ReAct scaffold errors by scanning for 'No actions to execute' in user messages, computing per-model error rates and percentage of affected trials. Log a summary section with LaTeX-ready scalar values for tokens, number of config–environment pairs, total API cost, and malformed-response rates for selected models.	analysis/compute_paper_stats.py

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 5b52b2fc-41ef-4670-befc-f331b21ccbb4

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch last_todoss

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[sourcery-ai]

Hey - I've found 1 issue, and left some high level feedback:

• The per-assistant-turn input token calculation currently does sum(msg_tok[:i]) inside the loop, which is O(n²) over messages; consider precomputing a prefix sum array so each input count is O(1) and the loop stays linear in the number of messages per trial.
• All the JSONL parsing and aggregation runs at import time; wrapping this logic in a main() function and guarding with if __name__ == "__main__": would make the module safer to import and easier to reuse programmatically.
• The top-level docstring still says "Compute the TODO statistics"—updating this to accurately describe the current outputs (tokens, config–env pairs, cost, malformed-response rates) will make the script’s purpose clearer to future readers.

Prompt for AI Agents

Please address the comments from this code review:

## Overall Comments
- The per-assistant-turn input token calculation currently does `sum(msg_tok[:i])` inside the loop, which is O(n²) over messages; consider precomputing a prefix sum array so each input count is O(1) and the loop stays linear in the number of messages per trial.
- All the JSONL parsing and aggregation runs at import time; wrapping this logic in a `main()` function and guarding with `if __name__ == "__main__":` would make the module safer to import and easier to reuse programmatically.
- The top-level docstring still says "Compute the TODO statistics"—updating this to accurately describe the current outputs (tokens, config–env pairs, cost, malformed-response rates) will make the script’s purpose clearer to future readers.

## Individual Comments

### Comment 1
<location path="analysis/illustrative_traces.tex" line_range="3" />
<code_context>
+% Illustrative trace excerpts for the four major reasoning-breakdown categories.
+% Requires: tcolorbox, tikz, xcolor, enumitem, listings
+% Usage: \input{analysis/results/illustrative_traces.tex}
+
+\definecolor{colENU}{RGB}{198,219,239}   % Evidence non-uptake — light blue
</code_context>
<issue_to_address>
**issue (bug_risk):** The documented \input path does not match the actual file location in the repo.

The usage line references `\input{analysis/results/illustrative_traces.tex}`, but the file is at `analysis/illustrative_traces.tex`. This mismatch will cause the include to fail if copied. Please either correct the documented path or move the file to match it.
</issue_to_address>

---

_{Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.}

[sourcery-ai]

issue (bug_risk): The documented \input path does not match the actual file location in the repo.

The usage line references \input{analysis/results/illustrative_traces.tex}, but the file is at analysis/illustrative_traces.tex. This mismatch will cause the include to fail if copied. Please either correct the documented path or move the file to match it.