docs: CONTRIBUTING, AGENTS, CI, issue templates, and unit tests

.env.example

@@ -0,0 +1,16 @@
+# ─── IGenBench API Keys ───────────────────────────────────────────────────────
+# Copy this file to .env and fill in your keys.
+# Never commit .env to version control.
+
+# Google AI Studio — generation + evaluation
+# https://aistudio.google.com/app/apikey
+GOOGLE_API_KEY=
+
+# OpenRouter — generation + evaluation (OpenAI-compatible)
+# https://openrouter.ai/keys
+OPENROUTER_API_KEY=
+
+# Replicate — generation only
+# Requires: uv sync --extra replicate
+# https://replicate.com/account/api-tokens
+REPLICATE_API_TOKEN=

.github/ISSUE_TEMPLATE/bug_report.yml

@@ -0,0 +1,65 @@
+name: Bug Report
+description: Report something that is not working correctly.
+labels: ["bug"]
+body:
+  - type: markdown
+    attributes:
+      value: |
+        Thanks for taking the time to fill out this bug report.
+        Please search existing issues before submitting.
+
+  - type: textarea
+    id: description
+    attributes:
+      label: What happened?
+      description: A clear description of the bug.
+    validations:
+      required: true
+
+  - type: textarea
+    id: reproduce
+    attributes:
+      label: Steps to reproduce
+      description: The exact commands or code that trigger the bug.
+      placeholder: |
+        1. Run `igenbench eval --info-path ...`
+        2. See error ...
+    validations:
+      required: true
+
+  - type: textarea
+    id: expected
+    attributes:
+      label: Expected behaviour
+    validations:
+      required: true
+
+  - type: textarea
+    id: logs
+    attributes:
+      label: Relevant logs / error output
+      render: shell
+
+  - type: input
+    id: version
+    attributes:
+      label: IGenBench version
+      placeholder: "e.g. 0.1.0 or git SHA"
+    validations:
+      required: true
+
+  - type: input
+    id: python
+    attributes:
+      label: Python version
+      placeholder: "e.g. 3.11.9"
+    validations:
+      required: true
+
+  - type: input
+    id: os
+    attributes:
+      label: Operating system
+      placeholder: "e.g. macOS 15, Ubuntu 24.04, Windows 11"
+    validations:
+      required: true

.github/ISSUE_TEMPLATE/feature_request.yml

@@ -0,0 +1,37 @@
+name: Feature Request
+description: Suggest a new feature or improvement.
+labels: ["enhancement"]
+body:
+  - type: markdown
+    attributes:
+      value: |
+        Thanks for the suggestion! Please search existing issues first.
+
+  - type: textarea
+    id: problem
+    attributes:
+      label: Problem or motivation
+      description: What problem does this feature solve, or what use case does it enable?
+    validations:
+      required: true
+
+  - type: textarea
+    id: solution
+    attributes:
+      label: Proposed solution
+      description: Describe the feature you'd like and how it should work.
+    validations:
+      required: true
+
+  - type: textarea
+    id: alternatives
+    attributes:
+      label: Alternatives considered
+      description: Any other approaches you considered?
+
+  - type: checkboxes
+    id: willing
+    attributes:
+      label: Are you willing to implement this?
+      options:
+        - label: Yes, I'd like to submit a PR.

.github/workflows/ci.yml

@@ -0,0 +1,49 @@
+name: CI
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+    branches: [main]
+
+jobs:
+  lint:
+    name: Lint & Format
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Set up uv
+        uses: astral-sh/setup-uv@v4
+        with:
+          version: "latest"
+
+      - name: Install dependencies
+        run: uv sync --dev
+
+      - name: Ruff lint
+        run: uv run ruff check .
+
+      - name: Ruff format check
+        run: uv run ruff format --check .
+
+  test:
+    name: Tests (Python ${{ matrix.python-version }})
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ["3.10", "3.11", "3.12"]
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Set up uv
+        uses: astral-sh/setup-uv@v4
+        with:
+          version: "latest"
+          python-version: ${{ matrix.python-version }}
+
+      - name: Install dependencies
+        run: uv sync --dev
+
+      - name: Run unit tests
+        run: uv run pytest tests/ -v --tb=short

AGENTS.md

@@ -0,0 +1,150 @@
+# AGENTS.md — IGenBench Codebase Guide for AI Coding Agents
+
+This file helps AI agents (Claude Code, GitHub Copilot, Cursor, etc.) navigate and modify this codebase correctly.
+
+---
+
+## What This Project Does
+
+IGenBench is a CLI benchmark tool for evaluating the **reliability** of text-to-infographic generation models. Given a dataset of `VISItem` JSON files (each containing a prompt, reference image, and evaluation questions), it:
+
+1. **Generates** infographics using a text-to-image model (`igenbench gen` / `igenbench batch-gen`)
+2. **Evaluates** generated images by asking an LLM to answer factual Q&A questions about the image (`igenbench eval` / `igenbench batch-eval`)
+3. **Scores** the results to compute Q-ACC and I-ACC metrics (`igenbench score`)
+
+---
+
+## Architecture
+
+```
+CLI (typer)
+  └── Workflow (orchestration + resume logic)
+        └── Engine (single-item gen/eval step)
+              └── LLMCaller (provider abstraction)
+                    ├── GoogleCaller
+                    ├── OpenrouterCaller
+                    └── ReplicateCaller
+```
+
+### Key Files
+
+| File | Purpose |
+|------|---------|
+| `igenbench/vis_item.py` | Core data model. **Read this first.** |
+| `igenbench/cli/main.py` | Typer app; registers all sub-commands |
+| `igenbench/cli/gen_cli.py` | `igenbench gen` command |
+| `igenbench/cli/eval_cli.py` | `igenbench eval` command |
+| `igenbench/cli/batch_cli.py` | `igenbench batch-gen` and `batch-eval` |
+| `igenbench/cli/score_cli.py` | `igenbench score` command |
+| `igenbench/workflow/gen_workflow.py` | Generation orchestration |
+| `igenbench/workflow/eval_workflow.py` | Evaluation orchestration with resume |
+| `igenbench/engine/gen_engine.py` | Calls LLMCaller.generate_image for one item |
+| `igenbench/engine/eval_engine.py` | Calls LLMCaller.understand_image + parses answer |
+| `igenbench/utils/llm/llm_caller.py` | All provider implementations + `LLMCaller` base |
+| `igenbench/utils/llm/caller_registry.py` | `@register_caller` decorator + lookup |
+| `igenbench/utils/state_manager.py` | Reads/writes VISItem JSON to disk |
+| `igenbench/utils/path_resolver.py` | Resolves output paths from item id + model name |
+| `prompts/gen_text2image.py` | Prompt template for image generation |
+| `prompts/eval_judge_factual_qa.py` | Prompt template for evaluation |
+
+---
+
+## Data Model
+
+```python
+VISItem
+  id: str                        # e.g. "001"
+  t2i_prompt: str                # generation prompt
+  chart_type: str                # e.g. "bar", "pie"
+  reference_image_url: str       # URL of ground-truth image
+  generation: dict[model, path]  # generated image paths keyed by model name
+  evaluation: list[EvalEntry]
+
+EvalEntry
+  source: str          # "prompt" | "seed"
+  question: str        # e.g. "How many bars are there?"
+  ground: str          # ground-truth answer
+  question_type: str   # reliability dimension (e.g. "count", "ordering")
+  judgments: list[Judgment]
+
+Judgment
+  gen_model: str       # model that generated the image
+  eval_model: str      # model that evaluated it
+  analysis: str        # chain-of-thought reasoning
+  answer: str          # "1" (correct) or "0" (incorrect)
+```
+
+All items are serialised to `{output_dir}/{item_id}/{item_id}.json`.
+
+---
+
+## How to Add a New Provider
+
+```python
+# igenbench/utils/llm/llm_caller.py
+from .caller_registry import register_caller
+from .llm_caller import LLMCaller
+
+@register_caller("my_provider")
+class MyProviderCaller(LLMCaller):
+    def __init__(self) -> None:
+        # initialise your SDK / client here
+        pass
+
+    def generate_image(self, model: str, prompt: str, **kwargs):
+        # return a PIL.Image.Image
+        ...
+
+    def understand_image(self, model: str, prompt: str, image_path: str, **kwargs) -> str:
+        # return the model's text response
+        ...
+```
+
+Then call with `--provider my_provider`.
+
+---
+
+## Dev Commands
+
+```bash
+# Install (including dev deps)
+uv sync --dev
+
+# Lint
+uv run ruff check .
+
+# Format
+uv run ruff format .
+
+# Tests (no API keys needed)
+uv run pytest tests/ -v
+
+# Run a single gen (requires GOOGLE_API_KEY)
+uv run igenbench gen \
+  --info-path hf_datasets/data/1.json \
+  --output-dir outputs/ \
+  --provider google \
+  --model gemini-2.5-flash-image
+```
+
+---
+
+## Output Directory Layout
+
+```
+outputs/
+└── {item_id}/
+    ├── {item_id}.json          # VISItem state (generation + evaluation results)
+    └── {gen_model}/
+        └── {item_id}.png       # generated image
+```
+
+---
+
+## Common Mistakes to Avoid
+
+- **Never** add API keys to source files. Use environment variables (see `.env.example`).
+- `VISItem.from_dict(path)` takes a **file path**, not a dict.
+- `EvalEntry.judgments` is a list — multiple (gen_model, eval_model) pairs can coexist on the same question (multi-model comparison).
+- The `Question` dataclass in `vis_item.py` is **deprecated**; use `EvalEntry` instead.
+- When editing CLI commands, register them via `igenbench/cli/main.py` imports (side-effect import pattern).

CHANGELOG.md

@@ -0,0 +1,21 @@
+# Changelog
+
+All notable changes to IGenBench are documented here.
+Format follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/);
+versioning follows [PEP 440](https://peps.python.org/pep-0440/).
+
+---
+
+## [0.1.0] — 2026-06
+
+### Added
+- `igenbench gen` — generate a single infographic from a VISItem JSON
+- `igenbench eval` — evaluate a single generated image against benchmark questions
+- `igenbench batch-gen` — generate infographics for the full dataset with `--resume`
+- `igenbench batch-eval` — evaluate the full dataset with `--resume`
+- `igenbench score` — aggregate Q-ACC / I-ACC scores with optional `--by-source` / `--by-type` breakdown
+- Provider support: Google (Gemini), OpenRouter, Replicate
+- `@register_caller` plugin API for adding custom LLM providers
+- `VISItem` data model with full JSON serialisation
+- Incremental progress: every question result is saved immediately so interrupted runs resume cleanly
+- ACL 2026 camera-ready release