Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
70 changes: 70 additions & 0 deletions .github/ISSUE_TEMPLATE/experiment_result.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
name: Experiment result
description: Submit a reproducible WAMProbe result for review
title: "[Result]: "
labels: ["experiment-result", "reproduction"]
body:
- type: markdown
attributes:
value: >-
Submit structured evidence, not only a screenshot. Do not include tokens,
credentials, private datasets, browser sessions, or model weights.
- type: input
id: wamprobe_revision
attributes:
label: WAMProbe version and commit
placeholder: 0.1.0rc1 / full Git commit SHA
validations:
required: true
- type: textarea
id: model_provenance
attributes:
label: Adapter and model provenance
description: Include adapter revision, upstream repository revision, checkpoint ID/hash, license, and preprocessing configuration.
validations:
required: true
- type: textarea
id: benchmark
attributes:
label: Benchmark configuration
description: Include benchmark ID, context count/IDs, action set, seed, horizon, split, and any deviations.
validations:
required: true
- type: textarea
id: environment
attributes:
label: Environment and compute
description: Include operating system, Python, accelerator/driver, dependency lock or image digest, peak memory, and elapsed time.
validations:
required: true
- type: textarea
id: commands
attributes:
label: Exact commands
render: shell
validations:
required: true
- type: textarea
id: artifacts
attributes:
label: Artifacts and SHA256 hashes
description: Link or attach summary.json, results.jsonl, report.md/report.html, evidence manifest, and hashes where public redistribution is allowed.
validations:
required: true
- type: textarea
id: interpretation
attributes:
label: Result and limitations
description: Report the metric profile, unsupported metrics, uncertainty unit, failures, and the narrow claim supported by this run.
validations:
required: true
- type: checkboxes
id: integrity
attributes:
label: Evidence integrity
options:
- label: Every action branch uses the same restored initial context.
required: true
- label: The submitted result was generated by the revisions and commands above.
required: true
- label: No secret, private data, checkpoint, or unverifiable substituted output is attached.
required: true
82 changes: 82 additions & 0 deletions .github/ISSUE_TEMPLATE/external_reproduction.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,82 @@
name: External reproduction
description: Report an independent clean-install WAMProbe smoke reproduction
title: "[External reproduction]: "
labels: ["reproduction"]
body:
- type: markdown
attributes:
value: >-
Thank you for independently checking WAMProbe. This form records the
evidence needed for the external-smoke release gate. Failed attempts are
useful too. Do not post credentials or private machine paths.
- type: input
id: reproducer
attributes:
label: Reproducer name or handle
validations:
required: true
- type: input
id: date
attributes:
label: Date and timezone
placeholder: 2026-07-16 UTC
validations:
required: true
- type: input
id: revision
attributes:
label: WAMProbe tag or full commit SHA
validations:
required: true
- type: textarea
id: environment
attributes:
label: Clean environment
description: Include OS/architecture, Python version, and whether this was a new venv, container, or machine.
validations:
required: true
- type: textarea
id: commands
attributes:
label: Installation and smoke commands
render: shell
validations:
required: true
- type: dropdown
id: result
attributes:
label: Result
options:
- PASS
- FAIL
validations:
required: true
- type: input
id: exit_status
attributes:
label: Exit status and elapsed time
placeholder: exit 0; 42 seconds
validations:
required: true
- type: input
id: output_hash
attributes:
label: Output summary.json SHA256
validations:
required: true
- type: textarea
id: deviations
attributes:
label: Deviations, errors, or notes
description: Write "None" when the documented procedure was followed exactly. For a failure, include the smallest non-sensitive error output.
validations:
required: true
- type: checkboxes
id: independence
attributes:
label: Independence confirmation
options:
- label: I performed this install and smoke test independently; this is not a maintainer-run substitute for an external user.
required: true
- label: I reviewed the form and removed tokens, credentials, private paths, and other secrets.
required: true
35 changes: 34 additions & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,27 @@

Thank you for helping make WAM evaluation more causal, reproducible, and control-grounded.

## Choose a contribution path

- **Connect a model:** run the
[custom adapter starter](docs/adapters/CUSTOM_ADAPTER.md), then open an
[Adapter proposal](https://github.com/myheart521/WAMProbe/issues/new?template=adapter_proposal.yml).
- **Add a paired diagnostic:** open a
[Benchmark proposal](https://github.com/myheart521/WAMProbe/issues/new?template=benchmark_proposal.yml)
before generating a large dataset.
- **Add or change a metric:** open a
[Metric proposal](https://github.com/myheart521/WAMProbe/issues/new?template=metric_proposal.yml)
and state how the metric can be gamed.
- **Share a run:** use the
[Experiment result](https://github.com/myheart521/WAMProbe/issues/new?template=experiment_result.yml)
form with standard artifacts and hashes.
- **Reproduce independently:** use the
[External reproduction](https://github.com/myheart521/WAMProbe/issues/new?template=external_reproduction.yml)
form. A maintainer-controlled run cannot satisfy the external-user gate.
- **Pick a bounded task:** see the
[roadmap](docs/NEXT_STEPS.md) and issues labeled
[`good first issue`](https://github.com/myheart521/WAMProbe/labels/good%20first%20issue).

## Development setup

```bash
Expand All @@ -10,6 +31,15 @@ source .venv/bin/activate
python -m pip install -e '.[dev]'
```

Verify the adapter starter before replacing its backend:

```bash
python -m examples.custom_adapter.run \
--output runs/custom-adapter \
--contexts 8 \
--seed 7
```

Before opening a pull request, run:

```bash
Expand All @@ -31,11 +61,14 @@ the audit itself.
- Follow test-driven development for behavior changes.
- Keep the core package CPU-only and avoid heavy mandatory dependencies.
- New adapters must publish a capability declaration and pin an upstream revision used by
their smoke test.
their smoke test. Preserve context/action identities and reject malformed model outputs
at the adapter boundary.
- New benchmarks must document generation, splits, licenses, and shared-state guarantees.
- New metrics must include a metric card, expected baseline ordering, and at least one
failure/anti-gaming test.
- Do not commit datasets, checkpoints, generated runs, tokens, or browser sessions.
- Result submissions must identify the WAMProbe, adapter, model, checkpoint, preprocessing,
and benchmark revisions; include exact commands and public artifact hashes.

Please use focused commits and explain both the scientific reason and user-visible impact
of a change.
22 changes: 20 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -241,6 +241,15 @@ Start with the [scope and capability RFC](docs/rfcs/0001-scope-and-capabilities.
[counterfactual metrics RFC](docs/rfcs/0002-counterfactual-metrics.md), and the
[adapter selection record](docs/research/ADAPTER_SELECTION.md).

For a working integration instead of pseudocode, run the tested
[custom adapter starter kit](examples/custom_adapter/README.md) and follow the
[step-by-step adapter guide](docs/adapters/CUSTOM_ADAPTER.md):

```bash
python -m examples.custom_adapter.run \
--output runs/custom-adapter --contexts 8 --seed 7
```

## Real-model artifacts

Model weights are never committed to Git. The first StarWAM spike requires approximately
Expand Down Expand Up @@ -280,11 +289,14 @@ Review the [reproducibility guide](docs/reproducibility/REPRODUCIBILITY.md) and
| Run the CPU demo | [Quick start](docs/QUICKSTART.md) |
| Understand the scientific motivation | [WAM/VLA failure-case evidence map](docs/research/WAM_VLA_FAILURE_CASES.md) |
| Interpret a score correctly | [Core metric cards](docs/metrics/CORE_METRICS.md) |
| Add a model integration | [Scope and capability RFC](docs/rfcs/0001-scope-and-capabilities.md) |
| Add a model integration | [Custom adapter starter guide](docs/adapters/CUSTOM_ADAPTER.md) |
| Submit a structured experiment | [Experiment result form](https://github.com/myheart521/WAMProbe/issues/new?template=experiment_result.yml) |
| Reproduce independently | [External reproduction form](https://github.com/myheart521/WAMProbe/issues/new?template=external_reproduction.yml) |
| Reproduce LIBERO-CF-Mini | [Benchmark card](docs/benchmarks/LIBERO_CF_MINI.md) |
| Inspect the closed-loop protocol | [Experiment card](docs/experiments/TOY_CLOSED_LOOP_V0.1.md) |
| Audit the release | [Reproducibility guide](docs/reproducibility/REPRODUCIBILITY.md) |
| Read the full implementation plan | [Detailed Chinese project plan](docs/WAMProbe_PLAN.md) |
| Choose the next milestone | [Dependency-aware roadmap](docs/NEXT_STEPS.md) |
| Browse all documentation | [Published documentation site](https://myheart521.github.io/WAMProbe/) |

## Project status and roadmap
Expand All @@ -293,7 +305,8 @@ The `v0.1.0rc1` engineering scope is implemented and published. The remaining fo
acceptance item is one genuine independent clean-install smoke report in
[Issue #2](https://github.com/myheart521/WAMProbe/issues/2).

Next research milestones:
The [dependency-aware roadmap](docs/NEXT_STEPS.md) tracks evidence gates separately from
repository work. Next research milestones:

1. expand LIBERO initial-state coverage;
2. evaluate action-conditioned real-WAM futures when an adapter exposes that capability;
Expand All @@ -306,6 +319,11 @@ Contributions are welcome—especially adapters, paired benchmark generators, me
anti-gaming tests, and independent reproductions. New metrics must document the failure
they detect and include a sanity check against reference baselines.

Start with a labeled
[`good first issue`](https://github.com/myheart521/WAMProbe/labels/good%20first%20issue),
an [Adapter proposal](https://github.com/myheart521/WAMProbe/issues/new?template=adapter_proposal.yml),
or the [Experiment result form](https://github.com/myheart521/WAMProbe/issues/new?template=experiment_result.yml).

```bash
python -m pip install -e '.[dev]'
ruff format --check .
Expand Down
21 changes: 19 additions & 2 deletions README.zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -235,6 +235,15 @@ class MyWorldModel:
[反事实指标 RFC](docs/rfcs/0002-counterfactual-metrics.md)和
[适配器选择记录](docs/research/ADAPTER_SELECTION.md)。

如果希望从可运行代码开始,而不是照着伪代码重新搭建,请运行已经测试过的
[自定义 Adapter Starter Kit](examples/custom_adapter/README.md),并按照
[适配器接入指南](docs/adapters/CUSTOM_ADAPTER.md)逐步替换后端:

```bash
python -m examples.custom_adapter.run \
--output runs/custom-adapter --contexts 8 --seed 7
```

## 真实模型文件

模型权重永远不会提交到 Git。第一个 StarWAM 验证需要约 46.3 GB 固定版本的 StarWAM 和
Expand Down Expand Up @@ -271,19 +280,22 @@ StarWAM 和 LIBERO 的隔离环境、GPU 预检、预处理来源及 smoke 命
| 运行 CPU 演示 | [快速开始](docs/QUICKSTART.md) |
| 理解项目的研究动机 | [WAM/VLA 失败案例证据图谱](docs/research/WAM_VLA_FAILURE_CASES.md) |
| 正确解释指标 | [核心指标卡](docs/metrics/CORE_METRICS.md) |
| 接入一个新模型 | [范围与能力 RFC](docs/rfcs/0001-scope-and-capabilities.md) |
| 接入一个新模型 | [自定义适配器接入指南](docs/adapters/CUSTOM_ADAPTER.md) |
| 提交结构化实验结果 | [实验结果表单](https://github.com/myheart521/WAMProbe/issues/new?template=experiment_result.yml) |
| 独立复现项目 | [外部复现表单](https://github.com/myheart521/WAMProbe/issues/new?template=external_reproduction.yml) |
| 复现 LIBERO-CF-Mini | [基准卡](docs/benchmarks/LIBERO_CF_MINI.md) |
| 检查闭环实验协议 | [实验卡](docs/experiments/TOY_CLOSED_LOOP_V0.1.md) |
| 审计发行版 | [可复现性指南](docs/reproducibility/REPRODUCIBILITY.md) |
| 阅读完整实施计划 | [WAMProbe 中文规划文档](docs/WAMProbe_PLAN.md) |
| 选择下一阶段任务 | [带依赖关系的后续路线图](docs/NEXT_STEPS.md) |
| 浏览全部文档 | [在线文档站](https://myheart521.github.io/WAMProbe/) |

## 项目状态与路线图

`v0.1.0rc1` 工程范围已经实现并发布。正式验收目前只剩一项:由真实独立用户在
[Issue #2](https://github.com/myheart521/WAMProbe/issues/2) 提交干净环境安装复现报告。

下一阶段研究计划:
[后续路线图](docs/NEXT_STEPS.md)会把外部证据门槛和仓库内工程任务分开跟踪。下一阶段研究计划:

1. 扩展 LIBERO 初始状态覆盖;
2. 在适配器真正暴露对应能力后,评测动作条件真实 WAM 未来;
Expand All @@ -295,6 +307,11 @@ StarWAM 和 LIBERO 的隔离环境、GPU 预检、预处理来源及 smoke 命
欢迎提交适配器、配对基准生成器、指标反作弊测试和独立复现报告。新增指标必须说明它要
发现的失败模式,并包含相对参考基线的 sanity check。

可以先认领带有
[`good first issue`](https://github.com/myheart521/WAMProbe/labels/good%20first%20issue) 标签的任务,
提交 [Adapter 提案](https://github.com/myheart521/WAMProbe/issues/new?template=adapter_proposal.yml),
或使用[实验结果表单](https://github.com/myheart521/WAMProbe/issues/new?template=experiment_result.yml)。

```bash
python -m pip install -e '.[dev]'
ruff format --check .
Expand Down
Loading