Skip to content

Build the disabled-by-default Ditto Proof v1 harness #14

Description

@ohad6k

Ditto has real usage signals, but it does not yet have controlled evidence showing when the personal layer helps.

Build a standalone, disabled-by-default benchmark harness for Ditto Proof v1. This issue covers the harness only, not provider runs or public claims.

Acceptance criteria

  • Freeze Ditto at v0.3.7 / 5f4008b0c0df40dcadb92c8fd1ba4dcf3aee40d0 for every scored cell.
  • Build exactly 24 paired comparisons and 48 isolated cell records: two systems, three task families, two variants, and two trials.
  • Keep host-native persistent personalization absent in both conditions.
  • Require fresh workspaces and dedicated homes for every cell.
  • Record exact system identity, host version, tools, permissions, budgets, instruction hashes, fixture commit/hash, attempts, and artifact hashes.
  • Make evidence append-only. A correction creates a superseding record instead of overwriting one.
  • Allow one retry only for a provider or host failure before meaningful output.
  • Require both --execute and an exact manifest-hash approval before any provider command can run.
  • Keep all private profiles, transcripts, reviewer records, and run artifacts outside Git.
  • Leave ditto.py, mining behavior, profiles, skills, and normal commands unchanged.
  • Keep the existing Ditto test suite green and add focused Windows, UTF-8, privacy, and isolation coverage.

Not part of this issue

  • Selecting or paying for systems
  • Running the scored benchmark
  • Recruiting reviewers
  • Publishing proof clips or performance claims
  • Antigravity changes

Implementation stops at a verified local harness and a separate execution approval gate.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestproof-v1Reproducible, privacy-safe Ditto Proof v1 work

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions