Download
+Start with the public, editable Gloss v1 deck.
+ Get the .pptx ↓ +diff --git a/.github/ISSUE_TEMPLATE/benchmark-case.yml b/.github/ISSUE_TEMPLATE/benchmark-case.yml index 79b8414..e02d31f 100644 --- a/.github/ISSUE_TEMPLATE/benchmark-case.yml +++ b/.github/ISSUE_TEMPLATE/benchmark-case.yml @@ -1,5 +1,5 @@ -name: Propose a benchmark case -description: Add a slide behavior that generated PowerPoint artifacts should prove. +name: Propose a challenge case +description: Add a native presentation behavior that generated artifacts should prove. title: "[case]: " labels: ["benchmark-case"] body: diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 3ce6090..24ace50 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,11 +1,13 @@ # Contributing to Gloss -Gloss improves when a benchmark claim becomes easier to inspect, reproduce, or challenge. Small, evidence-backed pull requests are welcome. +Gloss improves when the challenge presentation becomes harder to fake and easier to +understand. Small, evidence-backed pull requests are welcome. ## Good first contributions -- Add a PowerPoint construct the corpus does not cover. +- Add a native presentation behavior the deck does not cover. - Reproduce a grader false positive or false negative. +- Make one of the twenty slide explanations or prompts clearer. - Improve a candidate assertion and its mutation fixture. - Improve a deterministic comparative generation path and prove the score change. - Review a prompt requirement, evidence record, or release gate. @@ -30,6 +32,7 @@ Install [uv](https://docs.astral.sh/uv/), Python 3.12, LibreOffice Impress, and git clone https://github.com/aronchick/gloss.git cd gloss/gloss-v1/grader uv sync --extra dev --locked +uv run gloss check ../benchmark/deck/gold/gloss-v1-gold.pptx uv run ../benchmark/validate_corpus.py uv run pytest tests/test_mutation_fixtures.py -q ``` @@ -57,9 +60,9 @@ uv run pytest --cov=gloss_service --cov-fail-under=85 uv run pip-audit ``` -## Adding a benchmark case +## Adding a challenge case -A benchmark change should identify: +A deck change should identify: - the natural-language requirement; - the intended visual and native artifact behavior; @@ -83,12 +86,11 @@ Keep a pull request focused enough to review. In the description, include: CI must pass. Maintainers may ask for a smaller fixture, clearer provenance, or an explicit release-state caveat before merging. -## Launch surface +## Website -The static site lives in `site/`. Harness counts come from -`site/evidence/preview-v1.json`; comparative bars come from the byte-equivalent -public copy of `gloss-v1/benchmark/comparative-v1/summary.json`. Verify both -sources, local assets, copy, and media metadata with: +The static site lives in `site/` and should lead with the real deck, the exact +prompt, and GitHub contribution paths. Verify links, assets, copy, evidence, and +media metadata with: ```bash node launch/verify-launch.mjs diff --git a/README.md b/README.md index d0298a5..7390b5c 100644 --- a/README.md +++ b/README.md @@ -1,211 +1,315 @@ # Gloss -**Gloss — Generative Layout & OOXML Scoring System** is an open, deliberately -difficult PowerPoint deck that AI struggles to render and build as a real, editable -artifact. Its accompanying measurement tools make the failure modes easy to inspect, -reproduce, and improve. +**A deliberately complicated presentation that AI struggles to make natively.** -The tools inspect both sides of a deck: +Gloss is an open, 20-slide challenge for presentation-generating AI. The goal is +not to make screenshots that happen to look like slides. The goal is to make a +real presentation: editable text, native charts and tables, working masters and +layouts, live fields, connected shapes, correct reading order, and intentional +grouping all the way down. -- **Looks right:** rendered pixels, typography, geometry, and visual consistency. -- **Built right:** native shapes, charts, tables, layouts, masters, text semantics, - relationships, and package safety. +The deck is the challenge. Gloss also includes a small checker that makes native +object changes easy to see, plus deeper measurement tools for people developing +generators and evaluation systems. -**[gloss.tools](https://gloss.tools)** · -[Issues](https://github.com/aronchick/gloss/issues) · -[Pull requests](https://github.com/aronchick/gloss/pulls) · -[Contributing](CONTRIBUTING.md) +The name expands to **Generative Layout & Object Structure Standard**: a reminder +that layout is visible, while object structure is what keeps a presentation real. -> **A screenshot is not a PowerPoint.** The artifact is the product. +**[Open gloss.tools](https://gloss.tools)** · +**[Download the deck](https://github.com/aronchick/gloss/raw/refs/heads/main/gloss-v1/benchmark/deck/gold/gloss-v1-gold.pptx)** · +**[Copy the exact prompt](gloss-v1/benchmark/prompts/DESIGNER_BRIEF.md)** · +**[Join the work](https://github.com/aronchick/gloss/issues)** -## The ACID idea +> **A screenshot is not a presentation. The artifact is the product.** -The browser ACID tests made standards failures impossible to hand-wave away: every -browser received the same hostile, public artifact, and everyone could inspect the -result. They targeted open web standards rather than one browser's preferred output. +## Remember ACID? -Gloss brings that **ACID-test philosophy** to generated presentations: +[Acid1](https://www.w3.org/Style/CSS/Test/CSS1/current/sec53.htm), +[Acid2](https://www.webstandards.org/action/acid2/), and +[Acid3](https://acid3.acidtests.org/) gave every browser the same hostile public +artifact. You did not need a white paper to see what the browser understood and +what it did not. -- publish the torture cases instead of hiding them; -- grade the final `.pptx`, not a generation trace or a screenshot; -- test both visible output and standards-based OOXML structure; -- make every assertion, fixture, score, and failure reproducible; -- fail closed when required evidence or rendering stages are missing; -- invite the community to add cases and make the grader stricter. +**Gloss is ACID for presentation decks made by AI.** It publishes the hard +artifact, the exact instructions, the reference renders, and the checks so anyone +can inspect a result, find a failure, and improve the ecosystem. -> **Remember [Acid1](https://www.w3.org/Style/CSS/Test/CSS1/current/sec53.htm) and -> [Acid2](https://www.webstandards.org/action/acid2/)? Gloss is ACID for AI-made -> presentation decks.** +Here, ACID refers to the browser conformance-test tradition—not database +transactions. **Gloss** is the public project. **Gloss v1** is its first bundled +challenge suite. -Here, **ACID refers to the public browser conformance-test tradition, not database -transactions**. +## Try it in three minutes -## Product and suite +1. [Download the native Gloss v1 deck](https://github.com/aronchick/gloss/raw/refs/heads/main/gloss-v1/benchmark/deck/gold/gloss-v1-gold.pptx). +2. Move one object on one slide and a second object on another slide. Save it as + `edited.pptx`. +3. Run the checker. It reports those two native objects—one finding per object. -**Gloss** is the public challenge deck and collaboration project. Its companion -measurement tools make failure modes, evidence, and comparative results easy to -inspect and reproduce. +```bash +git clone https://github.com/aronchick/gloss.git +cd gloss/gloss-v1/grader +uv sync --extra dev --locked +uv run gloss check /path/to/edited.pptx +``` -**Gloss v1 is the first ACID-style PowerPoint conformance suite bundled with -Gloss.** It supplies the current prompt corpus, hostile deck cases, assertions, gold -fixture, mutation tests, and protocol adapter: +An untouched copy returns: -| Name | Role | -| --- | --- | -| **Gloss** | The deliberately difficult presentation deck and public project | -| **ACID tests** | The open conformance-testing philosophy | -| **Gloss v1** | One bundled benchmark protocol and corpus | +```text +Exact match. No native objects changed. +``` -> **Technical-preview status:** the bundled -> [`GLOSS_OPENSPEC.md`](GLOSS_OPENSPEC.md) contract is still Draft and its -> official-release gates remain incomplete. Gloss is live for public collaboration, -> but no result should be treated as an official model leaderboard until a release -> announcement names a frozen scoring cohort. +Move two objects and it returns only two findings: -## Technical preview evidence +```text +2 native objects changed: + Slide 02 · placeholder “Agenda” · changed: position + Slide 12 · placeholder “Document Fields” · changed: position +``` -The current Gloss v1 candidate suite contains 280 schema-valid checks across -rendered pixels and native package structure. Its generated operator suite passes -280/280 positive controls and detects 280/280 generated single-fault mutations. -This proves configured operator behavior only; it is not independent assertion -evidence and it is not a model leaderboard. +`gloss check` is the friendly front door: a strict native-object diff against the +public deck. The deeper `gloss grade` protocol remains available for rendered +comparison, semantic assertions, provenance, and release work. -Every public count is recorded in -[`site/evidence/preview-v1.json`](site/evidence/preview-v1.json) and checked against -the committed mutation reports: +## The deck -```bash -node launch/verify-launch.mjs -``` +Each slide isolates something that can look correct while being built incorrectly. +Open any reference image, then inspect the corresponding +[canonical slide instructions](gloss-v1/benchmark/prompts/variants/canonical). -## Frozen Gloss comparison +### 01 — Cover / Title Stress Test -Gloss includes a frozen comparative bundle produced against Gloss v1: -[`gloss-v1/benchmark/comparative-v1`](gloss-v1/benchmark/comparative-v1). -It contains four repository-owned generation paths, three public seeds per path, -and 12 editable 20-slide decks. The canonical Linux/amd64 grader completed all -240 slide renders. + -The current local artifact scores are 67.68% for the native paths and 62.32% for -the visual paths. These are reproducible workflow baselines, not model rankings. -Every report says `local artifact score; self-reported` and carries no model -attribution. +**Demonstrates:** a cinematic title composition with an image, translucent shapes, +and a master-driven footer. **Tests natively:** layout placeholders, image crop, +z-order, opacity, shadow, and inherited slide numbers. -Reproduce every deck, report, hash, and public bar: +### 02 — Dense Agenda with Layout Semantics -```bash -./gloss-v1/benchmark/comparative-v1/reproduce.sh -``` + -Release mode intentionally fails until independent prompt convergence, assertion -provenance and evidence, reviewer approvals, baselines, environment manifests, and -signed release indexes are complete. +**Demonstrates:** a dense agenda that still reads cleanly. **Tests natively:** +placeholder use, real bullet levels, paragraph spacing and indentation, flat +grouping, and exact alignment. -## Score provenance +### 03 — Native Table Stress Test -- **Local grading** measures a deck on the caller's machine. Local results are - self-reported and are not official leaderboard verification. -- **Hosted grading** runs the same protocol adapter inside the controlled Gloss - environment. Every public submission score must carry the exact label - `grading-verified artifact score; generation-attested`. -- Generation strategy, token count, cost, retries, and generation time are supplied - by submitters. They remain **generation-attested** even when the deck itself was - grading-verified. -- A report is not eligible when schema validation did not run or failed, the - canonical renderer or reference exports were unavailable, required slides were - not compared, or any required stage was incomplete. + -## Repository layout +**Demonstrates:** a formatted performance table with a connected annotation. +**Tests natively:** a real table—not a pile of rectangles—plus cell styling, +mixed border weights, paragraph indents, and connector placement. -```text -site/ Static gloss.tools source and public evidence -launch/ Gloss film renderer, verifier, and launch copy -GLOSS_OPENSPEC.md Draft contract for the bundled Gloss v1 suite -gloss-v1/ - benchmark/ ACID-style prompts, cases, checks, fixtures, and evidence - grader/ Gloss v1 protocol adapter and local CLI - schemas/ Bundled ECMA-376 Transitional XSD and report schemas - service/ Pre-release hosted grading components - Dockerfile Canonical Ubuntu 22.04 grading environment -``` +### 04 — Native Chart Stress Test -## Run the bundled suite locally + -Gloss currently exercises its first suite through the `gloss` CLI. Requirements: -[uv](https://docs.astral.sh/uv/), Python 3.12, LibreOffice Impress, and `pdftoppm` -from Poppler. +**Demonstrates:** a data-rich chart with annotations and takeaways. **Tests +natively:** an editable chart with correct source data, chart type, labels, +gridlines, legend, overlay shapes, and arrow connectors. -```bash -cd gloss-v1/grader -uv sync --extra dev --locked -uv run gloss validate ../benchmark/deck/gold/gloss-v1-gold.pptx -``` +### 05 — Master Reuse Enforcement -Full grading is fail-closed until a frozen release supplies signed cohort provenance -and the caller supplies a complete artifact-context handoff. During development, -use `validate` for the runnable quarantine/XSD smoke above. After the v1 release -artifacts are published: + -```bash -uv run gloss grade ../submission.pptx \ - --tier 3 \ - --artifact-context ./artifact-context.json \ - --format json -``` +**Demonstrates:** a repeatable card system. **Tests natively:** master and layout +inheritance, grouped card objects, equal distribution, placeholders, and a footer +that is inherited instead of manually copied. -Write a standalone report and private visual diffs: +### 06 — Multilingual Editorial -```bash -uv run gloss grade ../submission.pptx \ - --tier 3 \ - --artifact-context ./artifact-context.json \ - --format html \ - --output report.html \ - --artifacts private-diffs/ -``` + -The protocol CLI exits with status `2` when verification could not be completed. -A normally graded deck may receive a failing score without making the grading -command itself fail. +**Demonstrates:** English, Arabic, and Japanese sharing one editorial canvas. +**Tests natively:** RTL paragraph direction, CJK line breaking, script-specific +fonts, exact Unicode text, and an intentional cross-column overlap. -## Canonical container +### 07 — Image Crop and Mask -```bash -docker build -t gloss-v1 gloss-v1 -docker run --rm \ - --volume "$PWD:/workspace:ro" \ - gloss-v1 grade /workspace/submission.pptx --tier 3 \ - --artifact-context /workspace/artifact-context.json --format json -``` + -## Quality gates +**Demonstrates:** three treatments of approved source imagery. **Tests natively:** +independent crop rectangles, circle and rounded-rectangle masks, asset identity, +and a translucent overlay in the right z-order. -```bash -cd gloss-v1/grader -uv run ruff check . -uv run ruff format --check . -uv run mypy gloss tests -uv run pytest --cov=gloss --cov-fail-under=85 -uv build -uv run pip-audit -``` +### 08 — Overlap, Shadow, and Transparency + + + +**Demonstrates:** a cascading stack with convincing depth. **Tests natively:** +precise z-order, five opacity levels, shadows, selective grouping, and a real +gradient fill. + +### 09 — Dense Text Overflow + + + +**Demonstrates:** two code-heavy columns with deliberately different overflow +behavior. **Tests natively:** fixed-size clipping versus auto-shrink, preserved +whitespace, monospaced runs, exact indentation, and line spacing. + +### 10 — Connector and Alignment Diagram + + + +**Demonstrates:** a legible systems diagram. **Tests natively:** connectors that +remain attached to shapes, elbow routing, arrowheads, grid alignment, labels, and +nested backend groups. + +### 11 — Theme vs. Local Override + + + +**Demonstrates:** two rows of nearly identical swatches. **Tests natively:** the +invisible semantic difference between theme-linked colors and explicit RGB +overrides—the kind of distinction a screenshot cannot prove. + +### 12 — Native Field Slide + + + +**Demonstrates:** live document metadata next to a static lookalike. **Tests +natively:** slide-number and fixed-date fields, a master footer field, and the +difference between a live field and typed text. + +### 13 — Composite Stress + + -The bundled validator applies deterministic ECMA-376 Markup Compatibility -preprocessing before validating against the bundled Part 1 Transitional XSD set. -RELAX NG validation is outside the Gloss v1 contract. +**Demonstrates:** chart, table, image, Arabic annotation, and dense callouts in one +composition. **Tests natively:** multiple editable object types and their +relationships surviving together. + +### 14 — RTL-Heavy Comparison + + + +**Demonstrates:** a mirrored Arabic/English comparison. **Tests natively:** +right-to-left paragraphs, mixed bidirectional runs, script-aware fonts, and +mirrored alignment without flattening text. + +### 15 — Rotated Text + + + +**Demonstrates:** typography at 0°, 45°, 90°, 135°, and 270°. **Tests natively:** +real rotation values, anchor points, bounding boxes, and shapes aligned to rotated +text. + +### 16 — Intentional Off-Canvas Bleed + + + +**Demonstrates:** design elements that deliberately continue beyond the canvas. +**Tests natively:** negative coordinates and overhanging geometry without clipping, +normalizing, or “fixing” intentional overflow. + +### 17 — Deep Grouping + + + +**Demonstrates:** a composition built from nested visual systems. **Tests natively:** +three levels of groups, exact membership, child transforms, and z-order inside each +nesting level. + +### 18 — Multi-Column Editorial + + + +**Demonstrates:** a three-column English/Japanese magazine layout. **Tests +natively:** column structure, repeated image treatments, pull quotes, mixed-script +text, and a patterned background. + +### 19 — Repetition and Consistency + + + +**Demonstrates:** the deck’s design language as a consistency audit. **Tests +natively:** master-derived typography, colors, spacing, line weights, and reusable +layout behavior across the full document. + +### 20 — Final Torture Slide + + + +**Demonstrates:** the whole challenge in one dense synthesis. **Tests natively:** a +chart, table, cropped image, Arabic and Japanese text, rotated type, transparent +layers, connectors, fields, groups, and master inheritance at once. + +## Give the exact challenge to an AI + +The complete prompt is already assembled in one copyable file: + +**[Open the exact Gloss v1 authoring prompt →](gloss-v1/benchmark/prompts/DESIGNER_BRIEF.md)** + +Give the AI that prompt together with: + +- the [20 reference images](gloss-v1/benchmark/deck/exports); +- the [approved assets](gloss-v1/benchmark/assets); +- the [bundled font set](gloss-v1/benchmark/fonts). + +Do not give it the native gold deck. Ask for an editable native presentation, +save the result as `.pptx`, and run `gloss check` against it. The experiment is +intentionally reproducible: every instruction, reference, asset, and check is in +this repository. + +## Native presentations, not one app + +Gloss is about native presentations across PowerPoint, Google Slides, and Keynote. +The principle is format-independent: a chart should remain a chart, text should +remain text, and layout behavior should survive editing. + +Gloss v1 is honestly **PPTX-first**. OOXML is a published standard with an +inspectable package and object structure, which makes independent local checking +possible today. Google Slides and Keynote coverage should use the same public, +artifact-first philosophy as their reliable inspection surfaces mature. Help us +define those adapters in [Issues](https://github.com/aronchick/gloss/issues). + +## Measurement is the support layer + +The simple check answers the first question: *what native objects changed?* The +bundled advanced grader can also inspect rendered pixels, typography, geometry, +charts, tables, layouts, masters, text semantics, relationships, and package +safety. Those tools exist to make this hard presentation easier to understand and +improve; they are not the product’s identity. + +The current candidate suite contains 280 schema-valid checks and generated +operator fixtures. That evidence proves configured checker behavior, not an +official model leaderboard. See [GLOSS_OPENSPEC.md](GLOSS_OPENSPEC.md) for the +draft protocol and [the public evidence bundle](site/evidence/preview-v1.json) for +the receipts. + +## Public today, harder tomorrow + +Gloss v1 is public by design. That also means it may become training data for the +systems it tests. This release is the starting line, not a permanent secret exam. + +Future generations need automated ways to create difficult decks, hold cases back, +delay disclosure, rotate prompts, and publish them after evaluation. The public +suite remains valuable as a shared conformance target; hidden generations will +measure generalization. ## Build it with us -The ACID tradition works because hard cases are public. Help Gloss make generated -decks worth editing: +The browser ACID tests mattered because standards failures became visible and the +community could make the tests better. Gloss should work the same way. -- [Propose a benchmark case](https://github.com/aronchick/gloss/issues/new?template=benchmark-case.yml) -- [Report a grader gap](https://github.com/aronchick/gloss/issues/new?template=grader-bug.yml) +- [Propose a difficult presentation behavior](https://github.com/aronchick/gloss/issues/new?template=benchmark-case.yml) +- [Report a checker gap](https://github.com/aronchick/gloss/issues/new?template=grader-bug.yml) - [Challenge evidence or a public claim](https://github.com/aronchick/gloss/issues/new?template=evidence-challenge.yml) -- Read [CONTRIBUTING.md](CONTRIBUTING.md) before sending a patch +- [Send a pull request](https://github.com/aronchick/gloss/pulls) +- Read [CONTRIBUTING.md](CONTRIBUTING.md) + +## Repository map -## License +```text +gloss-v1/benchmark/deck/ Native challenge deck + 20 reference renders +gloss-v1/benchmark/prompts/ One exact brief + per-slide prompt variants +gloss-v1/benchmark/assets/ Approved source assets +gloss-v1/grader/ Simple checker + advanced grading protocol +site/ Static gloss.tools source +GLOSS_OPENSPEC.md Draft protocol for the bundled v1 suite +``` Code and benchmark materials are released under the -[Apache License 2.0](LICENSE), except third-party or asset files that carry their -own license notices. +[Apache License 2.0](LICENSE), except third-party assets with their own notices. diff --git a/gloss-v1/benchmark/README.md b/gloss-v1/benchmark/README.md index d08d929..c0b811b 100644 --- a/gloss-v1/benchmark/README.md +++ b/gloss-v1/benchmark/README.md @@ -6,6 +6,11 @@ Prompt-derived structural requirements are frozen before gold authoring. Reference exports later become authoritative only for explicitly provenance- linked visual assertions; gold OOXML never creates a scored requirement. +Start with [`prompts/DESIGNER_BRIEF.md`](prompts/DESIGNER_BRIEF.md): it is the +exact, copyable master prompt containing the deck-wide rules and all twenty slide instructions. Provide +it with the reference PNGs in `deck/exports/`, the approved assets, and the bundled +fonts. Do not provide the native gold deck to the system being tested. + ## Integrity validation From `gloss-v1/grader/` using the locked grader environment: diff --git a/gloss-v1/benchmark/prompts/DESIGNER_BRIEF.md b/gloss-v1/benchmark/prompts/DESIGNER_BRIEF.md index ed2df4c..27e4258 100644 --- a/gloss-v1/benchmark/prompts/DESIGNER_BRIEF.md +++ b/gloss-v1/benchmark/prompts/DESIGNER_BRIEF.md @@ -742,27 +742,88 @@ This slide's purpose is to verify that the master/layout is being used correctly ### SLIDE 13: Composite Stress Combines: native chart (pie chart, 6 segments), native table (4×3), one approved image, Arabic text annotation, repeated layout elements, dense callout annotations. +## Explicit v1 constraints + +- Use the **Blank with Footer** layout and title the slide `Composite Stress` in Carlito 36pt Bold at (1.5cm, 0.5cm). +- Add one native pie chart at (1.5cm, 3cm), size 10cm × 8cm. Use six labeled values: `Structure 28`, `Text 22`, `Visual 18`, `Assets 14`, `Fields 10`, `Other 8`; show percentage labels and no legend. +- Add one native 4-row × 3-column table at (13cm, 3cm), size 9cm × 7cm. Header: `Check`, `Status`, `Weight`. Rows: `Schema | Pass | 3`, `Rendering | Pass | 2`, `Assets | Review | 1`. +- Insert `cityscape.png` at (23cm, 3cm), size 8cm × 6cm, cropping 15% from the left. Use only asset ID `cityscape` from the manifest. +- Add the Arabic annotation `الهيكل والمحتوى والعرض في اختبار واحد` in Noto Sans Arabic 14pt with native RTL direction. +- Add three dense rounded callouts labeled `NATIVE CHART`, `NATIVE TABLE`, and `APPROVED ASSET`; connect each to its target and keep all data labels visible. + ### SLIDE 14: RTL-Heavy Comparison Two-column layout with heavy Arabic text on left (5+ paragraphs with mixed Arabic/English), English summary on right. Bidirectional text within single paragraphs (e.g., "The الذكاء الاصطناعي revolution"). Mirrored alignment: Arabic column is right-aligned, English column is left-aligned. +## Explicit v1 constraints + +- Use the **Two-Column** layout with title `RTL Systems Review`. +- The left column contains at least five separate Arabic paragraphs in Noto Sans Arabic 14pt with native RTL paragraph direction. Include these exact phrases: `مراجعة الأنظمة الموزعة`, `دقة العرض`, `سلامة البنية`, `The الذكاء الاصطناعي revolution`, and `الإصدار Gloss v1`. +- The right column contains five English summary paragraphs in Liberation Sans 14pt, beginning with `A structural benchmark must preserve meaning and direction.` +- Right-align the Arabic column and left-align the English column. Mirror their inner padding and keep a 1pt teal divider centered between them. +- Mixed Arabic/English runs must preserve Unicode character order and visible punctuation without converting text to outlines. + ### SLIDE 15: Rotated Text Multiple text boxes at different rotation angles: 0°, 45°, 90°, 135°, 270°. Supporting shapes aligned to the rotated text. Tests exact anchor points for rotated elements. +## Explicit v1 constraints + +- Use the **Blank with Footer** layout with title `Rotation Atlas`. +- Create five text boxes labeled exactly `Baseline 0°`, `Diagonal 45°`, `Vertical 90°`, `Reverse 135°`, and `Vertical 270°`. +- Apply rotations of 0°, 45°, 90°, 135°, and 270° respectively. Each text box is 5cm × 1.5cm, uses Liberation Sans 16pt Bold, and has a matching 6cm × 2cm supporting rectangle sharing its center and rotation. +- Distribute the five pairs evenly from x=2cm through x=28cm on a common y=8cm anchor line. Keep text editable and store rotations in native OOXML transforms. + ### SLIDE 16: Intentional Off-Canvas Bleed Objects deliberately extending beyond the slide canvas edges — a large circle at (-3cm, -2cm) partly visible, a rectangle extending 5cm past the right edge. These are intentional design choices, not errors. +## Explicit v1 constraints + +- Use the **Blank with Footer** layout with title `Beyond the Frame`. +- Place a 10cm diameter coral circle at (-3cm, -2cm), a 12cm × 6cm teal rectangle at (28.867cm, 6cm) so it extends 5cm beyond the right edge, and a 16cm × 3cm gold strip at (8cm, -1cm). +- Add on-canvas labels `INTENTIONAL BLEED` and `Negative coordinates are part of the composition.` +- Preserve negative and over-bound coordinates in OOXML. Do not replace the composition with a screenshot, hide content under opaque shapes, or move the objects fully on-canvas. + ### SLIDE 17: Deep Grouping 3 levels of nested groups. Inner groups contain exactly 3 shapes each. Middle groups contain exactly 2 inner groups. Outer group wraps everything. Z-order must be exact within each nesting level. +## Explicit v1 constraints + +- Use the **Blank with Footer** layout with title `Nested Systems`. +- Build one outer group containing three middle groups. Each middle group contains two inner groups; each inner group contains three leaf shapes (18 leaf shapes total). +- Use native group objects at all three levels. Label the middle groups `INPUT`, `PROCESS`, and `OUTPUT`; label inner groups `A1`, `A2`, `B1`, `B2`, `C1`, and `C2`. +- Within every inner group, z-order is circle below rounded rectangle below label. The full outer group's bounding box has top-left position (3cm, 3cm) and size 27cm × 13cm. + ### SLIDE 18: Multi-Column Editorial 3-column magazine-style layout with mixed English/Japanese text. Each column has a header image (from assets), body text, and a pull quote. Pattern fill on the background of one column. +## Explicit v1 constraints + +- Use the **Blank with Footer** layout with title `Three Cities / 三つの都市`. +- Build three equal 9.5cm-wide editorial columns with 0.9cm gutters. Each column contains a 9.5cm × 3.5cm approved header image, a heading, body copy, and a pull quote. +- Column images, left to right: `cityscape.png`, `texture-pattern.png`, `cityscape.png`. Center the crop of each image and do not use unlisted media. +- Headings: `Systems`, `システム`, `Shared Futures`. Include the exact Japanese line `生成されたスライドは構造と意味を保持します。` +- Apply the approved `texture-pattern.png` as a tiled background only to the center column at 20% opacity. Use 0.9cm gutters consistently. + ### SLIDE 19: Repetition and Consistency Visually identical to the design system — every element must match the master's font sizes, colors, spacing, and line weights exactly. Tests deck-wide consistency. Internal hyperlinks to Slides 1 and 5 (not external URLs). +## Explicit v1 constraints + +- Use the **Content Slide** layout with title `Design System Audit`. +- Show five samples labeled `Typography`, `Palette`, `Spacing`, `Line Weight`, and `Navigation`. Each sample must use the deck-level tokens exactly: Carlito 36pt, Liberation Sans 18pt, navy `#1B2A4A`, coral `#E8634A`, teal `#2AACB8`, and a 0.5pt teal rule. +- Create two native internal hyperlinks: `Return to Cover` targets Slide 1 and `Meet the Team` targets Slide 5. Do not use external URLs. +- Footer line, company name, accent bar, and slide number remain inherited from the master/layout; do not duplicate them as slide-local shapes. + ### SLIDE 20: Final Torture Slide Everything combined: chart (line chart), table (3×6), approved image (cropped), Arabic + Japanese text, rotated text box at 45°, overlapping semi-transparent shapes, nested group, master-inherited footer, gradient fill, bullet list, slide number field — all on one slide. +## Explicit v1 constraints + +- Use the **Blank with Footer** layout with title `Gloss Synthesis`. +- Add one native line chart with series `Structural`, `Visual`, and `Combined` across categories `L1`, `L2`, `L3`; values are `72, 84, 96`, `68, 82, 95`, and `70, 83, 97`. +- Add one native 3-row × 6-column table. Header: `Tier`, `Slides`, `Checks`, `SSIM`, `Schema`, `Status`; rows: `L1 | 5 | 70 | 0.9999 | Pass | Ready` and `L3 | 20 | 280 | 0.9999 | Pass | Ready`. +- Insert `hero-abstract.png` with a 20% left crop. Add Arabic `اختبار شامل` with native RTL direction and Japanese `総合テスト` in Noto Sans CJK JP. +- Include a 45° text box labeled `ROTATED`, three overlapping semi-transparent shapes inside a two-level group, a navy-to-teal gradient, a three-item bullet list, a live slide-number field, and the inherited master footer. + --- ## Asset Manifest diff --git a/gloss-v1/benchmark/prompts/validation/slide-20.md b/gloss-v1/benchmark/prompts/validation/slide-20.md index bfc932d..0fc4665 100644 --- a/gloss-v1/benchmark/prompts/validation/slide-20.md +++ b/gloss-v1/benchmark/prompts/validation/slide-20.md @@ -3,9 +3,9 @@ "authors": [], "pairwise_similarity_diagnostics": [], "prompt_hashes": { - "canonical": "sha256:69711e77502102776aae817c7417faad5d3ae0ea8e29c823554cbc27c5b989d9", - "paraphrase-a": "sha256:86adb541d0961cefeb3d1388cdc4d6fea1f989d1485bb6dafbc5043d177204a7", - "paraphrase-b": "sha256:943121b317a29b1be5cf83776833d3c0c8e25aef6176f3d1bdefd3e8d4e72e05" + "canonical": "sha256:ba8e51e1becddfd40e2f2a02c5161a1caa09481255fcb9ff586f6b32ff866245", + "paraphrase-a": "sha256:c00f3398a053de3f9ac2b9a34697a8cb4eebefaf90ee1e73bc878ba24cd58081", + "paraphrase-b": "sha256:e37739606bb0dd690de9ac19cd76c51d2a95f3310d46c2c4e1c4b18d00474ac4" }, "record_id": "gloss-prompt-validation-slide-20", "round_id": null, @@ -18,9 +18,9 @@ - Prompt contract status: complete - Canonical/paraphrase hard-constraint parity: pass -- Canonical SHA-256: `69711e77502102776aae817c7417faad5d3ae0ea8e29c823554cbc27c5b989d9` -- Paraphrase A SHA-256: `86adb541d0961cefeb3d1388cdc4d6fea1f989d1485bb6dafbc5043d177204a7` -- Paraphrase B SHA-256: `943121b317a29b1be5cf83776833d3c0c8e25aef6176f3d1bdefd3e8d4e72e05` +- Canonical SHA-256: `ba8e51e1becddfd40e2f2a02c5161a1caa09481255fcb9ff586f6b32ff866245` +- Paraphrase A SHA-256: `c00f3398a053de3f9ac2b9a34697a8cb4eebefaf90ee1e73bc878ba24cd58081` +- Paraphrase B SHA-256: `e37739606bb0dd690de9ac19cd76c51d2a95f3310d46c2c4e1c4b18d00474ac4` - Independent-author convergence: not run - Required blinded author count: 3 - Mandatory assertion pass status: pending for all 3 implementations diff --git a/gloss-v1/benchmark/prompts/variants/canonical/slide-20.md b/gloss-v1/benchmark/prompts/variants/canonical/slide-20.md index 7cace1a..959aa69 100644 --- a/gloss-v1/benchmark/prompts/variants/canonical/slide-20.md +++ b/gloss-v1/benchmark/prompts/variants/canonical/slide-20.md @@ -4,8 +4,6 @@ The natural-language requirements below are the primary directive. Use the refer Everything combined: chart (line chart), table (3×6), approved image (cropped), Arabic + Japanese text, rotated text box at 45°, overlapping semi-transparent shapes, nested group, master-inherited footer, gradient fill, bullet list, slide number field — all on one slide. ---- - ## Explicit v1 constraints - Use the **Blank with Footer** layout with title `Gloss Synthesis`. @@ -13,3 +11,5 @@ Everything combined: chart (line chart), table (3×6), approved image (cropped), - Add one native 3-row × 6-column table. Header: `Tier`, `Slides`, `Checks`, `SSIM`, `Schema`, `Status`; rows: `L1 | 5 | 70 | 0.9999 | Pass | Ready` and `L3 | 20 | 280 | 0.9999 | Pass | Ready`. - Insert `hero-abstract.png` with a 20% left crop. Add Arabic `اختبار شامل` with native RTL direction and Japanese `総合テスト` in Noto Sans CJK JP. - Include a 45° text box labeled `ROTATED`, three overlapping semi-transparent shapes inside a two-level group, a navy-to-teal gradient, a three-item bullet list, a live slide-number field, and the inherited master footer. + +--- diff --git a/gloss-v1/benchmark/prompts/variants/paraphrase-a/slide-20.md b/gloss-v1/benchmark/prompts/variants/paraphrase-a/slide-20.md index 930409f..7046460 100644 --- a/gloss-v1/benchmark/prompts/variants/paraphrase-a/slide-20.md +++ b/gloss-v1/benchmark/prompts/variants/paraphrase-a/slide-20.md @@ -4,8 +4,6 @@ The natural-language requirements that follow are the primary directive. Treat t Everything combined: chart (line chart), table (3×6), approved image (cropped), Arabic + Japanese text, rotated text box at 45°, overlapping semi-transparent shapes, nested group, master-inherited footer, gradient fill, bullet list, slide number field — all on one slide. ---- - ## Mandatory v1 requirements - Select the **Blank with Footer** layout with title `Gloss Synthesis`. @@ -13,3 +11,5 @@ Everything combined: chart (line chart), table (3×6), approved image (cropped), - Include one native 3-row × 6-column table. Header: `Tier`, `Slides`, `Checks`, `SSIM`, `Schema`, `Status`; rows: `L1 | 5 | 70 | 0.9999 | Pass | Ready` and `L3 | 20 | 280 | 0.9999 | Pass | Ready`. - Position `hero-abstract.png` with a 20% left crop. Include Arabic `اختبار شامل` with native RTL direction and Japanese `総合テスト` in Noto Sans CJK JP. - Include a 45° text box labeled `ROTATED`, three overlapping semi-transparent shapes inside a two-level group, a navy-to-teal gradient, a three-item bullet list, a live slide-number field, and the inherited master footer. + +--- diff --git a/gloss-v1/benchmark/prompts/variants/paraphrase-b/slide-20.md b/gloss-v1/benchmark/prompts/variants/paraphrase-b/slide-20.md index 36b29fe..69d9479 100644 --- a/gloss-v1/benchmark/prompts/variants/paraphrase-b/slide-20.md +++ b/gloss-v1/benchmark/prompts/variants/paraphrase-b/slide-20.md @@ -4,8 +4,6 @@ The primary directive is the following set of natural-language requirements. Con Everything combined: chart (line chart), table (3×6), approved image (cropped), Arabic + Japanese text, rotated text box at 45°, overlapping semi-transparent shapes, nested group, master-inherited footer, gradient fill, bullet list, slide number field — all on one slide. ---- - ## Fixed v1 acceptance constraints - Adopt the **Blank with Footer** layout with title `Gloss Synthesis`. @@ -13,3 +11,5 @@ Everything combined: chart (line chart), table (3×6), approved image (cropped), - Provide one native 3-row × 6-column table. Header: `Tier`, `Slides`, `Checks`, `SSIM`, `Schema`, `Status`; rows: `L1 | 5 | 70 | 0.9999 | Pass | Ready` and `L3 | 20 | 280 | 0.9999 | Pass | Ready`. - Embed `hero-abstract.png` with a 20% left crop. Provide Arabic `اختبار شامل` with native RTL direction and Japanese `総合テスト` in Noto Sans CJK JP. - Include a 45° text box labeled `ROTATED`, three overlapping semi-transparent shapes inside a two-level group, a navy-to-teal gradient, a three-item bullet list, a live slide-number field, and the inherited master footer. + +--- diff --git a/gloss-v1/benchmark/requirements/prompt-requirements.json b/gloss-v1/benchmark/requirements/prompt-requirements.json index 3f48a40..51e2092 100644 --- a/gloss-v1/benchmark/requirements/prompt-requirements.json +++ b/gloss-v1/benchmark/requirements/prompt-requirements.json @@ -149,7 +149,7 @@ "source_type": "prompt", "authority": "primary", "path": "benchmark/prompts/variants/canonical/slide-20.md", - "sha256": "69711e77502102776aae817c7417faad5d3ae0ea8e29c823554cbc27c5b989d9" + "sha256": "ba8e51e1becddfd40e2f2a02c5161a1caa09481255fcb9ff586f6b32ff866245" } ], "independent_reviews": [], @@ -15135,8 +15135,8 @@ "provenance": { "source_id": "slide-20-prompt", "source_type": "prompt", - "line_start": 11, - "line_end": 11, + "line_start": 9, + "line_end": 9, "excerpt": "- Use the **Blank with Footer** layout with title `Gloss Synthesis`.", "excerpt_sha256": "879c1e2c7eb467566cfda47f1b0c44f6f1e75e73e865c059a25c437ddbe2a857" }, @@ -15175,8 +15175,8 @@ "provenance": { "source_id": "slide-20-prompt", "source_type": "prompt", - "line_start": 12, - "line_end": 12, + "line_start": 10, + "line_end": 10, "excerpt": "- Add one native line chart with series `Structural`, `Visual`, and `Combined` across categories `L1`, `L2`, `L3`; values are `72, 84, 96`, `68, 82, 95`, and `70, 83, 97`.", "excerpt_sha256": "c9290c0038e5213fd6ae2aec7690f705665911923945219b13544c29ce36c99b" }, @@ -15215,8 +15215,8 @@ "provenance": { "source_id": "slide-20-prompt", "source_type": "prompt", - "line_start": 13, - "line_end": 13, + "line_start": 11, + "line_end": 11, "excerpt": "- Add one native 3-row × 6-column table. Header: `Tier`, `Slides`, `Checks`, `SSIM`, `Schema`, `Status`; rows: `L1 | 5 | 70 | 0.9999 | Pass | Ready` and `L3 | 20 | 280 | 0.9999 | Pass | Ready`.", "excerpt_sha256": "002dfc07ea3b536a91fb863588727de959fe95f3c9020ca0e766d2b0df28c028" }, @@ -15255,8 +15255,8 @@ "provenance": { "source_id": "slide-20-prompt", "source_type": "prompt", - "line_start": 14, - "line_end": 14, + "line_start": 12, + "line_end": 12, "excerpt": "- Insert `hero-abstract.png` with a 20% left crop. Add Arabic `اختبار شامل` with native RTL direction and Japanese `総合テスト` in Noto Sans CJK JP.", "excerpt_sha256": "393f213352644b945cbf5a9bd8587806259012cc36eca0c162e1322a26579589" }, @@ -15295,8 +15295,8 @@ "provenance": { "source_id": "slide-20-prompt", "source_type": "prompt", - "line_start": 15, - "line_end": 15, + "line_start": 13, + "line_end": 13, "excerpt": "- Include a 45° text box labeled `ROTATED`, three overlapping semi-transparent shapes inside a two-level group, a navy-to-teal gradient, a three-item bullet list, a live slide-number field, and the inherited master footer.", "excerpt_sha256": "641b7b5911d4964ef11e26e7ce454de8d7e7a0ca3cfd51785b79aca194f6aa64" }, diff --git a/gloss-v1/benchmark/tools/build_corpus.py b/gloss-v1/benchmark/tools/build_corpus.py index b039bc8..95d6f5f 100644 --- a/gloss-v1/benchmark/tools/build_corpus.py +++ b/gloss-v1/benchmark/tools/build_corpus.py @@ -368,79 +368,6 @@ class SlideSpec: } -LEVEL3_DETAILS = { - 13: """ -## Explicit v1 constraints - -- Use the **Blank with Footer** layout and title the slide `Composite Stress` in Carlito 36pt Bold at (1.5cm, 0.5cm). -- Add one native pie chart at (1.5cm, 3cm), size 10cm × 8cm. Use six labeled values: `Structure 28`, `Text 22`, `Visual 18`, `Assets 14`, `Fields 10`, `Other 8`; show percentage labels and no legend. -- Add one native 4-row × 3-column table at (13cm, 3cm), size 9cm × 7cm. Header: `Check`, `Status`, `Weight`. Rows: `Schema | Pass | 3`, `Rendering | Pass | 2`, `Assets | Review | 1`. -- Insert `cityscape.png` at (23cm, 3cm), size 8cm × 6cm, cropping 15% from the left. Use only asset ID `cityscape` from the manifest. -- Add the Arabic annotation `الهيكل والمحتوى والعرض في اختبار واحد` in Noto Sans Arabic 14pt with native RTL direction. -- Add three dense rounded callouts labeled `NATIVE CHART`, `NATIVE TABLE`, and `APPROVED ASSET`; connect each to its target and keep all data labels visible. -""", - 14: """ -## Explicit v1 constraints - -- Use the **Two-Column** layout with title `RTL Systems Review`. -- The left column contains at least five separate Arabic paragraphs in Noto Sans Arabic 14pt with native RTL paragraph direction. Include these exact phrases: `مراجعة الأنظمة الموزعة`, `دقة العرض`, `سلامة البنية`, `The الذكاء الاصطناعي revolution`, and `الإصدار Gloss v1`. -- The right column contains five English summary paragraphs in Liberation Sans 14pt, beginning with `A structural benchmark must preserve meaning and direction.` -- Right-align the Arabic column and left-align the English column. Mirror their inner padding and keep a 1pt teal divider centered between them. -- Mixed Arabic/English runs must preserve Unicode character order and visible punctuation without converting text to outlines. -""", - 15: """ -## Explicit v1 constraints - -- Use the **Blank with Footer** layout with title `Rotation Atlas`. -- Create five text boxes labeled exactly `Baseline 0°`, `Diagonal 45°`, `Vertical 90°`, `Reverse 135°`, and `Vertical 270°`. -- Apply rotations of 0°, 45°, 90°, 135°, and 270° respectively. Each text box is 5cm × 1.5cm, uses Liberation Sans 16pt Bold, and has a matching 6cm × 2cm supporting rectangle sharing its center and rotation. -- Distribute the five pairs evenly from x=2cm through x=28cm on a common y=8cm anchor line. Keep text editable and store rotations in native OOXML transforms. -""", - 16: """ -## Explicit v1 constraints - -- Use the **Blank with Footer** layout with title `Beyond the Frame`. -- Place a 10cm diameter coral circle at (-3cm, -2cm), a 12cm × 6cm teal rectangle at (28.867cm, 6cm) so it extends 5cm beyond the right edge, and a 16cm × 3cm gold strip at (8cm, -1cm). -- Add on-canvas labels `INTENTIONAL BLEED` and `Negative coordinates are part of the composition.` -- Preserve negative and over-bound coordinates in OOXML. Do not replace the composition with a screenshot, hide content under opaque shapes, or move the objects fully on-canvas. -""", - 17: """ -## Explicit v1 constraints - -- Use the **Blank with Footer** layout with title `Nested Systems`. -- Build one outer group containing three middle groups. Each middle group contains two inner groups; each inner group contains three leaf shapes (18 leaf shapes total). -- Use native group objects at all three levels. Label the middle groups `INPUT`, `PROCESS`, and `OUTPUT`; label inner groups `A1`, `A2`, `B1`, `B2`, `C1`, and `C2`. -- Within every inner group, z-order is circle below rounded rectangle below label. The full outer group's bounding box has top-left position (3cm, 3cm) and size 27cm × 13cm. -""", - 18: """ -## Explicit v1 constraints - -- Use the **Blank with Footer** layout with title `Three Cities / 三つの都市`. -- Build three equal 9.5cm-wide editorial columns with 0.9cm gutters. Each column contains a 9.5cm × 3.5cm approved header image, a heading, body copy, and a pull quote. -- Column images, left to right: `cityscape.png`, `texture-pattern.png`, `cityscape.png`. Center the crop of each image and do not use unlisted media. -- Headings: `Systems`, `システム`, `Shared Futures`. Include the exact Japanese line `生成されたスライドは構造と意味を保持します。` -- Apply the approved `texture-pattern.png` as a tiled background only to the center column at 20% opacity. Use 0.9cm gutters consistently. -""", - 19: """ -## Explicit v1 constraints - -- Use the **Content Slide** layout with title `Design System Audit`. -- Show five samples labeled `Typography`, `Palette`, `Spacing`, `Line Weight`, and `Navigation`. Each sample must use the deck-level tokens exactly: Carlito 36pt, Liberation Sans 18pt, navy `#1B2A4A`, coral `#E8634A`, teal `#2AACB8`, and a 0.5pt teal rule. -- Create two native internal hyperlinks: `Return to Cover` targets Slide 1 and `Meet the Team` targets Slide 5. Do not use external URLs. -- Footer line, company name, accent bar, and slide number remain inherited from the master/layout; do not duplicate them as slide-local shapes. -""", - 20: """ -## Explicit v1 constraints - -- Use the **Blank with Footer** layout with title `Gloss Synthesis`. -- Add one native line chart with series `Structural`, `Visual`, and `Combined` across categories `L1`, `L2`, `L3`; values are `72, 84, 96`, `68, 82, 95`, and `70, 83, 97`. -- Add one native 3-row × 6-column table. Header: `Tier`, `Slides`, `Checks`, `SSIM`, `Schema`, `Status`; rows: `L1 | 5 | 70 | 0.9999 | Pass | Ready` and `L3 | 20 | 280 | 0.9999 | Pass | Ready`. -- Insert `hero-abstract.png` with a 20% left crop. Add Arabic `اختبار شامل` with native RTL direction and Japanese `総合テスト` in Noto Sans CJK JP. -- Include a 45° text box labeled `ROTATED`, three overlapping semi-transparent shapes inside a two-level group, a navy-to-teal gradient, a three-item bullet list, a live slide-number field, and the inherited master footer. -""", -} - - PARAPHRASE_A = { "What to build": "Required construction", "must contain": "is required to include", @@ -555,8 +482,6 @@ def canonical_sections() -> dict[int, str]: "Use the reference image only as supplementary visual guidance.\n\n" ) content = heading + intro + body - if number in LEVEL3_DETAILS: - content += "\n\n" + LEVEL3_DETAILS[number].strip() + "\n" sections[number] = content.strip() + "\n" if set(sections) != set(range(1, 21)): raise RuntimeError(f"expected slide sections 1-20, found {sorted(sections)}") diff --git a/gloss-v1/grader/README.md b/gloss-v1/grader/README.md index cd747b3..7b4d1fc 100644 --- a/gloss-v1/grader/README.md +++ b/gloss-v1/grader/README.md @@ -1,11 +1,27 @@ -# Gloss Grader +# Gloss Checker -The Gloss CLI performs quarantine checks, deterministic ECMA-376 Markup Compatibility preprocessing, Part 1 Transitional XSD validation, LibreOffice export, SSIM comparison, native object inspection, checklist evaluation, and report generation. +The friendly front door is a native-object check against the public Gloss deck: + +```bash +uv sync --extra dev --locked +uv run gloss check ../benchmark/deck/gold/gloss-v1-gold.pptx +uv run gloss check /path/to/edited.pptx +``` + +An untouched deck returns zero findings. Change the position, size, text, type, +rotation, field, chart, table, image, fill, shadow, group, or z-order of one native +object and `gloss check` returns one aggregated finding for that object. Use +`--format json` for machine-readable output or `--reference other.pptx` to compare +against another deck. + +The same CLI also contains the deeper Gloss v1 grading protocol: quarantine checks, +deterministic ECMA-376 Markup Compatibility preprocessing, Part 1 Transitional XSD +validation, LibreOffice export, SSIM comparison, native inspection, checklist +evaluation, and release-grade reports. The wheel contains the benchmark corpus and schemas. Source and Docker layouts are also discovered automatically; `--benchmark-dir` or `GLOSS_BENCHMARK_DIR` can select an explicit corpus. ```bash -uv sync --extra dev --locked uv run gloss validate ../benchmark/deck/gold/gloss-v1-gold.pptx uv run gloss grade ../submission.pptx --tier 3 \ --artifact-context ./local-artifact-context.json --format json @@ -13,7 +29,9 @@ uv run gloss grade ../submission.pptx --tier 3 \ --artifact-context ./local-artifact-context.json --format html -o report.html ``` -Output formats are `text`, `json`, and standalone `html`. Use `--artifacts DIR` to retain `diff-slide-NN.png` visual diagnostics. Incomplete verification exits with status `2` and is always reported as `eligible: false`. +Advanced `grade` output formats are `text`, `json`, and standalone `html`. Use +`--artifacts DIR` to retain `diff-slide-NN.png` visual diagnostics. Incomplete +verification exits with status `2` and is always reported as `eligible: false`. `--artifact-context` is required and must name a complete JSON serialization of `ArtifactReportContext`, including explicit `null` values. Missing or unknown fields are rejected. The diff --git a/gloss-v1/grader/gloss/cli.py b/gloss-v1/grader/gloss/cli.py index dc5f4a8..56bb1a4 100644 --- a/gloss-v1/grader/gloss/cli.py +++ b/gloss-v1/grader/gloss/cli.py @@ -1,4 +1,4 @@ -"""Gloss CLI — benchmark grading tool.""" +"""Gloss CLI — native presentation challenge and checker.""" import json from pathlib import Path @@ -15,7 +15,81 @@ @click.group() @click.version_option(version=__version__) def main() -> None: - """Gloss benchmark grader — ECMA-376 slide generation fidelity testing.""" + """Gloss — the open challenge for native AI-made presentations.""" + + +@main.command() +@click.argument( + "submission", + type=click.Path(exists=True, dir_okay=False, readable=True, path_type=Path), +) +@click.option( + "--reference", + type=click.Path(exists=True, dir_okay=False, readable=True, path_type=Path), + default=None, + help="Reference .pptx; defaults to the bundled Gloss v1 deck", +) +@click.option( + "--format", + "fmt", + type=click.Choice(["text", "json"]), + default="text", + show_default=True, +) +def check(submission: Path, reference: Path | None, fmt: str) -> None: + """Report native objects changed from the public Gloss deck.""" + import zipfile + + from lxml import etree + + from gloss.compare import compare_native_decks + from gloss.mce import MCEProfileError + from gloss.resources import BenchmarkDataError, resolve_benchmark_dir + + try: + resolved_reference = reference + if resolved_reference is None: + resolved_reference = resolve_benchmark_dir() / "deck" / "gold" / "gloss-v1-gold.pptx" + if not resolved_reference.is_file(): + raise FileNotFoundError(f"Reference deck is missing: {resolved_reference}") + changes = compare_native_decks(submission, resolved_reference) + except ( + BenchmarkDataError, + FileNotFoundError, + MCEProfileError, + OSError, + ValueError, + zipfile.BadZipFile, + etree.XMLSyntaxError, + ) as exc: + console.print(f"[red]Check could not run:[/red] {exc}") + raise click.exceptions.Exit(2) from exc + + if fmt == "json": + click.echo( + json.dumps( + { + "status": "exact" if not changes else "changed", + "changed_objects": len(changes), + "changes": [change.as_dict() for change in changes], + }, + indent=2, + ) + ) + elif not changes: + console.print("[green]Exact match.[/green] No native objects changed.") + else: + noun = "object" if len(changes) == 1 else "objects" + console.print(f"[yellow]{len(changes)} native {noun} changed:[/yellow]") + for change in changes: + location = "Deck" if change.slide_number == 0 else f"Slide {change.slide_number:02d}" + fields = ", ".join(change.changed_fields) + console.print( + f" [bold]{location}[/bold] · {change.label} · {change.change_type}: {fields}" + ) + + if changes: + raise click.exceptions.Exit(1) @main.command("build-environment-candidate") diff --git a/gloss-v1/grader/gloss/compare.py b/gloss-v1/grader/gloss/compare.py index d878154..ad204dc 100644 --- a/gloss-v1/grader/gloss/compare.py +++ b/gloss-v1/grader/gloss/compare.py @@ -1,8 +1,9 @@ -"""Stage 2: Visual comparison — SSIM-based slide comparison.""" +"""Visual and native-object comparison for presentation artifacts.""" from __future__ import annotations -from typing import TYPE_CHECKING, cast +from dataclasses import dataclass +from typing import TYPE_CHECKING, Any, cast import numpy as np from PIL import Image @@ -13,9 +14,223 @@ if TYPE_CHECKING: from pathlib import Path + from gloss.inspect_ooxml import DeckGraph, SceneObject + SSIM_THRESHOLD = 0.9999 +@dataclass(frozen=True) +class NativeObjectChange: + """One changed native presentation object, aggregated into one finding.""" + + slide_number: int + object_path: str + label: str + changed_fields: tuple[str, ...] + change_type: str = "changed" + + def as_dict(self) -> dict[str, Any]: + return { + "slide": self.slide_number, + "object_path": self.object_path, + "label": self.label, + "change_type": self.change_type, + "changed_fields": list(self.changed_fields), + } + + +def compare_native_decks(submission: Path, reference: Path) -> list[NativeObjectChange]: + """Return one finding for each native object that differs from ``reference``. + + This is the intentionally small front-door check used after editing the public + Gloss deck. It compares the native object tree rather than rendered pixels and + collapses any number of property differences on one object into one finding. + The release grader remains responsible for full visual and protocol scoring. + """ + from gloss.inspect_ooxml import extract_deck_graph + + submission_graph = extract_deck_graph(submission) + reference_graph = extract_deck_graph(reference) + return _compare_deck_graphs(submission_graph, reference_graph) + + +def _compare_deck_graphs( + submission: DeckGraph, + reference: DeckGraph, +) -> list[NativeObjectChange]: + changes: list[NativeObjectChange] = [] + if len(submission.slides) != len(reference.slides): + changes.append( + NativeObjectChange( + slide_number=0, + object_path="deck", + label="slide count", + changed_fields=("slide_count",), + ) + ) + + shared_slides = min(len(submission.slides), len(reference.slides)) + for index in range(shared_slides): + actual_slide = submission.slides[index] + expected_slide = reference.slides[index] + slide_number = expected_slide.slide_number + slide_fields = tuple( + field + for field, actual, expected in ( + ("layout", actual_slide.layout_ref, expected_slide.layout_ref), + ("layout_name", actual_slide.layout_name, expected_slide.layout_name), + ("master", actual_slide.master_name, expected_slide.master_name), + ) + if actual != expected + ) + if slide_fields: + changes.append( + NativeObjectChange( + slide_number=slide_number, + object_path="slide", + label="slide properties", + changed_fields=slide_fields, + ) + ) + + actual_objects = dict(_flatten_objects(actual_slide.objects)) + expected_objects = dict(_flatten_objects(expected_slide.objects)) + for path in sorted(actual_objects.keys() | expected_objects.keys()): + actual = actual_objects.get(path) + expected = expected_objects.get(path) + object_path = _format_object_path(path) + if actual is None and expected is not None: + changes.append( + NativeObjectChange( + slide_number=slide_number, + object_path=object_path, + label=_object_label(expected, object_path), + changed_fields=("presence",), + change_type="missing", + ) + ) + continue + if expected is None and actual is not None: + changes.append( + NativeObjectChange( + slide_number=slide_number, + object_path=object_path, + label=_object_label(actual, object_path), + changed_fields=("presence",), + change_type="unexpected", + ) + ) + continue + if actual is None or expected is None: + continue + fields = _changed_object_fields(actual, expected) + if fields: + changes.append( + NativeObjectChange( + slide_number=slide_number, + object_path=object_path, + label=_object_label(expected, object_path), + changed_fields=fields, + ) + ) + + for slide in reference.slides[shared_slides:]: + changes.append( + NativeObjectChange( + slide_number=slide.slide_number, + object_path="slide", + label="entire slide", + changed_fields=("presence",), + change_type="missing", + ) + ) + for slide in submission.slides[shared_slides:]: + changes.append( + NativeObjectChange( + slide_number=slide.slide_number, + object_path="slide", + label="entire slide", + changed_fields=("presence",), + change_type="unexpected", + ) + ) + return changes + + +def _flatten_objects( + objects: list[SceneObject], + parent: tuple[int, ...] = (), +) -> list[tuple[tuple[int, ...], SceneObject]]: + flattened: list[tuple[tuple[int, ...], SceneObject]] = [] + for index, obj in enumerate(objects, 1): + path = parent + (index,) + flattened.append((path, obj)) + flattened.extend(_flatten_objects(obj.children, path)) + return flattened + + +def _format_object_path(path: tuple[int, ...]) -> str: + return ".".join(str(value) for value in path) + + +def _object_label(obj: SceneObject, object_path: str) -> str: + object_type = "placeholder" if obj.placeholder_type else obj.obj_type + text = " ".join(run.text.strip() for run in obj.text_runs if run.text.strip()) + if text: + excerpt = text if len(text) <= 42 else f"{text[:39].rstrip()}…" + return f"{object_type} “{excerpt}”" + if obj.name: + return f"{object_type} “{obj.name}”" + return f"{object_type} {object_path}" + + +def _changed_object_fields(actual: SceneObject, expected: SceneObject) -> tuple[str, ...]: + fields: list[str] = [] + if actual.bbox[:2] != expected.bbox[:2]: + fields.append("position") + if actual.bbox[2:] != expected.bbox[2:]: + fields.append("size") + comparisons = ( + ("type", actual.obj_type, expected.obj_type), + ("name", actual.name, expected.name), + ("z_order", actual.z_index, expected.z_index), + ("rotation", actual.rotation, expected.rotation), + ("text", _text_snapshot(actual), _text_snapshot(expected)), + ( + "placeholder", + (actual.placeholder_type, actual.placeholder_idx), + (expected.placeholder_type, expected.placeholder_idx), + ), + ( + "table", + (actual.is_table, actual.table_rows, actual.table_cols), + (expected.is_table, expected.table_rows, expected.table_cols), + ), + ("chart", (actual.is_chart, actual.chart_type), (expected.is_chart, expected.chart_type)), + ("field", actual.field_type, expected.field_type), + ("fill", (actual.fill_type, actual.opacity), (expected.fill_type, expected.opacity)), + ("shadow", actual.has_shadow, expected.has_shadow), + ( + "image", + (actual.is_picture, actual.asset_hash), + (expected.is_picture, expected.asset_hash), + ), + ("group", len(actual.children), len(expected.children)), + ) + fields.extend( + field + for field, actual_value, expected_value in comparisons + if actual_value != expected_value + ) + return tuple(fields) + + +def _text_snapshot(obj: SceneObject) -> tuple[tuple[str, str, int, bool, bool], ...]: + return tuple( + (run.text, run.font_family, run.font_size, run.bold, run.italic) for run in obj.text_runs + ) + + def compare_slides( submission_dir: Path, gold_dir: Path, diff --git a/gloss-v1/grader/pyproject.toml b/gloss-v1/grader/pyproject.toml index df01303..3cb5494 100644 --- a/gloss-v1/grader/pyproject.toml +++ b/gloss-v1/grader/pyproject.toml @@ -5,7 +5,7 @@ build-backend = "hatchling.build" [project] name = "gloss" version = "1.0.0a1" -description = "Gloss benchmark grader — ECMA-376 slide generation fidelity testing" +description = "The open challenge and checker for native AI-made presentations" readme = "README.md" requires-python = ">=3.12" license = "Apache-2.0" diff --git a/gloss-v1/grader/tests/test_cli.py b/gloss-v1/grader/tests/test_cli.py index f105635..52d4fb4 100644 --- a/gloss-v1/grader/tests/test_cli.py +++ b/gloss-v1/grader/tests/test_cli.py @@ -4,6 +4,7 @@ import json from dataclasses import replace +from pathlib import Path from typing import TYPE_CHECKING, Any from click.testing import CliRunner @@ -21,8 +22,6 @@ ) if TYPE_CHECKING: - from pathlib import Path - import pytest @@ -275,3 +274,14 @@ def test_validate_exits_two_when_schema_cannot_run( assert result.exit_code == 2 assert "not performed" in result.output assert "schemas unavailable" in result.output + + +def test_check_reports_exact_gold_deck() -> None: + root = Path(__file__).resolve().parents[2] + gold = root / "benchmark" / "deck" / "gold" / "gloss-v1-gold.pptx" + + result = CliRunner().invoke(main, ["check", str(gold)]) + + assert result.exit_code == 0 + assert "Exact match" in result.output + assert "No native objects changed" in result.output diff --git a/gloss-v1/grader/tests/test_compare.py b/gloss-v1/grader/tests/test_compare.py index cacad89..5d775b0 100644 --- a/gloss-v1/grader/tests/test_compare.py +++ b/gloss-v1/grader/tests/test_compare.py @@ -2,15 +2,14 @@ from __future__ import annotations -from typing import TYPE_CHECKING +import zipfile +from pathlib import Path import numpy as np +from lxml import etree from PIL import Image -from gloss.compare import compare_slides - -if TYPE_CHECKING: - from pathlib import Path +from gloss.compare import compare_native_decks, compare_slides def _image(path: Path, color: tuple[int, int, int], size: tuple[int, int] = (16, 16)) -> None: @@ -74,3 +73,72 @@ def test_diff_uses_signed_arithmetic_before_amplification(tmp_path: Path) -> Non assert result[0].diff_path == diffs / "diff-slide-01.png" with Image.open(result[0].diff_path) as diff: assert np.array(diff).max() == 255 + + +def _move_named_native_objects( + source: Path, + destination: Path, + targets: dict[str, str], +) -> None: + drawing_namespace = "http://schemas.openxmlformats.org/drawingml/2006/main" + presentation_namespace = "http://schemas.openxmlformats.org/presentationml/2006/main" + parser = etree.XMLParser(resolve_entities=False, no_network=True, load_dtd=False) + with zipfile.ZipFile(source) as original, zipfile.ZipFile(destination, "w") as mutated: + for info in original.infolist(): + content = original.read(info.filename) + if target_text := targets.get(info.filename): + root = etree.fromstring(content, parser=parser) + shape_tree = root.find(f".//{{{presentation_namespace}}}spTree") + assert shape_tree is not None + + def shape_text(candidate: etree._Element) -> str: + values = candidate.xpath( + ".//a:t/text()", + namespaces={"a": drawing_namespace}, + ) + assert isinstance(values, list) + return "".join(str(value) for value in values) + + shape = next( + ( + candidate + for candidate in shape_tree.findall(f"{{{presentation_namespace}}}sp") + if target_text in shape_text(candidate) + ), + None, + ) + assert shape is not None + offset = shape.find( + f"{{{presentation_namespace}}}spPr/" + f"{{{drawing_namespace}}}xfrm/{{{drawing_namespace}}}off" + ) + assert offset is not None + offset.set("x", str(int(offset.get("x", "0")) + 914_400)) + content = etree.tostring(root, xml_declaration=True, encoding="UTF-8") + mutated.writestr(info, content) + + +def test_native_check_gold_is_exact_and_two_moves_are_two_findings(tmp_path: Path) -> None: + root = Path(__file__).resolve().parents[2] + gold = root / "benchmark" / "deck" / "gold" / "gloss-v1-gold.pptx" + assert compare_native_decks(gold, gold) == [] + + edited = tmp_path / "edited.pptx" + _move_named_native_objects( + gold, + edited, + { + "ppt/slides/slide2.xml": "Agenda", + "ppt/slides/slide12.xml": "Document Fields", + }, + ) + + changes = compare_native_decks(edited, gold) + + assert len(changes) == 2 + assert [change.slide_number for change in changes] == [2, 12] + assert [change.label for change in changes] == [ + "placeholder “Agenda”", + "placeholder “Document Fields”", + ] + assert all(change.changed_fields == ("position",) for change in changes) diff --git a/gloss-v1/service/gloss_service/main.py b/gloss-v1/service/gloss_service/main.py index e67248e..6ccbe2e 100644 --- a/gloss-v1/service/gloss_service/main.py +++ b/gloss-v1/service/gloss_service/main.py @@ -205,8 +205,9 @@ async def lifespan(_app: FastAPI) -> AsyncIterator[None]: title="Gloss hosted service", version="1.0.0", description=( - "Gloss — Generative Layout & OOXML Scoring System. Controlled grading and " - "public grading-verified leaderboard." + "Supporting measurement service for Gloss — the Generative Layout & " + "Object Structure Standard presentation challenge. Provides controlled " + "grading and a public grading-verified leaderboard." ), docs_url="/v1/docs", redoc_url="/v1/redoc", diff --git a/gloss-v1/service/gloss_service/static/index.html b/gloss-v1/service/gloss_service/static/index.html index ee8a60b..a2ad55e 100644 --- a/gloss-v1/service/gloss_service/static/index.html +++ b/gloss-v1/service/gloss_service/static/index.html @@ -1,10 +1,10 @@ - +
- -Generative Layout & OOXML Scoring System
+Supporting measurement for the Gloss challenge
Twenty adversarial slides. Native tables, live charts, multilingual type, masters, groups, fields, and unforgiving geometry. Every published result is labeled exactly: grading-verified artifact score; generation-attested.
Gloss measurement stack
+Hosted checker stack
No screenshots. No traces. One final .pptx.
Organizations receive an API key from a benchmark maintainer, register immutable model and revision keys, then precommit a campaign. Each upload carries only its campaign_id plus generation attestation; tier, variant, benchmark, and cohort cannot be overridden.
curl https://your-gloss-host/v1/submissions \
-H "Authorization: Bearer $GLOSS_API_KEY" \
-F 'metadata=@submission.json;type=application/json' \
-F 'file=@model-output.pptx'
Open interactive API docs
- Open technical preview
-Gloss is a deliberately difficult presentation deck for AI to render and build as a real, editable PowerPoint. Its open measurement tools show whether the result looks right and is built right.
+An open ACID test for AI-made presentations
+Gloss is a deliberately complicated 20-slide presentation that AI struggles to recreate as a real, editable artifact. The deck is the challenge. The checker makes every native miss easy to see.
-Generative Layout & OOXML Scoring System · Apache-2.0
+Remember Acid1, Acid2, and Acid3? Gloss is ACID for presentation decks made by AI.