refactor(runner): abortOnFailure is input-only + scenarios cleanup + input-schema flatten

.actor/input_schema.json

@@ -24,7 +24,7 @@
             "type": "string",
             "description": "Markdown with YAML frontmatter + ## Test / ## Checkpoint sections separated by ---. Checkpoint supports: contains:, regex:, json-schema:, script: prefixes for deterministic checks, and plain text for LLM judge. Multiple checks in one checkpoint all must pass.",
             "editor": "textarea",
-            "prefill": "---\nname: example-eval\ndescription: Simple eval demonstrating checkpoint types\nabortOnFailure: false\n---\n\n## Test\nWhat is the largest planet in our solar system? Answer in one sentence.\n\n## Checkpoint\ncontains: Jupiter\nThe answer should be scientifically accurate.\n"
+            "prefill": "---\nname: example-eval\ndescription: Simple eval demonstrating checkpoint types\n---\n\n## Test\nWhat is the largest planet in our solar system? Answer in one sentence.\n\n## Checkpoint\ncontains: Jupiter\nThe answer should be scientifically accurate.\n"
         },
         "systemPrompt": {
             "title": "System Prompt",

CLAUDE.md

@@ -51,7 +51,6 @@ Markdown s YAML frontmatter. Validace při startu (throwne `ScenarioParseError`
 ---
 name: scenario-name (required)
 description: "Description"
-abortOnFailure: false
 ---
 
 ## Test
@@ -67,6 +66,8 @@ Plain text = LLM judge prompt
 (Optional) Follow-up question
 ```
 
+Abort semantics: nastav top-level runner input `abortOnFailure` — jakýkoli `abortOnFailure:` klíč uvnitř scénářového YAML frontmatter je tiše zahozen (input je jediný zdroj pravdy).
+
 Multi-test: bloky oddělené `---`.
 
 ## Checkpoint systém (`shared/src/judge.ts`)

actors/runner/.actor/input_schema.json

@@ -24,7 +24,7 @@
             "type": "string",
             "description": "Markdown with YAML frontmatter + ## Test / ## Checkpoint sections separated by ---. Checkpoint supports: contains:, regex:, json-schema:, script: prefixes for deterministic checks, and plain text for LLM judge. Multiple checks in one checkpoint all must pass.",
             "editor": "textarea",
-            "prefill": "---\nname: example-eval\ndescription: Simple eval demonstrating checkpoint types\nabortOnFailure: false\n---\n\n## Test\nWhat is the largest planet in our solar system? Answer in one sentence.\n\n## Checkpoint\ncontains: Jupiter\nThe answer should be scientifically accurate.\n"
+            "prefill": "---\nname: example-eval\ndescription: Simple eval demonstrating checkpoint types\n---\n\n## Test\nWhat is the largest planet in our solar system? Answer in one sentence.\n\n## Checkpoint\ncontains: Jupiter\nThe answer should be scientifically accurate.\n"
         },
         "systemPrompt": {
             "title": "System Prompt",
@@ -61,9 +61,9 @@
         "abortOnFailure": {
             "title": "Abort on test failure",
             "type": "boolean",
-            "description": "If true, the eval run aborts on the first failing test in the scenario. If false, subsequent tests still run independently. When set explicitly, overrides the abortOnFailure flag in the scenario's YAML frontmatter; when left unset, the scenario YAML value is used (which itself defaults to false if absent from the frontmatter).",
-            "nullable": true,
-            "sectionCaption": "Behavior"
+            "description": "If true, the eval run aborts on the first failing test in the scenario. If false, subsequent tests still run independently. This is the single source of truth — any `abortOnFailure:` key in the scenario's YAML frontmatter is ignored.",
+            "default": false,
+            "nullable": true
         },
         "envVariables": {
             "title": "Environment Variables",
@@ -77,8 +77,7 @@
             "title": "Pre-authenticate Apify CLI",
             "type": "boolean",
             "description": "If true (default), the runner populates ~/.apify/auth.json with APIFY_TOKEN via 'apify login --token' BEFORE the agent starts. This mirrors what every real Apify developer has after running 'apify login' once on their machine — 'apify push' from the agent will work without the agent having to discover auth flow. Set to false for raw-unauthed measurement (the agent has to figure out auth on its own — surfaces F11 in the eval pack's FINDINGS.md).",
-            "default": true,
-            "sectionCaption": "Authentication"
+            "default": true
         },
         "initPreset": {
             "title": "Tool Setup Preset",

actors/runner/README.md

@@ -27,7 +27,6 @@ A scenario is a Markdown file with one or more tests. Each test has a prompt and
 ---
 name: my-test
 description: What this scenario tests
-abortOnFailure: false
 ---
 
 ## Test
@@ -54,7 +53,7 @@ The answer should name a real, currently existing repository.
 - Tests are separated by `---`
 - Each test needs `## Test` (the prompt) and `## Checkpoint` (evaluation criteria)
 - Optional: `## Monitor` — a follow-up question about the agent's work
-- YAML frontmatter: `name` (required), `description`, `abortOnFailure` (stop on first failure). The top-level `abortOnFailure` input on the Actor overrides this when set explicitly — useful for flipping abort behaviour per-run without editing the scenario markdown.
+- YAML frontmatter: `name` (required), `description`. Whether the scenario aborts on the first failing test is controlled by the top-level `abortOnFailure` input on the Actor (default `false`); any `abortOnFailure:` key in the scenario YAML is ignored (accepted for backward compatibility, but not read).
 - **Retries (experimental):** `maxRetries` input re-runs a failed test. ⚠️ Retries create a fresh workspace but may produce inconsistent results due to agent caching, auth state, and non-determinism. Recommended: `maxRetries: 0` (default) — one run per actor call
 
 ## Checkpoint syntax
@@ -368,7 +367,7 @@ Use `maxBudgetUsd` to cap spending. The budget is a soft limit — checked betwe
 
 - Start with simple `contains:` checks to verify basic functionality, then add LLM judge for quality
 - Use `maxTurns: 3` for simple questions, `maxTurns: 10+` for complex multi-step tasks
-- Set `abortOnFailure: true` when tests build on each other (test 2 depends on test 1). Either in the scenario YAML frontmatter, or via the top-level `abortOnFailure` input — the input wins when both are set.
+- Set `abortOnFailure: true` on the top-level runner input when tests build on each other (test 2 depends on test 1). This is input-only — any `abortOnFailure:` key in the scenario YAML frontmatter is ignored.
 - Use `script:` checkpoints to verify side effects (files created, API state changed)
 - The Custom Init Script can install tools, download validators, or set up test fixtures
 - If the agent runs an Apify Actor, its dataset is automatically downloaded into `eval-datasets/<datasetId>.json` in the workspace — your script checks and the LLM judge can read it directly

actors/runner/src/main.ts

@@ -136,15 +136,10 @@ const agent = input.agent ?? 'claude-code';
 const maxRetries = input.maxRetries ?? 0;
 const maxTurns = input.maxTurns ?? 10;
 
-// Resolve abortOnFailure with precedence: input (top-level) > scenario YAML frontmatter.
-// parseScenario already fills meta.abortOnFailure with `false` when the YAML omits it,
-// so meta.abortOnFailure is always a concrete boolean. Use `??` (not `||`) so an explicit
-// `false` from input still overrides `true` from the YAML.
-const abortOnFailure = input.abortOnFailure ?? meta.abortOnFailure;
-// Treat null and undefined the same for source labeling — a caller that
-// sends `{"abortOnFailure": null}` explicitly meant "defer to scenario",
-// same as omitting the field. `!= null` matches both `null` and `undefined`.
-const abortOnFailureSource = input.abortOnFailure != null ? 'input' : 'scenario';
+// abortOnFailure is now driven exclusively by the runner input.
+// Any `abortOnFailure:` key in a scenario's YAML frontmatter is ignored.
+// Default to false when not explicitly set on the input.
+const abortOnFailure = input.abortOnFailure ?? false;
 
 const preset = (input.initPreset ?? 'none') as PresetName;
 
@@ -157,10 +152,7 @@ const scenarioSpan = startScenarioSpan(tracer, {
     initPreset: preset,
 });
 
-log.info(`Scenario "${meta.name}": ${tests.length} test(s), abortOnFailure=${abortOnFailure} (source: ${abortOnFailureSource})`);
-if (input.abortOnFailure != null && input.abortOnFailure !== meta.abortOnFailure) {
-    log.info(`Override: input.abortOnFailure=${input.abortOnFailure} takes precedence over scenario YAML value=${meta.abortOnFailure}`);
-}
+log.info(`Scenario "${meta.name}": ${tests.length} test(s), abortOnFailure=${abortOnFailure}`);
 if (parseWarnings) {
     for (const w of parseWarnings) log.warning(`[parse] ${w}`);
 }

actors/runner/test/e2e/run-e2e.ts

@@ -269,26 +269,6 @@ const tests: TestDef[] = [
             };
         },
     },
-    {
-        name: 'us5-abort-on-failure',
-        scenarioFile: 'us5-abort-on-failure.md',
-        input: {
-            scenario: readScenario('us5-abort-on-failure.md'),
-            maxTurns: 5,
-            maxBudgetUsd: 0.50,
-        },
-        checks: (results) => {
-            if (results.length !== 1) return {
-                pass: false,
-                details: `Expected 1 result (step 2 should NOT run), got ${results.length}`,
-            };
-            const v = (results[0] as any).overallVerdict;
-            return {
-                pass: v === 'fail',
-                details: `Step 1 failed (${v}), step 2 was correctly skipped (only 1 result in dataset)`,
-            };
-        },
-    },
     {
         name: 'us5-dependent-steps',
         scenarioFile: 'us5-dependent-steps.md',

docs/04-implementation-log.md

@@ -119,7 +119,6 @@
 - `us1-agent-fails`: judge správně detekuje selhání agenta (neexistující soubor) — PASS
 - `us1-partial-answer`: nuanced eval (5 zemí + math 17×23) — PASS
 - `us1-borderline-judge`: hraniční checkpointy (closure, tomato paradox) — PASS, confidence 0.98-0.99
-- `us5-abort-on-failure`: krok 1 fail → krok 2 přeskočen (1 result v datasetu) — PASS
 - `us5-dependent-steps`: 3-step řetěz (random → reverse → concat) — PASS
 - `us6-multi-secret`: 2 secrets maskované i v tool args + error messages — PASS
 - `us7-actual-abort`: budget $0.01 → error, run zastaveno — PASS

docs/reference.html

@@ -498,7 +498,7 @@ <h3>Struktura</h3>
 <pre><code><span class="cmt">---</span>
 <span class="prop">name</span>: scenario-id        <span class="cmt"># required</span>
 <span class="prop">description</span>: <span class="str">"Co testuje"</span>
-<span class="prop">abortOnFailure</span>: <span class="kw">false</span>   <span class="cmt"># stop on first fail</span>
+<span class="cmt"># abortOnFailure is input-only; scenario YAML is ignored</span>
 <span class="cmt">---</span>
 
 <span class="kw">## Test</span>
@@ -526,7 +526,6 @@ <h3>YAML frontmatter</h3>
     <tbody>
         <tr><td><code>name</code> <span class="badge badge-red">required</span></td><td>string</td><td>&mdash;</td><td>Identifikátor scénáře</td></tr>
         <tr><td><code>description</code></td><td>string</td><td><code>""</code></td><td>Co scénář testuje</td></tr>
-        <tr><td><code>abortOnFailure</code></td><td>boolean</td><td><code>false</code></td><td>Zastaví scénář po prvním failu</td></tr>
     </tbody>
 </table>
 
@@ -967,9 +966,8 @@ <h2 id="examples">Example scen&aacute;&rcaron;e</h2>
         <tr><td><code>us1-complex-tool-use</code></td><td>2</td><td><span class="badge badge-green">contains</span> <span class="badge badge-purple">llm</span></td><td>Multi-tool operace</td></tr>
         <tr><td><code>us1-agent-fails</code></td><td>1</td><td><span class="badge badge-purple">llm</span></td><td>Agent nemůže splnit úkol &mdash; judge musí detekovat</td></tr>
         <tr><td><code>us1-partial-answer</code></td><td>2</td><td><span class="badge badge-purple">llm</span></td><td>Částečně správná odpověď</td></tr>
-        <tr><td><code>us5-multi-step</code></td><td>2</td><td><span class="badge badge-green">contains</span></td><td>Multi-step s abortOnFailure</td></tr>
+        <tr><td><code>us5-multi-step</code></td><td>2</td><td><span class="badge badge-green">contains</span></td><td>Multi-step scénář s 2 nezávislými testy</td></tr>
         <tr><td><code>us5-dependent-steps</code></td><td>3</td><td><span class="badge badge-green">regex</span> <span class="badge badge-purple">llm</span></td><td>Závislé kroky (soubory z předchozího)</td></tr>
-        <tr><td><code>us5-abort-on-failure</code></td><td>2</td><td><span class="badge badge-purple">llm</span> <span class="badge badge-green">contains</span></td><td>Test 1 failne, test 2 se nesmí spustit</td></tr>
         <tr><td><code>us6-env-vars</code></td><td>1</td><td><span class="badge badge-green">contains</span></td><td>Injection + maskování env vars</td></tr>
         <tr><td><code>us6-multi-secret</code></td><td>1</td><td><span class="badge badge-purple">llm</span></td><td>Více secrets + error messages</td></tr>
         <tr><td><code>us7-budget-abort</code></td><td>1</td><td><span class="badge badge-purple">llm</span></td><td>Budget abort s nízkým limitem</td></tr>
@@ -1070,7 +1068,6 @@ <h2 id="types">TypeScript typy</h2>
 <pre><code><span class="kw">interface</span> <span class="type">ScenarioMeta</span> {
     name: <span class="type">string</span>;
     description: <span class="type">string</span>;
-    abortOnFailure: <span class="type">boolean</span>;
 }
 
 <span class="kw">interface</span> <span class="type">TestCase</span> {

scenarios/actor-dev/cheerio-scraper.md

@@ -1,7 +1,6 @@
 ---
 name: actor-dev-cheerio-scraper
 description: Agent creates a CheerioCrawler-based Apify Actor that scrapes product data
-abortOnFailure: true
 expectedTools:
   required: [Bash, Write]
   optional: [Read, Edit, Glob, Grep]

scenarios/ai-nondeterministic.md

@@ -1,7 +1,6 @@
 ---
 name: ai-nondeterministic
 description: "Non-deterministic AI test: tasks where agent can take different paths"
-abortOnFailure: false
 ---
 
 ## Test

scenarios/checkpoint-syntax-demo.md

@@ -1,7 +1,6 @@
 ---
 name: checkpoint-syntax-demo
 description: Demonstrates every checkpoint syntax variant — use as a reference
-abortOnFailure: false
 ---
 
 ## Test

scenarios/jq-tool-checks.md

@@ -1,7 +1,6 @@
 ---
 name: jq-tool-checks-demo
 description: Demonstrates jq checkpoint type for validating agent tool usage and trajectory
-abortOnFailure: false
 ---
 
 ## Test

scenarios/mcpc-preset-test.md

@@ -1,7 +1,6 @@
 ---
 name: mcpc-preset-test
 description: Test mcpc preset with real MCP server
-abortOnFailure: false
 ---
 
 ## Test

scenarios/multi-check-demo.md

@@ -1,7 +1,6 @@
 ---
 name: multi-check-demo
 description: Demonstrates all checkpoint types in one scenario
-abortOnFailure: false
 ---
 
 ## Test

scenarios/script-validation.md

@@ -1,7 +1,6 @@
 ---
 name: script-validation
 description: Tests script checkpoint with downloaded validator
-abortOnFailure: false
 ---
 
 ## Test

scenarios/security-isolation.md

@@ -1,7 +1,6 @@
 ---
 name: security-isolation
 description: Verify agent cannot modify runner files and works only in workspace
-abortOnFailure: false
 ---
 
 ## Test

scenarios/skill-injection.md

@@ -1,7 +1,6 @@
 ---
 name: skill-injection-test
 description: Test that init script can inject CLAUDE.md rules into workspace and agent follows them
-abortOnFailure: true
 ---
 
 ## Test

scenarios/smoke-test.md

@@ -1,7 +1,6 @@
 ---
 name: smoke-test
 description: Basic smoke test — simple question with checkpoint validation
-abortOnFailure: true
 ---
 
 ## Test

scenarios/trajectory-test.md

@@ -1,7 +1,6 @@
 ---
 name: trajectory-test
 description: Tests tool usage trajectory capture - agent must use multiple tools
-abortOnFailure: true
 ---
 
 ## Test

scenarios/us1-agent-fails.md

@@ -1,7 +1,6 @@
 ---
 name: us1-agent-fails
 description: "US1 deep: Agent cannot fulfill the task — judge should detect failure"
-abortOnFailure: false
 ---
 
 ## Test

scenarios/us1-ai-judge.md

@@ -1,7 +1,6 @@
 ---
 name: us1-ai-judge
 description: "US1 AI test: LLM judge evaluates nuanced answer"
-abortOnFailure: false
 ---
 
 ## Test

scenarios/us1-borderline-judge.md

@@ -1,7 +1,6 @@
 ---
 name: us1-borderline-judge
 description: "US1 deep: Borderline checkpoints that test judge accuracy"
-abortOnFailure: false
 ---
 
 ## Test

scenarios/us1-complex-tool-use.md

@@ -1,7 +1,6 @@
 ---
 name: us1-complex-tool-use
 description: "US1 deep: Agent must use multiple tools to solve a task"
-abortOnFailure: false
 ---
 
 ## Test

scenarios/us1-partial-answer.md

@@ -1,7 +1,6 @@
 ---
 name: us1-partial-answer
 description: "US1 deep: Agent gives partially correct answer — judge must detect"
-abortOnFailure: false
 ---
 
 ## Test

scenarios/us5-dependent-steps.md

@@ -1,7 +1,6 @@
 ---
 name: us5-dependent-steps
 description: "US5 deep: Steps that depend on each other"
-abortOnFailure: true
 ---
 
 ## Test

scenarios/us5-multi-step.md

@@ -1,7 +1,6 @@
 ---
 name: us5-multi-step
-description: "US5: Multi-step scenario with 2 tests and abortOnFailure"
-abortOnFailure: true
+description: "US5: Multi-step scenario with 2 tests"
 ---
 
 ## Test

scenarios/us6-env-vars.md

@@ -1,7 +1,6 @@
 ---
 name: us6-env-vars
 description: "US6: Verify env var injection and masking"
-abortOnFailure: false
 ---
 
 ## Test

scenarios/us6-multi-secret.md

@@ -1,7 +1,6 @@
 ---
 name: us6-multi-secret
 description: "US6 deep: Multiple secrets, tool call args, error messages"
-abortOnFailure: false
 ---
 
 ## Test

scenarios/us7-actual-abort.md

@@ -1,7 +1,6 @@
 ---
 name: us7-actual-abort
 description: "US7 deep: Budget so low that run MUST be aborted mid-execution"
-abortOnFailure: false
 ---
 
 ## Test

scenarios/us7-budget-abort.md

@@ -1,7 +1,6 @@
 ---
 name: us7-budget-abort
 description: "US7: Verify budget abort with very low limit"
-abortOnFailure: false
 ---
 
 ## Test

shared/src/__tests__/integration-scenario.test.ts

@@ -89,7 +89,6 @@ describe('integration: trajectory-test scenario', () => {
 
     it('parses correctly', () => {
         expect(meta.name).toBe('trajectory-test');
-        expect(meta.abortOnFailure).toBe(true);
         expect(tests).toHaveLength(2);
     });