Enable native BCAL multi-turn evaluation

.github/workflows/bcal-evaluation.yml

@@ -15,6 +15,7 @@ on:
         options:
           - gold
           - challenge
+          - multiturn
       bcal-version-mode:
         description: "Install the latest prerelease or pin an exact BCal NuGet version"
         required: false
@@ -194,7 +195,19 @@ jobs:
         if: always()
         with:
           name: evaluation-results-${{ github.run_id }}-${{ matrix.entry }}
-          path: ${{ env.EVALUATION_RESULTS_DIR }}/**/*.jsonl
+          path: |
+            ${{ env.EVALUATION_RESULTS_DIR }}/**/*.jsonl
+            !${{ env.EVALUATION_RESULTS_DIR }}/**/bcal-scenario/**
+          if-no-files-found: error
+          retention-days: ${{ inputs.test-run && 1 || 30 }}
+
+      - name: Upload BCal scenario execution metadata
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+        if: ${{ always() && !cancelled() }}
+        with:
+          name: bcal-scenario-${{ github.run_id }}-${{ matrix.entry }}
+          path: ${{ env.EVALUATION_RESULTS_DIR }}/**/bcal-scenario/**
+          if-no-files-found: ignore
           retention-days: ${{ inputs.test-run && 1 || 30 }}
 
   summarize-results:

.github/workflows/get-entries.yml

@@ -21,7 +21,7 @@ on:
         type: string
         default: "bug-fix"
       dataset:
-        description: Optional NL2AL dataset selector (gold or challenge)
+        description: Optional NL2AL dataset selector (gold, challenge, or multiturn)
         required: false
         type: string
         default: ""

.github/workflows/summarize-results.yml

@@ -25,7 +25,7 @@ on:
         required: true
         type: string
       dataset:
-        description: "Optional NL2AL dataset selector (gold or challenge)"
+        description: "Optional NL2AL dataset selector (gold, challenge, or multiturn)"
         required: false
         type: string
         default: ""

docs/bcal-datasets.md

@@ -1,13 +1,14 @@
 # BCAL dataset panels
 
-BC-Bench exposes two reviewed single-turn NL2AL panels for BCAL evaluation:
+BC-Bench exposes three reviewed NL2AL panels for BCAL evaluation:
 
-| Selection | Entries | Purpose |
-|---|---:|---|
-| `gold` | 110 | Default benchmark panel |
-| `challenge` | 65 | Harder held-out panel |
+| Selection | Entries | Input | Scoring scope |
+|---|---:|---|---|
+| `gold` | 110 | One user request | Generated AL/resources |
+| `challenge` | 65 | One user request | Generated AL/resources |
+| `multiturn` | 7 | Ordered user turns in one BCAL session | Final cumulative artifact |
 
-The **Evaluation with bcal** workflow defaults to `gold` for both scheduled and manual runs. Manual runs can select either panel from the `dataset` input. The selected panel is included in the workflow run name, concurrency group, bc-eval run name, and bc-eval tags.
+The **Evaluation with bcal** workflow defaults to `gold` for both scheduled and manual runs. Manual runs can select any panel from the `dataset` input. The selected panel is included in the workflow run name, concurrency group, bc-eval run name, and bc-eval tags.
 
 ## BCAL package version
 
@@ -26,30 +27,45 @@ uv run bcbench evaluate bcal <instance-id> --dataset gold --bcal-version 18.0.38
 ```powershell
 uv run bcbench dataset list --category nl2al --dataset gold
 uv run bcbench dataset list --category nl2al --dataset challenge
-uv run bcbench dataset view <instance-id> --category nl2al --dataset challenge
-uv run bcbench dataset version <instance-id> --category nl2al --dataset challenge
+uv run bcbench dataset list --category nl2al --dataset multiturn
+uv run bcbench dataset view <instance-id> --category nl2al --dataset multiturn
+uv run bcbench dataset version <instance-id> --category nl2al --dataset multiturn
 
 # Generation only
-uv run bcbench run bcal <instance-id> --dataset challenge
+uv run bcbench run bcal <instance-id> --dataset multiturn
 
 # Generation plus raw evaluation result
-uv run bcbench evaluate bcal <instance-id> --dataset challenge
+uv run bcbench evaluate bcal <instance-id> --dataset multiturn
 
 # Validate and summarize a completed run
-uv run bcbench result summarize --run-id <run-id> --category nl2al --dataset challenge
+uv run bcbench result summarize --run-id <run-id> --category nl2al --dataset multiturn
 ```
 
-`dataset version` writes both the Business Central version and the exact selected JSONL path when `--github-output` is used. The workflow passes that path to symbol resolution, so a challenge entry is never resolved against the gold file.
+`run bcal` and `evaluate bcal` isolate generated source under `<repo-path>\<instance-id>\workspace`, so one entry does not reset another entry's workspace or its diagnostics.
+
+`dataset version` writes both the Business Central version and the exact selected JSONL path when `--github-output` is used. The workflow passes that path to symbol resolution, so an entry is never resolved against a different panel.
+
+## Multi-turn execution and evidence boundary
+
+A multi-turn entry contains authored user messages and per-customization artifact checkpoints. BCAL executes all turns in one native `--scenario` / `--result` invocation, preserving one session and workspace. The runner never flattens the conversation into one prompt or starts a process per turn.
+
+Before invoking the model, the runner probes `bcal --help` and requires both native options. Each scripted customization step must succeed, report successful compilation, and appear in the authored order. The final result must also report consistent totals, no clarification interactions, a successful export, and a successful session archive.
+
+The native protocol exports the final workspace. Evaluation therefore scores the cumulative artifact checklist from the final customization checkpoint after all scripted turns complete. Earlier requirements that were explicitly withdrawn are not required again. This does not claim evidence for intermediate filesystem snapshots, deployed behavior, UI interaction, or business-record changes.
+
+The seven selectable conversations contain customization turns only. Seven additional conversations remain in `dataset\nl2al_multiturn_pending.jsonl` because they require reviewed active-page context or explicit interaction mappings for plan, inspect, or clarification turns. The runner rejects these mixed conversations rather than fabricating answers.
+
+Scenario input, native result data, session archives, and stdout/stderr are stored outside the generated AL workspace in a unique directory per invocation. GitHub Actions uploads these `bcal-scenario` diagnostics separately from result JSONL artifacts, so stale scenario output cannot mask a failed rerun and diagnostic JSON is not scored.
 
 ## Identity and scoring provenance
 
 Every named-panel raw result records:
 
-- `dataset`: `gold` or `challenge`
+- `dataset`: `gold`, `challenge`, or `multiturn`
 - `dataset_version`: the reviewed dataset revision from `dataset/nl2al_manifest.json`
 - `dataset_sha256`: the exact SHA-256 of the selected JSONL at evaluation time
 
-Summarization loads prompts and expected assertions from that same panel. It refuses to export results when the requested panel differs from the recorded panel, a named-panel result lacks identity, the JSONL hash changed, the recorded dataset version is wrong, or the result ID is absent from the selected panel.
+Summarization loads prompts, ordered turns, and expected assertions from that same panel. It refuses to export results when the requested panel differs from the recorded panel, a named-panel result lacks identity, the JSONL hash changed, the recorded dataset version is wrong, the result ID is absent from the selected panel, or the selected panel contains the wrong turn structure.
 
 Older results without panel identity keep their legacy behavior when summarization does not name a panel. They are not automatically labeled as gold.
 
@@ -58,12 +74,12 @@ Each named-panel bc-eval row includes:
 | Metadata | Value |
 |---|---|
 | `dataset`, `dataset_version`, `dataset_sha256` | Exact panel identity |
-| `dataset_mode` | `single_turn` |
-| `dataset_turn_count` | `1` |
-| `evaluation_scope` | `single_turn` |
+| `dataset_mode` | `single_turn` or `multiturn` |
+| `dataset_turn_count` | `1` or the authored turn count |
+| `evaluation_scope` | `single_turn` or `final_artifact` |
 | `area`, `family`, `scenario_tier` | Entry classification |
 
-The row tags include the dataset, reviewed version, single-turn mode, evaluation scope, and available area/family/tier classifications.
+For multi-turn rows, the exported task input preserves the authored turn order and intents, while the expected output is the final customization checkpoint. Row tags include the dataset, reviewed version, dataset mode, evaluation scope, and available area/family/tier classifications.
 
 ## Complete-panel safety
 

src/bcbench/agent/bcal/agent.py

@@ -1,6 +1,7 @@
 """BCal agent for NL2AL evaluation — generates AL code from natural language via bcal CLI."""
 
 import logging
+import re
 import shutil
 import subprocess
 import time
@@ -9,6 +10,7 @@
 from bcbench_core.artifacts import ALPACKAGES_DIRNAME
 from pydantic import BaseModel, ConfigDict, field_validator
 
+from bcbench.agent.bcal.scenario import BCalScenarioError, inspect_scenario_result, prepare_scenario
 from bcbench.config import get_config
 from bcbench.dataset import NL2ALEntry
 from bcbench.exceptions import AgentError, AgentTimeoutError
@@ -18,6 +20,7 @@
 _config = get_config()
 
 _BCAL_TOOL = "bcal"
+_BCAL_HELP_TIMEOUT = 30
 
 
 class BCalBackendConfig(BaseModel):
@@ -58,6 +61,28 @@ def _process_output(output: str | bytes | None) -> str:
     return (output or "").strip()
 
 
+def _require_scenario_support(executable: str) -> None:
+    try:
+        help_text = subprocess.check_output(
+            [executable, "--help"],
+            stderr=subprocess.STDOUT,
+            stdin=subprocess.DEVNULL,
+            timeout=_BCAL_HELP_TIMEOUT,
+            text=True,
+            encoding="utf-8",
+            errors="replace",
+        )
+    except (subprocess.CalledProcessError, subprocess.TimeoutExpired, OSError) as exc:
+        raise AgentError(f"Cannot verify native BCAL scenario support: {executable} --help failed: {exc}. No model was invoked.") from exc
+
+    missing = [option for option in ("--scenario", "--result") if not re.search(rf"(?<![\w-]){re.escape(option)}(?![\w-])", help_text)]
+    if missing:
+        raise AgentError(
+            f"Installed bcal CLI does not advertise native multi-turn scenario support: --help is missing {', '.join(missing)}. "
+            "Select a bcal package that advertises both --scenario and --result. No model was invoked."
+        )
+
+
 def _trim_prompt_echo(stdout: str, query: str) -> str:
     compact_query = "".join(query.split())
     if not compact_query:
@@ -119,6 +144,7 @@ def run_bcal_agent(
     entry: NL2ALEntry,
     repo_path: Path,
     backend_config: BCalBackendConfig,
+    result_dir: Path | None = None,
 ) -> tuple[AgentMetrics | None, ExperimentConfiguration]:
     logger.info(f"Running bcal CLI on: {entry.instance_id}")
 
@@ -129,11 +155,14 @@ def run_bcal_agent(
         raise AgentError(f"Package cache not found at: {package_cache_path}. Run the setup step first.")
 
     export_folder = repo_path / project_name / _config.file_patterns.nl2al_export_subdir
-    cmd_args = _bcal_cmd_args(entry, entry.get_task(), package_cache_path, export_folder, backend_config)
+    if entry.turns:
+        return _run_bcal_scenario(entry, repo_path, package_cache_path, export_folder, backend_config, result_dir)
+
+    cmd_args = _bcal_cmd_args(entry, entry.nl_prompt, package_cache_path, export_folder, backend_config)
 
     logger.info(f"Export folder: {export_folder}")
     logger.debug(f"Package cache path: {package_cache_path}")
-    logger.debug(f"Using prompt:\n{entry.get_task()}")
+    logger.debug(f"Using prompt:\n{entry.nl_prompt}")
     logger.debug(f"bcal CLI command: {cmd_args}")
 
     try:
@@ -159,6 +188,77 @@ def run_bcal_agent(
         raise
 
 
+def _run_bcal_scenario(
+    entry: NL2ALEntry,
+    repo_path: Path,
+    package_cache_path: Path,
+    export_folder: Path,
+    backend_config: BCalBackendConfig,
+    result_dir: Path | None,
+) -> tuple[AgentMetrics, ExperimentConfiguration]:
+    if any(turn.intent != "customize" for turn in entry.turns):
+        raise AgentError("This conversation requires explicit interaction/context mappings and is not supported by the native BCAL runner")
+    executable = _resolve_bcal_executable()
+    _require_scenario_support(executable)
+    cmd_args = [executable, f"--packagecachepath={package_cache_path}", *backend_config.cli_args(), f"--exportfolder={export_folder}"]
+    scenario_path, result_path = prepare_scenario(entry, repo_path, result_dir)
+    cmd_args.extend((f"--scenario={scenario_path}", f"--result={result_path}"))
+    logger.info(f"Scenario artifacts: {scenario_path.parent}")
+    logger.info(f"Export folder: {export_folder}")
+
+    process_error: subprocess.CalledProcessError | subprocess.TimeoutExpired | OSError | None = None
+    stdout = ""
+    stderr = ""
+    start = time.monotonic()
+    try:
+        result = subprocess.run(
+            cmd_args,
+            timeout=_config.timeout.bcal_execution,
+            stdin=subprocess.DEVNULL,
+            capture_output=True,
+            text=True,
+            encoding="utf-8",
+            errors="replace",
+            check=True,
+        )
+        stdout = _process_output(result.stdout)
+        stderr = _process_output(result.stderr)
+    except (subprocess.CalledProcessError, subprocess.TimeoutExpired) as exc:
+        process_error = exc
+        stdout = _process_output(exc.stdout)
+        stderr = _process_output(exc.stderr)
+    except OSError as exc:
+        process_error = exc
+    execution_time = time.monotonic() - start
+    if isinstance(process_error, subprocess.TimeoutExpired):
+        execution_time = _config.timeout.bcal_execution
+
+    metrics, scenario_error = inspect_scenario_result(result_path, len(entry.turns), execution_time)
+    diagnostics = [detail for detail in (scenario_error, stdout, stderr) if detail]
+    try:
+        (result_path.parent / "stdout.log").write_text(stdout, encoding="utf-8")
+        (result_path.parent / "stderr.log").write_text(stderr, encoding="utf-8")
+    except OSError as exc:
+        diagnostics.append(f"Could not save scenario diagnostics: {exc}")
+        scenario_error = scenario_error or str(exc)
+
+    config = ExperimentConfiguration()
+    if isinstance(process_error, subprocess.TimeoutExpired):
+        message = "\n".join([f"bcal CLI timed out after {_config.timeout.bcal_execution} seconds", *diagnostics])
+        raise AgentTimeoutError(message, metrics=metrics, config=config)
+    if isinstance(process_error, subprocess.CalledProcessError):
+        message = "\n".join([f"bcal CLI exited with status {process_error.returncode}", *diagnostics])
+        raise BCalScenarioError(message, metrics=metrics)
+    if process_error is not None:
+        message = "\n".join([f"bcal CLI execution failed: {process_error}", *diagnostics])
+        raise BCalScenarioError(message, metrics=metrics)
+    if scenario_error:
+        raise BCalScenarioError("\n".join(diagnostics), metrics=metrics)
+
+    logger.info(f"bcal CLI scenario complete for: {entry.instance_id}")
+    return metrics, config
+
+
 def run_bcal_prompt(
     entry: NL2ALEntry,
     query: str,