diff --git a/README.md b/README.md index 70d18d8..a32f12c 100644 --- a/README.md +++ b/README.md @@ -469,6 +469,20 @@ metadata: and `python -m inferedge_aiguard.cli build-runtime-intelligence-optional-stale-drop`. This lets downstream reviewers connect the EdgeEnv handoff to the AIGuard source fixture without making EdgeEnv produce `guard_analysis`. +Verify the EdgeEnv-owned replay/regression/handoff path locally with: + +```bash +bash scripts/smoke_runtime_intelligence_replay_regression_handoff.sh \ + --output-dir reports/runtime_intelligence_replay_regression_handoff +``` + +This smoke records two local runtime-telemetry runs, exports and inspects an +`edgeenv.runtime-telemetry-history.v1` replay artifact, generates a +comparability-first regression report with `--telemetry-history`, and writes +the Runtime Intelligence Lab handoff manifest. It checks that +`history_seed_run_config` markers reach the handoff summary, that regression +deltas remain gated by `same-condition` comparability, and that EdgeEnv still +does not produce AIGuard `guard_analysis`. Verify that producer-side source traceability path locally with: ```bash diff --git a/docs/compare-workflow-guide.md b/docs/compare-workflow-guide.md index 0f7bf76..6ad8262 100644 --- a/docs/compare-workflow-guide.md +++ b/docs/compare-workflow-guide.md @@ -116,6 +116,17 @@ If a runtime telemetry history artifact is available, pass regression report. Telemetry context remains supplemental evidence; it never bypasses the same-condition comparability gate. +To replay that path with local fixtures and a Lab handoff manifest, run: + +```bash +bash scripts/smoke_runtime_intelligence_replay_regression_handoff.sh \ + --output-dir reports/runtime_intelligence_replay_regression_handoff +``` + +The smoke exercises `runs telemetry export-history`, `runs telemetry +inspect-history`, `report regression --telemetry-history`, and `report +runtime-intelligence-handoff` without creating production monitoring state. + Committed replay-context examples are available when downstream tools need a small EdgeEnv-owned fixture without running a benchmark: diff --git a/docs/ko/README.md b/docs/ko/README.md index 7a97dca..d9ba5ac 100644 --- a/docs/ko/README.md +++ b/docs/ko/README.md @@ -187,6 +187,21 @@ mirror한다: `python -m inferedge_aiguard.cli build-runtime-intelligence-optional-stale-drop`. 이는 EdgeEnv handoff와 AIGuard source fixture를 추적 가능하게 할 뿐, EdgeEnv가 `guard_analysis`를 생성한다는 의미가 아니다. +아래 smoke는 EdgeEnv가 소유한 replay/regression/handoff 경로를 실제 CLI로 +검증한다. + +```bash +bash scripts/smoke_runtime_intelligence_replay_regression_handoff.sh \ + --output-dir reports/runtime_intelligence_replay_regression_handoff +``` + +이 smoke는 runtime telemetry가 있는 두 local run을 기록하고, +`edgeenv.runtime-telemetry-history.v1` artifact를 export/inspect한 뒤, +`report regression --telemetry-history`와 +`report runtime-intelligence-handoff`까지 실행한다. 통과 기준은 +`history_seed_run_config` marker가 handoff summary까지 보존되고, +regression delta가 여전히 `same-condition` comparability gate 뒤에서만 +계산되며, EdgeEnv가 AIGuard `guard_analysis`를 생성하지 않는 것이다. 아래 smoke는 이 producer-side source traceability 경로를 로컬에서 검증한다. sibling InferEdgeLab checkout이 있으면 생성된 EdgeEnv handoff manifest와 AIGuard optional-present alignment fixture를 Lab source traceability gate로도 diff --git a/docs/portfolio_summary.md b/docs/portfolio_summary.md index 80e6598..377b8e4 100644 --- a/docs/portfolio_summary.md +++ b/docs/portfolio_summary.md @@ -62,6 +62,13 @@ It also mirrors AIGuard optional-present source traceability through `InferEdgeAIGuard/examples/runtime_intelligence/aiguard_runtime_operation_guard_analysis_optional_stale_drop.json` and `python -m inferedge_aiguard.cli build-runtime-intelligence-optional-stale-drop`, without making EdgeEnv produce the external `guard_analysis`. +`scripts/smoke_runtime_intelligence_replay_regression_handoff.sh` exercises the +EdgeEnv-owned CLI path before source traceability handoff: local benchmark runs +with runtime telemetry, telemetry history export/inspect, comparability-first +`report regression --telemetry-history`, and Runtime Intelligence Lab handoff +manifest generation. The smoke verifies that `history_seed_run_config` replay +markers reach the handoff summary while regression deltas remain gated by +same-condition comparability. `scripts/smoke_runtime_intelligence_source_traceability.sh` regenerates a producer-side handoff manifest and, when a sibling InferEdgeLab checkout is available, runs Lab's source traceability gate against the AIGuard diff --git a/docs/runtime-telemetry-history.md b/docs/runtime-telemetry-history.md index 2105002..2136a64 100644 --- a/docs/runtime-telemetry-history.md +++ b/docs/runtime-telemetry-history.md @@ -303,6 +303,19 @@ edgeenv report regression \ --output-md /tmp/edgeenv-regression.md ``` +Reviewer smoke for the same path plus Runtime Intelligence Lab handoff: + +```bash +bash scripts/smoke_runtime_intelligence_replay_regression_handoff.sh \ + --output-dir reports/runtime_intelligence_replay_regression_handoff +``` + +This smoke uses public CLI commands to record local runs with runtime telemetry, +export and inspect replay history, generate comparability-first regression +evidence, and write the EdgeEnv-to-Lab handoff manifest. It validates +`history_seed_run_config` traceability without promoting telemetry into a +production monitoring stream. + The regression report records telemetry coverage and evidence gaps for the baseline/candidate pair. It still calculates regression deltas only after the normal same-condition comparability gate passes. diff --git a/scripts/smoke_runtime_intelligence_replay_regression_handoff.sh b/scripts/smoke_runtime_intelligence_replay_regression_handoff.sh new file mode 100755 index 0000000..9d00dfc --- /dev/null +++ b/scripts/smoke_runtime_intelligence_replay_regression_handoff.sh @@ -0,0 +1,434 @@ +#!/usr/bin/env bash +set -euo pipefail + +EDGEENV_DIR="${EDGEENV_DIR:-$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)}" +OUTPUT_DIR="${OUTPUT_DIR:-$EDGEENV_DIR/reports/runtime_intelligence_replay_regression_handoff}" + +usage() { + cat <<'EOF' +InferEdgeEnv Runtime Intelligence replay/regression/handoff smoke + +Usage: + bash scripts/smoke_runtime_intelligence_replay_regression_handoff.sh [--output-dir ] + bash scripts/smoke_runtime_intelligence_replay_regression_handoff.sh --help + +This smoke exercises the local-first EdgeEnv Runtime Intelligence path through +the public CLI: + bench run with runtime telemetry + -> runs telemetry export-history + -> runs telemetry inspect-history + -> report regression --telemetry-history + -> report runtime-intelligence-handoff + +It validates that telemetry history/replay and history_seed run_config context +remain supplemental evidence and that regression deltas are still gated by +same-condition comparability before Lab handoff metadata is produced. +EOF +} + +while [[ $# -gt 0 ]]; do + case "$1" in + --help|-h) + usage + exit 0 + ;; + --output-dir) + if [[ $# -lt 2 ]]; then + echo "Missing value for --output-dir" >&2 + exit 2 + fi + OUTPUT_DIR="$2" + shift + ;; + *) + echo "Unknown argument: $1" >&2 + usage >&2 + exit 2 + ;; + esac + shift +done + +cd "$EDGEENV_DIR" +mkdir -p "$OUTPUT_DIR" + +PYTHON_CMD=(python) +WORK_DIR="$OUTPUT_DIR/work" +EDGEENV_ROOT="$OUTPUT_DIR/.edgeenv" +EMITTER="$WORK_DIR/emit_runtime_intelligence_metrics.py" +TARGET_YAML="$WORK_DIR/local_runtime_intelligence_target.yaml" +BASELINE_CONFIG="$WORK_DIR/baseline_runtime_intelligence.yaml" +CANDIDATE_CONFIG="$WORK_DIR/candidate_runtime_intelligence.yaml" +BASELINE_LOG="$OUTPUT_DIR/baseline_bench_run.log" +CANDIDATE_LOG="$OUTPUT_DIR/candidate_bench_run.log" +EXPORT_LOG="$OUTPUT_DIR/telemetry_export_history.log" +INSPECT_LOG="$OUTPUT_DIR/telemetry_inspect_history.log" +REGRESSION_LOG="$OUTPUT_DIR/regression_report.log" +HANDOFF_LOG="$OUTPUT_DIR/runtime_intelligence_handoff.log" +HISTORY_JSON="$OUTPUT_DIR/runtime_telemetry_history.json" +REGRESSION_JSON="$OUTPUT_DIR/edgeenv_runtime_regression.json" +REGRESSION_MD="$OUTPUT_DIR/edgeenv_runtime_regression.md" +HANDOFF_JSON="$OUTPUT_DIR/edgeenv_runtime_intelligence_lab_handoff.json" +SUMMARY_MD="$OUTPUT_DIR/runtime_intelligence_replay_regression_handoff_summary.md" + +echo "== EdgeEnv Runtime Intelligence replay/regression/handoff smoke ==" +echo "Output: $OUTPUT_DIR" + +rm -rf "$WORK_DIR" "$EDGEENV_ROOT" +mkdir -p "$WORK_DIR" + +cat > "$EMITTER" <<'PY' +from __future__ import annotations + +import json +import os + + +VARIANTS = { + "baseline": { + "sequence_id": 1, + "timestamp": "2026-05-22T00:00:01Z", + "metrics": { + "latency_mean_ms": 100.0, + "latency_p50_ms": 96.0, + "latency_p95_ms": 120.0, + "latency_p99_ms": 130.0, + "throughput_fps": 50.0, + }, + }, + "candidate": { + "sequence_id": 2, + "timestamp": "2026-05-22T00:00:02Z", + "metrics": { + "latency_mean_ms": 118.0, + "latency_p50_ms": 114.0, + "latency_p95_ms": 132.0, + "latency_p99_ms": 171.6, + "throughput_fps": 39.0, + }, + }, +} + + +def runtime_history_seed(sequence_id: int, timestamp: str) -> dict: + return { + "schema_version": "inferedge-runtime-telemetry-history-seed-v1", + "evidence_role": "runtime_telemetry_history_seed", + "registry_owner": "edgeenv", + "decision_owner": "lab", + "source_result_schema_version": "inferedge-runtime-result-v1", + "source_telemetry_schema_version": "inferedge-runtime-telemetry-v1", + "replay_scope": "single_result_to_history", + "replay_ready": True, + "production_monitoring": False, + "missing_telemetry_is_failure": False, + "source_result": { + "compare_key": "runtime-intelligence-smoke__b1__h224w224__fp32", + "backend_key": "local-python__cpu", + "engine_backend": "local-python", + "device": "cpu", + "precision": "fp32", + "power_mode": "unknown", + }, + "run_config": { + "batch": 1, + "height": 224, + "width": 224, + "warmup": 1, + "runs": 3, + "timeout_ms": None, + "input_mode": "synthetic", + "input_preprocess": "none", + "power_mode": "unknown", + "jetson_clocks": "unknown", + }, + "recommended_registry_key_fields": [ + "compare_key", + "backend_key", + "device", + "precision", + "power_mode", + "run_config", + ], + "time_series_fields": [ + "telemetry_timestamp", + "execution_sequence_id", + "latency.mean_ms", + "latency.p99_ms", + "operation.timeout_observed", + ], + "points": [ + { + "execution_sequence_id": sequence_id, + "telemetry_timestamp": timestamp, + "mean_ms": 100.0 + sequence_id, + "p99_ms": 130.0 + sequence_id, + "timeout_observed": False, + } + ], + } + + +def runtime_telemetry(sequence_id: int, timestamp: str, metrics: dict) -> dict: + expected_fields = ["queue_depth", "gpu_temperature", "telemetry_timestamp"] + return { + "schema_version": "inferedge-runtime-telemetry-v1", + "collection_mode": "single_result_export", + "telemetry_timestamp": timestamp, + "execution_sequence_id": sequence_id, + "latency": { + "mean_ms": metrics["latency_mean_ms"], + "p99_ms": metrics["latency_p99_ms"], + }, + "resource": { + "telemetry_source": "runtime-result", + "gpu_temperature": 55.0 + sequence_id, + }, + "operation": { + "queue_depth": sequence_id - 1, + "timeout_observed": False, + }, + "coverage": { + "schema_version": "inferedge-runtime-telemetry-coverage-v1", + "expected_fields": expected_fields, + "observed_fields": expected_fields, + "missing_fields": [], + "expected_field_count": len(expected_fields), + "observed_field_count": len(expected_fields), + "missing_field_count": 0, + "coverage_ratio": 1.0, + "comparability_owner": "edgeenv", + "missing_telemetry_is_failure": False, + }, + "missing_fields": [], + "production_monitoring": False, + "history_seed": runtime_history_seed(sequence_id, timestamp), + } + + +def main() -> int: + variant_name = os.environ.get("RUNTIME_INTELLIGENCE_VARIANT", "baseline") + variant = VARIANTS.get(variant_name) + if variant is None: + valid = ", ".join(sorted(VARIANTS)) + print(f"Unknown RUNTIME_INTELLIGENCE_VARIANT={variant_name!r}; expected one of {valid}") + return 2 + + metrics = variant["metrics"] + telemetry = runtime_telemetry( + variant["sequence_id"], + variant["timestamp"], + metrics, + ) + print(f"runtime_intelligence_variant={variant_name}") + print("EDGEENV_METRICS_JSON=" + json.dumps(metrics, sort_keys=True)) + print("EDGEENV_RUNTIME_TELEMETRY_JSON=" + json.dumps(telemetry, sort_keys=True)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) +PY + +cat > "$TARGET_YAML" <<'EOF' +target_name: runtime-intelligence-local-smoke +target_type: local +board_name: local-dev-machine +os: local +runtime_tags: + - local + - runtime-intelligence-smoke +EOF + +cat > "$BASELINE_CONFIG" < "$CANDIDATE_CONFIG" < "$BASELINE_LOG" + +"${PYTHON_CMD[@]}" -m inferedge_env.cli bench run \ + --edgeenv-root "$EDGEENV_ROOT" \ + --target "$TARGET_YAML" \ + --config "$CANDIDATE_CONFIG" > "$CANDIDATE_LOG" + +BASELINE_RUN_ID="$(sed -n 's/^Run ID: //p' "$BASELINE_LOG" | tail -n 1)" +CANDIDATE_RUN_ID="$(sed -n 's/^Run ID: //p' "$CANDIDATE_LOG" | tail -n 1)" +if [[ -z "$BASELINE_RUN_ID" || -z "$CANDIDATE_RUN_ID" ]]; then + echo "Failed to parse run IDs from bench run logs" >&2 + exit 1 +fi + +BASELINE_RESULT="$EDGEENV_ROOT/runs/$BASELINE_RUN_ID/result.json" +CANDIDATE_RESULT="$EDGEENV_ROOT/runs/$CANDIDATE_RUN_ID/result.json" + +"${PYTHON_CMD[@]}" -m inferedge_env.cli runs telemetry export-history \ + --edgeenv-root "$EDGEENV_ROOT" \ + --output "$HISTORY_JSON" > "$EXPORT_LOG" + +"${PYTHON_CMD[@]}" -m inferedge_env.cli runs telemetry inspect-history \ + "$HISTORY_JSON" > "$INSPECT_LOG" + +"${PYTHON_CMD[@]}" -m inferedge_env.cli report regression \ + "$BASELINE_RUN_ID" \ + "$CANDIDATE_RUN_ID" \ + --edgeenv-root "$EDGEENV_ROOT" \ + --telemetry-history "$HISTORY_JSON" \ + --output-json "$REGRESSION_JSON" \ + --output-md "$REGRESSION_MD" > "$REGRESSION_LOG" + +"${PYTHON_CMD[@]}" -m inferedge_env.cli report runtime-intelligence-handoff \ + --baseline-result "$BASELINE_RESULT" \ + --candidate-result "$CANDIDATE_RESULT" \ + --edgeenv-regression-report "$REGRESSION_JSON" \ + --telemetry-history "$HISTORY_JSON" \ + --output "$HANDOFF_JSON" > "$HANDOFF_LOG" + +"${PYTHON_CMD[@]}" - "$HISTORY_JSON" "$REGRESSION_JSON" "$HANDOFF_JSON" <<'PY' +import json +import sys +from pathlib import Path + +history = json.loads(Path(sys.argv[1]).read_text(encoding="utf-8")) +regression = json.loads(Path(sys.argv[2]).read_text(encoding="utf-8")) +handoff = json.loads(Path(sys.argv[3]).read_text(encoding="utf-8")) + +summary = history.get("summary", {}) +if history.get("schema_version") != "edgeenv.runtime-telemetry-history.v1": + raise SystemExit("unexpected telemetry history schema") +if summary.get("telemetry_runs") != 2: + raise SystemExit("telemetry history must include two telemetry runs") +if summary.get("history_seed_runs") != 2: + raise SystemExit("telemetry history must preserve two history seeds") +if summary.get("history_seed_run_config_runs") != 2: + raise SystemExit("telemetry history must preserve two history seed run_config snapshots") +if summary.get("missing_telemetry_runs") != 0: + raise SystemExit("telemetry history smoke should not create missing telemetry gaps") + +if regression.get("mode") != "same-condition": + raise SystemExit("regression smoke must stay same-condition") +if regression.get("regression_detected") is not True: + raise SystemExit("regression smoke must detect the candidate regression") +triggered = { + item.get("name") + for item in regression.get("evidence", {}).get("triggered_thresholds", []) + if isinstance(item, dict) +} +if "p99_latency_high" not in triggered: + raise SystemExit("regression smoke must trigger p99 latency review evidence") +context = regression.get("runtime_telemetry_context") +if not isinstance(context, dict): + raise SystemExit("regression smoke must attach runtime telemetry context") +if context.get("role") != "supplemental_runtime_telemetry_context": + raise SystemExit("runtime telemetry context role drifted") +if "guard_analysis" in regression: + raise SystemExit("EdgeEnv regression must not include guard_analysis") +if "Regression deltas are still gated by same-condition comparability." not in context.get("notes", []): + raise SystemExit("regression smoke lost comparability-first note") +if context.get("history", {}).get("summary", {}).get("history_seed_run_config_runs") != 2: + raise SystemExit("regression telemetry context lost history seed run_config summary") + +if handoff.get("schema_version") != "edgeenv.runtime-intelligence-lab-handoff.v1": + raise SystemExit("unexpected handoff schema") +if "guard_analysis" in handoff: + raise SystemExit("EdgeEnv handoff must not include guard_analysis") +files = handoff.get("files", {}) +if "runtime_telemetry_history" not in files: + raise SystemExit("handoff must reference runtime telemetry history") +edgeenv_summary = handoff.get("edgeenv_report_summary", {}) +if edgeenv_summary.get("history_seed_runs") != 2: + raise SystemExit("handoff must summarize two history seed runs") +if edgeenv_summary.get("history_seed_run_config_runs") != 2: + raise SystemExit("handoff must summarize two history seed run_config snapshots") +if not edgeenv_summary.get("history_seed_run_config_markers"): + raise SystemExit("handoff must expose compact history seed run_config markers") +alignment = handoff.get("lab_bundle_alignment", {}) +flags = alignment.get("boundary_flags", {}) +if flags.get("edgeenv_does_not_generate_guard_analysis") is not True: + raise SystemExit("handoff must keep EdgeEnv guard_analysis boundary") +if flags.get("lab_is_final_decision_owner") is not True: + raise SystemExit("handoff must keep Lab final decision ownership") +if "aiguard_guard_analysis" not in alignment.get("external_file_keys", []): + raise SystemExit("handoff must keep AIGuard guard_analysis external") +PY + +grep -q "Runtime telemetry history valid" "$INSPECT_LOG" +grep -q "Runtime history seed run_config runs: 2" "$INSPECT_LOG" +grep -q "Scope: read-only local replay validation" "$INSPECT_LOG" +grep -q "Regression detected: true" "$REGRESSION_LOG" +grep -q "Runtime Telemetry Context:" "$REGRESSION_LOG" +grep -q "p99_latency_high" "$REGRESSION_LOG" +grep -q "role: supplemental context, not a comparability gate" "$REGRESSION_LOG" +grep -q "History seed entries: 2" "$HANDOFF_LOG" +grep -q "History seed run_config markers:" "$HANDOFF_LOG" +grep -q "Lab remains the final deployment decision owner." "$HANDOFF_LOG" + +cat > "$SUMMARY_MD" <