Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 29 additions & 0 deletions docs/audits/v0.5.0_dependency_integration_current_state.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
# v0.5.0 dependency integration current-state audit

Production scoring starts in `targetintel/scoring.py` (`score_all_profiles`)
and uses the three `configs/scoring_*.yaml` profiles. Production intent ranking
starts in `targetintel/intent_ranking.py` (`build_intent_rankings` and
`add_intent_ranks`); it adds separate rank columns to a dataframe and remains
the default path. Selected-state values are carried by baseline artifacts but
are not a registration mechanism. The global CLI (`targetintel/cli.py`) calls
the deterministic pipeline; it has no optional dependency-profile selector.

Issue 505 is isolated in `targetintel/functional_dependency/depmap_benchmark.py`.
It consumes explicit frozen universe/profile/baseline/policy inputs and writes
`dependency_benchmark_manifest.json`, metrics, bounded-overlay diagnostics, and
`integration_evidence.json`. Its manifest is analysis-only and has status
`human_review_required`; its overlay ranks are separate diagnostic fields.

There was no profile registry, auto-discovery, environment-variable enablement,
or global optional-profile selection mechanism. A file could not become the
production default through presence alone. The baseline benchmark loader reads
a TSV and calculates a fingerprint; it does not mutate the source file.

The canonical Issue 506 boundary is the new isolated
`targetintel.functional_dependency.dependency_integration` module. It verifies
the Issue 505 manifest and its profile/universe/baseline/policy identities,
uses an explicit context and evidence scope, builds a separately stored
candidate overlay, and proves baseline byte/fingerprint preservation. The
candidate retains baseline ranking/profile provenance and is selectable only by
a future explicit caller plus separate approved human authorization. It is not
connected to production scoring/ranking or the global CLI.
60 changes: 60 additions & 0 deletions docs/specs/v0.5.0_dependency_integration_gate.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
# v0.5.0 dependency integration gate

Issue 506 is an offline, isolated decision gate between the Issue 505
analysis-only dependency benchmark and any future optional profile. Its
canonical boundary is `targetintel.functional_dependency.dependency_integration`.
It reads only an explicit baseline TSV and Issue 505 artifacts; it does not
import production scoring, intent ranking, role classification, feature
construction, modality, CLI, reports, LLMs, or network code.

The controlled evidence scopes are `synthetic_fixture`, `local_real_data`,
and `externally_validated_real_data`. Scope is a required caller argument and
enters the decision identity. Synthetic fixture scope always produces
`blocked_fixture_evidence`; a real-data label alone does not establish
scientific adequacy.

The controlled states are `blocked_fixture_evidence`,
`blocked_insufficient_evidence`, `blocked_incompatible_inputs`,
`blocked_policy_failure`, `eligible_for_human_activation`,
`human_review_required`, and `explicitly_rejected`. Unknown values fail closed.
Eligibility is not activation.

`DependencyIntegrationPolicy` is immutable and content-addressed. It fixes
coverage, holdout, eligible-count, Recall@K, negative-control, bounded-overlay
stability, displacement, band-violation, ablation, and missing-profile
thresholds; construction method; fallback; opt-in; human-approval requirement;
and limitations. It rejects target-specific settings, tuning, callbacks,
expressions, credentials, and hidden reasoning. Results never enter policy
identity.

The gate verifies frozen-universe, benchmark-universe, profile-run, baseline
ranking and SHA-256 fingerprint, Issue 505 policy, and context identities.
Missing or conflicting identities fail closed and retain exact reasons. Every
criterion has an ID, source artifact/field, observed value, operator,
threshold, pass/fail/unavailable result, and limitations. Unavailable is never
pass; no aggregate score exists.

The deterministic candidate is named
`dependency_aware_melanoma_anti_pd1_candidate_v1`. It is context-bound,
analysis-only, non-default, and contains no benchmark or holdout labels,
target-specific weights, credentials, or clinical-validation claim. Its
bounded overlay retains baseline score and rank separately, only reorders
eligible rows inside fixed baseline bands, and preserves missing-profile order.

The baseline TSV bytes and fingerprint are checked before and after evaluation.
The gate emits separate proof that scores/ranks are retained and that production
configuration/defaults remain untouched; it cannot register a global profile.
`select_dependency_profile` returns baseline when selection is absent, rejects
unknown or blocked selection, and requires a future approved authorization even
after eligibility.

`DependencyProfileAuthorization` is a future contract only. It requires the
candidate, real-data benchmark, decision, context, reviewer reference, status,
and limitations. Fixture authorization is rejected. Issue 506 emits no approved
authorization.

The activation-readiness bundle is immutable and reports blocked, eligible for
human review, or rejected. The bundled synthetic fixture is blocked, although
it still emits an overlay for audit. Issue 507 must run the same gate after a
pinned local real-data benchmark, inspect every failure/unavailable result, and
obtain separate human authorization; it must not auto-enable the candidate.
17 changes: 17 additions & 0 deletions examples/depmap/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,3 +67,20 @@ python examples/depmap/run_dependency_benchmark.py \
--policy tests/fixtures/depmap/benchmark/evaluation_policy.json \
--output-dir /tmp/targetintel-dependency-benchmark
```

## Dependency integration gate

Issue 506 consumes that benchmark only through an explicit, offline gate. It
writes an analysis-only candidate overlay and never changes production scores,
ranks, defaults, or CLI behavior. The synthetic fixture is always blocked;
human authorization is not emitted.

```bash
python examples/depmap/run_dependency_integration_gate.py \
--benchmark-dir /tmp/targetintel-dependency-benchmark \
--baseline-ranking tests/fixtures/depmap/benchmark/baseline_ranking.tsv \
--policy tests/fixtures/depmap/integration/integration_policy.json \
--context tests/fixtures/depmap/integration/context.json \
--evidence-scope synthetic_fixture \
--output-dir /tmp/targetintel-dependency-integration
```
37 changes: 37 additions & 0 deletions examples/depmap/run_dependency_integration_gate.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
#!/usr/bin/env python3
"""Run the offline Issue 506 dependency-integration gate."""
from __future__ import annotations
import argparse
import json
from pathlib import Path
import sys

PROJECT_ROOT = Path(__file__).resolve().parents[2]
if str(PROJECT_ROOT) not in sys.path:
sys.path.insert(0, str(PROJECT_ROOT))

from targetintel.functional_dependency import (
DependencyIntegrationError, DependencyIntegrationPolicy,
build_dependency_integration, write_dependency_integration_artifacts,
)

def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--benchmark-dir", required=True)
parser.add_argument("--baseline-ranking", required=True)
parser.add_argument("--policy", required=True)
parser.add_argument("--context", required=True)
parser.add_argument("--evidence-scope", required=True)
parser.add_argument("--output-dir", required=True)
args = parser.parse_args()
try:
policy = DependencyIntegrationPolicy.from_dict(json.loads(Path(args.policy).read_text(encoding="utf-8")))
result = build_dependency_integration(Path(args.benchmark_dir).resolve(), Path(args.baseline_ranking).resolve(), policy, json.loads(Path(args.context).read_text(encoding="utf-8")), args.evidence_scope)
write_dependency_integration_artifacts(Path(args.output_dir).resolve(), result)
except (DependencyIntegrationError, ValueError) as error:
parser.error(str(error))
print(result["decision"]["decision_id"])
return 0

if __name__ == "__main__":
raise SystemExit(main())
16 changes: 16 additions & 0 deletions targetintel/functional_dependency/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,13 @@
DependencyBenchmarkPolicy, evaluate_dependency_benchmark,
load_baseline_ranking, write_dependency_benchmark_artifacts,
)
from .dependency_integration import (
DependencyAwareProfileCandidate, DependencyIntegrationError,
DependencyIntegrationPolicy, DependencyProfileAuthorization,
build_dependency_integration, select_dependency_profile,
validate_evidence_scope, validate_integration_state,
write_dependency_integration_artifacts,
)

__all__ = [
"DepMapFileManifest",
Expand Down Expand Up @@ -65,4 +72,13 @@
"evaluate_dependency_benchmark",
"load_baseline_ranking",
"write_dependency_benchmark_artifacts",
"DependencyAwareProfileCandidate",
"DependencyIntegrationError",
"DependencyIntegrationPolicy",
"DependencyProfileAuthorization",
"build_dependency_integration",
"select_dependency_profile",
"validate_evidence_scope",
"validate_integration_state",
"write_dependency_integration_artifacts",
]
Loading
Loading