feat(verify): tiered data verification layer (Tier 0 offline scoring)

[Seungpyo1007]

What

Adds app/verify/ — an existence/trust verification layer that sits on top of the structural validator (app/validate.py, left untouched). Where validate.py only checks "is this file well-formed?", this answers "does this record describe a real, actually-existing device/part — confidently enough to set verified:true?" — the goal being to lift the dataset's ~1.2% verified ratio.

Tiers

• Tier 0 — offline, deterministic, all ~102k records (offline.py/signals.py/hosts.py): four sub-scores (completeness + cross-field consistency + source-host trust + provenance) → a green/yellow/red band. Hard contradictions (threads<cores, boost<base, future release) force red. Full scores cached to gitignored data/_verify/state/; the tracked data/_verify/ledger.jsonl is reserved for promotion decisions.
• Tier 1 — http_check.py: source_urls HTTP liveness (stdlib urllib + ThreadPool, per-host rate limit, resumable TTL cache).
• Tier 2 — crossref.py: external cross-reference under a strict exact-heading rule — no fuzzy matching (fuzzy serves the wrong SKU ~35% of the time), so ambiguous candidates never auto-promote.
• Tier 3 — promote.py: hybrid escalation + surgical verified:false→true write-back (only that token, atomic, LF-preserved; never clobbers curated records or reformats inline arrays).

CLI

python -m app.verify score | report | check-urls | crossref | promote

CI

• Non-blocking verify-offline job in validate-data.yml (continue-on-error, scores changed records + runs unit tests) — never gates a merge.
• Scheduled/manual verify-network.yml for the network tiers, with a diff-scope guard that fails unless only verified flags + the ledger changed. No TechEngine/submodule ops.

Validation

40 tests pass. The golden-subset test confirms the offline scorer — blind to the verified flag — reproduces the human-curated verified-CPU set (976/976 land green), which is the empirical justification for using the score to drive promotion.

Tuning note: soc_not_after_device is a soft signal, not hard — the dataset's SoC release_date values are largely placeholder YYYY-01-01 that skew late, so a device-vs-SoC mismatch usually means the SoC date is wrong, not the device. Fixing those dates is a separate enrichment task.

Refs #1

[TechEngineBot]

TechEngine change review: PASS

• PR: feat(verify): tiered data verification layer (Tier 0 offline scoring) #52
• Ref: feat/verify-layer
• Commit: 228a38f
• Requested by: @Seungpyo1007
• Run: https://github.com/GetTechAPI/TechEngine/actions/runs/27926651797

Check	Result
python -m app.validate	PASS
python integrity_check.py TechAPI/data --strict	PASS

Changed data

Category	Added	Modified	Deleted	Added verified	Added unverified	Added Kaggle-sourced
brand	0	0	0	0	0	0
soc	0	0	0	0	0	0
smartphone	0	0	0	0	0	0
tablet	0	0	0	0	0	0
watch	0	0	0	0	0	0
pda	0	0	0	0	0	0
gpu	0	0	0	0	0	0
cpu	0	0	0	0	0	0

Changed record examples

• No data file changes detected.

Heuristic review

• Heuristic warnings: none found.

[TechEngineBot]

TechEngine validation stats: PASS

• PR: feat(verify): tiered data verification layer (Tier 0 offline scoring) #52
• Ref: feat/verify-layer
• Commit: 228a38f
• Run: https://github.com/GetTechAPI/TechEngine/actions/runs/27926651797

Data summary

Category	Total	Verified	Unverified	Missing verified	Tracked	Verified % of tracked
brand	189	0	189	0	189	0.0%
soc	2104	58	2046	0	2104	2.8%
smartphone	90118	184	89934	0	90118	0.2%
tablet	3048	0	3048	0	3048	0.0%
watch	378	0	378	0	378	0.0%
pda	110	0	110	0	110	0.0%
gpu	2030	0	2030	0	2030	0.0%
cpu	3977	976	3001	0	3977	24.5%
all	101954	1218	100736	0	101954	1.2%

Warning

Tracked verified coverage is below 50% for brand 0.0% (0/189), tablet 0.0% (0/3048), watch 0.0% (0/378), pda 0.0% (0/110), gpu 0.0% (0/2030), smartphone 0.2% (184/90118), all 1.2% (1218/101954), soc 2.8% (58/2104), and 1 more.
Tracked coverage excludes records missing the verified field; see the Missing verified column for those records.
This does not fail validation. Keep imported records verified: false until manual audit, but treat this as follow-up verification work before relying on the affected categories as curated data.

Validation notes

• Full advisory outlier listings are suppressed on successful runs because they are dataset-wide and mostly stable between PRs.
• Failure runs still include a detailed log excerpt for debugging.

Key output:

## app.validate
## integrity_check.py --strict
loaded CPU=3977 GPU=2030
✅ integrity gate: no hard anomalies.

Integrity section	Flagged lines
structural	0
CPU name/tier consistency (desktop mainstream only)	0
CPU single>multi (cinebench/geekbench — should be multi>=single)	0
CPU era-vs-score outliers	8
CPU cross-source ratio outliers (possible wrong-variant)	152
GPU cross-source ratio outliers + sanity	18

[Seungpyo1007]

🔎 Data verification — Tier 0 (offline existence/trust)

Scored by app.verify; posted by TechEngineBot. Informational only —
the structural gate (app.validate) is separate and authoritative for merge.

### Changed records in this PR
Tier 0 offline score — 0 record(s)

category        green   yellow      red    total
------------------------------------------------
------------------------------------------------
ALL                 0        0        0        0

bands: green 0.0%  yellow 0.0%  red 0.0%

### Full-dataset baseline
Tier 0 offline score — 101954 record(s)

category        green   yellow      red    total
------------------------------------------------
brand              10      179        0      189
soc               123      680     1301     2104
smartphone       8453    80547     1118    90118
tablet            174     2846       28     3048
watch              11      357       10      378
pda                27       77        6      110
gpu               245     1785        0     2030
cpu               976     3001        0     3977
------------------------------------------------
ALL             10019    89472     2463   101954

bands: green 9.8%  yellow 87.8%  red 2.4%

_{green = authoritative source + complete + consistent · yellow = plausible, needs confirmation · red = sparse/weak source or a hard contradiction. Promotion to verified runs in the scheduled verify-network workflow.}

[TechEngineBot]

🔎 Data verification — Tier 0 (on demand)

Requested by @Seungpyo1007 via /verify · scored by app.verify, posted by TechEngineBot. Informational only — the structural gate (app.validate) is separate.

### Changed records in this PR
Tier 0 offline score — 0 record(s)

category        green   yellow      red    total
------------------------------------------------
------------------------------------------------
ALL                 0        0        0        0

bands: green 0.0%  yellow 0.0%  red 0.0%

### Full-dataset baseline
Tier 0 offline score — 101954 record(s)

category        green   yellow      red    total
------------------------------------------------
brand              10      179        0      189
soc               123      680     1301     2104
smartphone       8453    80547     1118    90118
tablet            174     2846       28     3048
watch              11      357       10      378
pda                27       77        6      110
gpu               245     1785        0     2030
cpu               976     3001        0     3977
------------------------------------------------
ALL             10019    89472     2463   101954

bands: green 9.8%  yellow 87.8%  red 2.4%