feat(ingest): weekly crawler that opens PRs to TechAPI with new SKUs

[Seungpyo1007]

Purpose

Long-running TechEngine tracker for finding missing TechAPI records, drafting import candidates, requesting PR validation, and keeping TechAPI data PRs reviewable.

This issue is the TechEngine-side companion to GetTechAPI/TechAPI#1. It should stay open while the dataset rebuild continues. TechAPI PRs may use Closes GetTechAPI/TechAPI#1 for Development linking, but this TechEngine issue tracks the automation and validation work behind those PRs.

Current Status

• State: active TechEngine automation / validation tracker
• Companion TechAPI tracker: Massive dataset rebuild: CPU + brand + GPU + smartphone + SoC (1989-2026) TechAPI#1
• Current TechAPI data PR: data(smartphone): import 5000 PhoneDB raw variants TechAPI#25
• Current active import branch: data/import-staging
• Current active batch: +5,045 smartphone records from Kaggle / PhoneDB-derived sources
• TechEngine role: validate the PR, summarize changed data, report dataset stats, and warn about low verified coverage without blocking bulk imports
• Import policy: new bulk records should remain verified: false until manual audit or follow-up verification confirms them

Latest Dataset Snapshot From PR #25

Category	Total	Verified	Unverified	Missing verified	Verified %
brand	129	0	0	129	n/a
soc	195	58	137	0	29.7%
smartphone	6,544	184	6,360	0	2.8%
gpu	2,030	0	2,030	0	0.0%
cpu	3,977	976	3,001	0	24.5%
all	12,896	1,218	11,549	129	9.5%

TechEngine Responsibilities

• Maintain coverage reports that show missing CPU, GPU, smartphone, SoC, and brand records
• Generate or support import batches where source coverage is useful
• Validate TechAPI PRs from reusable branches such as data/import-staging and feat/site
• Post two useful PR comments:
  • Changed-data review: what changed, examples, source/verified counts, and heuristic warnings
  • Validation stats: totals, verified coverage, warning callouts, and key command output
• Keep validation warnings actionable without failing expected bulk-import cases
• Add site-build verification only when site/ files changed
• Keep project metadata, priority, labels, milestones, assignees, and issue linkage filled in automatically where possible

Validation Contract

Data PR validation should include:

Check	Expected behavior
python -m app.validate	Hard fail on schema/API validation errors
python integrity_check.py TechAPI/data --strict	Hard fail on structural anomalies; summarize stable advisory outliers
Verified coverage warning	Warn when category or overall verified coverage is low; do not fail bulk import PRs
Heuristic review	Flag suspicious names, typo-like patterns, duplicates, or source artifacts
Site build	Run only when TechAPI site/ files changed

Recent Linked Work

PR / Issue	Repository	Status	Main change
TechAPI#25	TechAPI	Open	Import 5,000 PhoneDB raw smartphone variants plus 45 Mobiles 2025 records
TechAPI#24	TechAPI	Merged	Add smartphone and SoC records, improve PR metadata and project automation
TechAPI#23	TechAPI	Merged	Import a larger smartphone batch
TechAPI#22	TechAPI	Merged	Add smartphone and SoC records from Kaggle-derived sources
TechEngine#19	TechEngine	Open	Auto-generated coverage gaps report
TechEngine#18	TechEngine	Open	Deterministic dump timestamps for daily refresh readiness

Remaining Work

• Keep coverage reports useful by reducing obvious table-artifact slugs and source noise
• Expand automated candidate generation beyond CPU/GPU into smartphone and SoC sources where structured data is reliable
• Continue improving PR comments so reviewers can see what changed without opening thousands of files
• Make low verified coverage warnings clear and category-specific
• Keep issue and project metadata synchronized for TechAPI and TechEngine PRs
• Preserve weekly coverage workflows unless a separate daily/PR workflow is explicitly added
• Avoid closing this tracker until the TechAPI dataset rebuild and supporting automation are mature

Operational Notes

• Assignees: @Seungpyo1007 and @TechEngineBot
• Labels: enhancement
• Milestone: Daily automation
• Project: TechEngine work
• Priority: High
• Start date: 2026-05-29
• Target date: 2026-09-30

TechEngineBot should comment on relevant PRs and update linked tracker issues whenever validation or metadata automation runs.

[Seungpyo1007]

Linked TechAPI PR update: data(smartphone): import 5000 PhoneDB raw variants #25

• TechAPI PR: data(smartphone): import 5000 PhoneDB raw variants TechAPI#25
• TechAPI tracker: Massive dataset rebuild: CPU + brand + GPU + smartphone + SoC (1989-2026) TechAPI#1
• Branch: data/import-staging
• Priority: High
• TechEngine role: PR validation, changed-data summary, verified-coverage warning, and project/issue metadata support
• Current state: validation checks passed on the latest PR update

Changed Data

Category	Added	Modified	Deleted
smartphone	5,045	0	0

Validation Snapshot

Check	Result
TechAPI self-validate	PASS
TechEngine engine-validate / validate	PASS
Metadata triage	PASS
Validation request workflow	PASS

Dataset After This PR

Category	Total	Verified	Unverified	Missing verified	Verified %
brand	129	0	0	129	n/a
soc	195	58	137	0	29.7%
smartphone	6,544	184	6,360	0	2.8%
gpu	2,030	0	2,030	0	0.0%
cpu	3,977	976	3,001	0	24.5%
all	12,896	1,218	11,549	129	9.5%

Notes

• Bulk records remain verified: false by design.
• Low verified coverage should be reported as a warning, not a hard validation failure.
• This TechEngine issue remains open as the long-running automation and validation tracker.