Skip to content

data(smartphone): import PhoneDB raw device variants (batch 1)#27

Merged
Seungpyo1007 merged 9 commits into
mainfrom
data/phonedb-import
Jun 19, 2026
Merged

data(smartphone): import PhoneDB raw device variants (batch 1)#27
Seungpyo1007 merged 9 commits into
mainfrom
data/phonedb-import

Conversation

@Seungpyo1007

@Seungpyo1007 Seungpyo1007 commented Jun 19, 2026 •
edited by github-actions Bot
Loading

Copy link
Copy Markdown
Member

Summary

Adds 2,661 variant-level smartphone records + 127 SoC seeds scraped from PhoneDB (phonedb.net), continuing the raw-import campaign. Each PhoneDB device id is one regional/storage variant, so these are variant-level records (distinct from the model-level GSMArena import). All records are raw seed (verified: false) with source_urls pointing at the exact device page.

Source & method

  • Source: https://phonedb.net device pages (robots.txt permits — only /ad/ and one path are disallowed). Attribution via per-record source_urls.
  • Throttled concurrent fetch; no CAPTCHA/Cloudflare bypass.
  • Coverage skews recent — 2023: 768, 2024: 993, 2025: 501 — filling the post-2020 gap left by the older GSMArena source.
  • Top brands: Samsung 629, Xiaomi 383, Oppo 256, Motorola 179, Apple 164, Huawei 158, Vivo 124, ZTE 86, …

Notes

  • SoCs auto-seeded where missing (verified: false).
  • Existing brands only; devices with brands not yet in data/brand were skipped (follow-up).
  • Local validator passes (python -m app.validate).
  • Data-only: the static site/public/v1 dump is left to the weekly-refresh pipeline.
  • First batch of an ongoing PhoneDB import (resumable; ~22k device ids remain).

Relates to #1.

Closes #1

@github-actions github-actions Bot added data Dataset changes enhancement New feature or request labels Jun 19, 2026
@github-actions github-actions Bot mentioned this pull request Jun 19, 2026
Open
@Seungpyo1007 Seungpyo1007 moved this from Todo to In Progress in TechAPI-Project Jun 19, 2026
@TechEngineBot

TechEngineBot commented Jun 19, 2026

Copy link
Copy Markdown
Member

TechEngine change review: PASS

Check Result
python -m app.validate PASS
python integrity_check.py TechAPI/data --strict PASS

Changed data

Category Added Modified Deleted Added verified Added unverified Added Kaggle-sourced
brand 0 0 0 0 0 0
soc 127 0 0 0 127 0
smartphone 2661 0 0 0 2661 0
gpu 0 0 0 0 0 0
cpu 0 0 0 0 0 0

Changed record examples

soc added

  • soc/apple/2023/apple-a16-bionic-apl1w10-t8120.json - Apple A16 Bionic APL1W10 T8120
  • soc/apple/2023/apple-a17-pro-apl1v02-t8130.json - Apple A17 Pro APL1V02 T8130
  • soc/apple/2024/apple-a18-pro-apl1v07-t8140.json - Apple A18 Pro APL1V07 T8140
  • soc/apple/2025/apple-a18-apl1v08-t8142.json - Apple A18 APL1V08 T8142
  • soc/apple/2025/apple-a19-apl1v13-t8150a.json - Apple A19 APL1V13 T8150a
  • soc/apple/2025/apple-a19-pro-apl1v12-t8150.json - Apple A19 Pro APL1V12 T8150
  • soc/apple/2025/apple-a19-pro-l-apl1v12-t8150.json - Apple A19 Pro L APL1V12 T8150
  • soc/arm/2025/xiaomi-xring-o1-v10100b.json - Xiaomi Xring O1 V10100B
  • soc/hisilicon/2021/hisilicon-honor-kirin990e-5g.json - HiSilicon Honor KIRIN990E 5G
  • soc/hisilicon/2022/hisilicon-honor-kirin9000l-5g.json - HiSilicon Honor KIRIN9000L 5G
  • soc/hisilicon/2023/hisilicon-honor-kirin710a.json - HiSilicon Honor KIRIN710A
  • soc/hisilicon/2023/hisilicon-kirin9000s-4g-hi36a0-gfcv120.json - HiSilicon KIRIN9000S 4G Hi36A0 GFCV120
  • soc/hisilicon/2024/hisilicon-kirin9000s1-4g-hi36a0-gfcv120.json - HiSilicon KIRIN9000S1 4G Hi36A0 GFCV120
  • soc/hisilicon/2024/hisilicon-kirin9000sl-4g-hi36a0.json - HiSilicon KIRIN9000SL 4G Hi36A0
  • soc/hisilicon/2025/hisilicon-honor-kirin710.json - HiSilicon Honor KIRIN710
  • ... 112 more

smartphone added

  • smartphone/apple/2023/apple-iphone-15-5g-a3089-dual-sim-td-lte-jp-ca-mx-sa-128gb-apple-iphone-154.json - Apple iPhone 15 5G A3089 Dual SIM TD-LTE JP CA MX SA 128GB (Apple iPhone 15,4)
  • smartphone/apple/2023/apple-iphone-15-5g-a3089-dual-sim-td-lte-jp-ca-mx-sa-256gb-apple-iphone-154.json - Apple iPhone 15 5G A3089 Dual SIM TD-LTE JP CA MX SA 256GB (Apple iPhone 15,4)
  • smartphone/apple/2023/apple-iphone-15-5g-a3089-dual-sim-td-lte-jp-ca-mx-sa-512gb-apple-iphone-154.json - Apple iPhone 15 5G A3089 Dual SIM TD-LTE JP CA MX SA 512GB (Apple iPhone 15,4)
  • smartphone/apple/2023/apple-iphone-15-5g-a3090-global-dual-sim-td-lte-128gb-apple-iphone-154.json - Apple iPhone 15 5G A3090 Global Dual SIM TD-LTE 128GB (Apple iPhone 15,4)
  • smartphone/apple/2023/apple-iphone-15-5g-a3090-global-dual-sim-td-lte-256gb-apple-iphone-154.json - Apple iPhone 15 5G A3090 Global Dual SIM TD-LTE 256GB (Apple iPhone 15,4)
  • smartphone/apple/2023/apple-iphone-15-5g-a3090-global-dual-sim-td-lte-512gb-apple-iphone-154.json - Apple iPhone 15 5G A3090 Global Dual SIM TD-LTE 512GB (Apple iPhone 15,4)
  • smartphone/apple/2023/apple-iphone-15-5g-a3092-dual-sim-td-lte-cn-hk-128gb-apple-iphone-154.json - Apple iPhone 15 5G A3092 Dual SIM TD-LTE CN HK 128GB (Apple iPhone 15,4)
  • smartphone/apple/2023/apple-iphone-15-5g-a3092-dual-sim-td-lte-cn-hk-256gb-apple-iphone-154.json - Apple iPhone 15 5G A3092 Dual SIM TD-LTE CN HK 256GB (Apple iPhone 15,4)
  • smartphone/apple/2023/apple-iphone-15-5g-a3092-dual-sim-td-lte-cn-hk-512gb-apple-iphone-154.json - Apple iPhone 15 5G A3092 Dual SIM TD-LTE CN HK 512GB (Apple iPhone 15,4)
  • smartphone/apple/2023/apple-iphone-15-5g-uw-a2846-dual-sim-td-lte-128gb-apple-iphone-154.json - Apple iPhone 15 5G UW A2846 Dual SIM TD-LTE 128GB (Apple iPhone 15,4)
  • smartphone/apple/2023/apple-iphone-15-5g-uw-a2846-dual-sim-td-lte-256gb-apple-iphone-154.json - Apple iPhone 15 5G UW A2846 Dual SIM TD-LTE 256GB (Apple iPhone 15,4)
  • smartphone/apple/2023/apple-iphone-15-5g-uw-a2846-dual-sim-td-lte-512gb-apple-iphone-154.json - Apple iPhone 15 5G UW A2846 Dual SIM TD-LTE 512GB (Apple iPhone 15,4)
  • smartphone/apple/2023/apple-iphone-15-plus-5g-a3093-dual-sim-td-lte-jp-ca-mx-sa-128gb-apple-iphone-155.json - Apple iPhone 15 Plus 5G A3093 Dual SIM TD-LTE JP CA MX SA 128GB (Apple iPhone 15,5)
  • smartphone/apple/2023/apple-iphone-15-plus-5g-a3093-dual-sim-td-lte-jp-ca-mx-sa-256gb-apple-iphone-155.json - Apple iPhone 15 Plus 5G A3093 Dual SIM TD-LTE JP CA MX SA 256GB (Apple iPhone 15,5)
  • smartphone/apple/2023/apple-iphone-15-plus-5g-a3093-dual-sim-td-lte-jp-ca-mx-sa-512gb-apple-iphone-155.json - Apple iPhone 15 Plus 5G A3093 Dual SIM TD-LTE JP CA MX SA 512GB (Apple iPhone 15,5)
  • ... 2646 more

Heuristic review

  • Added records by manufacturer/brand: samsung: 776, xiaomi: 383, huawei: 297, oppo: 256, motorola: 179, apple: 171, vivo: 124, google: 93

  • Added records by source class: other: 2788

  • Heuristic warnings: 2596 total; showing first 20.

    • smartphone: smartphone/apple/2023/apple-iphone-15-5g-a3089-dual-sim-td-lte-jp-ca-mx-sa-128gb-apple-iphone-154.json: double spaces in name
    • smartphone: smartphone/apple/2023/apple-iphone-15-5g-a3089-dual-sim-td-lte-jp-ca-mx-sa-256gb-apple-iphone-154.json: double spaces in name
    • smartphone: smartphone/apple/2023/apple-iphone-15-5g-a3089-dual-sim-td-lte-jp-ca-mx-sa-512gb-apple-iphone-154.json: double spaces in name
    • smartphone: smartphone/apple/2023/apple-iphone-15-5g-a3090-global-dual-sim-td-lte-128gb-apple-iphone-154.json: double spaces in name
    • smartphone: smartphone/apple/2023/apple-iphone-15-5g-a3090-global-dual-sim-td-lte-256gb-apple-iphone-154.json: double spaces in name
    • smartphone: smartphone/apple/2023/apple-iphone-15-5g-a3090-global-dual-sim-td-lte-512gb-apple-iphone-154.json: double spaces in name
    • smartphone: smartphone/apple/2023/apple-iphone-15-5g-a3092-dual-sim-td-lte-cn-hk-128gb-apple-iphone-154.json: double spaces in name
    • smartphone: smartphone/apple/2023/apple-iphone-15-5g-a3092-dual-sim-td-lte-cn-hk-256gb-apple-iphone-154.json: double spaces in name
    • smartphone: smartphone/apple/2023/apple-iphone-15-5g-a3092-dual-sim-td-lte-cn-hk-512gb-apple-iphone-154.json: double spaces in name
    • smartphone: smartphone/apple/2023/apple-iphone-15-5g-uw-a2846-dual-sim-td-lte-128gb-apple-iphone-154.json: double spaces in name
    • smartphone: smartphone/apple/2023/apple-iphone-15-5g-uw-a2846-dual-sim-td-lte-256gb-apple-iphone-154.json: double spaces in name
    • smartphone: smartphone/apple/2023/apple-iphone-15-5g-uw-a2846-dual-sim-td-lte-512gb-apple-iphone-154.json: double spaces in name
    • smartphone: smartphone/apple/2023/apple-iphone-15-plus-5g-a3093-dual-sim-td-lte-jp-ca-mx-sa-128gb-apple-iphone-155.json: double spaces in name
    • smartphone: smartphone/apple/2023/apple-iphone-15-plus-5g-a3093-dual-sim-td-lte-jp-ca-mx-sa-256gb-apple-iphone-155.json: double spaces in name
    • smartphone: smartphone/apple/2023/apple-iphone-15-plus-5g-a3093-dual-sim-td-lte-jp-ca-mx-sa-512gb-apple-iphone-155.json: double spaces in name
    • smartphone: smartphone/apple/2023/apple-iphone-15-plus-5g-a3094-global-dual-sim-td-lte-128gb-apple-iphone-155.json: double spaces in name
    • smartphone: smartphone/apple/2023/apple-iphone-15-plus-5g-a3094-global-dual-sim-td-lte-256gb-apple-iphone-155.json: double spaces in name
    • smartphone: smartphone/apple/2023/apple-iphone-15-plus-5g-a3094-global-dual-sim-td-lte-512gb-apple-iphone-155.json: double spaces in name
    • smartphone: smartphone/apple/2023/apple-iphone-15-plus-5g-a3096-dual-sim-td-lte-cn-hk-128gb-apple-iphone-155.json: double spaces in name
    • smartphone: smartphone/apple/2023/apple-iphone-15-plus-5g-a3096-dual-sim-td-lte-cn-hk-256gb-apple-iphone-155.json: double spaces in name

@TechEngineBot

TechEngineBot commented Jun 19, 2026

Copy link
Copy Markdown
Member

TechEngine validation stats: PASS

Data summary

Category Total Verified Unverified Missing verified Tracked Verified % of tracked
brand 189 0 60 129 60 0.0%
soc 1687 58 1629 0 1687 3.4%
smartphone 19205 184 19021 0 19205 1.0%
gpu 2030 0 2030 0 2030 0.0%
cpu 3977 976 3001 0 3977 24.5%
all 27088 1218 25741 129 26959 4.5%

Warning

Tracked verified coverage is below 50% for brand 0.0% (0/60), gpu 0.0% (0/2030), smartphone 1.0% (184/19205), soc 3.4% (58/1687), all 4.5% (1218/26959), cpu 24.5% (976/3977).
Tracked coverage excludes records missing the verified field; see the Missing verified column for those records.
This does not fail validation. Keep imported records verified: false until manual audit, but treat this as follow-up verification work before relying on the affected categories as curated data.

Validation notes

  • Full advisory outlier listings are suppressed on successful runs because they are dataset-wide and mostly stable between PRs.
  • Failure runs still include a detailed log excerpt for debugging.

Key output:

## app.validate
## integrity_check.py --strict
loaded CPU=3977 GPU=2030
✅ integrity gate: no hard anomalies.
Integrity section Flagged lines
structural 0
CPU name/tier consistency (desktop mainstream only) 0
CPU single>multi (cinebench/geekbench — should be multi>=single) 0
CPU era-vs-score outliers 8
CPU cross-source ratio outliers (possible wrong-variant) 152
GPU cross-source ratio outliers + sanity 18

@Seungpyo1007 Seungpyo1007 merged commit ffe83b6 into main Jun 19, 2026
4 checks passed
@github-project-automation github-project-automation Bot moved this from In Progress to Done in TechAPI-Project Jun 19, 2026
This was referenced Jun 19, 2026
Merged
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@Seungpyo1007 Seungpyo1007

@TechEngineBot TechEngineBot

Labels

data Dataset changes enhancement New feature or request

Projects

TechAPI-Project
Status: Done

Milestone

Massive dataset rebuild (1989-2026)

Development

Successfully merging this pull request may close these issues.

Massive dataset rebuild: CPU + brand + GPU + smartphone + SoC (1989-2026)

2 participants

@Seungpyo1007 @TechEngineBot