A rigorous, zero-AI pipeline for mining public ZTF light curves — and an honest field report on why solo variable-star discovery from public survey data is walled in 2026. The one path that isn't walled: O-C timing of known eclipsing binaries.
Only photometry, statistics, periodograms, and catalog cross-matching — no machine learning, no black boxes. Every layer is offline-testable; network I/O is isolated at the edges. This repository is as much a reproducible negative result as it is a toolkit: if you were about to spend a month hunting new variable stars in ZTF by yourself, read the decision log first — it will save you the month.
- Discovering / characterizing NEW variable stars, solo, from public ZTF is walled in every accessible regime. Not a code bug — the walls are physical (see the terrain map below). Every statistically significant candidate we produced was killed by an eyeball / magnitude / independent-instrument check. Nothing false was ever submitted.
- One path hits no wall and works today: O-C timing of known eclipsing binaries. Fold a catalogued EB on 7 years of ZTF, refine its period to ~ppm, and build an Observed−Calculated (O-C) diagram to look for a changing period (mass transfer / a third body). This produces a real, submittable contribution to O-C databases (BAV, JAAVSO).
- Built, adversarially reviewed, 59 tests green, defenses proven on real data.
A bright, isolated EA binary (CSS_J114242) folded on 577 clean ZTF epochs — a textbook double eclipse, catalog period recovered to ~4 ppm, O-C flat/stable over 5 years:
The screening tool then builds a primary-locked O-C diagram for each candidate and ranks by non-flatness. Below, the O-C ranking metric is (deliberately shown being) fooled by a single poorly-sampled segment — an honest illustration of a known limitation, not a real period change (see next steps):
Honest reward note: refining a known EB's ephemeris credits you as a contributor on a catalogued object (and a possible author line in an O-C database) — a real, citable record with your name. It is not "discovering" or "naming" a star (which is not a real thing for any path — see the decision log).
The clean, bright, periodic sky is ~85–95% already catalogued — Gaia DR3 (~9.5M variables), Chen et al. 2020 (~781k from ZTF), and SCoPe (scored ~210M ZTF light curves) swept it. What's left is walled from the other side:
| Regime | Wall |
|---|---|
| bright, uncrowded, any class | ~100% already in VSX |
| faint, short-period | diurnal-alias-dominated + no reachable independent confirmer |
| plane LPVs (dense, uncharacterized) | bright Mira saturates ZTF + blended in ASAS-SN |
| off-plane LPVs | too sparse to be worth scanning |
The full chronology — blind scans → cross-band confirmation → adversarial review → the
alias trap → the LPV pivot → the physical wall — is in docs/DECISION-LOG.md.
The red-team reports that shaped it are in docs/adversarial/.
Python 3.14. Clone, create a virtualenv, install requirements.txt:
git clone <your-fork-url> ztf-variable-finder && cd ztf-variable-finder
python3.14 -m venv .venv
.venv/bin/python -m pip install -r requirements.txtGotcha: always invoke tools in module form —
.venv/bin/python -m pytest,.venv/bin/python -m pip …. The venv's console-script shebangs can go stale after a directory rename. SetPYTHONPATH=srcsoimport ztfvarresolves.
Run the tests:
PYTHONPATH=src .venv/bin/python -m pytest -q # 59 passingScreen known eclipsing binaries for a changing period (the working path):
PYTHONPATH=src .venv/bin/python scripts/screen_eb.py \
--ra 173.0 --dec 25.0 --radius 8 --vmin 13 --vmax 15.5 --type EA --out eb_outIt fetches ZTF per EB, keeps clean-eclipse ones (via PDM), refines the period, builds a primary-locked O-C, ranks by non-flatness, and writes dossiers (phased + O-C plots) for the top candidates. Eyeball each top O-C plot — a genuine smooth slope or parabola is the candidate; a single jumping point is not.
Characterize / discovery scan (walled, kept for reproducibility):
PYTHONPATH=src .venv/bin/python scripts/characterize.py --source gaia-lpv \
--ra 288.0 --dec 10.0 --radius 0.25 --gmin 14 --gmax 17 \
--min-period 20 --max-period 1000 --archive archive_lpv --out lpv_outRuns are resumable (a local archive skips already-fetched data and survives network
timeouts). Add --vsx-mode require-unperiodized to target VSX objects that lack a period.
See docs/DECISION-LOG.md for why these paths are walled.
These defenses are the transferable value of the repo — they apply to any ground-based time-domain survey:
- Cross-band ≠ independent confirmation. ZTF g/r/i share the same nights, so the diurnal
sampling alias is coherent across bands. Only a different instrument (ASAS-SN / ATLAS) or
masking the alias comb can rule it out.
find_period(avoid_aliases=True)masks the solar/sidereal harmonic comb;vetting.is_aliasrejects survivors. - Independent-instrument confirmation (
asassn.py) is 3-state:True/False/None— a failed query never masquerades as agreement. - Calibrated significance — reduced χ² with an error floor, plus a bootstrap FAP, so red noise and underestimated errors don't manufacture detections.
- Injection–recovery (
injection.py) measures the real false-positive rate and completeness on the actual cadence, not an idealized one. - Fail closed —
decide_noveltytreats a failed catalog query as unknown, never NOVEL; masked IPAC nulls are dropped, not read as variability.
| Module | Responsibility |
|---|---|
lightcurve.py |
LightCurve dataclass + clean() (drops bad catflags / nan / zero-error epochs) |
variability.py |
reduced χ² (with error floor), MAD, Stetson J, is_variable |
period.py |
Lomb–Scargle find_period (analytic / bootstrap FAP, avoid_aliases comb mask), phase fold, classify |
oc.py |
the working path — PDM refine_period, oc_points, oc_trend (O-C diagram) |
source.py |
group_by_position (per-band oids → physical source), analyze_source (cross-band confirmation) |
novelty.py |
VizieR / SIMBAD cross-match; decide_novelty (fails closed); query_vsx (returns each match's period) |
vetting.py |
alias / band-consistency / FAP rejection |
asassn.py |
independent ASAS-SN Sky Patrol period confirmation (3-state) |
ephemeris.py |
T0 (epoch of extremum) + amplitude / type → VSX-grade entry |
sample.py |
ZTF fetch — fetch_ztf_lightcurves_fast (resolve oid via TAP → fetch by ID, not the slow POS cone) |
prefilter.py |
ZTF DR24 object-stats via IRSA TAP → download only suspects (~15–30× less data) |
archive.py |
persistent local light-curve cache (never re-download; reprocess offline) |
gaia_targets.py |
Gaia DR3 parent samples (variable / LPV / eclipsing-binary) |
injection.py |
injection–recovery (false-positive rate + completeness on real cadence) |
dossier.py |
per-target result dossier (period + T0 + type + confirmation + plot) |
Scripts (scripts/): screen_eb.py (the working O-C path), characterize.py
(characterization orchestrator), scan_grid.py (blind / prefilter scan), reprocess.py
(re-run detection over the archive, no network), plus validate_known.py,
injection_recovery.py, hunt_field.py, scrutinize.py.
The immediate, high-value fix and the path to a real result:
- Harden the O-C ranking metric.
oc.oc_trend's rawswing(max − min) is fooled by a single outlier segment — exactly the plot shown above. Rank by the quadratic curvature significance and/or a linear-slope significance instead, and sigma-clip single-point segments. A real period change is a smooth slope or parabola, never one jumping point. - Scale the EB screening. Period changes are rare — a small fraction of EBs — so you
must screen hundreds to expect one. Add caching to
screen_eb.py(reusearchive.LightCurveArchive) so large runs are resumable, then sweep bright, isolated EA binaries across several fields. - Package refined ephemerides. Even without a period change, the 7-year ZTF baseline gives ppm-level periods for known EBs — a genuine, submittable O-C-database contribution.
Detailed developer state and gotchas live in HANDOFF.md.
- Deterministic, zero AI/ML. No fetching light curves from pixels — consume calibrated ZTF light curves only.
- Lose no data useful for a discovery. The archive persists everything; ML brokers may be used for triage but never as a filter that discards.
- Any VSX / O-C-database submission is outward-facing and human-only. Eyeball every phased and O-C plot first, and let a human press submit.
Built on public data and open tools: the Zwicky Transient Facility (ZTF) and IRSA,
ASAS-SN Sky Patrol, Gaia DR3, VizieR / SIMBAD, the AAVSO VSX, and
astropy / astroquery / numpy / scipy.
MIT © 2026 psd (fraa.psd@gmail.com)