ngps_pipeline

Operational wrapper that drives PypeIt through a full night of NGPS (Palomar Next Generation Spectrograph) data: per-channel pypeit_setup → run_pypeit → pypeit_sensfunc → pypeit_flux_calib → cross-channel trace prediction, merge, and coadd.

This repository is the lean, public companion to the internal NGPS quicklook system at Palomar — analogous to LPipe (Perley 2019) sitting beside the LRIS PypeIt support. The quicklook database and GUI are deliberately not included; this build runs purely from the command line.

Requirements

• Python 3.11+ (uses tomllib).
• The ngps_ugri branch of cfremling/PypeIt (adds the four-channel p200_ngps_{u,g,r,i} spectrograph classes).
• numpy, astropy (pulled in by PypeIt).
• A directory of raw NGPS FITS files for the night you want to reduce.

End-to-end recipe

The complete process to reduce a night of NGPS data on a fresh machine.

1. Install the NGPS-enabled PypeIt and this wrapper

# In a fresh venv / conda env (Python 3.11+):
git clone -b ngps_ugri https://github.com/cfremling/PypeIt.git
pip install -e PypeIt/

git clone https://github.com/cfremling/ngps_pipeline.git
pip install -e ngps_pipeline/

That's it for installation. Verify both are importable:

python -c "import pypeit, ngps_pipeline; print(pypeit.__version__)"
which run_pypeit pypeit_setup pypeit_sensfunc pypeit_flux_calib

2. Configure where raw data and outputs live

Create ~/.config/ngps_pipeline.toml (or point $NGPS_PIPELINE_CONFIG at a TOML file anywhere):

# Absolute path — this is the root of all per-night reductions.
work_dir = "/Users/you/ngps_data/work"

# Not used by this public build (the GUI db lives elsewhere), but the
# loader still resolves the key. Leave it pointing somewhere harmless.
db_path  = "/Users/you/ngps_data/ngps_db.sqlite"

[palomar]
longitude_deg = -116.8639
latitude_deg  = 33.3563
altitude_m    = 1712.0

If you skip this step, the loader falls back to ~/ngps_data/work / ~/ngps_data/ngps_db.sqlite with the same Palomar coordinates.

3. Drop your raw FITS files into the per-date raw/ directory

mkdir -p /Users/you/ngps_data/work/20260501/raw
# Copy or rsync the night's raw frames in:
rsync -av observer@ngps:/data/20260501/ngps_260501_*.fits \
          /Users/you/ngps_data/work/20260501/raw/

The wrapper expects exactly this layout:

<work_dir>/
└── 20260501/
    └── raw/
        ├── ngps_260501_0001.fits
        ├── ngps_260501_0002.fits
        └── ...

A typical night is ~80 frames, ~1 GB. NGPS raw files have all four channels in image extensions (U, G, R, I) inside a single FITS container — don't pre-split them.

4. Run the night

python -m ngps_pipeline.reduce 20260501

That single command does the full pipeline for all four channels:

1. pypeit_setup per channel (auto-picks the dominant Setup letter, tolerating channels with missing image extensions).
2. run_pypeit per channel (parallelized across u/g/r/i — one subprocess per channel, threads share the GIL-free subprocess work).
3. pypeit_sensfunc from the auto-detected standard star (Feige 34, Feige 110, BD+28/25/33, G191B2B, HZ44).
4. pypeit_flux_calib applied to every science spec1d in each channel.
5. ngps_pipeline.finalize:
   • Central-slit trace selection (closest to the std's SPAT, with an atmospheric-dispersion fallback predicted by trace_predict.py).
   • Per-channel coadd (inverse-variance weighted).
   • Per-channel empirical telluric correction (std / CALSPEC Beer-Lambert on the airmass ratio).
   • Inverse-variance weighted multi-channel merge.

The final per-target merged 1D spectrum lands in <work_dir>/20260501/merged/<TARGET>.fits. Per-channel intermediate coadds (one per channel × target) live in <work_dir>/20260501/merged_per_channel/.

Useful flags

• --channels r,g — reduce only a subset.
• --force — re-run every step, ignoring existing outputs.
• --skip-setup / --skip-run / --skip-flux / --skip-coadd — resume partway through.
• --coadd-mode {1d,both} — 1d (default) is per-frame OPT + 1D coadd; both also runs pypeit_coadd_2dspec (currently has a known QA-dir race condition between channels, hence the default).
• --skip-db-import — silences the harmless [db] import failed: line that this public build prints at the end (the internal quicklook DB module isn't shipped here).

See CAPABILITIES.md for a detailed description of every step.

Quick reference

Step	Command
Install once	pip install -e PypeIt/ -e ngps_pipeline/
Configure once	edit ~/.config/ngps_pipeline.toml
Per night (place raw)	rsync into <work_dir>/<DATE>/raw/
Per night (reduce)	python -m ngps_pipeline.reduce <DATE>

Status

Pre-1.0, expect breaking changes. Targets the NGPS commissioning data taken during the 2024–2026 commissioning campaign.

License

BSD-3-Clause, matching PypeIt.