PypeIt Reduction State

What it is

While run_pypeit reduces your data it records a machine-readable state of the reduction to a JSON file in the reduction (configuration) directory, named after the .pypeit file:

<pypeit_root>_state.json      # e.g. shane_kast_blue_A_state.json

The state captures, per calibration group and per detector/mosaic, the status of each calibration step (bias, dark, arc, tiltimg, slits, wv_calib, tilts, scattlight, flats, align). For each step it records:

  • a required flag and a status — one of undone, running, success, complete, or fail;

  • the input_files used, the processed output_file, and any qa_files; and

  • step-specific metrics and per-slit/order detail — e.g. bias mean and standard deviation; slits count and per-slit center; wv_calib and tilts per-slit RMS; and flats corrections (pixel-to-pixel, spatial and spectral illumination), provenance, and per-slit per-correction mean/RMS.

In addition to the calibrations, the state records the science frames (see Science-frame state below): one entry per reduced science/standard exposure and detector, with the status of each science macro-step and the per-object / per-slit detail.

The state is modeled by RunPypeItState (a pydantic model); a tabular summary of the calibrations is available via get_status(), and of the science frames via get_science_status().

Science-frame state

Alongside the calibrations, the state records a list of science-frame entries — one per reduced science or standard exposure and detector/mosaic. The science reduction is tracked as four macro-steps,

process  ->  findobj  ->  skysub  ->  extract

(findobj and skysub are produced together). For each (frame, detector) entry the state records:

  • the objtype (science or standard) and the contributing raw frame(s);

  • the status of each of the four macro-steps (same vocabulary as the calibrations: undone / running / success / fail);

  • the data products — the spec2d and spec1d files — and the object count nobj;

  • per-slit detail (a status from the BADSKYSUB / BADEXTRACT slit bitmask, and the object count on the slit); and

  • per-object detail for each detected object — snr_find (the object-finding S/N), s2n (the extraction S/N), the spatial pixel position, FWHM, the trace sign, and whether it was extracted.

A tabular per-frame summary is available via get_science_status() (columns frame, detector, calib, objtype, the four step statuses, nobj, spec2d, spec1d). The PypeIt Dashboard renders this in its Science view.

How and when it is generated

  • During a run. run_pypeit updates the state as it goes: it marks a step running before building it and success / fail afterward, writing the JSON file at each transition — for both the calibrations and the science macro-steps. So the file is a live, per-step record of the run’s progress. pypeit_run_to_calibstep (rebuilding a single calibration) and pypeit_reduce_by_step (rebuilding a single science step) likewise write the state as they run and refresh it from disk on completion. Because each of these single-purpose runs only populates its own portion of the state, they first merge the existing on-disk state (merge_from_disk()) so a calibration build does not blank the science entries, nor a science build the calibration statuses.

  • Without running (read-only). The pypeit_status script derives the same state without performing any processing — it instantiates PypeIt in calib_only mode, checks what calibrations exist, and prints a status table; it writes a human-readable <pypeit_root>.status.log but, being a read, does not write <pypeit_root>_state.json. The dashboard derives the same way on launch when no state file is present, and additionally reconstructs the science state from the on-disk spec2d / spec1d products (and any Intermediate/ files).

The PypeIt Dashboard reads <pypeit_root>_state.json to render its Status, Calibrations, and Science views; if the file is absent it derives the state the same way pypeit_status does.

Warning

Do not edit the _state.json file by hand. It is generated and overwritten by PypeIt and is meant to be read by tools (the dashboard, pypeit_status), not edited. Hand edits will be lost on the next write and can make the file unreadable. To refresh the state, re-run the reduction with run_pypeit (or pypeit_run_to_calibstep to rebuild a single calibration).

Note

State I/O is deliberately non-essential to the reduction: PypeIt writes the state defensively, so a failure to write or update it is logged as a warning and never aborts a run.

See also