PypeIt’s Core Data Reduction Executable and Workflow
This document describes PypeIt’s fundamental data-reduction executable and summarizes its workflow. Links to more detail on specific algorithms and advice on adjustments that can/should be made to better reduce your data are included throughout.
The main script to run the PypeIt reduction is
run_pypeit and should be
executed from the command-line (once you’ve activated the relevant PypeIt python
environment; see Setup a clean python environment). Here, we briefly describe how to use the
script and some of its options.
run_pypeit, you must have
Inspected and fussed with your setups; see Setup.
Set your current working directory to one of the setup sub-folders (e.g.,
Edited the PypeIt Reduction File in that directory as recommended for a successful reduction.
(optional) Removed any calibration files in the
Calibrations/folder. This is particularly necessary if you’re re-reducing data that was previously reduced by an older PypeIt version. However, when in doubt, it’s good practice to perform a fresh reduction by removing these files instead of attempting to re-use them.
The script usage can be displayed by calling the script with the
$ run_pypeit -h usage: run_pypeit [-h] [-v VERBOSITY] [-r REDUX_PATH] [-m] [-s] [-o] [-c] pypeit_file ## [1;37;42mPypeIt : The Python Spectroscopic Data Reduction Pipeline v1.13.1.dev859+gf736f5f47[0m ## ## Available spectrographs include: ## bok_bc, gemini_flamingos1, gemini_flamingos2, gemini_gmos_north_e2v, ## gemini_gmos_north_ham, gemini_gmos_north_ham_ns, ## gemini_gmos_south_ham, gemini_gnirs_echelle, gemini_gnirs_ifu, ## gtc_maat, gtc_osiris, gtc_osiris_plus, jwst_nircam, jwst_nirspec, ## keck_deimos, keck_esi, keck_hires, keck_kcrm, keck_kcwi, ## keck_lris_blue, keck_lris_blue_orig, keck_lris_red, ## keck_lris_red_mark4, keck_lris_red_orig, keck_mosfire, keck_nires, ## keck_nirspec_low, lbt_luci1, lbt_luci2, lbt_mods1b, lbt_mods1r, ## lbt_mods2b, lbt_mods2r, ldt_deveny, magellan_fire, magellan_fire_long, ## magellan_mage, mdm_modspec, mdm_osmos_mdm4k, mmt_binospec, ## mmt_bluechannel, mmt_mmirs, not_alfosc, not_alfosc_vert, ntt_efosc2, ## p200_dbsp_blue, p200_dbsp_red, p200_tspec, shane_kast_blue, ## shane_kast_red, shane_kast_red_ret, soar_goodman_blue, ## soar_goodman_red, tng_dolores, vlt_fors2, vlt_sinfoni, ## vlt_xshooter_nir, vlt_xshooter_uvb, vlt_xshooter_vis, wht_isis_blue, ## wht_isis_red positional arguments: pypeit_file PypeIt reduction file (must have .pypeit extension) options: -h, --help show this help message and exit -v VERBOSITY, --verbosity VERBOSITY Verbosity level between 0 [none] and 2 [all] -r REDUX_PATH, --redux_path REDUX_PATH Path to directory for the reduction. Only advised for testing -m, --do_not_reuse_calibs Do not load previously generated calibrations, even ones made during the run. -s, --show Show reduction steps via plots (which will block further execution until clicked on) and outputs to ginga. Requires remote control ginga session via "ginga --modules=RC,SlitWavelength &" -o, --overwrite Overwrite any existing files/directories -c, --calib_only Only run on calibrations
A typical run of PypeIt is initiated with a command like:
run_pypeit keck_lris_blue_multi_600_4000_d560.pypeit -o
The code launches, reads the PypeIt Reduction File, initiates a few internals, and then proceeds to generate a lot of messages in the terminal window. We fear that some of those comments are outdated or even misleading. In short, only a PypeIt developer is likely to make too much sense of them.
There are a few standard options that you should consider. These are listed in the usage and we describe them in a bit more detail here with guidance on when to (or not to) use them.
--overwrite command will over-write any existing
files and directories. We recommend this be used the majority of the
time. But if you know you only want to re-reduce a few science frames,
then remove them and run without
--do_not_use_calibs flag tells PypeIt to avoid using any
existing calibration frames instead of loading from disk.
Using this can greatly slow down the code.
This is the main debugging mode of PypeIt. It will generate a lot of plots to the screen. It is probably too overwhelming for most users, i.e. best for developers. Instead, aim to inspect the output Calibrations and QA plots.
This will cause PypeIt to only reduce the calibrations. In fact, if your
PypeIt Reduction File only has calibrations and you execute
this option, an error will be raised that will suggest you run the code in this
A core design principle that allows PypeIt to reduce data from many different Spectrographs is that it isolates code that is spectrograph-specific from code that is generally applicable to any slit-based spectrograph. In this section, we briefly describe the general workflow used for all spectrographs. For caveats and advice specific to a given spectrograph, see Instrument-specific Details.
The most basic workflow used by PypeIt is:
Initialization: Parse the PypeIt Reduction File and consolidate with additional MetaData read from the file headers. Metadata in the PypeIt Reduction File takes precedence over any metadata read from the file headers. Setup the output directory structure.
Reduce Standard Star Frames: If any are included, first reduce any standard star exposures. As part of the
run_pypeitworkflow, the only use of the standard stars is to guide object tracing when the object flux is too faint to be traced directly. Standard stars are used for Fluxing via separate scripts.
Reduce Science Frames: Perform the main reduction procedures required for all the science exposures. Any standard stars associated with the science frames are used as a crutch for object tracing.
Data Reduction Workflow
For each of the two “reduction” steps listed above, the only difference is the crutch used for the object tracing: The standard star frames always use the slit-edge traces as the crutch, whereas the science frames will use the standard star trace as the crutch, if one is available. Whenever the code falls back to a tracing crutch, beware of differential atmospheric refraction!
From the outer-most to the inner-most loop, the data reduction iterates through:
For each detector or detector mosaic, PypeIt:
All frames undergo Basic Image Processing before more frame-specific processing.
For instruments with appropriate slit-mask metadata (e.g., Keck/DEIMOS), PypeIt will then attempt to match the detected objects to the expected locations of the objects from the slit-mask metadata, as well as determine which objects from the slit-mask design were not detected and which detected objects are serendipitous.
Finally, PypeIt loops through all of the detectors or detector mosaics again to perform the spectral Extraction.
PypeIt defines three main algorithms for reducing data, depending on the type of
MultiSlit (used for both long-slit and MOS observations),
IFU. The pipeline used for each spectrograph is provided
Key differences between the three algorithms include:
Object finding and extraction is performed independently for
Echellereductions match object traces found on each order
Calibrations are performed in a slightly different order for
Sky-subtraction and flat-fielding are treated differently for
Beyond this, more fine-grained control over the PypeIt workflow and the parameters used by many of the lower-level algorithms is enabled by User-level Parameters that can be changed via the PypeIt Reduction File.