PypeIt’s Core Data Reduction Executable and Workflow

Overview

This document describes PypeIt’s fundamental data-reduction executable and summarizes its workflow. Links to more detail on specific algorithms and advice on adjustments that can/should be made to better reduce your data are included throughout.

run_pypeit

The main script to run the PypeIt reduction is run_pypeit and should be executed from the command-line (once you’ve activated the relevant PypeIt python environment; see Setup a clean python environment). Here, we briefly describe how to use the script and some of its options.

Before executing run_pypeit, you must have

  1. Inspected and fussed with your setups; see Setup.

  2. Set your current working directory to one of the setup sub-folders (e.g., keck_deimos_A).

  3. Edited the PypeIt Reduction File in that directory as recommended for a successful reduction.

  4. (optional) Removed any calibration files in the Calibrations/ folder. This is particularly necessary if you’re re-reducing data that was previously reduced by an older PypeIt version. However, when in doubt, it’s good practice to perform a fresh reduction by removing these files instead of attempting to re-use them.

usage

The script usage can be displayed by calling the script with the -h option:

$ run_pypeit -h
usage: run_pypeit [-h] [-v VERBOSITY] [-r REDUX_PATH] [-m] [-s] [-o] [-c]
                  pypeit_file

##  PypeIt : The Python Spectroscopic Data Reduction Pipeline v1.16.1.dev635+g9540496b9
##
##  Available spectrographs include:
##   aat_uhrf, bok_bc, gemini_flamingos1, gemini_flamingos2,
##   gemini_gmos_north_e2v, gemini_gmos_north_ham,
##   gemini_gmos_north_ham_ns, gemini_gmos_south_ham, gemini_gnirs_echelle,
##   gemini_gnirs_ifu, gtc_maat, gtc_osiris, gtc_osiris_plus, jwst_nircam,
##   jwst_nirspec, keck_deimos, keck_esi, keck_hires, keck_kcrm, keck_kcwi,
##   keck_lris_blue, keck_lris_blue_orig, keck_lris_red,
##   keck_lris_red_mark4, keck_lris_red_orig, keck_mosfire, keck_nires,
##   keck_nirspec_high, keck_nirspec_high_old, keck_nirspec_low, lbt_luci1,
##   lbt_luci2, lbt_mods1b, lbt_mods1r, lbt_mods2b, lbt_mods2r, ldt_deveny,
##   magellan_fire, magellan_fire_long, magellan_mage, mdm_modspec,
##   mdm_osmos_mdm4k, mdm_osmos_r4k, mmt_binospec, mmt_bluechannel,
##   mmt_mmirs, not_alfosc, not_alfosc_vert, ntt_efosc2, p200_dbsp_blue,
##   p200_dbsp_red, p200_tspec, shane_kast_blue, shane_kast_red,
##   shane_kast_red_ret, soar_goodman_blue, soar_goodman_red, tng_dolores,
##   vlt_fors2, vlt_sinfoni, vlt_xshooter_nir, vlt_xshooter_uvb,
##   vlt_xshooter_vis, wht_isis_blue, wht_isis_red

positional arguments:
  pypeit_file           PypeIt reduction file (must have .pypeit extension)

options:
  -h, --help            show this help message and exit
  -v VERBOSITY, --verbosity VERBOSITY
                        Verbosity level between 0 [none] and 2 [all]
  -r REDUX_PATH, --redux_path REDUX_PATH
                        Path to directory for the reduction. Only advised for
                        testing
  -m, --do_not_reuse_calibs
                        Do not load previously generated calibrations, even ones
                        made during the run.
  -s, --show            Show reduction steps via plots (which will block further
                        execution until clicked on) and outputs to ginga.
                        Requires remote control ginga session via "ginga
                        --modules=RC,SlitWavelength &"
  -o, --overwrite       Overwrite any existing files/directories
  -c, --calib_only      Only run on calibrations

Standard Call

A typical run of PypeIt is initiated with a command like:

run_pypeit keck_lris_blue_multi_600_4000_d560.pypeit -o

The code launches, reads the PypeIt Reduction File, initiates a few internals, and then proceeds to generate a lot of messages in the terminal window. We fear that some of those comments are outdated or even misleading. In short, only a PypeIt developer is likely to make too much sense of them.

Options

There are a few standard options that you should consider. These are listed in the usage and we describe them in a bit more detail here with guidance on when to (or not to) use them.

-o

The -o or --overwrite command will over-write any existing files and directories. We recommend this be used the majority of the time. But if you know you only want to re-reduce a few science frames, then remove them and run without -o.

-m

This -m or --do_not_use_calibs flag tells PypeIt to avoid using any existing calibration frames instead of loading from disk.

Using this can greatly slow down the code.

-s

This is the main debugging mode of PypeIt. It will generate a lot of plots to the screen. It is probably too overwhelming for most users, i.e. best for developers. Instead, aim to inspect the output Calibrations and QA plots.

-c

This will cause PypeIt to only reduce the calibrations. In fact, if your PypeIt Reduction File only has calibrations and you execute run_pypeit without this option, an error will be raised that will suggest you run the code in this mode.

Workflow

A core design principle that allows PypeIt to reduce data from many different Spectrographs is that it isolates code that is spectrograph-specific from code that is generally applicable to any slit-based spectrograph. In this section, we briefly describe the general workflow used for all spectrographs. For caveats and advice specific to a given spectrograph, see Instrument-specific Details.

The most basic workflow used by PypeIt is:

  1. Initialization: Parse the PypeIt Reduction File and consolidate with additional MetaData read from the file headers. Metadata in the PypeIt Reduction File takes precedence over any metadata read from the file headers. Setup the output directory structure.

  2. Reduce Standard Star Frames: If any are included, first reduce any standard star exposures. As part of the run_pypeit workflow, the only use of the standard stars is to guide object tracing when the object flux is too faint to be traced directly. Standard stars are used for Fluxing via separate scripts.

  3. Reduce Science Frames: Perform the main reduction procedures required for all the science exposures. Any standard stars associated with the science frames are used as a crutch for object tracing.

Data Reduction Workflow

For each of the two “reduction” steps listed above, the only difference is the crutch used for the object tracing: The standard star frames always use the slit-edge traces as the crutch, whereas the science frames will use the standard star trace as the crutch, if one is available. Whenever the code falls back to a tracing crutch, beware of differential atmospheric refraction!

From the outer-most to the inner-most loop, the data reduction iterates through:

  1. the set of defined “calibration groups”,

  2. the set of “frame combination” groups (or individual frames if no groups are defined),

  3. and the set of spectrograph detectors or Detector Mosaics.

For each detector or detector mosaic, PypeIt:

  1. processes the relevant calibration frames

  2. performs a first round of estimating a global sky model (see Global) and Object Finding

All frames undergo Basic Image Processing before more frame-specific processing.

For instruments with appropriate slit-mask metadata (e.g., Keck/DEIMOS), PypeIt will then attempt to match the detected objects to the expected locations of the objects from the slit-mask metadata, as well as determine which objects from the slit-mask design were not detected and which detected objects are serendipitous.

Finally, PypeIt loops through all of the detectors or detector mosaics again to perform the spectral Extraction.

Workflow Alterations

PypeIt defines three main algorithms for reducing data, depending on the type of instrument used: MultiSlit (used for both long-slit and MOS observations), Echelle, and IFU. The pipeline used for each spectrograph is provided here.

Key differences between the three algorithms include:

  • Object finding and extraction is performed independently for MultiSlit, but Echelle reductions match object traces found on each order

  • Calibrations are performed in a slightly different order for IFU observations

  • Sky-subtraction and flat-fielding are treated differently for IFU observations

Beyond this, more fine-grained control over the PypeIt workflow and the parameters used by many of the lower-level algorithms is enabled by User-level Parameters that can be changed via the PypeIt Reduction File.