Outputs

PypeIt, despite being a pipeline for data reduction, is capable of generating an inordinate amount of data products. These pages document the various data products. Note that conventions used for the file names are discussed here.

Directory Structure

Assuming it was executed from within the directory created by pypeit_setup (e.g., ${RDXDIR}/keck_deimos_A), by default run_pypeit will produce the following directories:

${RDXDIR}/${PYP_SPEC}_${SETUP}/Calibrations: Directory with all the calibration frames.

${RDXDIR}/${PYP_SPEC}_${SETUP}/Science: Directory with all the reduced science and standard frames

${RDXDIR}/${PYP_SPEC}_${SETUP}/QA: Directory with all the quality assessment output

where $PYP_SPEC is the PypeIt name for the spectrograph used to obtain the data (e.g., keck_deimos) and $SETUP is the instrument setup identifier (e.g., A). When referencing output files, we refer to this default directory structure throughout this documentation.

Warning

PypeIt provides options and User-level Parameters that allow you to change the default output directory structure. BEWARE that these options are not well tested. For now, we strongly recommend you use PypeIt’s default output directory structure.

Core Processing Output Files

The primary output files from PypeIt’s core data processing steps are a set of calibrations, calibrated 2D spectral images, and 1D spectral extractions. The following links provide more information about these primary output files, how they’re produced, and their current datamodel.

Note

Nearly all of PypeIt’s main output files are written by python objects that subclass from DataContainer; see datamodel, as well as important details here or (repeated) below.

Further Processing Output Files

Output files from PypeIt’s Further Processing Scripts are discussed in their associated documentation pages. Here are some quick links to their descriptions:

Importantly note that:

execution of pypeit_flux_calib makes direct changes to the Spec1D Output

the output of pypeit_collate_1d is identical to pypeit_coadd_1dspec script

Generally, the further processing scripts that produce 1D spectra that do not make direct changes to the Spec1D Output, produce OneSpec files. See:

Common Output Components

Datamodels

Nearly all of PypeIt’s main output files are written by python objects that subclass from DataContainer. This class implements an interface that imposes strict adherence to a fixed datamodel (e.g., datamodel). The class also provides generalized I/O routines that parse the components of the datamodel into a series of FITS header entries and HDUs. The components of a datamodel can be nearly any object type, with general rules for how each object type is parsed into an output file; e.g., single value types (like integers) are generally written to a FITS header, 2D numpy.ndarray objects are written as an astropy.io.fits.ImageHDU, and astropy.table.Table objects are written as an astropy.io.fits.BinTableHDU.

When documenting the datamodels of our output files, it is important to keep a few things in mind:

Not every component of the datamodel needs to be defined. Anything that is not defined – either because it is not requested by the user or not available for a given instrument – will not be included in the output file. For example, the documentation for the datamodel provided by the Spec1D Output includes everything that could be in the file; however, none of the OPT_* columns will be available if optimal extraction was not performed.

Data models are version-controlled. This means that output files produced by one version of PypeIt may not be usable with a different version of PypeIt if the datamodel was updated in the meantime. If you get an obscure error when trying to load a file that might have been produced with a previous version of PypeIt, the first thing to try is to perform a completely fresh reduction using the new version of the code; either perform the reduction in a new directory or remove all the old output files.

Largely by automating the process, we do our best to keep the documentation of datamodels up-to-date. However, this automation is not always straight-forward. If you see something in the datamodel documentation that doesn’t make sense, make sure you’re reading the correct version of the documentation and then please Submit an issue.

Bitmasks

Bitmasks are used heavily throughout PypeIt, both to communicate to the end-user which quantities should and should not be trusted, but also as a book-keeping method for tracking why pixels were flagged. See Output Bitmasks for a description of the bitmasks provided by PypeIt output files (both calibrations and reduced spectra).

History

PypeIt logs a history of the processing steps used to create a given file in the fits header; see History Keyword.