PypeIt Conventions

Overview

This document briefly summarizes a set of conventions (generally) adopted by PypeIt.

Image Orientation

One of the first things PypeIt does during Basic Image Processing is re-orient the detector images so that they are consistent across all Spectrographs. This is important to the generality and simplicity of the low-level code base.

The PypeIt convention is to orient images such that spectra run along the first axis of an image — from blue wavelengths at small pixel coordinates to red wavelengths at large pixel coordinates — and the spatial or cross-dispersion direction runs along the second axis — with echelle orders running from the highest order at small pixel coordinates to the lowest order at large pixel coordinates. That is, the shape of the images is always (roughly) the number of spectral pixels by the number of spatial pixels, often referred to in our documentation as (nspec,nspat).

Regardless of the native orientation of the images, all images produced by PypeIt will have this orientation.

On-chip Identifiers

Generally, PypeIt constructs identifiers using combinations of the spatial pixel number, spectral pixel number, and detector or mosaic number.

For example, slits in multi-slit reductions are identified by their spatial position at a reference spectral position, rounded to the nearest pixel number; e.g, slit_id = 241 means that the center of the slit is at spatial pixel 241.

Object identifiers in multislit data are named after the spatial pixel nearest their center, the slit identifier, and the detector; e.g., SPAT0242-SLIT0241-DET07.

Datamodels

Nearly all of PypeIt’s main output files are written by python objects that subclass from DataContainer. This class implements an interface that imposes strict adherence to a fixed datamodel (e.g., datamodel). The class also provides generalized I/O routines that parse the components of the datamodel into a series of FITS header entries and HDUs. The components of a datamodel can be nearly any object type, with general rules for how each object type is parsed into an output file; e.g., single value types (like integers) are generally written to a FITS header, 2D numpy.ndarray objects are written as an astropy.io.fits.ImageHDU, and astropy.table.Table objects are written as an astropy.io.fits.BinTableHDU.

When documenting the datamodels of our output files, it is important to keep a few things in mind:

  • Not every component of the datamodel needs to be defined. Anything that is not defined – either because it is not requested by the user or not available for a given instrument – will not be included in the output file. For example, the documentation for the datamodel provided by the Spec1D Output includes everything that could be in the file; however, none of the OPT_* columns will be available if optimal extraction was not performed.

  • Data models are version-controlled. This means that output files produced by one version of PypeIt may not be usable with a different version of PypeIt if the datamodel was updated in the meantime. If you get an obscure error when trying to load a file that might have been produced with a previous version of PypeIt, the first thing to try is to perform a completely fresh reduction using the new version of the code; either perform the reduction in a new directory or remove all the old output files.

  • Largely by automating the process, we do our best to keep the documentation of datamodels up-to-date. However, this automation is not always straight-forward. If you see something in the datamodel documentation that doesn’t make sense, make sure you’re reading the correct version of the documentation and then please Submit an issue.

Calibration Frames

All reduced calibration frames are held in the Calibrations/ directory and given the named after their calibration type (e.g., Arc). See Directory Structure and Calibrations.

The calibration frames are also assigned a unique identifier that is a combination of their setup, calibration group, and detector (e.g., A_1_DET01). For details on the construction of this identifier, see Calibration Frame Naming.

Science Frames

All reduced science frames are held in the Science directory and have file names that follow the format {prefix}_{fileroot}-{target}_{camera}_{date}.*:

  • prefix provides the type of output file and is either spec1d or spec2d.

  • fileroot is the file name (or first file in the relevant combination group) stripped of the file suffixes (e.g., DE.20100913.22358 for file DE.20100913.22358.fits.gz).

  • target is the name of the observed target pulled from the file header with any spaces are removed

  • camera is the name of the spectrograph camera; see Spectrographs.

  • date is the UT date of the observation pulled from the file header

An example file produced by reducing a Keck/DEIMOS frame has the file name spec1d_DE.20100913.22358-CFHQS1_DEIMOS_20100913T061231.334.fits.

Date and Time

By including the UT observing date and time in the science frame file names, we ensure the uniqueness of the file name. The UT date + time are drawn from the header and refer to the start of the observation, if there are multiple time stamps. Other DRPs (e.g. LowRedux) have tended to use the frame number as the unique identifier. We have broken with that tradition: (1) to better follow archival naming conventions; (2) because of concerns that some facilities may not include the frame number in the header; (3) some users may intentionally or accidentally generate multiple raw files with the same frame number.

The adopted format is {year}{month}{day}T{hour}{min}{sec} where the seconds are allowed to have multiple decimal places; for example, 20100913T061231.334.