PypeIt Reduction File
Overview
The PypeIt Reduction File is the critical component to any successful run of PypeIt. It is where you set (1) how PypeIt is executed using its User-level Parameters and (2) what data files to include in the reduction.
The name of the file is expected to end with .pypeit, and it has a specific
format that we discuss below. The PypeIt reduction file should first be
automatically generated using the pypeit_setup script, but it can (and
often must) be edited by the user.
This document provides guidance on modifying the file. You may also wish to refer to Input File Format for additional information on formatting.
You must have a unique PypeIt file for each instrument configuration, or “setup” (modulo detectors), which often includes the name of the slit mask design for relevant instruments. It is possible that you will need to modify the settings for different gratings, etc. It will also enable you to more easily customize the associated calibration files to process.
File Format
The following describes the standard format for a PypeIt Reduction File. Users may also refer to Input File Format for additional details.
Here is an example PypeIt reduction file:
# Auto-generated PypeIt input file using PypeIt version: 1.12.2
# UTC 2023-04-05T22:42:29.971
# User-defined execution parameters
[rdx]
spectrograph = shane_kast_blue
# Setup
setup read
Setup A:
dichroic: d55
dispname: 600/4310
setup end
# Data block
data read
path /Users/westfall/Work/packages/PypeIt-development-suite/RAW_DATA/shane_kast_blue/600_4310_d55
filename | frametype | ra | dec | target | dispname | decker | binning | mjd | airmass | exptime | dichroic | calib
b1.fits.gz | arc,tilt | 140.44166666666663 | 37.43222222222222 | Arcs | 600/4310 | 0.5 arcsec | 1,1 | 57162.06664467593 | 1.0 | 30.0 | d55 | 0
b14.fits.gz | bias | 172.34291666666664 | 36.86833333333333 | Bias | 600/4310 | 2.0 arcsec | 1,1 | 57162.15420034722 | 1.0 | 0.0 | d55 | 0
b15.fits.gz | bias | 172.41833333333332 | 36.94444444444444 | Bias | 600/4310 | 2.0 arcsec | 1,1 | 57162.15440162037 | 1.0 | 0.0 | d55 | 0
b16.fits.gz | bias | 172.49124999999995 | 36.97833333333333 | Bias | 600/4310 | 2.0 arcsec | 1,1 | 57162.154603125 | 1.0 | 0.0 | d55 | 0
b17.fits.gz | bias | 172.5645833333333 | 37.04694444444444 | Bias | 600/4310 | 2.0 arcsec | 1,1 | 57162.15480474537 | 1.0 | 0.0 | d55 | 0
b18.fits.gz | bias | 172.63708333333332 | 37.11555555555556 | Bias | 600/4310 | 2.0 arcsec | 1,1 | 57162.15500949074 | 1.0 | 0.0 | d55 | 0
b19.fits.gz | bias | 172.71166666666664 | 37.18611111111111 | Bias | 600/4310 | 2.0 arcsec | 1,1 | 57162.15521145833 | 1.0 | 0.0 | d55 | 0
b20.fits.gz | bias | 172.78416666666666 | 37.254444444444445 | Bias | 600/4310 | 2.0 arcsec | 1,1 | 57162.15541377315 | 1.0 | 0.0 | d55 | 0
b21.fits.gz | bias | 172.85708333333332 | 37.32361111111111 | Bias | 600/4310 | 2.0 arcsec | 1,1 | 57162.15561504629 | 1.0 | 0.0 | d55 | 0
b22.fits.gz | bias | 172.93 | 37.3925 | Bias | 600/4310 | 2.0 arcsec | 1,1 | 57162.15581597222 | 1.0 | 0.0 | d55 | 0
b23.fits.gz | bias | 173.00166666666667 | 37.4225 | Bias | 600/4310 | 2.0 arcsec | 1,1 | 57162.156018981485 | 1.0 | 0.0 | d55 | 0
b10.fits.gz | pixelflat,illumflat,trace | 144.82041666666666 | 37.43222222222222 | Dome Flat | 600/4310 | 2.0 arcsec | 1,1 | 57162.07859895833 | 1.0 | 15.0 | d55 | 0
b11.fits.gz | pixelflat,illumflat,trace | 144.955 | 37.43222222222222 | Dome Flat | 600/4310 | 2.0 arcsec | 1,1 | 57162.07897476852 | 1.0 | 15.0 | d55 | 0
b12.fits.gz | pixelflat,illumflat,trace | 145.0908333333333 | 37.43222222222222 | Dome Flat | 600/4310 | 2.0 arcsec | 1,1 | 57162.079351388886 | 1.0 | 15.0 | d55 | 0
b13.fits.gz | pixelflat,illumflat,trace | 145.22791666666666 | 37.43222222222222 | Dome Flat | 600/4310 | 2.0 arcsec | 1,1 | 57162.079728240744 | 1.0 | 15.0 | d55 | 0
b3.fits.gz | pixelflat,illumflat,trace | 143.86791666666667 | 37.43222222222222 | Dome Flat | 600/4310 | 2.0 arcsec | 1,1 | 57162.07596400463 | 1.0 | 15.0 | d55 | 0
b4.fits.gz | pixelflat,illumflat,trace | 144.00458333333333 | 37.43222222222222 | Dome Flat | 600/4310 | 2.0 arcsec | 1,1 | 57162.076341782406 | 1.0 | 15.0 | d55 | 0
b5.fits.gz | pixelflat,illumflat,trace | 144.14041666666665 | 37.43222222222222 | Dome Flat | 600/4310 | 2.0 arcsec | 1,1 | 57162.07671956019 | 1.0 | 15.0 | d55 | 0
b6.fits.gz | pixelflat,illumflat,trace | 144.27708333333334 | 37.43222222222222 | Dome Flat | 600/4310 | 2.0 arcsec | 1,1 | 57162.077096064815 | 1.0 | 15.0 | d55 | 0
b7.fits.gz | pixelflat,illumflat,trace | 144.41291666666666 | 37.43222222222222 | Dome Flat | 600/4310 | 2.0 arcsec | 1,1 | 57162.07747175926 | 1.0 | 15.0 | d55 | 0
b8.fits.gz | pixelflat,illumflat,trace | 144.54874999999996 | 37.43222222222222 | Dome Flat | 600/4310 | 2.0 arcsec | 1,1 | 57162.077847569446 | 1.0 | 15.0 | d55 | 0
b9.fits.gz | pixelflat,illumflat,trace | 144.6845833333333 | 37.43222222222222 | Dome Flat | 600/4310 | 2.0 arcsec | 1,1 | 57162.078222916665 | 1.0 | 15.0 | d55 | 0
b27.fits.gz | science | 184.40291666666664 | 39.01111111111111 | J1217p3905 | 600/4310 | 2.0 arcsec | 1,1 | 57162.20663842592 | 1.0 | 1200.0 | d55 | 0
b28.fits.gz | science | 184.40416666666664 | 39.01111111111111 | J1217p3905 | 600/4310 | 2.0 arcsec | 1,1 | 57162.22085034722 | 1.0 | 1200.0 | d55 | 0
b24.fits.gz | standard | 189.47833333333332 | 24.99638888888889 | Feige 66 | 600/4310 | 2.0 arcsec | 1,1 | 57162.17554351852 | 1.039999961853 | 30.0 | d55 | 0
data end
Parameter Block
At the top of the file is the parameter block which allows the user to customize the parameters used to define how the data reduction proceeds. The two lines shown in this example are the only 2 that are required.
See User-level Parameters for how the parameter block should be formatted, how to change parameters, a detailed description of each parameter provided for any PypeIt algorithm, all the changes made to the default parameter values made for each Instrument-Specific Default Configuration, and the precedence given to changes made to the parameters w.r.t. their global defaults. Importantly, note that this provides the alterations that are always made to the default parameters for the instrument in question; i.e., these are not the parameters that you need to include in your PypeIt reduction file. You only need to include parameters in your PypeIt reduction file that you wish to change from the instrument-specific defaults (or the global defaults in the case that the instrument requires no change to the global default).
Suggested changes to the parameters are made throughout our documentation. If you’re having trouble getting the performance you want, first try reading through the documentation for the algorithm your interested in (start with those listed under “Processing Details” in the menu to the left) and then ping the PypeIt Users Slack (make sure you look at the pinned comment in the #guidlines channel). Also see our Reduction Tips.
Setup Block
The next block, beginning with line setup read and
ending with setup end, describes the
instrument configuration. There can only be one setup shown
(e.g., Setup A), and the parameters provided show the salient
metadata for that instrument configuration.
You should not edit any of this;
it is informational and required. See Setup
for further details.
Data Block
Last is the data block, beginning with the line data read
and ending with data end, which
includes the path(s) to the raw data files and a table
describing those files. It is common to edit this table
as described below.
The data block is a | deimited table as written by
the underlying astropy.table.Table object used by
pypeit_setup. The | symbols need not align.
Warning
Users are recommended to always generate the PypeIt reduction file using the pypeit_setup script. However, you will often need to edit it because it is virtually impossible to create an automated procedure that will work in all cases. The PypeIt reduction file is the ultimate authority in terms of how the data is reduced. As such, you should understand how edits to this file work because these edits will override anything derived from the FITS headers!
Most Spectrographs require at least one file with each of the following Frame Types:
arc: Wavelength calibration
trace: Slit/order definition
pixelflat: Flat fielding
science: Science exposure
Warning
The code will not run if your PypeIt Reduction File includes
entries with None. You must remove or modify those entries.
Common edits to the Data Block of the PypeIt file are as follows.
Add/Remove a File
You can add/remove files from the data block.
To add a file, the only safe move is to copy in a line from the
sorted file generated by pypeit_setup; see Setup.
It needs to be formatted just like the others.
To remove a file, you may delete the line or comment it out by
pre-pending a #.
Here is yet another reminder to not include bad calibration
frames in the reduction (i.e., frames that you do not want to use,
frames with incorrectly identified types, or frames that could not be
automatically classified and have a None type). Check them now
and remove them if they are bad.
frametype
The most common edit for a given data file is its Frame Types.
For almost all spectrographs supported by PypeIt, you will need
at least one of these: arc, tilt, pixelflat, trace
and science.
As you can see from the above example, a given file can have multiple frametypes. Simply provide a comma-separated list, without spaces.
Standard star exposures are very frequently mis-labeled as science (and to a lesser extent, vice-versa). So keep an eye out for those.
near-IR
One key difference is that you can and probably should make modifications to enable A-B (or AA-BB or whatever) subtraction. See A-B image differencing for a full discussion.