Cookbook
Here, we provide a quick-start cookbook that should help you run PypeIt on a batch of data. There are alternate ways to run these steps, but non-experts should adhere to the following approach.
Note that:
Essentially all PypeIt reduction steps are done via executing command-line scripts using a terminal window. We provide specific commands below and throughout our documentation.
This cookbook provides minimal detail, but serves as a basic introduction to the primary steps used to reduce your data with PypeIt. We guide you to other parts of our documentation that explain specific functionality in more detail.
If your spectrograph is an echelle, every place you read “slit” think “order”.
We also tend to use “spectrograph” and “instrument” interchangeably.
And “setup” and “configuration”, too.
Specific advice on Spectrographs is provided in their own doc page (although not every supported spectrograph has stand-alone documentation).
Invariably something will be out of date in our doc pages. When you see an egregious example, please holler on GitHub or Slack.
Finally, note that before you keep going, you should have already done the following:
Check that PypeIt can reduce your observations: See the list of supported Spectrographs.
Grab a buff computer: We recommend one with at least 16Gb of RAM, but this limit is very instrument specific. In particular, even 32Gb may not be enough for many-detector instruments (e.g., Keck DEIMOS). Multi-processors are good too, although only a portion of the code runs in parallel (essentially only what, e.g., individual numpy operations will do natively).
Install PypeIt: See Installation.
Ready? Okay, now lets
0. Get organized
While PypeIt can handle one or more nights of data with a mix of gratings, tilts, and masks, you will probably find it easier to isolate one set of files at a time. This includes mask by mask for multi-slit observations. Here is what we recommend:
Place the science + calibrations (see below) in one folder.
Copy bias (and dark) frames into each folder as needed.
We will refer to that folder as
RAWDIR
The raw images can be gzip-compressed, although this means opening files will be slower.
A word on calibration data
As with any DRP, PypeIt will work most smoothly if you have taken data with a good set of calibrations, e.g.
Bias frames (optional)
Flats without saturation
Arcs with most/all of the lamps on, without substantial saturation, and with a set of well separated lines that span the full spectral range of your data
Slitmasks designed without overlapping slits
Sensible detector binning and windowing
Data with poor calibration frames will always be hard to reduce. Please take extra care to insure you are not trying to reduce data with bad calibrations. This is the primary “failure mode” of PypeIt.
And heaven help you if you mixed binning, grating tilt, etc. between your calibrations and science (although this is supported for some instruments, by necessity).
Plan your output directory
PypeIt uses a preset directory structure for its Outputs. Typically, the
directory structure is generated starting from wherever you execute
run_pypeit. It is good practice to silo your raw data in a directory
that is not altered, but only accessed by the data reduction routines. That
is, once you have a RAWDIR, plan to perform the reductions in a separate
directory that we’ll refer to as RDXDIR.
1. Setup PypeIt
See Setup for more detail on setting up PypeIt to reduce your data.
PypeIt requires a configuration file, called the PypeIt Reduction File, that tells
it how to reduce your data. To generate this file, we recommend you first
navigate to your RDXDIR and run pypeit_obslog; e.g.:
cd $RDXDIR
pypeit_obslog keck_deimos -r $RAWDIR
This will provide an automatically generated log of all the files in RAWDIR,
including the metadata that PypeIt automatically ingests and will quickly
indicate if something is amiss (e.g., it can’t determine the frametype).
Then, run pypeit_setup to generate a “sorted” list of files and,
ultimately, the PypeIt Reduction File. By “sorted”, we mean that the data are
organized into unique instrument configuration, and these unique configurations
are assigned a unique alphabetic identifier (e.g., “Setup A”). Once instructed
by the -c command-line option, pypeit_setup will create one
PypeIt Reduction File per (unique) instrument setup/configuration. Each unique
setup contained within a single PypeIt Reduction File requires its own separate
execution of run_pypeit.
Note
Typically, pypeit_setup is executed multiple times; however, to get
the PypeIt Reduction File you need to successfully run run_pypeit, make
sure to include the -c option; e.g.:
pypeit_setup -s keck_deimos -r $RAWDIR -c all
This will, e.g., create a keck_deimos_A folder and write the relevant
keck_deimos_A.pypeit file to that folder.
2. Edit your PypeIt file
For more detail, see PypeIt Reduction File, particularly for tips on checking and editing.
Even after running pypeit_setup to produce a PypeIt Reduction File for each unique instrument setup, you may need to edit the file to correct any mistakes made by the automated routines and add or remove files to include in a given execution of run_pypeit.
To edit the file, enter the relevant sub-folder:
cd keck_deimos_A
and use your favorite text editor to edit the relevant file,
keck_deimos_A.pypeit.
3. Run the reduction
See PypeIt’s Core Data Reduction Executable and Workflow for more detail on executing run_pypeit.
With your inspected and corrected PypeIt Reduction File, you’re ready to execute the basic data reduction script, run_pypeit, e.g.:
run_pypeit keck_deimos_A.pypeit
This script requires no interaction and performs the end-to-end data reduction, from calibrations through to 2D and 1D spectra for each science and standard star frame, according the “instructions” it finds in the PypeIt Reduction File you provide.
Although run_pypeit provides a few command-line options, the PypeIt Reduction File is how you affect the detailed control flow, procedures, and User-level Parameters used by its algorithms.
In the above example, a successful execution of run_pypeit will yield a
set of calibration frames held in the
${RDXDIR}/keck_deimos_A/Calibrations directory, extracted 1D and 2D spectra in
the ${RDXDIR}/keck_deimos_A/Science directory, and a set of quality
assessment plots in the ${RDXDIR}/keck_deimos_A/QA directory.
4. Examine the calibrations
For more detail, see Calibrations.
As the code runs, when a new calibration is generated the default is to write it to disk as a “calibration frame” file. Quality assessment plots for some of these are written to the PypeIt QA folder for inspection. We encourage you to inspect these calibration outputs as they come, both the files themselves and the QA plots.
Here is the order they tend to be created with a separate doc for how to view each, what they should look like, and how to troubleshoot:
Note that only a subset of these files may be made, depending on your spectrograph and the calibration files available.
5. Examine the reduced spectra
Eventually (be patient), the code will start generating 2D and 1D spectra, one per standard and science frame.
To examine the 2D spectral images, see Spec2D Output
To examine the extracted 1D spectra, see Spec1D Output
If you are missing 1D spectra, this means that PypeIt did not find any objects in the corresponding frame; see Object Finding.
See also our Reduction Tips for help tuning parameters related to extraction and sky subtraction for your spectra.
6. Flux-calibrate your data (optional)
PypeIt separates the “basic” data reduction steps performed by run_pypeit from a series of further processing steps that can be performed using a separate set of command-line scripts. One of these is flux-calibration, or Fluxing.
Fluxing is a a two-stage process: (1) generate a sensitivity function and (2) apply it to your spectra.
7. Co-add multiple exposures/datasets (optional)
Another set of further processing steps include coadding data.
To coadd extracted spectra, see Coadd 1D Spectra.
To coadd 2D spectra, see Coadd 2D Spectra.
To coadd 2D spectra into a 3D datacube, see Coadd 3D Spectra.