Quick-Look Reductions
Overview
PypeIt provides a quick-look (QL) script pypeit_ql that executes PypeIt in a
mode that should produce results more quickly but with potentially lower
fidelity. These results should be used for quick inspection only; e.g., for
real-time decision-making while at the telescope.
The primary way this is currently achieved is by:
Enabling re-use of existing calibrations and/or facilitating the creation of calibrations that can be repeatedly applied to the data without reprocessing.
Setting parameters in the reduction that enable a more quick-and-dirty reduction; e.g., only boxcar extraction is performed.
Imposing a more-limited set of use cases, but automating effectively all of the “management” procedures (setting up directories, writing pypeit files, optimizing parameters) that are usually performed by the user when reducing data in those cases.
Particularly because of the latter, the quick-look script follows a specific Directory Structure and makes assumptions about which calibrations can be used that are more lenient than recommended for a robust reduction.
Importantly, pypeit_ql can only be used to reduce data for a single science
target in a single instrument configuration. All the science frames provided
will be combined. Standard star frames can be included and, as long as they are
automatically identified as standards, they will be reduced separately from the
science target. For instruments with dither patterns that PypeIt can parse,
images will be grouped by dither offset position. These dithers can then be
optionally combined using PypeIt’s 2D coadding script.
Here, we describe the algorithm and provide specific usage tutorials. The
script usage can be displayed by calling the script with the -h option:
$ pypeit_ql -h
usage: pypeit_ql [-h] [--raw_files RAW_FILES [RAW_FILES ...]]
[--raw_path RAW_PATH] [--sci_files SCI_FILES [SCI_FILES ...]]
[--redux_path REDUX_PATH] [--parent_calib_dir PARENT_CALIB_DIR]
[--setup_calib_dir SETUP_CALIB_DIR] [--clear_science]
[--calibs_only] [--overwrite_calibs] [--det DET [DET ...]]
[--slitspatnum SLITSPATNUM] [--maskID MASKID]
[--boxcar_radius BOXCAR_RADIUS] [--snr_thresh SNR_THRESH]
[--ignore_std] [--skip_display] [--coadd2d]
[--only_slits ONLY_SLITS [ONLY_SLITS ...]] [--offsets OFFSETS]
[--weights WEIGHTS] [--spec_samp_fact SPEC_SAMP_FACT]
[--spat_samp_fact SPAT_SAMP_FACT] [--try_old]
spectrograph
Script to produce quick-look PypeIt reductions
positional arguments:
spectrograph A valid spectrograph identifier: bok_bc,
gemini_flamingos1, gemini_flamingos2,
gemini_gmos_north_e2v, gemini_gmos_north_ham,
gemini_gmos_north_ham_ns, gemini_gmos_south_ham,
gemini_gnirs_echelle, gemini_gnirs_ifu, gtc_maat,
gtc_osiris, gtc_osiris_plus, jwst_nircam, jwst_nirspec,
keck_deimos, keck_esi, keck_hires, keck_kcrm, keck_kcwi,
keck_lris_blue, keck_lris_blue_orig, keck_lris_red,
keck_lris_red_mark4, keck_lris_red_orig, keck_mosfire,
keck_nires, keck_nirspec_low, lbt_luci1, lbt_luci2,
lbt_mods1b, lbt_mods1r, lbt_mods2b, lbt_mods2r,
ldt_deveny, magellan_fire, magellan_fire_long,
magellan_mage, mdm_modspec, mdm_osmos_mdm4k,
mdm_osmos_r4k, mmt_binospec, mmt_bluechannel, mmt_mmirs,
not_alfosc, not_alfosc_vert, ntt_efosc2, p200_dbsp_blue,
p200_dbsp_red, p200_tspec, shane_kast_blue,
shane_kast_red, shane_kast_red_ret, soar_goodman_blue,
soar_goodman_red, tng_dolores, vlt_fors2, vlt_sinfoni,
vlt_xshooter_nir, vlt_xshooter_uvb, vlt_xshooter_vis,
wht_isis_blue, wht_isis_red
options:
-h, --help show this help message and exit
--raw_files RAW_FILES [RAW_FILES ...]
Either a PypeIt-formatted input file with the list of
raw images to process and the relevant path, or a space-
separated list of the filenames (e.g., "img1.fits
img2.fits"). For the latter entry mode, the path
containing the files is set using --raw_path. (default:
None)
--raw_path RAW_PATH Directory with the raw files to process. Ignored if a
PypeIt-formatted file is provided using the --rawfiles
option. (default: current working directory)
--sci_files SCI_FILES [SCI_FILES ...]
A space-separated list of raw file names that are
science exposures. These files must *also* be in the
list of raw files. Use of this option overrides the
automated PypeIt frame typing. Should only be used of
automatic frame typing fails or is undesirable.
(default: None)
--redux_path REDUX_PATH
Path for the QL reduction outputs. (default: current
working directory)
--parent_calib_dir PARENT_CALIB_DIR
Directory with/for calibrations for *all* instrument
configurations/setups. If provided, the data for your
instrument configuration will be placed or pulled from a
relevant sub-directory. If None, the redux_path is used.
(default: None)
--setup_calib_dir SETUP_CALIB_DIR
Directory with/for calibrations specific to your
instrument configuration/setup. Use of this option
circumvents the automated naming system for the
configuration/setup sub-directories. If None, the code
will try to find relevant calibrations in the
parent_calib_dir. If no calibrations exist in that
directory that match the instrument setup/configuration
of the provided data, the code will construct new
calibrations (assuming relevant raw files are provided).
(default: None)
--clear_science Remove the existing output science directories to force
a fresh reduction. If False, any existing directory
structure will remain, and any alterations to existing
science files will follow the normal behavior of
run_pypeit. (default: False)
--calibs_only Reduce only the calibrations? (default: False)
--overwrite_calibs Re-process and overwrite any existing calibration files.
(default: False)
--det DET [DET ...] A space-separated set of detectors or detector mosaics
to reduce. By default, *all* detectors or default
mosaics for this instrument will be reduced. Detectors
in a mosaic must be a mosaic "allowed" by PypeIt and
should be provided as comma-separated integers (with no
spaces). For example, to separately reduce detectors 1
and 5 for Keck/DEIMOS, you would use --det 1 5; to
reduce mosaics made up of detectors 1,5 and 3,7, you
would use --det 1,5 3,7 (default: None)
--slitspatnum SLITSPATNUM
Reduce the slit(s) as specified by the slitspatnum
value(s) (default: None)
--maskID MASKID Reduce the slit(s) as specified by the maskID value(s)
(default: None)
--boxcar_radius BOXCAR_RADIUS
Set the radius for the boxcar extraction in arcseconds
(default: None)
--snr_thresh SNR_THRESH
Change the default S/N threshold used during source
detection (default: None)
--ignore_std If standard star observations are automatically
detected, ignore those frames. Otherwise, they are
included with the reduction of the science frames.
(default: False)
--skip_display Run the quicklook without displaying any results.
(default: True)
--coadd2d Perform default 2D coadding. (default: False)
--only_slits ONLY_SLITS [ONLY_SLITS ...]
If coadding, only coadd this space-separated set of
slits. If not provided, all slits are coadded. (default:
None)
--offsets OFFSETS If coadding, spatial offsets to apply to each image; see
the [coadd2d][offsets] parameter. Options are restricted
here to either maskdef_offsets or auto. If not
specified, the (spectrograph-specific) default is used.
(default: None)
--weights WEIGHTS If coadding, weights used to coadd images; see the
[coadd2d][weights] parameter. Options are restricted
here to either uniform or auto. If not specified, the
(spectrograph-specific) default is used. (default: None)
--spec_samp_fact SPEC_SAMP_FACT
If coadding, adjust the wavelength grid sampling by this
factor. For a finer grid, set value to <1.0; for coarser
sampling, set value to >1.0). (default: 1.0)
--spat_samp_fact SPAT_SAMP_FACT
If coadding, adjust the spatial grid sampling by this
factor. For a finer grid, set value to <1.0; for coarser
sampling, set value to >1.0). (default: 1.0)
--try_old Attempt to load old datamodel versions. A crash may
ensue.. (default: False)
At present, only a few spectrographs have been extensively tested:
keck_deimos, keck_lris_red, keck_nires, shane_kast_blue, and
shane_kast_red.
Requirements
To run pypeit_ql, you need to provide a minimal set of calibrations, either
as new raw frames taken during your run or via an applicable set of calibrations
taken from previous runs (e.g., for long-slit and/or fixed-format echelle
observations).
If you provide only a set of calibrations, pypeit_ql will process them.
If you also provide science exposures, they will be reduced using the provided
or linked calibrations.
The script effectively requires that PypeIt is able to correctly determine the
type of each input frame, without input from the user. If this fails, so too
will the script. The only exception to this is that you can specify which
frames are science frames, using the --sci_files argument. Importantly,
however, files listed using the --sci_files option must also be listed among
the raw files (see below).
Specifying the input raw files
The script provides a few ways that you can specify the files to reduce:
Provide a file with a specific format that lists the files to be reduced. The format must follow the standard PypeIt file Data Block; however, only the
filenamecolumn is required/used.Provide the directory and list of files directly on the command line.
Provide only the raw directory, which will be used to search for any
*.fits*files and reduce all files found.
An example file named input.rawfiles used in the first approach could look
like this:
# Data block
raw read
path /path/to/files
filename
b1.fits.gz
b10.fits.gz
b27.fits.gz
raw end
and you would pass it to the QL script using the --raw_files command-line argument:
pypeit_ql shane_kast_blue --raw_files input.rawfiles
You will get identical behavior if you instead used
pypeit_ql shane_kast_blue --raw_files b1.fits.gz b10.fits.gz b27.fits.gz --raw_path /path/to/files
Finally, if those three files are the only files with the relevant extension
in /path/to/files, the third entry option would look like this:
pypeit_ql shane_kast_blue --raw_path /path/to/files
In this example (see more below), the three files are an arc-lamp exposure
(b1.fits.gz), a dome-flat exposure (b10.fits.gz), and an on-sky science
exposure (b27.fits.gz). PypeIt is generally able to identify science frames
from the Shane/Kast spectrograph; however, you could specify the science frame
in the above example like so:
pypeit_ql shane_kast_blue --raw_files b1.fits.gz b10.fits.gz b27.fits.gz --raw_path /path/to/files --sci_files b27.fits.gz
Directory Structure
As with typical executions of run_pypeit, pypeit_ql yields
directories with calibrations, quality-assessment plots, and science products.
The difference is that pypeit_ql keeps the calibrations and science products
separate.
For example, executing:
pypeit_ql shane_kast_blue --raw_files b1.fits.gz b10.fits.gz b27.fits.gz --raw_path /path/to/files
will yield two directories where you executed the call: b27/ and
shane_kast_blue_A/. Both directories will look very similar to a normal
execution of run_pypeit (see Directory Structure), except the latter will
only contain calibrations (and related QA plots) and the former will only
contain the science results (and related QA plots). The name of the directory
with the reduction for the science frames is based on the name of the frame,
b27 in this example. The name of directory with the calibrations is always
the combination of the instrument name and setup/configuration identifier (e.g.
shane_kast_blue_A), just as produced by pypeit_setup. A symlink to
the shane_kast_blue_A/Calibrations/ directory is included in b27/ such
that the directory mimics the normal output from run_pypeit.
If multiple science frames are provided, the name of the output directory combines the names of the first and last science frames in the stack. For example:
pypeit_ql shane_kast_blue --raw_files b1.fits.gz b10.fits.gz b27.fits.gz b28.fits.gz b29.fits.gz --raw_path /path/to/files
would produce a b27-b29 directory (assuming b27.fits.gz,
b28.fits.gz, and b29.fits.gz are all science frames) with the results
produced by stacking all 3 science frames. For Shane/Kast, images are stacked
via simple coaddition of the frames, not a “2D coadd” that includes spatial and
spectral registration of the slit images.
Use of existing calibrations
None of the examples above have provided a path with/for the processed calibration files. This means the code uses the current working directory as the “parent” calibration directory.
The “parent” calibration directory potentially contains calibrations for many
different instrument setups/configurations, each in their own directory that
follows the PypeIt naming scheme; e.g., shane_kast_blue_A/,
shane_kast_blue_B/, etc. After pypeit_ql parses the input files, the
script compares the setup/configuration of the science frames to the available
calibrations. If any exist, they will be used and any provided calibration
frames will be ignored unless you set the --overwrite_calibs option.
In terms of a workflow, this means that you might first run:
pypeit_ql shane_kast_blue --raw_files b1.fits.gz b10.fits.gz b27.fits.gz --raw_path /path/to/files
and each subsequent call can omit the calibration files, like so:
pypeit_ql shane_kast_blue --raw_files b28.fits.gz --raw_path /path/to/files
pypeit_ql shane_kast_blue --raw_files b27.fits.gz b28.fits.gz --raw_path /path/to/files
pypeit_ql shane_kast_blue --raw_files b27.fits.gz b28.fits.gz b29.fits.gz --raw_path /path/to/files
...
assuming the instrument setup/configuration has not changed. This forces
pypeit_ql to match the instrument setup to the available calibrations;
however, you can force the code to use a specific set of calibrations by
specifying it directly, like so:
pypeit_ql shane_kast_blue --raw_files b28.fits.gz --raw_path /path/to/files --setup_calib_dir ./shane_kast_blue_A/Calibrations
Alternatively, calibrations that maintain long-term stability (or at least
stable enough for quick-look) can be stored in a parent directory and you can
use them with pypeit_ql without needed to provide any raw calibrations
frames. To do so, call:
pypeit_ql shane_kast_blue --raw_files b27.fits.gz --raw_path /path/to/files --parent_calib_dir /path/to/calibration/archive
If no appropriate calibrations are found, the code will fault.
Algorithm
The quick-look script proceeds as follows:
Parse the list of files to be processed
Extract metadata from the file headers, just as done by pypeit_setup. If any frames provided cannot be typed, the code will fault during this “setup phase.”
Isolate all of the science frames. Standard frames will also be isolated, unless the user selects to ignore standards using
--ignore_std.All science+standard frames must come from the same instrument setup/configuration.
All science images (and separately all standard frames) either set to the same combination group (meaning they will be stacked) or, for some instruments, parsed into dither sequences. Separate dither sequences are combined; i.e., all “A” frames with a given offset are combined.
All science+standard images are forced into the same calibration group, if not automatically done.
If a “parent” calibration directory is provided (using the
--parent_calib_diroption), the script matches the instrument setup/configuration to the available calibrations. If none are found, the code will fault; otherwise, this matching is used to set the calibrations directory (i.e., the same thing that can be done manually using the--setup_calib_diroption).
If the calibration directory is not yet defined (either because it wasn’t set directly or no parent directory was set), the code will first try to find any matching calibrations in the current working directory.
If matches are found, the calibrations will either be used as they are or, if the user sets the
--overwrite_calibsflag, reprocessed using the provided files.If matches are not found, new calibrations are processed using the provided files, and the setup/configuration identifier is forced to be consistent with any existing calibrations in the directory. For example, if
shane_kast_blue_Aexists in your current working directory but the new files you are trying to process do not match the setup for those calibrations, a new directory calledshane_kast_blue_B/will be created with the newly processed calibrations.
Any new calibrations are processed using the pypeit file written to the new calibrations directory (e.g.,
shane_kast_blue_B). Do not delete these pypeit files! They are what’s used to match the instrument setup/configuration to new frames to be processed. Note that adjustments can be made directly to that file and the calibrations can be re-processed using a normal execution of run_pypeit.The quick-look script can be run by only providing calibrations frames. The code will end here if that is the case.
If science (and/or standard) frames are provided, the code creates the output directory and writes a pypeit file used to process the data. Similar to the calibrations, this file can be edited directly and the data can be reprocessed using a normal execution of run_pypeit.
If the user selects to coadd 2D frames (e.g., coadding different dither positions) using the
--coaddtag, the code will executepypeit_setup_coadd2dusing the written pypeit file and then execute the 2D coadd.
Processing Details
The details of the pypeit_ql script given above is essentially just
management of a provided set of files to construct a directory structure and a
set of pypeit files. Beyond that, it executes run_pypeit exactly as one
would from the command line. The only difference is that it sets specific
parameters that execute PypeIt in a mode that should proceed more quickly.
For example, cosmic rays are not identified/rejected in the science image and only boxcar extraction is performed.
Longslit
The default quicklook mode for longslit reductions
is to use the boxcar extraction method, skip
bias image subtraction (maintain overscan), and skip
CR rejection.
Standard call
Here is a sample call for a standard
longslit run on files from the shane_kast_blue
Development suite. In the example, the files
b1.fits.gz, b10.fits.gz, and b27.fits.gz are an
arc, flat, and science frame. But their ordering
is not important.
pypeit_ql shane_kast_blue --raw_files b1.fits.gz b10.fits.gz b27.fits.gz --raw_path /path/to/files
This call first generates a shane_kast_blue_A folder with the processed
calibrations (assuming there are no other calibrations in the current working
directory) and associated QA Outputs. It then generates a separate
folder named b27, which holds the Science folder with the processed 2D
spectral image and the extracted spectra.
Other Options
Here are a few more options
–box_radius
Over-ride the default boxcar extraction radius with
--box_radius. The value is given in arcseconds.
This will over-ride the defaut as described
in ExtractionPar Keywords.
–det
It will greatly speed things up to isolate
the detector(s) of interest. Use --det
with the same syntax as the parameter detnum,
as described in ReduxPar Keywords.
Here is an example with the (old) keck_lris_red
detector:
pypeit_ql keck_lris_red --raw_path /path/to/files --raw_files LR.20160216.40478.fits.gz --setup_calib_dir /path/to/parent/calib/dir/keck_lris_red/long_600_7500_d560/Calibrations --det 2
This will only process the second detector.
Multislit
Here are some options specific to multi-slit observations.
Isolating a slit
In general, reducing all of the slits from a multi-slit observation will not be quick. Therefore, you may wish to isolate a single slit.
This can be done in two ways.
–slitspatnum
Specify the detector and spatial position of the slit you wish to reduce.
Here is an example with keck_deimos:
pypeit_ql keck_deimos --raw_path /path/to/files --raw_files d1010_0056.fits.gz --setup_calib_dir /path/to/parent/calib/dir/keck_deimos/600ZD_M_6500/Calibrations --slitspatnum MSC02:452
Here we have specified --slitspatnum as
MSC02:452, which means use the 2nd mosaic
and the slit closest to position 452.
This requires that the detector(s) with this
slit have been calibrated (or will be calibrated, e.g. by
specfiying --det).
–maskID
Specify the user defined maskID value for the slit of interest. This will be an integer, e.g. 958454.
Here is an example with keck_deimos:
pypeit_ql keck_deimos --raw_path /path/to/files --raw_files d1010_0056.fits.gz --setup_calib_dir /path/to/parent/calib/dir/keck_deimos/600ZD_M_6500/Calibrations --maskID 958454
This requires that the detector(s) with this
slit have been calibrated (or will be calibrated, e.g. by
specifying --det).