Collating 1D Spectra

Overview

PypeIt provides a tool called pypeit_collate_1d to go through a large number of processed Spec1D Output files, group the spectra by object, and coadd all matching spectra.

Grouping can be done by sky coordinates if available or by pixel coordinates. Coadding is done using flux-calibrated spectra when available.

Fluxing is performed using archived sensitivity files.

Usage

All collate options are accessible via either the command line or a .collate1d file. Additional coadd 1D parameters can also be passed in via configuration files.

An example run of pypeit_collate_1d requires a tolerance and a list of spec1d files:

pypeit_collate_1d --tolerance 3 --spec1d_files Science/spec1d*.fits

pypeit_collate_1d can also be run in dry run mode to try out different threshold values without doing any processing on the input:

$ pypeit_collate_1d --tolerance 3 --spec1d_files Science/spec1d*

Writing the parameters to collate1d.par
UserWarning: Selected configuration file already exists and will be overwritten! (parset.py:649)
[INFO]    :: Creating J132402.48+271212.38_DEIMOS_20130409.fits from the following sources:
[INFO]    ::     Science/spec1d_DE.20130409.20629-S13A-SDF-z6clus_DEIMOS_2013Apr09T054342.730.fits: SPAT0287-SLIT0284-DET01 (sbzk_1440)
[INFO]    ::     Science/spec1d_DE.20130409.22509-S13A-SDF-z6clus_DEIMOS_2013Apr09T061459.683.fits: SPAT0286-SLIT0284-DET01 (sbzk_1440)
[INFO]    :: Creating J132401.95+271237.80_DEIMOS_20130409.fits from the following sources:
[INFO]    ::     Science/spec1d_DE.20130409.20629-S13A-SDF-z6clus_DEIMOS_2013Apr09T054342.730.fits: SPAT0363-SLIT0359-DET01 (sbzk_1796)
[INFO]    ::     Science/spec1d_DE.20130409.22509-S13A-SDF-z6clus_DEIMOS_2013Apr09T061459.683.fits: SPAT0360-SLIT0359-DET01 (sbzk_1796)
[INFO]    :: Creating J132401.46+271303.86_DEIMOS_20130409.fits from the following sources:
[INFO]    ::     Science/spec1d_DE.20130409.20629-S13A-SDF-z6clus_DEIMOS_2013Apr09T054342.730.fits: SPAT0438-SLIT0433-DET01 (NB711_007606)
[INFO]    ::     Science/spec1d_DE.20130409.22509-S13A-SDF-z6clus_DEIMOS_2013Apr09T061459.683.fits: SPAT0438-SLIT0433-DET01 (NB711_007606)
[INFO]    :: Creating J132401.22+271321.92_DEIMOS_20130409.fits from the following sources:
[INFO]    ::     Science/spec1d_DE.20130409.20629-S13A-SDF-z6clus_DEIMOS_2013Apr09T054342.730.fits: SPAT0503-SLIT0500-DET01 (NB921_024976)
[INFO]    ::     Science/spec1d_DE.20130409.22509-S13A-SDF-z6clus_DEIMOS_2013Apr09T061459.683.fits: SPAT0497-SLIT0494-DET05 (NB921_024976)
[INFO]    :: Creating J132405.90+271402.23_DEIMOS_20130409.fits from the following sources:
[INFO]    ::     Science/spec1d_DE.20130409.20629-S13A-SDF-z6clus_DEIMOS_2013Apr09T054342.730.fits: SPAT1137-SLIT1134-DET01 (NB973_031779)
[INFO]    ::     Science/spec1d_DE.20130409.22509-S13A-SDF-z6clus_DEIMOS_2013Apr09T061459.683.fits: SPAT1136-SLIT1134-DET01 (NB973_031779)
[INFO]    :: Creating J132407.49+271432.92_DEIMOS_20130409.fits from the following sources:
[INFO]    ::     Science/spec1d_DE.20130409.20629-S13A-SDF-z6clus_DEIMOS_2013Apr09T054342.730.fits: SPAT1434-SLIT1433-DET01 (photz_3929)
[INFO]    ::     Science/spec1d_DE.20130409.22509-S13A-SDF-z6clus_DEIMOS_2013Apr09T061459.683.fits: SPAT1434-SLIT1433-DET01 (photz_3929)
[INFO]    :: Creating J132401.27+271430.39_DEIMOS_20130409.fits from the following sources:
[INFO]    ::     Science/spec1d_DE.20130409.20629-S13A-SDF-z6clus_DEIMOS_2013Apr09T054342.730.fits: SPAT0847-SLIT0838-DET05 (NB921_032374)
[INFO]    :: Creating J132412.25+271322.19_DEIMOS_20130409.fits from the following sources:
[INFO]    ::     Science/spec1d_DE.20130409.20629-S13A-SDF-z6clus_DEIMOS_2013Apr09T054342.730.fits: SPAT1510-SLIT1544-DET05 (photz_2590)
[INFO]    ::     Science/spec1d_DE.20130409.22509-S13A-SDF-z6clus_DEIMOS_2013Apr09T061459.683.fits: SPAT1510-SLIT1544-DET05 (photz_2590)
[INFO]    :: Creating J132401.80+271145.36_DEIMOS_20130409.fits from the following sources:
[INFO]    ::     Science/spec1d_DE.20130409.22509-S13A-SDF-z6clus_DEIMOS_2013Apr09T061459.683.fits: SPAT0079-SLIT0092-DET01 (photz_835)
[INFO]    :: Creating J132404.67+271226.61_DEIMOS_20130409.fits from the following sources:
[INFO]    ::     Science/spec1d_DE.20130409.22509-S13A-SDF-z6clus_DEIMOS_2013Apr09T061459.683.fits: SPAT0561-SLIT0569-DET01 (photz_1674)
[INFO]    ::     Science/spec1d_DE.20130409.22509-S13A-SDF-z6clus_DEIMOS_2013Apr09T061459.683.fits: SPAT0558-SLIT0564-DET05 (photz_1674)
[INFO]    :: Creating J132406.95+271357.19_DEIMOS_20130409.fits from the following sources:
[INFO]    ::     Science/spec1d_DE.20130409.22509-S13A-SDF-z6clus_DEIMOS_2013Apr09T061459.683.fits: SPAT1211-SLIT1207-DET01 (NB973_031059)
[INFO]    :: Creating J132407.01+271649.64_DEIMOS_20130409.fits from the following sources:
[INFO]    ::     Science/spec1d_DE.20130409.22509-S13A-SDF-z6clus_DEIMOS_2013Apr09T061459.683.fits: SPAT2038-SLIT1998-DET05 (SERENDIP)
[INFO]    :: Total duration: 0:00:03.043560

Command Line

The script usage can be displayed by calling the script with the -h option:

$ pypeit_collate_1d -h
usage: pypeit_collate_1d [-h] [--spec1d_files [SPEC1D_FILES ...]]
                         [--par_outfile PAR_OUTFILE] [--outdir OUTDIR]
                         [--spec1d_outdir SPEC1D_OUTDIR] [--tolerance TOLERANCE]
                         [--match_using MATCH_USING] [--dry_run] [--ignore_flux]
                         [--flux] [--exclude_slit_bm EXCLUDE_SLIT_BM]
                         [--exclude_serendip] [--wv_rms_thresh WV_RMS_THRESH]
                         [--refframe {observed,heliocentric,barycentric}]
                         [--chk_version] [-v VERBOSITY]
                         [input_file]

Flux/Coadd multiple 1d spectra from multiple nights and prepare a directory for
the KOA.

positional arguments:
  input_file            (Optional) File for guiding the collate process.
                        Parameters in this file are overidden by the command
                        line. The file must have the following format:

                        [collate1d]
                          tolerance             <tolerance>
                          outdir                <directory to place output files>
                          spec1d_outdir         <directory to place modified spec1ds, if any>
                          exclude_slit_trace_bm <slit types to exclude>
                          exclude_serendip      If set serendipitous objects are skipped.
                          match_using           Whether to match using "pixel" or
                                                "ra/dec"
                          dry_run               If set the matches are displayed
                                                without any processing
                          flux                  Flux calibrate using archived sensfuncs.
                          ignore_flux           Ignore any flux calibration information in
                                                spec1d files.
                          wv_rms_thresh         If set, any objects with a wavelength rms > than the input
                                                value are skipped, else all wavelength rms values are accepted.
                          refframe              Perform reference frame correction prior to coadding.
                                                Options are ['observed', 'heliocentric', 'barycentric']. Defaults to None.

                        spec1d read
                        <path to spec1d files, wildcards allowed>
                        ...
                        end

options:
  -h, --help            show this help message and exit
  --spec1d_files [SPEC1D_FILES ...]
                        One or more spec1d files to flux/coadd/archive. Can
                        contain wildcards
  --par_outfile PAR_OUTFILE
                        Output to save the parameters
  --outdir OUTDIR       The path where all coadded output files and report files
                        will be placed. Defaults to the current directory.
  --spec1d_outdir SPEC1D_OUTDIR
                        The path where all modified spec1d files are placed.
                        These are only created if flux calibration or refframe
                        correction are asked for. Defaults to overwriting
                        existing spec1ds.
  --tolerance TOLERANCE
                        The tolerance used when comparing the coordinates of
                        objects. If two objects are within this distance from
                        each other, they are considered the same object. If
                        match_using is 'ra/dec' (the default) this is an angular
                        distance. The defaults units are arcseconds but other
                        units supported by astropy.coordinates.Angle can be used
                        (`e.g.`, '0.003d' or '0h1m30s'). If match_using is
                        'pixel' this is a float.
  --match_using MATCH_USING
                        Determines how 1D spectra are matched as being the same
                        object. Must be either 'pixel' or 'ra/dec'.
  --dry_run             If set, the script will display the matching File and
                        Object Ids but will not flux, coadd or archive.
  --ignore_flux         If set, the script will only coadd non-fluxed spectra
                        even if flux data is present. Otherwise fluxed spectra
                        are coadded if all spec1ds have been fluxed calibrated.
  --flux                If set, the script will flux calibrate using archived
                        sensfuncs before coadding.
  --exclude_slit_bm EXCLUDE_SLIT_BM
                        A list of slit trace bitmask bits that should be
                        excluded. Comma separated.
  --exclude_serendip    Whether to exclude SERENDIP objects from collating.
  --wv_rms_thresh WV_RMS_THRESH
                        If set, any objects with a wavelength RMS > this value
                        are skipped, else all wavelength RMS values are
                        accepted.
  --refframe {observed,heliocentric,barycentric}
                        Perform reference frame correction prior to coadding.
                        Options are: observed, heliocentric, barycentric
  --chk_version         If True enforce strict PypeIt version checking to ensure
                        that all files were created with the current version of
                        PypeIt. If set to False, the code will attempt to read
                        out-of-date files and keep going. Beware (!!) that this
                        can lead to unforeseen bugs that either cause the code
                        to crash or lead to erroneous results. I.e., you really
                        need to know what you are doing if you set this to
                        False!
  -v VERBOSITY, --verbosity VERBOSITY
                        Verbosity level between 0 [none] and 2 [all]. Default:
                        1. Level 2 writes a log with filename
                        collate_1d_YYYYMMDD-HHMM.log

.collate1D Configuration File

The cofiguration file for pypeit_collate_1d consists of a set of Collate1DPar Keywords, followed by a list of spec1d files. An example configuration file is shown below:

# User-defined coadding and fluxing parameters can be given but are not required
[coadd1d]
sn_clip = 20

[fluxcalib]
extinct_correct = True

# User-defined collating parameters
[collate1d]

# Whether to match using ra/dec sky coordinates or via spatial pixel coordinates on the exposure.
match_using ra/dec

# How close two spectra must be in order to be considered the same object.
# This can be specified in a variety of formats.

# For ra/dec matching, this is an angular distance. For pixel matching it's an
# integer or floating point number.
# For example:
#     3.5      Arcseconds (the default unit)
#     2m       Arcminutes
#     0.0003d  Decimal degrees
#     0d3m4.3s Degrees, arcminutes, arcseconds
#     1h2m3s   Hours, minutes, seconds
#     300      Pixel distance
#
tolerance = 3.5

# What slit bitmask flags to exclude from the matching.
# If this list is not empty, each spec1d file to be coadded
# must have a matching spec2d file.

slit_exclude_flags = BOXSLIT

# Exclude SERENDIP objects
exclude_serendip = False

# Whether to flux calibrate spec1d files using archival senfuncs.
# Defaults to False
#flux = False

# Whether to perform reference frame correciton on spec1d files.
# Options can be None, observed,heliocentric,barycentric. Defaults to None.
#refframe = heliocentric

# Whether to ignore existing flux calibrated data in the spec1ds.
# Defaults to False. Even when this is False, if the flux calibration data
# (e.g. OPT_FLAM or BOX_FLAM) is not available the uncalibrated data is coadded.
#ignore_flux = False

# Exclude any object with a wavelength wave_rms > than this threshold
# wv_rms_thresh = 0.2

# Where to place coadded files and report files. Defaults to
# current directory.
#outdir = /work/output

# Whether to check that spec1d files and archival sensfunc files have an
# up to date datamodel version. If false (the default) version numbers are
# not checked.
#chk_version = True

# A list of the spec1d files. Wildcards are allowed.
# This follows the input file data block format.
spec1d read
filename
Science/spec1d*.fits
spec1d end

Coadd1D and Fluxing Configuration

Coadd1d configuration can be set in the .collate1d as shown above. Coadd parameters can also be specified in a separate .coadd1d file with the same base name as the .collate1d file; see Coadd1DPar Keywords and Coadd 1D Spectra.

Fluxing configuration can also be configured as shown above; see FluxCalibratePar Keywords and Fluxing. However pypeit_collate_1d will always set extrap_sens and use_archived_sens to True when fluxing.

Flux Calibration

pypeit_collate_1d will coadd flux calibrated data if it is available in all of the spec1ds being used (i.e. a OPT_FLAM or BOX_FLAM entry exists, see Spec1D Output). To override this behavior so that pypeit_collate_1d always coadds using counts, pass --ignore_flux to the command line or set the ignore_flux = True in the configuration file.

Alternatively pypeit_collate_1d can perform flux calibration itself using archived sensitivity functions. To do so pass --flux to the command line or set flux = True in the configuration file. If spec1d_outdir is set on the command line or in a configuration file, the new spec1d will be written to that directory. Otherwise this will overwrite any flux calibration data already in the spec1d files. Currently archived sensitivity functions are experimental and only supported for DEIMOS.

Reference Frame correciton

For data that was reduced without reference-frame correction (i.e. refframe in WavelengthSolutionPar Keywords was set to observed in the PypeIt Reduction File), pypeit_collate_1d can perform this correction before coadding the files. To do so pass either refframe heliocentric or refframe barycentric via the command line or a configuration file. This correction will only be performed on spec1d files that have not already been corrected. If spec1d_outdir is specified, the corrected spec1d files will be written to that location. Otherwise the existing spec1d fils will be modified.

Reporting

pypeit_collate_1d creates two files to report on the results of collating: collate_report.dat and collate_warnings.txt.

collate_report.dat

The collate_report.dat is an IPAC file containing metadata about what objects were coadded. The file is organized by the output file name, and it has multiple rows per output file: one row per extracted spectra that was coadded to create the file. Below is a description of its columns.

Column Name

Description

filename

The filename of the coadded output file.

maskdef_objname

The name of the object being coadded.

maskdef_id

The slit id for the according to the mask definition.

det

The detector the spectrum was captured on.

objra

The RA of the source object, determined from the mask definition.

objdec

The DEC of the source object, determined from the mask definition.

s2n

The signal to noise ratio of the extracted object.

wave_rms

The RMS in pixels of the wavelength solution.

spec1d_filename

The name of the spec1d file containing the spectrum.

dispname

The grating used for the source image.

slmsknam

The slitmask used for the source image.

binning

Binning from the source image header.

mjd

Modified Julian Date from the the source image header.

airmass

Airmass from the source image header.

exptime

Exposure time from the source image header.

guidfwhm

Guide star FWHM value from the source image header.

progpi

Program Principle Investigator from the source image header.

semester

Semester from the source image header.

progid

Program ID from the source image header.

collate_warnings.txt

The collate_warnings.txt file contains information about any failures that occurred during collating and/or archiving. Below is an example collate_warnings.txt:

pypeit_collate_1d warnings

Started 2021-07-26 12:02:39.156118
Duration: 0:00:47.245288

Excluded Objects:

Excluding SERENDIP object from SPAT1510-SLIT1544-DET05 in Science/spec1d_DE.20130409.20629-S13A-SDF-z6clus_DEIMOS_20130409T054342.730.fits
Excluding SERENDIP object from SPAT0071-SLIT0086-DET05 in Science/spec1d_DE.20130409.22509-S13A-SDF-z6clus_DEIMOS_20130409T061459.683.fits


Failed to Coadd:


Missing Archive Files:

Could not archive matching text file for Science/spec1d_DE.20130409.20629-S13A-SDF-z6clus_DEIMOS_20130409T054342.730.fits, file not found.
Could not archive matching text file for Science/spec1d_DE.20130409.22509-S13A-SDF-z6clus_DEIMOS_20130409T061459.683.fits, file not found.

Matching

To decide if two spectra match, pypeit_collate_1d performs the following checks.

  1. The slit_exclude_flags are checked against the slit the spectrum was found in. If a match is found, the spectrum is skipped and not coadded.

  2. The two spectra are compared to make sure they are from the same spectrograph, in the same configuration.

  3. The position of the two spectra are compared to see that they are within a given tolerance of each other.

If a spectrum does not match any others, it is still output using the Current Coadd1D Data Model.

Step 1: Exclude by Slit Bitmask

PypeIt assigns a bitmask to each slit in a slit mask. Spectra from slits of certain types can be excluded from coadding. If this feature is used, there must be a Spec2D Output file corresponding for each spec1d file. The bitmask values:

SHORTSLIT

Slit formed by left and right edge is too short. Not ignored for flexure.

BOXSLIT

Slit formed by left and right edge is valid (large enough to be a valid slit), but too short to be a science slit.

USERIGNORE

User has specified to ignore this slit. Not ignored for flexure.

BADWVCALIB

Wavelength calibration failed for this slit.

BADTILTCALIB

Tilts analysis failed for this slit.

SKIPFLATCALIB

Flat field generation failed for this slit. Skip flat fielding.

BADFLATCALIB

Flat field generation failed for this slit. Ignore it fully.

BADREDUCE

Skysub/extraction failed for this slit.

Step 2: Match by Spectrograph Configuration

There are a set of configuration keys that define a unique configuration for each spectrograph PypeIt supports; see Use of Metadata to Identify Instrument Configurations. For example, for DEIMOS the keys are DISPNAME, DECKER, BINNING, DISPANGLE, and AMP. pypeit_collate_1d will only match spectra if the values for their configuration keys, except for DECKER, are the same. The DECKER keyword is not checked so that spectra from the same object through different slit masks can be coadded.

Step 3: Match by Position

RA/DEC

If RA/DEC matching is being used, the tolerance is specified as an angular distance. By default, it is treated as arcseconds, but any format supported by astropy Angles can be used. The matching is done using the Astropy SkyCoord separation method.

Currently only DEIMOS and MOSFIRE supports RA/DEC matching.

Pixel

If pixel matching is being used, the tolerance can be specified as an integer or floating point number. The matching is done as the distance along the spatial axis of the exposure.