Automated sorting of DEIMOS frames by instrument configuration
Version History
Version |
Author |
Date |
|
---|---|---|---|
1.0 |
Kyle Westfall |
13 Oct 2020 |
1.1.2dev |
1.1 |
Debora Pelliccia |
12 Jul 2021 |
1.9.2.dev |
Basics
Sorting frames by the configuration of the instrument, ensuring that this configuration is the same for all coupled sets of calibration and science frames, is performed by the pypeit_setup script; see Setup. pypeit_setup uses automated procedures to sort the frames and write a PypeIt Reduction File for each unique configuration (or for some down-selected set).
DEIMOS configuration identification
The DEIMOS instrument configuration is determined by a unique combination of the following keywords:
|
Header Key |
---|---|
|
|
|
|
|
|
|
|
|
|
|
|
as determined by
pypeit.metadata.PypeItMetaData.unique_configurations()
. The
AMPMODE
value is included, even though PypeIt
(currently)
restricts itself to only attempting to reduce frames read by the B
and A amplifiers; see
valid_configuration_values()
.
Additionally, PypeIt
requires all frames to have MOSMODE ==
'Spectral'
. Frames that do not match these header keyword
restrictions will not be included in the automatically generated
PypeIt Reduction File created by pypeit_setup.
For DEIMOS, the unique configurations are determined by collating the
relevant metadata from the headers of all frames found by a run of
pypeit_setup, except those that are designated as bias or
dark frames. The reason is that bias and darks can have header data
(e.g., dispangle
) that do not match the instrument configuration
that an observer intended for their use; e.g., the frames were taken
before the instrument was fully configured for the night’s
observations. To match these frames to a specific configuration,
PypeIt
uses the DATE-OBS
header keyword to match the frames
to the configurations with frames taken on the same date.
DEIMOS calibration groups
PypeIt
uses the concept of a “calibration group” to define a
complete set of calibration frames (e.g., arcs, flats, biases) and
the science frame to which these calibration frames should be
applied. By default, pypeit_setup uses the configuration
identifier (e.g., A
) to assign frames to a single calibration
group. No automated procedure exists to do anything except this.
However, a user can edit the PypeIt Reduction File to, within a given
configuration, assign specific calibration frames to specific science
frames using the data in the calib
column. For example, if the
observer takes both evening and morning calibration frames, the
PypeIt
user can use the PypeIt Reduction File to associate
calibrations taken at the end of the night with the morning
calibrations and vice versa. For DEIMOS specifically, columns with
the DATE-OBS
and UTC
header data for each frame are provided
to assist with this.
Testing
Requirement PD-3 states: “As a user, I want the pipeline to automatically and correctly associate calibrations with science data.”
PypeIt
meets this requirement in the majority of use cases. One
exception to this is given in the warning above with respect to how
bias and dark frames are assigned to “calibration groups”.
The test used to demonstrate PD-1 is satisfied
(Automated typing of DEIMOS frames) is also relevant here in that each
directory in the PypeIt
dev suite with DEIMOS data correctly
identifies the frame types and associates them with a single
configuration, all written to a single pypeit file.
To test that PypeIt
can successfully identify multiple
configurations among a set of files, we have added the
test_setup_keck_deimos_multiconfig
and
test_setup_keck_deimos_multiconfig_clean
tests to
${PYPEIT_DEV}/unit_tests/test_setups.py
.
To run these tests:
cd ${PYPEIT_DEV}/unit_tests
pytest test_setup.py::test_setup_keck_deimos_multiconfig -W ignore
pytest test_setup.py::test_setup_keck_deimos_multiconfig_clean -W ignore
The tests require that you have downloaded the PypeIt
PypeIt Development Suite and defined the PYPEIT_DEV
environmental
variable that points to the relevant directory.
Both tests collect the names of all files in the following two directories:
${PYPEIT_DEV}/RAW_DATA/keck_deimos/830G_L_8100
${PYPEIT_DEV}/RAW_DATA/keck_deimos/830G_L_8400
test_setup_keck_deimos_multiconfig
The algorithm for this test is as follows:
Use
PypeItSetup
to automatically identify the configurations for these files.Check that the code found two configurations and wrote the pypeit files for each.
For each configuration:
Read the pypeit file
Check that the name for the setup is correct (‘A’ or ‘B’)
Check that the calibration group is the same for all frames (‘0’ or ‘1’)
test_setup_keck_deimos_multiconfig_clean
The algorithm for this test is as follows:
Similar to the previous test, use
PypeItSetup
to automatically identify the configurations for these files.Check that the code found two configurations.
Check that the only bias frame in the list was correctly identified.
Check that the full table containing both configurations contains the correct number of files (25).
Check that “cleaning” the configurations of frames that cannot be reduced by
PypeIt
(those withMOSMODE != 'Spectral'
orAMPMODE != SINGLE:B
orAMPMODE != SINGLE:A
), usingclean_configurations()
does not remove any file because all of the dev-suite files are valid.After artificially changing the metadata for two files so that they do not satisfy the necessary metadata restrictions, check that
clean_configurations()
removes both of these files.
Because these tests are now included in the PypeIt
Unit Tests (GitHub CI), these configuration checks are performed by the
developers for every new version of the code.