Wavelength Calibration

Overview

Wavelength calibration is performed using arc lamp spectra or the night sky lines, dependent on the instrument. In all cases, the solution is provided in vacuum.

This doc describes the wavelength calibration Automated Algorithms, the By-Hand Approach including the pypeit_identify script, Common Failure Modes, and more.

See WaveCalib for a discussion of the main outputs and good/bad examples.

If you wish to use your own line lists (i.e., you have reliable identifications using your instrument, but those lines are not in one of the PypeIt-supplied files), see Line Lists.

Arc Processing

If you are combining multiple arc images that have different arc lamps (e.g., one with He and another with Hg+Ne) then be sure to process without clipping. This may be the default for your spectrograph (e.g., Keck DEIMOS), but you can be certain by adding the following to the PypeIt Reduction File (for longslit observations):

[calibrations]
    [[arcframe]]
        [[[process]]]
            clip = False
            subtract_continuum = True
    [[tiltframe]]
        [[[process]]]
            clip = False
            subtract_continuum = True

For a multislit observation, you should keep clip = False, and change subtract_continuum = True to subtract_continuum = False.

Line Lists

PypeIt-Included Line Lists

Without exception, arc line wavelengths are taken from the NIST database, in vacuum. These data are stored as ASCII tables in the “arc_lines” directory of the repository. Here are the available lamps:

Lamp	Range (Å)	Last updated
ArI	3100-11000	7 October 2018
CdI	3000-6500	28 February 2022
CuI	4200-6100	4 October 2018
FeI	3000-10000	26 April 2020
HeI	3800-6000	21 December 2016
HgI	2900-12000	28 February 2022
KrI	4000-10000	3 May 2018
NeI	5000-12000	3 May 2018
XeI	4000-12000	3 May 2018
ZnI	3000-5000	6 Sep 2023
ThAr	3000-11000	9 January 2018
FeAr	3000-9000	6 Sep 2023

In the case of the ThAr list, all of the lines are taken from the NIST database, and they are labeled with a ‘MURPHY’ flag if the line also appears in the list of lines identified by Murphy et al. (2007, MNRAS, 378, 221).

User-Supplied Line Lists

Occasionally users reliably find lines in their arc spectra that are not included in the lists above. For experimenting with adding particular lines or for instrument-specific arc line detections, PypeIt allows the use of user-supplied arc line lists for wavelength calibration. The ability to use arbitrary line lists is included caveat emptor, and users MUST ensure that ALL lists used have wavelength measurements in vacuum.

Note

Users who are confident that their new lines would benefit PypeIt broadly (i.e., beyond this single instrument) should submit a pull request adding their lines to the appropriate line list file (see Development Procedures and Guidelines).

A script pypeit_install_linelist is included that installs a user-supplied line list into the PypeIt cache for use. The script usage can be displayed by calling it with the -h option:

$ pypeit_install_linelist -h
usage: pypeit_install_linelist [-h] files [files ...]

Script to install user-created arc line lists

positional arguments:
  files       Filename(s) of the line list files to be installed in the PypeIt
              cache

options:
  -h, --help  show this help message and exit

For example, you might be using the MMT Blue Channel Spectrograph and want to use various blue mercury and cadmium lines that are not included in the lists above. You would create a new line list file with a name like HgCd_MMT_lines.dat, then install it in the PypeIt cache using the command:

$ pypeit_install_linelist HgCd_MMT_lines.dat

To access this list in your reduction, you would need to include it in your lamp list in the PypeIt Reduction File along with the built-in lists:

[calibrations]
    [[wavelengths]]
        lamps = ArI, CdI, HgI, HgCd_MMT

Note

PypeIt expects all arc line list filenames to be of the form <ion name>_lines.dat. When creating a user-supplied list, be sure to include the _lines.dat portion in the filename, but exclude the _lines portion when specifying the list either in the PypeIt Reduction File or with the pypeit_identify routine, as in the example above.

The format of user-supplied line lists must match that of the built-in line lists. The best course of action is to make a copy of one of the official line lists from GitHub, and then add your new lines following the formatting of the original file. When adding lines, be sure you are using the vacuum wavelength from the NIST database tables (select Show Advanced Settings, then Vacuum (all wavelengths)) to ensure your additional lines are on the same scale as PypeIt-included lines to minimize redisuals in the wavelength fit.

By way of example, the first few lines of the neutral mercury list (HgI_lines.dat) are:

# Creation Date: 2022-Feb-28
# VACUUM -- MUST BE IN NIST
| ion |       wave | NIST | Instr | amplitude |                    Source  |
| HgI |  2968.1495 |    1 |     0 |      3000 | ldt_deveny_300_HgCdAr.fits |
| HgI |  3022.384  |    1 |     0 |      1200 | ldt_deveny_300_HgCdAr.fits |
| HgI |  3342.4450 |    1 |     0 |       700 | ldt_deveny_300_HgCdAr.fits |
| HgI |  3651.1980 |    1 |     6 |      7408 |  lrisb_600_4000_PYPIT.json |
| HgI |  3664.3270 |    1 |     2 |      1042 |  lrisb_600_4000_PYPIT.json |

Only the ion and wavelength columns are used by PypeIt for the wavelength calibration, but all must be present else the code will crash with an error.

Automated Algorithms

These notes will describe the algorithms used to perform wavelength calibration in 1D (i.e., down the slit/order) with PypeIt. The basic steps are:

1. Extract 1D arc spectra down the center of each slit/order

2. Load the parameters guiding wavelength calibration

3. Generate the 1D wavelength fits

The code is guided by the WaveCalib class, partially described by this notebook (BEWARE, this may be out of date).

For the primary step (#3), we have developed several algorithms, finding it challenging to have one that satisfies all instruments in all configurations. We briefly describe each and where they tend to be most effective. Each of these is used only to identify known arc lines in the spectrum. Fits to the identified lines (vs. pixel) are performed with the same, iterative algorithm to generate the final wavelength solution.

Holy Grail

This algorithm is based on pattern matching the detected lines with that expected from the lamps observed. It has worked well for the low dispersion spectrographs and has been used to generate the templates used for most of the other algorithms.

It has the great positive of requiring limited developer effort once a vetted line-list for the observed lamps has been generated.

However, we have found this algorithm is not highly robust (e.g., slits fail at ~5-10% rate) and it struggles with high dispersion data (e.g., ThAr lamps). At this stage, we recommend it be used primarily by developers to generate template spectra.

Reidentify

Following on our success using archived templates with the LowRedux code, we have implemented an improved version in PypeIt. Each input arc spectrum is cross-correlated against one or more archived spectra, allowing for both a shift and a stretch.

Archived spectra that yield a high cross-correlation score are used to identify arc lines based on their recorded wavelength solutions.

This algorithm is optimal for fixed-format spectrographs (e.g., X-Shooter, ESI).

Full Template

This algorithm is similar to Reidentify with two exceptions: (i) there is only a single template used (occasionally one per detector for spectra that span multiple detectors; e.g., DEIMOS); (ii) IDs from the input arc spectrum are generally performed on snippets of the full input array. The motivation for the latter is to reduce non-linearities that are not well captured by the shift+stretch analysis of Reidentify.

We recommend implementing this method for multi-slit observations, long-slit observations where wavelengths vary (e.g., grating tilts). We are likely to implement this for echelle observations (e.g., HIRES).

Echelle Spectrographs

Echelle spectrographs are a special case for wavelength solutions, primarily because the orders follow the grating equation.

In general, the approach is:

1. Identify the arc lines in each order

2. Fit the arc lines in each order to a polynomial, individually

3. Fit a 2D solution to the lines using the order number as a basis

4. Reject orders where the RMS of the fit (measured in binned pixels) exceeds a certain threshold set by the user (see RMS threshold)

5. Attempt to recover the missing orders using the 2D fit and a higher RMS threshold

6. Refit the 2D solution

One should always inspect the outputs, especially the 2D solution (global and orders). One may then need to modify the rms_thresh_frac_fwhm parameter and/or hand-fit a few of the orders to improve the solution.

RMS threshold

The parameter that controls the RMS threshold is rms_thresh_frac_fwhm, which is a fraction of the FWHM. If the parameter fwhm_fromlines is set to True, FWHM (in binned pixels) will be computed from the arc lines in each slits, otherwise the value set by the parameter fwhm will be used.

That is, each order must satisfy the following:

RMS < rms_thresh_frac_fwhm * FWHM     # FWHM in binned pixels

Mosaics

For echelle spectrographs with multiple detectors per camera that are mosaiced (e.g. Keck/HIRES), we fit the 2D solutions on a per detector basis. Ths is because we have found the mosaic solutions to be too difficult to make sufficiently accurate.

By-Hand Approach

Identify

If you would prefer to manually wavelength calibrate, then you can do so with the pypeit_identify GUI. To use this script, you must have successfully traced the slit edges (i.e., a Edges file must exist) and generated a Arc calibration frame.

pypeit_identify

usage

The script usage can be displayed by calling the script with the -h option:

$ pypeit_identify -h
usage: pypeit_identify [-h] [--lamps LAMPS] [-s] [--wmin WMIN] [--wmax WMAX]
                       [--slits SLITS] [-m] [-n] [--det DET] [--rmstol RMSTOL]
                       [--fwhm FWHM] [--sigdetect SIGDETECT] [--pixtol PIXTOL]
                       [--linear] [--force_save] [--rescale_resid]
                       [-v VERBOSITY] [--try_old]
                       arc_file slits_file

Launch PypeIt pypeit_identify tool, display extracted Arc, and load linelist.

positional arguments:
  arc_file              PypeIt Arc file
  slits_file            PypeIt Slits file

options:
  -h, --help            show this help message and exit
  --lamps LAMPS         Comma separated list of calibration lamps (no spaces)
                        (default: None)
  -s, --solution        Load a wavelength solution from the arc_file (if it
                        exists) (default: False)
  --wmin WMIN           Minimum wavelength range (default: 3000.0)
  --wmax WMAX           Maximum wavelength range (default: 50000.0)
  --slits SLITS         Which slit to load for wavelength calibration. Format
                        should be [0,1,...] for multiple slits, 0 for only one
                        slit. If creating a new WaveCalib with the -n flag, this
                        is not necessary. (default: 0)
  -m, --multi           Set this flag to create wavelength solutions for muliple
                        slits (default: False)
  -n, --new_sol         Set this flag to construct a new WaveCalib file, rather
                        than using the exising one (default: False)
  --det DET             Detector index (default: 1)
  --rmstol RMSTOL       RMS tolerance (default: 0.1)
  --fwhm FWHM           FWHM for line finding (default: 4.0)
  --sigdetect SIGDETECT
                        sigma detection for line finding (default: None)
  --pixtol PIXTOL       Pixel tolerance for Auto IDs (default: 0.1)
  --linear              Show the spectrum in linear (rather than log) scale
                        (default: False)
  --force_save          Save the solutions, despite the RMS (default: False)
  --rescale_resid       Rescale the residual plot to include all points?
                        (default: False)
  -v VERBOSITY, --verbosity VERBOSITY
                        Verbosity level between 0 [none] and 2 [all]. Default:
                        1. Level 2 writes a log with filename identify_YYYYMMDD-
                        HHMM.log (default: 1)
  --try_old             Attempt to load old datamodel versions. A crash may
                        ensue.. (default: False)

To launch the GUI, use the following command:

$ pypeit_identify Arc_A_1_01.fits Slits_A_1_01.fits.gz

basics

Instructions on how to use this GUI are available by pressing the ‘?’ key while hovering your mouse over the plotting window. You might find it helpful to specify the wavelength range of the linelist and the lamps to use the pypeit_identify command-line options. The full list of identify operations is copied below:

cursor : Select lines (LMB click)
         Select regions (LMB drag = add, RMB drag = remove)
         Navigate (LMB drag = pan, RMB drag = zoom)
left   : Advance the line list slider to the left by one
right  : Advance the line list slider to the right by one
p      : Toggle pan/zoom with the cursor
q      : Close Identify window and continue PypeIt reduction
a      : Automatically identify lines using current solution
c      : Clear automatically identified lines
d      : Delete all line identifications (start from scratch)
f      : Fit the wavelength solution
g      : Toggle ghost solution (show predicted line positions when wavelength is on the x-axis)
h      : Reset ghost parameters
i      : Include an undetected line to the detected line list
         First select fitting pixels (LMB drag = add, RMB drag = remove)
         Then press 'i' to perform a fit.         NOTE: ghost solution must be turned off to select fit regions.
l      : Load saved line IDs from file (waveids.ascii in local directory)
m      : Select a line
r      : Refit a line
s      : Save current line IDs to a file
w      : Toggle wavelength/pixels on the x-axis of the main panel
y      : Toggle the y-axis scale between logarithmic and linear
z      : Delete a single line identification
+/-    : Raise/Lower the order of the fitting polynomial

Here is a standard sequence of moves once the GUI pops up:

0. Load an existing ID list if you made one already (type ‘l’). If so, skip to step 7.

1. Compare the arc lines to a calibrated spectrum

2. Use the Magnifying glass to zoom in on one you recognize and which is in the PypeIt linelist(s)

3. To select a line, use ‘m’ to mark the line near the cursor, or use a left mouse button click near the line (a red line will appear on the selected line)

4. Use the slider bar to select the wavelength (vacuum)

5. Click on Assign Line (it will be blue when you move the mouse back in the plot window)

6. Repeat steps 1-5 until you have identified 4+ lines across the spectrum

7. Use ‘f’ to fit the current set of lines

8. Use ‘+/-’ to modify the order of the polynomial fit

9. Use ‘a’ to auto ID the rest

10. Use ‘f’ to fit again

11. Use ‘s’ to save the line IDs and the wavelength solution if the RMS of the latter is within tolerance.

Some tips: Pressing the left/right keys will advance the line list by one. You may find it helpful to toggle between pixel coordinates and wavelength coordinates (use the ‘w’ key to toggle between these two settings). Wavelength coordinates can only be accessed once you have a preliminary fit to the spectrum. When plotting in wavelength coordinates, you can overplot a ‘ghost’ spectrum (press the ‘g’ key to activate or deactivate) based on the linelist which may help you to identify lines. You can shift and stretch the ghost spectrum by clicking and dragging the left and right mouse buttons, respectively (if you’re not in ‘pan’ mode). To reset the shift/stretch, press the ‘h’ key.

If your solution is good enough (rms < 0.1 pixels), then pypeit_identify will automatically prompt you after you quit the GUI to see if you want to save the solution. Note, you can increase this tolerance using the command-line option pixtol, or by setting the force_save command-line option.

In addition to writing the wavelength solution to the current working directory, PypeIt now also saves the solution in the PypeIt cache (identified by spectrograph and the current time for uniqueness) and prints a message indicating how to use it, such as:

[INFO]    :: Your arxiv solution has been written to ./wvarxiv_ldt_deveny_20220426T0958.fits
[INFO]    :: Your arxiv solution has also been cached.
            To utilize this wavelength solution, insert the
            following block in your PypeIt Reduction File:
            [calibrations]
               [[wavelengths]]
                  reid_arxiv = wvarxiv_ldt_deveny_20220426T0958.fits
                  method = full_template

Replace the reid_arxiv filename with the filename output on your screen from pypeit_identify, and run PypeIt in the standard Full Template mode.

We also recommend that you send your solution to the PypeIt development team (e.g., post it on GitHub or the Users Slack), so that others can benefit from your wavelength calibration solution.

customizing

If your arclines are over-sampled (e.g., Gemini/GMOS) you may need to increase the fwhm from the default value of 4. And also the pixel tolerance pixtol for auto ID’ng lines from its default of 0.1 pixels. And the rmstol, if you wish to save the solution to disk!

Common Failure Modes

Most of the failures should only be in MultiSlit mode or if the calibrations for Echelle are considerably different from expectation.

As regards Multislit, the standard failure modes of the Full Template method that is now preferred are:

1. The lamps used are different from those archived.

2. The slit spans much bluer/redder than the archived template.

In either case, a new template may need to be generated. If you are confident this is the case, Submit an issue.

Items to Modify

There are several parameters in the Wavelength Calibration WavelengthSolutionPar Keywords that one needs to occasionally customize for your specific observations. We describe the most common below.

FWHM

The arc lines are identified and fitted with an expected knowledge of their FWHM (future versions should solve for this). A fiducial value for a standard slit is assumed for each instrument but if you are using particularly narrow/wide slits then in your PypeIt Reduction File you may need to modify it like so:

[calibrations]
    [[wavelengths]]
        fwhm=X.X

Alternatively, PypeIt can compute the arc line FWHM from the arc lines themselves (only the ones with the highest detection significance). The FWHM measured in this way will override the value set by fwhm, which will still be used as a first guess and for the Wavelength Tilts. This is particularly advantageous for multi-slit observations that have masks with different slit widths (e.g., DEIMOS LVM slit-masks). The keyword that controls this option is called fwhm_fromlines and is set to False by default (see User-level Parameters). To switch it on, add the following to your PypeIt Reduction File:

[calibrations]
    [[wavelengths]]
        fwhm_fromlines = True

Flexure Correction

By default, the code will calculate a flexure shift based on the (boxcar) extracted sky spectrum. See Flexure Correction for further details.

Developers

Adding a new solution

When adding a new instrument or grating, one generally has to perform a series of steps to enable accurate and precise wavelength calibration with PypeIt. We recommend the following procedure, when possible:

• Perform wavelength calibration with a previous pipeline:

  • Record a calibrated, arc spectrum (i.e., wavelength vs. counts)

  • In vaccuum or convert from air to vacuum

• If no other DRP exists:

  • Try running PypeIt with the Holy Grail algorithm and use that output.

  • If that fails, generate a solution with the By-Hand Approach.

• Build a template from the arc spectrum:

  • For fixed-format spectrographs, one spectrum (or one per order) should be sufficient.

  • For gratings that tilt, one may need to splice together a series of arc spectra to cover the full spectral range.

  • Follow the guidance here and see examples in templates and

• Augment the line list

  • We are very conservative about adding new lines to the existing line lists. One bad line can have significant, negative consequences.

  • Therefore, carefully vet the line by insuring it is frequently detected

  • And that it does not have large systematic residuals in good wavelength solutions.

  • Then add to one of the files to data/arc_lines/lists

Full Template

The preferred method for multi-slit calibration is now called full_template which cross-matches an input spectrum against an archived template. The latter must be constructed by a developer, using templates. The following table summarizes the existing ones (all of which are in the data/arc_lines/reid_arxiv folder):

Instrument	Setup	Name
keck_deimos	600ZD grating, all lamps	keck_deimos_600ZD.fits
keck_deimos	830G grating, all lamps	keck_deimos_830G.fits
keck_deimos	1200G grating, all lamps	keck_deimos_1200G.fits
keck_deimos	1200B grating, all lamps	keck_deimos_1200B.fits
keck_deimos	900ZD grating, all lamps	keck_deimos_900ZD.fits
keck_lris_blue	B300 grism, all lamps	keck_lris_blue_300_d680.fits
keck_lris_blue	B400 grism, all lamps?	keck_lris_blue_400_d560.fits
keck_lris_blue	B600 grism, all lamps	keck_lris_blue_600_d560.fits
keck_lris_blue	B1200 grism, all lamps	keck_lris_blue_1200_d460.fits
keck_lris_red	R400 grating, all lamps	keck_lris_red_400.fits
keck_lris_red	R1200/9000 , all lamps	keck_lris_red_1200_9000.fits
shane_kast_blue	452_3306 grism, all lamps	shane_kast_blue_452.fits
shane_kast_blue	600_4310 grism, all lamps	shane_kast_blue_600.fits
shane_kast_blue	830_3460 grism, all lamps	shane_kast_blue_830.fits

Follow the guidance here and see examples in templates and

One of the key parameters (and the only one that can be modified) for full_template is the number of snippets to break the input spectrum into for cross-matching. The default is 2 and the concept is to handle non-linearities by simply reducing the length of the spectrum. For relatively linear dispersers, nsnippet = 1 may frequently suffice.

For instruments where the spectrum runs across multiple detectors in the spectral dimension (e.g., DEIMOS), it may be necessary to generate detector specific templates (ugh). This is especially true if the spectrum is partial on the detector (e.g., the 830G grating).

---

Additional Reading

• Flexure Correction
• Heliocentric Correction
• Wavelength Tilts
• Constructing a New Wavelength Template