What’s New in PypeIt

For a detailed log of code edits (including releases earlier than version 1.5.0), see CHANGELOG.

Version 1.14.0

Dependency Changes

  • Main dependency bumps: numpy>=1.22, matplotlib>=3.7, ginga>=4.1.1, qtpy>=2.0.1

Functionality/Performance Improvements and Additions

  • Improvements to wavelength grids and masking in coadd routines.

  • Refactored coadding routines to work with lists to support coadding data from different setups.

  • Sensitivity function models can now be computed relative to the flat-field spectrum.

  • Improvements in 2D coaddition

    • Fix a bug in pypeit_setup_coadd2d for the output file name of the .coadd2d file

    • Added possibility to specify more than one Science folder in pypeit_setup_coadd2d

    • Now only_slits parameter in pypeit_coadd_2dspec includes the detector number (similar to slitspatnum)

    • Added exclude_slits parameter in pypeit_coadd_2dspec to exclude specific slits

    • Fix wrong RA and Dec for 2D coadded serendips

  • Allow wavelength calibrations for specific slits/orders to be redone (instead of adopting the value from a processed calibration frame); see new redo_slits parameter.

Instrument-specific Updates

  • Adds/Improves support for Gemini/GNIRS (IFU), Keck/KCRM, Keck/ESI, MDM/Modspec, Keck/HIRES, JWST

  • HIRES wavelength solution improvements galore

  • Improvements for Keck/LRIS

    • Generated wavelength templates for all the LRIS grism & grating

    • Added FeAr line list

    • Improved calibration association and frame typing

    • Improved and added documentation

    • Changes to metadata.py including commenting out, in the pypeit file, files that have frametype None (this prevent run_pypeit to crash)

    • Added a function check_spectrograph() (currently only defined for LRIS), that checks (during pypeit_setup) if the selected spectrograph is the corrected one for the data used.

Script Changes

  • Added a script to convert a wavelength solution into something that can be placed in the reid archive.

  • Store user-generated wavelength solution in pypeit cache

Datamodel Changes

  • Changed calibration frame naming as an attempt to avoid very long names for files with many calibration groups. Sequential numbers are reduced to a range; e.g., '0-1-2-3-4' becomes '0+4' and '3-5-6-10-11-12-15-18-19' becomes '3-5+6-10+12-15-18+19'

  • Instrumental FWHM map is calculated and output in Calibrations and spec1d files.

Under-the-hood Improvements

  • Change how masking is dealt with in extraction to fix a bug in how masks were being treated for echelle data

  • Refactored function that loads wavelength calibration line lists

Bug Fixes

  • Hotfix for GTC/OSIRIS lamp list

  • Hotfix for Arc1D stats annotations on the QA

  • Hotfix for metadata:

    • correctly set config_independent_frames when multiple configurations are being setup

    • support lists in config_independent_frames

  • Hotfix for rebin (speed-up and conserves flux)

  • Hotfix for skysub regions GUI that used np.bool

  • Hotfix to stop pypeit_setup from crashing on data from lbt_luci1, lbt_luci2, magellan_fire, magellan_fire_long, p200_tspec, or vlt_sinfoni.

  • Hotfix to set BPM for each type of calibration file.

  • Fixed a bug in echelle coadding where the wrong coadded spectra were being used in final stacks.

  • Fix a bug in spectrograph.select_detectors, where a list of slitspatnum could not be used.

Version 1.13.0

Functionality/Performance Improvements and Additions

  • Allow user control of the local sky subtraction window

  • Implemented a resample algorithm when generating datacubes

  • Sensitivity function masking and output updates

  • Improve speed in ginga visualization of traces and added pypeit_chk_tilts. Note that this script uses an update of the tilts datamodel, so it will not work on older reductions.

  • Fixed a bug in the variance_model calculation for combined images.

Datamodel Changes

  • Removed use of the term “master”, renamed to calibration frames/files. Default output directory for calibration frames is now Calibrations. Calibration frames renamed from, e.g., MasterArc* to Arc*.

  • Names of associated calibration frames now written to spec2d file headers.

Dependency Changes

  • Deprecate support for python 3.8 with PypeIt, allow python 3.11

  • Make pypeit_show_2dspec (somewhat) backwards compatible.

  • Added the option to disable strict version checking for 1d coadds.

  • Start to deprecate use of os.path in favor of pathlib

Script Changes

  • In PypeIt files, the 'calib' column is now always added to the pypeit file, regardless of whether or not you also request the 'comb_id' and 'bkg_id' columns.

  • In PypeIt files, the calibration group numbers can now be anything, as long as there are no more than 63 unique integers.

  • Deprecated pypeit_parse_calib_id script, but improved the .calib file provided by pypeit_setup. The .calib file is now always written, and provides a direct association between input raw files and output calibration files. Discussed in new docs.

  • Added coadd2d setup script

  • Added the possibility to use dither offsets saved in the header of the science frames for coadding 2D spectra (dithoff must be part of the spectrograph metadata).

  • Major quicklook updates; see documentation. ql_multislit.py deprecated.

Instrument-specific Updates

  • Updates to reduction parameters for LDT/DeVeny

Hotfixes

  • Hotfix to docs to ensure pypeit_loaders api doc is generated

  • Hotfix for KCWI when using alignment (aka ContBars) frames for the astrometric correction.

Version 1.12.2

Instrument-specific Updates

  • Include Gemini/GMOS mask design slurping and usage documentation

  • New Gemini/GMOS wavelength solution

  • Added NIRES tutorial doc

  • reid_arxiv templates for all MMTO Blue Channel gratings and for MMTO Binospec G600 and G1000

Usage Updates

  • Added new specutils interface

Datamodel Updates

  • Include the S/N of extracted spectra in the SpecObj datamodel

Hotfixes

  • Various bug fixes and enhancements to mmt_bluechannel and mmt_binospec support

  • Fixed bugs when only performing calibrations and (1) calib groups are all set to ‘all’ or (2) anything other than ‘0’.

Version 1.12.1

Instrument-specific Updates

  • Implemented the upgraded GTC/OSIRIS+

  • Added support for more NOT/ALFOSC grisms as well as NOT recommended standards

  • Implemented the SOAR/Goodman (blue) M1 only

Minor Updates

  • Modify install notes to allow python 3.10; python3.8 not longer explicitly supported

  • Allow for bad orders during extraction without crashing

Hotfixes

  • Specify sphinx versions to correctly work with sphinx_rtd_theme

  • Fixed bug that caused crash of sensfunc routines using telluric grids in offline processing

  • Fixed error when showing flats in Ginga when the fine correction is not performed

  • Fixed keymap error when displaying GUIs

  • New docs on OneSpec

Version 1.12.0

Functionality Improvements / User-level Updates

  • Refactor quicklook scripts

  • Added spectral flexure and reference frame corrections for IFU data

  • Allow separate sky frame to be used for sky subtraction with IFU data

  • Limit the images written to the Edges file to only the trace image, mask, and detector.

Instrument-specific Updates

  • Limit LRISr header crashes

  • Parse Keck/NIRES dither patterns, similar to MOSFIRE

  • Started modifications and support for JWST NIRSpec

Minor / Under-the-hood Updates and Bug Fixes

  • Fixed bug that allowed science frames to be assigned to multiple instrument configurations

  • Fixed typo related to GitHub download for offline processing

  • OpenMP link fix

  • Enable boxcar_radius for manual extraction

  • Prevent flexure crash

  • Fixed error with deprecated numpy types

  • Improved optimization of bspline c code

  • Introduce BitMaskArray class to ease use of bitmasks

  • Fixed memory hogging by matplotlib when using version >= 3.6.1

Version 1.11.0

Significant Updates

  • Add ability for users to specify custom arc line lists for wavelength calibration, saved in the user’s PypeIt cache

  • Allow for the specification of a specific UVIS extinction file for sensitivity function computation and flux calibration.

  • Now more than one setup can be assigned to the same calibration frame, allowing to associate the same calibration frames to different science/standard frames, if desired.

  • Refactor and general update of documentation

Instrument-specific Updates

  • Correctly associate calibrations with science data for MOSFIRE longslit and long2pos masks.

  • Automatically assign comb_id and bkg_id to MOSFIRE science data, using the information on the dither pattern.

Minor

  • Added Keck/NIRES frame-typing development doc.

  • Allow verbosity specification for various post-processing command-line scripts.

Version 1.10.0

Major Changes

  • Refactor PypeIt input files. Main pypeit file remains the same, except that inclusion of leading and trailing | characters in the data table (required in previous versions) will now result in DeprecationWarnings. All post-processing scripts (coadding, fluxing, etc) must use the new format. See the main documentation pages.

Minor Changes/Improvements

  • Apply find_min_max when clipping the image for object finding

  • Mask bad detector regions for global sky flexure calculation

  • Added wavelength diagnostics to the spec2d output

Instrument-specific Changes/Improvements

  • Modify tweak_standard for Keck-MOSFIRE/J2

  • Detector structure correction included in flatfield calibration (Keck-KCWI only)

Version 1.9.1

Addresses bug related to downloading from the reid_arxiv when using the reidentify wavelength calibration method.

Version 1.9.0

Major Changes

  • Reduction of Keck/DEIMOS now defaults to mosaics of red and blue detector pairs

  • Package data is no longer distributed via PyPI. Instead, we use astropy’s downloading/caching system as needed by the user

  • Significant refactor of object finding and 2D extraction algorithms

  • Added support for: - VLT FORS2 600z grism - VLT XShooter UVB arm

Minor Changes/Improvements

  • Default comb_id ordering matches the sorted file name in pypeit_setup

  • Save output wavelength calibration from pypeit_identify to the cache for direct reuse in data reduction.

  • The pypeit_identify GUI can now toggle between linear and log scaling of the arc spectrum flux.

  • Dark subtraction now ignores any difference in exposure time, by default

  • Added more flexible quicklook that can handle dithering.

  • Bug fixes in local sky subtraction and extraction

  • Fixed pypeit_setup issues due to bad LRIS headers.

  • Fixed a bug in 2d coadding when objects were not being identified.

  • Refactored 2d extraction.

  • Minor enhancements to pypeit_identify GUI

  • Refactoring of pypeit_show_wvcalib GUI

Instrument-specific Changes/Improvements

  • Improve Keck/KCWI automatic frame typing.

  • Added wavelength templates for - Keck/MOSFIRE (OH and arc lines) - bok_bc 300 grating template

  • Improved wavelength solution for Gemini-Nort E2V detector

  • Keck/DEIMOS now uses gain/RN values measured periodically by WMKO

  • Added enhancements and fixes for Keck lris red Mark4.

  • Added code to better parse Gemini/GNIRS dither sequences

Version 1.8.1

  • Various hotfixes

  • Include preliminary support for fluxing with archived SensFunc files for DEIMOS.

Version 1.8.0

Significant changes

  • Code to allow for mosaicing multiple detectors into a single reduction. This is now the default for Gemini GMOS and improves stability of wavelength calibration.

  • Introduces pypeit_parse_calib_id script

  • Refactored manual extraction

  • Update for LDT/DeVeny including support for binned data, use_header for reading arc lamps used from frames, and reid_arxiv templates for three additional gratings.

  • Slurps in and uses slitmask design for Keck/LRIS (limited usage)

  • Significant improvements in 2D coadding.

  • Scripts to explore the noise residuals in PypeIt

Datamodel changes and algorithmic improvements

  • Improved performance of L.A. Cosmic implementation

  • Now uses stars in alignment boxes for default calculation of slitmask offsets in DEIMOS reductions.

  • 2D wavelength calibration image now added to Flat output

  • Improved treatment of saturation.

  • Dark counts used for calculating the shot noise now includes measured dark images if provided.

  • Include sky model in 2nd pass of global sky subtraction (not for IR redux).

  • Skymask is now computed also for the maskdef_extract objects.

  • Added dedicated fwhm and boxcar_radius for maskdef_extract objects.

  • Added pypeit_version to the pypeit file header.

  • Set DEIMOS find_fwhm default to 0.8” in binned pixels.

  • Added row-dependent pattern-noise calculation for KCWI

Bug fixes

  • Fixed a bug about how maskdef_offset is assigned to each detector

  • Fixed 2Dcoadd spec bugs for central wavelength dithers.

Version 1.7.0

Major Changes

  • MOSFIRE improvements: - improved frame typing

    • ingestion of slitmask metadata for MOSFIRE with association of extracted spectra to object name and coordinates

    • extraction of undetected objects

    • incorporates dither pattern from file headers

  • Implements new Mark4 detector for Keck/LRISr; selected as the keck_lris_red_mark4 “spectrograph”

Minor Changes

  • Introduces pypeit_parse_calib_id script

  • Throw a warning if the chosen spectrograph has a header which does not match expectation

  • Pypeit can now read (currently for Keck DEIMOS only) the list of arc lamps from the header and use it for wavelength calibration.

  • Allow one to restrict the wavelength range of the arxiv template

  • Set DEIMOS FWHM default to 10 pixels

  • Fixed a bug in HolyGrail that did not allow for sigdetect and rms_wavelength to be slit dependent lists

Version 1.6.0

  • Improved basic image processing, particularly error propagation and documentation

  • Minor changes to Bias, Dark, and spec1d datamodel.

Version 1.5.0

  • Improvement to installation docs and scripts to ease installation of telluric and quick-look files.

  • Support dithered observations for DEIMOS.