What’s New in PypeIt

For a detailed log of code edits (including releases earlier than version 1.5.0), see CHANGELOG.

Version 1.13.0

Funtionality/Performance Improvements and Additions

  • Allow user control of the local sky subtraction window

  • Implemented a resample algorithm when generating datacubes

  • Sensitivity function masking and output updates

  • Improve speed in ginga visualization of traces and added pypeit_chk_tilts. Note that this script uses an update of the tilts datamodel, so it will not work on older reductions.

  • Fixed a bug in the variance_model calculation for combined images.

Datamodel Changes

  • Removed use of the term “master”, renamed to calibration frames/files. Default output directory for calibration frames is now Calibrations. Calibration frames renamed from, e.g., MasterArc* to Arc*.

  • Names of associated calibration frames now written to spec2d file headers.

Dependency Changes

  • Deprecate support for python 3.8 with PypeIt, allow python 3.11

  • Make pypeit_show_2dspec (somewhat) backwards compatible.

  • Added the option to disable strict version checking for 1d coadds.

  • Start to deprecate use of os.path in favor of pathlib

Script Changes

  • In PypeIt files, the 'calib' column is now always added to the pypeit file, regardless of whether or not you also request the 'comb_id' and 'bkg_id' columns.

  • In PypeIt files, the calibration group numbers can now be anything, as long as there are no more than 63 unique integers.

  • Deprecated pypeit_parse_calib_id script, but improved the .calib file provided by pypeit_setup. The .calib file is now always written, and provides a direct association between input raw files and output calibration files. Discussed in new docs.

  • Added coadd2d setup script

  • Added the possibility to use dither offsets saved in the header of the science frames for coadding 2D spectra (dithoff must be part of the spectrograph metadata).

  • Major quicklook updates; see documentation. ql_multislit.py deprecated.

Instrument-specific Updates

  • Updates to reduction parameters for LDT/DeVeny

Hotfixes

  • Hotfix to docs to ensure pypeit_loaders api doc is generated

  • Hotfix for KCWI when using alignment (aka ContBars) frames for the astrometric correction.

Version 1.12.2

Instrument-specific Updates

  • Include Gemini/GMOS mask design slurping and usage documentation

  • New Gemini/GMOS wavelength solution

  • Added NIRES tutorial doc

  • reid_arxiv templates for all MMTO Blue Channel gratings and for MMTO Binospec G600 and G1000

Usage Updates

  • Added new specutils interface

Datamodel Updates

  • Include the S/N of extracted spectra in the SpecObj datamodel

Hotfixes

  • Various bug fixes and enhancements to mmt_bluechannel and mmt_binospec support

  • Fixed bugs when only performing calibrations and (1) calib groups are all set to ‘all’ or (2) anything other than ‘0’.

Version 1.12.1

Instrument-specific Updates

  • Implemented the upgraded GTC/OSIRIS+

  • Added support for more NOT/ALFOSC grisms as well as NOT recommended standards

  • Implemented the SOAR/Goodman (blue) M1 only

Minor Updates

  • Modify install notes to allow python 3.10; python3.8 not longer explicitly supported

  • Allow for bad orders during extraction without crashing

Hotfixes

  • Specify sphinx versions to correctly work with sphinx_rtd_theme

  • Fixed bug that caused crash of sensfunc routines using telluric grids in offline processing

  • Fixed error when showing flats in Ginga when the fine correction is not performed

  • Fixed keymap error when displaying GUIs

  • New docs on OneSpec

Version 1.12.0

Functionality Improvements / User-level Updates

  • Refactor quicklook scripts

  • Added spectral flexure and reference frame corrections for IFU data

  • Allow separate sky frame to be used for sky subtraction with IFU data

  • Limit the images written to the Edges file to only the trace image, mask, and detector.

Instrument-specific Updates

  • Limit LRISr header crashes

  • Parse Keck/NIRES dither patterns, similar to MOSFIRE

  • Started modifications and support for JWST NIRSpec

Minor / Under-the-hood Updates and Bug Fixes

  • Fixed bug that allowed science frames to be assigned to multiple instrument configurations

  • Fixed typo related to GitHub download for offline processing

  • OpenMP link fix

  • Enable boxcar_radius for manual extraction

  • Prevent flexure crash

  • Fixed error with deprecated numpy types

  • Improved optimization of bspline c code

  • Introduce BitMaskArray class to ease use of bitmasks

  • Fixed memory hogging by matplotlib when using version >= 3.6.1

Version 1.11.0

Significant Updates

  • Add ability for users to specify custom arc line lists for wavelength calibration, saved in the user’s PypeIt cache

  • Allow for the specification of a specific UVIS extinction file for sensitivity function computation and flux calibration.

  • Now more than one setup can be assigned to the same calibration frame, allowing to associate the same calibration frames to different science/standard frames, if desired.

  • Refactor and general update of documentation

Instrument-specific Updates

  • Correctly associate calibrations with science data for MOSFIRE longslit and long2pos masks.

  • Automatically assign comb_id and bkg_id to MOSFIRE science data, using the information on the dither pattern.

Minor

  • Added Keck/NIRES frame-typing development doc.

  • Allow verbosity specification for various post-processing command-line scripts.

Version 1.10.0

Major Changes

  • Refactor PypeIt input files. Main pypeit file remains the same, except that inclusion of leading and trailing | characters in the data table (required in previous versions) will now result in DeprecationWarnings. All post-processing scripts (coadding, fluxing, etc) must use the new format. See the main documentation pages.

Minor Changes/Improvements

  • Apply find_min_max when clipping the image for object finding

  • Mask bad detector regions for global sky flexure calculation

  • Added wavelength diagnostics to the spec2d output

Instrument-specific Changes/Improvements

  • Modify tweak_standard for Keck-MOSFIRE/J2

  • Detector structure correction included in flatfield calibration (Keck-KCWI only)

Version 1.9.1

Addresses bug related to downloading from the reid_arxiv when using the reidentify wavelength calibration method.

Version 1.9.0

Major Changes

  • Reduction of Keck/DEIMOS now defaults to mosaics of red and blue detector pairs

  • Package data is no longer distributed via PyPI. Instead, we use astropy’s downloading/caching system as needed by the user

  • Significant refactor of object finding and 2D extraction algorithms

  • Added support for: - VLT FORS2 600z grism - VLT XShooter UVB arm

Minor Changes/Improvements

  • Default comb_id ordering matches the sorted file name in pypeit_setup

  • Save output wavelength calibration from pypeit_identify to the cache for direct reuse in data reduction.

  • The pypeit_identify GUI can now toggle between linear and log scaling of the arc spectrum flux.

  • Dark subtraction now ignores any difference in exposure time, by default

  • Added more flexible quicklook that can handle dithering.

  • Bug fixes in local sky subtraction and extraction

  • Fixed pypeit_setup issues due to bad LRIS headers.

  • Fixed a bug in 2d coadding when objects were not being identified.

  • Refactored 2d extraction.

  • Minor enhancements to pypeit_identify GUI

  • Refactoring of pypeit_show_wvcalib GUI

Instrument-specific Changes/Improvements

  • Improve Keck/KCWI automatic frame typing.

  • Added wavelength templates for - Keck/MOSFIRE (OH and arc lines) - bok_bc 300 grating template

  • Improved wavelength solution for Gemini-Nort E2V detector

  • Keck/DEIMOS now uses gain/RN values measured periodically by WMKO

  • Added enhancements and fixes for Keck lris red Mark4.

  • Added code to better parse Gemini/GNIRS dither sequences

Version 1.8.1

  • Various hotfixes

  • Include preliminary support for fluxing with archived SensFunc files for DEIMOS.

Version 1.8.0

Significant changes

  • Code to allow for mosaicing multiple detectors into a single reduction. This is now the default for Gemini GMOS and improves stability of wavelength calibration.

  • Introduces pypeit_parse_calib_id script

  • Refactored manual extraction

  • Update for LDT/DeVeny including support for binned data, use_header for reading arc lamps used from frames, and reid_arxiv templates for three additional gratings.

  • Slurps in and uses slitmask design for Keck/LRIS (limited usage)

  • Significant improvements in 2D coadding.

  • Scripts to explore the noise residuals in PypeIt

Datamodel changes and algorithmic improvements

  • Improved performance of L.A. Cosmic implementation

  • Now uses stars in alignment boxes for default calculation of slitmask offsets in DEIMOS reductions.

  • 2D wavelength calibration image now added to Flat output

  • Improved treatment of saturation.

  • Dark counts used for calculating the shot noise now includes measured dark images if provided.

  • Include sky model in 2nd pass of global sky subtraction (not for IR redux).

  • Skymask is now computed also for the maskdef_extract objects.

  • Added dedicated fwhm and boxcar_radius for maskdef_extract objects.

  • Added pypeit_version to the pypeit file header.

  • Set DEIMOS find_fwhm default to 0.8” in binned pixels.

  • Added row-dependent pattern-noise calculation for KCWI

Bug fixes

  • Fixed a bug about how maskdef_offset is assigned to each detector

  • Fixed 2Dcoadd spec bugs for central wavelength dithers.

Version 1.7.0

Major Changes

  • MOSFIRE improvements: - improved frame typing

    • ingestion of slitmask metadata for MOSFIRE with association of extracted spectra to object name and coordinates

    • extraction of undetected objects

    • incorporates dither pattern from file headers

  • Implements new Mark4 detector for Keck/LRISr; selected as the keck_lris_red_mark4 “spectrograph”

Minor Changes

  • Introduces pypeit_parse_calib_id script

  • Throw a warning if the chosen spectrograph has a header which does not match expectation

  • Pypeit can now read (currently for Keck DEIMOS only) the list of arc lamps from the header and use it for wavelength calibration.

  • Allow one to restrict the wavelength range of the arxiv template

  • Set DEIMOS FWHM default to 10 pixels

  • Fixed a bug in HolyGrail that did not allow for sigdetect and rms_wavelength to be slit dependent lists

Version 1.6.0

  • Improved basic image processing, particularly error propagation and documentation

  • Minor changes to Bias, Dark, and spec1d datamodel.

Version 1.5.0

  • Improvement to installation docs and scripts to ease installation of telluric and quick-look files.

  • Support dithered observations for DEIMOS.