What’s New in PypeIt
For a detailed log of code edits (including releases earlier than version 1.5.0), see CHANGELOG.
Main dependency bumps: numpy>=1.22, matplotlib>=3.7, ginga>=4.1.1, qtpy>=2.0.1
Functionality/Performance Improvements and Additions
Improvements to wavelength grids and masking in coadd routines.
Refactored coadding routines to work with lists to support coadding data from different setups.
Sensitivity function models can now be computed relative to the flat-field spectrum.
Improvements in 2D coaddition
Fix a bug in
pypeit_setup_coadd2dfor the output file name of the
Added possibility to specify more than one Science folder in
pypeit_coadd_2dspecincludes the detector number (similar to
pypeit_coadd_2dspecto exclude specific slits
Fix wrong RA and Dec for 2D coadded serendips
Allow wavelength calibrations for specific slits/orders to be redone (instead of adopting the value from a processed calibration frame); see new
Adds/Improves support for Gemini/GNIRS (IFU), Keck/KCRM, Keck/ESI, MDM/Modspec, Keck/HIRES, JWST
HIRES wavelength solution improvements galore
Improvements for Keck/LRIS
Generated wavelength templates for all the LRIS grism & grating
Added FeAr line list
Improved calibration association and frame typing
Improved and added documentation
metadata.pyincluding commenting out, in the pypeit file, files that have frametype None (this prevent
Added a function
check_spectrograph()(currently only defined for LRIS), that checks (during
pypeit_setup) if the selected spectrograph is the corrected one for the data used.
Added a script to convert a wavelength solution into something that can be placed in the reid archive.
Store user-generated wavelength solution in pypeit cache
Changed calibration frame naming as an attempt to avoid very long names for files with many calibration groups. Sequential numbers are reduced to a range; e.g.,
Instrumental FWHM map is calculated and output in
Change how masking is dealt with in extraction to fix a bug in how masks were being treated for echelle data
Refactored function that loads wavelength calibration line lists
Hotfix for GTC/OSIRIS lamp list
Hotfix for Arc1D stats annotations on the QA
Hotfix for metadata:
config_independent_frameswhen multiple configurations are being setup
support lists in
Hotfix for rebin (speed-up and conserves flux)
Hotfix for skysub regions GUI that used
Hotfix to stop pypeit_setup from crashing on data from lbt_luci1, lbt_luci2, magellan_fire, magellan_fire_long, p200_tspec, or vlt_sinfoni.
Hotfix to set BPM for each type of calibration file.
Fixed a bug in echelle coadding where the wrong coadded spectra were being used in final stacks.
Fix a bug in
spectrograph.select_detectors, where a list of
slitspatnumcould not be used.
Functionality/Performance Improvements and Additions
Allow user control of the local sky subtraction window
Implemented a resample algorithm when generating datacubes
Sensitivity function masking and output updates
Improve speed in ginga visualization of traces and added pypeit_chk_tilts. Note that this script uses an update of the tilts datamodel, so it will not work on older reductions.
Fixed a bug in the variance_model calculation for combined images.
Removed use of the term “master”, renamed to calibration frames/files. Default output directory for calibration frames is now
Calibrations. Calibration frames renamed from, e.g.,
Names of associated calibration frames now written to
Deprecate support for python 3.8 with PypeIt, allow python 3.11
Make pypeit_show_2dspec (somewhat) backwards compatible.
Added the option to disable strict version checking for 1d coadds.
Start to deprecate use of
os.pathin favor of
In PypeIt files, the
'calib'column is now always added to the pypeit file, regardless of whether or not you also request the
In PypeIt files, the calibration group numbers can now be anything, as long as there are no more than 63 unique integers.
pypeit_parse_calib_idscript, but improved the
.calibfile provided by
.calibfile is now always written, and provides a direct association between input raw files and output calibration files. Discussed in new docs.
Added coadd2d setup script
Added the possibility to use dither offsets saved in the header of the science frames for coadding 2D spectra (
dithoffmust be part of the spectrograph metadata).
Major quicklook updates; see documentation. ql_multislit.py deprecated.
Updates to reduction parameters for LDT/DeVeny
Hotfix to docs to ensure
pypeit_loadersapi doc is generated
Hotfix for KCWI when using alignment (aka ContBars) frames for the astrometric correction.
Include Gemini/GMOS mask design slurping and usage documentation
New Gemini/GMOS wavelength solution
Added NIRES tutorial doc
reid_arxiv templates for all MMTO Blue Channel gratings and for MMTO Binospec G600 and G1000
Added new specutils interface
Include the S/N of extracted spectra in the SpecObj datamodel
Various bug fixes and enhancements to mmt_bluechannel and mmt_binospec support
Fixed bugs when only performing calibrations and (1) calib groups are all set to ‘all’ or (2) anything other than ‘0’.
Implemented the upgraded GTC/OSIRIS+
Added support for more NOT/ALFOSC grisms as well as NOT recommended standards
Implemented the SOAR/Goodman (blue) M1 only
Modify install notes to allow python 3.10; python3.8 not longer explicitly supported
Allow for bad orders during extraction without crashing
Specify sphinx versions to correctly work with sphinx_rtd_theme
Fixed bug that caused crash of sensfunc routines using telluric grids in offline processing
Fixed error when showing flats in Ginga when the fine correction is not performed
Fixed keymap error when displaying GUIs
New docs on OneSpec
Functionality Improvements / User-level Updates
Refactor quicklook scripts
Added spectral flexure and reference frame corrections for IFU data
Allow separate sky frame to be used for sky subtraction with IFU data
Limit the images written to the Edges file to only the trace image, mask, and detector.
Limit LRISr header crashes
Parse Keck/NIRES dither patterns, similar to MOSFIRE
Started modifications and support for JWST NIRSpec
Minor / Under-the-hood Updates and Bug Fixes
Fixed bug that allowed science frames to be assigned to multiple instrument configurations
Fixed typo related to GitHub download for offline processing
OpenMP link fix
Enable boxcar_radius for manual extraction
Prevent flexure crash
Fixed error with deprecated numpy types
Improved optimization of bspline c code
Introduce BitMaskArray class to ease use of bitmasks
Fixed memory hogging by matplotlib when using version >= 3.6.1
Add ability for users to specify custom arc line lists for wavelength calibration, saved in the user’s PypeIt cache
Allow for the specification of a specific UVIS extinction file for sensitivity function computation and flux calibration.
Now more than one setup can be assigned to the same calibration frame, allowing to associate the same calibration frames to different science/standard frames, if desired.
Refactor and general update of documentation
Correctly associate calibrations with science data for MOSFIRE longslit and long2pos masks.
Automatically assign comb_id and bkg_id to MOSFIRE science data, using the information on the dither pattern.
Added Keck/NIRES frame-typing development doc.
Allow verbosity specification for various post-processing command-line scripts.
Refactor PypeIt input files. Main pypeit file remains the same, except that inclusion of leading and trailing | characters in the data table (required in previous versions) will now result in DeprecationWarnings. All post-processing scripts (coadding, fluxing, etc) must use the new format. See the main documentation pages.
Apply find_min_max when clipping the image for object finding
Mask bad detector regions for global sky flexure calculation
Added wavelength diagnostics to the spec2d output
Modify tweak_standard for Keck-MOSFIRE/J2
Detector structure correction included in flatfield calibration (Keck-KCWI only)
Addresses bug related to downloading from the reid_arxiv when using the reidentify wavelength calibration method.
Reduction of Keck/DEIMOS now defaults to mosaics of red and blue detector pairs
Package data is no longer distributed via PyPI. Instead, we use astropy’s downloading/caching system as needed by the user
Significant refactor of object finding and 2D extraction algorithms
Added support for: - VLT FORS2 600z grism - VLT XShooter UVB arm
Default comb_id ordering matches the sorted file name in pypeit_setup
Save output wavelength calibration from pypeit_identify to the cache for direct reuse in data reduction.
The pypeit_identify GUI can now toggle between linear and log scaling of the arc spectrum flux.
Dark subtraction now ignores any difference in exposure time, by default
Added more flexible quicklook that can handle dithering.
Bug fixes in local sky subtraction and extraction
Fixed pypeit_setup issues due to bad LRIS headers.
Fixed a bug in 2d coadding when objects were not being identified.
Refactored 2d extraction.
Minor enhancements to pypeit_identify GUI
Refactoring of pypeit_show_wvcalib GUI
Improve Keck/KCWI automatic frame typing.
Added wavelength templates for - Keck/MOSFIRE (OH and arc lines) - bok_bc 300 grating template
Improved wavelength solution for Gemini-Nort E2V detector
Keck/DEIMOS now uses gain/RN values measured periodically by WMKO
Added enhancements and fixes for Keck lris red Mark4.
Added code to better parse Gemini/GNIRS dither sequences
Include preliminary support for fluxing with archived SensFunc files for DEIMOS.
Code to allow for mosaicing multiple detectors into a single reduction. This is now the default for Gemini GMOS and improves stability of wavelength calibration.
Introduces pypeit_parse_calib_id script
Refactored manual extraction
Update for LDT/DeVeny including support for binned data, use_header for reading arc lamps used from frames, and reid_arxiv templates for three additional gratings.
Slurps in and uses slitmask design for Keck/LRIS (limited usage)
Significant improvements in 2D coadding.
Scripts to explore the noise residuals in PypeIt
Datamodel changes and algorithmic improvements
Improved performance of L.A. Cosmic implementation
Now uses stars in alignment boxes for default calculation of slitmask offsets in DEIMOS reductions.
2D wavelength calibration image now added to Flat output
Improved treatment of saturation.
Dark counts used for calculating the shot noise now includes measured dark images if provided.
Include sky model in 2nd pass of global sky subtraction (not for IR redux).
Skymask is now computed also for the maskdef_extract objects.
Added dedicated fwhm and boxcar_radius for maskdef_extract objects.
Added pypeit_version to the pypeit file header.
Set DEIMOS find_fwhm default to 0.8” in binned pixels.
Added row-dependent pattern-noise calculation for KCWI
Fixed a bug about how maskdef_offset is assigned to each detector
Fixed 2Dcoadd spec bugs for central wavelength dithers.
MOSFIRE improvements: - improved frame typing
ingestion of slitmask metadata for MOSFIRE with association of extracted spectra to object name and coordinates
extraction of undetected objects
incorporates dither pattern from file headers
Implements new Mark4 detector for Keck/LRISr; selected as the keck_lris_red_mark4 “spectrograph”
Introduces pypeit_parse_calib_id script
Throw a warning if the chosen spectrograph has a header which does not match expectation
Pypeit can now read (currently for Keck DEIMOS only) the list of arc lamps from the header and use it for wavelength calibration.
Allow one to restrict the wavelength range of the arxiv template
Set DEIMOS FWHM default to 10 pixels
Fixed a bug in HolyGrail that did not allow for sigdetect and rms_wavelength to be slit dependent lists
Improved basic image processing, particularly error propagation and documentation
Minor changes to Bias, Dark, and spec1d datamodel.
Improvement to installation docs and scripts to ease installation of telluric and quick-look files.
Support dithered observations for DEIMOS.