What’s New in PypeIt

Version 2.0.1

  • Add key-binding documentation from pypeit/display/pypeit_modes.py to the main documentation page, pypeit_show_1dspec.

  • Small changes to default parameter values for the APF/Levy spectrograph

  • Change an exception raised when a coding-level check failed that was meant to ensure consistency between calibration groups and science frames. This was causing unintended faults for APF/Levy data.

Version 2.0.0

Breaking Changes

  • FindObj parameters find_maxdev and find_extrap_npoly have been renamed to trace_maxdev and trace_extrap_npoly, respectively, to indicate their use in object tracing rather than object finding. No change to the functionality of these parameters has been made. Users with these parameters in their pre-existing PypeIt Input Files will need to change the parameter names in order to run said files.

Documentation Updates

  • Added tables with the list of available telluric model spectra.

Dependency Changes

  • Removed dependency on linetools package (details below).

Functionality/Performance Improvements and Additions

  • Added the possibility to use pypeit_sensfunc to compute a sensitivity function using the the 1D spectrum generated by pypeit_coadd_1dspec, i.e., the OneSpec datamodel. Note that this is only possible for non-echelle data.

  • Added a new class to interface with the atmospheric extinction data.

  • Significant refactor of interface with flux-calibration standards.

  • Addition of two new FindObj object tracing prameters trace_maxshift and trace_min_max that make it easier to work with spectral images that are subject to high atmospheric dispersion (former) and/or regions of the detector that extend beyond the detected object (latter).

  • Include an object tracing QA plot (<objname>_<det>_<slit>_obj_trace.png) in the collection of QA plots produced in the main PypeIt run.

  • Added the possibility to use the maskdesign information to set specmin and specmax in the edge tracing process for each slit (only if this information is available). Currently implemented only for Gemini/GMOS.

  • When creating a sensitivity function from a standard star reduced using the mosaic approach, the code now splits the standard star spec1d into the single detectors and, after the sensitivity function for each detector is computed, it splices them into a single sensfunc covering the full wavelength range.

  • Add the possibility to use more than one standard star spec1d to create the sensitivity function. This can be helpful in case of spectrally dithered observations.

Instrument-specific Updates

  • Improved support for LBT/LUCI, including updated detector parameters, support for the N3.75 camera, wavelength solutions for the G200 grating, and a basic non-linearity correction. Edge finding parameters have been better optimized, and spatial flexure correction has now been turned on.

  • Updated object tracing parameters for LDT/DeVeny to allow more robust extraction of objects observed at low elevation where atmospheric dispersion is significant.

  • Updated the gain and ronoise values for Gemini/GMOS spectrographs to account for the different values measured in each amplifier.

  • Enables MMT/Binospec reductions to match the slitmask design input to the detected slits, similar to procedures used for Keck/DEIMOS, Keck/MOSFIRE, etc.

Script Changes

  • Added a new script, pypeit_rectify_2dspec, which creates a rectified 2D spectral image. Each slit/order in the spec2d output file is rectified onto a common wavelength grid. This rectified 2D image must be used only for quick-look visualization purposes, and not for scientific analysis.

Testing

  • Added tests for some coadd functions.

Under-the-hood Improvements

  • Improved extension checking for post-processing scripts to explicitly allow PypeIt-produced “.fits” files regardless of a given spectrograph’s raw data extension.

Bug Fixes

  • Removed last vestiges of support for Python 3.10 in the form of datetime.UTC crutches left over from the start of support of Python 3.12.

  • Fixed a bug with configuration-specific parameters where the raw data FITS header was polled for metadata information used to set parameters rather than relying on the metadata in the .pypeit file. For some spectrographs, this required adding new columns to the .pypeit file format; to retain backwards- compatibility when an older .pypeit file without these columns is used, the code still pulls the desired metadata from the raw data FITS file as before.

  • Fixed Coadd2d filenames to remove spaces from objet names (as is done in the main PypeIt run).

AI Use

Deprecation of linetools dependency

We used Claude to aid in the removal of the linetools dependency from PypeIt with the following prompts:

  • On this deprecate_linetools branch, I wish to remove all imports of linetools. Can you confirm that I have successfully done so?

  • Yes, please do, but ignore the ones in the Documentation files

  • The test test_arc.py in tests/ is failing. Please suggest a fix. If you need to run it, use the “pypeit” environment in conda.

Additional by-hand changes were made to remove the linetools references from the docs, remove lingering instances of xspectrum1d, and fix remaining issues.

We also added tests for the new pypeit.utils.loadjson function using the following prompt (GPT-5 mini):

  • Please add test functions to test_utils.py that test the functionality of the pypeit.utils.loadjson function. As necessary, the test functions should create test JSON files in the pypeit/tests directory that are then deleted when each test is complete. The tests should cover usage with regular and gzip compressed files. The tests should also cover failure modes when the file does not exist, the file is not JSON format, and any of the loading functions fail.

  • Please make the following edits to your contribution. Remove the definition of TEST_DIR; instead simply save the files in the current working directory. Move all the new tests to the end of the file. Make sure all imports are at the top of the file, after any initial file docstring.

Version 1.18.1

  • Improves documentation for LDT/DeVeny

  • Removes C extension used to accelerate some bspline functions.

Version 1.18.0

Organization Updates

  • Governance policies drafted

Documentation Updates

  • Updated development guidelines with new procedures related to using forks. Apart from a small number of maintainers, code contributions are now expected to come from forks. Also, all the developer-specific installation details have been removed from the main installation doc.

  • Cleaned up documentation referring to the telluric atmospheric grids. Much of the old documentation had not been updated to reflect the PCA models, leading to some confusion.

Dependency Changes

  • Bump dependency to specutils>=2.0

Functionality/Performance Improvements and Additions

  • The WCS for datacubes now adopts the convention of North is up and East is left. In previous version of PypeIt, East was right.

  • Improved the code to compute the spatial flexure correction and added a QA plot to inspect the results. Two new parset parameters, spat_flexure_sigdetect and spat_flexure_vrange, were added to control the detection of the slit edges (used to compute the flexure) and the stretching of the image in the QA plot, respectively.

  • Setup GUI allows resizing panels when viewing pypeit files.

  • Setup GUI allows selecting which detector to view when viewing raw data.

  • Added backwards compatibilty for old frames taken with the Bok B&C

  • The object trace can now be iterated over with the find_numiterfit parameter.

  • Added EdgeTracePar parameter mask_off_detector to allow the user to mask spectral regions in each slit/order where more than 50% of the slit falls off the detector

  • Adds Subaru/FOCAS to the instrument suite

  • Added SensfuncPar parameter trim_std_pixs to allow the user to “trim” the standard star 1D spectrum by a user-defined number of pixels before computing the sensitivity function.

  • We include new, brighter blackbody standard stars, as reported in Cooke, Suzuki and Prochaska (2025).

  • We have removed J1255p1924 as a blackbody flux calibration standard star. Based on X-SHOOTER observations by Cooke, Suzuki and Prochaska (2025), this white dwarf star has Ca II absorption.

  • Allow the user to skip writing 2D science spectrum files. Note that this should only be invoked after careful inspection of a sample of 2D reduced spectra and when the user is completely satisfied of the quality of the extracted 1D spectra.

  • Added ExtractionPar parameter min_frac_prof to allow the user to set the minimum fraction of the object spatial profile that is not masked in the optimal extraction.

  • Changed the implementation on how to set the reference objects, used to compute spatial offsets and/or weights when combining 2D spectra (i.e., pypeit_coadd_2dspec). The user can now provide a list of reference objects (one per spec2d file) using the SPAT_PIXPOS_ID or ECH_FRACPOS_ID (for MultiSlit and Echelle data, respectively) attributes in SpecObj as unique identifiers for reference objects. These identifiers can be provided using the user_obj_ids parameter in the Coadd2DPar parsets. The user_obj parameter has been removed.

  • Restricted the range of stretch corrections to the wavelength solution in telluric fits to a more reasonable one to avoid catastrophic edge cases.

  • Passed the TelluricPar parameter resln_frac_bounds through to the tellfit script functions (qso_telluric, star_telluric, poly_telluric)

Instrument-specific Updates

  • Added wavelength template for Gemini GMOS-N/GMOS-S R150 grating.

  • Added support for the Levy spectrograph on the Automated Planet Finder at Lick Observatory (APF/Levy)

Script Changes

  • pypeit_setup_gui script has been removed. The Setup GUI can be started with pypeit_setup -G.

  • Added pypeit_run_to_calibstep to run PypeIt to an input calibration step. This script is useful to re-generate calibration files for a given configuration without running the full reduction.

  • Added finer debugging control for pypeit_trace_edges

  • Added "-p", "--param_block_file" option to pypeit_setup that allows the user to specify the name of a file containing pre-formatted user parameters to be added to the Parameter Block of the .pypeit file. This option is useful when project-specific parameters need to be repeatedly specified across many .pypeit files.

Datamodel Changes

  • The SpecObj datamodel has been updated (v1.1.12) to include the BOX_R_ASEC attribute, which contains the information about the size of the boxcar radius in arcsec. In addition, the already existing BOX_RADIUS attribute, which contains the boxcar radius in pixels, had the name changed to BOX_R_PIX.

  • The SpecObj datamodel has been updated (v1.1.13) to include the SPAT_PIXPOS_ID and ECH_FRACPOS_ID attributes, which are unique identifiers for the extracted objects from MultiSlit and Echelle data, respectively.

Under-the-hood Improvements

  • Updates the pypeit_trace_edges script to use lampoffflats, if they’re provided by the pypeit file passed to the script.

  • Moved deprecated, paper, presentations, and proposals directories into a new repository PypeIt-misc.

  • Abstracted generation of GitHub remote paths in the cache system to make sure functionality works when working in a fork of the code.

  • Basic maintenance on bspline: Removed *kwargs from instantiation call, removed redundant (placed vs. bkpts) arguments, removed unused to/from dict functionality, cleaned up construction of breakpoints, added docs.

Bug Fixes

  • Fix the WCS for the datacube generation. There was an offset in both spatial dimensions equal to half the field of view.

  • Fix the code for the extraction of the 1D flat spectrum, so that the spectrum is extracted even when pixelflat_model does not exist.

  • The variance cube previously had a cubic term, and this has been changed to a quadratic term. This has been cross-checked with simple subpixel calculations.

  • Fixes bug associated with trying to remove or add slits for reductions that mosaic multiple detectors.

  • Fix Setup GUI sizing issues on MacOS.

  • Fixed a bug with pypeit_identify when the number of echelle orders has more than two digits.

  • Fixed bug when defining manual extraction apertures for 2D coadds of echelle data

  • Fix a bug in the fluxed std star QA plot generated by pypeit_sensfunc, where the plot was showing an inconsistency between the observed and the true fluxes.

  • For Gemini GMOS, PypeIt now correctly uses mask definition files with tilted slits. (Before, it would simply fail with an error message if any slits were tilted.)

  • Fix a bug in the order_platescale() method for the keck_esi spectrograph, where the wrong binning was used to compute the platescale.

  • Fix a bug in how the box_width, showed in arcsec in the “spec1d*.txt” file, was computed for the echelle data. The value was previously not accounting for the order-specific plate scale.

  • Fix a bug in the core.findobj_skymask.ech_fill_in_orders() method, where attributes of objects only found on some orders were not properly propagated to the other orders.

  • Fixed a bug causing 2D coadding of SOAR/Goodman data to fault based on the extensions of the spec2d files.

Version 1.17.4

  • Hotfix for deprecation of certain behavior of numpy.fromstring in numpy >= 2.3.

Version 1.17.3

  • Hotfix for file name typo in data/standards/xshooter/xshooter_info.txt. See issue #1903.

Version 1.17.2

  • Hotfix for typo in wavetilts.py, changing legend.legendHandles to legend.legend_handles in use of matplotlib.

Version 1.17.1

  • Bumped to numpy>=1.26

  • Lifts restriction from python<3.13 to python<3.14

Version 1.17.0

Installation Changes

  • Significant expansion of PypeIt’s use of the cache system; see Additional Data and the PypeIt Cache. Important changes include that most cached files are now version dependent. When upgrading to new versions of PypeIt, users should delete their cache and start fresh.

Dependency Changes

  • Support added for numpy>=2.0.0; numpy<=2.0.0 should still be supported

  • Deprecated support for python 3.10

  • General update to dependencies to be roughly consistent with release of python 3.11

Functionality/Performance Improvements and Additions

  • Added the max_overlap parameter, which limits the set of new order traces added, to compensate for orders missed during automated edge-tracing, to those that have less than a given fractional overlap with adjacent orders.

  • Added the order_fitrej and order_outlier parameters used to set the sigma-clipping threshold used when fitting Legendre functions to the order widths and gaps.

  • Added the possibility to decide if the extracted standard star spectrum should be used as a crutch for tracing the object in the science frame (before it was done as default). This is done by setting the parameter use_std_trace in FindObjPar.

  • Now PypeIt can handle the case where “Standard star trace does not match the number of orders in the echelle data” both in run_pypeit and in pypeit_coadd_1dspec.

  • Added the functionality to use slitless flats to create pixelflats. Note: new frametype slitless_pixflat is added to the PypeIt frametype list.

  • The created pixelflats are stored in the reduction directory and in the PypeIt cache directory data/pixelflats.

  • Added a functionality that allows, when multiple frames are combined, to scale each frame to have the same mean value before combining. To use this functionality, the new parameter scale_mean should be set to True.

  • Added the possibility to use the parameter fwhm_fromlines also for the tilts calibration.

Instrument-specific Updates

  • Improved LRIS frame typing, including the typing of slitless flats and sky flats.

  • Improved HIRES frame typing and configuration setup.

  • Added support for Keck/KCWI BH3 grating configuration.

  • Updated the requirements of a spectral flip for KCWI (blue) data. If all amplifiers are used, the data will not be flipped in the spectral direction. Otherwise, the data will be flipped.

  • Added support for the (decommissioned) AAT/UHRF instrument

  • Updated X-Shooter detector gain and read noise to come from header, and updated plate scales to the most recent values from the manual. Detailed changes are:

    • NIR arm:

      • Platescale updated from 0.197 to 0.245 arcsec/pixel

      • Dark current updated from 0. to 72. e-/pixel/hr

      • Gain updated from 2.12 to 2.29 e-/DN

    • VIS arm:

      • Platescale updated from an order-dependent value, to being 0.154 arcsec/pixel for all orders

    • UVB arm:

      • Platescale updated from an order-dependent value, to being 0.164 arcsec/pixel for all orders

  • Add new P200/DBSP reid_arxiv template for 1200/7100 with D55 dichroic

  • Add B480 as a supported option for Gemini-S/GMOS

Script Changes

  • Modifications to the cache-related Installation Scripts to accommodate expansion of and changes to the cache system.

  • Added pypeit_clean_cache script to facilitate both viewing and removing files in the cache.

  • Changed the name of the multi-dimensional specdata to specdata_multi in pypeit_identify and improved the robustness of the saving dialog when calibrating single trace spectra.

  • Fixed a read-in error for the high resolution A0V PHOENIX model.

  • A new script, called pypeit_extract_datacube, allows 1D spectra of point sources to be extracted from datacubes.

  • The sensitivity function is now generated outside of datacube generation.

  • The grating_corr column is now used to select the correct grating correction file for each spec2d file when generating the datacube.

  • Added the --extr parameter in the pypeit_sensfunc script (also as a SensFuncPar) to allow the user to specify the extraction method to use when computing the sensitivity function (before only optimal extraction was used).

  • Added pypeit_show_pixflat script to inspect the (slitless) pixel flat generated during the reduction and stored in data/pixelflats.

  • Added pypeit_chk_flexure script to check both spatial and spectral flexure applied to the reduced data.

  • Treatment of file names is now more formal. Compression signatures are now considered, and filename matching is now more strict.

  • Removed --spec_samp_fact and --spat_samp_fact command line options from pypeit_coadd_2d. These options are now parameters in Coadd2dPar.

  • pypeit_show_2dspec now shows the first available detector in the 2D spectrum by default. The user can specify the detector to show with the --det option.

  • Added --removetrace command line option to pypeit_ql to not show the object trace when displaying the 2D spectrum.

  • Change the default value for --skip_display in pypeit_ql to True.

Datamodel Changes

  • Adjusted spec1d datamodel to enable use with UVES_popler GUI tool

Under-the-hood Improvements

  • Introduced PypeItDataPaths to handle all interactions with the pypeit/data directory, which provides a unified interface for accessing on-disk and cached files.

  • When adding missing orders, the full syncing procedure is no longer performed. The code now only checks that the edges are still synced after the missed orders are added.

  • When detecting overlapping orders/slits, the code now forces each edge used to have been directly detected; i.e., if an edge is inserted, the fact that the resulting slit is abnormally short should not trigger the overlap detection.

  • Improved the QA plot resulting from fitting order widths and gaps as a function of spatial position.

  • Updated general raw image reader so that it correctly accounts for spectrographs that read the data and overscan sections directly from the file headers.

Bug Fixes

  • Fix “The system cannot find the file specified” errors when installing on Windows.

  • Fixed a fault caused when all frames in a pypeit file are identified as being part of all calibration groups.

  • Allow for empty 2D wavecal solution in HDU extension of WaveCalib file

  • Fixed a bug in the ginga display function, when the user doesn’t provide the trc_name argument.

  • Fix a MAJOR BUT SUBTLE bug in the use of numpy.argsort. When using numpy.argsort the parameter kind='stable' should be used to ensure that a sorting algorithm more robust than “quicksort” is used.

  • Fix error “ValueError: setting an array element with a sequence. The requested array has an inhomogeneous shape after 1 dimensions…” occurring when unpacking the SpecObj spectrum but having an attribute of the SpecObj object that is None.

  • Fixed a hidden bug that was causing the spatial flexure to fail. The bug was in the SlitTraceBitMask class, where the function exclude_for_flexure() was not returning the 'BOXSLIT' flag.

  • Fix a bug in pypeit_coadd_2d related to how the binning was taken into account in the mask definition, and in the calculation of the offset between frames.

  • Fix bug when trying to open mosaic data from previous versions; version checking flag was not being propagated.

Version 1.16.0

Installation Changes

Dependency Changes

  • PypeIt now only supports python versions >=3.10, numpy>=1.23, and ginga>=5.1.0

  • sphinx-rtd-theme updated to 2.0, along with relevant dependencies

Functionality/Performance Improvements and Additions

  • Allow the selection of a secondary filter option for all spectrographs that need it; see new filter2 parameter.

  • When dealing with bkg_redux reduction (i.e., A-B), a sky model (called bkg_redux_skymodel) is generated using the A frame, while the existing skymodel is generated using the A-B frame. This allows to have an actual 1D extracted sky spectrum and to perform the flexure correction for bkg_redux reduction.

  • Various improvements in the flexure correction and added the possibility to use a modeled archive sky spectrum generated with pypeit.wavemodel.nearIR_modelsky() to perform the flexure correction.

  • Updated the list of available CALSPEC standard star with the latest version from the CALSPEC website.

Instrument-specific Updates

  • Improvements for SOAR/Goodman

    • Added WAVMODE header keyword to configuration keys and raw header cards.

  • Added example of improved wavelength solution for GEMINI/GMOS

  • Introduces reduction pipeline for high-resolution modes of Keck/NIRSPEC

  • Expanded the template for shane_kast_red and the 1200/5000 grating

  • Fixed and improved the wavelength template magellan_fire_long

  • Updated Gemini/GMOS-S with the new detector parameters for the Hamamatsu CCDs replaced on 2023-12-14

Script Changes

  • When interactively fitting a single peak with the pypeit_identify script (i.e. using the i command), line centroids can now be both updated and added interactively.

Datamodel Changes

  • Spec2DObj now has an additional image, called bkg_redux_skymodel, which is the “actual” sky model in the case of bkg_redux reduction (i.e., dithered observations). This image is not generated if bkg_redux is False. In any case, the skymodel is always generated.

Bug Fixes

  • The subpixel algorithm used to resample datacubes that are generated by pypeit_coadd_datacube produced error cubes that were not properly propagating the noise. The error cubes of the NGP algorithm were unaffected. The error cubes are now regularly inspected with vet tests to ensure the error cubes are reliable.

  • Fix a bug (introduced in a recent PR) that was generating an error if less than 2 spec1d files were used with pypeit_coadd_1dspec. Now the script can be run with only one file (as it was before).

  • Allow spec_flex_shift() to take as input either the name of an archive sky spectrum or directly a sky spectrum. This fixes a bug in the flexure correction for SlicerIFU.

  • The –show option of the pypeit_coadd_1dspec script was not properly displaying the x-axis of the plot. Both the top and bottom panels now scale together.

  • Fix a bug in pypeit_sensfunc that was causing the script to crash because par[‘fluxcalib’] was not being passed to sensfunc.SensFunc.get_instance().

Version 1.15.0

Installation Changes

  • PypeIt will no longer be available through conda-forge and must be installed via pip from PyPI. Users may still use conda for management of PypeIt’s virtual environment.

Dependency Changes

  • Removes use of deprecated pkg_resources.

  • Require version >=1.12 for specutils.

  • Add psutil as a dependency for [dev] installs.

  • Add pytest-qt as a dependency for [dev] installs.

  • Require version >=0.3.1 for linetools

Functionality/Performance Improvements and Additions

  • Started the development of instrument-specific scattered light removal. In this release, we only model KCWI/KCRM scattered light.

  • Turned on usage of the use_2dmodel_mask parameter for echelle spectrographs

  • Added support for Keck/KCRM RL data reduction.

  • Allow one to turn off reinit of reduce_bpm in global_skysub and made this the default for the final pass

  • Allow pypeit Spectrum1D loaders to circumvent the requirement that reduced spectrum have monotonically increasing wavelength data. Non-monotonic wavelength data usually indicate a problem with the data reduction, but this at least lets users ingest the spectrum.

  • Add a sensible error message to the pypeit Spectrum1D loaders in the event a user inadvertently tries to use Spectrum1D instead of SpectrumList for a spec1d file.

  • Enabled interpolation and extrapolation of missing echelle orders. This is currently only used for Keck/HIRES, but the code is general.

  • Allow the specification of wavelength limits on the flexure cross-correlation against a sky template to limit the effect of unconstrained wavelength solutions at the ends of the spectrum.

  • Change overscan_method = 'polynomial' to overscan_method = 'chebyshev' to reflect the type of polynomial being fit. Both names point to the same code, but the name 'polynomial' is deprecated and will be removed in the future.

  • Introduced PCA method for telluric corrections

  • Added slicer subpixel sampling for DataCube generation

  • Added trace_rms_tol parameter for edge tracing, which helps guard against poorly constrained traces for spectrally truncated slits/orders.

Instrument-specific Updates

  • Add support for the R4K detector for MDM OSMOS

  • Updated archival sensitivity functions for DEIMOS 1200G, 800G, and 600ZD gratings.

  • Keck/KCWI and Keck/KCRM: Turned on polynomial correction for sky subtraction.

  • We now support the reduction of VLT/FORS2 data taken in MOS mode.

  • Fixed fluxing file format in the Keck-DEIMOS HOWTO documentation.

  • LDT/DeVeny: default parameter & wavelength template updates, add instrument- specific line lists, use SCITARG for target name if none entered in the LOUI.

  • Added support for NTT/EFOSC2 Gr#4

  • Keck/KCWI: FeAr are used for wavelength calibration, while the ThAr lamp is used to determine the wavelength tilt.

  • Improvement to the wavelength calibration for Keck/HIRES. New arxiv templates added. Templates available for orders 35-117.

  • Various updates to default parameters for Keck/HIRES.

  • Added Keck-LRIS HOWTO documentation.

Script Changes

  • When making datacubes, the user can select a separate frame to use for the sky subtraction. In this case, it is the processed data that will be used for sky subtraction (akin to nodding).

  • Allow a list of files to be passed to pypeit_chk_wavecalib.

  • A new script has been written (chk_scattlight) to check the generation of the scattered light calibration frame model.

  • Added the pypeit_edge_inspector script, which allows users to edit the edge traces by-hand.

  • pypeit_parse_slits now prints also the echelle order number, if applicable.

  • Column SpatID in the output of pypeit_chk_wavecalib changed to SpatOrderID and now show the echelle order number, if applicable, otherwise the slit number.

  • pypeit_chk_edges now load SlitTraceSet (if available) to be able to overlay the echelle order numbers.

  • Added a new GUI for creating and editing PypeIt input files: pypeit_setup_gui

  • Added a -G option to pypeit_setup and pypeit_obslog that will start the new Setup GUI.

  • Improvements and bug fixes for how the mask is displayed by pypeit_show_2dspec, and added pypeit_print_bpm to allow for a quick report on the flags associated with a given bit value.

Datamodel Changes

  • A wavelength array is now stored for DataCube()

  • WaveCalib and WaveFit datacontainers now store information about echelle order number, if applicable.

  • Change to how SlitTraceSet datamodel stores and checks bit flag values.

Under-the-hood Improvements

  • The CoAdd3D code has been refactored into a series of core modules and PypeIt-specific routines.

  • Polynomial overscan fitting now uses the numpy.polynomial.Chebyshev class instead of the deprecated numpy.polyfit() function.

  • Improvements to echelle wavelength calibration. Code changes in: pypeit/core/wavecal/wvutils.py, pypeit/core/wavecal/echelle.py, pypeit/core/wavecal/autoid.py, pypeit/wavecalib.py.

  • Fixed some failure modes when matching measured order positions to expected positions for fixed-format echelle spectrographs.

  • More extensive propagation of turning off datamodel version checking (using try_old and chk_version) throughout the code.

Bug Fixes

  • Fixed bug associated with finding more than one file with the same name (but presumably different extensions).

  • Fixed differential atmospheric refraction (DAR) correction bug. This bug affected datacubes combined using CoAdd3D(). Previously, the DAR was being computed, applied, and then later overwritten. The new routine is faster and more accurate.

  • Fixed a bug associated with an incorrect date for the transition to the Mark4 detector for Keck/LRIS RED.

  • Fixed a bug with the pypeit_identify script when using echelle data. Previously, the sigdetect parameter was a list of all orders, instead of a single value.

  • Fixed a bug with the GUI pypeit_skysub_regions. Previously, the calib header information was not included, and this led to a calibration error.

  • Corrected the binning scaling for BOXCAR_RADIUS in the object-finding algorithm. Previously, the platescale was multiplied by the spectral binning and not the spatial binning.

  • Aligned the documentation and parameters with the code for polynomial overscan subtraction. Docs/params specified 3 values to be passed in overscan_par for overscan_method = 'polynomial', but the code only uses one (poly order).

  • Fixed duplicate object bug reported in Issue #1690.

Version 1.14.0

Dependency Changes

  • Main dependency bumps: numpy>=1.22, matplotlib>=3.7, ginga>=4.1.1, qtpy>=2.0.1

Functionality/Performance Improvements and Additions

  • Improvements to wavelength grids and masking in coadd routines.

  • Refactored coadding routines to work with lists to support coadding data from different setups.

  • Sensitivity function models can now be computed relative to the flat-field spectrum.

  • Improvements in 2D coaddition

    • Fix a bug in pypeit_setup_coadd2d for the output file name of the .coadd2d file

    • Added possibility to specify more than one Science folder in pypeit_setup_coadd2d

    • Now only_slits parameter in pypeit_coadd_2dspec includes the detector number (similar to slitspatnum)

    • Added exclude_slits parameter in pypeit_coadd_2dspec to exclude specific slits

    • Fix wrong RA and Dec for 2D coadded serendips

  • Allow wavelength calibrations for specific slits/orders to be redone (instead of adopting the value from a processed calibration frame); see new redo_slits parameter.

Instrument-specific Updates

  • Adds/Improves support for Gemini/GNIRS (IFU), Keck/KCRM, Keck/ESI, MDM/Modspec, Keck/HIRES, JWST

  • HIRES wavelength solution improvements galore

  • Improvements for Keck/LRIS

    • Generated wavelength templates for all the LRIS grism & grating

    • Added FeAr line list

    • Improved calibration association and frame typing

    • Improved and added documentation

    • Changes to metadata.py including commenting out, in the pypeit file, files that have frametype None (this prevent run_pypeit to crash)

    • Added a function check_spectrograph() (currently only defined for LRIS), that checks (during pypeit_setup) if the selected spectrograph is the corrected one for the data used.

Script Changes

  • Added a script to convert a wavelength solution into something that can be placed in the reid archive.

  • Store user-generated wavelength solution in pypeit cache

Datamodel Changes

  • Changed calibration frame naming as an attempt to avoid very long names for files with many calibration groups. Sequential numbers are reduced to a range; e.g., '0-1-2-3-4' becomes '0+4' and '3-5-6-10-11-12-15-18-19' becomes '3-5+6-10+12-15-18+19'

  • Instrumental FWHM map is calculated and output in Calibrations and spec1d files.

Under-the-hood Improvements

  • Change how masking is dealt with in extraction to fix a bug in how masks were being treated for echelle data

  • Refactored function that loads wavelength calibration line lists

Bug Fixes

  • Hotfix for GTC/OSIRIS lamp list

  • Hotfix for Arc1D stats annotations on the QA

  • Hotfix for metadata:

    • correctly set config_independent_frames when multiple configurations are being setup

    • support lists in config_independent_frames

  • Hotfix for rebin (speed-up and conserves flux)

  • Hotfix for skysub regions GUI that used np.bool

  • Hotfix to stop pypeit_setup from crashing on data from lbt_luci1, lbt_luci2, magellan_fire, magellan_fire_long, p200_tspec, or vlt_sinfoni.

  • Hotfix to set BPM for each type of calibration file.

  • Fixed a bug in echelle coadding where the wrong coadded spectra were being used in final stacks.

  • Fix a bug in spectrograph.select_detectors, where a list of slitspatnum could not be used.

Version 1.13.0

Functionality/Performance Improvements and Additions

  • Allow user control of the local sky subtraction window

  • Implemented a resample algorithm when generating datacubes

  • Sensitivity function masking and output updates

  • Improve speed in ginga visualization of traces and added pypeit_chk_tilts. Note that this script uses an update of the tilts datamodel, so it will not work on older reductions.

  • Fixed a bug in the variance_model calculation for combined images.

Datamodel Changes

  • Removed use of the term “master”, renamed to calibration frames/files. Default output directory for calibration frames is now Calibrations. Calibration frames renamed from, e.g., MasterArc* to Arc*.

  • Names of associated calibration frames now written to spec2d file headers.

Dependency Changes

  • Deprecate support for python 3.8 with PypeIt, allow python 3.11

  • Make pypeit_show_2dspec (somewhat) backwards compatible.

  • Added the option to disable strict version checking for 1d coadds.

  • Start to deprecate use of os.path in favor of pathlib

Script Changes

  • In PypeIt files, the 'calib' column is now always added to the pypeit file, regardless of whether or not you also request the 'comb_id' and 'bkg_id' columns.

  • In PypeIt files, the calibration group numbers can now be anything, as long as there are no more than 63 unique integers.

  • Deprecated pypeit_parse_calib_id script, but improved the .calib file provided by pypeit_setup. The .calib file is now always written, and provides a direct association between input raw files and output calibration files. Discussed in new docs.

  • Added coadd2d setup script

  • Added the possibility to use dither offsets saved in the header of the science frames for coadding 2D spectra (dithoff must be part of the spectrograph metadata).

  • Major quicklook updates; see documentation. ql_multislit.py deprecated.

Instrument-specific Updates

  • Updates to reduction parameters for LDT/DeVeny

Hotfixes

  • Hotfix to docs to ensure pypeit_loaders api doc is generated

  • Hotfix for KCWI when using alignment (aka ContBars) frames for the astrometric correction.

Version 1.12.2

Instrument-specific Updates

  • Include Gemini/GMOS mask design slurping and usage documentation

  • New Gemini/GMOS wavelength solution

  • Added NIRES tutorial doc

  • reid_arxiv templates for all MMTO Blue Channel gratings and for MMTO Binospec G600 and G1000

Usage Updates

  • Added new specutils interface

Datamodel Updates

  • Include the S/N of extracted spectra in the SpecObj datamodel

Hotfixes

  • Various bug fixes and enhancements to mmt_bluechannel and mmt_binospec support

  • Fixed bugs when only performing calibrations and (1) calib groups are all set to ‘all’ or (2) anything other than ‘0’.

Version 1.12.1

Instrument-specific Updates

  • Implemented the upgraded GTC/OSIRIS+

  • Added support for more NOT/ALFOSC grisms as well as NOT recommended standards

  • Implemented the SOAR/Goodman (blue) M1 only

Minor Updates

  • Modify install notes to allow python 3.10; python3.8 not longer explicitly supported

  • Allow for bad orders during extraction without crashing

Hotfixes

  • Specify sphinx versions to correctly work with sphinx_rtd_theme

  • Fixed bug that caused crash of sensfunc routines using telluric grids in offline processing

  • Fixed error when showing flats in Ginga when the fine correction is not performed

  • Fixed keymap error when displaying GUIs

  • New docs on OneSpec

Version 1.12.0

Functionality Improvements / User-level Updates

  • Refactor quicklook scripts

  • Added spectral flexure and reference frame corrections for IFU data

  • Allow separate sky frame to be used for sky subtraction with IFU data

  • Limit the images written to the Edges file to only the trace image, mask, and detector.

Instrument-specific Updates

  • Limit LRISr header crashes

  • Parse Keck/NIRES dither patterns, similar to MOSFIRE

  • Started modifications and support for JWST NIRSpec

Minor / Under-the-hood Updates and Bug Fixes

  • Fixed bug that allowed science frames to be assigned to multiple instrument configurations

  • Fixed typo related to GitHub download for offline processing

  • OpenMP link fix

  • Enable boxcar_radius for manual extraction

  • Prevent flexure crash

  • Fixed error with deprecated numpy types

  • Improved optimization of bspline c code

  • Introduce BitMaskArray class to ease use of bitmasks

  • Fixed memory hogging by matplotlib when using version >= 3.6.1

Version 1.11.0

Significant Updates

  • Add ability for users to specify custom arc line lists for wavelength calibration, saved in the user’s PypeIt cache

  • Allow for the specification of a specific UVIS extinction file for sensitivity function computation and flux calibration.

  • Now more than one setup can be assigned to the same calibration frame, allowing to associate the same calibration frames to different science/standard frames, if desired.

  • Refactor and general update of documentation

Instrument-specific Updates

  • Correctly associate calibrations with science data for MOSFIRE longslit and long2pos masks.

  • Automatically assign comb_id and bkg_id to MOSFIRE science data, using the information on the dither pattern.

Minor

  • Added Keck/NIRES frame-typing development doc.

  • Allow verbosity specification for various post-processing command-line scripts.

Version 1.10.0

Major Changes

  • Refactor PypeIt input files. Main pypeit file remains the same, except that inclusion of leading and trailing | characters in the data table (required in previous versions) will now result in DeprecationWarnings. All post-processing scripts (coadding, fluxing, etc) must use the new format. See the main documentation pages.

Minor Changes/Improvements

  • Apply find_min_max when clipping the image for object finding

  • Mask bad detector regions for global sky flexure calculation

  • Added wavelength diagnostics to the spec2d output

Instrument-specific Changes/Improvements

  • Modify tweak_standard for Keck-MOSFIRE/J2

  • Detector structure correction included in flatfield calibration (Keck-KCWI only)

Version 1.9.1

Addresses bug related to downloading from the reid_arxiv when using the reidentify wavelength calibration method.

Version 1.9.0

Major Changes

  • Reduction of Keck/DEIMOS now defaults to mosaics of red and blue detector pairs

  • Package data is no longer distributed via PyPI. Instead, we use astropy’s downloading/caching system as needed by the user

  • Significant refactor of object finding and 2D extraction algorithms

  • Added support for: - VLT FORS2 600z grism - VLT XShooter UVB arm

Minor Changes/Improvements

  • Default comb_id ordering matches the sorted file name in pypeit_setup

  • Save output wavelength calibration from pypeit_identify to the cache for direct reuse in data reduction.

  • The pypeit_identify GUI can now toggle between linear and log scaling of the arc spectrum flux.

  • Dark subtraction now ignores any difference in exposure time, by default

  • Added more flexible quicklook that can handle dithering.

  • Bug fixes in local sky subtraction and extraction

  • Fixed pypeit_setup issues due to bad LRIS headers.

  • Fixed a bug in 2d coadding when objects were not being identified.

  • Refactored 2d extraction.

  • Minor enhancements to pypeit_identify GUI

  • Refactoring of pypeit_show_wvcalib GUI

Instrument-specific Changes/Improvements

  • Improve Keck/KCWI automatic frame typing.

  • Added wavelength templates for - Keck/MOSFIRE (OH and arc lines) - bok_bc 300 grating template

  • Improved wavelength solution for Gemini-Nort E2V detector

  • Keck/DEIMOS now uses gain/RN values measured periodically by WMKO

  • Added enhancements and fixes for Keck lris red Mark4.

  • Added code to better parse Gemini/GNIRS dither sequences

Version 1.8.1

  • Various hotfixes

  • Include preliminary support for fluxing with archived SensFunc files for DEIMOS.

Version 1.8.0

Significant changes

  • Code to allow for mosaicing multiple detectors into a single reduction. This is now the default for Gemini GMOS and improves stability of wavelength calibration.

  • Introduces pypeit_parse_calib_id script

  • Refactored manual extraction

  • Update for LDT/DeVeny including support for binned data, use_header for reading arc lamps used from frames, and reid_arxiv templates for three additional gratings.

  • Slurps in and uses slitmask design for Keck/LRIS (limited usage)

  • Significant improvements in 2D coadding.

  • Scripts to explore the noise residuals in PypeIt

Datamodel changes and algorithmic improvements

  • Improved performance of L.A. Cosmic implementation

  • Now uses stars in alignment boxes for default calculation of slitmask offsets in DEIMOS reductions.

  • 2D wavelength calibration image now added to Flat output

  • Improved treatment of saturation.

  • Dark counts used for calculating the shot noise now includes measured dark images if provided.

  • Include sky model in 2nd pass of global sky subtraction (not for IR redux).

  • Skymask is now computed also for the maskdef_extract objects.

  • Added dedicated fwhm and boxcar_radius for maskdef_extract objects.

  • Added pypeit_version to the pypeit file header.

  • Set DEIMOS find_fwhm default to 0.8” in binned pixels.

  • Added row-dependent pattern-noise calculation for KCWI

Bug fixes

  • Fixed a bug about how maskdef_offset is assigned to each detector

  • Fixed 2Dcoadd spec bugs for central wavelength dithers.

Version 1.7.0

Major Changes

  • MOSFIRE improvements: - improved frame typing

    • ingestion of slitmask metadata for MOSFIRE with association of extracted spectra to object name and coordinates

    • extraction of undetected objects

    • incorporates dither pattern from file headers

  • Implements new Mark4 detector for Keck/LRISr; selected as the keck_lris_red_mark4 “spectrograph”

Minor Changes

  • Introduces pypeit_parse_calib_id script

  • Throw a warning if the chosen spectrograph has a header which does not match expectation

  • Pypeit can now read (currently for Keck DEIMOS only) the list of arc lamps from the header and use it for wavelength calibration.

  • Allow one to restrict the wavelength range of the arxiv template

  • Set DEIMOS FWHM default to 10 pixels

  • Fixed a bug in HolyGrail that did not allow for sigdetect and rms_wavelength to be slit dependent lists

Version 1.6.0

  • Improved basic image processing, particularly error propagation and documentation

  • Minor changes to Bias, Dark, and spec1d datamodel.

Version 1.5.0

  • Improvement to installation docs and scripts to ease installation of telluric and quick-look files.

  • Support dithered observations for DEIMOS.

Starting after version 1.14.0, the logged changes are now only recorded by these release docs. Before that, a detailed log of code edits (including releases earlier than version 1.5.0) was kept in a single location; see the CHANGELOG.