Installation

Warning

Apple Silicon users who are having issues installing PypeIt may need to set up an environment configured for x86-64. See User Installation on Apple Silicon-based Macs for detailed steps. If that also fails, please Submit an issue and/or reach out to our user Community.

Below, we provide detailed instructions for installing PypeIt. For troubleshooting, please consult the PypeIt Community and/or submit an issue on GitHub.

User Installation

Setup a clean python environment

PypeIt is available from the Python Package Index (PyPI) and is installed via pip. This process also installs and/or upgrades PypeIt’s Package Dependencies, and for this reason you should always first set up a clean python environment in which to install PypeIt. This mitigates any possible dependency conflicts with other packages you use.

You can set up a new python environment using either conda:

conda create -n pypeit python=3.11
conda activate pypeit

or venv:

python -m venv pypeit
source pypeit/bin/activate

See the Managing Environments with Conda and/or the venv documentation for more details. The conda installation method described below creates an environment for you.

Install via pip

To install the latest release of PypeIt and its required dependencies, within your virtual environment execute:

pip install pypeit

Optional Dependencies

PypeIt has a few optional dependencies that improve and/or expand functionality.

  • If you are generating datacubes (and performing an astrometric correction), you will also need the scikit-image package. It can be installed by including it in the optional dependencies, e.g.:

    pip install "pypeit[scikit-image]"

  • To take advantage of an interface that allows you to ingest PypeIt outputs into its Spectrum1D and SpectrumList objects (see Spec1D Output), you can include specutils in the installation like so:

    pip install "pypeit[specutils]"

Note

Whether or not it is correct syntax to use the quotes in the commands above depends on your shell. The above commands are specific to ZShell, whereas you don’t need the quotes in Bash. But, in any case, you should avoid copying these commands from your browser since the unicode for quotation marks may not be correct, leading to errors when they are directly pasted into a terminal window.

Install via conda

conda is a popular and widely-used package and environment manager. We provide a YAML file that can be used to set up the virtual environment for you. This file creates a conda environment called pypeit, and then installs pypeit, all of the required dependencies, and the optional dependency specutils via pip. To use this:

  1. Download environment.yml.

  2. Create the conda environment and install pypeit into it:

    conda env create -f environment.yml

  3. Activate it:

    conda activate pypeit

  4. Verify that the new environment was installed correctly and contains pypeit:

    conda list

    Most of the packages listed will show as coming from the pypi channel.

This environment should now be ready to use and contain the latest official pypeit release.

Upgrading to a new version of PypeIt

Since either installation method above ultimately uses pip to install PypeIt, upgrading the package should simply be a matter of executing:

pip install pypeit --upgrade

If this causes problems (e.g., a new PypeIt script is unavailable or you encounter script errors), first try uninstalling (pip uninstall pypeit) and then reinstalling. There are two important things to keep in mind when upgrading:

  • PypeIt datamodels are not necessarily backwards-compatible. This means that, e.g., pypeit_show_2dspec may fault when trying to view spec2d* files produced with your existing PypeIt version after upgrading to a new version. The best approach is to always re-reduce data you’re still working with anytime you update PypeIt.

  • As opposed to earlier versions of PypeIt, the cached files are now version-specific. Every time you upgrade pypeit, we recommend deleting your existing cache and starting fresh! The only caveat to this is if you are actively using multiple versions of PypeIt, meaning you will still be using old versions of the cached files. Otherwise, you will end up with multiple versions of the same file on disk. Importantly, the code also considers local files you have installed (using, e.g., pypeit_install_linelist) to be version specific. If you have installed such files, you will need to re-install them after upgrading.

If you have locally installed files, your upgrade may look something like this:

pypeit_clean_cache --remove_all
pip install pypeit --upgrade
pypeit_install_linelist /path/to/my/linelists/*_lines.dat

Note

If you find particular data files useful for your reductions, please consider issuing a PR to include them file in the PypeIt repository. This helps the community, and it means you’ll avoid these upgrading complications.

User Installation on Apple Silicon-based Macs

Both the pip and conda installation methods should be successful for Macs that uses Apple Silicon processors. The full Anaconda installers also now include support for Apple Silicon.

If the above does not work, you may need to set up a virtual environment configured for x86-64:

  1. Install miniconda.

  2. Use conda to create an environment configured for x86-64:

    conda create -n pypeit -y
conda activate pypeit
conda config --env --set subdir osx-64

  3. Install whichever version of Python you want (e.g.):

    conda install python=3.11

  4. Install PypeIt via pip as above.

Solutions/Recommendations/Feedback for these installation options are welcome; please Submit an issue.

User Installation on Windows

  1. Download Python for Windows.

  2. Run the installer.

    • Make sure “Add python.exe to Path” or “Add Python to environment variables” is selected before installing.

    • If you have Admin privileges click “Disable path length limit” after the installation succeeds.

  3. Download and run the Visual Studio build tools installer.

    • Only “Desktop Development with C++” needs to be checked.

    • Click install

  4. Create a virtual environment as in Setup a clean python environment and install PypeIt as described above.

If running python on Windows brings up a window for the Microsoft Store you may want to change the application alias. This is under Settings -> Apps -> App execution aliases on Windows 10 and Settings -> Apps -> Advanced app settings -> App execution aliases on Windows 11. Disable the App Installer options for the python.exe and python3.exe executables.

An alternative for running under Windows is to install the Windows Subsystem for Linux (WSL). This in effect allows you to run PypeIt under Linux under Windows.

Additional Data and the PypeIt Cache

To limit the disk-space required for installation, most of PypeIt’s static data files are either not kept in the GitHub repository or distributed via pip. PypeIt uses the generalized cache system provided by Astropy to interface with the remote data, which maintains copies of the data files in a user-writeable location that is independent of the PypeIt installation. For most users, this will be ~/.pypeit/cache, but the exact location can be set directly using astropy’s configuration system. By default, PypeIt will download necessary files at runtime if they are not already cached. Regardless of their location, remote or local, PypeIt essentially organizes all its reference data into subdirectories of the pypeit/data directory in your package installation. The following table gives the reference name, subdirectory, and remote host for data in this directory tree:

Reference

Subdirectory

Host

arc_plot

arc_lines/plots

…

arclines

arc_lines

…

extinction

extinction

…

filters

filters

…

linelist

arc_lines/lists

…

nist

arc_lines/NIST

github

pixelflat

pixelflats

github

reid_arxiv

arc_lines/reid_arxiv

github

sensfunc

sensfuncs

github

skisim

skisim

github

sky_spec

sky_spec

…

spectrographs

spectrographs

…

standards

standards

github

static_calibs

static_calibs

…

tel_model

telluric/models

…

telgrid

telluric/atm_grids

s3_cloud

tests

tests

github

Although most cached files are hosted on GitHub, a few particularly large files are hosted on Amazon S3 cloud storage. Note that a host of ... means that all files should be distributed with your package installation for these directories.

As stated above, PypeIt will download remote files and store them in your cache as they’re needed to reduce your data. I.e., you should mostly be able to ignore the fact that the relevant files are remote, as long as you’re running the reductions while connected to the internet. However, if you’re preparing to run a set of reductions and you would prefer to pre-load data that you expect to need, we provide a few specific scripts for interacting the cache, as described below.

Viewing/Removing Files in the Cache

The pypeit_clean_cache script allows you to view and/or delete files in the cache. To list the cache contents, use the -l option:

% pypeit_clean_cache -l
       HOST               BRANCH               SUBDIR FILE
     github               1.15.1                tests gemini_gnirs_32_1_spat_fit.npz
     github               1.15.1            sensfuncs keck_deimos_600ZD_sensfunc.fits
   s3_cloud                  ...   telluric/atm_grids TellPCA_3000_26000_R10000.fits
     github               1.15.1                tests solution_arrays.npz

Note that the files hosted on GitHub will be specific to a branch or version of PypeIt. Every time you upgrade pypeit, we recommend deleting your existing cache and starting fresh!

Local files that have been installed into the cache (e.g., using pypeit_install_linelist) will appear as being hosted on GitHub and be specific to the version of the code used to install it. When you install local files, keep two things in mind:

  1. The current cache system does not keep track of the original on-disk location of these files. When you install these local files into the cache, the original file will remain (as long as you don’t move/delete it yourself), and they will not be removed by pypeit_clean_cache.

  2. However, as far as the cache is concerned, these files are specific to a given PypeIt version. This means you’ll need to re-install them when you upgrade PypeIt; otherwise, PypeIt will not recognize their existence in the cache. We discuss upgrading above.

Some example uses for removing files include:

  • To remove your entire cache: pypeit_clean_cache --remove_all.

  • To remove cached files for a specific version: pypeit_clean_cache -v 1.15.0

  • To remove a specific file: pypeit_clean_cache -p gemini_gnirs_32_1_spat_fit.npz

Pre-loading Cache Data

Because a fresh install of PypeIt does not contain all of the ancillary data that might be required for data reduction, users planning to run the pipeline without an internet connection will need to cache the necessary data files ahead of time. To ease this process, we provide the pypeit_cache_github_data script. For example, to download the needed files for the keck_deimos spectrograph, you would execute:

$ pypeit_cache_github_data keck_deimos

(Alternatively, you can get all of the cached files hosted on GitHub by performing a developer installation, if you prefer). Once cached, the data will be accessed by PypeIt without requiring an internet connection. By default, this script also downloads any files it finds that are not specific to a given spectrograph. (Unlike previous versions) This script does not download any files hosted in s3_cloud (see the table above); instead, use the scripts below.

Note

Beware of rate limits imposed by GitHub. If you run into this, try setting up an access token and export it as the GITHUB_TOKEN environmental variable; see here.

Atmospheric Model Grids

Calculation of the sensitivity functions and general fitting of telluric absorption uses a grid of model atmosphere spectra. Files with these pre-computed model grids are large (few GB). Recent updates allow for the use of a PCA decomposition of these models to provided smaller (few MB) reference files and faster performance; however, both methods are still functional.

For the larger, atmospheric-grid files, note that we provide model spectra for atmospheric conditions specific to an observatory; however, a model grid is not provided for all observatories with spectrographs supported by PypeIt. If you do not find models for your observatory, you can use the Maunakea model as an approximation. It includes a large grid of different parameters and should be good enough for most purposes.

Note

Instruments that anticipate needing a telluric grid have its filename already included in the telgridfile TelluricPar keyword. The needed model grid will download automatically when required by the code, but given the size of these files and your downlink speed, this may take some time.

To install an atmospheric grid or PCA file independent of a reduction, use the pypeit_install_telluric script, calling the filename of the grid required. For example, if you need the file TelFit_MaunaKea_3100_26100_R20000.fits, you would execute:

$ pypeit_install_telluric TelFit_MaunaKea_3100_26100_R20000.fits

The downloaded file will exist in the PypeIt cache. Unlike files hosted on github, these files will persist through upgrades of your installation. To force the update of a telluric model grid file to the latest version, simply run pypeit_install_telluric with the --force_update option.

If you require a telluric grid that is not presently hosted in the cloud, the code will instruct you to download the file separately from the PypeIt dev-suite Google Drive. Users may select any of the files in the Google Drive for their telluric correction, download them separately, then install them using the --local option to pypeit_install_telluric.

User-provided atmospheric extinction files and wavelength-calibration line lists

As needed to improve their data reduction, users can “install” their own atmospheric extinction files and/or wavelength-calibration line lists. PypeIt manages these local files within its cache system. To install such files, use the pypeit_install_extinctfile or pypeit_install_linelist script, respectively; see Installation Scripts, Specifying an extinction file to use, and User-Supplied Line Lists.

If you find specific files are generally useful/important to your data reduction, we encourage you to submit a GitHub pull-request so that these files can be included in the PypeIt repository.

Important

Because PypeIt uses the cache system to manage the local files, it will associate each file with the version of the code used to install it in the cache. Every time you upgrade your pypeit version, you should delete the local files from the cache (this will not remove the local file itself) and re-install them using the upgraded version of PypeIt. See Viewing/Removing Files in the Cache and Upgrading to a new version of PypeIt.

Quick-Look Calibration Files

Note

We continue to work on cleaner installation solutions for these data products. In the meantime, note that you will likely need to re-run the data-specific installation scripts described below every time you upgrade your installation.

Some of the quick-look reductions provided by PypeIt require canned calibration files to speed up the data-reduction process, as appropriate for a quick-look result. These files are hosted in the QL_CALIB directory in the PypeIt dev-suite Google Drive.

To install the quick-look calibration files:

  1. Right-click on the QL_CALIB folder in the PypeIt dev-suite Google Drive and select the “Download” option from the drop-down menu. This will download a zip file containing the full directory contents. Its current size (as of 22 July 2021) is about 35 MB.

  2. Run the pypeit_install_ql_calibs script. E.g.:

    $ pypeit_install_ql_calibs --zip ~/Downloads/QL_CALIB-20210722T162355Z-001.zip --odir my_path

The pypeit_install_ql_calibs script will unzip the downloaded file in the my_path directory and create a symlink to the extracted directory in the pypeit/data/ directory of your PypeIt installation. The script can automatically delete the zip file using the --rmzip option. If you already have the QL_CALIB directory, you can also use the script to simply create the symlink using the --ql_path option.

Warning

The installation script simply creates symlinks to the downloaded data. This means that if you move the original data, the symlinks will become broken and you will need to rerun the installation script.

Raw Data

Example raw data for all supported spectrographs are used in extensive testing of the code base during development; see PypeIt Development Suite. General users should not need access to these data; however, they may be useful for learning how to use PypeIt before running it on your own data from the same instrument. These data are stored in the RAW_DATA directory in the PypeIt dev-suite Google Drive, divided into subdirectories for each instrument and instrument setup. See also the PypeIt Development Suite GitHub repository, which includes a PypeIt Reduction File for each instrument and setup used during development testing.

Important Package Notes

Interactive Tools

Interactive tools in PypeIt are built using the QT windowing toolkit. The qtpy package is used to provide an abstract interface to the two most widely used QT bindings for Python (see Package Dependencies):

At least one of those bindings must be installed for the interactive GUIs to work. By default pypeit will install pyqt6. Other backends can be used by installing them manually via pip or conda and then setting the QT_API environment variable. See the QtPy documentation for more details.

C code

Significant speed gains in PypeIt can be enabled via compilation of the C versions of the b-spline fitting code. Compilation of the C code should happen automatically when you install PypeIt. However, you can check that the C code was compiled successfully by running the pypeit_c_enabled script. What you should see is:

$ pypeit_c_enabled
Successfully imported bspline C utilities.

If no message is printed, the C code could not be imported.

Some notes if you have problems installing the C code:

  • the code will still run successfully by falling back to slower, pure-python implementations

  • to successfully compile the C code, you may need to update gcc and/or Xcode for Mac users

  • for some Mac users, you may also need to update your OS if you’re using a particularly old version (e.g., 10.10 Yosemite)

Some of the C code uses OpenMP to parallelize loops and take advantage of multiple cores/threads. This support is transparent and the code will work single-threaded if OpenMP is not available. GCC supports OpenMP out of the box, however the clang compiler that Apple’s XCode provides does not. So for optimal performance on Apple hardware, you will want to install GCC via homebrew or macports and specify its use when installing pypeit. For example, if you installed GCC via homebrew, you would get pypeit to use it by doing, for example:

$ export CC=/opt/homebrew/bin/gcc
$ pip install pypeit

Basically, pypeit checks the CC environment variable for what compiler to use so configure that as needed to use your desired compiler. The pypeit_c_enabled script can be used to check if your compiler has OpenMP support.

ginga Plugins

PypeIt requires the ginga viewer and uses at least one ginga plugin to enable specific display functionality. No special considerations are needed to have these plugins installed; however, you can check that they’re enabled by running the following script with the following result:

$ pypeit_chk_plugins
[INFO]    :: All required plugins found: SlitWavelength

If the check is unsuccessful, you will see an error message listing the missing plugins. If you have a problem, please submit an issue.

Package Dependencies

All PypeIt dependencies are installed along with the installation of PypeIt itself. Beware this means that packages in your current environment may be updated depending on the PypeIt version requirements (which is why we recommend you Setup a clean python environment for PypeIt). The current version requirements for both users and developers are:

Python Version

>=3.11,<3.14

Required for users

IPython>=8.0.0, PyERFA>=2.0.0, PyYAML>=6.0, astropy>=6.0, bottleneck, configobj>=5.0.6, extension-helpers>=1.0, fast-histogram>=0.11, ginga>=5.1.0, linetools>=0.3.2, matplotlib>=3.7, numpy>=1.26, packaging>=22.0, pygithub, pyqt6, qtpy>=2.2.0, scikit-learn>=1.2, scipy>=1.9

Required for developers

coverage, docutils<0.21, psutil, pygit2, pytest-astropy, pytest-cov, pytest-qt, pytest>=7.0.0, scikit-image>=0.23, specutils>=1.13, sphinx-automodapi, sphinx>=1.6,<8, sphinx_rtd_theme==2.0.0, tox

Dependency Caveats

Some users have run into the following complications when installing the PypeIt dependencies. If you run into any more, please submit an issue.

  • At the moment, an implicit dependency on QT bindings remains because of our dependence on linetools.

Developer Installation

We, of course, welcome and encourage community development of PypeIt. Please see our Code of Conduct and the Development Procedures and Guidelines.

Developer install via pip

Install pre-release or development versions of PypeIt directly from GitHub using pip as follows. If you already have a pypeit environment set up, run:

pip install --upgrade "git+https://github.com/pypeit/PypeIt#egg=pypeit"

If you’re installing in a clean environment, be sure to include the optional dependencies as well:

pip install --upgrade "git+https://github.com/pypeit/PypeIt#egg=pypeit"

These commands will install the default branch, release. You can also specify a different branch, such as the main develop branch:

pip install --upgrade "git+https://github.com/pypeit/PypeIt.git@develop#egg=pypeit"

Commit hashes, tag names, or git refs can also be specified; see the VCS Support documentation for details and examples.

Developer install from source

Developers doing code development will likely want to set up an “editable” install that points to a locally checked out copy of the GitHub repository. We highly recommended using pip to install the repository and to Setup a clean python environment for code development.

To install from source (after setting up the python environment), first clone (your fork of) the repository:

git clone https://github.com/pypeit/PypeIt.git

Then install the code, include the development dependencies:

cd PypeIt
pip install -e ".[dev]"

An “editable” install means that any changes you make in the repository directory tree will become immediately available the next time the code is imported. Including the [dev] set of optional dependencies ensures that all of the tools you need to test and build PypeIt are installed. (Again, note that you may or may not need the quotes above depending on your shell, and that you should avoid cutting and pasting these commands into a terminal window.)

Finally, you may want to add lines to your relevant shell configuration file (e.g., .zshrc or .bashrc) that activate the relevant environment whenever you start up a new shell. For example:

conda activate pypeit

Otherwise you will need to always type this command at the terminal prompt to activate the pypeit environment.

Test Your Installation

Tagged versions of PypeIt are extensively tested before distribution. However, it is worth testing that your installation has been successful, as follows.

User Tests

The most basic tests that PypeIt has been properly installed is to get the help dialog for one of its main executables. I.e., from a terminal widow, type:

run_pypeit -h

A second basic test is to try to import PypeIt from within a python session. For example:

python
>>> import pypeit

To ensure that your installation of ``pyqt6`` works, you can try to use pypeit_show_1dspec on one of the test files distributed with the package. Below is a zshell command-line incantation (it’s likely the same in bash) that will locate a test spec1D file and attempt to use pypeit_show_1dspec to show it:

python -c "from importlib import resources; print(resources.files('pypeit') / 'tests/files/spec1d_r153-J0025-0312_KASTr_20150123T025323.850.fits')" | xargs -I {} pypeit_show_1dspec {}

If pyqt6 or another Qt backend is correctly installed, this should show a test spectrum from the Shane/KAST spectrograph.

Developer Tests

If you performed a developer installation by cloning the repository into a local directory (e.g., ~/PypeIt), you can run the standard unit tests within the PypeIt environment by executing:

cd ~/PypeIt
pytest

To test within isolated environments and against different versions of various dependencies, we recommend using tox:

cd PypeIt
tox -e test

or, e.g.:

tox -e test-astropydev

Run tox -a to see a list of available test environments.

In either case, over 100 tests should pass, nearly 100 will be skipped and none should fail. The skipped tests only run if the PypeIt development is installed and configured; see PypeIt Development Suite.

Troubleshooting

If you have trouble installing pypeit, you’re encouraged to join our PypeIt Users Slack and post your issue to the #installing channel. Here is an incomplete list of issues that users have reported in the past. In addition to posting to the Users Slack if your issue isn’t among those listed below, please let us know if these suggestions do not work for you.

I am trying to install pypeit for the first time and it fails!: The root problem of this can be system dependent:

  • First, always make sure you install the code into a fresh environment.

  • If you’re on Windows, make sure you follow the User Installation on Windows instructions. If you’re still having trouble, it may be because PypeIt includes some C code to accelerate some computations. If the issue is because the C compiler is not properly linking, you can try typing set CC=help at the command prompt before running the pip install command.

  • Occasionally, the installation may fail because of incompatible dependencies. This may be because of recent releases of one of PypeIt’s dependencies; i.e., updates to packages since the most recent PypeIt release. Please let us know if this happens, and we will try to issue a new release asap that corrects the incompatibility. In the short-term, we may ask you to install old versions of packages that we know work.

I am trying to upgrade pypeit and it fails!: First try uninstalling your current pypeit version:

pip uninstall pypeit

Then reinstall it. If that also fails, try creating a fresh environment and reinstalling pypeit in that new environment.

The installation process succeeded, but the code is faulting!: This could be for a few reasons:

  • Recall that pypeit isn’t necessarily backwards compatible. If you’ve upgraded pypeit and tried to use it with data that was reduced by a previous version, the fault may because of changes between versions. You will either need to revert to your previous version or reprocess the data.

  • This may be because of dependency changes. A tell-tale signature of this is if you get errors associate with missing or unknown keywords or arguments. This is may be because of recent releases of one of PypeIt’s dependencies; i.e., updates to packages since the most recent PypeIt release. Please let us know if this happens, and we will try to issue a new release asap that corrects the incompatibility. In the short-term, we may ask you to install old versions of packages that we know work.

The installation process succeeded and the code completes without faulting, but the output looks wrong!: This could happen for any number of reasons. We always welcome reports of failures! Either submit an issue or report it on the PypeIt Users Slack. However, here are a few things to note and/or try:

  • Make sure you have checked your calibrations; see Calibrations. The issue may be related to a parameter that you can change.

  • If you don’t see any spec1d files in your Science folder, this is likely because the code didn’t find any objects; see Object Finding.

  • If you’ve recently upgraded the code, this may be related to changes in dependencies that the developers didn’t catch. PypeIt performs a lot of testing before issuing a new release, but does not have complete test coverage and performance validation. This means silent failures are the most difficult to catch.

  • And, of course, the code will have bugs. If you find one, the more information you provide the developers, the easier it will be for us to track down the issue. Valuable information includes your OS, OS version, python version, and pypeit version, as well as QA plots and ginga screen grabs that illustrate the issue.