pypeit.io module

Provides a set of I/O routines.

pypeit.io.compress_file(ifile, overwrite=False, rm_original=True)[source]

Compress a file using gzip package.

Parameters:

  • ifile (str) – Name of file to compress. Output file with have the same name with ‘.gz’ appended.

  • overwrite (bool, optional) – Overwrite any existing file.

  • rm_original (bool, optional) – The method writes the compressed file such that both the uncompressed and compressed file will exist when the compression is finished. If this is True, the original (uncompressed) file is removed.

Raises:

ValueError – Raised if the file name already has a ‘.gz’ extension or if the file exists and overwrite is False.

Create a symlink to the input file in the provided directory.

Warning

If the directory provided by symlink_dir does not already exist, this function will create it.

Parameters:

  • filename (str) – The name of the file to symlink. The name of the symlink is identical to this file name.

  • symlink_dir (str) – The directory for the symlink. If the directory does not already exist, it will be created.

  • relative_symlink (bool, optional) – If True, the path to the file is relative to the directory with the symlink.

  • overwrite (bool, optional) – Overwrite any existing symlink of the same name.

  • quiet (bool, optional) – Suppress output to stdout.

pypeit.io.dict_to_hdu(d, name=None, hdr=None, force_to_bintbl=False)[source]

Write a dictionary to a fits HDU.

Elements in the dictionary that are integers, floats, or strings (specific numpy types or otherwise) are written to the HDU header. The header keywords are identical to the dictionary keys.

If any of the elements in the dictionary are an astropy.table.Table, that dictionary can only contain that table and single values that will be written to the extension header. That is, there can be only one astropy.table.Table element, and none of the elements can be a list or numpy.ndarray. By default the extension name is the dictionary key for the astropy.table.Table item; this can be overridden using the name argument.

Elements in the dictionary that are a list or a numpy.ndarray are written as either an image (if there is only one array and a binary table is not specifically requested using force_to_bintbl) or a series of table columns. The lists are assumed to be interpretable as the array argument of astropy.io.fits.Column (for a table) or the data argument of astropy.io.fits.ImageHDU (for an image).

  • If an image is to be written, the extension name, by default, is the dictionary key for the array item; this can be overridden using the name argument.

  • If a table is to be written, the method checks that the relevant arrays have a consistent number of rows. If they do not, the format and dimensions of the table written are set so that the arrays are contained in a single row. The column names in the table are identical to the dictionary keywords. In this case, name must be provided if you want the extension to have a name; there is no default name.

Parameters:

  • d (dict) – Dictionary with data to write to the astropy.io.fits.BinTableHDU.

  • name (str, optional) – Name to give the HDU extension. If None and the input is a dictionary with a single array or astropy.table.Table to write, the name of the extension is the relevant dictionary keyword. Any provided value for name will override this behavior. If the provided dictionary is used to construct a table, where the dictionary keys are used for the table column names, there is no default name for the extension (i.e., no extension name is used if name is None).

  • hdr (astropy.io.fits.Header, optional) – Base-level header to include in the HDU. If None, an empty header is used and then added to.

  • force_to_bintbl (bool, optional) – Force construction of a astropy.io.fits.BinTableHDU instead of an astropy.io.fits.ImageHDU when either there are no arrays or tables to write or only a single array is provided.

Returns:

HDU with the data. An astropy.io.fits.ImageHDU object is returned if there is 1 (or fewer) array-like objects in the dictionary. Otherwise, an astropy.io.fits.BinTableHDU object is returned with the data.

Return type:

astropy.io.fits.ImageHDU, astropy.io.fits.BinTableHDU

Raises:
pypeit.io.dict_to_lines(d, level=0, use_repr=False)[source]

Dump a dictionary to a set of string lines to be written to a file.

Parameters:

  • d (dict) – The dictionary to convert

  • level (int, optional) – An indentation level. Each indentation level is 4 spaces.

  • use_repr (bool, optional) – Instead of using string type casting (i.e., str(...)), use the objects __repr__ attribute.

Returns:

A list of strings that represent the lines in a file.

Return type:

list

pypeit.io.files_from_extension(raw_path, extension: str = 'fits')[source]

Grab the list of files with a given extension

Parameters:

  • raw_path (str or list) –

    Path(s) to raw files, which may or may not include the prefix of the files to search for.

    For a string input, for example, this can be the directory '/path/to/files/' or the directory plus the file prefix '/path/to/files/prefix', which yeilds the search strings '/path/to/files/*fits' or '/path/to/files/prefix*fits', respectively.

    For a list input, this can use wildcards for multiple directories.

  • extension (str, optional) – File extension to search on.

Returns:

List of raw data filenames (sorted) with full path

Return type:

list

pypeit.io.fits_open(filename, **kwargs)[source]

Thin wrapper around astropy.io.fits.open that handles empty padding bytes.

Parameters:
Returns:

List of all the HDUs in the fits file.

Return type:

astropy.io.fits.HDUList

Raises:

PypeItError – Raised if the file does not exist.

pypeit.io.hdu_iter_by_ext(hdu, ext=None, hdu_prefix=None)[source]

Convert the input to lists that can be iterated through by an extension index/name.

Importantly, note that the function does not alter the provided HDUs. If hdu is an astropy.io.fits.HDUList on input, it is simply returned; otherwise, the provided HDU is returned as the only element of a new astropy.io.fits.HDUList object; however, the HDU is not copied! The returned HDU is always the second item in the returned tuple.

If ext is None and hdu is an astropy.io.fits.HDUList, the returned list of extensions includes all extensions in the provided hdu. The extensions are selected by their name, if the HDU has one, or by their index number, otherwise. If ext is None and hdu is not an astropy.io.fits.HDUList, the returned list of extensions just selects the individual HDU provided, either using an integer or the name of the provided hdu (hdu.name), if it has one.

The hdu_prefix parameter can be used to downselect the set of extensions to only those extension strings that start with this prefix (for those extensions that can be identified by a string name).

Warning

The method does not check that all input ext are valid for the provided hdu!

Parameters:
Returns:

Returns two objects: a list with the extensions to iterate through and an astropy.io.fits.HDUList with the list of HDUs.

Return type:

tuple

Raises:

TypeError – Raised if ext is not a string, integer, or list, if any element of ext is not a string or integer, or if hdu is not one of the approved types.

pypeit.io.header_version_check(hdr, warning_only=True)[source]

Check the package versions in the header match the system versions.

Note

The header must contain the keywords written by initialize_header().

Parameters:

  • hdr (astropy.io.fits.Header) – The header to check

  • warning_only (bool, optional) – If the versions are discrepant, only throw a warning instead of raising an exception.

Returns:

Returns True if the check was successful, False otherwise. If warning_only is False, the method will either raise an exception or return True.

Return type:

bool

Raises:

ValueError – Raised if warning_only is False and the system versions are different from those logged in the header.

pypeit.io.init_record_array(shape, dtype)[source]

Utility function that initializes a record array using a provided input data type. For example:

dtype = [ ('INDX', numpy.int, (2,) ),
          ('VALUE', numpy.float) ]

Defines two columns, one named INDEX with two integers per row and the one named VALUE with a single float element per row. See numpy.recarray.

Parameters:

  • shape (int, tuple) – Shape of the output array.

  • dtype (list) – List of the tuples that define each element in the record array.

Returns:

Zeroed record array

Return type:

numpy.recarray

pypeit.io.initialize_header(hdr=None)[source]

Initialize FITS header for all PypeIt output fits images.

Parameters:

hdr (astropy.io.fits.Header, optional) – Header object to update with basic summary information. The object is modified in-place and also returned. If None, an empty header is instantiated, edited, and returned.

Returns:

The initialized (or edited) fits header.

Return type:

astropy.io.fits.Header

pypeit.io.load_object(module, obj=None)[source]

Load an abstracted module and object.

Thanks to: https://stackoverflow.com/questions/67631/how-to-import-a-module-given-the-full-path?rq=1

Parameters:

  • module (str) – The name of a global python module, the root name of a local file with the object to import, or the full module + object type. If obj is None, this must be the latter.

  • obj (str, optional) – The name of the object to import. If None, module must be the full module + object type name.

Returns:

The imported object.

Return type:

type

Raises:

ImportError – Raised if unable to import module.

pypeit.io.parse_hdr_key_group(hdr, prefix='F')[source]

Parse a group of fits header values grouped by a keyword prefix.

If the prefix is ‘F’, the header keywords are expected to be, e.g., ‘F1’, ‘F2’, ‘F3’, etc. The list of values returned are then, e.g., [ hdr[‘F1’], hdr[‘F2’], hdr[‘F3’], … ]. The function performs no retyping, so the values in the returned list have whatever type they had in the fits header.

Parameters:

  • hdr (fits.Header) – Astropy Header object

  • prefix (str, optional) – The prefix used for the header keywords.

Returns:

The list of header values ordered by their keyword index.

Return type:

list

pypeit.io.rec_to_bintable(arr, name=None, hdr=None)[source]

Construct an astropy.io.fits.BinTableHDU from a record array.

Parameters:

  • arr (numpy.recarray) – The data array to write to a binary table.

  • name (str, optional) – The name for the binary table extension.

  • hdr (astropy.io.fits.Header, optional) – Header for the BinTableHDU extension.

Returns:

The binary fits table that can be included in an astropy.io.fits.HDUList and written to disk.

Return type:

astropy.io.fits.BinTableHDU

pypeit.io.rec_to_fits_col_dim(col_element, single_row=False)[source]

Return the string representation of the dimensions for the fits table column based on the provided column element.

The shape is inverted because the first element is supposed to be the most rapidly varying; i.e. the shape is supposed to be written as row-major, as opposed to the native column-major order in python.

Parameters:
Returns:

String representation of the column dimensions. Return None if the object is not multidimensional.

Return type:

str

pypeit.io.rec_to_fits_type(col_element, single_row=False)[source]

Return the string representation of a fits binary table data type based on the provided column element.

Parameters:
Returns:

String representation of the format for the column.

Return type:

str

pypeit.io.remove_suffix(file)[source]

Remove the suffix of a file name.

For normal filenames, this simply returns the file string without its last suffix. For gzipped files, this removes both the ‘.gz’ suffix, and the one preceding it.

Parameters:

file (str) – File name or full path to use

Returns:

The file without its suffix or its input path, if provided.

Return type:

str

Examples

>>> remove_suffix('unzipped_file.txt')
'unzipped_file'
>>> remove_suffix('/path/to/unzipped_file.fits')
'unzipped_file'
>>> remove_suffix('dot.separated.file.name.txt')
'dot.separated.file.name'
>>> remove_suffix('gzipped_file.fits.gz')
'gzipped_file'
pypeit.io.write_to_fits(d, ofile, name=None, hdr=None, overwrite=False, checksum=True)[source]

Write the provided object to a fits file.

This is either a convenience wrapper for write_to_hdu() that adds a primary HDU and writes the result to the provided file, or a convenience wrapper for an already formed astropy.io.fits.HDUList passed as (d).

If the provided file name includes the ‘.gz’ extension, the file is first written using astropy.io.fits.HDUList.writeto and then compressed using compress_file().

Note

  • If the root directory of the output does not exist, this method will create it.

  • Compressing the file is generally slow, but following the two-step process of running astropy.io.fits.HDUList.writeto and then compress_file() is generally faster than having astropy.io.fits.HDUList.writeto do the compression, particularly for files with many extensions (or at least this was true in the past).

Parameters:
pypeit.io.write_to_hdu(d, name=None, hdr=None, force_to_bintbl=False)[source]

Write the input to an astropy.io.fits HDU extension.

List and numpy.ndarray items are written as an ImageHDU, astropy.table.Table items are written as a BinTableHDU, and dictionaries are passed to dict_to_hdu().

Warning

If d is a list, the method assumes that the list is essentially an array that can be sensibly converted to a numpy.ndarray.

Parameters:
Returns:

HDU with the data.

Return type:

astropy.io.fits.ImageHDU, astropy.io.fits.BinTableHDU

Raises:

TypeError – Raised if the input object is not one of the allowed types.