pypeit.io module
Provides a set of I/O routines.
- pypeit.io.compress_file(ifile, overwrite=False, rm_original=True)[source]
Compress a file using gzip package.
- Parameters:
ifile (
str
) – Name of file to compress. Output file with have the same name with ‘.gz’ appended.overwrite (
bool
, optional) – Overwrite any existing file.rm_original (
bool
, optional) – The method writes the compressed file such that both the uncompressed and compressed file will exist when the compression is finished. If this is True, the original (uncompressed) file is removed.
- Raises:
ValueError – Raised if the file name already has a ‘.gz’ extension or if the file exists and overwrite is False.
- pypeit.io.create_symlink(filename, symlink_dir, relative_symlink=False, overwrite=False, quiet=False)[source]
Create a symlink to the input file in the provided directory.
Warning
If the directory provided by
symlink_dir
does not already exist, this function will create it.- Parameters:
filename (
str
) – The name of the file to symlink. The name of the symlink is identical to this file name.symlink_dir (
str
) – The directory for the symlink. If the directory does not already exist, it will be created.relative_symlink (
bool
, optional) – If True, the path to the file is relative to the directory with the symlink.overwrite (
bool
, optional) – Overwrite any existing symlink of the same name.quiet (
bool
, optional) – Suppress output to stdout.
- pypeit.io.dict_to_hdu(d, name=None, hdr=None, force_to_bintbl=False)[source]
Write a dictionary to a fits HDU.
Elements in the dictionary that are integers, floats, or strings (specific numpy types or otherwise) are written to the HDU header. The header keywords are identical to the dictionary keys.
If any of the elements in the dictionary are an astropy.table.Table, that dictionary can only contain that table and single values that will be written to the extension header. That is, there can be only one astropy.table.Table element, and none of the elements can be a
list
or numpy.ndarray. By default the extension name is the dictionary key for the astropy.table.Table item; this can be overridden using thename
argument.Elements in the dictionary that are a list or a numpy.ndarray are written as either an image (if there is only one array and a binary table is not specifically requested using
force_to_bintbl
) or a series of table columns. The lists are assumed to be interpretable as thearray
argument of astropy.io.fits.Column (for a table) or thedata
argument of astropy.io.fits.ImageHDU (for an image).If an image is to be written, the extension name, by default, is the dictionary key for the array item; this can be overridden using the
name
argument.If a table is to be written, the method checks that the relevant arrays have a consistent number of rows. If they do not, the format and dimensions of the table written are set so that the arrays are contained in a single row. The column names in the table are identical to the dictionary keywords. In this case,
name
must be provided if you want the extension to have a name; there is no default name.
- Parameters:
d (
dict
) – Dictionary with data to write to the astropy.io.fits.BinTableHDU.name (
str
, optional) – Name to give the HDU extension. If None and the input is a dictionary with a single array or astropy.table.Table to write, the name of the extension is the relevant dictionary keyword. Any provided value forname
will override this behavior. If the provided dictionary is used to construct a table, where the dictionary keys are used for the table column names, there is no default name for the extension (i.e., no extension name is used ifname is None
).hdr (astropy.io.fits.Header, optional) – Base-level header to include in the HDU. If None, an empty header is used and then added to.
force_to_bintbl (
bool
, optional) – Force construction of a astropy.io.fits.BinTableHDU instead of an astropy.io.fits.ImageHDU when either there are no arrays or tables to write or only a single array is provided.
- Returns:
HDU with the data. An astropy.io.fits.ImageHDU object is returned if there is 1 (or fewer) array-like objects in the dictionary. Otherwise, an astropy.io.fits.BinTableHDU object is returned with the data.
- Return type:
- Raises:
TypeError – Raised if the input object is not a dictionary or the method cannot interpret how to use an element of the dictionary.
ValueError – Raised if dictionary contains another dictionary, more than one astropy.table.Table object, or both an astropy.table.Table and an array-like object (
list
or numpy.ndarray).
- pypeit.io.dict_to_lines(d, level=0, use_repr=False)[source]
Dump a dictionary to a set of string lines to be written to a file.
- Parameters:
- Returns:
A list of strings that represent the lines in a file.
- Return type:
- pypeit.io.files_from_extension(raw_path, extension='.fits')[source]
Find files from one or more paths with one or more extensions.
This is a recursive function. If
raw_path
is a list, the function is called for every item in the list and the results are concatenated.- Parameters:
raw_path (
str
, Path,list
) – One or more paths to search for files, which may or may not include the prefix of the files to search for. For string input, this can be the directory'/path/to/files/'
or the directory plus the file prefix'/path/to/files/prefix'
, which yeilds the search strings'/path/to/files/*fits'
or'/path/to/files/prefix*fits'
, respectively. For a list input, this can use wildcards for multiple directories.extension (
str
,list
, optional) – One or more file extensions to search on.
- Returns:
List of Path objects with the full path to the set of unique raw data filenames that match the provided criteria search strings.
- Return type:
- pypeit.io.fits_open(filename, **kwargs)[source]
Thin wrapper around astropy.io.fits.open that handles empty padding bytes.
- Parameters:
**kwargs – Passed directly to astropy.io.fits.open.
- Returns:
List of all the HDUs in the fits file.
- Return type:
- Raises:
PypeItError – Raised if the file does not exist.
- pypeit.io.hdu_iter_by_ext(hdu, ext=None, hdu_prefix=None)[source]
Convert the input to lists that can be iterated through by an extension index/name.
Importantly, note that the function does not alter the provided HDUs. If
hdu
is an astropy.io.fits.HDUList on input, it is simply returned; otherwise, the provided HDU is returned as the only element of a new astropy.io.fits.HDUList object; however, the HDU is not copied! The returned HDU is always the second item in the returned tuple.If
ext
is None andhdu
is an astropy.io.fits.HDUList, the returned list of extensions includes all extensions in the providedhdu
. The extensions are selected by their name, if the HDU has one, or by their index number, otherwise. Ifext
is None andhdu
is not an astropy.io.fits.HDUList, the returned list of extensions just selects the individual HDU provided, either using an integer or the name of the provided hdu (hdu.name
), if it has one.The
hdu_prefix
parameter can be used to downselect the set of extensions to only those extension strings that start with this prefix (for those extensions that can be identified by a string name).Warning
The method does not check that all input
ext
are valid for the providedhdu
!- Parameters:
hdu (astropy.io.fits.HDUList, astropy.io.fits.ImageHDU, astropy.io.fits.BinTableHDU) – The HDU(s) to iterate through.
ext (
int
,str
,list
, optional) – One or more extensions to include in the iteration. If None, the returned list will enable iteration through all HDU extensions.hdu_prefix (
str
, optional) – In addition to the restricted list of extensions (ext
), only include extensions with this prefix.
- Returns:
Returns two objects: a
list
with the extensions to iterate through and an astropy.io.fits.HDUList with the list of HDUs.- Return type:
- Raises:
TypeError – Raised if
ext
is not a string, integer, or list, if any element ofext
is not a string or integer, or ifhdu
is not one of the approved types.
- pypeit.io.header_version_check(hdr, warning_only=True)[source]
Check the package versions in the header match the system versions.
Note
The header must contain the keywords written by
initialize_header()
.- Parameters:
hdr (astropy.io.fits.Header) – The header to check
warning_only (
bool
, optional) – If the versions are discrepant, only throw a warning instead of raising an exception.
- Returns:
Returns True if the check was successful, False otherwise. If warning_only is False, the method will either raise an exception or return True.
- Return type:
- Raises:
ValueError – Raised if warning_only is False and the system versions are different from those logged in the header.
- pypeit.io.init_record_array(shape, dtype)[source]
Utility function that initializes a record array using a provided input data type. For example:
dtype = [ ('INDX', numpy.int, (2,) ), ('VALUE', numpy.float) ]
Defines two columns, one named INDEX with two integers per row and the one named VALUE with a single float element per row. See numpy.recarray.
- pypeit.io.initialize_header(hdr=None)[source]
Initialize FITS header for all PypeIt output fits images.
- Parameters:
hdr (astropy.io.fits.Header, optional) – Header object to update with basic summary information. The object is modified in-place and also returned. If None, an empty header is instantiated, edited, and returned.
- Returns:
The initialized (or edited) fits header.
- Return type:
astropy.io.fits.Header
- pypeit.io.load_object(module, obj=None)[source]
Load an abstracted module and object.
Thanks to: https://stackoverflow.com/questions/67631/how-to-import-a-module-given-the-full-path?rq=1
- Parameters:
- Returns:
The imported object.
- Return type:
- Raises:
ImportError – Raised if unable to import
module
.
- pypeit.io.load_sky_spectrum(sky_file: str) XSpectrum1D [source]
Load a sky spectrum from the PypeIt data directory into an XSpectrum1D object.
Todo
Try to eliminate the XSpectrum1D dependancy
- Parameters:
sky_file (
str
) – The filename (NO PATH) of the sky file to use.- Returns:
Sky spectrum
- Return type:
- pypeit.io.load_telluric_grid(filename: str)[source]
Load a telluric atmospheric grid from the PypeIt data directory.
- Parameters:
filename (
str
) – The filename (NO PATH) of the telluric atmospheric grid to use.- Returns:
Telluric Grid FITS HDU list
- Return type:
- pypeit.io.load_thar_spec()[source]
Load the archived ThAr spectrum from the PypeIt data directory.
The spectrum read is always
pypeit/data/arc_lines/thar_spec_MM201006.fits
.- Returns:
ThAr Spectrum FITS HDU list
- Return type:
- pypeit.io.parse_hdr_key_group(hdr, prefix='F')[source]
Parse a group of fits header values grouped by a keyword prefix.
If the prefix is ‘F’, the header keywords are expected to be, e.g., ‘F1’, ‘F2’, ‘F3’, etc. The list of values returned are then, e.g., [ hdr[‘F1’], hdr[‘F2’], hdr[‘F3’], … ]. The function performs no retyping, so the values in the returned list have whatever type they had in the fits header.
- pypeit.io.rec_to_bintable(arr, name=None, hdr=None)[source]
Construct an astropy.io.fits.BinTableHDU from a record array.
- Parameters:
arr (numpy.recarray) – The data array to write to a binary table.
name (
str
, optional) – The name for the binary table extension.hdr (astropy.io.fits.Header, optional) – Header for the BinTableHDU extension.
- Returns:
The binary fits table that can be included in an astropy.io.fits.HDUList and written to disk.
- Return type:
astropy.io.fits.BinTableHDU
- pypeit.io.rec_to_fits_col_dim(col_element, single_row=False)[source]
Return the string representation of the dimensions for the fits table column based on the provided column element.
The shape is inverted because the first element is supposed to be the most rapidly varying; i.e. the shape is supposed to be written as row-major, as opposed to the native column-major order in python.
- Parameters:
col_element (numpy.ndarray) – The example data to write to a astropy.io.fits.BinTableHDU used to determine the column dimension.
single_row (
bool
, optional) – Flag that the provided object is the data written to a single row for the astropy.io.fits.BinTableHDU column.
- Returns:
String representation of the column dimensions. Return None if the object is not multidimensional.
- Return type:
- pypeit.io.rec_to_fits_type(col_element, single_row=False)[source]
Return the string representation of a fits binary table data type based on the provided column element.
- Parameters:
col_element (numpy.ndarray) – The example data to write to a astropy.io.fits.BinTableHDU used to determine the column format.
single_row (
bool
, optional) – Flag that the provided object is the data written to a single row for the astropy.io.fits.BinTableHDU column.
- Returns:
String representation of the format for the column.
- Return type:
- pypeit.io.remove_suffix(file)[source]
Remove the suffix of a file name.
For normal filenames, this simply returns the file string without its last suffix. For gzipped files, this removes both the ‘.gz’ suffix, and the one preceding it.
- Parameters:
file (
str
) – File name or full path to use- Returns:
The file without its suffix or its input path, if provided.
- Return type:
Examples
>>> remove_suffix('unzipped_file.txt') 'unzipped_file' >>> remove_suffix('/path/to/unzipped_file.fits') 'unzipped_file' >>> remove_suffix('dot.separated.file.name.txt') 'dot.separated.file.name' >>> remove_suffix('gzipped_file.fits.gz') 'gzipped_file'
- pypeit.io.write_to_fits(d, ofile, name=None, hdr=None, overwrite=False, checksum=True)[source]
Write the provided object to a fits file.
This is either a convenience wrapper for
write_to_hdu()
that adds a primary HDU and writes the result to the provided file, or a convenience wrapper for an already formed astropy.io.fits.HDUList passed as (d
).If the provided file name includes the ‘.gz’ extension, the file is first written using astropy.io.fits.HDUList.writeto and then compressed using
compress_file()
.Note
If the root directory of the output does not exist, this method will create it.
Compressing the file is generally slow, but following the two-step process of running astropy.io.fits.HDUList.writeto and then
compress_file()
is generally faster than having astropy.io.fits.HDUList.writeto do the compression, particularly for files with many extensions (or at least this was true in the past).
- Parameters:
d (
dict
,list
, numpy.ndarray, astropy.table.Table, astropy.io.fits.HDUList) – Object to write to the HDU. Seewrite_to_hdu()
.ofile (
str
) – File name (path) for the fits file.name (
str
, optional) – Name for the extension with the data. If None, the extension is not given a name. However, if the input object is a dictionary, seedict_to_hdu()
for how the name will overwrite any dictionary keyword associated with the data to write. Ignored ifd
is an astropy.io.fits.HDUList.hdr (astropy.io.fits.Header, optional) – Base-level header to use for all HDUs. Ignored if
d
is an astropy.io.fits.HDUList.overwrite (
bool
, optional) – Overwrite any existing file.checksum (
bool
, optional) – Passed to astropy.io.fits.HDUList.writeto to add the DATASUM and CHECKSUM keywords fits header(s).
- pypeit.io.write_to_hdu(d, name=None, hdr=None, force_to_bintbl=False)[source]
Write the input to an astropy.io.fits HDU extension.
List and numpy.ndarray items are written as an ImageHDU, astropy.table.Table items are written as a BinTableHDU, and dictionaries are passed to
dict_to_hdu()
.Warning
If
d
is a list, the method assumes that the list is essentially an array that can be sensibly converted to a numpy.ndarray.- Parameters:
d (
dict
,list
, numpy.ndarray, astropy.table.Table) – Object to write to the HDU.name (
str
, optional) – Name for the HDU extension.hdr (astropy.io.fits.Header, optional) – Header to include in the HDU.
force_to_bintbl (
bool
, optional) – Force construction of a astropy.io.fits.BinTableHDU instead of an astropy.io.fits.ImageHDU when either there are no arrays or tables to write or only a single array is provided.
- Returns:
HDU with the data.
- Return type:
- Raises:
TypeError – Raised if the input object is not one of the allowed types.