pypeit.inputfiles module

Class for I/O of PypeIt input files

class pypeit.inputfiles.Coadd1DFile(config=None, file_paths=None, data_table=None, setup=None, vet=True, preserve_comments=False)[source]

Bases: InputFile

Child class for coaddition in 1D

data_block = 'coadd1d': Defines the name of data block.

datablock_required = True: Sets whether or not the data block is required.

flavor = 'Coadd1D': Defines the type of the input file.

property objids

required_columns = ['filename', 'obj_id']: Sets the required columns in the data table.

property sensfiles

Generate a list of the sensitivity files with the full path. The files must exist and be within one of the paths (or the current folder with not other paths specified) for this to succeed.

Returns:: List of full path to each data file or None if filename is not part of the data table or there is no data table!
Return type:: list

property setup_id

setup_required = False: Sets whether or not the setup block is required.

class pypeit.inputfiles.Coadd2DFile(config=None, file_paths=None, data_table=None, setup=None, vet=True, preserve_comments=False)[source]

Bases: InputFile

Child class for coaddition in 2D

data_block = 'spec2d': Defines the name of data block.

datablock_required = True: Sets whether or not the data block is required.

flavor = 'Coadd2D': Defines the type of the input file.

required_columns = ['filename']: Sets the required columns in the data table.

setup_required = False: Sets whether or not the setup block is required.

vet()[source]: Check for required bits and pieces of the .coadd2d file besides the input objects themselves

class pypeit.inputfiles.Coadd3DFile(config=None, file_paths=None, data_table=None, setup=None, vet=True, preserve_comments=False)[source]

Bases: InputFile

Child class for coadding spec2d files into datacubes

data_block = 'spec2d': Defines the name of data block.

datablock_required = True: Sets whether or not the data block is required.

flavor = 'Coadd3d': Defines the type of the input file.

property options

Parse the options associated with a cube block. Here is a description of the available options:

sensfunc: The name of an a sensitivity function file that is used
for the flux calibration. The file provided here should be generated by (or of the same format as the output of) the command pypeit_sensfunc. This parameter can also be set for all frames with the default command:
[reduce] [[cube]] sensfile = sens.fits
scale_corr: The name of an alternative spec2d file that is used for the relative spectral scale correction. This parameter can also be set for all frames with the default command:
```
[reduce]
    [[cube]]
        scale_corr = spec2d_alternative.fits
```
grating_corr: The name of a Flat calibrations file that is used
for the grating tilt correction. This parameter can also be set for all frames with the default command:
[reduce] [[cube]] grating_corr = Flat_A_0_DET01.fits
skysub_frame: The name of an alternative spec2d file that is used for the sky subtraction. This parameter can also be set for all frames with the default command:
```
[reduce]
    [[cube]]
        skysub_frame = spec2d_alternative.fits
```
ra_offset: The RA offset to apply to the WCS of the cube.
dec_offset: The DEC offset to apply to the WCS of the cube.

Returns:: opts – Dictionary containing cube options.
Return type:: dict

required_columns = ['filename']: Sets the required columns in the data table.

setup_required = False: Sets whether or not the setup block is required.

vet()[source]: Check for required bits and pieces of the .coadd3d file besides the input objects themselves

class pypeit.inputfiles.Collate1DFile(config=None, file_paths=None, data_table=None, setup=None, vet=True, preserve_comments=False)[source]

Bases: InputFile

Child class for collate 1D script

data_block = 'spec1d': Defines the name of data block.

datablock_required = True: Sets whether or not the data block is required.

property filenames

List of path + filename’s

Allows for wildcards

Returns:: List of the full paths to each data file or None if filename is not part of the data table or there is no data table!
Return type:: list or None

flavor = 'Collate1D': Defines the type of the input file.

required_columns = ['filename']: Sets the required columns in the data table.

setup_required = False: Sets whether or not the setup block is required.

class pypeit.inputfiles.ExtractFile(config=None, file_paths=None, data_table=None, setup=None, vet=True, preserve_comments=False)[source]

Bases: InputFile

Child class for the Extraction input file

data_block = 'extract': Defines the name of data block.

datablock_required = False: Sets whether or not the data block is required.

flavor = 'Extract': Defines the type of the input file.

setup_required = False: Sets whether or not the setup block is required.

class pypeit.inputfiles.FlexureFile(config=None, file_paths=None, data_table=None, setup=None, vet=True, preserve_comments=False)[source]

Bases: InputFile

Child class for flexure corrections

data_block = 'flexure': Defines the name of data block.

datablock_required = True: Sets whether or not the data block is required.

flavor = 'Flexure': Defines the type of the input file.

required_columns = ['filename']: Sets the required columns in the data table.

setup_required = False: Sets whether or not the setup block is required.

vet()[source]: Check for required bits and pieces of the .flex file besides the input objects themselves

class pypeit.inputfiles.FluxFile(config=None, file_paths=None, data_table=None, setup=None, vet=True, preserve_comments=False)[source]

Bases: InputFile

Child class for the Fluxing input file

data_block = 'flux': Defines the name of data block.

datablock_required = True: Sets whether or not the data block is required.

flavor = 'Flux': Defines the type of the input file.

required_columns = ['filename']: Sets the required columns in the data table.

property sensfiles

Generate a list of the sensitivity files with the full path. The files must exist and be within one of the paths (or the current folder with not other paths specified) for this to succeed.

Returns:: List of full path to each data file or None if filename is not part of the data table or there is no data table!
Return type:: list

setup_required = False: Sets whether or not the setup block is required.

vet()[source]: Check for required parts of the Fluxing input file and handle the various options for sensfile

class pypeit.inputfiles.InputFile(config=None, file_paths=None, data_table=None, setup=None, vet=True, preserve_comments=False)[source]

Bases: object

Generic class to load, process, and write PypeIt input files

In practice, we use one of the children of this class, e.g. PypeItFile

This class has limited support for preserving comments in the input files. We currently support comments in the configuration section, and commented out data lines in the data sections. Other comments are not preserved.

Parameters:

config (dict, list, optional) – Configuration dictionary or list of config lines. This is converted to a ConfigObj instance.
file_paths (list, optional) – List of string or Path objects with the file paths for the data files
data_table (astropy.table.Table, optional) – Data block
setup (dict, optional) – Dictionary with the instrument setup/configuration. The first key contains the setup name.
vet (bool, optional) – If True, vet the file after iniitialization. The vet() method can be called after initialization, if needed.
preserve_comments (bool, optional) – If True, (try to) preserve the comments in the input file.

Variables:

data (astropy.table.Table) – Table with the data in the data block
file_paths (list) – List of Path objects with the directory paths for data files
setup (dict) – Dictionary with the instrument setup/configuration.
preserve_comments (bool) – Flag to preserve comments in the input file.
config (ConfigObj) – Parameters parsed from the configuration block of the input file.

static _parse_setup_lines(lines)[source]

Return a list of the setup names and corresponding Setup dict

Parameters:: lines (numpy.ndarray) – Setup lines as an array
Returns:: list, dict. List of setup name, setup dict
Return type:: tuple

static _read_data_file_table(lines, preserve_comments)[source]

Read the file table format.

Because we allow (even encourage!) the users to modify entries by hand, we have a custom way to parse what is largely a standard fixed_width ASCII table

Parameters:

lines (list) – List of lines within the data block read from the input file.
preserve_comments (bool) – Whether or not to preserve comments in the input file.

Returns:

A list with the paths provided (can be empty) and an astropy.table.Table with the data provided in the input file.

Return type:

tuple

property cfg_lines

Return a list containing the configuration lines: If no configuration is available (config is None), None is returned.

Returns:: List of the configuration lines prepared for writing to a file (and other usages).
Return type:: list

data_block = None: Defines the name of data block.

datablock_required = False: Sets whether or not the data block is required.

property filenames: list[str]

List of path + filename’s Wrapper to path_and_files(). See that function for a full description.

Returns:: List of the full paths to each data file or None if filename is not part of the data table or there is no data table!
Return type:: list or None

static find_block(lines, block)[source]

Find a specific block of lines

These must be wrapped within block read and block end, e.g.:

setup read
Setup A:
...
setup end

Parameters:

lines (list) – List of file lines
block (str) – Name of block to parse

Returns:

Starting,ending line of the block; -1 if not present

Return type:

int, int

flavor = 'Generic': Defines the type of the input file.

classmethod from_file(input_file, vet=True, preserve_comments=False)[source]

Parse the user-provided input file.

Parameters:

input_file (str) – Name of input file
vet (bool,Optional) – Whether or not to vet the file after iniitialization. Defaults to True. The vet() method can be called after initialization if needed.
preserve_comments (bool,Optional) – Whether or not to preserve comments and blank lines in the congiguration section and the data table. Defaults to False.

Returns:

An instance of the InputFile class

Return type:

InputFile

get_pypeitpar(config_specific_file=None, spectrograph_name=None, pypeit_fits=False)[source]

Use the configuration lines and a configuration-specific example file to build the full parameter set.

Parameters:

config_specific_file (str, list, Path, astropy.io.fits.Header, astropy.table.Table, optional) – The object used to generate the default, configuration-specific parameters using config_specific_par(). If None and filenames are available, use the first file. If None and filenames are not available, use default_pypeit_par() to generate the parameters instead.
spectrograph_name (str, optional) – The name of the spectrograph to load. If None, the spectrograph name is pulled from the configuration. If the configuration is None or the ['rdx']['spectrograph'] configuration entry is not available, an exception is raised.
pypeit_fits (bool, optional) – The spectrograph loader is being called from a post-processing script where the expected input files are PypeIt-written FITS files only. This has the effect of overriding the allowed_extensions attribute to be [".fits"].

Returns:

spec (Spectrograph) – Spectrograph subclass instance.
par (PypeItPar) – Parameter set generated by merging the configuration-specific parameters with the configuration parameters.
config_specific_file (str, list, Path, astropy.io.fits.Header, astropy.table.Table) – The object used to generate the configuration-specific parameters, which can be None.

get_spectrograph(spectrograph_name=None, pypeit_fits=False)[source]

Use the configuration lines to instantiate the relevant Spectrograph subclass.

Parameters:

spectrograph_name (str, optional) – The name of the spectrograph to load. If None, the spectrograph name is pulled from the configuration. If the configuration is None or the ['rdx']['spectrograph'] configuration entry is not available, an exception is raised.
pypeit_fits (bool, optional) – The spectrograph loader is being called from a post-processing script where the expected input files are PypeIt-written FITS files only. This has the effect of overriding the allowed_extensions attribute to be [".fits"].

Returns:

Spectrograph subclass instance using the relevant configuration parameter.

Return type:

Spectrograph

Raises:

PypeItError – Raised if the configuration or the relevant parameter is not available.

path_and_files(key, skip_blank=False, include_commented_out=False, check_exists=True)[source]

Generate a list of the filenames with the full path from the column of the data astropy.table.Table specified by key. The files must exist and be within one of the paths for this to succeed.

Parameters:

key (str) – Column in data with the filenames of interest
skip_blank (bool, optional) – If True, ignore any entry that is ‘’, ‘none’ or ‘None’.
check_exists (bool, optional) – If False, PypeIt will not check if the files exist.
include_commented_out (bool, optional) – If False, commented out files will not be included. If True, they are included, without the “#” character.

Returns:

List of strings, where each string provides the full path to each data file. None is returned if data is not defined or if key is not one of its columns.

Return type:

list

Raises:

FileNotFoundError: – Raised if check_exists is True and any of the files do not exist.

static readlines(ifile)[source]

General parser for a PypeIt input file. Used for many of our input files, including the main PypeIt file.

Checks that the file exists.
Reads all the lines in the file
Replaces special characters.

Applies to settings, setup, and user-level reduction files.

Parameters:: ifile (str) – Name of the file to parse.
Returns:: Returns a list of the valid lines in the files.
Return type:: numpy.ndarray

static remove_comments_and_blanks(lines)[source]

Return type:: ndarray

required_columns = []: Sets the required columns in the data table.

property setup_name

Return the setup name

Returns:: Setup name. Usually a single, capitalized Letter
Return type:: str

setup_required = False: Sets whether or not the setup block is required.

vet()[source]: Check for required bits and pieces of the Input file besides the input objects themselves

write(input_file, version_override=None, date_override=None)[source]

Write an Input file to disk

Parameters:

input_file (str) – Name of PypeIt file to be generated
version_override (str, optional) – Override the current version and use this one instead. For documentation purposes only!
date_override (str, optional) – Override the current date and use this one instead. For documentation purposes only!

class pypeit.inputfiles.PypeItFile(config=None, file_paths=None, data_table=None, setup=None, vet=True, preserve_comments=False)[source]

Bases: InputFile

Child class for the PypeIt file

data_block = 'data': Defines the name of data block.

datablock_required = True: Sets whether or not the data block is required.

flavor = 'PypeIt': Defines the type of the input file.

property frametypes

Return a dict of the frametypes with key, item the filename, frametype

Return type:: dict

get_pypeitpar(config_specific_file=None, spectrograph_name=None, pypeit_fits=False)[source]

Use the configuration lines and a configuration-specific example file to build the full parameter set.

This overrides the base class function to use files with specific frametypes and the contents of the PypeIt File (including and modifications away from values in the FITS headers) for the config-specific parameters.

Warning

The calling sequence is purposely kept consistent with the base class, however only the config_specific_file is used. An exception will be raised if spectrograph_name is not None or pypeit_fits is True.

Parameters:

config_specific_file (str, Path, optional) –
The file used to generate the default, configuration-specific parameters. Note that this must be a file name, unlike the type options available to the base class. The behavior of this parameter is defined as follows:
- If None and filenames are present, the file to use is determined as follows:
  If the frametypes are not defined or there are no science, standard, arc, or trace frames, the first data file is used.
  
  If the frametypes are defined, the first science or standard frame is used.
  
  If the frametypes are defined and there are no science or standard frames, the first arc or trace frame is used.
- If not None, the filenames are expected to be present and the provided string must be a file that is listed by the data table. If it is not, an exception is raised.
spectrograph_name (str, optional) – Provided for consistency with the base class. This must always be None. The name of the spectrograph in pypeit files is always defined by the “rdx” parameter set.
pypeit_fits (bool, optional) – Provided for consistency with the base class. The files provided by the pypeit file are, by definition, raw files, meaning this parameter must always be False.

Returns:

spec (Spectrograph) – Spectrograph subclass instance.
par (PypeItPar) – Parameter set used to controle the data processing. If a valid config_specific_file is defined (see above), the paraemters are determined using config_specific_par(). Otherwise, the parameters are determined by default_pypeit_par()
config_specific_file (Path) – The path to the file used to generate the configuration-specific parameters, which can be None.

required_columns = ['filename', 'frametype']: Sets the required columns in the data table.

setup_required = True: Sets whether or not the setup block is required.

vet()[source]: Check for required bits and pieces of the PypeIt file besides the input objects themselves

class pypeit.inputfiles.RawFiles(config=None, file_paths=None, data_table=None, setup=None, vet=True, preserve_comments=False)[source]

Bases: InputFile

Child class for a list of raw files

data_block = 'raw': Defines the name of data block.

datablock_required = True: Sets whether or not the data block is required.

flavor = 'Raw': Defines the type of the input file.

required_columns = ['filename']: Sets the required columns in the data table.

setup_required = False: Sets whether or not the setup block is required.

vet()[source]: Check for required bits and pieces of the .coadd2d file besides the input objects themselves

class pypeit.inputfiles.SensFile(config=None, file_paths=None, data_table=None, setup=None, vet=True, preserve_comments=False)[source]

Bases: InputFile

Child class for the Sensitivity input file

data_block = 'sens': Defines the name of data block.

datablock_required = False: Sets whether or not the data block is required.

flavor = 'Sens': Defines the type of the input file.

setup_required = False: Sets whether or not the setup block is required.

class pypeit.inputfiles.TelluricFile(config=None, file_paths=None, data_table=None, setup=None, vet=True, preserve_comments=False)[source]

Bases: InputFile

Child class for telluric corrections

data_block = None: Defines the name of data block.

datablock_required = False: Sets whether or not the data block is required.

flavor = 'Telluric': Defines the type of the input file.

setup_required = False: Sets whether or not the setup block is required.

pypeit.inputfiles.grab_rawfiles(file_of_files=None, list_of_files=None, raw_paths=None, extension='.fits')[source]

Parse a set of raw files from the input.

Although all arguments are optional, one of file_of_files, list_of_files, or raw_paths must be true. Precedence is given in that order; i.e., if file_of_files is provided, all other arguments are ignored.

Parameters:

file_of_files (str, optional) – File with list of raw files. Format must follow the Data Block of a PypeIt file, and the only required column is the filename.
list_of_files (list, optional) – List of raw files (str). Ignored if file_of_files is provided. If raw_paths is None, the path is assumed to be the current working directory.
raw_paths (list, optional) – One or more paths with the raw files. Ignored if file_of_files is provided. If list_of_files is None, all files with the provided extension are assumed to be raw files.
extension (str, optional) – File extension to search on. Ignored if file_of_files or list_of_files is provided.

Returns:

List of raw data filenames with full path

Return type:

list