pypeit.datamodel module

Implements classes and function for the PypeIt data model.

DataContainer

DataContainer objects provide a utility for enforcing a specific datamodel on an object, and provides convenience routines for writing data to fits files. The class itself is an abstract base class that cannot be directly instantiated. As a base class, DataContainer objects are versatile, but they have their limitations.

Derived classes must do the following:

• Define a class attribute called datamodel. See the examples below for their format.

• Provide an __init__ method that defines the instantiation calling sequence and passes the relevant dictionary to this base-class instantiation.

• Provide a _validate method, if necessary, that processes the data provided in the __init__ into a complete instantiation of the object. This method and the __init__ method are the only places where attributes can be added to the class.

• Provide a _bundle() method that reorganizes the datamodel into partitions that can be written to one or more fits extensions. More details are provided in the description of DataContainer._bundle().

• Provide a _parse() method that parses information in one or more fits extensions into the appropriate datamodel. More details are provided in the description of DataContainer._parse().

Note

The attributes of the class are not required to be a part of the datamodel; however, it makes the object simpler when they are. Any attributes that are not part of the datamodel must be defined in either the __init__ or _validate methods; otherwise, the class with throw an AttributeError.

Here are some examples of how to and how not to use them.

Defining the datamodel

The format of the datamodel needed for each implementation of a DataContainer derived class is as follows.

The datamodel itself is a class attribute (i.e., it is a member of the class, not just of an instance of the class). The datamodel is a dictionary of dictionaries: Each key of the datamodel dictionary provides the name of a given datamodel element, and the associated item (dictionary) for the datamodel element provides the type and description information for that datamodel element. For each datamodel element, the dictionary item must provide:

• otype: This is the type of the object for this datamodel item. E.g., for a float or a numpy.ndarray, you would set otype=float and otype=np.ndarray, respectively. The otype can also be a tuple with optional types. Beware optional types that are themselves DataContainers. This works for PypeItImage, but it likely needs more testing.

• descr: This provides a text description of the datamodel element. This is used to construct the datamodel tables in the pypeit documentation.

If the object type is a numpy.ndarray, you should also provide the atype keyword that sets the type of the data contained within the array. E.g., for a floating point array containing an image, your datamodel could be simply:

datamodel = {'image' : dict(otype=np.ndarray, atype=float, descr='My image')}

More advanced examples are given below.

Basic container

Here’s how to create a derived class for a basic container that holds two arrays and a metadata parameter:

import numpy as np
import inspect

from pypeit.datamodel import DataContainer

class BasicContainer(DataContainer):
    datamodel = {'vec1': dict(otype=np.ndarray, atype=float, descr='Test'),
                 'meta1': dict(otype=str, decr='test'),
                 'arr1': dict(otype=np.ndarray, atype=float, descr='test')}

    def __init__(self, vec1, meta1, arr1):
        # All arguments are passed directly to the container
        # instantiation
        args, _, _, values = inspect.getargvalues(inspect.currentframe())
        super(BasicContainer, self).__init__({k: values[k] for k in args[1:]})

    def _bundle(self):
        # Use the base class _bundle Specify the extension
        return super(BasicContainer, self)._bundle(ext='basic')

With this implementation:

• You can instantiate the container so that number of data table rows would be the same (10):

  data = BasicContainer(np.arange(10), 'length=10', np.arange(30).reshape(10,3))
  

• After instantiating, access the data like this:

  # Get the data model keys
  keys = list(data.keys())
  
  # Access the datamodel as attributes ...
  print(data.vec1)
  # ... or as items
  print(data['meta1'])
  

• Attributes and items are case-sensitive:

  # Faults because of case-sensitive attributes/items
  print(data.Vec1)
  print(data['MeTa1'])
  

• The attributes of the container can only be part of the datamodel or added in either the DataContainer.__init__() or DataContainer._validate() methods:

  # Faults with KeyError
  data['newvec'] = np.arange(10)
  test = data['newvec']
  
  # Faults with AttributeError
  data.newvec = np.arange(10)
  test = data.newvec
  

• The DataContainer also enforces strict types for members of the datamodel:

  # Faults with TypeError
  data.vec1 = 3
  data.meta1 = 4.
  

• Read/Write the data from/to a fits file. In this instantiation, the data is written to a astropy.io.fits.BinTableHDU object; the table has 10 rows because the shape of the arrays match this. The file I/O routines look like this:

  # Write to a file
  ofile = 'test.fits'
  data.to_file(ofile)
  
  # Write to a gzipped file
  ofile = 'test.fits.gz'
  data.to_file(ofile)
  
  # Test written data against input
  with fits.open(ofile) as hdu:
      print(len(hdu))
      # 2: The primary extension and a binary table with the data
  
      print(hdu[1].name)
      # BASIC: As set by the _bundle method
  
      print(len(hdu['BASIC'].data))
      # 10: The length of the data table
  
      print(hdu['BASIC'].columns.names)
      # ['vec1', 'arr1']: datamodel elements written to the table columns
  
      print(hdu['BASIC'].header['meta1'])
      # 'length=10': int, float, or string datamodel components are written to headers
  

• If the shape of the first axis of the arrays (number of rows) do not match, the arrays are written as single elements of a table with one row:

  # Number of rows are mismatched
  data = BasicContainer(np.arange(10), 'length=1', np.arange(30).reshape(3,10))
  data.to_file(ofile)
  
  with fits.open(ofile) as hdu:
      print(len(hdu))
      # 2: The primary extension and a binary table with the data
      print(len(hdu['BASIC'].data))
      # 1: All of the data is put into a single row
  

Mixed Object Containers

DataContainer objects can also contain multiple arrays and/or astropy.table.Table objects. However, multiple Tables or combinations of arrays and Tables cannot be bundled into individual extensions. Here are two implementations of a mixed container, a good one and a bad one:

import numpy as np
import inspect

from astropy.table import Table

from pypeit.datamodel import DataContainer

class GoodMixedTypeContainer(DataContainer):
    datamodel = {'tab1': dict(otype=Table, descr='Test'),
                 'tab1len': dict(otype=int, descr='test'),
                 'arr1': dict(otype=np.ndarray, descr='test'),
                 'arr1shape': dict(otype=tuple, descr='test')}

    def __init__(self, tab1, arr1):
        # All arguments are passed directly to the container
        # instantiation, but the list is incomplete
        args, _, _, values = inspect.getargvalues(inspect.currentframe())
        super(GoodMixedTypeContainer, self).__init__({k: values[k] for k in args[1:]})

    def _validate(self):
        # Complete the instantiation
        self.tab1len = len(self.tab1)
        # NOTE: DataContainer does allow for tuples, but beware because
        # they have to saved to the fits files by converting them to strings
        # and writing them to the fits header.  So the tuples should be
        # short!  See _bundle below, and in DataContainer.
        self.arr1shape = self.arr1.shape

    def _bundle(self):
        # Bundle so there's only one Table and one Image per extension
        return [{'tab1len': self.tab1len, 'tab1': self.tab1},
                {'arr1shape': str(self.arr1shape), 'arr1': self.arr1}]


class BadMixedTypeContainer(GoodMixedTypeContainer):
    def _bundle(self):
        # Use default _bundle method, which will try to put both tables
        # in the same extension.  NOTE: Can't use super here because
        # GoodMixedTypeContainer doesn't have an 'ext' argument
        return DataContainer._bundle(self, ext='bad')

With this implementation:

• To instantiate:

  x = np.arange(10)
  y = np.arange(10)+5
  z = np.arange(30).reshape(10,3)
  
  arr1 = np.full((3,3,3), -1)
  tab1 = Table(data=({'x':x,'y':y,'z':z}), meta={'test':'this'})
  
  data = GoodMixedTypeContainer(tab1, arr1)
  

• Data access:

  print(data.tab1.keys())
  # ['x', 'y', 'z']
  print(assert data.tab1.meta['test'])
  # 'this'
  print(data.arr1shape)
  # (3,3,3)
  

• Construct an astropy.io.fits.HDUList:

  hdu = data.to_hdu(add_primary=True)
  
  print(len(hdu))
  # 3: Includes the primary HDU, one with the table, and one with the array
  
  print([h.name for h in hdu])
  # ['PRIMARY', 'TAB1', 'ARR1']
  

• Tuples are converted to strings:

  print(hdu['ARR1'].header['ARR1SHAPE'])
  # '(3,3,3)'
  

• The tuples are converted back from strings when they’re read from the HDU:

  _data = GoodMixedTypeContainer.from_hdu(hdu)
  print(_data.arr1shape)
  # (3,3,3)
  

• Table metadata is also written to the header, which can be accessed with case-insensitive keys:

  print(hdu['TAB1'].header['TEST'])
  print(hdu['TAB1'].header['TesT'])
  # Both print: 'this'
  

• However, it’s important to note that the keyword case gets mangled when you read it back in. This has to do with the Table.read method and I’m not sure there’s anything we can do about it without the help of the astropy folks. We recommend table metadata use keys that are in all caps:

  # Fails because of the string case
  print(_data.tab1.meta['test'])
  # This is okay
  print(_data.tab1.meta['TEST'])
  

• The difference between the implementation of BadMixedTypeContainer and GoodMixedTypeContainer has to do with how the data is bundled into HDU extensions. The BadMixedTypeContainer will instantiate fine:

  data = BadMixedTypeContainer(tab1, arr1)
  print(data.tab1.keys())
  # ['x', 'y', 'z']
  print(data.tab1.meta['test'])
  # 'this'
  

• But it will barf when you try to reformat/write the data because you can’t write both a Table and an array to a single HDU:

  # Fails
  hdu = data.to_hdu()
  

Complex Instantiation Methods

All of the DataContainer above have had simple instatiation methods. DataContainer can have more complex instantiation methods, but there are significant limitations to keep in made. Consider:

class BadInitContainer(DataContainer):
    datamodel = {'inp1': dict(otype=np.ndarray, descr='Test'),
                 'inp2': dict(otype=np.ndarray, descr='test'),
                 'out': dict(otype=np.ndarray, descr='test'),
                 'alt': dict(otype=np.ndarray, descr='test')}

    def __init__(self, inp1, inp2, func='add'):
        args, _, _, values = inspect.getargvalues(inspect.currentframe())
        super(BadInitContainer, self).__init__({k: values[k] for k in args[1:]})


class DubiousInitContainer(DataContainer):
    datamodel = {'inp1': dict(otype=np.ndarray, descr='Test'),
                 'inp2': dict(otype=np.ndarray, descr='test'),
                 'out': dict(otype=np.ndarray, descr='test'),
                 'alt': dict(otype=np.ndarray, descr='test')}

    def __init__(self, inp1, inp2, func='add'):
        # If any of the arguments of the init method aren't actually
        # part of the datamodel, you can't use the nominal two lines
        # used in all the other examples above.  You have to be specific
        # about what gets passed to super.__init__.  See the
        # BadInitContainer example above and the test below.  WARNING:
        # I'm not sure you would ever want to do this because it can
        # lead to I/O issues; see the _validate function.
        self.func = func
        super(DubiousInitContainer, self).__init__({'inp1': inp1, 'inp2':inp2})

    def _validate(self):
        # Because func isn't part of the data model, it won't be part of
        # self if the object is instantiated from a file.  So I have to
        # add it here.  But I don't know what the value of the attribute
        # was for the original object that was written to disk.  This is
        # why you likely always want anything that's critical to setting
        # up the object to be part of the datamodel so that it gets
        # written to disk.  See the testing examples for when this will
        # go haywire.
        if not hasattr(self, 'func'):
            self.func = None
        if self.func not in [None, 'add', 'sub']:
            raise ValueError('Function must be either 'add' or 'sub'.')

        # This is here because I don't want to overwrite something that
        # might have been read in from an HDU, particularly given that
        # func will be None if reading from a file!
        if self.out is None:
            print('Assigning out!')
            if self.func is None:
                raise ValueError('Do not know how to construct out attribute!')
            self.out = self.inp1 + self.inp2 if self.func == 'add' else self.inp1 - self.inp2

    # I'm not going to overwrite _bundle, so that the nominal approach
    # is used.


class ComplexInitContainer(DataContainer):
    datamodel = {'inp1': dict(otype=np.ndarray, descr='Test'),
                 'inp2': dict(otype=np.ndarray, descr='test'),
                 'out': dict(otype=np.ndarray, descr='test'),
                 'alt': dict(otype=np.ndarray, descr='test'),
                 'func': dict(otype=str, descr='test')}

    def __init__(self, inp1, inp2, func='add'):
        # Since func is part of the datamodel now, we can use the normal
        # two intantiation lines.
        args, _, _, values = inspect.getargvalues(inspect.currentframe())
        super(ComplexInitContainer, self).__init__({k: values[k] for k in args[1:]})

    def _validate(self):
        if self.func not in ['add', 'sub']:
            raise ValueError('Function must be either 'add' or 'sub'.')
        # This is here because I don't want to overwrite something that
        # might have been read in from an HDU, even though they should
        # nominally be the same!
        if self.out is None:
            print('Assigning out!')
            if self.func is None:
                raise ValueError('Do not know how to construct out attribute!')
            self.out = self.inp1 + self.inp2 if self.func == 'add' else self.inp1 - self.inp2

With this implementation:

• The instantiation of the BadInitContainer will fail because the init arguments all need to be part of the datamodel for it to work:

  x = np.arange(10)
  y = np.arange(10)+5
  
  data = BadInitContainer(x,y)
  # Fails with AttributeError
  

• The following instantiation is fine because DubiousInitContainer handles the fact that some of the arguments to __init__ are not part of the datamodel:

  data = DubiousInitContainer(x,y)
  print(np.array_equal(data.out, data.inp1+data.inp2))
  # True
  

• One component of the data model wasn’t instantiated, so it will be None:

  print(data.alt is None)
  # True
  

• The problem with DubiousInitContainer is that it has attributes that cannot be reinstantiated from what’s written to disk. That is, DataContainers aren’t really fully formed objects unless all of its relevant attributes are components of the data model:

  _data = DubiousInitContainer.from_hdu(hdu)
  print(_data.func == DubiousInitContainer(x,y).func)
  # False
  

• This is solved by adding func to the datamodel:

  data = ComplexInitContainer(x,y)
  _data = ComplexInitContainer.from_hdu(data.to_hdu(add_primary=True))
  print(data.func == _data.func)
  # True
  

class pypeit.datamodel.DataContainer(d=None)

Bases: object

Defines an abstract class for holding and manipulating data.

The primary utilities of the class are:

• Attributes can be accessed normally or as expected for a dict

• Attributes and items are restricted to conform to a specified data model.

This abstract class should only be used as a base class.

Derived classes must do the following:

• Define a datamodel

• Provide an __init__ method that defines the instantiation calling sequence and passes the relevant dictionary to this base-class instantiation.

• Provide a _validate method, if necessary, that processes the data provided in the __init__ into a complete instantiation of the object. This method and the __init__ method are the only places where attributes can be added to the class.

• Provide a _bundle() method that reorganizes the datamodel into partitions that can be written to one or more fits extensions. More details are provided in the description of _bundle().

• Provide a _parse() method that parses information in one or more fits extensions into the appropriate datamodel. More details are provided in the description of _parse().

Note

The attributes of the class are not required to be a part of the datamodel; however, it makes the object simpler when they are. Any attributes that are not part of the datamodel must be defined in either the __init__ or _validate methods; otherwise, the class with throw an AttributeError.

Todo

Add a copy method

Parameters:

d (dict, optional) – Dictionary to copy to the internal attribute dictionary. All of the keys in the dictionary must be elements of the datamodel. Any attributes that are not part of the datamodel can be set in the __init__ or _validate methods of a derived class. If None, the object is instantiated with all of the relevant data model attributes but with all of those attributes set to None.

__getattr__(item)

Maps values to attributes. Only called if there isn’t an attribute with this name

__getitem__(item)

Get an item directly from the internal dict.

__repr__()

Over-ride print representation

Returns:

Basics of the Data Container

Return type:

str

__setattr__(item, value)

Set the attribute value.

Attributes are restricted to those defined by the datamodel.

__setitem__(item, value)

Access and set an attribute identically to a dictionary item.

Items are restricted to those defined by the datamodel.

_base_header(hdr=None)

Construct a base header that is included with all HDU extensions produced by to_hdu() (unless they are overwritten by a nested DataContainer).

Additional data can be added to the header for individual HDU extensions in to_hdu() as desired, but this function should not add any elements from the datamodel to the header.

Parameters:

hdr (astropy.io.fits.Header, optional) – Baseline header to add to all returned HDUs. If None, set by pypeit.io.initialize_header().

Returns:

Header object to include in all HDU extensions.

Return type:

astropy.io.fits.Header

_bundle(ext=None, transpose_arrays=False)

Bundle the data into a series of objects to be written to fits HDU extensions.

The returned object must be a list. The list items should be one of the following:

• a dictionary with a single key/item pair, where the key is the name for the extension and the item is the data to be written.

• a single item (object) to be written, which will be written in the provided order without any extension names (although see the caveat in dict_to_hdu() for dictionaries with single array or astropy.table.Table items).

The item to be written can be a single array for an astropy.io.fits.ImageHDU, an astropy.table.Table for a astropy.io.fits.BinTableHDU, or a dictionary; see write_to_hdu().

For how these objects are parsed into the HDUs, see to_hdu().

The default behavior implemented by this base class just parses the attributes into a single dictionary based on the datamodel, under the expectation that it is all written to a single fits extension. Note that this will fault if the datamodel contains:

• a dictionary object

• more than one astropy.table.Table, or

• the combination of an astropy.table.Table and one or more array-like objects (list or numpy.ndarray)

Certain restrictions apply to how the data can be bundled for the general parser implementation (_parse()) to work correctly. These restrictions are:

• The shape and orientation of any input arrays are assumed to be correct.

• Datamodel keys for arrays or astropy.table.Table objects written to an HDU should match the HDU extension name. Otherwise, the set of HDU extension names and datamodel keys must be unique.

• Datamodel keys will be matched to header values: to_hdu() will write any items in a dictionary that is an integer, float, or string (specific numpy types or otherwise) to the header. This means header keys in all extensions should be unique and should not be the same as any extension name.

• Datamodels can contain tuples, but these must be reasonably converted to strings such they are included in (one of) the HDU header(s).

Parameters:

• ext (str, optional) – Name for the HDU extension. If None, no name is given.

• transpose_arrays (bool, optional) – Transpose the arrays before writing them to the HDU. This option is mostly meant to correct orientation of arrays meant for tables so that the number of rows match.

Returns:

A list of dictionaries, each list element is written to its own fits extension. See the description above.

Return type:

list

classmethod _check_parsed(version_passed, type_passed, chk_version=True)

Convenience function that issues the warnings/errors caused by parsing a file into a datamodel.

Parameters:

• version_passed (bool) – Flag that the datamodel version is correct.

• type_passed (bool) – Flag that the datamodel class type is correct.

• chk_version (bool, optional) – Flag to impose strict version checking.

_init_internals()

Add internal variables to the object before initialization completes

These should be set to None

classmethod _parse(hdu, ext=None, ext_pseudo=None, transpose_table_arrays=False, hdu_prefix=None, allow_subclasses=False)

Parse data read from one or more HDUs.

This method is the counter-part to _bundle(), and parses data from the HDUs into a dictionary that can be used to instantiate the object.

Warning

• Beware that this may read data from the provided HDUs that could then be changed by _validate(). Construct your _validate() methods carefully!

• Although _bundle() methods will likely need to be written for each derived class, this parsing method is very general to what DataContainer can do. Before overwriting this function in a derived class, make sure and/or test that this method doesn’t meet your needs, and then tread carefully regardless.

• Because the astropy.table.Table methods are used directly, any metadata associated with the Table will also be included in the HDUs constructed by to_hdu(). However, the astropy.table.Table.read method always returns the metadata with capitalized keys. This means that, regardless of the capitalization of the metadata keywords when the data is written, they will be upper-case when read by this function!

• Use allow_subclasses with care! The expectation is that all the subclasses have exactly the same datamodel and version number as the main class.

Note

Currently, the only test for the object type (otype) given explicitly by the class datamodel is to check if the type should be a tuple. If it is, the parser reads the string from the header and evaluates it so that it’s converted to a tuple on output. See the restrictions listed for _bundle().

All the other type conversions are implicit or based on the HDU type.

Parameters:

• hdu (astropy.io.fits.HDUList, astropy.io.fits.ImageHDU, astropy.io.fits.BinTableHDU) – The HDU(s) to parse into the instantiation dictionary.

• ext (int, str, list, optional) – One or more extensions with the data. If None, the function trolls through the HDU(s) and parses the data into the datamodel.

• ext_pseudo (int, str, list, optional) –

  Pseudonyms to use for the names of the HDU extensions instead of the existing extension names. Similar to the use of hdu_prefix, this is useful for parsing nested DataContainer objects; i.e., a child DataContainer may be assigned to an extension based on the datamodel of its parent, meaning that it can’t be properly parsed by the parent’s method. If used, the number of extension pseudonyms provided must match the list returned by:

  io.hdu_iter_by_ext(hdu, ext=ext, hdu_prefix=hdu_prefix)
  

• transpose_table_arrays (bool, optional) – Tranpose all the arrays read from any binary tables. This is meant to invert the use of transpose_arrays in _bound().

• hdu_prefix (str, optional) – Only parse HDUs with extension names matched to this prefix. If None, ext is used. If the latter is also None, all HDUs are parsed. See pypeit.io.hdu_iter_by_ext().

• allow_subclasses (bool, optional) – Allow subclasses of the calling class to successfully pass the check on whether or not it can parse the provided HDUs. That is, if the class provided in the HDU header is “A”, setting this parameter to True means that parsing will continue as long as “A” is a subclass of the calling class. In PypeIt this is needed, e.g., for SensFunc to successfully parse files written by either of its subclasses, IRSensFunc or UVISSensFunc. This is possible because the datamodel is defined by the parent class and not altered by the subclasses! Use with care! See function warnings.

Returns:

Return four objects: (1) a dictionary with the datamodel contents, (2) a boolean flagging if the datamodel version checking has passed, (3) a boolean flagging if the datamodel type checking has passed, and (4) the list of parsed HDUs.

Return type:

tuple

_primary_header(hdr=None)

Construct a primary header that is included with the primary HDU extension produced by to_hdu().

Additional data can be added to the header for individual HDU extensions in to_hdu() as desired, but this function should not add any elements from the datamodel to the header.

Parameters:

hdr (astropy.io.fits.Header, optional) – Header for the primary extension. If None, set by pypeit.io.initialize_header().

Returns:

Header object to include in the primary HDU.

Return type:

astropy.io.fits.Header

_validate()

Validate the data container.

The purpose of this function is to check the input data provided by the instantiation and flesh out any details of the object.

Derived classes should override this function, unless there is nothing to validate.

Attributes can be added to the object in this function because it is called before the datamodel is frozen.

check_populated(dm_items)

Check that a set of datamodel items are populated.

Parameters:

dm_items (list, str) – One or more items in the datamodel to check.

Returns:

Flag that all the requested datamodel items are populated (not None).

Return type:

bool

classmethod confirm_class(name, allow_subclasses=False)

Confirm that the provided name of a datamodel class matches the calling class.

Parameters:

• name (str) – The name of the class to check against.

• allow_subclasses (bool, optional) – Allow subclasses of the calling class to be included in the check.

Returns:

Class match confirmed if True

Return type:

bool

datamodel = None

Provides the class data model. This is undefined in the abstract class and should be overwritten in the derived classes.

The format of the datamodel needed for each implementation of a DataContainer derived class is as follows.

The datamodel itself is a class attribute (i.e., it is a member of the class, not just of an instance of the class). The datamodel is a dictionary of dictionaries: Each key of the datamodel dictionary provides the name of a given datamodel element, and the associated item (dictionary) for the datamodel element provides the type and description information for that datamodel element. For each datamodel element, the dictionary item must provide:

• otype: This is the type of the object for this datamodel item. E.g., for a float or a numpy.ndarray, you would set otype=float and otype=np.ndarray, respectively.

• descr: This provides a text description of the datamodel element. This is used to construct the datamodel tables in the pypeit documentation.

If the object type is a numpy.ndarray, you should also provide the atype keyword that sets the type of the data contained within the array. E.g., for a floating point array containing an image, your datamodel could be simply:

datamodel = {'image' : dict(otype=np.ndarray, atype=float, descr='My image')}

More advanced examples are given in the top-level module documentation.

Currently, datamodel components are restricted to have otype that are tuple, int, float, numpy.integer, numpy.floating, numpy.ndarray, or astropy.table.Table objects. E.g., datamodel values for otype cannot be dict.

classmethod from_dict(d=None)

Instantiate from a dictionary.

This is primarily to allow for instantiating classes from a file where the data has already been parsed. E.g., see how the TracePCA objects are instantiated in pypeit.edgetrace.EdgeTraceSet.from_hdu. However, note that this does the bare minimum to instantiate the object. Any class-specific operations that are needed to complete the instantiation should be done by ovewritting this method; e.g., see pypeit.tracepca.TracePCA.from_dict().

Parameters:

d (dict, optional) – Dictionary with the data to use for instantiation.

classmethod from_file(ifile, verbose=True, chk_version=True, **kwargs)

Instantiate the object from an extension in the specified fits file.

This is a convenience wrapper for from_hdu().

Parameters:

• ifile (str, Path) – Fits file with the data to read

• verbose (bool, optional) – Print informational messages

• chk_version (bool, optional) – Passed to from_hdu().

• kwargs (dict, optional) – Arguments passed directly to from_hdu().

Raises:

FileNotFoundError – Raised if the specified file does not exist.

classmethod from_hdu(hdu, chk_version=True, **kwargs)

Instantiate the object from an HDU extension.

This is primarily a wrapper for _parse().

Parameters:

• hdu (astropy.io.fits.HDUList, astropy.io.fits.ImageHDU, astropy.io.fits.BinTableHDU) – The HDU(s) with the data to use for instantiation.

• chk_version (bool, optional) – If True, raise an error if the datamodel version or type check failed. If False, throw a warning only.

• **kwargs – Passed directly to _parse().

hdu_prefix = None

If set, all HDUs generated for this DataContainer will have this prefix. This can be set independently for each DataContainer derived class; however, it always defaults to None for the base class. Be wary of nested DataContainer’s!!

internals = None

A list of strings identifying a set of internal attributes that are not part of the datamodel.

keys()

Return the keys for the data objects only

Returns:

The iterable with the data model keys.

Return type:

dict_keys

one_row_table = False

Force the full datamodel to be encapsulated into an astropy.table.Table with a single row when written to disk. Beware that this requires that this is possible! See, e.g., DetectorContainer.

output_to_disk = None

If set, this limits the HDU extensions that are written to the output file. Note this is the name of the extension, including the hdu_prefix, not necessarily the names of specific datamodel components.

to_file(ofile, overwrite=False, checksum=True, **kwargs)

Write the data to a file.

This is a convenience wrapper for to_hdu() and pypeit.io.write_to_fits(). The add_primary parameter of to_hdu() is always true such that the first extension of the written fits file is always an empty primary header.

Parameters:

• ofile (str, Path) – Fits file for the data. File names with ‘.gz’ extensions will be gzipped; see pypeit.io.write_to_fits().

• overwrite (bool, optional) – Flag to overwrite any existing file.

• checksum (bool, optional) – Passed to astropy.io.fits.HDUList.writeto to add the DATASUM and CHECKSUM keywords fits header(s).

• kwargs (dict, optional) – Passed directly to to_hdu().

to_hdu(hdr=None, add_primary=False, primary_hdr=None, force_to_bintbl=False, hdu_prefix=None)

Construct one or more HDU extensions with the data.

The organization of the data into separate HDUs is performed by _bundle(), which returns a list of objects. The type of each element in the list determines how it is handled. If the object is a dictionary with a single key/item pair, the key is used as the extension header. Otherwise, the objects are written to unnamed extensions. The object or dictionary item is passed to pypeit.io.write_to_hdu() to construct the HDU.

Parameters:

• hdr (astropy.io.fits.Header, optional) – Baseline header to add to all returned HDUs. If None, set by pypeit.io.initialize_header().

• add_primary (bool, optional) –

  If False, the returned object is a simple list, with a list of HDU objects (either astropy.io.fits.ImageHDU or astropy.io.fits.BinTableHDU). If true, the method constructs an astropy.io.fits.HDUList with a primary HDU, such that this call:

  hdr = io.initialize_header()
  hdu = fits.HDUList([fits.PrimaryHDU(header=primary_hdr)] + self.to_hdu(hdr=hdr))
  

  and this call:

  hdu = self.to_hdu(add_primary=True)
  

  are identical.

• primary_hdr (astropy.io.fits.Header, optional) – Header to add to the primary if add_primary=True

• force_to_bintbl (bool, optional) – Force construction of a astropy.io.fits.BinTableHDU instead of an astropy.io.fits.ImageHDU when either there are no arrays or tables to write or only a single array is provided (as needed for, e.g., SpecObj). See write_to_hdu().

• hdu_prefix (str, optional) – Prefix for the HDU names. If None, will use hdu_prefix. If the latter is also None, no prefix is added.

Returns:

A list of HDUs, where the type depends on the value of add_primary: If True, an astropy.io.fits.HDUList is returned, otherwise a list is returned.

Return type:

list, astropy.io.fits.HDUList

static valid_write_to_hdu_type(obj)

Check if the provided object can be written to an astropy.io.fits.HDUList.

This needs to be consistent with write_to_hdu().

Parameters:

obj (object) – Object to write

Returns:

Flag that object can be written.

Return type:

bool

version = None

Provides the string representation of the class version.

This is currently put to minimal use so far, but will used for I/O verification in the future.

Each derived class should provide a version to guard against data model changes during development.

pypeit.datamodel.obj_is_data_container(obj)

Simple method to check whether an object is a data container

Parameters:

obj –

Returns:

True if it is

Return type:

bool