pypeit.bitmask module

Base class for handling bit masks.

Class usage examples

Bitmasks allow you to define a set of bit values signified by strings, and then toggle and interpret bits held by a numpy.ndarray. For example, say you’re processing an image and you want to setup a set of bits that indicate that the pixel is part of a bad-pixel mask, has a cosmic ray, or is saturated. You can define the following:

from pypeit.bitmask import BitMask

bits = {'BPM':'Pixel is part of a bad-pixel mask',
        'COSMIC':'Pixel is contaminated by a cosmic ray',
        'SATURATED':'Pixel is saturated.'}
image_bm = BitMask(list(bits.keys()), descr=list(bits.values()))

Note

Consistency in the order of the dictionary keywords is critical to the repeatability of the BitMask instances. The above is possible because dict objects automatically maintain the order of the provided keywords since Python 3.7.

Or, better yet, define a derived class:

from pypeit.bitmask import BitMask

class ImageBitMask(BitMask):
    def __init__(self):
        bits = {'BPM':'Pixel is part of a bad-pixel mask',
                'COSMIC':'Pixel is contaminated by a cosmic ray',
                'SATURATED':'Pixel is saturated.'}
        super(ImageBitMask, self).__init__(list(bits.keys()), descr=list(bits.values()))

image_bm = ImageBitMask()

In either case, you can see the list of bits and their bit numbers by running:

>>> image_bm.info()
         Bit: BPM = 0
 Description: Pixel is part of a bad-pixel mask

         Bit: COSMIC = 1
 Description: Pixel is contaminated by a cosmic ray

         Bit: SATURATED = 2
 Description: Pixel is saturated.
>>> image_bm.bits
{'BPM': 0, 'COSMIC': 1, 'SATURATED': 2}
>>> image_bm.keys()
['BPM', 'COSMIC', 'SATURATED']

Now you can define a numpy.ndarray to hold the mask value for each image pixel; the minimum_dtype() returns the smallest data type required to represent the list of defined bits. The maximum number of bits that can be defined is 64. Assuming you have an image img:

import numpy
mask = numpy.zeros(img.shape, dtype=image_bm.minimum_dtype())

Assuming you have boolean or integer arrays that identify pixels to mask, you can turn on the mask bits as follows:

mask[cosmics_indx] = image_bm.turn_on(mask[cosmics_indx], 'COSMIC')
mask[saturated_indx] = image_bm.turn_on(mask[saturated_indx], 'SATURATED')

or make sure certain bits are off:

mask[not_a_cosmic] = image_bm.turn_off(mask[not_a_cosmic], 'COSMIC')

The form of these methods is such that the array passed to the method are not altered. Instead the altered bits are returned, which is why the lines above have the form m = bm.turn_on(m, flag).

Some other short usage examples:

• To find which flags are set for a single value:

  image_bm.flagged_bits(mask[0,10])
  

• To find the list of unique flags set for any pixel:

  unique_flags = numpy.sort(numpy.unique(numpy.concatenate(
                      [image_bm.flagged_bits(b) for b in numpy.unique(mask)]))).tolist()
  

• To get a boolean array that selects pixels with one or more mask bits:

  cosmics_indx = image_bm.flagged(mask, flag='COSMIC')
  all_but_bpm_indx = image_bm.flagged(mask, flag=['COSMIC', 'SATURATED'])
  any_flagged = image_bm.flagged(mask)
  

• To construct masked arrays, following from the examples above:

  masked_img = numpy.ma.MaskedArray(img, mask=image_bm.flagged(mask))
  

BitMask objects can be defined programmatically, as shown above for the ImageBitMask derived class, but they can also be defined by reading formatted files. The current options are:

1. Fits headers: There are both reading and writing methods for bitmask I/O using astropy.io.fits.Header objects. Using the ImageBitMask class as an example:

   >>> from astropy.io import fits
   >>> hdr = fits.Header()
   >>> image_bm = ImageBitMask()
   >>> image_bm.to_header(hdr)
   >>> hdr
   BIT0    = 'BPM     '           / Pixel is part of a bad-pixel mask
   BIT1    = 'COSMIC  '           / Pixel is contaminated by a cosmic ray
   BIT2    = 'SATURATED'          / Pixel is saturated.
   >>> copy_bm = BitMask.from_header(hdr)
   

class pypeit.bitmask.BitMask(keys, descr=None)

Bases: object

Generic class to handle and manipulate bitmasks. The input list of bit names (keys) must be unique, except that values of ‘NULL’ are ignored. The index in the input keys determines the bit value; ‘NULL’ keys are included in the count. For example:

>>> from pypeit.bitmask import BitMask
>>> keys = [ 'key1', 'key2', 'NULL', 'NULL', 'key3' ]
>>> bm = BitMask(keys)
>>> bm.info()
         Bit: key1 = 0

         Bit: key2 = 1

         Bit: key3 = 4

Todo

• Have the class keep the mask values internally instead of having it only operate on the mask array…

Parameters:

• keys (str, list) – List of keys (or single key) to use as the bit name. Each key is given a bit number ranging from 0..N-1.

• descr (str, list, optional) – List of descriptions (or single discription) provided by info() for each bit. No descriptions by default.

Raises:

• ValueError – Raised if more than 64 bits are provided.

• TypeError – Raised if the provided keys do not have the correct type.

nbits

Number of bits

Type:

int

bits

A dictionary with the bit name and value

Type:

dict

descr

List of bit descriptions

Type:

numpy.ndarray

max_value

The maximum valid bitmask value given the number of bits.

Type:

int

static _fill_sequence(keys, vals, descr=None)

Fill bit sequence with NULL keys if bit values are not sequential.

The instantiation of BitMask does not include the value of the bit, it just assumes that the bits are in sequence such that the first key has a value of 0, and the last key has a value of N-1. This is a convenience function that fills the list of keys with ‘NULL’ for bit values that are non-sequential. This is used primarily for instantiation the BitMask from bits written to a file where the NULL bits have been skipped.

Parameters:

• keys (list, str) – Bit names

• vals (list, int) – Bit values

• descr (list, str, optional) – The description of each bit. If None, no bit descriptions are defined.

Returns:

Three 1D arrays with the filled keys, values, and descriptions.

Return type:

numpy.ndarray

Raises:

ValueError – Raised if a bit value is less than 0.

_prep_flags(flag)

Prep the flags for use.

consolidate(value, flag_set, consolidated_flag)

Consolidate a set of flags into a single flag.

correct_flag_order(flags)

Check if the provided flags are in the correct order compared to the current definition of the object.

Parameters:

flags (list) – A list of strings that must be in the order of the bit numbers. I.e., bit 0 uses the string in the first element of the list, bit 1 uses the second element, etc. The number of flags does not need to exactly match the current set of flags. It can be longer or shorter, so long as it begins with the first flag.

Returns:

Indicates if the provides flags are in the correct order.

Return type:

bool

flagged(value, flag=None, exclude=None, and_not=None)

Determine if a bit is on in the provided integer(s). The function can be used to determine if any individual bit is on or any one of many bits is on.

Parameters:

• value (int, array-like) – Bitmask value. It should be less than or equal to max_value; however, that is not checked.

• flag (str, array-like, optional) – One or more bit names to check. If None, then it checks if any bit is on.

• exclude (str, array-like, optional) – One or more bit names to exclude from the check. If None, all flags are included in the check.

• and_not (str, array-like, optional) – One or more bit names to ensure are not selected by the check. I.e., if this bit is flagged the returned value is false, even if other selected bits are flagged. See examples. If None, functionality ignored.

Returns:

Boolean flags that the provided flags (or any flag) is on for the provided bitmask value. If a numpy array, the shape is the same as value.

Return type:

bool, numpy.ndarray

Raises:

• KeyError – Raised by the dict data type if the input flag is not one of the valid bitmask names.

• TypeError – Raised if the provided flag does not contain one or more strings.

Example

Let the BitMask object have two flags, ‘A’ and ‘B’, and a bit array v, such that:

>>> import numpy
>>> from pypeit.bitmask import BitMask
>>> bm = BitMask(['A', 'B'])
>>> v = numpy.arange(4).astype(numpy.int16)
>>> v
array([0, 1, 2, 3], dtype=int16)
>>> print([bm.flagged_bits(_v) for _v in v])
[[], ['A'], ['B'], ['A', 'B']]

This function will return a boolean array that indicates where flags are turned on in each bit value.

• To find if a specific bit is on:

  >>> bm.flagged(v, flag='A')
  array([False,  True, False,  True])
  

• To find if any bit is on:

  >>> bm.flagged(v)
  array([False,  True,  True,  True])
  

  This is identical to:

  >>> bm.flagged(v, flag=['A', 'B'])
  array([False,  True,  True,  True])
  

  or a logical-or combination of all the bits:

  >>> bm.flagged(v, flag='A') | bm.flagged(v, flag='B')
  array([False,  True,  True,  True])
  

• To find if any bit is on except for a given subset, use the exclude keyword:

  >>> bm.flagged(v, exclude='A')
  array([False, False,  True,  True])
  

  This is identical to:

  >>> bm.flagged(v, flag='B')
  array([False, False,  True,  True])
  

  Obviously, this is more useful when there are many flags and you’re only trying to exclude a few.

• To find if a set of bits are on and a different set of bits are not on, use the and_not keyword:

  >>> bm.flagged(v, and_not='B')
  array([False,  True, False, False])
  

  This is equivalent to:

  >>> bm.flagged(v, flag=['A', 'B'], and_not='B')
  array([False,  True, False, False])
  

  and:

  >>> bm.flagged(v) & numpy.logical_not(bm.flagged(v, flag='B'))
  array([False,  True, False, False])
  

  Note that the returned values are False if the bit indicates that flag ‘B’ is on, even though flag ‘B’ was in the list provided to the flag keyword; i.e., the use of and_not supersedes any elements of flag.

• Currently there is no functionality that performs a logical-and combination of flags.

flagged_bits(value)

Return the list of flagged bit names for a single bit value.

Parameters:

value (int) – Bitmask value. It should be less than or equal to max_value; however, that is not checked.

Returns:

List of flagged bit value keywords.

Return type:

list

Raises:

• KeyError – Raised by the dict data type if the input flag is not one of the valid bitmask names.

• TypeError – Raised if the provided flag does not contain one or more strings.

classmethod from_header(hdr, prefix=None)

Instantiate the BitMask using data parsed from a fits header.

Todo

• This is very similar to the function in ParSet. Abstract to a general routine?

• If comments are truncated by the comment line length, they’ll be different than a direct instantiation.

Parameters:

• hdr (astropy.io.fits.Header) – Header object with the bits.

• prefix (str, optional) – Prefix of the relevant header keywords, which overwrites the string defined for the class. If None, uses the default for the class.

info()

Print the list of bits and, if available, their descriptions.

keys()

Return a list of the bit keywords.

Keywords are sorted by their bit value and ‘NULL’ keywords are ignored.

Returns:

List of bit keywords.

Return type:

list

minimum_dtype(asuint=False)

Return the smallest int datatype that is needed to contain all the bits in the mask. Output as an unsigned int if requested.

Parameters:

asuint (bool, optional) – Return an unsigned integer type. Signed types are returned by default.

Warning

uses int16 if the number of bits is less than 8 and asuint=False because of issue astropy.io.fits has writing int8 values.

static parse_bits_from_hdr(hdr, prefix)

Parse bit names, values, and descriptions from a fits header.

Todo

• This is very similar to the function in ParSet. Abstract to a general routine?

Parameters:

• hdr (astropy.io.fits.Header) – Header object with the bits.

• prefix (str) – The prefix used for the header keywords.

Returns:

Three lists are returned providing the bit names, values, and descriptions.

prefix = 'BIT'

to_dict(prefix=None)

Write the bits to a dictionary.

The keys of the dictionary are identical to those use to write the bitmask to a FITS header.

Parameters:

prefix (str, optional) – Prefix to use for the dictionary keywords, which overwrites the string defined for the class. If None, uses the default for the class.

Returns:

Dictionary where the keyword is the prefix and the bit number, and the value is the bit flag name.

Return type:

dict

to_header(hdr, prefix=None)

Write the bits to a fits header.

The header is edited in place!

Todo

• This is very similar to the function in ParSet. Abstract to a general routine?

• The comment might have a limited length and be truncated.

Parameters:

• hdr (astropy.io.fits.Header) – Header object for the parameters. Modified in-place.

• prefix (str, optional) – Prefix to use for the header keywords, which overwrites the string defined for the class. If None, uses the default for the class.

toggle(value, flag)

Toggle a bit in the provided bitmask value.

Parameters:

• value (int, array-like) – Bitmask value. It should be less than or equal to max_value; however, that is not checked.

• flag (str, array-like) – Bit name(s) to toggle.

Returns:

New bitmask value after toggling the selected bit.

Return type:

array-like

Raises:

ValueError – Raised if the provided flag is None.

turn_off(value, flag)

Ensure that a bit is turned off in the provided bitmask value.

Parameters:

• value (int, array-like) – Bitmask value. It should be less than or equal to max_value; however, that is not checked.

• flag (str, array-like) – Bit name(s) to turn off.

Returns:

New bitmask value after turning off the selected bit.

Return type:

int

Raises:

ValueError – Raised by the dict data type if the input flag is not one of the valid bitmask names or if it is None.

turn_on(value, flag)

Ensure that a bit is turned on in the provided bitmask value.

Parameters:

• value (int, numpy.ndarray) – Bitmask value. It should be less than or equal to max_value; however, that is not checked.

• flag (list, numpy.ndarray, str) – Bit name(s) to turn on.

Returns:

New bitmask value after turning on the selected bit.

Return type:

int

Raises:

ValueError – Raised by the dict data type if the input flag is not one of the valid bitmask names or if it is None.

unpack(value, flag=None)

Construct boolean arrays with the selected bits flagged.

Parameters:

• value (numpy.ndarray) – The bitmask values to unpack.

• flag (str, list, optional) – The specific bits to unpack. If None, all values are unpacked.

Returns:

A tuple of boolean numpy.ndarray objects flagged according to each bit.

Return type:

tuple

version = None