pypeit.archive module

Module for abstracting common code used in archiving files and metadata.

This module splits archiving functionality into ArchiveMetadata and ArchiveDir classes. ArchiveMetadata is responsible for archiving metadata from objects and ArchiveDir is responsible for archiving files.

What types of files and what metadata is archived is delegated to a get_metdadata_func function provided by the caller. It’s signature is:

Args:

object:
The object to be archived.

Returns:
tuple: metadata, files_to_copy

metadata (list of list of str):
The metadata rows read from the object. One object can result in multiple rows. Each row is a list of strings.

files_to_copy (iterable):
An iterable of tuples. Each tuple has a src file to copy to the archive and a destination pathname for that file in the archive. The destination pathname is relative to the archive’s root directory.

Below is an example get_metadata_func function that gets metadata for a single fits file and archives it in a directory based on observation date:

import os.path
from astropy.io import fits
from astropy.time import Time

def get_target_metadata(file_info):

    header = fits.getheader(file_info)

    # Determine the path within the archive to store the file. The subdir
    # has the format YYYYMM based on the observation date.
    mjd = header['MJD']
    subdir_name = Time(mjd, format='mjd').strftime("%Y%m")
    dest_pathname = os.path.join(subdir_name, os.path.basename(file_info))

    # Return a single data row with a column for file name, date and target
    data_rows = [ [dest_pathname, mjd, header['TARGET']] ]

    return (data_rows, [(file_info, dest_pathname)])

class pypeit.archive.ArchiveDir(archive_root, metadata_list, copy_to_archive=True)[source]

Bases: object

Copies files to a directory for archival purposes.

One or more ArchiveMetadata objects are associated with this ArchiveDir, and those metadata files are also stored in the archive. These objects also translate the passed in objects to filenames to copy into the archive.

Parameters:

archive_root (str) – Root directory where the files and metadata will be placed. This will be created if needed.
metadata_list (ArchiveMetadata) – One or more ArchiveMetadata objects responsible for translating objects passed to the archive to paths for copying.

_archive_file(orig_file, dest_file)[source]

Copies a file to the archive directory, if copying is enable.

Parameters:

orig_file (str) – Path to the file to copy.
dest_file (str) – Relative path to the file to copy.

Returns:

The full path to the new copy in the archive.

Return type:

str

add(items)[source]

Adds items to the archive.

Parameters:: items (object, or iterable) – The item or iterable object of items to add to the archive. The items will be passed to a ArchiveMetadata object so that metadata and file names can be read from it.

save()[source]: Saves the metadata in this class to IPAC files in the archive directory

class pypeit.archive.ArchiveMetadata(metadata_file, col_names, get_metadata_func, append, formats={})[source]

Bases: object

Reads metadata from objects and writes it to a file for archiving. This class can be used on its own for saving metadata or passed to an ArchiveDir object for saving files and metadata. Currently the files are saved in ipac format.

Parameters:

metadata_file (str) – File the metadata should be written to.
col_names (list of str) – The column names of the metadata
get_metadata_func (callable) – Function that reads metadata and file information from the objects being archived.
append (bool) – If true append new metadata to an existing file. If this is false any existing file will be overwritten.
formats (dict, optional) – Dictionary mapping column names to a format. The format can be any format accepted by a Astropy Table Column.

add(item)[source]

Adds an item to the ArchiveMetadata.

Parameters:: item – The object to add to the archive.
Returns:: An iterable of tuples, as returned by get_metadata_func. Each tuple has a src file to copy to the archive and a destination pathname for that file in the archive. The destination pathname is relative to the archive’s root directory.
Return type:: iterable

save()[source]: Saves the metadata in this class to IPAC files in the archive directory