Fluxing
Overview
Fluxing is done after the main run of PypeIt, and requires the extracted 1D spectra.
It is a two step process of generating a Sensitivity Function and then Applying the Sensitivity Function. We describe each in turn. However, we first discuss the basic fluxing concepts/approach.
Theoretical Underpinnings of Flux Calibration
Sensitivity Function Units and Definitions
The sensitivity function in PypeIt is defined to be the function \(S_\lambda\) satisfying
with units of \([{\rm erg/cm^2/photons}]\), where \(F_\lambda\) is the specific energy flux in units of \([{\rm erg/s/cm^2/\mathrm{\mathring{A}}}]\), and \(N_\lambda\) is the specific photon flux with units \([{\rm photons/s/\mathrm{\mathring{A}}}]\).
PypeIt spec1d files contain \(N_{\rm pix}\) with units \([{\rm photons/pixel}]\). To generate flux-calibrated spectra \(F_\lambda\), \(S_\lambda\) must be computed from a spectrophotometric standard-star observations.
\(N_{\lambda}\) must be determined from \(N_{\rm pix}\) via
where \(\Delta t\) is the exposure time and \(\frac{d\lambda}{d{\rm pix}}\) is the wavelength spacing per pixel, which is in general not a constant since PypeIt spec1d spectra (i.e. \(N_{\rm pix}\)) are extracted on an irregularly spaced wavelength grid.
After flux calibration, flux calibrated spectra (i.e. \(F_\lambda\)) are
reported in the spec1d files as OPT_FLAM and BOX_FLAM for optimal
and boxcar extractions, respectively, in units of \([10^{-17} {\rm
erg/s/cm^2/\mathrm{\mathring{A}}}]\).
Spectroscopic Zeropoints
Flux calibration of PypeIt spectra is expressed via the “spectroscopic zeropoint”, which, by analogy with the imaging zeropoint, is defined to be:
With this definition we see that an astronomical source with a flat spectrum in frequency \(\nu\), i.e. \(F_\nu = {\rm const}\) and AB magnitude equal to the Zeropoint will produce \(N_\lambda = 1\ {\rm photon/s/\mathrm{\mathring{A}}}\) on the detector, that is the sum of the \(N_{\rm pix}\) photons per pixel over all pixels corresponding to a \(\Delta \lambda = 1 \mathrm{\mathring{A}}\) interval will be equal to unity.
From the definition of the spectroscopic zeropoint above, it follows that
where \(ZPCONST = 40.09\) is a dimensionless number defined by
In practice, PypeIt fits and stores the spectroscopic zerpoints and uses the equation above to compute \(F_\lambda\) from \(N_\lambda\) and vice-versa.
The sensitivity function script, pypeit_sensfunc, produces a QA plot showing the the zeropoint fit, as shown below. For echelle observations this zeropoint QA is shown for each order.
Spectroscopic Throughput
The zeropoint is closely related to the spectroscopic throughput. The number of counts per pixel in a spectrum of an object with flux \(F_\lambda\)
where \(A\) is the effective aperture of the telescope, \(T(\lambda)\) is the spectroscopic throughput, \({\rm Atm(\lambda)}\) is the attenuation caused by the Earth’s atmosphere, \(\frac{d\lambda}{d_{\rm pix}}\) is the number of \(\mathrm{\mathring{A}}\) per pixel defined above, \(h\nu\) is the photon energy, and \(\Delta t\) is the exposure time.
Based on this equation and the definitions above it follows that the spectroscopic throughput can be written \(T(\lambda) = \frac{h\nu}{A S_\lambda}\).
Note \(T(\lambda)\) is clearly dimensionless given the units of \(S_\lambda\): \([{\rm erg/cm^2/photons}]\). As \(S_\lambda\) is specified by the zeropoint, throughput curves can be computed once the zeropoints given the effective aperture of the telescope.
In addition to the zeropoint QA shown above, the sensitivity function script
pypeit_sensfunc also produces a QA plot showing throughput curve(s),
computed directly from the spectroscopic zeropoints.
Note that we have defined the spectroscopic throughput above to be that of the
telescope + instrument system, but it does not include the attenuation
caused by the Earth’s atmosphere (see below).
Also, the zeropoints, sensitivity functions,
and PypeIt flux calibration algorithms in general do not attempt to remove
the impact of slit losses. In the limit where your standard star observations
and science observations have exactly the same seeing, the flux calibration will
be perfect. In the more realistic scenario where they differ, this will
manifest as a wavelength-dependent systematic error in the flux calibration,
with the direction of the error depending on the relative seeing between the
standard-star and science observations. In future versions, we hope to implement
a better treatment of slit losses. For the time being, we recommend that users
that require very accurate flux calibration force PypeIt flux-calibrated
spectra to agree with photometry. This can be done using the filter parameter
option for 1D coadding (see Coadd1DPar Keywords), which can be
set in the .coadd1d file, which is used to guide 1D coaddition with the
pypeit_coadd1d script (see Coadd 1D Spectra).
Extinction Correction
Note
This discussion only applies to data at λ < 7000Å using the UVIS sensitivity function algorithm (see pypeit_sensfunc). In this regime, there is no detailed modeling of telluric absorption, but atmospheric extinction is corrected for. See Telluric correction for discussion of how to handle atmospheric effects for IR data.
Attenuation caused by the Earth’s atmosphere is called atmospheric extinction.
Extinction is a function of wavelength and airmass, and needs to be corrected for
in science spectra for proper flux calibration. It is generally measured by the
observatory in a dedicated campaign (sometimes decades ago), and published for use.
PypeIt includes measured extinction files for many of the observatories
whose spectrographs are supported by the pipeline. The list of current extinction
files is in pypeit/data/extinction/README, and shown below:
# Airmass Extinction files
# NOTE: Longitudes are measured increasing to the east, so west longitudes are
# negative. Generally grabbed from the Internet. Buyer beware.
File Lon Lat Elevation
ctioextinct.dat -70.8150 -30.1653 2215.0
kpnoextinct.dat -111.600 31.9633 2120.0
lapalmaextinct.dat -17.8947 28.7636 2396.0
mkoextinct.dat -155.47833 19.82833 4160.0
mthamextinct.dat -121.6428 37.34139 1290.0
paranalextinct.dat -70.4048305 -24.627167 2635.43
palomarextinct.dat -116.86489 33.35631 1713.
lasillaextinct.dat -70.73 -29.2567 2400.
The extinction correction is applied twice in the flux calibration process:
During the creation of the sensitivity function, the spectrophotometric standard star is corrected to what its flux would have been above the atmosphere.
While applying the sensitivity function to science spectra, those spectra are also corrected because it is highly unlikely that both the standard and science objects were observed at the same airmass.
PypeIt selects the observatory’s extinction file whose geographic coordinates are closest to those of the telescope associated with the spectrograph being used for the reduction. For most users, this is an automatic selection and nothing further need be done. If, however, you are working with a telescope situated more than 5 deg (geographic coordinates) from one of the listed observatories, the code will crash unless you specify a particular extinction file to use or install a custom extinction file in the PypeIt cache.
Specifying an extinction file to use
To specify a particular extinction file, it must be specified in both steps of the flux calibration process:
In the Sensitivity Input File, add the following parameter block to the top of the file:
[sensfunc] [[UVIS]] extinct_file = <name_of_file>In the Flux File, add the following parameter block to the top of the file:
[fluxcalib] extinct_file = <name_of_file>
These steps may be used to either specify one of the PypeIt-included
extinction files listed above or to employ a user-supplied extinction file
(possibly for an observatory not on the list). The latter may be installed
in the user’s PypeIt cache using the script pypeit_install_extinctfile,
whose usage is described in Installation Scripts.
The above steps are meant as a scaffold during development of new instruments in PypeIt, and users are encouraged to submit a pull request to the main repository with new extinction files for other observatories.
Creating a PypeIt Sensitivity Function
The sensitivity function is generated from the Spec1D Output file of a processed standard star.
PypeIt uses an archived fluxed spectrum from either the CALSPEC calibration database or one of the files we have grabbed from ESO. If you observed something else, see Adding a Standard Star.
The sensitivity function is generated by dividing the standard star’s flux in units of \(10^{-17} {\rm erg/s/cm^2/\mathrm{\mathring{A}}}\) by the standard star’s counts per second per Angstrom [photons/s/\(\mathrm{\mathring{A}}\)]. The sensitivity function is written to disk as a FITS file. It has units of \([10^{-17} {\rm erg/s/cm^2/\mathrm{\mathring{A}}}]/[{\rm e-/s/\mathrm{\mathring{A}}}]\). For more information see Flux Calibration.
To flux calibrate spectra, the spectrum in counts from the spec1D files is used to compute counts per second per Angstrom, by which the sensitivity function is multiplied to yield a fluxed science spectrum in units of \(F_\lambda\ [10^{-17} {\rm erg/s/cm^2/\mathrm{\mathring{A}}}]\).
The sensitivity function is written to disk as a FITS file. It has units of \([10^{-17} {\rm erg/s/cm^2/\mathrm{\mathring{A}}}]/[{\rm photons/s/\mathrm{\mathring{A}}}]\) or \([10^{-17} {\rm erg/cm^2/photons}]\).
If you are using an IR instrument or choose the IR mode (see below) then you will need to have grabbed the telluric files. See Atmospheric Model Grids.
pypeit_sensfunc
The process is mediated by the pypeit_sensfunc script.
The script usage can be displayed by calling the script with the
-h option:
$ pypeit_sensfunc -h
usage: pypeit_sensfunc [-h] [--algorithm {UVIS,IR}] [--multi MULTI] [-o OUTFILE]
[-s SENS_FILE] [-f FLATFILE] [--debug]
[--par_outfile PAR_OUTFILE] [-v VERBOSITY]
spec1dfile
Compute a sensitivity function
positional arguments:
spec1dfile spec1d file for the standard that will be used to
compute the sensitivity function
options:
-h, --help show this help message and exit
--algorithm {UVIS,IR}
Override the default algorithm for computing the
sensitivity function. Note that it is not possible to
set --algorithm and simultaneously use a .sens file with
the --sens_file option. If you are using a .sens file,
set the algorithm there via:
[sensfunc]
algorithm = IR
The algorithm options are:
UVIS = Should be used for data with lambda < 7000A. No
detailed model of telluric absorption but corrects for
atmospheric extinction.
IR = Should be used for data with lambbda > 7000A.
Performs joint fit for sensitivity function and telluric
absorption using HITRAN models.
--multi MULTI List of detector numbers to splice together for
instruments with multiple detectors arranged in the
spectral direction, e.g. --multi = '3,7'. Note that it
is not possible to set --multi and simultaneously use a
.sens file with the --sens_file option. If you are
using a .sens file, set the multi_spec_det param there
via:
[sensfunc]
multi_spec_det = 3,7
-o OUTFILE, --outfile OUTFILE
Output file for sensitivity function. If not specified,
the sensitivity function will be written out to a
standard filename in the current working directory, i.e.
if the standard spec1d file is named
spec1d_b24-Feige66_KASTb_foo.fits the sensfunc will be
written to sens_b24-Feige66_KASTb_foo.fits. A QA file
will also be written as
sens_spec1d_b24-Feige66_KASTb_foo_QA.pdf and a file
showing throughput plots to
sens_spec1d_b24-Feige66_KASTb_foo_throughput.pdf. The
same extensions for QA and throughput will be used if
outfile is provided but with .fits trimmed off if it is
in the filename.
-s SENS_FILE, --sens_file SENS_FILE
Configuration file with sensitivity function parameters
-f FLATFILE, --flatfile FLATFILE
Use the flat file for computing the sensitivity
function. Note that it is not possible to set
--flatfile and simultaneously use a .sens file with the
--sens_file option. If you are using a .sens file, set
the flatfile there via e.g.:
[sensfunc]
flatfile = Calibrations/Flat_A_0_DET01.fits
Where Flat_A_0_DET01.fits is the flat file in your
Calibrations directory
--debug show debug plots?
--par_outfile PAR_OUTFILE
Name of output file to save the parameters used by the
fit
-v VERBOSITY, --verbosity VERBOSITY
Verbosity level between 0 [none] and 2 [all]. Default:
1. Level 2 writes a log with filename sensfunc_YYYYMMDD-
HHMM.log
Here is a typical call:
pypeit_sensfunc spec1dfile -o Keck_LRISr_600_7500_sens.fits
This analyzes the standard star spectrum in spec1dfile and writes
the sensitivity file to Keck_LRISr_600_7500_sens.fits. Note that the
spec1dfile to be used in the above argument is the spec1d file
associated with the reduced standard star exposure (not the spec1d
file of the reduced science frame).
Here are the common options used:
–multi
For some instruments (e.g. keck_deimos, gemini_gmos), the spectrum spans
across multiple detectors. You can have the sensitivity function
handle this by using the --multi option, e.g.:
pypeit_sensfunc --multi 3,7
–debug
Throws a number of plots to the screen
–algorithm
The algorithm options are:
UVIS = Should be used for data with \(\lambda < 7000 \mathrm{\mathring{A}}\). No detailed model of telluric absorption but corrects for atmospheric extinction.
IR = Should be used for data with \(\lambda > 7000 \mathrm{\mathring{A}}\). Performs joint fit for sensitivity function and telluric absorption using HITRAN models.
–sens
Provide a Sensitivity Input File to guide the process. Do this if your changes to the defaults are not accommodated by the script inputs.
Sensitivity Input File
This type of Input File Format
contains only a Parameter Block
where you specify the sensfunc parameters.
For example, if you wish to use the MaunaKea telluric grid with your data, you would create a sens file containing:
# User-defined execution parameters
[sensfunc]
algorithm = IR
[[IR]]
telgridfile = TelFit_MaunaKea_3100_26100_R20000.fits
IR without a Standard
If you wish to generate a sensitivity function on a standard star that is not part of the PypeIt database and are working in the IR, you can feed the stellar parameters. Here is an example of the lines for a Sensitivity Input File.
[sensfunc]
algorithm = IR
star_mag = 12.1
star_type = A0
Then run on the spec1d file as you would otherwise. For an A0 star, we use the Vega spectrum. Otherwise, we use the Kurucz93 stellar SED.
Alternatively, see Adding a Standard Star.
Sensitivity Output File
The name of the file containing the sensitivity function can be directly
provided (using the -o command-line option). Otherwise, the file name will
default to one that is identical to the provided spec1d file, but with
spec1d replaced by sens.
The sensitivity data is built and written by SensFunc,
which subclasses from DataContainer. Note that the
telluric data are only included when using the IR algorithm.
Here is its datamodel:
Version 1.0.1
HDU Name |
HDU Type |
Data Type |
Description |
|---|---|---|---|
|
… |
Empty data HDU. Contains basic header information. |
|
|
… |
Telluric model; see |
|
|
… |
Table with the sensitivity function |
|
|
… |
Wavelength vectors. May be combined from many detectors; see the |
|
|
… |
Sensitivity function zeropoints. May be combined from many detectors; see the |
|
|
… |
Spectrograph throughput measurements. May be combined from many detectors; see the |
TELLURIC table (if present)
Column |
Data Type |
Description |
|---|---|---|
|
float64 |
Wavelength vector |
|
float64 |
Best-fitting telluric model spectrum |
|
float64 |
Best-fitting object model spectrum |
|
float64 |
Best-fitting telluric model parameters |
|
float64 |
Best-fitting telluric atmospheric parameters or PCA coefficients |
|
float64 |
Best-fitting telluric model spectral resolution |
|
float64 |
Best-fitting shift applied to the telluric model spectrum |
|
float64 |
Best-fitting stretch applied to the telluric model spectrum |
|
float64 |
Best-fitting object model parameters |
|
float64 |
Chi-square of the best-fit model |
|
bool |
Flag that fit was successful |
|
int64 |
Number of fit iterations |
|
int64 |
Echelle order for this specrum (echelle data only) |
|
int64 |
Polynomial order for each slit/echelle (if applicable) |
|
int64 |
Lowest index of a spectral pixel included in the fit |
|
int64 |
Highest index of a spectral pixel included in the fit |
|
float64 |
Minimum wavelength included in the fit |
|
float64 |
Maximum wavelength included in the fit |
SENS table
Column |
Data Type |
Description |
|---|---|---|
|
float64 |
Wavelength vector |
|
float64 |
Flux in counts per angstrom |
|
float64 |
Log10 of the blaze function for each slit/order |
|
float64 |
Measured sensitivity zero-point data |
|
bool |
Good-pixel mask for the measured zero points |
|
float64 |
Best-fit smooth model to the zero points |
|
bool |
Good-pixel mask for the model zero points |
|
float64 |
Coefficients of smooth model fit to zero points |
|
int64 |
Echelle order for this specrum (echelle data only) |
|
int64 |
Polynomial order for each slit/echelle (if applicable) |
|
float64 |
Minimum wavelength included in the fit |
|
float64 |
Maximum wavelength included in the fit |
|
float64 |
The wavelength array for the fluxed standard star spectrum |
|
float64 |
The F_lambda for the fluxed standard star spectrum |
|
float64 |
The inverse variance of F_lambda for the fluxed standard star spectrum |
|
bool |
The good pixel mask for the fluxed standard star spectrum |
Applying the PypeIt Sensitivity Function
Once you have generated a Sensitivity Function, you may apply
it to one or more Spec1D Output files.
The files are modified in place, filling the OPT_FLAM, BOX_FLAM, etc.
entries, as described in Spec1D Output.
Flux File
To flux one or more spec1d files, one provides a Flux File with the following format with a Parameter Block (optional) and a Data Block (required).
If one wishes to modify the FluxCalibratePar Keywords, add a Parameter Block at the top of the file, e.g.:
[fluxcalib]
extrap_sens = True
There are several ways to provide the Data Block, which always begins and
ends with flux read and flux end, respectively.
First, with one senstivity file and a list of spec1dfiles to be fluxed:
flux read
filename | sensfile
spec1dfile1 | sensfile1
spec1dfile2 |
... |
... |
flux end
Second, with a (presumably unique) sensitivity file for each spec1dfile:
flux read
filename | sensfile
spec1dfile1 | sensfile1
spec1dfile2 | sensfile2
spec1dfile3 | sensfile3
... | ...
flux end
Third, if the spectrograph has an archived sensitivity function (only DEIMOS to date) then a list of spec1d files:
flux read
filename
spec1dfile1
spec1dfile2
...
flux end
Here is an actual example:
flux read
filename | sensfile
spec1d_UnknownFRBHostY_vlt_fors2_2018Dec05T020241.687.fits | VLT_FORS2_sens.fits
spec1d_UnknownFRBHostY_vlt_fors2_2018Dec05T021815.356.fits
spec1d_UnknownFRBHostY_vlt_fors2_2018Dec05T023349.816.fits
flux end
To aid generating this file, we provide the pypeit_flux_setup script.
The script usage can be displayed by calling the script with the
-h option:
$ pypeit_flux_setup -h
usage: pypeit_flux_setup [-h] [--name NAME] [--objmodel {qso,star,poly}]
paths [paths ...]
Setup configuration files to perform flux calibration, 1D coadding, and telluric
correction.
positional arguments:
paths One or more paths for Science folders or sensitivity
functions. Sensitivity functions must start with 'sens_'
to be detected.
options:
-h, --help show this help message and exit
--name NAME The base name to use for the output files. Defaults to
the instrument name is used.
--objmodel {qso,star,poly}
science object model used in the telluric fitting. The
options are:
qso = For quasars. You might need to set redshift,
bal_wv_min_max in the tell file.
star = For stars. You need to set star_type, star_ra,
star_dec, and star_mag in the tell_file.
poly = For other type object, You might need to set
fit_wv_min_max, and norder in the tell_file.
pypeit_flux_calib
Fluxing is performed with the pypeit_flux_calib script.
The script usage can be displayed by calling the script with the
-h option:
$ pypeit_flux_calib -h
usage: pypeit_flux_calib [-h] [--par_outfile] [-v VERBOSITY] [--try_old]
flux_file
Flux calibrate 1D spectra produced by PypeIt
positional arguments:
flux_file File to guide fluxing process. This file must have the
following format:
flux read
filename | sensfile
spec1dfile1 | sensfile1
spec1dfile2 |
...
flux end
OR
flux read
filename | sensfile
spec1dfile1 | sensfile1
spec1dfile2 | sensfile2
spec1dfile3 | sensfile3
...
flux end
OR
[fluxcalib]
use_archived_sens = True
flux read
filename
spec1dfile1
spec1dfile2
spec1dfile3
...
flux end
That is, you must specify either a sensfile for all
spec1dfiles on the first line, specify one sensfile for
each spec1dfile, or specify no sensfiles and use an
archived one.
Archived sensfiles are available for the following
spectrographs: keck_deimos
options:
-h, --help show this help message and exit
--par_outfile Output to save the parameters
-v VERBOSITY, --verbosity VERBOSITY
Verbosity level between 0 [none] and 2 [all]. Default:
1. Level 2 writes a log with filename
flux_calib_YYYYMMDD-HHMM.log
--try_old Attempt to load old datamodel versions. A crash may
ensue..
Here is a typical call:
pypeit_flux_calib flux_file.txt
Again, the Spec1D Output files are modified in place; see there for the related datamodel changes. Also see pypeit_show_1dspec for details on how to view them.
Archival Sensitivity Functions
PypeIt supports using archived sensitivity functions for flux calibration. They can be applied by adding
use_archived_sens = True to the flux file passed to pypeit_flux_calib. For example:
[fluxcalib]
use_archived_sens = True
flux read
spec1d_d1010_0056-HIT2015-mask03_DEIMOS_20151010T045816.550.fits
flux end
Disclaimer: Currently only keck_deimos sensitivity files are available. These DEIMOS archival
sensitivity functions do not provide an absolute flux calibration. Instead, they are only intended to
remove the instrumental response, providing a relative flux calibration up to some unknown normalization.
Troubleshooting
Problem with Empty filename
If you encounter this error when doing flux calibration with the IR algorithm, please do the following steps:
Make sure you have installed the relevant atmosphere telluric models. See the instructions for installing this Additional Data.
Write the filename of the corresponding file for your observatory in the parameter telgridfile (i.e. keck_lris_sens.txt), e.g.:
[sensfunc] algorithm = IR polyorder = 8 [[IR]] telgridfile = TelFit_MaunaKea_3100_26100_R20000-006.fitsRun pypeit_sensfunc with the –sens_file option, e.g.:
pypeit_sensfunc your_spec1dfile -o your_output.fits --sens_file keck_lris_sens.txt
Adding a Standard Star
If your star is not in the repository you can add in a new solution if it is in the ESO database.
You will need to place their .dat file in pypeit/data/standards/esofil/ and then edit
the esofil_info.txt file accordingly. Make sure the flux column is in flux units rather
than magnitudes (i.e. those files starting with f in the ESO database), and these fluxes
are in units of \([10^{-16} {\rm erg/s/cm^2/\mathrm{\mathring{A}}}]\).
Extra kudos if you submit this as a PR for others benefit.
If your standard star is even more non-traditional, contact the developers.
Additional Reading
Here are additional docs on somewhat common edits that PypeIt users make:
FluxSpec Class
The guts of the flux algorithms are guided by the
apply_flux_calib() class.