PypeIt Dashboard: Design

Dashboard documentation version: 1.4.0

This page explains how the dashboard is built — its component structure, how it acquires the reduction state, and how it stays in sync with a running reduction. For the user-facing description of the views and controls, see PypeIt Dashboard.

The diagrams below are rendered with Mermaid.

Component architecture

The dashboard follows a Model–View–Controller organization with a strict split: a Qt-free model holds all reduction knowledge, thin Qt views render what the model exposes, and a small set of helpers turn user actions into subprocess launches. Nothing in the model imports Qt, so it is unit-testable without a display.

        flowchart TB
    subgraph disk["On disk (a reduction directory)"]
        PF[".pypeit file"]
        SJ["*_state.json"]
        PROD["Calibrations/, Science/,<br/>Intermediate/, QA/PNGs/"]
        LOG["reduction .log"]
    end

    subgraph model["Model layer (Qt-free)"]
        DM["DashboardModel<br/>loads / derives state"]
        RS["RunPypeItState<br/>(pydantic)"]
        PAL["palette<br/>status to color+glyph"]
        INS["inspect<br/>builds viewer / (Re)Build argv"]
    end

    subgraph view["View layer (Qt)"]
        MW["DashboardMainWindow<br/>header + tabs + status bar"]
        SV["StatusView"]
        CV["CalibrationsView"]
        SCV["ScienceView"]
        AB["ActivityBar<br/>Build + Inspection channels"]
    end

    subgraph ctrl["Run control"]
        RL["RunLock<br/>polls .log + state.json"]
        LA["Launcher<br/>QProcess"]
    end

    PF --> DM
    SJ --> DM
    PROD --> DM
    DM --> RS
    DM --> MW
    MW --> SV & CV & SCV
    SV & CV & SCV --> PAL
    CV & SCV --> INS
    INS --> LA
    LA --> SUB["subprocesses:<br/>pypeit_chk_*, ginga,<br/>pypeit_show_*,<br/>pypeit_run_to_calibstep,<br/>pypeit_reduce_by_step"]
    LA --> AB
    LOG --> RL
    SJ --> RL
    RL --> MW

The model never embeds plots: every viewer is an existing PypeIt script, launched as a subprocess by the Launcher and reported on the ActivityBar. The status palette (palette) is the one place that maps a status to a color, a glyph, and a label.

State acquisition on launch

On launch the model acquires the reduction state once. A present *_state.json is loaded (fast, authoritative); when it is absent the state is derived from disk the way pypeit_status does — a read that performs no processing and writes no state file.

        flowchart TD
    START["pypeit_dashboard &lt;file&gt;.pypeit"] --> ACQ["DashboardModel._acquire_state"]
    ACQ --> Q{"*_state.json present?"}
    Q -- "yes" --> LOAD["Load JSON into RunPypeItState<br/>(load_status = state_file)"]
    Q -- "no" --> DERIVE["Derive (read-only):<br/>instantiate PypeIt in calib_only mode"]
    DERIVE --> CAL["calib_all(status_only) for calibrations"]
    DERIVE --> SCI["derive_science_from_disk()<br/>for science frames"]
    CAL --> RS["RunPypeItState<br/>(load_status = derived,<br/>NOT written to disk)"]
    SCI --> RS
    LOAD --> RS
    RS --> SEED["seed planned science/standard frames<br/>(from cached .pypeit metadata)"]
    SEED --> ACCESS["Model accessors:<br/>status_table / slit_table /<br/>science_table / ..."]
    ACCESS --> RENDER["Views render"]

The user’s Refresh button re-runs this acquisition. Mid-run (see below) the model is re-read from *_state.json only — it is never re-derived while a run is active, so a transient mid-write file is skipped rather than triggering a heavy re-derivation.

Planned science frames. On both paths the model also seeds the planned science/standard frames — the upcoming exposures read from the .pypeit metadata — so the Science view lists what is coming even before any reduction (mirroring the planned calibrations), and keeps showing them after a calibration build replaces the state file. The planned-frame list is computed once and cached per .pypeit (a one-time metadata read), so re-seeding it on every state reload is cheap.

Live monitoring and (Re)Build

The dashboard observes a reduction whether it was launched from the dashboard’s (Re)Build controls or started independently in a terminal. A single RunLock is the heart of this: one QTimer polls the reduction .log (to detect an active run) and the *_state.json mtime (to drive live updates), emitting lockChanged and stateChanged.

        sequenceDiagram
    actor User
    participant View as Calibrations / Science view
    participant Launcher
    participant Proc as PypeIt subprocess
    participant Lock as RunLock (QTimer ~2s)
    participant Win as MainWindow

    User->>View: click (Re)Build (confirm clobber)
    View->>Launcher: run(run_to_calibstep / reduce_by_step)
    Launcher->>Proc: start QProcess
    Launcher->>Lock: engage lock
    Lock-->>Win: lockChanged(true)
    Win->>View: set_locked(true)  (buttons orange, disabled)
    loop while running
        Proc->>Proc: write *_state.json per step
        Lock-->>Win: stateChanged
        Win->>Win: re-read state (no derive)
        Win->>View: refresh (preserve scope/selection)
    end
    Proc-->>Launcher: finished
    Lock-->>Win: lockChanged(false)
    Win->>Win: final refresh + idle

The same path runs for a terminal-started run_pypeit: the .log activity engages the lock, stateChanged drives the live refresh, and the views update on their own with no manual Refresh. The refresh preserves the user’s scope (group/detector) and selected step/frame, and inspection launches use a separate ActivityBar channel so monitoring messages and viewer feedback never overwrite each other. The Science view’s Run PypeIt button launches a full run_pypeit -o through this same lock-governed path.

One shared state file, two writers. pypeit_run_to_calibstep (a calibration build) and pypeit_reduce_by_step (a science step-build) each only populate their own portion of *_state.json. To avoid one blanking the other’s portion when it writes, both call RunPypeItState.merge_from_disk() first — overlaying the existing on-disk calibration and science statuses onto their fresh state — so a science (re)build keeps the calibration statuses (and vice versa).

Deriving science state from disk

When there is no state file, the science portion is reconstructed from the on-disk products, with the final Science products authoritative and the Intermediate/ files (written only by pypeit_reduce_by_step) filling steps the Science products do not yet cover.

        flowchart TD
    subgraph auth["Authoritative: Science/ products"]
        S2["spec2d_*.fits present"] --> P1["process = findobj = skysub = success<br/>+ per-slit BADSKYSUB/BADEXTRACT"]
        S1["spec1d_*.fits present"] --> P2["extract = success<br/>+ per-object metrics (S/N, FWHM, ...)"]
    end
    subgraph fb["Fallback: Intermediate/ (reduce_by_step)"]
        IM1["sciImg_* -> process"]
        IM2["Sky_* -> skysub"]
        IM3["spec1d_*_all -> findobj"]
    end
    P1 --> INFER["process inferred 'success'<br/>if any later step succeeded"]
    P2 --> INFER
    fb --> INFER
    INFER --> ENTRY["ScienceFrameState per (frame, det)"]

This mirrors the calibration derive and is why a launch on a finished reduction shows full science status even with no state file. See PypeIt Reduction State for the state model itself.

See also