Running the measurement scripts
================================

``measure_joint_sumstat.py`` — unified entry point
----------------------------------------------------

This script supersedes all per-statistic BGS joint scripts.  Any combination
of SMF, LF, WP, ESD_HSC, ESD_DES, ESD_KIDS, WTHETA, and KNN can be measured
jointly with a shared jackknife or bootstrap covariance.  Bin edges, number of
JK regions, and weight variants are all configurable from the command line.

**Dataset flags**

.. list-table::
   :widths: 30 15 55
   :header-rows: 1

   * - Flag
     - Default
     - Description
   * - ``--survey``
     - ``bgs``
     - ``bgs`` · ``mock`` · ``custom``
   * - ``--mstar-min``
     - —
     - Filter BGS/mock samples by stellar-mass lower edge
   * - ``--mstar-max``
     - —
     - Filter by upper edge
   * - ``--z-min`` / ``--z-max``
     - from filename
     - Redshift range (auto-read for bgs/mock; required for custom + SMF/LF)
   * - ``--name``
     - —
     - Sample identifier (``custom`` only)
   * - ``--data-file`` / ``--rand-file``
     - —
     - Lens and random FITS paths (``custom`` only)
   * - ``--weight-file``
     - auto
     - Systematic-weight FITS (custom; auto-discovered for bgs/mock)
   * - ``--weight-variant``
     - ``sys-comb``
     - ``uniform`` · ``sys-add`` · ``sys-comb`` · ``all``
   * - ``--abs-mag-col``
     - —
     - FITS column for absolute magnitude (needed for LF)

**Statistics flags**

.. list-table::
   :widths: 30 35 35
   :header-rows: 1

   * - Flag
     - Default
     - Choices
   * - ``--stats``
     - ``SMF WP ESD_HSC ESD_DES``
     - ``SMF`` ``LF`` ``WP`` ``ESD_HSC`` ``ESD_DES`` ``ESD_KIDS`` ``WTHETA`` ``KNN``

**Binning flags** (see :ref:`bin-spec` for format syntax)

.. list-table::
   :widths: 30 35 35
   :header-rows: 1

   * - Flag
     - Default
     - Applies to
   * - ``--bins-rp SPEC``
     - ``log:30:0.01:60``
     - WP projected separation [Mpc]
   * - ``--bins-rp-lens SPEC``
     - ``log:30:0.01:60``
     - ESD projected separation [Mpc]
   * - ``--bins-mstar SPEC``
     - ``arange:9.0:12.6:0.25``
     - SMF log10(M★/M☉) [dex]
   * - ``--bins-theta SPEC``
     - ``log:30:0.01:60``
     - WTHETA angular separation [arcmin]
   * - ``--bins-mag SPEC``
     - ``arange:-24.0:-16.0:0.5``
     - LF absolute magnitude [mag]
   * - ``--bins-knn-r SPEC``
     - ``log:20:0.1:50``
     - KNN separation [Mpc]
   * - ``--knn-k K …``
     - ``1 2 3 4 5``
     - Neighbor orders
   * - ``--knn-n-query N``
     - ``100000``
     - Number of kNN query points

**Covariance flags**

.. list-table::
   :widths: 30 15 55
   :header-rows: 1

   * - Flag
     - Default
     - Description
   * - ``--covariance``
     - ``jackknife``
     - ``jackknife`` or ``bootstrap``
   * - ``--n-jk``
     - ``100``
     - Number of jackknife regions
   * - ``--n-bootstrap``
     - ``1000``
     - Number of bootstrap draws

**Misc flags**

.. list-table::
   :widths: 30 15 55
   :header-rows: 1

   * - Flag
     - Default
     - Description
   * - ``--output-dir``
     - ``data/``
     - Root directory for HDF5 output
   * - ``--output-file``
     - auto
     - Override auto-generated output path
   * - ``--skip-existing``
     - on
     - Skip if ``joint_covariance/cov`` already exists in output
   * - ``--n-threads``
     - ``nproc``
     - CPU threads for Corrfunc / treecorr
   * - ``--pi-max``
     - ``100.0``
     - π\ :sub:`max` for w\ :sub:`p` [Mpc]
   * - ``--dry-run``
     - off
     - Print plan without writing files or loading sources
   * - ``--fast-test``
     - off
     - Smoke-test: n_jk=10, max_ngal=5000, knn_n_query=5000,
       ``--subsample`` enabled for BGS
   * - ``--max-ngal N``
     - —
     - Randomly subsample to N galaxies after loading (randoms → 5N)
   * - ``--log-level``
     - ``INFO``
     - ``DEBUG`` · ``INFO`` · ``WARNING``

**Checkpoint / resume**

Each JK or bootstrap iteration is written to ``_checkpoint/subsamples`` in the
output HDF5 immediately after completion.  Re-running the same command resumes
from where it stopped — already-finished rows are skipped.  To add more
iterations, increase ``--n-jk`` (or ``--n-bootstrap``) and re-run; new rows
are appended to the checkpoint and the covariance is re-computed at the end.

**Example commands — LS10 M★≥11.5, sys-comb weights**

See ``scripts/run_ls10_m11.5_examples.sh`` for the full example set, or run
it directly::

    bash scripts/run_ls10_m11.5_examples.sh [--fast-test] [--dry-run]

Selected examples:

.. code-block:: bash

    # Default: SMF + WP + ESD_HSC + ESD_DES  (sys-comb, 100 JK)
    JAX_PLATFORMS=cpu python scripts/measure_joint_sumstat.py \
        --mstar-min 11.5 --weight-variant sys-comb

    # All lensing surveys
    JAX_PLATFORMS=cpu python scripts/measure_joint_sumstat.py \
        --mstar-min 11.5 --weight-variant sys-comb \
        --stats ESD_HSC ESD_DES ESD_KIDS

    # All summary statistics (no LF — requires abs_mag column)
    JAX_PLATFORMS=cpu python scripts/measure_joint_sumstat.py \
        --mstar-min 11.5 --weight-variant sys-comb \
        --stats SMF WP ESD_HSC ESD_DES ESD_KIDS WTHETA KNN

    # Bootstrap covariance, custom bins
    JAX_PLATFORMS=cpu python scripts/measure_joint_sumstat.py \
        --mstar-min 11.5 --weight-variant sys-comb \
        --stats SMF WP ESD_HSC ESD_DES \
        --covariance bootstrap --n-bootstrap 500 \
        --bins-rp log:15:0.1:40 --bins-mstar arange:11.5:12.6:0.1

    # Smoke-test (5000 galaxies, 10 JK regions, subsample BGS footprint)
    JAX_PLATFORMS=cpu python scripts/measure_joint_sumstat.py \
        --mstar-min 11.5 --weight-variant sys-comb \
        --fast-test --stats SMF WP

----

Output directory layout
------------------------

.. code-block:: none

   data/
   ├── lf_smf/         BGS and mock SMF (one .h5 per sample × weight variant)
   ├── twopcf/         BGS and mock wp, wtheta
   ├── lensing/        BGS ΔΣ (HSC/DES)
   ├── BGS_Mstar{X}/   joint covariance files per mass threshold
   ├── mocks/
   │   ├── lf_smf/     mock SMF
   │   ├── twopcf/     mock wp, wtheta
   │   └── joint/      mock joint SMF + wp
   ├── COSMOS/         cosmos_smf_z*.fits, cosmos_lf_{band}_z*.fits
   └── GAMA/           gama_smf_z*.fits, gama_lf_r_z*.fits