Running the measurement scripts

measure_joint_sumstat.py — unified entry point

This script supersedes all per-statistic BGS joint scripts. Any combination of SMF, LF, WP, ESD_HSC, ESD_DES, ESD_KIDS, WTHETA, and KNN can be measured jointly with a shared jackknife or bootstrap covariance. Bin edges, number of JK regions, and weight variants are all configurable from the command line.

Dataset flags

Flag	Default	Description
--survey	bgs	bgs · mock · custom
--mstar-min	—	Filter BGS/mock samples by stellar-mass lower edge
--mstar-max	—	Filter by upper edge
--z-min / --z-max	from filename	Redshift range (auto-read for bgs/mock; required for custom + SMF/LF)
--name	—	Sample identifier (custom only)
--data-file / --rand-file	—	Lens and random FITS paths (custom only)
--weight-file	auto	Systematic-weight FITS (custom; auto-discovered for bgs/mock)
--weight-variant	sys-comb	uniform · sys-add · sys-comb · all
--abs-mag-col	—	FITS column for absolute magnitude (needed for LF)

Statistics flags

Flag	Default	Choices
--stats	SMF WP ESD_HSC ESD_DES	SMF LF WP ESD_HSC ESD_DES ESD_KIDS WTHETA KNN

Binning flags (see Bin specification for format syntax)

Flag	Default	Applies to
--bins-rp SPEC	log:30:0.01:60	WP projected separation [Mpc]
--bins-rp-lens SPEC	log:30:0.01:60	ESD projected separation [Mpc]
--bins-mstar SPEC	arange:9.0:12.6:0.25	SMF log10(M★/M☉) [dex]
--bins-theta SPEC	log:30:0.01:60	WTHETA angular separation [arcmin]
--bins-mag SPEC	arange:-24.0:-16.0:0.5	LF absolute magnitude [mag]
--bins-knn-r SPEC	log:20:0.1:50	KNN separation [Mpc]
--knn-k K …	1 2 3 4 5	Neighbor orders
--knn-n-query N	100000	Number of kNN query points

Covariance flags

Flag	Default	Description
--covariance	jackknife	jackknife or bootstrap
--n-jk	100	Number of jackknife regions
--n-bootstrap	1000	Number of bootstrap draws

Misc flags

Flag	Default	Description
--output-dir	data/	Root directory for HDF5 output
--output-file	auto	Override auto-generated output path
--skip-existing	on	Skip if joint_covariance/cov already exists in output
--n-threads	nproc	CPU threads for Corrfunc / treecorr
--pi-max	100.0	πmax for wp [Mpc]
--dry-run	off	Print plan without writing files or loading sources
--fast-test	off	Smoke-test: n_jk=10, max_ngal=5000, knn_n_query=5000, --subsample enabled for BGS
--max-ngal N	—	Randomly subsample to N galaxies after loading (randoms → 5N)
--log-level	INFO	DEBUG · INFO · WARNING

Checkpoint / resume

Each JK or bootstrap iteration is written to _checkpoint/subsamples in the output HDF5 immediately after completion. Re-running the same command resumes from where it stopped — already-finished rows are skipped. To add more iterations, increase --n-jk (or --n-bootstrap) and re-run; new rows are appended to the checkpoint and the covariance is re-computed at the end.

Example commands — LS10 M★≥11.5, sys-comb weights

See scripts/run_ls10_m11.5_examples.sh for the full example set, or run it directly:

bash scripts/run_ls10_m11.5_examples.sh [--fast-test] [--dry-run]

Selected examples:

# Default: SMF + WP + ESD_HSC + ESD_DES  (sys-comb, 100 JK)
JAX_PLATFORMS=cpu python scripts/measure_joint_sumstat.py \
    --mstar-min 11.5 --weight-variant sys-comb

# All lensing surveys
JAX_PLATFORMS=cpu python scripts/measure_joint_sumstat.py \
    --mstar-min 11.5 --weight-variant sys-comb \
    --stats ESD_HSC ESD_DES ESD_KIDS

# All summary statistics (no LF — requires abs_mag column)
JAX_PLATFORMS=cpu python scripts/measure_joint_sumstat.py \
    --mstar-min 11.5 --weight-variant sys-comb \
    --stats SMF WP ESD_HSC ESD_DES ESD_KIDS WTHETA KNN

# Bootstrap covariance, custom bins
JAX_PLATFORMS=cpu python scripts/measure_joint_sumstat.py \
    --mstar-min 11.5 --weight-variant sys-comb \
    --stats SMF WP ESD_HSC ESD_DES \
    --covariance bootstrap --n-bootstrap 500 \
    --bins-rp log:15:0.1:40 --bins-mstar arange:11.5:12.6:0.1

# Smoke-test (5000 galaxies, 10 JK regions, subsample BGS footprint)
JAX_PLATFORMS=cpu python scripts/measure_joint_sumstat.py \
    --mstar-min 11.5 --weight-variant sys-comb \
    --fast-test --stats SMF WP

---

Output directory layout

data/
├── lf_smf/         BGS and mock SMF (one .h5 per sample × weight variant)
├── twopcf/         BGS and mock wp, wtheta
├── lensing/        BGS ΔΣ (HSC/DES)
├── BGS_Mstar{X}/   joint covariance files per mass threshold
├── mocks/
│   ├── lf_smf/     mock SMF
│   ├── twopcf/     mock wp, wtheta
│   └── joint/      mock joint SMF + wp
├── COSMOS/         cosmos_smf_z*.fits, cosmos_lf_{band}_z*.fits
└── GAMA/           gama_smf_z*.fits, gama_lf_r_z*.fits