Data format
===========

All output files are HDF5. Each file contains one or more groups (one per statistic).
The file root carries a ``git_hash`` attribute.

HDF5 group schema
-----------------

``smf/{sample_name}`` and ``lf/{sample_name}``::

   smf/{sample_name}/
   ├── attrs            (provenance — see table below)
   ├── cosmology/       H0, Om0, Ok0, Ob0
   ├── bin_edges        (N+1,)  [dex or mag]
   ├── log10mstar_centres | mag_centres  (N,)
   ├── phi              (N,)    [Mpc⁻³ dex⁻¹ or Mpc⁻³ mag⁻¹]
   ├── phi_err          (N,)
   └── cov              (N, N)

``twopcf/{stat}/{sample_name}`` (stat = wp · wtheta · xi)::

   ├── bin_edges    (N+1,)  [Mpc or arcmin]
   ├── sep_centres  (N,)
   ├── xi           (N,)    [Mpc or dimensionless]
   └── cov          (N, N)

``esd/{sample_name}``::

   ├── rp_bin_edges  (N+1,)  [Mpc]
   ├── rp_centres    (N,)
   ├── delta_sigma   (N,)    [M_sun/pc²]
   └── cov           (N, N)

``joint_covariance`` (joint files only)::

   ├── data_vector    (N_total,)          [phi | wp | ΔΣ]
   ├── err_jackknife  (N_total,)
   ├── cov            (N_total, N_total)
   ├── jk_subsamples  (N_JK, N_total)
   ├── mstar_centres  (N_smf,)  [dex]
   └── rp_centres     (N_wp,)   [Mpc]

Provenance attributes
----------------------

Every group carries these ``attrs``:

.. list-table::
   :widths: 22 12 10 56
   :header-rows: 1

   * - Attribute
     - Type
     - gga_model
     - Description
   * - ``script_name``
     - str
     - —
     - Script that produced this group
   * - ``git_hash``
     - str
     - —
     - Short git commit hash at run time
   * - ``run_date``
     - str
     - —
     - UTC ISO-8601 timestamp
   * - ``survey``
     - str
     - —
     - ``bgs`` · ``mock`` · ``cosmos`` · ``gama``
   * - ``statistic``
     - str
     - —
     - ``smf`` · ``lf`` · ``wp`` · ``wtheta`` · ``esd`` · …
   * - ``input_catalogue``
     - str
     - —
     - Semicolon-separated absolute input paths
   * - ``z_eff``
     - float
     - **yes**
     - ``np.average(z, weights=w)``
   * - ``n_gal``
     - int
     - —
     - Number of galaxies
   * - ``n_rand``
     - int
     - —
     - Number of randoms (``-1`` if not applicable)
   * - ``estimator``
     - str
     - **yes**
     - ``vmax`` · ``swml`` · ``cminus``
   * - ``z_min``, ``z_max``
     - float
     - **yes**
     - Redshift interval
   * - ``area_deg2``, ``area_sr``
     - float
     - **yes**
     - Survey footprint
   * - ``cov_method``
     - str
     - **yes**
     - ``jackknife`` · ``diagonal_poisson``
   * - ``weight_variant``
     - str
     - **yes**
     - ``uniform`` · ``sys-add`` · ``sys-comb``

Reading example
----------------

.. code-block:: python

   import h5py

   with h5py.File("data/lf_smf/BGS_Mstar9.00_z0.05_0.08-smf-vmax.h5") as f:
       grp = f["smf/BGS_Mstar9.00_z0.05_0.08"]
       phi     = grp["phi"][:]
       cov     = grp["cov"][:]
       z_eff   = grp.attrs["z_eff"]
       area_sr = grp.attrs["area_sr"]

gga_model required fields
--------------------------

.. list-table::
   :widths: 25 75
   :header-rows: 1

   * - Field
     - Notes
   * - ``phi``, ``cov``
     - Measurement and covariance
   * - ``log10mstar_centres``
     - Must match model grid
   * - ``z_eff``
     - Evaluates model at correct epoch
   * - ``area_sr``
     - Normalises volume integral
   * - ``estimator``
     - Selects matching likelihood
   * - ``z_min``, ``z_max``
     - Redshift window for selection function
   * - ``weight_variant``
     - Selects systematic branch
   * - ``cov_method``
     - Hartlap correction flag