# Dataset target provenance

## Pinned source

- Dataset: `jglaser/binding_affinity`
- Revision: `11e49b7ece33d62afd7f65bc05ce60ad37f9ba7b`
- File: `data/all.parquet`
- SHA-256: `d50f4cae794298f32c39f75a9e8603528d20e50e4fa757025def412f5b2a78d8`

## Why the target is pAffinity, not pKd

The combined Parquet exposes `affinity_uM` and `neg_log10_affinity_M`, but it does not retain the measurement type or source row.

The pinned `bindingdb_single.ipynb` selects:

```python
['IC50 (nM)', 'Ki (nM)', 'Kd (nM)', 'EC50 (nM)']
```

Its `to_uM` function converts every available field to micromolar, appends the values to one list, and returns:

```python
np.mean(vals[~np.isnan(vals)])
```

It stores the result in one column:

```python
df_affinity['affinity_uM'] = df_affinity[
    ['IC50 (nM)', 'Ki (nM)', 'Kd (nM)', 'EC50 (nM)']
].parallel_apply(to_uM, axis=1)
```

The pinned `combine_dbs.ipynb` then keeps only `seq`, `smiles`, and `affinity_uM` from each source and computes:

```python
df_nr['neg_log10_affinity_M'] = 6 - np.log(df_nr['affinity_uM']) / np.log(10)
```

Therefore:

- the numeric transform is `-log10(affinity in M)`;
- not every row is a dissociation constant;
- some BindingDB rows may average heterogeneous measurement types before the logarithm;
- the final file cannot separate `Ki`, `Kd`, `IC50`, or `EC50`;
- the final file cannot reconstruct assay conditions, censoring operators, or original source provenance.

MitoInteract recovery calls this value **pAffinity**. It must not be presented as pKd or converted to a physical Kd.

## Source-aware measurement contract (implemented)

`src/mitointeract_recovery/source_measurement.py` now implements, as tested code, the field-level contract the successor dataset must satisfy. Each raw record is captured as an immutable `AffinityMeasurement` with full provenance, and the module enforces:

- **Explicit Kd/Ki/IC50/EC50 semantics** via the closed `MeasurementType` enum; types are never merged, averaged, or silently relabeled.
- **Exact units**: only `M`, `mM`, `uM`, `µM`, `nM`, `pM` are accepted, conversions use exact molar factors, and the originally reported value and unit are preserved.
- **Relation-aware censoring**: the operator (`=`, `<`, `<=`, `>`, `>=`, `~`) is stored per record, survives unit conversion, and the inequality is reversed correctly under the `-log10` transform (`pbound`), since the transform flips the sign.
- **Exact-Kd-only pKd gating**: `exact_kd_pkd` raises unless the record is a Kd measurement with an exact `=` relation; censored and `~` records are never promoted to pKd.
- **Stable canonical pair IDs matching preparation**: the same sequence normalization, SMILES canonicalization, `stable_id` hashing, and `sequence + "\0" + canonical_smiles` pair convention as `scripts/prepare_balm_kd.py`, so IDs are interoperable with the prepared manifests.
- **Fail-closed replicate aggregation with source provenance**: `aggregate_exact_replicates` rejects mixed pairs, mixed measurement types, and non-exact relations instead of guessing, and returns the median value and median `-log10(M)` together with ordered provenance — bare `source_record_ids`, collision-proof `(source_database, source_record_id)` `sources`, and `assay_ids` / `citations` with `None` entries kept.

The contract now has a real publisher-source implementation. `bindingdb_adapter.py` and `prepare_bindingdb_gold_kd.py` parse the pinned BindingDB 202607 curated-articles export, join reactant sets to assay names/descriptions, and emit both the full source-record stream and an exact-Kd projection. See [`BINDINGDB_SOURCE_AWARE.md`](BINDINGDB_SOURCE_AWARE.md).

## BindingDB source-aware pilot

The checked pilot processes all 93,712 rows from the SHA-256-pinned monthly BindingDB 202607 files under CC BY 3.0. It emits:

- 87,315 source records with explicit measurement type and relation;
- 2,615 Kd records, of which 2,432 are exact, assay-joined Kd source records;
- 2,198 unique protein–ligand pairs, 442 proteins, and 245 assays in that exact-Kd projection;
- explicit retention and audit of 695 source records without a completed assay join;
- complete row accounting, including multichain, missing/invalid SMILES, and no-supported-measurement rejections;
- deterministic output hashes recorded in `reports/bindingdb-source-aware-audit.json`.

The raw exact-Kd records are not aggregated across assays. The 47 records with pKd below 3 and three records above 12 are review flags, not automatic filters. Repeated BindingDB reactant-set identifiers are disambiguated by the physical source-row ordinal, while the original identifier remains separately preserved.

This is a data-quality result, not a model result. No split, baseline, neural training, or model-performance claim exists for the source-aware data yet.

## What is still missing

1. **No cross-assay aggregation policy has passed review.** The 2,432 exact-Kd records remain source records. Any aggregation must remain assay-aware and retain every contributing record.
2. **No source-aware train/validation/test manifests exist.** Protein-cluster, scaffold, and publication-date leakage gates must be generated before modeling.
3. **No source-aware baseline has been run.** Descriptor, character, and mean controls must be repeated on the new manifests before neural work.
4. **The monthly publisher URLs are volatile.** A publication benchmark must use BindingDB's long-term quarterly archive or preserve the exact pinned 202607 files under their recorded checksums.
5. **Extreme pKd records need human review.** Diagnostic flags identify the tails without silently deleting potentially valid measurements.
6. **BALM remains a derived benchmark.** Its `BindingDB_filtered/data.csv` still lacks raw value, unit, relation, assay, and source-record fields. It remains useful for continuity but cannot substitute for the new provenance track.

## Consequence for v2

The mixed 10,000-pair pAffinity experiment remains useful only for pipeline recovery and leakage analysis. The BALM pKd benchmark remains the prior reproducible derived benchmark. The BindingDB source-aware track is now the primary data-quality track because it retains measurement type, original value and unit, relation operator, assay and citation fields, canonical entities, and source release identity end to end.

The next defensible milestone is an assay-aware exact-Kd pair table with full contributing provenance, followed by protein-cluster, scaffold, and temporal split manifests and cheap controls. Neural scaling remains gated on those artifacts.