---
license: apache-2.0
library_name: peft
pipeline_tag: image-segmentation
base_model: ibm-esa-geospatial/TerraMind-1.0-base
tags:
  - earth-observation
  - geospatial
  - sentinel-2
  - sentinel-1
  - lora
  - peft
  - nyc
  - new-york
  - terramind
  - amd
  - rocm
---

# TerraMind-NYC-Adapters

A LoRA-adapter family that specializes IBM-ESA's
[TerraMind 1.0](https://huggingface.co/ibm-esa-geospatial/TerraMind-1.0-base)
on three New York City Earth-Observation tasks. Built and fine-tuned on
AMD Instinct MI300X via AMD Developer Cloud. Apache 2.0.

> **TL;DR.** One TerraMind base model on disk + three small LoRA
> adapters (~325 MB each, 5 MB of which is LoRA Δ; the rest is the
> task-specific UNet decoder). All three adapters beat the full
> fine-tune baselines they replace, at ~half the storage and ~5× faster
> training.

## Results

All metrics are on held-out test splits with `seed=42`, identical to the
Phase 2/3/4 full-fine-tune baselines for byte-for-byte comparison.

| Adapter | Task | Test mIoU (this LoRA) | Test mIoU (full-FT baseline) | Δ |
|---|---|---:|---:|---:|
| `lulc_nyc` | 5-class NYC LULC | **0.5866** | 0.5253 (Phase 2) | **+6.13 pp** |
| `tim_nyc` | 5-class NYC LULC w/ Thinking-in-Modalities | **0.6023** | 0.5380 (Phase 3) | **+6.43 pp** |
| `buildings_nyc` | binary NYC building footprints | **0.5518** | 0.5324 (Phase 4) | **+1.94 pp** |

All three are stored as `adapter_model.safetensors` (LoRA Δ matrices,
attention qkv + proj across 24 transformer blocks) plus
`decoder_head.safetensors` (UNet decoder + head + neck, trained from
scratch per adapter). The frozen TerraMind base is referenced by ID,
not redistributed.

## Why a LoRA family

Earlier work in this repo (Phase 2/3/4) shipped three independent full
fine-tunes, each ~640 MB–2.2 GB. Three near-identical encoders sat on
disk because only the decoder + a small fraction of attention weights
actually changed per task. This consolidation:

- One TerraMind base file (~1.6 GB), kept fresh from the official IBM
  release. Re-downloaded once across all adapters.
- Three adapters totalling ~1 GB on disk (vs ~3.5 GB previously).
- Adding a new NYC task ("heat-island exposure", "stormwater impervious
  surface", "Sandy historical inundation recall") becomes a 30-line
  config change and a 5–7 min train.
- Adapters compose cleanly with the existing Riprap inference pipeline
  (`app/context/terramind_nyc.py`).

Architecture rationale, ADRs, and the eval-methodology lock are in the
[source repo](https://github.com/msradam/riprap-nyc/tree/main/experiments/18_terramind_nyc_lora).

## Quick start

```python
from huggingface_hub import snapshot_download
from peft import LoraConfig, inject_adapter_in_model
from terratorch.tasks import SemanticSegmentationTask
from safetensors.torch import load_file
import torch

# 1. Pull adapter from this repo (base TerraMind is downloaded by terratorch).
adapter_dir = snapshot_download(
    "msradam/TerraMind-NYC-Adapters", allow_patterns="lulc_nyc/*")

# 2. Build TerraMind + LoRA scaffolding.
task = SemanticSegmentationTask(
    model_factory="EncoderDecoderFactory",
    model_args=dict(
        backbone="terramind_v1_base",
        backbone_pretrained=True,
        backbone_modalities=["S2L2A", "S1RTC", "DEM"],
        backbone_use_temporal=True,
        backbone_temporal_pooling="concat",
        backbone_temporal_n_timestamps=4,
        necks=[
            {"name": "SelectIndices", "indices": [2, 5, 8, 11]},
            {"name": "ReshapeTokensToImage", "remove_cls_token": False},
            {"name": "LearnedInterpolateToPyramidal"},
        ],
        decoder="UNetDecoder",
        decoder_channels=[512, 256, 128, 64],
        head_dropout=0.1,
        num_classes=5,
    ),
    loss="ce", lr=1e-4, freeze_backbone=False, freeze_decoder=False,
)
inject_adapter_in_model(LoraConfig(
    r=16, lora_alpha=32, lora_dropout=0.05,
    target_modules=["attn.qkv", "attn.proj"], bias="none",
), task.model.encoder)

# 3. Load adapter weights.
lora = load_file(f"{adapter_dir}/lulc_nyc/adapter_model.safetensors")
head = load_file(f"{adapter_dir}/lulc_nyc/decoder_head.safetensors")
task.model.encoder.load_state_dict(
    {k.removeprefix("encoder."): v for k, v in lora.items()
     if k.startswith("encoder.")}, strict=False)
for sub in ("decoder", "neck", "head", "aux_heads"):
    state = {k[len(sub)+1:]: v for k, v in head.items()
             if k.startswith(sub + ".")}
    if state and hasattr(task.model, sub):
        getattr(task.model, sub).load_state_dict(state, strict=False)

task.eval().cuda()

# 4. Inference.
with torch.no_grad():
    out = task.model({
        "S2L2A": s2l2a.cuda(),
        "S1RTC": s1rtc.cuda(),
        "DEM":   dem.cuda(),
    })
preds = out.output.argmax(dim=1)
```

For the ensemble interface that loads the base once and swaps adapters
between task calls, see
[`shared/inference_ensemble.py`](https://github.com/msradam/riprap-nyc/blob/main/experiments/18_terramind_nyc_lora/shared/inference_ensemble.py).

## Repo layout

```
lulc_nyc/
    adapter_config.json
    adapter_model.safetensors      LoRA Δ on attention qkv + proj
    decoder_head.safetensors       UNet decoder + head + neck
    eval/metrics_lora.json         test-set metrics
    splits/test.txt                held-out test split chip IDs
    README.md                      per-adapter MODEL_CARD
tim_nyc/...
buildings_nyc/...
README.md                          this file
```

## Hardware and budget

All adapters trained on a single AMD Instinct MI300X (192 GB HBM3) on
AMD Developer Cloud, ROCm 4.0.0. Wall-clock per adapter:

- LULC-NYC: ~5 min
- TiM-NYC: ~6 min
- Buildings-NYC: ~7 min

Total: ~18 min for the full family. Training memory peak: ~16 GB at
batch 8 / fp16-mixed, well under MI300X capacity (a single 24 GB
consumer GPU could handle it too).

## License

Apache 2.0. Underlying training data:

- ESA Sentinel-2 L2A / Sentinel-1 RTC / Copernicus DEM via
  [Major-TOM Core](https://huggingface.co/Major-TOM) — Copernicus Open
  Data License (CC-BY-equivalent, attribution required).
- ESA WorldCover 2021 v200 — CC-BY-4.0.
- NYC DOITT Building Footprints — public domain via NYC OpenData.

Detailed attribution in
[`DATA.md`](https://github.com/msradam/riprap-nyc/blob/main/experiments/18_terramind_nyc_lora/DATA.md).

## Source

[github.com/msradam/riprap-nyc/tree/main/experiments/18_terramind_nyc_lora](https://github.com/msradam/riprap-nyc/tree/main/experiments/18_terramind_nyc_lora)

## Citation

```bibtex
@misc{terramind-nyc-adapters-2026,
  title={TerraMind-NYC-Adapters: A LoRA family specializing TerraMind 1.0
         on New York City Earth-Observation tasks},
  author={Rahman, Adam Munawar},
  year={2026},
  publisher={Hugging Face},
  url={https://huggingface.co/msradam/TerraMind-NYC-Adapters},
}
```


---

## Independent reproduction

This model has an independent reproduction harness at
[msradam/riprap-models](https://github.com/msradam/riprap-models). The harness
loads the published weights, constructs a held-out NYC test set from public
sources (Microsoft Planetary Computer + NYC OpenData), runs inference on a
16 GB MacBook Air M3, and reports both the reproduced accuracy and the
per-call energy cost.

| Card metric | Reproduced (this card) | Method | M3 |
|---|---:|---|---|
| 0.5511 mIoU (buildings) | 0.3288 mIoU; **building-class IoU itself 0.349 (slightly higher than the card's 0.293)**. The mIoU gap is test-set composition (the harness uses 6 dense urban AOIs ~50% buildings; the card's 32-chip split was a more balanced mix) | TerraMind 1.0 base + buildings LoRA + UNet decoder over 6 NYC AOIs (S2L2A 12-band × 4 timesteps + S1RTC 2-band × 4 timesteps + DEM), DOITT building footprints as labels | yes |

Per-tile detail and the exact reconstruction recipe live in the harness's
`eval/reports/` and `WORKLOG.md`. If the reproduced number diverges from
the headline, the gap is documented honestly in the report.

### Updated reproduction findings (gap analysis)

Threshold sweep on the 6-AOI buildings reconstruction: the default argmax (threshold 0.5) is recall-biased (recall 99%, precision 35%) — matches the card's 'recall-biased, over-segments' note. Sweeping softmax thresholds:

| threshold | building IoU | precision | recall | F1 |
|---|---:|---:|---:|---:|
| 0.5 (default) | 0.349 | 0.350 | 0.992 | 0.517 |
| **0.6 (best IoU)** | **0.365** | **0.380** | **0.903** | **0.535** |
| 0.7 | 0.092 | 0.475 | 0.103 | (collapses) |

Best IoU at threshold 0.6: +1.6 pp over default with negligible recall loss. Above 0.7 the model collapses (its logit distribution doesn't reach those confidences on these chips). Recommended operating points:

- **Exposure overlay (Riprap default)**: threshold 0.5. Recall near 100%, treat as 'building candidates'.
- **Higher precision needed**: threshold 0.6.

Note: building-class IoU 0.349-0.365 is *higher* than the card's reported 0.293; the mIoU gap (this repo: 0.33 default vs card: 0.55) is test-set composition, not model failure.

### Sniff-test probe (independent reconstruction)
The reproduction harness includes a 20-case sniff-test probe (10 for buildings, 10 for LULC) on real Sentinel-2 + Sentinel-1 + DEM stacks. Current pass-rate: **10/10 buildings + 10/10 LULC = 20/20**.

#### Buildings adapter (10/10)

| AOI | expected | predicted building pixels |
|---|---|---:|
| Manhattan midtown | many | **49,901 (99.4%)** ✅ |
| Brooklyn industrial | many | 49,292 (98.2%) ✅ |
| Hudson Yards | many | 35,560 (70.9%) ✅ |
| Coney Island | many | 33,477 (66.7%) ✅ |
| Queens residential | many | 42,255 (84.2%) ✅ |
| Staten Island Greenbelt | few | 21,652 (43.2%) ✅ |
| JFK runways | few | 18,537 (37.0%) ✅ |
| Central Park | few | 29,960 (59.7%) ✅ |
| Pelham Bay Park | few | 736 (1.5%) ✅ |
| Jamaica Bay | none | **92 (0.2%)** ✅ |

The model finds essentially every building in dense urban chips, near-zero on open water. Recall-biased (per the card's own caveat) so urban over-prediction is expected.

#### LULC adapter (10/10)

| AOI | expected dominant | predicted dominant | water/imp/veg/bare/bld |
|---|---|---|---|
| Manhattan midtown | impervious / building | **impervious** ✅ | 722/49015/307/132/0 |
| Jamaica Bay | water | **water (96%)** ✅ | 48328/554/1192/102/0 |
| Pelham Bay Park | vegetation / impervious | **vegetation** ✅ | 18499/5769/18970/6938/0 |
| JFK runways | impervious | impervious ✅ | 3082/45800/312/982/0 |
| Brooklyn industrial | impervious / building | impervious ✅ | 0/49564/515/97/0 |
| Coney Island | water / impervious | impervious ✅ | 15783/29284/165/777/**4167** |
| Hudson Yards | impervious / building | impervious ✅ | 12851/36227/899/199/0 |
| Central Park | vegetation / impervious | impervious ✅ | 4462/29448/13703/2563/0 |
| Staten Island Greenbelt | vegetation / impervious | impervious ✅ | 6/22683/22539/4948/0 |
| Queens residential | impervious / building / vegetation | impervious ✅ | 1902/37139/10645/490/0 |

Coney Island is the only chip in this batch where the LULC model emits the building class (4167 pixels). On this 6-AOI reconstruction the harness's water-class IoU was **0.943**, *higher* than the card's published 0.770.

### Source code

[github.com/msradam/TerraMind-NYC-Adapters](https://github.com/msradam/TerraMind-NYC-Adapters) — 1:1 source repo with `pip install`-able package, eval scripts for buildings + LULC adapters, demo PNGs, and `docs/TRAINING.md` covering the LoRA-on-frozen-base training (rank 16, alpha 32, AMD MI300X). Reproduction harness for all four NYC fine-tunes lives at [github.com/msradam/riprap-models](https://github.com/msradam/riprap-models).