Initial SHARM model card (training in progress, no weights yet)

README.md

@@ -0,0 +1,153 @@
+---
+license: mit
+model-index:
+  - name: hab3-social-rearrange-sharm
+    results: []
+links:
+  - name: GitHub
+    url: https://github.com/bkubwimana/ivalab
+tags:
+  - reinforcement-learning
+  - habitat-baselines
+  - habitat
+  - embodied-ai
+  - multi-agent
+  - social-rearrangement
+  - dd-ppo
+  - robotics
+  - shared-memory
+  - distributed-shared-memory
+  - recurrent-memory
+  - stigmergy
+  - communication
+  - physical-ai
+  - embodied-coordination
+datasets:
+  - ai-habitat/hab3_episodes
+---
+
+# Habitat 3.0 Social Rearrangement — SHARM (Shared Hierarchical Recurrent Memory)
+
+> **Status: training in progress.** This card describes the architecture and
+> training plan. Trained weights will be uploaded once the Stage 3/4 runs
+> complete and have been evaluated on the fixed validation split.
+
+The **Pereason + Go + Fabric + SHARM** multi-agent coordination model.
+Two embodied agents (a Spot robot arm and a humanoid) cooperate to
+rearrange objects across HSSD home scenes.
+
+SHARM extends the [Fabric model](https://huggingface.co/edge-inference/hab3-social-rearrange-fabric)
+with a learned, persistent, *shared* memory layer. Where Fabric
+exchanges *per-step perceptual messages* between agents, SHARM gives
+each agent a typed slot bank that accumulates state across an episode
+and is gossiped to the partner — directly inspired by stigmergic
+coordination (ant pheromone trails) re-cast as a learned policy module.
+
+This work is part of the thesis *"Scalable Multi-Agent Coordination
+Using a Shared-Context Architecture for Embodied Robotics"* by
+Benjamin Kubwimana. SHARM is the **learned** counterpart to the
+hand-coded CollabTime DSM evaluated earlier in the thesis.
+
+## What's in this repo
+
+| File | Description |
+|---|---|
+| *(weights pending)* | Trained Stage 3 / Stage 4 SHARM checkpoints will land here once eval is complete |
+
+## Architecture
+
+```
+RGB+lang ─→ SmolVLM2 (frozen, 350M)         ┐
+depth ────→ DepthAnythingV2 (trained, 25M) ─┴→ fused tokens (B, S, 960)
+                                                       ↓
+                          Fabric: encode 128-d msg, broadcast,
+                          cross-attend partner msg, gated residual
+                                                       ↓
+                          SHARM: write to typed slot bank,
+                          self-attend over slots, encode gossip,
+                          cross-attend partner gossip
+                                                       ↓
+                          Go transformer (PPO) ─→ skill choice
+```
+
+### SHARM specifics
+
+| Component | Detail |
+|---|---|
+| Slot bank | 8 typed slots: 4 perception + 2 task + 2 intent |
+| Slot dim | 64 |
+| Memory state | 520 floats per env, packed into Habitat's `recurrent_hidden_states` buffer |
+| Write head | Sparse content-addressed: produces (key, content, gate) per slot, soft-routed by similarity to learned slot key embeddings |
+| Decay | Stigmergic — multiplicative attenuation by slot age, half-life ~100 steps |
+| Read | Self-attention over own slots + cross-attention over partner gossip |
+| Gossip | `(K, slot_dim) → (K, msg_dim=128)` encoder, decoded by partner |
+| Training | Truncated BPTT through Habitat's `rnn_build_seq_info` |
+
+### Auxiliary losses (training-time only)
+
+SHARM is bootstrapped with two non-PPO gradient signals; both annealed:
+
+| Loss | What it pushes | Schedule |
+|---|---|---|
+| Reconstruction | Slots must encode partner's hand-coded task state (skill, holding, target) | weight 1.0 → 0.0 over first 50% of training |
+| Future-latent | Slots must enable predicting own pooled fused features at horizon h=4 | weight 0.5 → 0.1, sustained |
+
+The reconstruction loss serves as a teacher (so the write head learns
+*what* to encode) and is released so the model can discover signals
+beyond the human-designed schema.
+
+## Training plan
+
+| Stage | Frames | Purpose |
+|---|---|---|
+| 3a (sanity) | ~5M | Verify wiring, no crashes, recon loss converges |
+| 3b (full) | ~55M | Headline Stage 3 result |
+| 4 (ablation) | ~30M | Drop reconstruction aux, warm-start from Stage 3 best |
+
+Frame budget is capped at 60M per run. Total compute budget ~90M
+frames across all stages.
+
+## Evaluation (planned)
+
+All checkpoints will be evaluated on the same fixed 100-episode
+validation split used for the thesis Table 4.1, with metrics:
+
+- `pddl_success` (primary)
+- `num_agents_collide`
+- `episode_steps`
+
+For comparison, prior work on this benchmark:
+
+| Condition | Success | Collide | Source |
+|---|---|---|---|
+| Oracle baseline | 0.28 | 0.71 | Thesis Table 4.1 |
+| Oracle + CollabTime (hand-coded DSM) | 0.51 | 0.48 | Thesis Table 4.1 |
+| Trained RL baseline (ResNet-LSTM) | 0.15 | 0.47 | Thesis Table 4.1 |
+| Trained RL + CollabTime | 0.37 | 0.32 | Thesis Table 4.1 |
+| **Fabric** ([model](https://huggingface.co/edge-inference/hab3-social-rearrange-fabric)) | 0.43 | 0.16 | Thesis Table 4.1 |
+| **SHARM Stage 3** | *(pending)* | *(pending)* | — |
+| **SHARM Stage 4** | *(pending)* | *(pending)* | — |
+
+## How to use (once weights land)
+
+Weights will load via the same Habitat-baselines harness as the Fabric
+release. See the [GitHub repository](https://github.com/bkubwimana/ivalab)
+for the full training and evaluation pipeline.
+
+```bash
+git clone https://github.com/bkubwimana/ivalab.git
+cd ivalab && git submodule update --init --recursive
+git checkout feature/fabric-dsm  # SHARM lives on this branch until merged
+bash scripts/eval_trained.sh pereason_go_fabric_dsm
+```
+
+## Citation
+
+```
+@thesis{kubwimana_thesis_2026,
+  title  = {Scalable Multi-Agent Coordination Using a Shared-Context
+            Architecture for Embodied Robotics},
+  author = {Kubwimana, Benjamin},
+  year   = {2026}
+}
+```