# Running the Full Extraction

## What's Been Proven

The extraction code (`scripts/04_extract_survey_reading_lists.py`) works — it:
- ✅ Streams unarXive 2024 from HuggingFace
- ✅ Detects CS survey papers by title pattern
- ✅ Extracts arXiv IDs from bib entries (3 sources: ids field, DOI, raw text)
- ✅ Classifies sections (Related Work, Methods, Introduction, Experiments)
- ✅ Assigns 5-tier relevance labels based on section + frequency
- ✅ Checkpoints every 25 surveys (resume-safe)

## Why It Didn't Complete on Sandbox

The unarXive dataset is 97GB stored as 3 tar.gz parts. Streaming from HuggingFace:
- Downloads chunks over network
- At 26 papers/sec, scanning 800K papers takes ~8 hours
- Network drops cause `tarfile.ReadError: unexpected end of data`
- Sandbox has 4-hour session limits

## How to Run It (3 options)

### Option A: Google Colab (FREE, recommended)

```python
# In a Colab notebook:
!pip install datasets pyarrow numpy tqdm huggingface_hub
!huggingface-cli login

# Clone the script
!wget https://huggingface.co/siddhm11/researchit-reranker-phase6/resolve/main/scripts/04_extract_survey_reading_lists.py

# Run (takes 2-4 hours on Colab's stable network)
!python 04_extract_survey_reading_lists.py --max-surveys 300 --push-to-hub
```

Colab has a direct fast connection to HuggingFace servers and 12-hour session limits.

### Option B: Your laptop

```bash
pip install datasets pyarrow numpy tqdm huggingface_hub
huggingface-cli login
python scripts/04_extract_survey_reading_lists.py --max-surveys 300 --push-to-hub
```

### Option C: Kaggle Notebook (free GPU not needed, just stable network)

Same as Colab but on Kaggle. 9-hour session limit.

## What Happens After Extraction

Once `eval_v2_final/` is populated in the dataset repo, run:
```bash
python scripts/05_evaluate_on_surveys.py --eval-file eval_v2_final/eval_survey_reading_lists.parquet
```

Then retrain V6 on the proper data — the new labels from real survey paper sections will be genuinely different from the citation-graph-only labels.

## Expected Yield

From the 2 surveys extracted in testing:
- CS survey papers appear at ~1 per 2500 papers scanned
- At 300 target surveys × 2500 = need to scan ~750K papers
- At 26/sec = ~8 hours (Colab can handle this)
- arXiv resolution rate: 14-37% (older papers lower, newer ML papers higher)
- Expected dataset: ~15K-30K citation rows with section context