seqSight
/

seqlens-v2-micro-16k

@@ -112,6 +112,43 @@ Pre-trained on prokaryotic genomes from [OpenGenome2](https://huggingface.co/dat
 Note: Genus classification is composition-dominated (GC content, tetranucleotide frequencies), where k-mer baselines are expected to be competitive. The coding/non-coding task better reflects the model's learned structural and positional representations.
 ## Usage
 ```python
@@ -155,7 +192,7 @@ with torch.no_grad():
 - Pre-trained on prokaryotic genomes only — may underperform on eukaryotic tasks
 - 16K context may truncate long contigs; longer variants planned
 - Current model trained for ~1.2B tokens; extended training may improve performance
-- Not validated on community-standard benchmarks (GenomicBenchmarks, GUE) yet
 ## Citation

 Note: Genus classification is composition-dominated (GC content, tetranucleotide frequencies), where k-mer baselines are expected to be competitive. The coding/non-coding task better reflects the model's learned structural and positional representations.
+## Extended Benchmark Comparison
+All evaluations below use the frozen linear probe protocol (frozen embeddings → LogisticRegression). Published fine-tuned baselines (e.g., ProkBERT's fine-tuned MCC scores) are not directly comparable — this is an apples-to-apples comparison across models with a fixed downstream classifier.
+### ProkBERT Prokaryotic Benchmarks (accuracy)
+| Task | v2_novel | v2_standard | NTv3-8M | Caduceus | SeqLens_v1 | NT-v2 | DNABERT-2 |
+|---|---|---|---|---|---|---|---|
+| Phage ID (L512) | 0.751 | 0.730 | 0.762 | 0.653 | 0.690 | 0.626 | 0.652 |
+| Phage ID (L1024) | 0.805 | 0.775 | 0.808 | 0.650 | 0.713 | 0.659 | 0.700 |
+| Phage ID (L2048) | 0.838 | 0.815 | 0.834 | 0.675 | 0.770 | 0.713 | 0.723 |
+| Promoter (sigma70) | 0.653 | 0.657 | 0.674 | 0.665 | 0.628 | 0.594 | 0.591 |
+| Promoter (multispecies) | 0.582 | 0.584 | 0.630 | 0.608 | 0.588 | 0.572 | 0.545 |
+| Lifestyle (BASEL) | 0.705 | 0.737 | 0.753 | 0.702 | 0.735 | 0.669 | 0.700 |
+| Lifestyle (E. coli, held-out) | 0.648 | 0.667 | 0.706 | 0.627 | 0.660 | 0.607 | 0.638 |
+| Lifestyle (Extremophile) | 0.813 | 0.871 | 0.835 | 0.852 | 0.775 | 0.823 | 0.797 |
+| **Average** | **0.724** | **0.729** | **0.751** | **0.679** | **0.695** | **0.658** | **0.668** |
+NTv3-8M leads this suite (trained on 9T bp across all species, ~36× more data than our 248B prokaryotic tokens). Our v2_standard (this model) is the best of our own three recipes here — the opposite ranking from CDS/GenomicBenchmarks, where the novel recipe wins. See `PROJECT_STATE.md` §3.4 for a per-task recipe breakdown.
+### GenomicBenchmarks — Eukaryotic Tasks (accuracy)
+| Task | v2_novel | v2_standard | NTv3-8M | Caduceus | SeqLens_v1 | NT-v2 | DNABERT-2 |
+|---|---|---|---|---|---|---|---|
+| Mouse Enhancers | 0.822 | 0.826 | 0.810 | 0.744 | 0.806 | 0.802 | 0.727 |
+| Coding vs Intergenic | 0.895 | 0.904 | 0.927 | 0.936 | 0.929 | 0.886 | 0.948 |
+| Human vs Worm | 0.906 | 0.928 | 0.955 | 0.967 | 0.969 | 0.941 | 0.981 |
+| Enhancers (Cohn) | 0.730 | 0.726 | 0.734 | 0.746 | 0.754 | 0.721 | 0.809 |
+| Enhancers (Ensembl) | 0.715 | 0.729 | 0.735 | 0.747 | 0.771 | 0.749 | 0.758 |
+| NonTATA Promoters | 0.830 | 0.842 | 0.852 | 0.865 | 0.864 | 0.831 | 0.891 |
+| OCR (Ensembl) | 0.661 | 0.676 | 0.670 | 0.682 | 0.692 | 0.670 | 0.677 |
+| **Average (excl. Regulatory)** | **0.794** | **0.805** | **0.812** | **0.813** | **0.826** | **0.800** | **0.827** |
+The `human_ensembl_regulatory` task is excluded: our single-nucleotide tokenizer preserves exact sequence length, and length alone is a near-complete shortcut for this task's 3-way label (a Random Forest trained on sequence length alone reaches 91.3% accuracy). Our models score ~0.99–1.00 on this task as an artifact of that leakage, not because of learned regulatory biology. Despite training exclusively on prokaryotic genomes, our 10M models remain competitive with human/multi-species models up to 10× our size on the remaining 7 eukaryotic tasks.
+Full results: `experiments/eval_reports/genomic_benchmarks/comparison.json` and `experiments/eval_reports/prokbench/comparison.json`.
 ## Usage
 ```python
 - Pre-trained on prokaryotic genomes only — may underperform on eukaryotic tasks
 - 16K context may truncate long contigs; longer variants planned
 - Current model trained for ~1.2B tokens; extended training may improve performance
+- The `human_ensembl_regulatory` GenomicBenchmarks task is excluded from our averages — it's a confirmed sequence-length artifact of single-nucleotide tokenization, not a genuine capability (see Extended Benchmark Comparison above)
 ## Citation