---
language:
  - dna
tags:
  - genomics
  - biology
  - dna
  - masked-language-model
  - prokaryotic
  - metagenomics
  - bioinformatics
license: apache-2.0
datasets:
  - arcinstitute/opengenome2
library_name: pytorch
pipeline_tag: fill-mask
model-index:
  - name: seqlens-v2-micro-16k
    results:
      - task:
          type: text-classification
          name: Coding vs Non-coding Classification
        metrics:
          - name: Accuracy (linear probe, frozen)
            type: accuracy
            value: 0.9111
      - task:
          type: text-classification
          name: Genus Classification (50 held-out genera)
        metrics:
          - name: Accuracy (linear probe, frozen)
            type: accuracy
            value: 0.7978
---

# SeqLens v2 Micro 16K

A compact genomic language model pre-trained on prokaryotic genomes for microbial bioinformatics tasks.

## Model Description

SeqLens v2 is a bidirectional genomic language model built on BiMamba (bidirectional Mamba2 SSM) with interleaved sliding-window attention. It is designed for microbial genomics — taxonomic classification, antimicrobial resistance detection, plasmid identification, and metagenomic analysis.

The **Micro** variant is the smallest in the SeqLens v2 family, targeting high-throughput, low-latency inference.

| Property | Value |
|---|---|
| Parameters | 10.3M |
| Hidden dimension | 256 |
| Layers | 8 (BiMamba) + 2 (sliding-window attention at layers 3, 7) |
| Context length | 16,384 tokens (single nucleotide) |
| Vocabulary | A, T, G, C, N, [CLS], [SEP], [PAD], [MASK] (9 tokens) |
| Pre-training objective | Masked Language Modeling (MLM), 15% mask rate |
| Architecture | BiMamba2 + chunked sliding-window attention + SwiGLU FFN |

## Architecture Details

- **BiMamba blocks:** Bidirectional Mamba2 SSM — processes sequences in both forward and reverse directions using shared weights. Provides O(L) scaling with sequence length.
- **Sliding-window attention:** Applied every 4th layer with window size 512. Captures fine-grained local patterns (codons, motifs) that SSMs can miss.
- **Attention-weighted pooling:** Learned pooling for sequence-level embeddings (superior to mean pooling for downstream tasks).
- **SwiGLU FFN:** Gated feed-forward with 4× expansion at each layer.

## Training

### Data

Pre-trained on prokaryotic genomes from [OpenGenome2](https://huggingface.co/datasets/arcinstitute/opengenome2) (Apache 2.0):
- **GTDB v220:** 113,379 species-cluster representative genomes
- Single-nucleotide tokenization, 16,384 bp chunks
- Quality filtered: sequences with >10% N or low Shannon entropy excluded

### Hyperparameters

| Parameter | Value |
|---|---|
| Optimizer | AdamW (β₁=0.9, β₂=0.98, ε=1e-8) |
| Learning rate | 1e-3 (cosine decay to 1e-5) |
| Warmup | 500 steps |
| Weight decay | 0.1 |
| Gradient clipping | 1.0 |
| Precision | BF16 mixed |
| Batch size | 64 effective (8 × 8 GPUs) |
| Total steps | 10,000 |
| Tokens seen | ~1.2B |

### Compute

| Resource | Value |
|---|---|
| Hardware | 8× NVIDIA A100-SXM4-80GB |
| Training time | 57 minutes |
| Framework | PyTorch 2.6.0 + mamba-ssm 2.2.4 |

## Evaluation

### Coding vs Non-coding Classification (linear probe, frozen backbone)

| Model | Params | Accuracy | F1 |
|---|---|---|---|
| **SeqLens v2 Micro** | **10M** | **0.911** | **0.911** |
| SeqLens v1 (89M) | 89M | 0.687 | 0.687 |
| 4-mer baseline | — | 0.588 | 0.588 |
| Random init | 10M | 0.596 | 0.596 |

### Genus Classification (50 held-out genera, linear probe, frozen backbone)

| Model | Params | Accuracy | F1 |
|---|---|---|---|
| 4-mer baseline | — | 0.865 | 0.838 |
| Random init | 10M | 0.826 | 0.768 |
| **SeqLens v2 Micro** | **10M** | **0.798** | **0.730** |

Note: Genus classification is composition-dominated (GC content, tetranucleotide frequencies), where k-mer baselines are expected to be competitive. The coding/non-coding task better reflects the model's learned structural and positional representations.

## Usage

```python
import torch
from model import SeqLensForMLM
from config import SeqLensConfig, MICRO_CONFIG
from tokenizer import NucleotideTokenizer

# Load model
device = torch.device("cuda")
ckpt = torch.load("seqlens-v2-micro-16k.pt", map_location=device)
model = SeqLensForMLM(MICRO_CONFIG).to(device).to(torch.bfloat16)
model.load_state_dict(ckpt["model"])
model.eval()

# Tokenize a DNA sequence
tokenizer = NucleotideTokenizer(max_len=16384)
seq = "ATGCGATCGATCG..." # your DNA sequence
token_ids = torch.tensor([tokenizer.encode(seq)], dtype=torch.long).to(device)

# Get sequence-level embeddings (for classification tasks)
with torch.no_grad():
    embeddings = model.get_embeddings(token_ids, pool="attention")  # (1, 256)

# Or get per-position predictions (MLM)
with torch.no_grad():
    output = model(token_ids)
    logits = output["logits"]  # (1, L, 9)
```

## Model Family

| Variant | Params | Layers | Dim | Context | Status |
|---|---|---|---|---|---|
| **Micro** | 10M | 8 | 256 | 16K | ✅ Released |
| Base | ~100M | 12 | 512 | 32K | In development |
| Large | ~400M | 24 | 768 | 64K | Planned |

## Limitations

- Pre-trained on prokaryotic genomes only — may underperform on eukaryotic tasks
- 16K context may truncate long contigs; longer variants planned
- Current model trained for ~1.2B tokens; extended training may improve performance
- Not validated on community-standard benchmarks (GenomicBenchmarks, GUE) yet

## Citation

```bibtex
@misc{seqlens-v2-2026,
  title={SeqLens v2: Compact Genomic Language Models for Microbial Bioinformatics},
  author={SeqSight Team},
  year={2026},
  url={https://huggingface.co/seqSight/seqlens-v2-micro-16k}
}
```

## License

Apache 2.0