File size: 7,633 Bytes


---
language:
- en

tags:
- nanopore
- dna
- sequencing
- genomics
- bioinformatics
- foundation-model
- transformer
- bert
- masked-language-modeling
- signal-processing
- vector-quantization
- vq-tokenizer

pipeline_tag: feature-extraction

library_name: transformers

base_model:
- none

datasets:
- ShuaiAnwo/PoreDNA_S1_HG002_MOD_250F701901011_A50

metrics:
- loss

license:
- openrail
---

# PoreBERT-DNA-VQI-256-526

A compact BERT-style foundation model for nanopore DNA sequencing signal representation learning.

PoreBERT-DNA-VQI-256-526 learns contextual representations from discretized nanopore electrical signal tokens generated by a Vector Quantization (VQ) tokenizer.

The model is pretrained with a Masked Language Modeling (MLM) objective on large-scale nanopore DNA sequencing token sequences.

The model is designed for downstream nanopore sequencing applications including:

- Basecalling
- Modified base detection
- Signal representation learning
- Read-level embedding


---

# Model Summary

| Property | Description |
|---|---|
| Model Type | BERT Encoder |
| Domain | Nanopore DNA sequencing |
| Architecture | Transformer Encoder |
| Parameters | ~8M |
| Training Objective | Masked Language Modeling (MLM) |
| Input | Discrete nanopore signal tokens |
| Tokenizer | VQ-based neural codec |
| Vocabulary Size | 2560 |
| Release Version | 526 |


---

# Model Name Explanation

```PoreBERT-DNA-VQI-256-526```

## PoreBERT

Nanopore sequencing foundation model based on the BERT encoder architecture.


## DNA

The model is trained on nanopore DNA sequencing electrical signal representations.


## VQI

Vector Quantization based minimal vocabulary tokenizer.

The raw nanopore electrical signal is converted into discrete token IDs using a neural codec based on vector quantization.


## 256

The hidden representation dimension of the Transformer encoder.

```

hidden_size = 256

```


## 526

Internal development release identifier.


---

# Architecture Overview

The complete representation pipeline:

```

Nanopore Electrical Signal

```
      |
      v
```

PoreCodec
(CNN Encoder + Vector Quantization)

```
      |
      v
```

Discrete Signal Tokens

```
      |
      v
```

PoreBERT-DNA-VQI-256-526

```
      |
      v
```

Contextual Token Embeddings

```
      |
      v
```

Downstream Applications

```


The PoreBERT model itself does not directly process raw electrical signals.

The input of PoreBERT is the discrete token sequence generated by the PoreCodec tokenizer.


---

# Tokenizer

This model uses the following tokenizer:

```

ShuaiAnwo/pore-codec-rsq742c12a-511

```


The tokenizer converts nanopore electrical signals into discrete token IDs through a VQ-based neural codec.


Workflow:

```

Raw signal

```
|
v
```

PoreCodec VQ tokenizer

```
|
v
```

Discrete Token IDs

```
|
v
```

PoreBERT Encoder

```
|
v
```

Contextual Embeddings

```


---

# Model Architecture Details

Configuration:

| Parameter | Value |
|---|---:|
| Architecture | Transformer Encoder |
| Hidden Size | 256 |
| Transformer Layers | 8 |
| Attention Heads | 8 |
| Attention Head Dimension | 32 |
| Intermediate Size | 1024 |
| Maximum Sequence Length | 1536 |
| Vocabulary Size | 2560 |
| Parameters | ~8M |


Architecture:

```

Token Embedding
|
Position Embedding
|
LayerNorm + Dropout
|
8 Transformer Encoder Layers
|
LayerNorm
|
Contextual Token Representation

```


---

# Pretraining Objective

The model is pretrained using Masked Language Modeling (MLM).


During training:

1. Nanopore electrical signals are converted into discrete tokens.
2. Random tokens are masked.
3. The Transformer predicts the original tokens using bidirectional context.


Example:

```

Input:

A B [MASK] D E

Prediction:

C

```


Training configuration:

| Parameter | Value |
|---|---:|
| MLM Probability | 0.15 |
| Optimizer | AdamW |
| Learning Rate | 8e-4 |
| Weight Decay | 0.01 |
| Adam beta1 | 0.9 |
| Adam beta2 | 0.98 |
| Precision | BF16 |
| Sequence Length | 1536 |


---

# Learning Rate Schedule

The model uses:

```

cosine_with_restarts

```


Configuration:

| Parameter | Value |
|---|---:|
| Scheduler | cosine_with_restarts |
| Number of Cycles | 2 |
| Warmup Steps | 2000 |


The learning rate schedule consists of a warmup phase followed by cosine decay with restart cycles.


---

# Usage

PoreBERT-DNA-VQI-256-526 operates on discrete signal tokens generated by the corresponding PoreCodec VQ tokenizer.


Pipeline:

```

Nanopore Raw Signal

```
    |
    v
```

PoreCodec VQ Tokenizer

```
    |
    v
```

Discrete Token IDs

```
    |
    v
```

PoreBERT Encoder

```
    |
    v
```

Contextual Signal Embeddings

````


## Quick Start

```python
import numpy as np
import torch

from transformers import AutoFeatureExtractor, AutoModel


codec_name = "ShuaiAnwo/pore-codec-rsq742c12a-511"
bert_name = "ShuaiAnwo/PoreBERT-DNA-VQI-256-526"


codec = AutoModel.from_pretrained(
    codec_name,
    trust_remote_code=True,
).eval()


feature_extractor = AutoFeatureExtractor.from_pretrained(
    codec_name,
    trust_remote_code=True,
)


bert = AutoModel.from_pretrained(
    bert_name,
).eval()



raw_signal = np.random.normal(
    70,
    8,
    1855,
).astype(np.float32)



with torch.no_grad():

    # Raw signal -> VQ token IDs

    signal = feature_extractor(
        raw_signal,
        return_tensors="pt",
    )["signal"]


    token_ids = codec.encode_signal(
        signal,
        layer=2,
    )


    # Token IDs -> contextual embeddings

    outputs = bert(
        input_ids=token_ids,
    )


    embeddings = outputs.last_hidden_state



print(
    "Embedding shape:",
    embeddings.shape
)
````

Output:

```
Embedding shape:

(batch_size, sequence_length, 256)
```

The generated embeddings can be used for downstream nanopore sequencing tasks:

* Basecalling
* Modified base detection
* Signal representation learning
* Read-level embedding
* Sequence classification

---

# Training Data

The model was pretrained on nanopore DNA sequencing token sequences.

Dataset:

```
ShuaiAnwo/PoreDNA_S1_HG002_MOD_250F701901011_A50
```

Training pipeline:

```
Nanopore Electrical Signal

        |
        v

PoreCodec VQ Tokenizer

        |
        v

Discrete Token Sequence

        |
        v

Masked Language Modeling

        |
        v

PoreBERT Encoder
```

---

# Training Configuration

Training configuration from the original experiment:

```yaml
model:
  bert_model_type: electra
  vocab_size: 2560
  hidden_size: 256
  num_hidden_layers: 8
  num_attention_heads: 8
  intermediate_size: 1024
  max_position_embeddings: 1536


mlm_config:
  mlm_probability: 0.15


optimizer:
  name: adamw
  learning_rate: 8.0e-4
  weight_decay: 0.01
  beta1: 0.9
  beta2: 0.98


scheduler:
  name: cosine_with_restarts
  num_cycles: 2
  t_warmup: 2000


precision:
  amp_bf16
```

---

# Gradient Checkpointing

The training framework supports gradient checkpointing.

Gradient checkpointing reduces GPU memory usage by recomputing intermediate activations during backward propagation.

Benefits:

* Lower GPU memory consumption
* Longer sequence training
* Larger models on limited hardware

Trade-off:

* Increased training computation time

---

# Limitations

* The model does not directly accept raw nanopore electrical signals.
* Raw signals must first be converted into VQ token IDs.
* Performance depends on tokenizer quality and training data distribution.
* The model is optimized for nanopore DNA sequencing representation learning.

---

# Citation

Coming soon.

---

# License

Please refer to the LICENSE file for usage conditions.

```
```