ShuaiAnwo/PoreBERT-DNA-VQI-007M-526

PoreBERT-DNA-VQI-256-526

A compact BERT-style foundation model for nanopore DNA sequencing signal representation learning.

PoreBERT-DNA-VQI-256-526 learns contextual representations from discretized nanopore electrical signal tokens generated by a Vector Quantization (VQ) tokenizer.

The model is pretrained with a Masked Language Modeling (MLM) objective on large-scale nanopore DNA sequencing token sequences.

The model is designed for downstream nanopore sequencing applications including:

• Basecalling
• Modified base detection
• Signal representation learning
• Read-level embedding

---

Model Summary

Property	Description
Model Type	BERT Encoder
Domain	Nanopore DNA sequencing
Architecture	Transformer Encoder
Parameters	~8M
Training Objective	Masked Language Modeling (MLM)
Input	Discrete nanopore signal tokens
Tokenizer	VQ-based neural codec
Vocabulary Size	2560
Release Version	526

---

Model Name Explanation

PoreBERT-DNA-VQI-256-526

PoreBERT

Nanopore sequencing foundation model based on the BERT encoder architecture.

DNA

The model is trained on nanopore DNA sequencing electrical signal representations.

VQI

Vector Quantization based minimal vocabulary tokenizer.

The raw nanopore electrical signal is converted into discrete token IDs using a neural codec based on vector quantization.

256

The hidden representation dimension of the Transformer encoder.

hidden_size = 256

526

Internal development release identifier.

---

Architecture Overview

The complete representation pipeline:

Nanopore Electrical Signal

  |
  v

PoreCodec
(CNN Encoder + Vector Quantization)

  |
  v

Discrete Signal Tokens

  |
  v

PoreBERT-DNA-VQI-256-526

  |
  v

Contextual Token Embeddings

  |
  v

Downstream Applications

The PoreBERT model itself does not directly process raw electrical signals.

The input of PoreBERT is the discrete token sequence generated by the PoreCodec tokenizer.

---

Tokenizer

This model uses the following tokenizer:

ShuaiAnwo/pore-codec-rsq742c12a-511

The tokenizer converts nanopore electrical signals into discrete token IDs through a VQ-based neural codec.

Workflow:

Raw signal

| v

PoreCodec VQ tokenizer

| v

Discrete Token IDs

| v

PoreBERT Encoder

| v

Contextual Embeddings

---

Model Architecture Details

Configuration:

Parameter	Value
Architecture	Transformer Encoder
Hidden Size	256
Transformer Layers	8
Attention Heads	8
Attention Head Dimension	32
Intermediate Size	1024
Maximum Sequence Length	1536
Vocabulary Size	2560
Parameters	~8M

Architecture:

Token Embedding
|
Position Embedding
|
LayerNorm + Dropout
|
8 Transformer Encoder Layers
|
LayerNorm
|
Contextual Token Representation

---

Pretraining Objective

The model is pretrained using Masked Language Modeling (MLM).

During training:

1. Nanopore electrical signals are converted into discrete tokens.
2. Random tokens are masked.
3. The Transformer predicts the original tokens using bidirectional context.

Example:

Input:

A B [MASK] D E

Prediction:

C

Training configuration:

Parameter	Value
MLM Probability	0.15
Optimizer	AdamW
Learning Rate	8e-4
Weight Decay	0.01
Adam beta1	0.9
Adam beta2	0.98
Precision	BF16
Sequence Length	1536

---

Learning Rate Schedule

The model uses:

cosine_with_restarts

Configuration:

Parameter	Value
Scheduler	cosine_with_restarts
Number of Cycles	2
Warmup Steps	2000

The learning rate schedule consists of a warmup phase followed by cosine decay with restart cycles.

---

Usage

PoreBERT-DNA-VQI-256-526 operates on discrete signal tokens generated by the corresponding PoreCodec VQ tokenizer.

Pipeline:

Nanopore Raw Signal

|
v

PoreCodec VQ Tokenizer

|
v

Discrete Token IDs

|
v

PoreBERT Encoder

|
v

Contextual Signal Embeddings

Quick Start

import numpy as np
import torch

from transformers import AutoFeatureExtractor, AutoModel


codec_name = "ShuaiAnwo/pore-codec-rsq742c12a-511"
bert_name = "ShuaiAnwo/PoreBERT-DNA-VQI-256-526"


codec = AutoModel.from_pretrained(
    codec_name,
    trust_remote_code=True,
).eval()


feature_extractor = AutoFeatureExtractor.from_pretrained(
    codec_name,
    trust_remote_code=True,
)


bert = AutoModel.from_pretrained(
    bert_name,
).eval()



raw_signal = np.random.normal(
    70,
    8,
    1855,
).astype(np.float32)



with torch.no_grad():

    # Raw signal -> VQ token IDs

    signal = feature_extractor(
        raw_signal,
        return_tensors="pt",
    )["signal"]


    token_ids = codec.encode_signal(
        signal,
        layer=2,
    )


    # Token IDs -> contextual embeddings

    outputs = bert(
        input_ids=token_ids,
    )


    embeddings = outputs.last_hidden_state



print(
    "Embedding shape:",
    embeddings.shape
)

Output:

Embedding shape:

(batch_size, sequence_length, 256)

The generated embeddings can be used for downstream nanopore sequencing tasks:

• Basecalling
• Modified base detection
• Signal representation learning
• Read-level embedding
• Sequence classification

---

Training Data

The model was pretrained on nanopore DNA sequencing token sequences.

Dataset:

ShuaiAnwo/PoreDNA_S1_HG002_MOD_250F701901011_A50

Training pipeline:

Nanopore Electrical Signal

        |
        v

PoreCodec VQ Tokenizer

        |
        v

Discrete Token Sequence

        |
        v

Masked Language Modeling

        |
        v

PoreBERT Encoder

---

Training Configuration

Training configuration from the original experiment:

model:
  bert_model_type: electra
  vocab_size: 2560
  hidden_size: 256
  num_hidden_layers: 8
  num_attention_heads: 8
  intermediate_size: 1024
  max_position_embeddings: 1536


mlm_config:
  mlm_probability: 0.15


optimizer:
  name: adamw
  learning_rate: 8.0e-4
  weight_decay: 0.01
  beta1: 0.9
  beta2: 0.98


scheduler:
  name: cosine_with_restarts
  num_cycles: 2
  t_warmup: 2000


precision:
  amp_bf16

---

Gradient Checkpointing

The training framework supports gradient checkpointing.

Gradient checkpointing reduces GPU memory usage by recomputing intermediate activations during backward propagation.

Benefits:

• Lower GPU memory consumption
• Longer sequence training
• Larger models on limited hardware

Trade-off:

• Increased training computation time

---

Limitations

• The model does not directly accept raw nanopore electrical signals.
• Raw signals must first be converted into VQ token IDs.
• Performance depends on tokenizer quality and training data distribution.
• The model is optimized for nanopore DNA sequencing representation learning.

---

Citation

Coming soon.

---

License

Please refer to the LICENSE file for usage conditions.