Create README.md

README.md

@@ -0,0 +1,580 @@
+下面是完整 `README.md` Markdown 源码格式，可以直接复制保存为 `README.md` 上传 HuggingFace。
+
+```markdown
+---
+language:
+- en
+
+tags:
+- nanopore
+- dna
+- sequencing
+- genomics
+- bioinformatics
+- foundation-model
+- transformer
+- bert
+- masked-language-modeling
+- signal-processing
+- vector-quantization
+- vq-tokenizer
+
+pipeline_tag: feature-extraction
+
+library_name: transformers
+
+base_model:
+- none
+
+datasets:
+- ShuaiAnwo/PoreDNA_S1_HG002_MOD_250F701901011_A50
+
+new_version:
+- v526
+
+metrics:
+- loss
+
+license:
+- openrail
+---
+
+# PoreBERT-DNA-VQI-256-526
+
+A compact BERT-style foundation model for nanopore DNA sequencing signal representation learning.
+
+PoreBERT-DNA-VQI-256-526 learns contextual representations from discretized nanopore electrical signal tokens generated by a Vector Quantization (VQ) tokenizer.
+
+The model is pretrained with a Masked Language Modeling (MLM) objective on large-scale nanopore DNA sequencing token sequences.
+
+The model is designed for downstream nanopore sequencing applications including:
+
+- Basecalling
+- Modified base detection
+- Signal representation learning
+- Read-level embedding
+
+
+---
+
+# Model Summary
+
+| Property | Description |
+|---|---|
+| Model Type | BERT Encoder |
+| Domain | Nanopore DNA sequencing |
+| Architecture | Transformer Encoder |
+| Parameters | ~8M |
+| Training Objective | Masked Language Modeling (MLM) |
+| Input | Discrete nanopore signal tokens |
+| Tokenizer | VQ-based neural codec |
+| Vocabulary Size | 2560 |
+| Release Version | 526 |
+
+
+---
+
+# Model Name Explanation
+
+```
+
+PoreBERT-DNA-VQI-256-526
+
+```
+
+## PoreBERT
+
+Nanopore sequencing foundation model based on the BERT encoder architecture.
+
+
+## DNA
+
+The model is trained on nanopore DNA sequencing electrical signal representations.
+
+
+## VQI
+
+Vector Quantization based minimal vocabulary tokenizer.
+
+The raw nanopore electrical signal is converted into discrete token IDs using a neural codec based on vector quantization.
+
+
+## 256
+
+The hidden representation dimension of the Transformer encoder.
+
+```
+
+hidden_size = 256
+
+```
+
+
+## 526
+
+Internal development release identifier.
+
+
+---
+
+# Architecture Overview
+
+The complete representation pipeline:
+
+```
+
+Nanopore Electrical Signal
+
+```
+      |
+      v
+```
+
+PoreCodec
+(CNN Encoder + Vector Quantization)
+
+```
+      |
+      v
+```
+
+Discrete Signal Tokens
+
+```
+      |
+      v
+```
+
+PoreBERT-DNA-VQI-256-526
+
+```
+      |
+      v
+```
+
+Contextual Token Embeddings
+
+```
+      |
+      v
+```
+
+Downstream Applications
+
+```
+
+
+The PoreBERT model itself does not directly process raw electrical signals.
+
+The input of PoreBERT is the discrete token sequence generated by the PoreCodec tokenizer.
+
+
+---
+
+# Tokenizer
+
+This model uses the following tokenizer:
+
+```
+
+ShuaiAnwo/pore-codec-rsq742c12a-511
+
+```
+
+
+The tokenizer converts nanopore electrical signals into discrete token IDs through a VQ-based neural codec.
+
+
+Workflow:
+
+```
+
+Raw signal
+
+```
+|
+v
+```
+
+PoreCodec VQ tokenizer
+
+```
+|
+v
+```
+
+Discrete Token IDs
+
+```
+|
+v
+```
+
+PoreBERT Encoder
+
+```
+|
+v
+```
+
+Contextual Embeddings
+
+```
+
+
+---
+
+# Model Architecture Details
+
+Configuration:
+
+| Parameter | Value |
+|---|---:|
+| Architecture | Transformer Encoder |
+| Hidden Size | 256 |
+| Transformer Layers | 8 |
+| Attention Heads | 8 |
+| Attention Head Dimension | 32 |
+| Intermediate Size | 1024 |
+| Maximum Sequence Length | 1536 |
+| Vocabulary Size | 2560 |
+| Parameters | ~8M |
+
+
+Architecture:
+
+```
+
+Token Embedding
+|
+Position Embedding
+|
+LayerNorm + Dropout
+|
+8 Transformer Encoder Layers
+|
+LayerNorm
+|
+Contextual Token Representation
+
+```
+
+
+---
+
+# Pretraining Objective
+
+The model is pretrained using Masked Language Modeling (MLM).
+
+
+During training:
+
+1. Nanopore electrical signals are converted into discrete tokens.
+2. Random tokens are masked.
+3. The Transformer predicts the original tokens using bidirectional context.
+
+
+Example:
+
+```
+
+Input:
+
+A B [MASK] D E
+
+Prediction:
+
+C
+
+```
+
+
+Training configuration:
+
+| Parameter | Value |
+|---|---:|
+| MLM Probability | 0.15 |
+| Optimizer | AdamW |
+| Learning Rate | 8e-4 |
+| Weight Decay | 0.01 |
+| Adam beta1 | 0.9 |
+| Adam beta2 | 0.98 |
+| Precision | BF16 |
+| Sequence Length | 1536 |
+
+
+---
+
+# Learning Rate Schedule
+
+The model uses:
+
+```
+
+cosine_with_restarts
+
+```
+
+
+Configuration:
+
+| Parameter | Value |
+|---|---:|
+| Scheduler | cosine_with_restarts |
+| Number of Cycles | 2 |
+| Warmup Steps | 2000 |
+
+
+The learning rate schedule consists of a warmup phase followed by cosine decay with restart cycles.
+
+
+---
+
+# Usage
+
+PoreBERT-DNA-VQI-256-526 operates on discrete signal tokens generated by the corresponding PoreCodec VQ tokenizer.
+
+
+Pipeline:
+
+```
+
+Nanopore Raw Signal
+
+```
+    |
+    v
+```
+
+PoreCodec VQ Tokenizer
+
+```
+    |
+    v
+```
+
+Discrete Token IDs
+
+```
+    |
+    v
+```
+
+PoreBERT Encoder
+
+```
+    |
+    v
+```
+
+Contextual Signal Embeddings
+
+````
+
+
+## Quick Start
+
+```python
+import numpy as np
+import torch
+
+from transformers import AutoFeatureExtractor, AutoModel
+
+
+codec_name = "ShuaiAnwo/pore-codec-rsq742c12a-511"
+bert_name = "ShuaiAnwo/PoreBERT-DNA-VQI-256-526"
+
+
+codec = AutoModel.from_pretrained(
+    codec_name,
+    trust_remote_code=True,
+).eval()
+
+
+feature_extractor = AutoFeatureExtractor.from_pretrained(
+    codec_name,
+    trust_remote_code=True,
+)
+
+
+bert = AutoModel.from_pretrained(
+    bert_name,
+).eval()
+
+
+
+raw_signal = np.random.normal(
+    70,
+    8,
+    1855,
+).astype(np.float32)
+
+
+
+with torch.no_grad():
+
+    # Raw signal -> VQ token IDs
+
+    signal = feature_extractor(
+        raw_signal,
+        return_tensors="pt",
+    )["signal"]
+
+
+    token_ids = codec.encode_signal(
+        signal,
+        layer=2,
+    )
+
+
+    # Token IDs -> contextual embeddings
+
+    outputs = bert(
+        input_ids=token_ids,
+    )
+
+
+    embeddings = outputs.last_hidden_state
+
+
+
+print(
+    "Embedding shape:",
+    embeddings.shape
+)
+````
+
+Output:
+
+```
+Embedding shape:
+
+(batch_size, sequence_length, 256)
+```
+
+The generated embeddings can be used for downstream nanopore sequencing tasks:
+
+* Basecalling
+* Modified base detection
+* Signal representation learning
+* Read-level embedding
+* Sequence classification
+
+---
+
+# Training Data
+
+The model was pretrained on nanopore DNA sequencing token sequences.
+
+Dataset:
+
+```
+ShuaiAnwo/PoreDNA_S1_HG002_MOD_250F701901011_A50
+```
+
+Training pipeline:
+
+```
+Nanopore Electrical Signal
+
+        |
+        v
+
+PoreCodec VQ Tokenizer
+
+        |
+        v
+
+Discrete Token Sequence
+
+        |
+        v
+
+Masked Language Modeling
+
+        |
+        v
+
+PoreBERT Encoder
+```
+
+---
+
+# Training Configuration
+
+Training configuration from the original experiment:
+
+```yaml
+model:
+  bert_model_type: electra
+  vocab_size: 2560
+  hidden_size: 256
+  num_hidden_layers: 8
+  num_attention_heads: 8
+  intermediate_size: 1024
+  max_position_embeddings: 1536
+
+
+mlm_config:
+  mlm_probability: 0.15
+
+
+optimizer:
+  name: adamw
+  learning_rate: 8.0e-4
+  weight_decay: 0.01
+  beta1: 0.9
+  beta2: 0.98
+
+
+scheduler:
+  name: cosine_with_restarts
+  num_cycles: 2
+  t_warmup: 2000
+
+
+precision:
+  amp_bf16
+```
+
+---
+
+# Gradient Checkpointing
+
+The training framework supports gradient checkpointing.
+
+Gradient checkpointing reduces GPU memory usage by recomputing intermediate activations during backward propagation.
+
+Benefits:
+
+* Lower GPU memory consumption
+* Longer sequence training
+* Larger models on limited hardware
+
+Trade-off:
+
+* Increased training computation time
+
+---
+
+# Limitations
+
+* The model does not directly accept raw nanopore electrical signals.
+* Raw signals must first be converted into VQ token IDs.
+* Performance depends on tokenizer quality and training data distribution.
+* The model is optimized for nanopore DNA sequencing representation learning.
+
+---
+
+# Citation
+
+Coming soon.
+
+---
+
+# License
+
+Please refer to the LICENSE file for usage conditions.
+
+```
+```