# Q-TensorFormer v3
**Quantum-Enhanced Tensor Network LLM Compression Engine**
[](LICENSE)
[](https://www.python.org/downloads/)
[](https://pytorch.org/)
[](https://pennylane.ai/)
[]()
[](https://huggingface.co/Premchan369/q-tensorformer)
> **"A hybrid quantumโtensor model that adaptively compresses itself using entanglement, achieving major efficiency gains with minimal performance loss."**
---
## What is Q-TensorFormer?
Q-TensorFormer replaces the dense feed-forward layers (FFN) of a Transformer with **Tensor-Train (TT) decomposition** โ reducing parameters by 50-70%. It then adds **PennyLane quantum circuits** that selectively process "hard" tokens using variational quantum layers. Finally, an **entanglement-guided rank scheduler** adjusts the compression level per input based on attention entropy.
### The 3 Pillars
| Pillar | What It Does | Impact |
|--------|-------------|--------|
| ๐งฎ **Tensor Compression** | Replaces dense FFN with TT cores | 1.5โ3ร parameter reduction |
| โ๏ธ **Quantum Feature Layer** | PennyLane circuit processes selected tokens | Richer token representations |
| ๐ง **Entropy โ Rank Scheduler** | Attention entropy adapts TT ranks dynamically | Input-aware compute efficiency |
### Core Formula
```
r(input) = r_min + ฮฑ ร S_norm(attention) ร (r_max - r_min)
where S_norm = entropy / log(seq_len) โ [0, 1]
```
---
## ๐ Quick Start
### Installation
```bash
git clone https://huggingface.co/Premchan369/q-tensorformer
cd q-tensorformer
pip install -e .
```
Or via pip:
```bash
pip install torch pennylane datasets
git clone https://huggingface.co/Premchan369/q-tensorformer
pip install -e ./q-tensorformer
```
### 30-Second Example
```python
import torch
from src.config import ModelConfig
from src.models import create_model
# Create a tiny Q-TensorFormer
config = ModelConfig(
d_model=64, n_heads=4, n_layers=2, tt_rank=4,
vocab_size=10000, use_quantum=True, n_qubits=4,
)
model = create_model(config, "qtensor")
print(f"Params: {model.total_params:,}")
print(f"Compression ratio: {model.compression_ratio:.1f}x")
# Forward pass
x = torch.randint(0, 10000, (4, 64)) # batch=4, seq=64
logits, stats = model(x, return_stats=True)
for i, s in enumerate(stats):
print(f"Layer {i}: rank={s['rank']}, "
f"entropy={s.get('entropy', 0):.2f}")
```
### Train on WikiText-2
```bash
# Benchmark all models (Q-TensorFormer vs. baselines)
python scripts/benchmark.py --preset small --epochs 5
# Hyperparameter sweep
python scripts/sweep.py --epochs 5
# Knowledge distillation
python scripts/distill.py --teacher_config small --student_rank 4
# Or directly from Python
python -c "
from src.config import ModelConfig, TrainingConfig, ExperimentConfig
from src.models import create_model
from src.data import load_wikitext2
from src.training import Trainer
config = ExperimentConfig(
model=ModelConfig(d_model=128, n_layers=2, tt_rank=8),
training=TrainingConfig(max_epochs=5, batch_size=16),
)
train, val, test, tok = load_wikitext2(seq_len=128, batch_size=16)
config.model.vocab_size = tok.vocab_size
model = create_model(config, 'qtensor')
trainer = Trainer(model, config, train, val, test)
trainer.train()
"
```
---
## ๐ Project Structure
```
q-tensorformer/
โโโ README.md # This file
โโโ LICENSE # Apache 2.0
โโโ CITATION.cff # Citation metadata
โโโ MODEL_CARD.md # Model card
โโโ setup.py # pip install
โโโ requirements.txt # Dependencies
โ
โโโ configs/ # YAML configuration presets
โ โโโ default.yaml # Small-scale config
โ โโโ production.yaml # Full-scale with budget constraints
โ โโโ sweep.yaml # Sweep configuration
โ
โโโ src/ # Core library
โ โโโ __init__.py # Version and metadata
โ โโโ config.py # Dataclass config + presets
โ โโโ tensor_layers.py # TTLinear, TTFeedForward with SVD truncation
โ โโโ quantum_layers.py # PennyLane angle embedding, fallback
โ โโโ scheduler.py # RankScheduler, BudgetAwareScheduler
โ โโโ router.py # QuantumRouter with straight-through gate
โ โโโ attention.py # MultiHeadAttention + HybridQAttention
โ โโโ blocks.py # HybridBlock = Attn + Router + TT-FFN
โ โโโ models.py # QTensorFormer + DenseBaseline
โ โโโ baselines.py # StandardTransformer, Distilled, Pruned
โ โโโ data.py # CharTokenizer, WikiText-2 loader
โ โโโ training.py # Trainer + DistillationTrainer
โ โโโ metrics.py # evaluate_model, Pareto frontier, efficiency score
โ โโโ budget.py # BudgetTracker, EnergyEstimator
โ
โโโ scripts/ # Executable scripts
โ โโโ benchmark.py # Full multi-model benchmark
โ โโโ sweep.py # Hyperparameter grid search
โ โโโ distill.py # Knowledge distillation training
โ
โโโ tests/ # Unit tests
โโโ test_tensor_layers.py # TT decomposition tests
โโโ test_quantum_layers.py # Quantum layer tests
```
---
## ๐๏ธ Architecture
```
Input Tokens
โ
โผ
โโโโโโโโโโโโโโโโโโโโโโโ
โ Embedding + PosEnc โ
โโโโโโโโโโโฌโโโโโโโโโโโโ
โ
โโโโโโโผโโโโโโโ (ร N layers)
โ HybridBlock โ
โ โ
โ LN โ Attention โ Entropy โ RankScheduler โ
โ LN โ QuantumRouter โ TTFeedForward โ
โ Residual connection โ
โโโโโโโฌโโโโโโโ
โ
โโโโโโโผโโโโโโโ
โ LN โ LM Head โ
โโโโโโโฌโโโโโโโ
โ
โผ
Logits (next token prediction)
```
### Data Flow Through One Block
1. **LayerNorm** โ normalize
2. **Multi-Head Attention** โ classical self-attention
3. **Entropy Monitor** โ compute attention entropy S(ฯ) per head
4. **RankScheduler** โ entropy โ TT-rank: `r = r_min + ฮฑ ร S_norm ร (r_max - r_min)`
5. **Apply** `set_rank(r)` โ SVD-based truncation on all TT-FFN cores
6. **LayerNorm** โ normalize residual
7. **QuantumRouter** โ learn which tokens need quantum (straight-through gate)
8. **TTFeedForward** โ up-project (TT) โ GELU โ down-project (TT)
9. **Residual connection** โ combined output
---
## ๐ง Model Variants
| Name | TT Decomp? | Quantum? | Adaptive Rank? | Use Case |
|------|-----------|----------|---------------|----------|
| **QTensorFormer** | โ
| โ
| โ
| Full hybrid (default) |
| **TensorOnly** | โ
| โ | โ
| Pure tensor compression |
| **StandardTransformer** | โ | โ | โ | Dense baseline |
| **Distilled** | โ | โ | โ | Smaller dense via KD |
| **Pruned** | โ | โ | โ | Magnitude-pruned dense |
---
## ๐ Benchmarks
### FFN-Only Compression
The TT decomposition compresses FFN layers by ~7-8ร at rank 8:
| d_model | Dense FFN Params | TT FFN Params (r=8) | Compression |
|---------|-----------------|---------------------|-------------|
| 128 | 131,072 | 18,112 | **7.2ร** |
| 256 | 524,288 | 67,904 | **7.7ร** |
| 512 | 2,097,152 | 265,792 | **7.9ร** |
### Overall Model Compression
| d_model | QTensorFormer | Dense Baseline | Compression |
|---------|--------------|---------------|-------------|
| 128 | 1.6M | 2.1M | **1.3ร** |
| 256 | 4.0M | 5.7M | **1.4ร** |
| 512 | 10.7M | 17.7M | **1.7ร** |
*Note: Overall compression is lower because embeddings (vocab ร d_model) don't get compressed. This is standard for any weight-level compression approach.*
### Verification (22/22 tests pass)
```
tests/test_tensor_layers.py .......... (10/10)
tests/test_quantum_layers.py ........ (8/8)
integration: qtensor, tensor_only, dense all pass โ
```
---
## โ๏ธ Quantum Details
### Circuit Architecture
```
q0: โโRX(input[0])โโRY(ฮธโโ)โโโโโโโโโโโโโโโโโโโโโโโโจZโฉโโ
โ โ
q1: โโRX(input[1])โโRY(ฮธโโ)โโXโโโโโโโโโโโโโโโโโโโโโจZโฉโโ
โ โ
q2: โโRX(input[2])โโRY(ฮธโโ)โโโโโXโโโโโโโโโโโโโโโโโโจZโฉโโ
โ โ
q3: โโRX(input[3])โโRY(ฮธโโ)โโโโโโโโXโโRY(ฮธโโ)โโโโโโจZโฉโโ
```
- **4 qubits** (NISQ-compatible)
- **Angle encoding**: input features โ RX rotations
- **2 variational layers**: RY rotation + CNOT ladder + cyclic entanglement
- **Measurement**: Pauli-Z expectation values โ classical output
- **Differentiation**: Backprop (`diff_method="backprop"`) for batched inputs
### Selective Quantum Routing
Not every token needs quantum. The `QuantumRouter` uses a learned gate:
```python
soft_mask = sigmoid(gate_proj(token) / temperature)
hard_mask = (soft_mask > 0.5) # binary decision
# Straight-through estimator:
# Forward: hard binary (fast, sparse)
# Backward: soft gradient (differentiable)
mask = hard.detach() + soft - soft.detach()
```
**Target sparsity**: 70% (default). Only ~30% of tokens pass through the quantum circuit.
---
## ๐ฏ Use Cases & Recipes
### 1. Edge NLP (Mobile / Low-GPU)
```bash
python scripts/benchmark.py --preset tiny --epochs 3
```
Config: `d_model=64, tt_rank=2, n_qubits=4`. Model < 1M params.
### 2. Enterprise Cost Reduction
```bash
# Knowledge-distilled compression
python scripts/distill.py \
--teacher_config medium \
--student_rank 4 \
--alpha 0.5 --temperature 3.0
```
Train a dense teacher (5M params), distill into a compressed student (1.5M params).
### 3. Research: Comparing Compression Methods
```python
from src.metrics import compare_models, print_comparison_table, compute_pareto_frontier
results = compare_models({
"standard": standard_model,
"pruned_50": pruned_model,
"distilled": distilled_model,
"qtensor_r8": qtensor_rank8,
"qtensor_r4": qtensor_rank4,
}, test_loader)
print_comparison_table(results)
pareto = compute_pareto_frontier(results)
```
### 4. Multilingual Low-Resource
```python
from src.data import CharTokenizer
texts = load_your_language_data()
tokenizer = CharTokenizer()
tokenizer.fit(texts)
config = ModelConfig(vocab_size=tokenizer.vocab_size, d_model=128,
tt_rank=4, n_layers=3)
```
### 5. Budget-Constrained Deployment
```yaml
budget:
max_params: 2000000
max_latency_ms: 50.0
max_energy_per_query: 500.0
target_compression_ratio: 2.0
```
---
## ๐งช Evaluation Metrics
| Metric | What It Measures | Tool |
|--------|-----------------|------|
| **Perplexity (PPL)** | Language modeling quality | `metrics.evaluate_model()` |
| **Total/compressed params** | Memory efficiency | `model.total_params` |
| **Compression ratio** | vs. dense equivalent | `model.compression_ratio` |
| **Latency (p50, p95)** | Inference speed | Benchmarked with warmup |
| **Energy (FLOPs proxy)** | Power consumption | `budget.EnergyEstimator` |
| **Pareto frontier** | Optimal PPL-params tradeoff | `metrics.compute_pareto_frontier()` |
| **Efficiency score** | Combined metric | `metrics.compute_efficiency_score()` |
| **Rank trajectory** | How ranks evolve during training | `metrics.rank_trajectory_analysis()` |
| **Quantum sparsity** | % tokens bypassing quantum | `model.stats['quantum_usage']` |
---
## ๐ฌ Scientific Background
### Tensor-Train Decomposition
Given a weight matrix **W โ R^{I ร O}**, TT decomposition factorizes it into **d cores**:
```
W(iโ,...,i_d, oโ,...,o_d) = โ G_k[i_k, o_k]
```
where **G_k โ R^{r_{k-1} ร i_k ร o_k ร r_k}** and rโ = r_d = 1. The TT-rank **r** controls the compression.
Q-TensorFormer uses **SVD-based rank truncation**: when reducing rank, we merge adjacent cores and keep the top-k singular values at each bond, preserving dominant signal directions (Eckart-Young theorem).
### Quantum-Classical Hybrid
We simulate **NISQ-era quantum circuits** using PennyLane's `default.qubit` backend. Compatible with real quantum hardware by changing the device.
### Entanglement โ Rank Correspondence
The core insight: **attention entropy** is a classical proxy for **quantum entanglement entropy**. When attention is diffuse (uniform over many tokens), the representation is more "complex" โ we allocate higher TT-rank. When attention is concentrated, we compress aggressively.
---
## ๐ Roadmap
### v3.1 (Next)
- [ ] Apply to real pretrained models (GPT-2 small, DistilBERT)
- [ ] Structured pruning baseline comparison
- [ ] GLUE/SuperGLUE classification benchmarks
### v3.2
- [ ] Actual quantum hardware support (Braket, IBM Q)
- [ ] Multi-modal extension (ViT + TT)
- [ ] ONNX export for production deployment
### v4.0
- [ ] Post-training quantization (int8 TT cores)
- [ ] Speculative decoding with adaptive TT-rank
- [ ] Online learning with adaptive compression
---
## ๐ค Contributing
1. Fork the repo
2. Create a feature branch
3. Make changes + add tests
4. Run `pytest tests/` to verify
5. Submit a PR
---
## ๐ References
- **Tensor Networks**: Cichocki et al., "Tensor Networks for Dimensionality Reduction and Large-scale Optimization" (arXiv:2007.02779)
- **Tensor-Train**: Oseledets, "Tensor-Train Decomposition" (SIAM J. Sci. Comp., 2011)
- **Quixer**: "Quantum Transformer for Language Modeling" (arXiv:2406.04305)
- **QKSAN**: "Quantum Kernel Self-Attention Network" (arXiv:2308.13422, IEEE TPAMI 2024)
- **PennyLane**: Bergholm et al., "Automatic differentiation of hybrid quantum-classical computations" (arXiv:1811.04968)
- **Knowledge Distillation**: Hinton et al., "Distilling the Knowledge in a Neural Network" (arXiv:1503.02531)
---
## ๐ License
Apache 2.0 โ see [LICENSE](LICENSE).
## ๐ Acknowledgments
Built with:
- [PyTorch](https://pytorch.org/) โ Deep learning framework
- [PennyLane](https://pennylane.ai/) โ Quantum computing library
- [HuggingFace Datasets](https://huggingface.co/docs/datasets) โ WikiText-2 loading
---
**Q-TensorFormer v3** ยท Made with โ๏ธ + ๐งฎ
[๐ค Model on Hub](https://huggingface.co/Premchan369/q-tensorformer)