---
language: en
tags:
- transformer
- moe
- mixture-of-experts
- hyper-connections
- research
- experimental-model
license: mit
base_model: persadian/persadian-Nano-V4
---

# persadian-Nano-V4

**MoE architecture optimized for T4-class GPUs**

## Model Details

- **Architecture:** Mixture of Experts (8 experts) + Adaptive Hyper-Connections + Compressed Sparse Attention
- **Parameters:** ~160M
- **Context Length:** 8,192 tokens
- **Target Hardware:** T4 / consumer-class GPUs
- **Inference Focus:** Lightweight active-path computation for research environments

## Three Novel Innovations

1. **Adaptive Hyper-Connections** - Input-dependent routing weights (not fixed Sinkhorn)
2. **Progressive Expert Activation** - Starts with 1 expert, grows to 2 during inference
3. **Online Compressed KV Cache** - Adaptive compression based on sequence length

| Feature                | Real V4-Pro                     | Persadian-Nano-V4                             |
| ---------------------- | ------------------------------- | --------------------------------------------- |
| Hyper-Connections      | Fixed Sinkhorn routing          | ✅Adaptive input-dependent routing            |
| Expert Activation      | Static top-k experts            | ✅Progressive expert scaling during inference |
| KV Cache               | Static KV compression           | ✅Online adaptive KV compression              |
| Attention Design       | Standard Hybrid Attention       | ✅Compressed Sparse Hybrid Attention          |
| MoE Routing            | Fixed expert utilization        | ✅Dynamic progressive routing                 |
| Context Optimization   | Enterprise-scale infrastructure | ✅Colab-optimized memory efficiency           |
| Hardware Requirement   | Designed for hyperscale clusters | ✅Optimized for single-GPU research environments              |
| Parameter Count        | ~1.6T parameters                | ✅~160M parameters                            |
| Active Compute         | Extremely high                  | ✅Lightweight active-path compute             |
| Deployment Target      | Hyperscale datacenters          | ✅Prosumer laptops + edge GPUs                |
| Training Accessibility | Large AI labs only              | ✅Independent researchers & startups          |
| Training Cost          | Estimated $10M+                 | ✅Near-zero using T4 GPU             |
| Research Direction     | Closed frontier model           | ✅Experimental open nano-architecture         |
| Inference Efficiency   | Infrastructure-heavy            | ✅Optimized for constrained hardware          |
| Innovation Focus       | Scale-first                     | ✅Efficiency-first with adaptive systems      |

## Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

# Load the custom model with its tokenizer
model = AutoModelForCausalLM.from_pretrained(
    "persadian/persadian-nano-v4",
    trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained("persadian/persadian-nano-v4")

# Generate text
prompt = "Once upon a time"
inputs = tokenizer(prompt, return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=50)
print(tokenizer.decode(output[0], skip_special_tokens=True))
```

## Citation
```bibtex
@misc{persadian2026nano,
  author = {Persadh, Darshani},
  title = {persadian-Nano-V4},
  year = {2026},
  publisher = {Hugging Face},
  url = {https://huggingface.co/persadian/persadian-Nano-V4}
}
```