kennethge123/qwen3-t5-inversion

Qwen3 T5 Inversion Model

Vec2Text embedding inversion model using T5-base (multilingual) as the decoder. Converts Qwen3-Embedding-8B (4096-dim) embeddings back to text.

Model Details

• Base model: t5-base
• Embedding model: Qwen3-Embedding-8B (4096-dim)
• Training data: 1M sentences from ClimbMix corpus
• Architecture: Vec2Text embedding inversion (embedding -> text)

Evaluation Results (200 held-out samples)

Metric	Score
Token F1	0.6389 (+/-0.1804)
BLEU-4	0.2952 (+/-0.2632)
ROUGE-L	0.5409 (+/-0.2177)
Exact Match	8/200 (4.0%)

Example Reconstructions

Original	Reconstructed	Token F1
"This is important because childhood sets the stage for the robustness of the im...	"This is important because the immune system sets the stage for robust immunity ...	0.865
Chapter 8: Made in the USA - A Statement About Quality and Pride

Have you ever ... | Chapter 8: Making a Statement in the USA: Made in the USA Question: Have you eve... | 0.667 | | Less than 30% of the 36 million cat owners in the U.S. know that the beautiful s... | Fewer than 10 percent of U.S. cat owners and feline lovers know that lilac-lovin... | 0.578 | | The key ingredients of Turkish meals are meat, vegetables, and legumes.... | The meat components of Turkish meals include vegetables, meat, and herbs.... | 0.690 | | The platform will then generate a list of available options with prices, travel ... | The platform will provide options such as price, availability, time of travel, a... | 0.621 |

Usage

import torch, torch.nn as nn, transformers
from safetensors.torch import load_file

# Load model
model = transformers.AutoModelForSeq2SeqLM.from_pretrained("t5-base")
tokenizer = transformers.AutoTokenizer.from_pretrained("t5-base")
hidden = model.config.hidden_size

embedding_transform = nn.Sequential(
    nn.Linear(4096, 4096), nn.LayerNorm(4096), nn.Dropout(0.1), nn.GELU(),
    nn.Linear(4096, hidden * 16),
)

# Load weights (download from this repo)
state = load_file("model.safetensors")  # or torch.load("bart_noisy.pt")
et_state = {k.replace("embedding_transform.", ""): v for k, v in state.items() if k.startswith("embedding_transform.")}
embedding_transform.load_state_dict(et_state)
ed_state = {k.replace("encoder_decoder.", ""): v for k, v in state.items() if k.startswith("encoder_decoder.")}
model.load_state_dict(ed_state, strict=False)

# Invert a Qwen3-Embedding-8B embedding (4096-dim)
device = torch.device("cuda")
model, embedding_transform = model.to(device).eval(), embedding_transform.to(device).eval()

with torch.no_grad():
    emb = torch.tensor(your_embedding, dtype=torch.float32).unsqueeze(0).to(device)
    proj = embedding_transform(emb).reshape(1, 16, hidden)
    out = model.generate(inputs_embeds=proj, attention_mask=torch.ones(1, 16, device=device),
                         max_length=128, num_beams=4, early_stopping=True)
    text = tokenizer.decode(out[0], skip_special_tokens=True)