---
library_name: transformers
license: apache-2.0
base_model:
- ai-sage/Giga-Embeddings-instruct
pipeline_tag: feature-extraction
tags:
- embeddings
- quantized
- 4bit
- nf4
- bnb
---

# Giga-Embeddings-instruct (4-bit NF4 Quantized)

This is a **4-bit quantized version** of the original model [`ai-sage/Giga-Embeddings-instruct`](https://huggingface.co/ai-sage/Giga-Embeddings-instruct), created using `bitsandbytes` with the following configuration:

```python
bnb_cfg = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=torch.bfloat16
)
```

## Giga-Embeddings-instruct
- Base Decoder-only LLM: GigaChat-3b
- Pooling Type: Latent-Attention
- Embedding Dimension: 2048


> ⚠️ **Note**: This model is **not fine-tuned** — it is the original model loaded in 4-bit precision using `transformers` + `bitsandbytes`. It requires `bitsandbytes` and `accelerate` to run.

## Usage

```python
from transformers import AutoModel, AutoTokenizer, BitsAndBytesConfig
import torch

bnb_cfg = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=torch.bfloat16
)

model = AutoModel.from_pretrained(
    "iMiW/Giga-Embeddings-instruct-4bit-nf4", 
    quantization_config=bnb_cfg, 
    trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained(
    "iMiW/Giga-Embeddings-instruct-4bit-nf4", 
    trust_remote_code=True
)

def get_detailed_instruct(task_description: str, query: str) -> str:
    return f'Instruct: {task_description}\nQuery: {query}'

# Each query must come with a one-sentence instruction that describes the task
task = 'Given a web search query, retrieve relevant passages that answer the query'

queries = [
    get_detailed_instruct(task, 'What is the capital of Russia?'),
    get_detailed_instruct(task, 'Explain gravity')
]
# No need to add instruction for retrieval documents
documents = [
    "The capital of Russia is Moscow.",
    "Gravity is a force that attracts two bodies towards each other. It gives weight to physical objects and is responsible for the movement of planets around the sun."
]
input_texts = queries + documents

model.eval()
model.cuda()

max_length = 4096

# Tokenize the input texts
batch_dict = tokenizer(
    input_texts,
    padding=True,
    truncation=True,
    max_length=max_length,
    return_tensors="pt",
)
batch_dict.to(model.device)
embeddings = model(**batch_dict, return_embeddings=True)

scores = (embeddings[:2] @ embeddings[2:].T)
print(scores.tolist())
```