---
language:
- ko
library_name: transformers
tags:
- sparse-retrieval
- splade
- opensearch
- neural-sparse
- information-retrieval
license: apache-2.0
datasets:
- ko-strategyqa
- miracl-ko
- mrtydi-ko
pipeline_tag: fill-mask
base_model: klue/roberta-large
---

# Korean Neural Sparse Encoder (klue/roberta-large)

A Korean sparse retrieval model trained with SPLADE-max on `klue/roberta-large` (337M params).

This model generates sparse lexical representations for Korean text, enabling efficient keyword-expanded retrieval via OpenSearch neural_sparse queries.

## Model Details

| Property | Value |
|----------|-------|
| Base Model | `klue/roberta-large` |
| Parameters | 337M (24 layers, 1024 hidden, 32K vocab) |
| Training Method | SPLADE-max (log(1+ReLU) + max pooling) |
| Loss | InfoNCE + FLOPS regularization |
| Training Data | 4.84M Korean triplets (46 shards) |
| Epochs | 25 |
| Effective Batch Size | 2048 (128/GPU x 2 grad_accum x 8 GPUs) |
| Hardware | 8x NVIDIA B200 (183GB VRAM) |
| Training Time | 13.4 hours |

## Benchmark Results (Recall@1)

| Benchmark | Queries | BM25 | V33-base (149M) | This Model (337M) | Delta vs V33 |
|-----------|---------|------|-----------------|-------------------|-------------|
| Ko-StrategyQA | 592 | 53.7% | 62.2% | **64.2%** | +2.0pp |
| MIRACL-ko | 213 | 44.1% | 61.0% | **70.4%** | +9.4pp |
| Mr.TyDi-ko | 421 | 55.6% | 72.7% | **76.2%** | +3.6pp |

### Full Metrics

| Benchmark | R@1 | R@5 | R@10 | MRR@10 | nz_q | nz_d |
|-----------|-----|-----|------|--------|------|------|
| Ko-StrategyQA | 64.2% | 82.4% | 85.3% | 72.1% | 37 | 71 |
| MIRACL-ko | 70.4% | 89.7% | 96.2% | 79.1% | 35 | 77 |
| Mr.TyDi-ko | 76.2% | 92.6% | 96.0% | 83.4% | 35 | 75 |

## Usage

### With Transformers (PyTorch)

```python
import torch
from transformers import AutoModelForMaskedLM, AutoTokenizer

model_name = "sewoong/korean-neural-sparse-encoder-base-klue-large"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForMaskedLM.from_pretrained(model_name)
model.eval()

text = "서울 맛집 추천해주세요"
inputs = tokenizer(text, return_tensors="pt", max_length=256, truncation=True, padding=True)

with torch.no_grad():
    logits = model(**inputs).logits
    sparse = torch.log1p(torch.relu(logits))
    mask = inputs["attention_mask"].unsqueeze(-1).float()
    sparse_vec = (sparse * mask).max(dim=1).values.squeeze()

# Top activated tokens
top_indices = torch.nonzero(sparse_vec).squeeze()
top_values = sparse_vec[top_indices]
sorted_idx = top_values.argsort(descending=True)

for idx in sorted_idx[:20]:
    token_id = top_indices[idx].item()
    weight = top_values[idx].item()
    token = tokenizer.decode([token_id])
    print(f"  {token}: {weight:.3f}")
```

### With OpenSearch

```json
PUT /my-index
{
  "settings": {
    "index": {
      "default_pipeline": "sparse-pipeline"
    }
  },
  "mappings": {
    "properties": {
      "text": { "type": "text" },
      "sparse_embedding": { "type": "rank_features" }
    }
  }
}
```

## Training Configuration

```yaml
model:
  name: "klue/roberta-large"
  dropout: 0.1
loss:
  lambda_q: 0.01
  lambda_d: 0.003
  temperature: 1.0
  flops_warmup_steps: 20000
  lambda_initial_ratio: 0.1
data:
  batch_size: 128
  query_max_length: 64
  doc_max_length: 256
training:
  num_epochs: 25
  learning_rate: 5.0e-5
  weight_decay: 0.01
  warmup_ratio: 0.06
  gradient_accumulation_steps: 2
```

## Related Models

- [sewoong/korean-neural-sparse-encoder](https://huggingface.co/sewoong/korean-neural-sparse-encoder) - V33 base model (149M, skt/A.X-Encoder-base)
- [opensearch-project/opensearch-neural-sparse-encoding-multilingual-v1](https://huggingface.co/opensearch-project/opensearch-neural-sparse-encoding-multilingual-v1) - OpenSearch official multilingual sparse model

## License

Apache License 2.0