XiaSheng/FreeChunk-nomic

FreeChunker-Nomic

FreeChunker is a training-free embedding optimization method that dynamically chunks text to improve retrieval performance. This repository contains the FreeChunker model initialized with nomic-ai/nomic-embed-text-v1.5 embeddings.

Features

• Dynamic Chunking: Automatically groups sentences into semantically coherent chunks.
• Optimized for RAG: Improves retrieval augmented generation by providing better context segments.
• Backbone: Built on top of nomic-ai/nomic-embed-text-v1.5 sentence embeddings.

Requirements

pip install torch transformers sentence-transformers numpy

Usage

Abstract Usage

from transformers import AutoModel
import torch

# 1. Load Model (UnifiedEncoder)
model = AutoModel.from_pretrained("XiaSheng/FreeChunk-nomic", trust_remote_code=True)

# 2. Build Vector Store from Text
text = "Your text..."
model.build_vector_store(text)

# 3. Query with Post-Aggregation (Default)
query = "Your query..."
results = model.query(query, top_k=1, aggregation_mode='post')

print(f"Query: {query}")
print(f"Result: {results}")

Files

• model.safetensors: The FreeChunker model weights.
• encoder.py: High-level interface (UnifiedEncoder) for end-to-end usage.
• sentenizer.py: Helper for text splitting and backbone embedding.
• aggregator.py: Helper for aggregating retrieved results.
• configuration_freechunker.py & modeling_freechunker.py: Model definition.

Citation

If you use this model in your research, please cite:

@article{zhang2025freechunker, 
   title={FreeChunker: A Cross-Granularity Chunking Framework}, 
   author={Zhang, Wenxuan and Jiang, Yuan-Hao and Wu, Yonghe}, 
   journal={arXiv preprint arXiv:2510.20356}, 
   year={2025} 
 }