Update README.md
Browse files
README.md
CHANGED
|
@@ -15,4 +15,46 @@ base_model:
|
|
| 15 |
pipeline_tag: text-generation
|
| 16 |
tags:
|
| 17 |
- LaTeX
|
| 18 |
-
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 15 |
pipeline_tag: text-generation
|
| 16 |
tags:
|
| 17 |
- LaTeX
|
| 18 |
+
---
|
| 19 |
+
|
| 20 |
+
# IntelliTeX: Natural Language to LaTeX Conversion
|
| 21 |
+
|
| 22 |
+
## Model Description
|
| 23 |
+
**IntelliTeX** is a specialized, highly efficient Small Language Model (SLM) designed to convert **colloquial, natural-language descriptions of mathematical formulas into accurate, compilable LaTeX expressions**.
|
| 24 |
+
|
| 25 |
+
By focusing on efficiency, IntelliTeX is ideal for real-time, on-device deployment, such as web applications using libraries like `transformer.js`. It is built upon the [CodeT5+](https://huggingface.co/Salesforce/codet5p-220m) architecture, with a 220-million-parameter Encoder-Decoder base.
|
| 26 |
+
|
| 27 |
+
## Training Data & Strategy
|
| 28 |
+
IntelliTeX was trained using a custom **two-stage training strategy** to maximize performance while minimizing parameter count.
|
| 29 |
+
|
| 30 |
+
1. **Stage 1: Domain-Adaptive Pre-training**: The model was continually pre-trained on the [TeXTeller](https://huggingface.co/OleehyO/TexTeller) dataset (approx. 4B Token) to adapt its encoder to mathematical syntax using span denoising and causal language modeling.
|
| 31 |
+
2. **Stage 2: Supervised Fine-Tuning**: The model was fine-tuned on the high-quality [Speech2LaTeX (S2L)](https://huggingface.co/datasets/marsianin500/Speech2Latex) dataset, which provides parallel pairs of transcribed natural-language input and corresponding LaTeX output.
|
| 32 |
+
|
| 33 |
+
## Key Capabilities
|
| 34 |
+
* **Long Context Handling:** Unlike standard fine-tuned models which often degrade when processing lengthy descriptions, IntelliTeX leverages its Stage 1 pre-training to maintain high accuracy even when inputs exceed 115 characters or require generating complex LaTeX sequences longer than 60 characters.
|
| 35 |
+
* **High Compilability:** Achieves a 99.8% compilable rate, ensuring that generated code is ready to render without syntax errors.
|
| 36 |
+
|
| 37 |
+
## Evaluation Results
|
| 38 |
+
The specialized training resulted in a model that achieves high accuracy and superior compilability compared to other small models on the S2L test dataset (2,745 samples).
|
| 39 |
+
|
| 40 |
+
| Metric | IntelliTeX (CodeT5+ 220M) | Qwen2.5-Coder-0.5B |
|
| 41 |
+
| :--- | :---: | :---: |
|
| 42 |
+
| **Compilable Rate (CR)** | **0.998** | 0.990 |
|
| 43 |
+
| **TexBLEU** | **0.915** | 0.902 |
|
| 44 |
+
| **Character Error Rate (CER)** | **0.22** | 0.24 |
|
| 45 |
+
|
| 46 |
+
## Running Proof-of-Concept Application (POC)
|
| 47 |
+
|
| 48 |
+
The design and small size of IntelliTeX enable significantly faster inference on CPU-constrained environments like web browsers compared to comparable decoder-only models.
|
| 49 |
+
|
| 50 |
+
A live, in-browser demonstration using the `transformer.js` library showcases this efficiency advantage on typical CPU hardware.
|
| 51 |
+
|
| 52 |
+
| Model | Average Inference Latency (CPU) |
|
| 53 |
+
| :--- | :---: |
|
| 54 |
+
| **IntelliTeX** | **~500 ms** |
|
| 55 |
+
| **Qwen2.5-Coder-0.5B-Instruct** | **~3000 ms** |
|
| 56 |
+
|
| 57 |
+
|
| 58 |
+
| IntelliTeX (Fast CPU Inference) | Qwen2.5-Coder-0.5B-Instruct (Slower CPU Inference) |
|
| 59 |
+
| :---: | :---: |
|
| 60 |
+
|  | |
|