---
license: mit
base_model: nvidia/Nemotron-Mini-4B-Instruct
tags:
  - typst
  - code-generation
  - qlora
  - fine-tuned
  - experimental
datasets:
  - jalasoft/typst-instruct
language:
  - en
  - es
pipeline_tag: text-generation
library_name: transformers
---

# Nemotron-Mini-4B-IT Fine-Tuned for Typst (Experimental)

<p align="center">
  <a href="https://huggingface.co/jalasoft/nemotron-mini-4B-it-ft-typ"><img src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Model-blue" alt="HuggingFace Model"></a>
  <a href="https://opensource.org/licenses/MIT"><img src="https://img.shields.io/badge/License-MIT-green.svg" alt="License: MIT"></a>
  <img src="https://img.shields.io/badge/Status-Experimental-red" alt="Status: Experimental">
  <img src="https://img.shields.io/badge/Base-Nemotron--Mini--4B-purple" alt="Base Model">
  <img src="https://img.shields.io/badge/Method-QLoRA-orange" alt="Training Method">
</p>

<p align="center">
  <b>An experimental QLoRA fine-tuned model for Typst code generation - shared for research and educational purposes</b>
</p>

---

> **Warning**: This is an experimental model that did not achieve production-quality results. It is shared for research transparency and educational purposes only.

## Table of Contents

- [Nemotron-Mini-4B-IT Fine-Tuned for Typst (Experimental)](#nemotron-mini-4b-it-fine-tuned-for-typst-experimental)
  - [Table of Contents](#table-of-contents)
  - [Model Summary](#model-summary)
  - [Intended Use Cases (Not Achieved)](#intended-use-cases-not-achieved)
  - [Training Details](#training-details)
    - [Training Procedure](#training-procedure)
    - [Training Hyperparameters](#training-hyperparameters)
    - [Training Data](#training-data)
  - [Known Limitations](#known-limitations)
    - [Observed Issues](#observed-issues)
    - [Recommendations](#recommendations)
  - [Evaluation](#evaluation)
  - [How to Use](#how-to-use)
  - [Citation and Licensing](#citation-and-licensing)
    - [License](#license)
    - [Citation](#citation)
  - [Additional Information](#additional-information)
    - [Model Card Authors](#model-card-authors)
    - [Related Resources](#related-resources)

---

## Model Summary

This model is a QLoRA fine-tuned version of [nvidia/Nemotron-Mini-4B-Instruct](https://huggingface.co/nvidia/Nemotron-Mini-4B-Instruct) trained on the [jalasoft/typst-instruct](https://huggingface.co/datasets/jalasoft/typst-instruct) dataset for generating [Typst](https://typst.app/) markup language documents from natural language instructions.

| Attribute | Value |
|-----------|-------|
| **Base Model** | nvidia/Nemotron-Mini-4B-Instruct |
| **Fine-tuning Method** | QLoRA (4-bit quantization) |
| **Training Framework** | Axolotl |
| **Dataset** | jalasoft/typst-instruct (1,016 examples) |
| **Task** | Text-to-Typst code generation |
| **Status** | Experimental (not production-ready) |

## Intended Use Cases (Not Achieved)

The model was developed with the goal of:

- **Document generation**: Creating complete Typst documents from natural language descriptions
- **Template creation**: Generating reusable Typst templates for various document types

However, the model did not meet the quality threshold required for these use cases due to the limitations described below.

## Training Details

### Training Procedure

- **Method**: QLoRA (Quantized Low-Rank Adaptation)
- **Framework**: Axolotl
- **Precision**: BF16 with 4-bit quantization
- **Flash Attention**: Enabled
- **LoRA Target**: All linear layers (auto-detected)

### Training Hyperparameters

| Parameter | Value |
|-----------|-------|
| LoRA Rank (r) | 64 |
| LoRA Alpha | 128 |
| LoRA Dropout | 0.1 |
| LoRA Target Linear | true (all linear layers) |
| Learning Rate | 2e-4 |
| LR Scheduler | Cosine |
| Optimizer | AdamW (fused) |
| Epochs | 5 |
| Sequence Length | 4096 |
| Micro Batch Size | 4 |
| Gradient Accumulation | 4 |
| Warmup Ratio | 0.1 |
| Weight Decay | 0.01 |
| Sample Packing | Enabled |
| Gradient Checkpointing | Enabled |

### Training Data

- **Dataset**: [jalasoft/typst-instruct](https://huggingface.co/datasets/jalasoft/typst-instruct)
- **Size**: 1,016 instruction-completion pairs
- **Content**: Natural language instructions paired with Typst code and feature summaries
- **Coverage**: Academic documents, technical tutorials, reports, tables, figures, and advanced typography

## Known Limitations

This model is marked as **experimental** due to significant issues observed during evaluation.

### Observed Issues

| Issue | Description |
|-------|-------------|
| **Hallucinations** | The model frequently generates incorrect or non-existent Typst syntax and features |
| **Code Quality** | Generated Typst code often contains syntax errors and fails to compile |

These limitations make the model unsuitable for production use in document generation workflows.

### Recommendations

If you experiment with this model:

- **Always validate** generated Typst code before use
- **Expect compilation errors** in most complex task outputs
- **Use for research purposes** to understand fine-tuning challenges
- **Consider larger models** or more training data for production use cases

## Evaluation

Performance was evaluated qualitatively across multiple model versions using diverse Typst generation test cases. Results consistently fell below the quality threshold required for the intended use cases.

## How to Use

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "jalasoft/nemotron-mini-4B-it-ft-typ"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype="float16")

# Nemotron uses a specific chat format
prompt = """<extra_id_0>System
You are an expert in Typst markup language. Generate clean, well-formatted Typst code based on user instructions.

<extra_id_1>User
Create a simple document with a title and two paragraphs
<extra_id_1>Assistant
"""

inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=1000)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

## Citation and Licensing

### License

This model is released under the **MIT License**. You are free to use, modify, and distribute this model for any purpose, including commercial applications.

### Citation

If you use this model in your research, please cite:

```bibtex
@misc{nemotron-typst-experimental,
  author = {{Jala R\&D}},
  title = {Nemotron-Mini-4B Fine-Tuned for Typst (Experimental)},
  year = {2025},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/jalasoft/nemotron-mini-4B-it-ft-typ}}
}
```

## Additional Information

### Model Card Authors

Created by Jalasoft R&D ([@jalasoft](https://huggingface.co/jalasoft))

### Related Resources

- **Training Dataset**: [jalasoft/typst-instruct](https://huggingface.co/datasets/jalasoft/typst-instruct)
- **Base Model**: [nvidia/Nemotron-Mini-4B-Instruct](https://huggingface.co/nvidia/Nemotron-Mini-4B-Instruct)
- **Typst Official Documentation**: <https://typst.app/docs>
- **Typst GitHub Repository**: <https://github.com/typst/typst>