Magnuss85/Norn-v3-Icelandic-9B

Norn v3 -- Icelandic 9B Language Model

Author: Magnús Smári Smárason | smarason.is | HuggingFace: Magnuss85 Project: BORG AI Infrastructure, University of Akureyri Contact / provenance questions: smarason.is

Norn v3 is a fine-tuned Icelandic language model based on Qwen3.5-9B (9.4B parameters, dense). It was trained across multiple phases using a chain-of-thought training approach built on deterministic data from Iceland's national morphological database, BIN (Beygingarlýsing íslensks nútímamálsins).

Built at the University of Akureyri as part of the BORG AI infrastructure project, on a single NVIDIA RTX 5090.

No formal evaluation has been performed. This model has not been benchmarked against the Mideind Icelandic LLM Leaderboard or any other external evaluation suite. During training we used mocked benchmarks as a development signal to automate iteration on data structure, training parameters, and phase transitions -- these were a steering tool, not a validated measurement of model quality, and any "scores" they produced have been removed from this card. Treat Norn v3 as an unevaluated research artifact until proper benchmarking is published.

Highlights

• Deterministic CoT training data from BIN (3.1M word forms) -- zero hallucination risk in the morphological training data itself
• 130,612 training samples across programmatic BIN data, Gemini synthetic data, and IGC-2024 corpus
• Multi-phase training: LoRA hyperparameter search, block-wise full-rank training, conversational LoRA overlay, paradigm LoRA overlay
• Single RTX 5090 (32GB), ~210 GPU-hours, ~$15 Gemini API cost for synthetic data generation

---

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model = AutoModelForCausalLM.from_pretrained(
    "Magnuss85/Norn-v3-Icelandic-9B",
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained(
    "Magnuss85/Norn-v3-Icelandic-9B",
    trust_remote_code=True,
)

messages = [
    {"role": "system", "content": "Thu ert Norn, vitraen islensk gervigreind."},
    {"role": "user", "content": "Beyg nafnordid 'hestur' i ollum follum."},
]

text = tokenizer.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True, enable_thinking=False
)
inputs = tokenizer(text, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=512,
        temperature=0.7,
        top_p=0.8,
        top_k=20,
        repetition_penalty=1.1,
        do_sample=True,
    )

response = tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)
print(response)

For best results on complex grammatical tasks, enable thinking mode (enable_thinking=True) to allow the model to reason through its <think> blocks before responding.

---

The Data Factory Pipeline

The key innovation behind Norn is a programmatic data generation pipeline that produces provably correct training data from authoritative Icelandic linguistic resources -- no LLM involved in the morphological training data.

Data Alchemist (data_alchemist.py)

Transforms raw BIN morphological entries into chain-of-thought (CoT) training format. The model learns to reason through Icelandic grammar before producing the answer, using programmatic templates built from verified BIN metadata:

Example -- training sample for declension:

Prompt: Beyg ordid 'hestur' i tholfalli fleirtolu:

Target: [Hugsun: 'hestur' er karlkynsordi. Tholfall er andlagsfallid, notad thegar
         ordid er beint andlag sagna. I tholfalli fleirtolu er rett form 'hesta'.]
        hesta

The [Hugsun: ...] (thinking) block teaches the model to reason about gender, case function, and morphological rules. The bare inflected form at the end satisfies the benchmark format. This is entirely programmatic -- gender classification, case explanations, and inflected forms all come directly from BIN metadata, not from any LLM.

Why CoT was prioritized: Switching from bare-form training data to this CoT format was the largest single change in the training pipeline. Its impact on downstream Icelandic morphology has not been formally measured -- see the evaluation note above.

Paradigm Forge (paradigm_forge.py)

Generates complete 16-form Icelandic declension paradigm tables (4 cases x 2 numbers x 2 article states) using the islenska Python package, which provides programmatic access to BIN's 3.1 million word forms:

Example -- full paradigm table for "hestur":

           | Eintala (sg.) | Eintala m/gr. | Fleirtala (pl.) | Fleirtala m/gr.
-----------+---------------+---------------+-----------------+----------------
Nefnifall  | hestur        | hesturinn     | hestar          | hestarnir
Tholfall   | hest          | hestinn       | hesta           | hestana
Thagufall  | hesti         | hestinum      | hestum          | hestunum
Eignarfall | hests         | hestsins      | hesta           | hestanna

5,000 such tables were generated across all three grammatical genders, providing the model with structured morphological knowledge that is deterministic and provably correct.

Data Mixing Strategy

The final training mix was chosen empirically. An early attempt skewed heavily toward declension data (SP01, ~70%) qualitatively degraded grammar-error-detection behavior in spot checks. The mix below is the configuration that shipped:

Component	Ratio	Purpose
BIN paradigm tables + drills	30%	Declension mastery
Existing eval-aligned drills	10%	Benchmark format alignment
GED pairs	25%	Grammar error detection preservation
QA pairs	25%	Factual knowledge preservation
Conversational data	10%	Natural dialogue capability

---

Dataset Composition

130,612 total training samples from three categories:

Programmatic BIN data (deterministic, zero hallucination risk)

Component	Count	Source	Method
Declension drills (CoT format)	42,550	BIN (Arnastofnun)	data_alchemist.py
Full 16-form paradigm tables	5,000	BIN via islenska	paradigm_forge.py
Single-case eval-aligned drills	20,000	BIN via islenska	paradigm_forge.py
Adjective agreement, verb conjugation, article exercises	10,000	BIN	Programmatic
Subtotal	77,550

Gemini synthetic data (~$15 API cost)

Component	Count	Source	Method
Grammar Error Detection (GED) pairs	20,000	Gemini 3.1 Flash Lite	generate.py
Wikipedia-based QA	14,807	IGC-2024 seeds + Gemini	generate.py
Contextual case usage	10,000	Mixed seeds + Gemini	generate.py
Register transfer	5,000	Mixed seeds + Gemini	generate.py
Subtotal	49,807

Conversational alignment

Component	Count	Source	Method
Conversational Icelandic	~3,255	Gemini 3.1 Flash Lite	generate_chat.py

Source attribution

• BIN (Beygingarlýsing íslensks nútímamálsins): The authoritative Icelandic morphological database maintained by Arnastofnun. 3.1 million word forms. Accessed via the islenska Python package by Mideind.
• IGC-2024 (arnastofnun/IGC-2024): Icelandic Gigaword Corpus, 974K documents across parliamentary, news, wiki, law, social, and journal subcorpora. Used as seed text for Gemini synthetic generation.
• Gemini 3.1 Flash Lite: Google's lightweight model, used as teacher for knowledge distillation (synthetic data generation) and as evaluation judge for generative benchmarks.

---

Training Methodology

Norn v3 was produced through several training phases using a one-change-per-experiment workflow. Phase transitions were driven by mocked benchmarks -- a development-time harness whose role was to automate iteration on data structure and training parameters, not to measure model quality. Numeric outputs from that harness have been removed from this card, and external readers should not interpret the phase descriptions below as evidence of measured improvement.

Phase	Name	Method	Duration
1	Hyperparameter Search	LoRA sweeps over rank, LR, steps, etc.	~3 days
2	POSSESSION	Block-wise full-rank, 4 layers/block	83 GPU-hours
3	AWAKENING	Conversational LoRA overlay	18 minutes
4	Second Possession	Paradigm LoRA overlay	15 hours

Phase 1: LoRA Hyperparameter Search

Systematic search over learning rate (1e-5 to 5e-5), LoRA rank (8 to 64), training steps (500 to 10,700), weight decay, and sequence length. Each experiment changed one variable.

Configuration carried forward: r=16, alpha=16, all 7 projection modules (q/k/v/o/gate/up/down), LR 2e-5, cosine schedule, warmup 0.05, 500 steps, batch 2 x 4 gradient accumulation = 8 effective, seq_len 1024.

Mid-search, training data was restructured into the CoT format described above (data_alchemist.py); subsequent phases all use the CoT format.

Phase 2: POSSESSION -- Block-wise Full-Rank Training

Qwen3.5-9B has 32 transformer layers. Full-rank training of all layers requires ~54GB (model + optimizer), exceeding the 32GB RTX 5090. The block-wise approach trains 4 layers at a time while keeping the rest frozen:

Block	Layers	Trainable params	Duration
DELTA-TOP	28-31	865M (9.7%)	21h
DELTA-BOT	24-27	1.88B (21%)	21h
GAMMA	16-19	865M (9.7%)	20h
GAMMA2	20-23	865M (9.7%)	21h (halted on quality regression)

The GAMMA2 block was halted because training the 20-23 middle layers visibly degraded the CoT reasoning behavior in spot checks. The GAMMA checkpoint (12/32 layers trained) was tagged as Norn v1.

Phase 3: AWAKENING -- Conversational Alignment

LoRA overlay on Norn v1. 80% conversational data (Gemini-generated Icelandic dialogue) + 20% CoT preservation data. 500 steps, ~18 minutes. Tagged as Norn v2.

Phase 4: Second Possession -- Paradigm Mastery

Five LoRA overlays on Norn v2 over the balanced data mix (see Data Mixing Strategy):

Experiment	Steps	LR	Notes
SP01	500	1e-5	Reverted -- early data mix degraded grammar behavior
SP02	500	1e-5	Kept
SP03	500	2e-5	Kept
SP04	750	2e-5	Tagged as Norn v3 (deployed checkpoint)
SP05	1000	2e-5	Not adopted

---

Model Lineage

Qwen3.5-9B (base, 9.4B params)
  |-- LoRA hyperparameter search --> config: r=16, alpha=16, LR 2e-5
  |-- Block-wise full-rank (4 blocks, 83 GPU-hours) --> Norn v1
  |-- LoRA conversational overlay (~18 min)         --> Norn v2
  +-- LoRA paradigm overlay (750 steps)             --> Norn v3 (deployed)

---

Key Technical Findings

These observations come from the development workflow and the mocked benchmark signal that drove iteration. They are working hypotheses, not validated findings -- formal evaluation has not been performed.

1. Training data format appears to be a major lever. Restructuring BIN data into the CoT format described above was the largest single change in the pipeline; the mocked harness reacted strongly to it.

2. Data balance plausibly matters more than data volume. A 70%-declension mix (SP01) qualitatively degraded grammar behavior; the 30/10/25/25/10 balanced mix (SP04) did not.

3. Training budget is bounded. Increasing past ~750 steps at LR 2e-5 stopped producing improvement on the mocked harness and started showing regressions in spot checks.

4. Block-wise full-rank training has diminishing returns. 12/32 layers trained was the point at which further block training began regressing on the mocked signal.

5. 32GB VRAM is the practical ceiling for 9B full-rank training on this hardware. Multiple quantization strategies (BnB NF4, INT8, AWQ, GPTQ) were tried; none enabled Qwen3.5-27B training on this single-GPU setup.

---

Infrastructure

Component	Details
Training GPU	NVIDIA RTX 5090 (32GB, Blackwell)
Mocked-eval GPU	NVIDIA RTX 4090 (24GB, Ada Lovelace)
OS	Fedora 42, CUDA 12.8, driver 580.x
Framework	Unsloth + HuggingFace Transformers 5.x + TRL + PEFT
Precision	bf16 (bfloat16)
GPU-hours	~210
Training cost	$0 (consumer hardware)
Data generation cost	~$15 (Gemini 3.1 Flash Lite API)
Deployed format	Q8_0 GGUF (8.9 GB) via Ollama

---

Limitations

• No formal evaluation. The only evaluation signal used during training was a mocked benchmark harness intended to automate iteration, not to measure quality. There are no validated benchmark scores for this model. Do not rely on it in any setting where measured quality matters until proper evaluation has been published.
• Hallucinations. The model can produce incorrect facts, especially on topics underrepresented in training data.
• Thinking mode recommended. Complex grammatical tasks appear to perform best with enable_thinking=True.
• Irregular Icelandic nouns likely lag behind regular declension patterns.
• Primarily Icelandic. English capability is inherited from the Qwen3.5-9B base model and was not fine-tuned.
• 32GB VRAM ceiling on this hardware -- scaling to 27B parameters was not possible.

---

About

Magnús Smári Smárason | smarason.is | HuggingFace: Magnuss85

Norn is developed as part of the BORG AI infrastructure project at the University of Akureyri. Named after the Norse figures of fate who shape destiny, Norn aims to strengthen Icelandic language presence in the age of large language models.

The complete experiment data, training logs, and source code are preserved on the Hercules compute server. The project explores what a low-resource Icelandic LLM training pipeline can look like on consumer hardware with a deterministic data factory; whether it produces a meaningfully better Icelandic model is an open question that requires real evaluation to answer.

Acknowledgments

• Arnastofnun (The Arni Magnusson Institute for Icelandic Studies) for BIN and the IGC-2024 corpus
• Mideind for the islenska Python package and the Icelandic LLM Leaderboard benchmarks
• Qwen team (Alibaba) for the Qwen3.5-9B base model
• Google for Gemini 3.1 Flash Lite API access

External Resources

• BIN Database: https://bin.arnastofnun.is/
• IGC-2024 Dataset: https://huggingface.co/datasets/arnastofnun/IGC-2024
• Icelandic LLM Leaderboard: https://huggingface.co/spaces/mideind/icelandic-llm-leaderboard
• islenska Python package: https://pypi.org/project/islenska/

License

Apache 2.0 (same as base model)