HTML-LM

HTML-LM is a compact encoder model designed to generate general-purpose embeddings for HTML web pages, capturing both textual content and HTML structure. The embeddings can be used as inputs to lightweight downstream models for various classification and regression tasks.

HTML-LM representations can be reused across a wide range of downstream applications, supporting tasks such as classification, regression, clustering, and broader web-document understanding. Specifically, we use them for:

Explicit content classification — determining whether a webpage contains explicit or adult content.
Article-page detection — identifying whether a webpage primarily contains article or editorial content.
Product-page detection — determining whether a webpage represents a product or e-commerce listing.
Page clustering — grouping webpages with similar content and structural characteristics.
Page-quality regression — estimating the overall quality and usefulness of a webpage.
Web-spam detection — estimating the degree of spam or low-quality content on a webpage.

Model details

Property	Value
Architecture	ModernBERT
Parameters	154M
Hidden size	768
Layers	22
Attention heads	12
Context length	8192 (trained with 4096)
Vocabulary	57K

The model was trained from scratch for one pass over the training corpus using a combination of:

Masked Language Modeling (MLM)
Bag-of-Words prediction from [CLS]
Contrastive distillation from Qwen3-Embedding-8B and Seznam SeLLMa 8B

Training data

HTML-LM was trained on 100M HTML documents sampled from the Seznam.cz crawl database.

The corpus contains approximately:

53% Czech domains
34% primarily English-language domains
9% other European domains
4% other domains

HTML preprocessing

⚠️ HTML-LM expects HTML documents to be preprocessed in a specific way, which is handled by the bundled AutoProcessor.from_pretrained("Seznam/html-lm").

The included HTMLLMProcessor performs the following preprocessing steps:

Parsing the document into a DOM.
Removing scripts, styles, comments, and irrelevant subtrees.
Preserving text and selected structurally meaningful HTML tags.
Removing HTML attributes.
Simplifying the DOM hierarchy.
Normalizing whitespace.

Performance note: The bundled processor may be slow out of the box for high-throughput or large-scale workloads. For improved performance, we recommend using it with torch.utils.data.DataLoader and setting num_workers > 1. See the example below.

Usage

pip install transformers torch beautifulsoup4 lxml

import torch
from transformers import AutoModel, AutoProcessor

model_id = "Seznam/html-lm"

# <class 'transformers_modules.hf.processor.HTMLLMProcessor'>
processor = AutoProcessor.from_pretrained(
    model_id,
    trust_remote_code=True,
)
# <class 'transformers.models.modernbert.modeling_modernbert.ModernBertModel'>
model = AutoModel.from_pretrained(model_id).to("cuda")
model.eval()

html_pages = [
    "<html><body><h1>Example</h1><p>Some text.</p></body></html>",
]

inputs = processor(
    html_pages,
    padding=True,
    truncation=True,
    max_length=4096,
    return_tensors="pt",
)

with torch.no_grad():
    outputs = model(
        input_ids=inputs["input_ids"].to("cuda"),
        attention_mask=inputs["attention_mask"].to("cuda")
    )

# [CLS] representation = document embedding
document_embeddings = outputs.last_hidden_state[:, 0, :]

print(document_embeddings.shape)
# torch.Size([1, 768])

print(document_embeddings)
# tensor([[-2.0882e-01,  9.9287e-01, -1.0417e+00,  9.4297e-01, -8.4005e-01,
#          1.1509e+00,  8.4631e-01, -3.2688e-01,  7.1889e-01,  3.8875e-02,
#          ... 
#          -2.9921e-01, -1.0583e+00,  1.4555e+00]], device='cuda:0')

Performance

The models were evaluated using the following methodology: we first froze each model and extracted embeddings from the task-specific datasets. These embeddings were then used to train and evaluate lightweight, task-specific MLPs across five downstream tasks. The performance was then aggregated using NMM (Normalized Metric Mean), which represents the mean normalized improvement over a random baseline.

Model	Parameters	NMM ↑
ModernBERT base	149M	0.4835
text-embedding-3-small	—	0.6544
jina-embeddings-v3	570M	0.6466
Qwen3-Embedding-8B	8B	0.6571
SeLLMa 8B	8B	0.6103
HTML-LM Base	154M	0.7001

HTML-LM achieves the highest aggregated score in this evaluation despite being substantially smaller than the evaluated large embedding models.

Examples

Preprocessing Example

HTMLLMProcessor exposes preprocess_html(html: str) for obtaining the cleaned HTML before tokenization.

from transformers import AutoProcessor

model_id = "Seznam/html-lm"

# <class 'transformers_modules.hf.processor.HTMLLMProcessor'>
processor = AutoProcessor.from_pretrained(
    model_id,
    trust_remote_code=True,
)

html = """
<html>
    <body>
        <div></div>
        <h1 class='big'>Example</h1>
        <p class='small'>Some article text.</p>
    </body>
</html>
"""

print(processor.preprocess_html(html))
# <html><body><h1>Example</h1><p>Some article text.</p></body></html>

Faster Preprocessing

For faster preprocessing, you can use the code snippet below, which leverages multiprocessing to accelerate the preprocessor.

import torch
from transformers import AutoProcessor, BatchEncoding

# must be an instance of `torch.utils.data.Dataset` yielding raw HTML strings
dataset = YOUR_DATASET_HERE

model_id = "Seznam/html-lm"

processor = AutoProcessor.from_pretrained(
    model_id,
    trust_remote_code=True,
)

def collate_fn(html_inputs: list[str]) -> BatchEncoding:
    return processor(
        html_inputs,
        padding=True,
        truncation=True,
        max_length=4096,
        return_tensors="pt"
    )

loader = torch.utils.data.DataLoader(
    dataset, 
    batch_size=16, 
    collate_fn=collate_fn,
    num_workers=32
)

Testing

To verify your own integration, the repository ships a small end-to-end example in test/. It contains one sample page together with the expected output of every stage of the pipeline, so you can check each step in isolation:

File	Content
`test/raw.html`	Input: a raw HTML page with attributes, `<meta>` tags and a deeply nested DOM.
`test/processed.html`	Expected output of `processor.preprocess_html(raw_html)`.
`test/tokenized.npy`	Expected `input_ids` — `int64`, shape `(133,)`, no padding, no truncation.
`test/embedded.npy`	Expected document embedding — `float32`, shape `(768,)`.

The reference embedding was produced with the model loaded in torch.float32 and taken as the [CLS] vector (outputs.last_hidden_state[:, 0, :]).

import numpy as np
import torch
from transformers import AutoModel, AutoProcessor

model_id = "Seznam/html-lm"

processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
model = AutoModel.from_pretrained(model_id, torch_dtype=torch.float32).to("cuda")
model.eval()

raw_html = open("test/raw.html").read()
expected_html = open("test/processed.html").read()
expected_ids = np.load("test/tokenized.npy")
expected_embedding = np.load("test/embedded.npy")

# 1) preprocessing
processed_html = processor.preprocess_html(raw_html)
assert processed_html == expected_html

# 2) tokenization
inputs = processor([raw_html], return_tensors="pt")
assert np.array_equal(inputs["input_ids"][0].numpy(), expected_ids)

# 3) embedding
with torch.no_grad():
    outputs = model(
        input_ids=inputs["input_ids"].to("cuda"),
        attention_mask=inputs["attention_mask"].to("cuda"),
    )
embedding = outputs.last_hidden_state[0, 0, :].float().cpu().numpy()

cos = np.dot(embedding, expected_embedding) / (
    np.linalg.norm(embedding) * np.linalg.norm(expected_embedding)
)
assert cos > 0.999, cos

License

This model is released under the CC BY-NC 4.0 license.

Citation

If you use HTML-LM, please cite:

@inproceedings{dvorak2026html-lm,
  title     = {Size Matters: Foundation Model for Czech HTML documents},
  author    = {Dvořák, Martin and Tlustoš, Vít and Voronin, Artyom and Habrovec, Martin and Podlesná, Kateřina and Rišová, Barbora and Vonášek, Josef},
  year      = {TBD},
  publisher = {TBD}
}

Acknowledgements

HTML-LM was developed by the Seznam.cz Research team as part of the HTML-LM project.

Downloads last month: -

Safetensors

Model size

0.2B params

Tensor type

F32

Paper for Seznam/html-lm

Drop your Decoder: Pre-training with Bag-of-Word Prediction for Dense Passage Retrieval

Paper • 2401.11248 • Published Jan 20, 2024 • 1