CoLMLM-Standard-LM-Baseline-360M-FW

The 360M-parameter standard LM baseline from the paper Co-LMLM: Continuous-Query Limited Memory Language Models, trained on FineWeb-Edu.

This is the data-matched control for CoLMLM-360M-FW: an ordinary causal language model trained from scratch on the same corpus, with the <FACT> annotations stripped out to plain text.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "lil-lab/CoLMLM-Standard-LM-Baseline-360M-FW"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

inputs = tokenizer("The Eiffel Tower is located in", return_tensors="pt")
print(tokenizer.decode(model.generate(**inputs, max_new_tokens=32)[0]))

This model is part of the Co-LMLM collection.

Citation

@misc{feldman2026colmlmcontinuousquerylimitedmemory,
      title={Co-LMLM: Continuous-Query Limited Memory Language Models},
      author={Yair Feldman and Linxi Zhao and Nathan Godey and Dongyoung Go and Yilun Hua and Kilian Q. Weinberger and Jennifer J. Sun and Yoav Artzi},
      year={2026},
      eprint={2607.07707},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2607.07707},
}