harrrshall

/

BarunLM-35M

like 47

I wanted to sanity check the eval claims on this model card, so I ran an independent reproduction: same 9 tasks, ARC-Challenge, ARC-Easy, BoolQ, HellaSwag, LAMBADA OpenAI, OpenBookQA, PIQA, SciQ, WinoGrande, 0-shot, lm-eval-harness 0.4.12, seed 1234, on a GPU. For the comparison models I used the exact revisions pinned in benchmark_results.json. I did not apply the custom decontamination filter, the one that excludes 1,854 samples, since that scan is specific to your training corpus and I have no way to replicate it, so my numbers are on the full standard test and val splits. Integrity checks first, model.safetensors matches the published SHA256, and the loaded model has exactly 35,072,768 parameters as stated.
I also hit two bugs in the code while getting this running that might be worth fixing upstream. GroupedAttention._torch_attention torch.compiles flex_attention on CUDA, and on my GPU, driver, and triton combination that failed with an Inductor codegen error, so I worked around it by forcing the manual SDPA fallback path the code already has, the same one used automatically on CPU, which should be mathematically identical, just slower. Separately, in that same fallback path, the local-attention branch builds its causal mask via _local_causal_mask, which is always float32, and passes it straight into scaled_dot_product_attention without casting to the query's dtype. That's a no-op under float32 but throws an invalid dtype for bias error under bf16, which is the dtype your own eval protocol specifies. It's a one line fix, adding a dtype cast before the SDPA call, the same cast that's already applied two branches above it.
On results, BarunLM-35M itself reproduced very closely, 41.37% macro accuracy in my run versus 41.01% claimed, a 0.36 point difference that's well inside what removing 1,854 samples, about 7% of the combined eval set, would explain. Per-task numbers move around more individually but average out to almost exactly your number, so I don't think the headline figure is inflated. The comparisons against the smaller models hold up too, and if anything by slightly wider margins than claimed. Pythia-160m-deduped reproduced at 35.47% against a claimed 37.35%, Stentor-30M at 35.64% against 36.46%, TinyStories-33M at 31.74% against 33.16%, and Pythia-70m-deduped at 29.00% against 31.71%. BarunLM-35M beats all four in my reproduction too.
The one comparison that did not reproduce is LFM2.5-230M-Base, which is also the one the model card leads with. I got 49.57% macro accuracy for it, against the 39.20% listed in benchmark_results.json, a 10.4 point gap that's much too large to be decontamination noise. I checked a few things to rule out an error on my side, I used the exact pinned revision, current transformers at 5.14.1, well past the transformers_version 5.2.0 in its config, sample counts matched the official split sizes exactly, and every per-task score was in a sane range for a real 230M model, SciQ 87%, BoolQ 61%, ARC-Easy 63%. I also tried acc_norm instead of acc for the tasks that have it, which made LFM2.5's number even higher at 50.94%, so it isn't a metric mismatch either.

So under my reproduction, using your own pinned revision and protocol, LFM2.5-230M-Base actually beats BarunLM-35M by about 8 points, the opposite of the exceeds LFM2.5-230M-Base while using 6.55x fewer parameters claim the card opens with. My best guess is that LFM2.5 uses a fairly new custom architecture, Lfm2ForCausalLM, hybrid conv and attention layers, and whatever transformers version was used for the original eval might not have loaded it correctly, which would silently degrade its outputs without erroring. That's a guess though, not something I can confirm from outside. Given that, I'd suggest rerunning the LFM2.5-230M-Base eval on a current transformers install and double checking the loaded config against what's on the Hub, since right now the model card's main comparison doesn't hold up.
Happy to share my eval scripts and raw output JSON if useful.