fusion-llm-demo / scripts /fix_think_tokens.py

Commit History

fix: F1 mask double-conversion NaN, F2 think token old format, S3 vocab sync, S1/S2 TODO, N1-N5
4cfb0d8

zhan1206 commited on

fix: v5 remove empty tokenizer.json, unify think tokens, integrate tokenizer into training pipeline
bae07a4

zhan1206 commited on