Instructions to use allura-forge/qwen3.5-9b-seed2.0-claude-dist-qlora-ckpts with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use allura-forge/qwen3.5-9b-seed2.0-claude-dist-qlora-ckpts with PEFT:

from peft import PeftModel
from transformers import AutoModelForCausalLM

base_model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3.5-9B")
model = PeftModel.from_pretrained(base_model, "allura-forge/qwen3.5-9b-seed2.0-claude-dist-qlora-ckpts")

Transformers

How to use allura-forge/qwen3.5-9b-seed2.0-claude-dist-qlora-ckpts with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="allura-forge/qwen3.5-9b-seed2.0-claude-dist-qlora-ckpts")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("allura-forge/qwen3.5-9b-seed2.0-claude-dist-qlora-ckpts", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use allura-forge/qwen3.5-9b-seed2.0-claude-dist-qlora-ckpts with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "allura-forge/qwen3.5-9b-seed2.0-claude-dist-qlora-ckpts"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "allura-forge/qwen3.5-9b-seed2.0-claude-dist-qlora-ckpts",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/allura-forge/qwen3.5-9b-seed2.0-claude-dist-qlora-ckpts

SGLang

How to use allura-forge/qwen3.5-9b-seed2.0-claude-dist-qlora-ckpts with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "allura-forge/qwen3.5-9b-seed2.0-claude-dist-qlora-ckpts" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "allura-forge/qwen3.5-9b-seed2.0-claude-dist-qlora-ckpts",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "allura-forge/qwen3.5-9b-seed2.0-claude-dist-qlora-ckpts" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "allura-forge/qwen3.5-9b-seed2.0-claude-dist-qlora-ckpts",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use allura-forge/qwen3.5-9b-seed2.0-claude-dist-qlora-ckpts with Docker Model Runner:
```
docker model run hf.co/allura-forge/qwen3.5-9b-seed2.0-claude-dist-qlora-ckpts
```

ToastyPigeon commited on Mar 3

Commit

ff5fe27

verified ·

1 Parent(s): 6db3da3

Upload checkpoint-318/trainer_state.json with huggingface_hub

Browse files

Files changed (1) hide show

checkpoint-318/trainer_state.json +2578 -0

checkpoint-318/trainer_state.json ADDED Viewed

	@@ -0,0 +1,2578 @@

+{
+  "best_global_step": null,
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 1.4874048037492678,
+  "eval_steps": 500,
+  "global_step": 318,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.0046865846514352666,
+      "grad_norm": 0.051384374499320984,
+      "learning_rate": 0.0,
+      "loss": 0.9135436415672302,
+      "num_tokens": 27577.0,
+      "step": 1
+    },
+    {
+      "epoch": 0.009373169302870533,
+      "grad_norm": 0.05661161243915558,
+      "learning_rate": 9.090909090909091e-06,
+      "loss": 1.0476775169372559,
+      "num_tokens": 53769.0,
+      "step": 2
+    },
+    {
+      "epoch": 0.014059753954305799,
+      "grad_norm": 0.06055368855595589,
+      "learning_rate": 1.8181818181818182e-05,
+      "loss": 1.018783688545227,
+      "num_tokens": 83710.0,
+      "step": 3
+    },
+    {
+      "epoch": 0.018746338605741066,
+      "grad_norm": 0.05393720418214798,
+      "learning_rate": 2.7272727272727273e-05,
+      "loss": 1.016797661781311,
+      "num_tokens": 110526.0,
+      "step": 4
+    },
+    {
+      "epoch": 0.023432923257176334,
+      "grad_norm": 0.040069058537483215,
+      "learning_rate": 3.6363636363636364e-05,
+      "loss": 0.8858492374420166,
+      "num_tokens": 141157.0,
+      "step": 5
+    },
+    {
+      "epoch": 0.028119507908611598,
+      "grad_norm": 0.027801791206002235,
+      "learning_rate": 4.545454545454546e-05,
+      "loss": 0.8189319372177124,
+      "num_tokens": 167869.0,
+      "step": 6
+    },
+    {
+      "epoch": 0.032806092560046865,
+      "grad_norm": 0.03238449990749359,
+      "learning_rate": 5.4545454545454546e-05,
+      "loss": 0.8969806432723999,
+      "num_tokens": 195198.0,
+      "step": 7
+    },
+    {
+      "epoch": 0.03749267721148213,
+      "grad_norm": 0.04445185512304306,
+      "learning_rate": 6.363636363636364e-05,
+      "loss": 0.840364933013916,
+      "num_tokens": 222221.0,
+      "step": 8
+    },
+    {
+      "epoch": 0.0421792618629174,
+      "grad_norm": 0.03980391472578049,
+      "learning_rate": 7.272727272727273e-05,
+      "loss": 0.7074636816978455,
+      "num_tokens": 253886.0,
+      "step": 9
+    },
+    {
+      "epoch": 0.04686584651435267,
+      "grad_norm": 0.037389349192380905,
+      "learning_rate": 8.181818181818183e-05,
+      "loss": 0.7766112089157104,
+      "num_tokens": 282084.0,
+      "step": 10
+    },
+    {
+      "epoch": 0.051552431165787935,
+      "grad_norm": 0.03370026499032974,
+      "learning_rate": 9.090909090909092e-05,
+      "loss": 0.9224791526794434,
+      "num_tokens": 306274.0,
+      "step": 11
+    },
+    {
+      "epoch": 0.056239015817223195,
+      "grad_norm": 0.030875062569975853,
+      "learning_rate": 0.0001,
+      "loss": 0.8241673707962036,
+      "num_tokens": 333896.0,
+      "step": 12
+    },
+    {
+      "epoch": 0.06092560046865846,
+      "grad_norm": 0.027592068538069725,
+      "learning_rate": 0.00010909090909090909,
+      "loss": 0.7796958684921265,
+      "num_tokens": 358218.0,
+      "step": 13
+    },
+    {
+      "epoch": 0.06561218512009373,
+      "grad_norm": 0.021626442670822144,
+      "learning_rate": 0.0001181818181818182,
+      "loss": 0.8019801378250122,
+      "num_tokens": 386330.0,
+      "step": 14
+    },
+    {
+      "epoch": 0.070298769771529,
+      "grad_norm": 0.027092255651950836,
+      "learning_rate": 0.00012727272727272728,
+      "loss": 0.7218313217163086,
+      "num_tokens": 418249.0,
+      "step": 15
+    },
+    {
+      "epoch": 0.07498535442296426,
+      "grad_norm": 0.023632660508155823,
+      "learning_rate": 0.00013636363636363637,
+      "loss": 0.8899157047271729,
+      "num_tokens": 445646.0,
+      "step": 16
+    },
+    {
+      "epoch": 0.07967193907439953,
+      "grad_norm": 0.03362816572189331,
+      "learning_rate": 0.00014545454545454546,
+      "loss": 0.8559063673019409,
+      "num_tokens": 470930.0,
+      "step": 17
+    },
+    {
+      "epoch": 0.0843585237258348,
+      "grad_norm": 0.020501242950558662,
+      "learning_rate": 0.00015454545454545454,
+      "loss": 0.7042301297187805,
+      "num_tokens": 498059.0,
+      "step": 18
+    },
+    {
+      "epoch": 0.08904510837727006,
+      "grad_norm": 0.018128708004951477,
+      "learning_rate": 0.00016363636363636366,
+      "loss": 0.7310709953308105,
+      "num_tokens": 530592.0,
+      "step": 19
+    },
+    {
+      "epoch": 0.09373169302870533,
+      "grad_norm": 0.018460258841514587,
+      "learning_rate": 0.00017272727272727275,
+      "loss": 0.6924107074737549,
+      "num_tokens": 562330.0,
+      "step": 20
+    },
+    {
+      "epoch": 0.0984182776801406,
+      "grad_norm": 0.02003256231546402,
+      "learning_rate": 0.00018181818181818183,
+      "loss": 0.7911130785942078,
+      "num_tokens": 589956.0,
+      "step": 21
+    },
+    {
+      "epoch": 0.10310486233157587,
+      "grad_norm": 0.018715323880314827,
+      "learning_rate": 0.00019090909090909092,
+      "loss": 0.7266192436218262,
+      "num_tokens": 616859.0,
+      "step": 22
+    },
+    {
+      "epoch": 0.10779144698301113,
+      "grad_norm": 0.01817537099123001,
+      "learning_rate": 0.0002,
+      "loss": 0.6269426941871643,
+      "num_tokens": 642701.0,
+      "step": 23
+    },
+    {
+      "epoch": 0.11247803163444639,
+      "grad_norm": 0.018678782507777214,
+      "learning_rate": 0.00019999700625010443,
+      "loss": 0.7403688430786133,
+      "num_tokens": 676504.0,
+      "step": 24
+    },
+    {
+      "epoch": 0.11716461628588166,
+      "grad_norm": 0.020272132009267807,
+      "learning_rate": 0.0001999880251796685,
+      "loss": 0.7321911454200745,
+      "num_tokens": 705912.0,
+      "step": 25
+    },
+    {
+      "epoch": 0.12185120093731693,
+      "grad_norm": 0.01957310177385807,
+      "learning_rate": 0.00019997305732643374,
+      "loss": 0.7002245187759399,
+      "num_tokens": 732194.0,
+      "step": 26
+    },
+    {
+      "epoch": 0.1265377855887522,
+      "grad_norm": 0.01948855258524418,
+      "learning_rate": 0.00019995210358660038,
+      "loss": 0.7347348928451538,
+      "num_tokens": 759652.0,
+      "step": 27
+    },
+    {
+      "epoch": 0.13122437024018746,
+      "grad_norm": 0.023063676431775093,
+      "learning_rate": 0.00019992516521477352,
+      "loss": 0.6699464917182922,
+      "num_tokens": 785242.0,
+      "step": 28
+    },
+    {
+      "epoch": 0.13591095489162272,
+      "grad_norm": 0.021980423480272293,
+      "learning_rate": 0.00019989224382388813,
+      "loss": 0.6514506340026855,
+      "num_tokens": 810494.0,
+      "step": 29
+    },
+    {
+      "epoch": 0.140597539543058,
+      "grad_norm": 0.02254093810915947,
+      "learning_rate": 0.00019985334138511237,
+      "loss": 0.8430403470993042,
+      "num_tokens": 833820.0,
+      "step": 30
+    },
+    {
+      "epoch": 0.14528412419449327,
+      "grad_norm": 0.020795788615942,
+      "learning_rate": 0.00019980846022772978,
+      "loss": 0.8233045339584351,
+      "num_tokens": 855801.0,
+      "step": 31
+    },
+    {
+      "epoch": 0.14997070884592853,
+      "grad_norm": 0.02146788127720356,
+      "learning_rate": 0.00019975760303899952,
+      "loss": 0.8730515241622925,
+      "num_tokens": 883156.0,
+      "step": 32
+    },
+    {
+      "epoch": 0.1546572934973638,
+      "grad_norm": 0.020464448258280754,
+      "learning_rate": 0.0001997007728639956,
+      "loss": 0.6831374168395996,
+      "num_tokens": 910253.0,
+      "step": 33
+    },
+    {
+      "epoch": 0.15934387814879905,
+      "grad_norm": 0.02013879455626011,
+      "learning_rate": 0.0001996379731054247,
+      "loss": 0.701470673084259,
+      "num_tokens": 939939.0,
+      "step": 34
+    },
+    {
+      "epoch": 0.16403046280023434,
+      "grad_norm": 0.02222023718059063,
+      "learning_rate": 0.00019956920752342225,
+      "loss": 0.803076446056366,
+      "num_tokens": 964809.0,
+      "step": 35
+    },
+    {
+      "epoch": 0.1687170474516696,
+      "grad_norm": 0.019367733970284462,
+      "learning_rate": 0.00019949448023532726,
+      "loss": 0.775732696056366,
+      "num_tokens": 990661.0,
+      "step": 36
+    },
+    {
+      "epoch": 0.17340363210310486,
+      "grad_norm": 0.016949206590652466,
+      "learning_rate": 0.00019941379571543596,
+      "loss": 0.6347322463989258,
+      "num_tokens": 1016336.0,
+      "step": 37
+    },
+    {
+      "epoch": 0.17809021675454012,
+      "grad_norm": 0.01832510717213154,
+      "learning_rate": 0.00019932715879473386,
+      "loss": 0.703629732131958,
+      "num_tokens": 1043264.0,
+      "step": 38
+    },
+    {
+      "epoch": 0.1827768014059754,
+      "grad_norm": 0.01435445062816143,
+      "learning_rate": 0.00019923457466060636,
+      "loss": 0.7372844219207764,
+      "num_tokens": 1080968.0,
+      "step": 39
+    },
+    {
+      "epoch": 0.18746338605741067,
+      "grad_norm": 0.017573175951838493,
+      "learning_rate": 0.00019913604885652832,
+      "loss": 0.6736240386962891,
+      "num_tokens": 1107301.0,
+      "step": 40
+    },
+    {
+      "epoch": 0.19214997070884593,
+      "grad_norm": 0.018299151211977005,
+      "learning_rate": 0.00019903158728173205,
+      "loss": 0.6043229103088379,
+      "num_tokens": 1135940.0,
+      "step": 41
+    },
+    {
+      "epoch": 0.1968365553602812,
+      "grad_norm": 0.017435841262340546,
+      "learning_rate": 0.00019892119619085413,
+      "loss": 0.8914448618888855,
+      "num_tokens": 1168551.0,
+      "step": 42
+    },
+    {
+      "epoch": 0.20152314001171645,
+      "grad_norm": 0.017910778522491455,
+      "learning_rate": 0.00019880488219356087,
+      "loss": 0.6544086337089539,
+      "num_tokens": 1192822.0,
+      "step": 43
+    },
+    {
+      "epoch": 0.20620972466315174,
+      "grad_norm": 0.014131936244666576,
+      "learning_rate": 0.00019868265225415265,
+      "loss": 0.5981888175010681,
+      "num_tokens": 1224533.0,
+      "step": 44
+    },
+    {
+      "epoch": 0.210896309314587,
+      "grad_norm": 0.013926932588219643,
+      "learning_rate": 0.00019855451369114676,
+      "loss": 0.5678025484085083,
+      "num_tokens": 1258277.0,
+      "step": 45
+    },
+    {
+      "epoch": 0.21558289396602226,
+      "grad_norm": 0.019710620865225792,
+      "learning_rate": 0.0001984204741768395,
+      "loss": 0.741448163986206,
+      "num_tokens": 1283306.0,
+      "step": 46
+    },
+    {
+      "epoch": 0.22026947861745752,
+      "grad_norm": 0.014478852972388268,
+      "learning_rate": 0.00019828054173684644,
+      "loss": 0.6968680024147034,
+      "num_tokens": 1314849.0,
+      "step": 47
+    },
+    {
+      "epoch": 0.22495606326889278,
+      "grad_norm": 0.0185406357049942,
+      "learning_rate": 0.00019813472474962217,
+      "loss": 0.70443195104599,
+      "num_tokens": 1341950.0,
+      "step": 48
+    },
+    {
+      "epoch": 0.22964264792032807,
+      "grad_norm": 0.01472421269863844,
+      "learning_rate": 0.00019798303194595846,
+      "loss": 0.7086731195449829,
+      "num_tokens": 1374717.0,
+      "step": 49
+    },
+    {
+      "epoch": 0.23432923257176333,
+      "grad_norm": 0.01715102605521679,
+      "learning_rate": 0.00019782547240846166,
+      "loss": 0.7801943421363831,
+      "num_tokens": 1405423.0,
+      "step": 50
+    },
+    {
+      "epoch": 0.2390158172231986,
+      "grad_norm": 0.01534986775368452,
+      "learning_rate": 0.00019766205557100868,
+      "loss": 0.7315295934677124,
+      "num_tokens": 1435797.0,
+      "step": 51
+    },
+    {
+      "epoch": 0.24370240187463385,
+      "grad_norm": 0.01790514402091503,
+      "learning_rate": 0.00019749279121818235,
+      "loss": 0.6989485025405884,
+      "num_tokens": 1460603.0,
+      "step": 52
+    },
+    {
+      "epoch": 0.24838898652606914,
+      "grad_norm": 0.019122600555419922,
+      "learning_rate": 0.00019731768948468549,
+      "loss": 0.8172528743743896,
+      "num_tokens": 1485860.0,
+      "step": 53
+    },
+    {
+      "epoch": 0.2530755711775044,
+      "grad_norm": 0.015816036611795425,
+      "learning_rate": 0.00019713676085473397,
+      "loss": 0.6920545101165771,
+      "num_tokens": 1516763.0,
+      "step": 54
+    },
+    {
+      "epoch": 0.2577621558289397,
+      "grad_norm": 0.0181569941341877,
+      "learning_rate": 0.00019695001616142915,
+      "loss": 0.6596686840057373,
+      "num_tokens": 1541863.0,
+      "step": 55
+    },
+    {
+      "epoch": 0.2624487404803749,
+      "grad_norm": 0.017863502725958824,
+      "learning_rate": 0.00019675746658610917,
+      "loss": 0.8410529494285583,
+      "num_tokens": 1571150.0,
+      "step": 56
+    },
+    {
+      "epoch": 0.2671353251318102,
+      "grad_norm": 0.017747659236192703,
+      "learning_rate": 0.0001965591236576794,
+      "loss": 0.6777825355529785,
+      "num_tokens": 1596156.0,
+      "step": 57
+    },
+    {
+      "epoch": 0.27182190978324544,
+      "grad_norm": 0.018964149057865143,
+      "learning_rate": 0.0001963549992519223,
+      "loss": 0.7582053542137146,
+      "num_tokens": 1620023.0,
+      "step": 58
+    },
+    {
+      "epoch": 0.27650849443468073,
+      "grad_norm": 0.01840493083000183,
+      "learning_rate": 0.00019614510559078625,
+      "loss": 0.7568569183349609,
+      "num_tokens": 1645111.0,
+      "step": 59
+    },
+    {
+      "epoch": 0.281195079086116,
+      "grad_norm": 0.014947458170354366,
+      "learning_rate": 0.00019592945524165374,
+      "loss": 0.7009609341621399,
+      "num_tokens": 1676825.0,
+      "step": 60
+    },
+    {
+      "epoch": 0.28588166373755125,
+      "grad_norm": 0.018854359164834023,
+      "learning_rate": 0.00019570806111658898,
+      "loss": 0.7680045366287231,
+      "num_tokens": 1704692.0,
+      "step": 61
+    },
+    {
+      "epoch": 0.29056824838898654,
+      "grad_norm": 0.018227359279990196,
+      "learning_rate": 0.0001954809364715648,
+      "loss": 0.7715598940849304,
+      "num_tokens": 1731109.0,
+      "step": 62
+    },
+    {
+      "epoch": 0.29525483304042177,
+      "grad_norm": 0.030492570251226425,
+      "learning_rate": 0.00019524809490566877,
+      "loss": 0.7712352275848389,
+      "num_tokens": 1759415.0,
+      "step": 63
+    },
+    {
+      "epoch": 0.29994141769185706,
+      "grad_norm": 0.017521576955914497,
+      "learning_rate": 0.00019500955036028922,
+      "loss": 0.760728120803833,
+      "num_tokens": 1785521.0,
+      "step": 64
+    },
+    {
+      "epoch": 0.30462800234329235,
+      "grad_norm": 0.015448614954948425,
+      "learning_rate": 0.00019476531711828027,
+      "loss": 0.7066532969474792,
+      "num_tokens": 1817775.0,
+      "step": 65
+    },
+    {
+      "epoch": 0.3093145869947276,
+      "grad_norm": 0.017962772399187088,
+      "learning_rate": 0.00019451540980310676,
+      "loss": 0.7092620730400085,
+      "num_tokens": 1842665.0,
+      "step": 66
+    },
+    {
+      "epoch": 0.31400117164616287,
+      "grad_norm": 0.017784444615244865,
+      "learning_rate": 0.0001942598433779687,
+      "loss": 0.7012664079666138,
+      "num_tokens": 1869577.0,
+      "step": 67
+    },
+    {
+      "epoch": 0.3186877562975981,
+      "grad_norm": 0.017005180940032005,
+      "learning_rate": 0.00019399863314490526,
+      "loss": 0.6345712542533875,
+      "num_tokens": 1897675.0,
+      "step": 68
+    },
+    {
+      "epoch": 0.3233743409490334,
+      "grad_norm": 0.01969163678586483,
+      "learning_rate": 0.00019373179474387858,
+      "loss": 0.883948802947998,
+      "num_tokens": 1925451.0,
+      "step": 69
+    },
+    {
+      "epoch": 0.3280609256004687,
+      "grad_norm": 0.015260924585163593,
+      "learning_rate": 0.0001934593441518374,
+      "loss": 0.5999019742012024,
+      "num_tokens": 1955944.0,
+      "step": 70
+    },
+    {
+      "epoch": 0.3327475102519039,
+      "grad_norm": 0.015242267400026321,
+      "learning_rate": 0.00019318129768176032,
+      "loss": 0.7795001268386841,
+      "num_tokens": 1992104.0,
+      "step": 71
+    },
+    {
+      "epoch": 0.3374340949033392,
+      "grad_norm": 0.015679946169257164,
+      "learning_rate": 0.00019289767198167916,
+      "loss": 0.6518503427505493,
+      "num_tokens": 2024006.0,
+      "step": 72
+    },
+    {
+      "epoch": 0.34212067955477443,
+      "grad_norm": 0.016625264659523964,
+      "learning_rate": 0.0001926084840336821,
+      "loss": 0.655728280544281,
+      "num_tokens": 2048339.0,
+      "step": 73
+    },
+    {
+      "epoch": 0.3468072642062097,
+      "grad_norm": 0.016117312014102936,
+      "learning_rate": 0.00019231375115289696,
+      "loss": 0.6389369964599609,
+      "num_tokens": 2079167.0,
+      "step": 74
+    },
+    {
+      "epoch": 0.351493848857645,
+      "grad_norm": 0.015863342210650444,
+      "learning_rate": 0.00019201349098645434,
+      "loss": 0.6088578104972839,
+      "num_tokens": 2110608.0,
+      "step": 75
+    },
+    {
+      "epoch": 0.35618043350908024,
+      "grad_norm": 0.0166738610714674,
+      "learning_rate": 0.00019170772151243107,
+      "loss": 0.6107393503189087,
+      "num_tokens": 2142233.0,
+      "step": 76
+    },
+    {
+      "epoch": 0.36086701816051553,
+      "grad_norm": 0.01740742288529873,
+      "learning_rate": 0.0001913964610387738,
+      "loss": 0.7315651774406433,
+      "num_tokens": 2169376.0,
+      "step": 77
+    },
+    {
+      "epoch": 0.3655536028119508,
+      "grad_norm": 0.021772760897874832,
+      "learning_rate": 0.00019107972820220267,
+      "loss": 0.8029112815856934,
+      "num_tokens": 2189864.0,
+      "step": 78
+    },
+    {
+      "epoch": 0.37024018746338605,
+      "grad_norm": 0.021346885710954666,
+      "learning_rate": 0.00019075754196709572,
+      "loss": 0.7283374667167664,
+      "num_tokens": 2212915.0,
+      "step": 79
+    },
+    {
+      "epoch": 0.37492677211482134,
+      "grad_norm": 0.015978064388036728,
+      "learning_rate": 0.00019042992162435302,
+      "loss": 0.6502755880355835,
+      "num_tokens": 2243293.0,
+      "step": 80
+    },
+    {
+      "epoch": 0.37961335676625657,
+      "grad_norm": 0.018250679597258568,
+      "learning_rate": 0.0001900968867902419,
+      "loss": 0.6920264363288879,
+      "num_tokens": 2271476.0,
+      "step": 81
+    },
+    {
+      "epoch": 0.38429994141769186,
+      "grad_norm": 0.018970279023051262,
+      "learning_rate": 0.00018975845740522244,
+      "loss": 0.8388804197311401,
+      "num_tokens": 2301194.0,
+      "step": 82
+    },
+    {
+      "epoch": 0.38898652606912715,
+      "grad_norm": 0.01729811541736126,
+      "learning_rate": 0.0001894146537327533,
+      "loss": 0.7298757433891296,
+      "num_tokens": 2331066.0,
+      "step": 83
+    },
+    {
+      "epoch": 0.3936731107205624,
+      "grad_norm": 0.016176382079720497,
+      "learning_rate": 0.00018906549635807861,
+      "loss": 0.6759993433952332,
+      "num_tokens": 2364333.0,
+      "step": 84
+    },
+    {
+      "epoch": 0.39835969537199767,
+      "grad_norm": 0.02077021636068821,
+      "learning_rate": 0.00018871100618699554,
+      "loss": 0.6763718128204346,
+      "num_tokens": 2391743.0,
+      "step": 85
+    },
+    {
+      "epoch": 0.4030462800234329,
+      "grad_norm": 0.018082395195961,
+      "learning_rate": 0.0001883512044446023,
+      "loss": 0.5660803318023682,
+      "num_tokens": 2414298.0,
+      "step": 86
+    },
+    {
+      "epoch": 0.4077328646748682,
+      "grad_norm": 0.014369606971740723,
+      "learning_rate": 0.00018798611267402746,
+      "loss": 0.6493896245956421,
+      "num_tokens": 2448533.0,
+      "step": 87
+    },
+    {
+      "epoch": 0.4124194493263035,
+      "grad_norm": 0.01696608029305935,
+      "learning_rate": 0.00018761575273514003,
+      "loss": 0.7302640676498413,
+      "num_tokens": 2475822.0,
+      "step": 88
+    },
+    {
+      "epoch": 0.4171060339777387,
+      "grad_norm": 0.014188491739332676,
+      "learning_rate": 0.00018724014680324057,
+      "loss": 0.6129812002182007,
+      "num_tokens": 2507447.0,
+      "step": 89
+    },
+    {
+      "epoch": 0.421792618629174,
+      "grad_norm": 0.020873507484793663,
+      "learning_rate": 0.0001868593173677335,
+      "loss": 0.7562912106513977,
+      "num_tokens": 2533325.0,
+      "step": 90
+    },
+    {
+      "epoch": 0.42647920328060923,
+      "grad_norm": 0.02266933023929596,
+      "learning_rate": 0.00018647328723078038,
+      "loss": 0.7829717993736267,
+      "num_tokens": 2557554.0,
+      "step": 91
+    },
+    {
+      "epoch": 0.4311657879320445,
+      "grad_norm": 0.01618797518312931,
+      "learning_rate": 0.000186082079505935,
+      "loss": 0.6683343648910522,
+      "num_tokens": 2584256.0,
+      "step": 92
+    },
+    {
+      "epoch": 0.4358523725834798,
+      "grad_norm": 0.01636049523949623,
+      "learning_rate": 0.00018568571761675893,
+      "loss": 0.6867488622665405,
+      "num_tokens": 2611538.0,
+      "step": 93
+    },
+    {
+      "epoch": 0.44053895723491504,
+      "grad_norm": 0.019674455747008324,
+      "learning_rate": 0.00018528422529541952,
+      "loss": 0.7042540311813354,
+      "num_tokens": 2634546.0,
+      "step": 94
+    },
+    {
+      "epoch": 0.44522554188635033,
+      "grad_norm": 0.01726832054555416,
+      "learning_rate": 0.0001848776265812687,
+      "loss": 0.7717908024787903,
+      "num_tokens": 2663498.0,
+      "step": 95
+    },
+    {
+      "epoch": 0.44991212653778556,
+      "grad_norm": 0.019708722829818726,
+      "learning_rate": 0.0001844659458194036,
+      "loss": 0.7058840394020081,
+      "num_tokens": 2690495.0,
+      "step": 96
+    },
+    {
+      "epoch": 0.45459871118922085,
+      "grad_norm": 0.015666253864765167,
+      "learning_rate": 0.00018404920765920896,
+      "loss": 0.6890168190002441,
+      "num_tokens": 2721805.0,
+      "step": 97
+    },
+    {
+      "epoch": 0.45928529584065614,
+      "grad_norm": 0.017277831211686134,
+      "learning_rate": 0.00018362743705288125,
+      "loss": 0.7441021800041199,
+      "num_tokens": 2750865.0,
+      "step": 98
+    },
+    {
+      "epoch": 0.46397188049209137,
+      "grad_norm": 0.019693659618496895,
+      "learning_rate": 0.00018320065925393468,
+      "loss": 0.7525553107261658,
+      "num_tokens": 2774941.0,
+      "step": 99
+    },
+    {
+      "epoch": 0.46865846514352666,
+      "grad_norm": 0.018467402085661888,
+      "learning_rate": 0.00018276889981568906,
+      "loss": 0.7478335499763489,
+      "num_tokens": 2798754.0,
+      "step": 100
+    },
+    {
+      "epoch": 0.47334504979496195,
+      "grad_norm": 0.016140319406986237,
+      "learning_rate": 0.00018233218458973984,
+      "loss": 0.6552085876464844,
+      "num_tokens": 2830956.0,
+      "step": 101
+    },
+    {
+      "epoch": 0.4780316344463972,
+      "grad_norm": 0.01590798981487751,
+      "learning_rate": 0.00018189053972441025,
+      "loss": 0.7423396110534668,
+      "num_tokens": 2862140.0,
+      "step": 102
+    },
+    {
+      "epoch": 0.48271821909783247,
+      "grad_norm": 0.01869652047753334,
+      "learning_rate": 0.00018144399166318572,
+      "loss": 0.7427437901496887,
+      "num_tokens": 2886612.0,
+      "step": 103
+    },
+    {
+      "epoch": 0.4874048037492677,
+      "grad_norm": 0.01759265922009945,
+      "learning_rate": 0.0001809925671431304,
+      "loss": 0.7087360620498657,
+      "num_tokens": 2912450.0,
+      "step": 104
+    },
+    {
+      "epoch": 0.492091388400703,
+      "grad_norm": 0.02047079987823963,
+      "learning_rate": 0.00018053629319328662,
+      "loss": 0.7138317227363586,
+      "num_tokens": 2937623.0,
+      "step": 105
+    },
+    {
+      "epoch": 0.4967779730521383,
+      "grad_norm": 0.017713995650410652,
+      "learning_rate": 0.00018007519713305605,
+      "loss": 0.8792675733566284,
+      "num_tokens": 2966948.0,
+      "step": 106
+    },
+    {
+      "epoch": 0.5014645577035736,
+      "grad_norm": 0.017888452857732773,
+      "learning_rate": 0.00017960930657056438,
+      "loss": 0.7580028176307678,
+      "num_tokens": 2995391.0,
+      "step": 107
+    },
+    {
+      "epoch": 0.5061511423550088,
+      "grad_norm": 0.01546698808670044,
+      "learning_rate": 0.00017913864940100808,
+      "loss": 0.6153715252876282,
+      "num_tokens": 3027800.0,
+      "step": 108
+    },
+    {
+      "epoch": 0.510837727006444,
+      "grad_norm": 0.016936223953962326,
+      "learning_rate": 0.00017866325380498416,
+      "loss": 0.7080944180488586,
+      "num_tokens": 3059479.0,
+      "step": 109
+    },
+    {
+      "epoch": 0.5155243116578794,
+      "grad_norm": 0.014087708666920662,
+      "learning_rate": 0.000178183148246803,
+      "loss": 0.6673498749732971,
+      "num_tokens": 3090276.0,
+      "step": 110
+    },
+    {
+      "epoch": 0.5202108963093146,
+      "grad_norm": 0.01673322543501854,
+      "learning_rate": 0.0001776983614727838,
+      "loss": 0.7345165014266968,
+      "num_tokens": 3117983.0,
+      "step": 111
+    },
+    {
+      "epoch": 0.5248974809607498,
+      "grad_norm": 0.020172668620944023,
+      "learning_rate": 0.00017720892250953373,
+      "loss": 0.7241357564926147,
+      "num_tokens": 3142487.0,
+      "step": 112
+    },
+    {
+      "epoch": 0.5295840656121851,
+      "grad_norm": 0.017690038308501244,
+      "learning_rate": 0.00017671486066220965,
+      "loss": 0.7055312395095825,
+      "num_tokens": 3168339.0,
+      "step": 113
+    },
+    {
+      "epoch": 0.5342706502636204,
+      "grad_norm": 0.02020364999771118,
+      "learning_rate": 0.00017621620551276366,
+      "loss": 0.8319218158721924,
+      "num_tokens": 3188724.0,
+      "step": 114
+    },
+    {
+      "epoch": 0.5389572349150556,
+      "grad_norm": 0.01781732589006424,
+      "learning_rate": 0.00017571298691817177,
+      "loss": 0.6731840372085571,
+      "num_tokens": 3212012.0,
+      "step": 115
+    },
+    {
+      "epoch": 0.5436438195664909,
+      "grad_norm": 0.01930844970047474,
+      "learning_rate": 0.00017520523500864625,
+      "loss": 0.7961735725402832,
+      "num_tokens": 3234866.0,
+      "step": 116
+    },
+    {
+      "epoch": 0.5483304042179262,
+      "grad_norm": 0.01762983947992325,
+      "learning_rate": 0.0001746929801858317,
+      "loss": 0.7620041966438293,
+      "num_tokens": 3263912.0,
+      "step": 117
+    },
+    {
+      "epoch": 0.5530169888693615,
+      "grad_norm": 0.01918371021747589,
+      "learning_rate": 0.00017417625312098452,
+      "loss": 0.7817209362983704,
+      "num_tokens": 3291531.0,
+      "step": 118
+    },
+    {
+      "epoch": 0.5577035735207967,
+      "grad_norm": 0.015562949702143669,
+      "learning_rate": 0.0001736550847531366,
+      "loss": 0.5484626889228821,
+      "num_tokens": 3319717.0,
+      "step": 119
+    },
+    {
+      "epoch": 0.562390158172232,
+      "grad_norm": 0.014110572636127472,
+      "learning_rate": 0.00017312950628724295,
+      "loss": 0.6404852867126465,
+      "num_tokens": 3353800.0,
+      "step": 120
+    },
+    {
+      "epoch": 0.5670767428236673,
+      "grad_norm": 0.01469608023762703,
+      "learning_rate": 0.0001725995491923131,
+      "loss": 0.5291256904602051,
+      "num_tokens": 3383165.0,
+      "step": 121
+    },
+    {
+      "epoch": 0.5717633274751025,
+      "grad_norm": 0.012918651103973389,
+      "learning_rate": 0.00017206524519952697,
+      "loss": 0.6107466220855713,
+      "num_tokens": 3422397.0,
+      "step": 122
+    },
+    {
+      "epoch": 0.5764499121265377,
+      "grad_norm": 0.020249634981155396,
+      "learning_rate": 0.00017152662630033505,
+      "loss": 0.8056011199951172,
+      "num_tokens": 3446864.0,
+      "step": 123
+    },
+    {
+      "epoch": 0.5811364967779731,
+      "grad_norm": 0.01469720620661974,
+      "learning_rate": 0.00017098372474454277,
+      "loss": 0.5424352884292603,
+      "num_tokens": 3480661.0,
+      "step": 124
+    },
+    {
+      "epoch": 0.5858230814294083,
+      "grad_norm": 0.018604544922709465,
+      "learning_rate": 0.00017043657303837963,
+      "loss": 0.7502825260162354,
+      "num_tokens": 3506115.0,
+      "step": 125
+    },
+    {
+      "epoch": 0.5905096660808435,
+      "grad_norm": 0.018334681168198586,
+      "learning_rate": 0.000169885203942553,
+      "loss": 0.8402462005615234,
+      "num_tokens": 3538294.0,
+      "step": 126
+    },
+    {
+      "epoch": 0.5951962507322789,
+      "grad_norm": 0.016648396849632263,
+      "learning_rate": 0.0001693296504702862,
+      "loss": 0.6800752878189087,
+      "num_tokens": 3566829.0,
+      "step": 127
+    },
+    {
+      "epoch": 0.5998828353837141,
+      "grad_norm": 0.01815720461308956,
+      "learning_rate": 0.00016876994588534234,
+      "loss": 0.6151847839355469,
+      "num_tokens": 3593345.0,
+      "step": 128
+    },
+    {
+      "epoch": 0.6045694200351494,
+      "grad_norm": 0.01732209511101246,
+      "learning_rate": 0.00016820612370003221,
+      "loss": 0.6896803975105286,
+      "num_tokens": 3616682.0,
+      "step": 129
+    },
+    {
+      "epoch": 0.6092560046865847,
+      "grad_norm": 0.016574790701270103,
+      "learning_rate": 0.000167638217673208,
+      "loss": 0.6374701261520386,
+      "num_tokens": 3643326.0,
+      "step": 130
+    },
+    {
+      "epoch": 0.6139425893380199,
+      "grad_norm": 0.01585974544286728,
+      "learning_rate": 0.00016706626180824186,
+      "loss": 0.6000485420227051,
+      "num_tokens": 3674755.0,
+      "step": 131
+    },
+    {
+      "epoch": 0.6186291739894552,
+      "grad_norm": 0.02168286219239235,
+      "learning_rate": 0.00016649029035099,
+      "loss": 0.6615947484970093,
+      "num_tokens": 3702146.0,
+      "step": 132
+    },
+    {
+      "epoch": 0.6233157586408905,
+      "grad_norm": 0.01953584887087345,
+      "learning_rate": 0.0001659103377877423,
+      "loss": 0.631389856338501,
+      "num_tokens": 3724254.0,
+      "step": 133
+    },
+    {
+      "epoch": 0.6280023432923257,
+      "grad_norm": 0.01740172505378723,
+      "learning_rate": 0.0001653264388431572,
+      "loss": 0.782500684261322,
+      "num_tokens": 3754889.0,
+      "step": 134
+    },
+    {
+      "epoch": 0.632688927943761,
+      "grad_norm": 0.01756126992404461,
+      "learning_rate": 0.00016473862847818277,
+      "loss": 0.6240105628967285,
+      "num_tokens": 3780049.0,
+      "step": 135
+    },
+    {
+      "epoch": 0.6373755125951962,
+      "grad_norm": 0.018631862476468086,
+      "learning_rate": 0.00016414694188796345,
+      "loss": 0.6254751682281494,
+      "num_tokens": 3805994.0,
+      "step": 136
+    },
+    {
+      "epoch": 0.6420620972466315,
+      "grad_norm": 0.01772165298461914,
+      "learning_rate": 0.00016355141449973256,
+      "loss": 0.7138672471046448,
+      "num_tokens": 3838692.0,
+      "step": 137
+    },
+    {
+      "epoch": 0.6467486818980668,
+      "grad_norm": 0.02134813368320465,
+      "learning_rate": 0.0001629520819706912,
+      "loss": 0.6843606233596802,
+      "num_tokens": 3861136.0,
+      "step": 138
+    },
+    {
+      "epoch": 0.651435266549502,
+      "grad_norm": 0.019024241715669632,
+      "learning_rate": 0.00016234898018587337,
+      "loss": 0.7739150524139404,
+      "num_tokens": 3884299.0,
+      "step": 139
+    },
+    {
+      "epoch": 0.6561218512009374,
+      "grad_norm": 0.017018890008330345,
+      "learning_rate": 0.0001617421452559971,
+      "loss": 0.7982111573219299,
+      "num_tokens": 3911990.0,
+      "step": 140
+    },
+    {
+      "epoch": 0.6608084358523726,
+      "grad_norm": 0.017716465517878532,
+      "learning_rate": 0.0001611316135153026,
+      "loss": 0.6516788005828857,
+      "num_tokens": 3937590.0,
+      "step": 141
+    },
+    {
+      "epoch": 0.6654950205038078,
+      "grad_norm": 0.018196681514382362,
+      "learning_rate": 0.00016051742151937655,
+      "loss": 0.6962698101997375,
+      "num_tokens": 3968492.0,
+      "step": 142
+    },
+    {
+      "epoch": 0.6701816051552432,
+      "grad_norm": 0.019828489050269127,
+      "learning_rate": 0.0001598996060429634,
+      "loss": 0.7837550640106201,
+      "num_tokens": 3992077.0,
+      "step": 143
+    },
+    {
+      "epoch": 0.6748681898066784,
+      "grad_norm": 0.019504865631461143,
+      "learning_rate": 0.00015927820407776353,
+      "loss": 0.8257341384887695,
+      "num_tokens": 4017854.0,
+      "step": 144
+    },
+    {
+      "epoch": 0.6795547744581136,
+      "grad_norm": 0.024247588589787483,
+      "learning_rate": 0.0001586532528302183,
+      "loss": 0.7519227266311646,
+      "num_tokens": 4040498.0,
+      "step": 145
+    },
+    {
+      "epoch": 0.6842413591095489,
+      "grad_norm": 0.017198674380779266,
+      "learning_rate": 0.00015802478971928242,
+      "loss": 0.6746048331260681,
+      "num_tokens": 4064754.0,
+      "step": 146
+    },
+    {
+      "epoch": 0.6889279437609842,
+      "grad_norm": 0.018579604104161263,
+      "learning_rate": 0.0001573928523741832,
+      "loss": 0.8102442622184753,
+      "num_tokens": 4088447.0,
+      "step": 147
+    },
+    {
+      "epoch": 0.6936145284124194,
+      "grad_norm": 0.017206164076924324,
+      "learning_rate": 0.00015675747863216801,
+      "loss": 0.6687013506889343,
+      "num_tokens": 4118049.0,
+      "step": 148
+    },
+    {
+      "epoch": 0.6983011130638547,
+      "grad_norm": 0.018775764852762222,
+      "learning_rate": 0.00015611870653623825,
+      "loss": 0.7222660183906555,
+      "num_tokens": 4141820.0,
+      "step": 149
+    },
+    {
+      "epoch": 0.70298769771529,
+      "grad_norm": 0.01597282849252224,
+      "learning_rate": 0.00015547657433287183,
+      "loss": 0.7305333018302917,
+      "num_tokens": 4171467.0,
+      "step": 150
+    },
+    {
+      "epoch": 0.7076742823667252,
+      "grad_norm": 0.014281094074249268,
+      "learning_rate": 0.0001548311204697331,
+      "loss": 0.6703628301620483,
+      "num_tokens": 4210286.0,
+      "step": 151
+    },
+    {
+      "epoch": 0.7123608670181605,
+      "grad_norm": 0.021154943853616714,
+      "learning_rate": 0.00015418238359337077,
+      "loss": 0.6685652732849121,
+      "num_tokens": 4234631.0,
+      "step": 152
+    },
+    {
+      "epoch": 0.7170474516695958,
+      "grad_norm": 0.018537605181336403,
+      "learning_rate": 0.00015353040254690393,
+      "loss": 0.6078515648841858,
+      "num_tokens": 4260789.0,
+      "step": 153
+    },
+    {
+      "epoch": 0.7217340363210311,
+      "grad_norm": 0.01671479269862175,
+      "learning_rate": 0.0001528752163676964,
+      "loss": 0.7512734532356262,
+      "num_tokens": 4291494.0,
+      "step": 154
+    },
+    {
+      "epoch": 0.7264206209724663,
+      "grad_norm": 0.015522940084338188,
+      "learning_rate": 0.00015221686428501928,
+      "loss": 0.6290964484214783,
+      "num_tokens": 4321851.0,
+      "step": 155
+    },
+    {
+      "epoch": 0.7311072056239016,
+      "grad_norm": 0.018737254664301872,
+      "learning_rate": 0.00015155538571770218,
+      "loss": 0.6476538777351379,
+      "num_tokens": 4345682.0,
+      "step": 156
+    },
+    {
+      "epoch": 0.7357937902753369,
+      "grad_norm": 0.016887914389371872,
+      "learning_rate": 0.0001508908202717729,
+      "loss": 0.7374083995819092,
+      "num_tokens": 4377398.0,
+      "step": 157
+    },
+    {
+      "epoch": 0.7404803749267721,
+      "grad_norm": 0.019042199477553368,
+      "learning_rate": 0.00015022320773808612,
+      "loss": 0.727727472782135,
+      "num_tokens": 4407190.0,
+      "step": 158
+    },
+    {
+      "epoch": 0.7451669595782073,
+      "grad_norm": 0.015718845650553703,
+      "learning_rate": 0.00014955258808994096,
+      "loss": 0.7678956389427185,
+      "num_tokens": 4443267.0,
+      "step": 159
+    },
+    {
+      "epoch": 0.7498535442296427,
+      "grad_norm": 0.01766720600426197,
+      "learning_rate": 0.00014887900148068735,
+      "loss": 0.6393877267837524,
+      "num_tokens": 4471233.0,
+      "step": 160
+    },
+    {
+      "epoch": 0.7545401288810779,
+      "grad_norm": 0.01771809719502926,
+      "learning_rate": 0.0001482024882413222,
+      "loss": 0.7477349042892456,
+      "num_tokens": 4498609.0,
+      "step": 161
+    },
+    {
+      "epoch": 0.7592267135325131,
+      "grad_norm": 0.01701246201992035,
+      "learning_rate": 0.00014752308887807427,
+      "loss": 0.6386726498603821,
+      "num_tokens": 4523329.0,
+      "step": 162
+    },
+    {
+      "epoch": 0.7639132981839485,
+      "grad_norm": 0.02258412167429924,
+      "learning_rate": 0.00014684084406997903,
+      "loss": 0.751873254776001,
+      "num_tokens": 4546858.0,
+      "step": 163
+    },
+    {
+      "epoch": 0.7685998828353837,
+      "grad_norm": 0.018019448965787888,
+      "learning_rate": 0.00014615579466644292,
+      "loss": 0.6369242072105408,
+      "num_tokens": 4573004.0,
+      "step": 164
+    },
+    {
+      "epoch": 0.773286467486819,
+      "grad_norm": 0.015171061269938946,
+      "learning_rate": 0.00014546798168479756,
+      "loss": 0.6253129243850708,
+      "num_tokens": 4600141.0,
+      "step": 165
+    },
+    {
+      "epoch": 0.7779730521382543,
+      "grad_norm": 0.01836264133453369,
+      "learning_rate": 0.00014477744630784378,
+      "loss": 0.6729415655136108,
+      "num_tokens": 4622472.0,
+      "step": 166
+    },
+    {
+      "epoch": 0.7826596367896895,
+      "grad_norm": 0.016997816041111946,
+      "learning_rate": 0.00014408422988138584,
+      "loss": 0.7881853580474854,
+      "num_tokens": 4654084.0,
+      "step": 167
+    },
+    {
+      "epoch": 0.7873462214411248,
+      "grad_norm": 0.017668506130576134,
+      "learning_rate": 0.00014338837391175582,
+      "loss": 0.6245524287223816,
+      "num_tokens": 4678962.0,
+      "step": 168
+    },
+    {
+      "epoch": 0.79203280609256,
+      "grad_norm": 0.01410279143601656,
+      "learning_rate": 0.00014268992006332846,
+      "loss": 0.6286853551864624,
+      "num_tokens": 4714998.0,
+      "step": 169
+    },
+    {
+      "epoch": 0.7967193907439953,
+      "grad_norm": 0.016260406002402306,
+      "learning_rate": 0.00014198891015602646,
+      "loss": 0.5806441307067871,
+      "num_tokens": 4744061.0,
+      "step": 170
+    },
+    {
+      "epoch": 0.8014059753954306,
+      "grad_norm": 0.016793547198176384,
+      "learning_rate": 0.0001412853861628166,
+      "loss": 0.6900240778923035,
+      "num_tokens": 4771394.0,
+      "step": 171
+    },
+    {
+      "epoch": 0.8060925600468658,
+      "grad_norm": 0.01554949302226305,
+      "learning_rate": 0.0001405793902071964,
+      "loss": 0.5430771708488464,
+      "num_tokens": 4797695.0,
+      "step": 172
+    },
+    {
+      "epoch": 0.8107791446983011,
+      "grad_norm": 0.021834636107087135,
+      "learning_rate": 0.00013987096456067236,
+      "loss": 0.7372509837150574,
+      "num_tokens": 4818473.0,
+      "step": 173
+    },
+    {
+      "epoch": 0.8154657293497364,
+      "grad_norm": 0.017153380438685417,
+      "learning_rate": 0.00013916015164022852,
+      "loss": 0.7403316497802734,
+      "num_tokens": 4850247.0,
+      "step": 174
+    },
+    {
+      "epoch": 0.8201523140011716,
+      "grad_norm": 0.017557591199874878,
+      "learning_rate": 0.00013844699400578696,
+      "loss": 0.5496473908424377,
+      "num_tokens": 4875104.0,
+      "step": 175
+    },
+    {
+      "epoch": 0.824838898652607,
+      "grad_norm": 0.018106888979673386,
+      "learning_rate": 0.00013773153435765964,
+      "loss": 0.656714677810669,
+      "num_tokens": 4902439.0,
+      "step": 176
+    },
+    {
+      "epoch": 0.8295254833040422,
+      "grad_norm": 0.014749976806342602,
+      "learning_rate": 0.00013701381553399145,
+      "loss": 0.514419674873352,
+      "num_tokens": 4934804.0,
+      "step": 177
+    },
+    {
+      "epoch": 0.8342120679554774,
+      "grad_norm": 0.015780452638864517,
+      "learning_rate": 0.00013629388050819547,
+      "loss": 0.6916260719299316,
+      "num_tokens": 4964820.0,
+      "step": 178
+    },
+    {
+      "epoch": 0.8388986526069128,
+      "grad_norm": 0.02181050553917885,
+      "learning_rate": 0.00013557177238637986,
+      "loss": 0.789571225643158,
+      "num_tokens": 4987774.0,
+      "step": 179
+    },
+    {
+      "epoch": 0.843585237258348,
+      "grad_norm": 0.019028659909963608,
+      "learning_rate": 0.00013484753440476692,
+      "loss": 0.7827271819114685,
+      "num_tokens": 5012356.0,
+      "step": 180
+    },
+    {
+      "epoch": 0.8482718219097832,
+      "grad_norm": 0.018884286284446716,
+      "learning_rate": 0.00013412120992710425,
+      "loss": 0.7337446808815002,
+      "num_tokens": 5039247.0,
+      "step": 181
+    },
+    {
+      "epoch": 0.8529584065612185,
+      "grad_norm": 0.015975872054696083,
+      "learning_rate": 0.00013339284244206847,
+      "loss": 0.6652353405952454,
+      "num_tokens": 5069912.0,
+      "step": 182
+    },
+    {
+      "epoch": 0.8576449912126538,
+      "grad_norm": 0.019088014960289,
+      "learning_rate": 0.00013266247556066122,
+      "loss": 0.6951537132263184,
+      "num_tokens": 5098146.0,
+      "step": 183
+    },
+    {
+      "epoch": 0.862331575864089,
+      "grad_norm": 0.014904006384313107,
+      "learning_rate": 0.000131930153013598,
+      "loss": 0.5959742069244385,
+      "num_tokens": 5128454.0,
+      "step": 184
+    },
+    {
+      "epoch": 0.8670181605155243,
+      "grad_norm": 0.01810140162706375,
+      "learning_rate": 0.0001311959186486898,
+      "loss": 0.7200735807418823,
+      "num_tokens": 5154213.0,
+      "step": 185
+    },
+    {
+      "epoch": 0.8717047451669596,
+      "grad_norm": 0.019789377227425575,
+      "learning_rate": 0.0001304598164282176,
+      "loss": 0.6804766654968262,
+      "num_tokens": 5175623.0,
+      "step": 186
+    },
+    {
+      "epoch": 0.8763913298183948,
+      "grad_norm": 0.017321709543466568,
+      "learning_rate": 0.00012972189042630044,
+      "loss": 0.7105655670166016,
+      "num_tokens": 5201293.0,
+      "step": 187
+    },
+    {
+      "epoch": 0.8810779144698301,
+      "grad_norm": 0.022706087678670883,
+      "learning_rate": 0.00012898218482625606,
+      "loss": 0.7062239646911621,
+      "num_tokens": 5227922.0,
+      "step": 188
+    },
+    {
+      "epoch": 0.8857644991212654,
+      "grad_norm": 0.014490657486021519,
+      "learning_rate": 0.0001282407439179557,
+      "loss": 0.6353857517242432,
+      "num_tokens": 5260546.0,
+      "step": 189
+    },
+    {
+      "epoch": 0.8904510837727007,
+      "grad_norm": 0.01803654059767723,
+      "learning_rate": 0.0001274976120951723,
+      "loss": 0.6230901479721069,
+      "num_tokens": 5288394.0,
+      "step": 190
+    },
+    {
+      "epoch": 0.8951376684241359,
+      "grad_norm": 0.023029128089547157,
+      "learning_rate": 0.00012675283385292212,
+      "loss": 0.6824889183044434,
+      "num_tokens": 5313596.0,
+      "step": 191
+    },
+    {
+      "epoch": 0.8998242530755711,
+      "grad_norm": 0.02031250298023224,
+      "learning_rate": 0.00012600645378480082,
+      "loss": 0.7509975433349609,
+      "num_tokens": 5340851.0,
+      "step": 192
+    },
+    {
+      "epoch": 0.9045108377270065,
+      "grad_norm": 0.01629417948424816,
+      "learning_rate": 0.00012525851658031352,
+      "loss": 0.6494298577308655,
+      "num_tokens": 5372120.0,
+      "step": 193
+    },
+    {
+      "epoch": 0.9091974223784417,
+      "grad_norm": 0.01852346956729889,
+      "learning_rate": 0.0001245090670221987,
+      "loss": 0.615811288356781,
+      "num_tokens": 5400573.0,
+      "step": 194
+    },
+    {
+      "epoch": 0.9138840070298769,
+      "grad_norm": 0.0194899570196867,
+      "learning_rate": 0.00012375814998374712,
+      "loss": 0.8159194588661194,
+      "num_tokens": 5424108.0,
+      "step": 195
+    },
+    {
+      "epoch": 0.9185705916813123,
+      "grad_norm": 0.017043786123394966,
+      "learning_rate": 0.00012300581042611492,
+      "loss": 0.6191376447677612,
+      "num_tokens": 5448551.0,
+      "step": 196
+    },
+    {
+      "epoch": 0.9232571763327475,
+      "grad_norm": 0.01926240883767605,
+      "learning_rate": 0.00012225209339563145,
+      "loss": 0.7384488582611084,
+      "num_tokens": 5475526.0,
+      "step": 197
+    },
+    {
+      "epoch": 0.9279437609841827,
+      "grad_norm": 0.020789049565792084,
+      "learning_rate": 0.00012149704402110243,
+      "loss": 0.7020618915557861,
+      "num_tokens": 5502611.0,
+      "step": 198
+    },
+    {
+      "epoch": 0.9326303456356181,
+      "grad_norm": 0.01582273840904236,
+      "learning_rate": 0.00012074070751110751,
+      "loss": 0.6311229467391968,
+      "num_tokens": 5529480.0,
+      "step": 199
+    },
+    {
+      "epoch": 0.9373169302870533,
+      "grad_norm": 0.02086408995091915,
+      "learning_rate": 0.00011998312915129371,
+      "loss": 0.7435621619224548,
+      "num_tokens": 5553458.0,
+      "step": 200
+    },
+    {
+      "epoch": 0.9420035149384886,
+      "grad_norm": 0.01609129272401333,
+      "learning_rate": 0.0001192243543016637,
+      "loss": 0.6395267844200134,
+      "num_tokens": 5584758.0,
+      "step": 201
+    },
+    {
+      "epoch": 0.9466900995899239,
+      "grad_norm": 0.01869116723537445,
+      "learning_rate": 0.00011846442839386003,
+      "loss": 0.6983841061592102,
+      "num_tokens": 5615997.0,
+      "step": 202
+    },
+    {
+      "epoch": 0.9513766842413591,
+      "grad_norm": 0.03048761747777462,
+      "learning_rate": 0.00011770339692844483,
+      "loss": 0.6789813041687012,
+      "num_tokens": 5640891.0,
+      "step": 203
+    },
+    {
+      "epoch": 0.9560632688927944,
+      "grad_norm": 0.01786402240395546,
+      "learning_rate": 0.00011694130547217554,
+      "loss": 0.75504469871521,
+      "num_tokens": 5670686.0,
+      "step": 204
+    },
+    {
+      "epoch": 0.9607498535442296,
+      "grad_norm": 0.01641128584742546,
+      "learning_rate": 0.0001161781996552765,
+      "loss": 0.5403404235839844,
+      "num_tokens": 5699611.0,
+      "step": 205
+    },
+    {
+      "epoch": 0.9654364381956649,
+      "grad_norm": 0.01727338694036007,
+      "learning_rate": 0.00011541412516870684,
+      "loss": 0.6894916296005249,
+      "num_tokens": 5729248.0,
+      "step": 206
+    },
+    {
+      "epoch": 0.9701230228471002,
+      "grad_norm": 0.01573982834815979,
+      "learning_rate": 0.00011464912776142494,
+      "loss": 0.5996260046958923,
+      "num_tokens": 5757744.0,
+      "step": 207
+    },
+    {
+      "epoch": 0.9748096074985354,
+      "grad_norm": 0.017988774925470352,
+      "learning_rate": 0.00011388325323764888,
+      "loss": 0.6654211282730103,
+      "num_tokens": 5788566.0,
+      "step": 208
+    },
+    {
+      "epoch": 0.9794961921499707,
+      "grad_norm": 0.014979843981564045,
+      "learning_rate": 0.00011311654745411425,
+      "loss": 0.6495011448860168,
+      "num_tokens": 5824022.0,
+      "step": 209
+    },
+    {
+      "epoch": 0.984182776801406,
+      "grad_norm": 0.01659063622355461,
+      "learning_rate": 0.00011234905631732819,
+      "loss": 0.6396357417106628,
+      "num_tokens": 5853973.0,
+      "step": 210
+    },
+    {
+      "epoch": 0.9888693614528412,
+      "grad_norm": 0.016888733953237534,
+      "learning_rate": 0.00011158082578082089,
+      "loss": 0.6784655451774597,
+      "num_tokens": 5883848.0,
+      "step": 211
+    },
+    {
+      "epoch": 0.9935559461042766,
+      "grad_norm": 0.01959056593477726,
+      "learning_rate": 0.00011081190184239419,
+      "loss": 0.6131283044815063,
+      "num_tokens": 5907061.0,
+      "step": 212
+    },
+    {
+      "epoch": 0.9982425307557118,
+      "grad_norm": 0.016141943633556366,
+      "learning_rate": 0.00011004233054136725,
+      "loss": 0.5719535946846008,
+      "num_tokens": 5937101.0,
+      "step": 213
+    },
+    {
+      "epoch": 1.0,
+      "grad_norm": 0.025195077061653137,
+      "learning_rate": 0.00010927215795582012,
+      "loss": 0.5020159482955933,
+      "num_tokens": 5947501.0,
+      "step": 214
+    },
+    {
+      "epoch": 1.0046865846514352,
+      "grad_norm": 0.01583407074213028,
+      "learning_rate": 0.00010850143019983474,
+      "loss": 0.4937314987182617,
+      "num_tokens": 5970493.0,
+      "step": 215
+    },
+    {
+      "epoch": 1.0093731693028705,
+      "grad_norm": 0.017454253509640694,
+      "learning_rate": 0.0001077301934207339,
+      "loss": 0.6226227879524231,
+      "num_tokens": 6003077.0,
+      "step": 216
+    },
+    {
+      "epoch": 1.0140597539543057,
+      "grad_norm": 0.014966587536036968,
+      "learning_rate": 0.00010695849379631813,
+      "loss": 0.5236163139343262,
+      "num_tokens": 6033093.0,
+      "step": 217
+    },
+    {
+      "epoch": 1.0187463386057412,
+      "grad_norm": 0.015618878416717052,
+      "learning_rate": 0.00010618637753210085,
+      "loss": 0.5199332237243652,
+      "num_tokens": 6060819.0,
+      "step": 218
+    },
+    {
+      "epoch": 1.0234329232571764,
+      "grad_norm": 0.01610647886991501,
+      "learning_rate": 0.00010541389085854176,
+      "loss": 0.5474415421485901,
+      "num_tokens": 6091049.0,
+      "step": 219
+    },
+    {
+      "epoch": 1.0281195079086116,
+      "grad_norm": 0.015623976476490498,
+      "learning_rate": 0.00010464108002827882,
+      "loss": 0.48489633202552795,
+      "num_tokens": 6122790.0,
+      "step": 220
+    },
+    {
+      "epoch": 1.0328060925600469,
+      "grad_norm": 0.015523376874625683,
+      "learning_rate": 0.00010386799131335889,
+      "loss": 0.5330443382263184,
+      "num_tokens": 6157043.0,
+      "step": 221
+    },
+    {
+      "epoch": 1.037492677211482,
+      "grad_norm": 0.017354318872094154,
+      "learning_rate": 0.00010309467100246713,
+      "loss": 0.49863070249557495,
+      "num_tokens": 6182364.0,
+      "step": 222
+    },
+    {
+      "epoch": 1.0421792618629173,
+      "grad_norm": 0.014648743905127048,
+      "learning_rate": 0.00010232116539815558,
+      "loss": 0.5403507947921753,
+      "num_tokens": 6217015.0,
+      "step": 223
+    },
+    {
+      "epoch": 1.0468658465143528,
+      "grad_norm": 0.017757907509803772,
+      "learning_rate": 0.00010154752081407066,
+      "loss": 0.5948547720909119,
+      "num_tokens": 6246427.0,
+      "step": 224
+    },
+    {
+      "epoch": 1.051552431165788,
+      "grad_norm": 0.03653530776500702,
+      "learning_rate": 0.00010077378357218021,
+      "loss": 0.5761139392852783,
+      "num_tokens": 6275025.0,
+      "step": 225
+    },
+    {
+      "epoch": 1.0562390158172232,
+      "grad_norm": 0.016985690221190453,
+      "learning_rate": 0.0001,
+      "loss": 0.539291501045227,
+      "num_tokens": 6304885.0,
+      "step": 226
+    },
+    {
+      "epoch": 1.0609256004686585,
+      "grad_norm": 0.017139555886387825,
+      "learning_rate": 9.92262164278198e-05,
+      "loss": 0.48060083389282227,
+      "num_tokens": 6331616.0,
+      "step": 227
+    },
+    {
+      "epoch": 1.0656121851200937,
+      "grad_norm": 0.015508485957980156,
+      "learning_rate": 9.845247918592937e-05,
+      "loss": 0.4607234299182892,
+      "num_tokens": 6361325.0,
+      "step": 228
+    },
+    {
+      "epoch": 1.070298769771529,
+      "grad_norm": 0.01898042857646942,
+      "learning_rate": 9.767883460184443e-05,
+      "loss": 0.5976904034614563,
+      "num_tokens": 6387046.0,
+      "step": 229
+    },
+    {
+      "epoch": 1.0749853544229642,
+      "grad_norm": 0.01673644594848156,
+      "learning_rate": 9.69053289975329e-05,
+      "loss": 0.49898090958595276,
+      "num_tokens": 6417864.0,
+      "step": 230
+    },
+    {
+      "epoch": 1.0796719390743996,
+      "grad_norm": 0.020018402487039566,
+      "learning_rate": 9.613200868664112e-05,
+      "loss": 0.5435779094696045,
+      "num_tokens": 6444931.0,
+      "step": 231
+    },
+    {
+      "epoch": 1.0843585237258349,
+      "grad_norm": 0.021194079890847206,
+      "learning_rate": 9.53589199717212e-05,
+      "loss": 0.6362026929855347,
+      "num_tokens": 6475713.0,
+      "step": 232
+    },
+    {
+      "epoch": 1.08904510837727,
+      "grad_norm": 0.017888659611344337,
+      "learning_rate": 9.458610914145826e-05,
+      "loss": 0.5617181658744812,
+      "num_tokens": 6504178.0,
+      "step": 233
+    },
+    {
+      "epoch": 1.0937316930287053,
+      "grad_norm": 0.020780451595783234,
+      "learning_rate": 9.381362246789917e-05,
+      "loss": 0.49862992763519287,
+      "num_tokens": 6529543.0,
+      "step": 234
+    },
+    {
+      "epoch": 1.0984182776801406,
+      "grad_norm": 0.01968269981443882,
+      "learning_rate": 9.304150620368188e-05,
+      "loss": 0.6710590124130249,
+      "num_tokens": 6557912.0,
+      "step": 235
+    },
+    {
+      "epoch": 1.1031048623315758,
+      "grad_norm": 0.01871589943766594,
+      "learning_rate": 9.226980657926614e-05,
+      "loss": 0.44637927412986755,
+      "num_tokens": 6582586.0,
+      "step": 236
+    },
+    {
+      "epoch": 1.1077914469830112,
+      "grad_norm": 0.021373949944972992,
+      "learning_rate": 9.149856980016529e-05,
+      "loss": 0.5362990498542786,
+      "num_tokens": 6605854.0,
+      "step": 237
+    },
+    {
+      "epoch": 1.1124780316344465,
+      "grad_norm": 0.018750222399830818,
+      "learning_rate": 9.072784204417995e-05,
+      "loss": 0.543184757232666,
+      "num_tokens": 6633003.0,
+      "step": 238
+    },
+    {
+      "epoch": 1.1171646162858817,
+      "grad_norm": 0.01677524298429489,
+      "learning_rate": 8.995766945863277e-05,
+      "loss": 0.5202764868736267,
+      "num_tokens": 6665844.0,
+      "step": 239
+    },
+    {
+      "epoch": 1.121851200937317,
+      "grad_norm": 0.01806574873626232,
+      "learning_rate": 8.918809815760585e-05,
+      "loss": 0.5611273646354675,
+      "num_tokens": 6698052.0,
+      "step": 240
+    },
+    {
+      "epoch": 1.1265377855887522,
+      "grad_norm": 0.022728972136974335,
+      "learning_rate": 8.841917421917912e-05,
+      "loss": 0.5777214765548706,
+      "num_tokens": 6719727.0,
+      "step": 241
+    },
+    {
+      "epoch": 1.1312243702401874,
+      "grad_norm": 0.018225179985165596,
+      "learning_rate": 8.765094368267186e-05,
+      "loss": 0.4442354440689087,
+      "num_tokens": 6745434.0,
+      "step": 242
+    },
+    {
+      "epoch": 1.1359109548916226,
+      "grad_norm": 0.020616615191102028,
+      "learning_rate": 8.688345254588578e-05,
+      "loss": 0.5914826989173889,
+      "num_tokens": 6773920.0,
+      "step": 243
+    },
+    {
+      "epoch": 1.140597539543058,
+      "grad_norm": 0.018396707251667976,
+      "learning_rate": 8.611674676235115e-05,
+      "loss": 0.4618031084537506,
+      "num_tokens": 6800391.0,
+      "step": 244
+    },
+    {
+      "epoch": 1.1452841241944933,
+      "grad_norm": 0.016975924372673035,
+      "learning_rate": 8.535087223857508e-05,
+      "loss": 0.5435763001441956,
+      "num_tokens": 6832207.0,
+      "step": 245
+    },
+    {
+      "epoch": 1.1499707088459286,
+      "grad_norm": 0.015872633084654808,
+      "learning_rate": 8.458587483129316e-05,
+      "loss": 0.471432626247406,
+      "num_tokens": 6864377.0,
+      "step": 246
+    },
+    {
+      "epoch": 1.1546572934973638,
+      "grad_norm": 0.020818166434764862,
+      "learning_rate": 8.382180034472353e-05,
+      "loss": 0.5589307546615601,
+      "num_tokens": 6890689.0,
+      "step": 247
+    },
+    {
+      "epoch": 1.159343878148799,
+      "grad_norm": 0.020436430349946022,
+      "learning_rate": 8.305869452782446e-05,
+      "loss": 0.5633253455162048,
+      "num_tokens": 6916605.0,
+      "step": 248
+    },
+    {
+      "epoch": 1.1640304628002343,
+      "grad_norm": 0.021504931151866913,
+      "learning_rate": 8.229660307155518e-05,
+      "loss": 0.5318947434425354,
+      "num_tokens": 6941217.0,
+      "step": 249
+    },
+    {
+      "epoch": 1.1687170474516697,
+      "grad_norm": 0.015647897496819496,
+      "learning_rate": 8.153557160613998e-05,
+      "loss": 0.41660377383232117,
+      "num_tokens": 6971845.0,
+      "step": 250
+    },
+    {
+      "epoch": 1.173403632103105,
+      "grad_norm": 0.01813015528023243,
+      "learning_rate": 8.077564569833632e-05,
+      "loss": 0.4969935417175293,
+      "num_tokens": 7000145.0,
+      "step": 251
+    },
+    {
+      "epoch": 1.1780902167545402,
+      "grad_norm": 0.02095090039074421,
+      "learning_rate": 8.00168708487063e-05,
+      "loss": 0.5970808267593384,
+      "num_tokens": 7028674.0,
+      "step": 252
+    },
+    {
+      "epoch": 1.1827768014059754,
+      "grad_norm": 0.01675095222890377,
+      "learning_rate": 7.92592924888925e-05,
+      "loss": 0.5089432001113892,
+      "num_tokens": 7057684.0,
+      "step": 253
+    },
+    {
+      "epoch": 1.1874633860574106,
+      "grad_norm": 0.018853550776839256,
+      "learning_rate": 7.85029559788976e-05,
+      "loss": 0.6490843892097473,
+      "num_tokens": 7088398.0,
+      "step": 254
+    },
+    {
+      "epoch": 1.1921499707088459,
+      "grad_norm": 0.01821407489478588,
+      "learning_rate": 7.774790660436858e-05,
+      "loss": 0.5658373236656189,
+      "num_tokens": 7119190.0,
+      "step": 255
+    },
+    {
+      "epoch": 1.196836555360281,
+      "grad_norm": 0.017882108688354492,
+      "learning_rate": 7.699418957388512e-05,
+      "loss": 0.5955532789230347,
+      "num_tokens": 7153203.0,
+      "step": 256
+    },
+    {
+      "epoch": 1.2015231400117163,
+      "grad_norm": 0.017674054950475693,
+      "learning_rate": 7.624185001625292e-05,
+      "loss": 0.5457881689071655,
+      "num_tokens": 7184609.0,
+      "step": 257
+    },
+    {
+      "epoch": 1.2062097246631518,
+      "grad_norm": 0.01938200183212757,
+      "learning_rate": 7.549093297780132e-05,
+      "loss": 0.5812852382659912,
+      "num_tokens": 7212942.0,
+      "step": 258
+    },
+    {
+      "epoch": 1.210896309314587,
+      "grad_norm": 0.021312085911631584,
+      "learning_rate": 7.474148341968652e-05,
+      "loss": 0.5053731203079224,
+      "num_tokens": 7235722.0,
+      "step": 259
+    },
+    {
+      "epoch": 1.2155828939660223,
+      "grad_norm": 0.015914585441350937,
+      "learning_rate": 7.39935462151992e-05,
+      "loss": 0.4808054566383362,
+      "num_tokens": 7268853.0,
+      "step": 260
+    },
+    {
+      "epoch": 1.2202694786174575,
+      "grad_norm": 0.016284095123410225,
+      "learning_rate": 7.324716614707793e-05,
+      "loss": 0.5059459209442139,
+      "num_tokens": 7301735.0,
+      "step": 261
+    },
+    {
+      "epoch": 1.2249560632688927,
+      "grad_norm": 0.02114776149392128,
+      "learning_rate": 7.250238790482773e-05,
+      "loss": 0.5616275072097778,
+      "num_tokens": 7326126.0,
+      "step": 262
+    },
+    {
+      "epoch": 1.2296426479203282,
+      "grad_norm": 0.018290745094418526,
+      "learning_rate": 7.175925608204428e-05,
+      "loss": 0.4677993357181549,
+      "num_tokens": 7353023.0,
+      "step": 263
+    },
+    {
+      "epoch": 1.2343292325717634,
+      "grad_norm": 0.018801284953951836,
+      "learning_rate": 7.101781517374398e-05,
+      "loss": 0.495519757270813,
+      "num_tokens": 7378830.0,
+      "step": 264
+    },
+    {
+      "epoch": 1.2390158172231986,
+      "grad_norm": 0.01867852732539177,
+      "learning_rate": 7.027810957369957e-05,
+      "loss": 0.6183238625526428,
+      "num_tokens": 7409846.0,
+      "step": 265
+    },
+    {
+      "epoch": 1.2437024018746339,
+      "grad_norm": 0.021188955754041672,
+      "learning_rate": 6.954018357178241e-05,
+      "loss": 0.5982891917228699,
+      "num_tokens": 7435647.0,
+      "step": 266
+    },
+    {
+      "epoch": 1.248388986526069,
+      "grad_norm": 0.016743872314691544,
+      "learning_rate": 6.880408135131022e-05,
+      "loss": 0.4980001151561737,
+      "num_tokens": 7466694.0,
+      "step": 267
+    },
+    {
+      "epoch": 1.2530755711775043,
+      "grad_norm": 0.019149333238601685,
+      "learning_rate": 6.806984698640202e-05,
+      "loss": 0.6516995429992676,
+      "num_tokens": 7496653.0,
+      "step": 268
+    },
+    {
+      "epoch": 1.2577621558289396,
+      "grad_norm": 0.019356200471520424,
+      "learning_rate": 6.733752443933878e-05,
+      "loss": 0.5884249210357666,
+      "num_tokens": 7523023.0,
+      "step": 269
+    },
+    {
+      "epoch": 1.2624487404803748,
+      "grad_norm": 0.02171611227095127,
+      "learning_rate": 6.660715755793154e-05,
+      "loss": 0.49592259526252747,
+      "num_tokens": 7542791.0,
+      "step": 270
+    },
+    {
+      "epoch": 1.2671353251318103,
+      "grad_norm": 0.017002740874886513,
+      "learning_rate": 6.587879007289576e-05,
+      "loss": 0.5243790149688721,
+      "num_tokens": 7577560.0,
+      "step": 271
+    },
+    {
+      "epoch": 1.2718219097832455,
+      "grad_norm": 0.020400408655405045,
+      "learning_rate": 6.515246559523312e-05,
+      "loss": 0.5987250804901123,
+      "num_tokens": 7603100.0,
+      "step": 272
+    },
+    {
+      "epoch": 1.2765084944346807,
+      "grad_norm": 0.019193653017282486,
+      "learning_rate": 6.442822761362015e-05,
+      "loss": 0.6312786936759949,
+      "num_tokens": 7630292.0,
+      "step": 273
+    },
+    {
+      "epoch": 1.281195079086116,
+      "grad_norm": 0.01957247033715248,
+      "learning_rate": 6.370611949180457e-05,
+      "loss": 0.5242177844047546,
+      "num_tokens": 7655668.0,
+      "step": 274
+    },
+    {
+      "epoch": 1.2858816637375512,
+      "grad_norm": 0.017094500362873077,
+      "learning_rate": 6.298618446600856e-05,
+      "loss": 0.4949776828289032,
+      "num_tokens": 7684925.0,
+      "step": 275
+    },
+    {
+      "epoch": 1.2905682483889866,
+      "grad_norm": 0.02443353645503521,
+      "learning_rate": 6.22684656423404e-05,
+      "loss": 0.5696538090705872,
+      "num_tokens": 7719008.0,
+      "step": 276
+    },
+    {
+      "epoch": 1.2952548330404219,
+      "grad_norm": 0.057685527950525284,
+      "learning_rate": 6.155300599421306e-05,
+      "loss": 0.5062693953514099,
+      "num_tokens": 7751344.0,
+      "step": 277
+    },
+    {
+      "epoch": 1.2999414176918571,
+      "grad_norm": 0.02328214980661869,
+      "learning_rate": 6.0839848359771536e-05,
+      "loss": 0.5536410212516785,
+      "num_tokens": 7777488.0,
+      "step": 278
+    },
+    {
+      "epoch": 1.3046280023432923,
+      "grad_norm": 0.02110743522644043,
+      "learning_rate": 6.012903543932766e-05,
+      "loss": 0.6596872806549072,
+      "num_tokens": 7805208.0,
+      "step": 279
+    },
+    {
+      "epoch": 1.3093145869947276,
+      "grad_norm": 0.019983140751719475,
+      "learning_rate": 5.9420609792803604e-05,
+      "loss": 0.5163890719413757,
+      "num_tokens": 7831286.0,
+      "step": 280
+    },
+    {
+      "epoch": 1.3140011716461628,
+      "grad_norm": 0.019265560433268547,
+      "learning_rate": 5.871461383718344e-05,
+      "loss": 0.5131065845489502,
+      "num_tokens": 7859261.0,
+      "step": 281
+    },
+    {
+      "epoch": 1.318687756297598,
+      "grad_norm": 0.022508597001433372,
+      "learning_rate": 5.801108984397354e-05,
+      "loss": 0.6271864175796509,
+      "num_tokens": 7885431.0,
+      "step": 282
+    },
+    {
+      "epoch": 1.3233743409490333,
+      "grad_norm": 0.017685849219560623,
+      "learning_rate": 5.7310079936671545e-05,
+      "loss": 0.4501059353351593,
+      "num_tokens": 7913746.0,
+      "step": 283
+    },
+    {
+      "epoch": 1.3280609256004687,
+      "grad_norm": 0.016956064850091934,
+      "learning_rate": 5.6611626088244194e-05,
+      "loss": 0.47412410378456116,
+      "num_tokens": 7942905.0,
+      "step": 284
+    },
+    {
+      "epoch": 1.332747510251904,
+      "grad_norm": 0.01661519892513752,
+      "learning_rate": 5.59157701186142e-05,
+      "loss": 0.47091153264045715,
+      "num_tokens": 7974222.0,
+      "step": 285
+    },
+    {
+      "epoch": 1.3374340949033392,
+      "grad_norm": 0.02206377312541008,
+      "learning_rate": 5.522255369215622e-05,
+      "loss": 0.45931947231292725,
+      "num_tokens": 8001662.0,
+      "step": 286
+    },
+    {
+      "epoch": 1.3421206795547744,
+      "grad_norm": 0.021219369024038315,
+      "learning_rate": 5.453201831520245e-05,
+      "loss": 0.6900108456611633,
+      "num_tokens": 8027798.0,
+      "step": 287
+    },
+    {
+      "epoch": 1.3468072642062097,
+      "grad_norm": 0.018379267305135727,
+      "learning_rate": 5.38442053335571e-05,
+      "loss": 0.5277714729309082,
+      "num_tokens": 8060964.0,
+      "step": 288
+    },
+    {
+      "epoch": 1.3514938488576451,
+      "grad_norm": 0.016046447679400444,
+      "learning_rate": 5.3159155930021e-05,
+      "loss": 0.5390745401382446,
+      "num_tokens": 8097006.0,
+      "step": 289
+    },
+    {
+      "epoch": 1.3561804335090804,
+      "grad_norm": 0.021015141159296036,
+      "learning_rate": 5.247691112192577e-05,
+      "loss": 0.5474106073379517,
+      "num_tokens": 8127461.0,
+      "step": 290
+    },
+    {
+      "epoch": 1.3608670181605156,
+      "grad_norm": 0.019383052363991737,
+      "learning_rate": 5.179751175867784e-05,
+      "loss": 0.5751417875289917,
+      "num_tokens": 8156154.0,
+      "step": 291
+    },
+    {
+      "epoch": 1.3655536028119508,
+      "grad_norm": 0.018249724060297012,
+      "learning_rate": 5.112099851931265e-05,
+      "loss": 0.4899910092353821,
+      "num_tokens": 8181611.0,
+      "step": 292
+    },
+    {
+      "epoch": 1.370240187463386,
+      "grad_norm": 0.021499330177903175,
+      "learning_rate": 5.044741191005908e-05,
+      "loss": 0.6149919629096985,
+      "num_tokens": 8208053.0,
+      "step": 293
+    },
+    {
+      "epoch": 1.3749267721148213,
+      "grad_norm": 0.018448660150170326,
+      "learning_rate": 4.9776792261913896e-05,
+      "loss": 0.5378507375717163,
+      "num_tokens": 8236427.0,
+      "step": 294
+    },
+    {
+      "epoch": 1.3796133567662565,
+      "grad_norm": 0.020480727776885033,
+      "learning_rate": 4.910917972822713e-05,
+      "loss": 0.5937903523445129,
+      "num_tokens": 8261482.0,
+      "step": 295
+    },
+    {
+      "epoch": 1.3842999414176917,
+      "grad_norm": 0.021023282781243324,
+      "learning_rate": 4.844461428229782e-05,
+      "loss": 0.475787878036499,
+      "num_tokens": 8283675.0,
+      "step": 296
+    },
+    {
+      "epoch": 1.3889865260691272,
+      "grad_norm": 0.022145070135593414,
+      "learning_rate": 4.7783135714980744e-05,
+      "loss": 0.6385347843170166,
+      "num_tokens": 8308823.0,
+      "step": 297
+    },
+    {
+      "epoch": 1.3936731107205624,
+      "grad_norm": 0.019477400928735733,
+      "learning_rate": 4.712478363230362e-05,
+      "loss": 0.6258922815322876,
+      "num_tokens": 8336711.0,
+      "step": 298
+    },
+    {
+      "epoch": 1.3983596953719977,
+      "grad_norm": 0.022344205528497696,
+      "learning_rate": 4.646959745309609e-05,
+      "loss": 0.49651363492012024,
+      "num_tokens": 8355982.0,
+      "step": 299
+    },
+    {
+      "epoch": 1.403046280023433,
+      "grad_norm": 0.018516717478632927,
+      "learning_rate": 4.581761640662927e-05,
+      "loss": 0.5565701723098755,
+      "num_tokens": 8392293.0,
+      "step": 300
+    },
+    {
+      "epoch": 1.4077328646748681,
+      "grad_norm": 0.02049725316464901,
+      "learning_rate": 4.516887953026691e-05,
+      "loss": 0.5584365725517273,
+      "num_tokens": 8418190.0,
+      "step": 301
+    },
+    {
+      "epoch": 1.4124194493263036,
+      "grad_norm": 0.018973683938384056,
+      "learning_rate": 4.452342566712818e-05,
+      "loss": 0.5133151412010193,
+      "num_tokens": 8446616.0,
+      "step": 302
+    },
+    {
+      "epoch": 1.4171060339777388,
+      "grad_norm": 0.02378571778535843,
+      "learning_rate": 4.388129346376178e-05,
+      "loss": 0.48315972089767456,
+      "num_tokens": 8470185.0,
+      "step": 303
+    },
+    {
+      "epoch": 1.421792618629174,
+      "grad_norm": 0.023281428962945938,
+      "learning_rate": 4.3242521367832015e-05,
+      "loss": 0.5316978096961975,
+      "num_tokens": 8494772.0,
+      "step": 304
+    },
+    {
+      "epoch": 1.4264792032806093,
+      "grad_norm": 0.020904844626784325,
+      "learning_rate": 4.260714762581677e-05,
+      "loss": 0.5123225450515747,
+      "num_tokens": 8519300.0,
+      "step": 305
+    },
+    {
+      "epoch": 1.4311657879320445,
+      "grad_norm": 0.022328970953822136,
+      "learning_rate": 4.197521028071765e-05,
+      "loss": 0.46863052248954773,
+      "num_tokens": 8544147.0,
+      "step": 306
+    },
+    {
+      "epoch": 1.4358523725834798,
+      "grad_norm": 0.02324979566037655,
+      "learning_rate": 4.13467471697817e-05,
+      "loss": 0.6753160953521729,
+      "num_tokens": 8568351.0,
+      "step": 307
+    },
+    {
+      "epoch": 1.440538957234915,
+      "grad_norm": 0.01805717498064041,
+      "learning_rate": 4.0721795922236496e-05,
+      "loss": 0.5179596543312073,
+      "num_tokens": 8598849.0,
+      "step": 308
+    },
+    {
+      "epoch": 1.4452255418863502,
+      "grad_norm": 0.016879988834261894,
+      "learning_rate": 4.010039395703664e-05,
+      "loss": 0.5598073601722717,
+      "num_tokens": 8635197.0,
+      "step": 309
+    },
+    {
+      "epoch": 1.4499121265377855,
+      "grad_norm": 0.019025171175599098,
+      "learning_rate": 3.948257848062351e-05,
+      "loss": 0.5200734734535217,
+      "num_tokens": 8668440.0,
+      "step": 310
+    },
+    {
+      "epoch": 1.454598711189221,
+      "grad_norm": 0.020963799208402634,
+      "learning_rate": 3.8868386484697417e-05,
+      "loss": 0.5326945185661316,
+      "num_tokens": 8689757.0,
+      "step": 311
+    },
+    {
+      "epoch": 1.4592852958406561,
+      "grad_norm": 0.02187330089509487,
+      "learning_rate": 3.825785474400291e-05,
+      "loss": 0.48833855986595154,
+      "num_tokens": 8716599.0,
+      "step": 312
+    },
+    {
+      "epoch": 1.4639718804920914,
+      "grad_norm": 0.01959657296538353,
+      "learning_rate": 3.7651019814126654e-05,
+      "loss": 0.5740939974784851,
+      "num_tokens": 8747984.0,
+      "step": 313
+    },
+    {
+      "epoch": 1.4686584651435266,
+      "grad_norm": 0.022587157785892487,
+      "learning_rate": 3.7047918029308815e-05,
+      "loss": 0.5537501573562622,
+      "num_tokens": 8768742.0,
+      "step": 314
+    },
+    {
+      "epoch": 1.473345049794962,
+      "grad_norm": 0.01806885376572609,
+      "learning_rate": 3.6448585500267485e-05,
+      "loss": 0.5617669224739075,
+      "num_tokens": 8799757.0,
+      "step": 315
+    },
+    {
+      "epoch": 1.4780316344463973,
+      "grad_norm": 0.017536548897624016,
+      "learning_rate": 3.5853058112036596e-05,
+      "loss": 0.5035026669502258,
+      "num_tokens": 8832220.0,
+      "step": 316
+    },
+    {
+      "epoch": 1.4827182190978325,
+      "grad_norm": 0.018568003550171852,
+      "learning_rate": 3.5261371521817244e-05,
+      "loss": 0.5434101819992065,
+      "num_tokens": 8862171.0,
+      "step": 317
+    },
+    {
+      "epoch": 1.4874048037492678,
+      "grad_norm": 0.01621602475643158,
+      "learning_rate": 3.467356115684284e-05,
+      "loss": 0.5029686093330383,
+      "num_tokens": 8896875.0,
+      "step": 318
+    }
+  ],
+  "logging_steps": 1,
+  "max_steps": 428,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 2,
+  "save_steps": 106,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": false
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 6.10671473662922e+17,
+  "train_batch_size": 2,
+  "trial_name": null,
+  "trial_params": null
+}