BK-MOO-Pretrain · mgda · fineweb

Continual pretraining checkpoint of meta-llama/Meta-Llama-3-8B following the BK-MOO-Pretrain recipe (https://github.com/longhp1618/BK-MOO-Pretrain).

Method: mgda
Training stage: fineweb
Base model: meta-llama/Meta-Llama-3-8B
Languages: ar, bn, en, id, it, ko, sw, zh
Training data: FineWeb-2 (and LIMA+S1 for *_limas1 stages)

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "Yiran0924/bk-moo-llama3-8b-mgda-fineweb"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, torch_dtype="bfloat16", device_map="auto")

See the BK-MOO-Pretrain repo for the multi-objective gradient surgery details (MGDA / CAGrad / PCGrad / SGD).

Downloads last month: 3

Safetensors

Model size

8B params

Tensor type

BF16

Inference Providers NEW

This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for Yiran0924/bk-moo-llama3-8b-mgda-fineweb

Base model

meta-llama/Meta-Llama-3-8B

Finetuned

(598)

this model