This model was converted to MLX format from Rakuten/RakutenAI-2.0-8x7B using mlx-lm version 0.31.0.
Refer to the original model card for more details on the model.
Use with mlx
pip install mlx-lm
from mlx_lm import load, generate
model, tokenizer = load("mlx-community/RakutenAI-2.0-8x7B-MLX-4bit")
prompt = "hello"
if tokenizer.chat_template is not None:
messages = [{"role": "user", "content": prompt}]
prompt = tokenizer.apply_chat_template(
messages, add_generation_prompt=True
)
response = generate(model, tokenizer, prompt=prompt, verbose=True)
Inference: M3 Ultra
==========
world - CSDN
精华内容
下载资源
问答
我要提问
Hello World
2019-04-09 11:20:00
Hello World
展开全文
Hello World
2019-04-09 11:20:00
Hello World
展开全文
Hello World
2019-04-09 11:20:00
Hello World
展开全文
Hello World
2019-04-09 11:20:00
Hello World
展开全文
Hello World
2019-04-09 11:20:00
Hello World
展开全文
Hello World
2019-04-09 11:20:00
Hello World
展开全文
Hello World
2019-04-09 11:20:00
Hello World
展开全文
Hello World
2019-04-09 11
==========
Prompt: 2 tokens, 7.076 tokens-per-sec
Generation: 256 tokens, 70.175 tokens-per-sec
Peak memory: 26.421 GB
- Downloads last month
- 20
Model size
47B params
Tensor type
BF16
·
U32 ·
Hardware compatibility
Log In to add your hardware
4-bit
Model tree for mlx-community/RakutenAI-2.0-8x7B-MLX-4bit
Base model
mistralai/Mistral-7B-v0.1 Finetuned
Rakuten/RakutenAI-7B Finetuned
Rakuten/RakutenAI-2.0-8x7B