mlx-community/RakutenAI-2.0-8x7B-MLX-8bit

This model was converted to MLX format from Rakuten/RakutenAI-2.0-8x7B using mlx-lm version 0.31.0. Refer to the original model card for more details on the model.

Use with mlx

pip install mlx-lm

from mlx_lm import load, generate

model, tokenizer = load("mlx-community/RakutenAI-2.0-8x7B-MLX-8bit")

prompt = "hello"

if tokenizer.chat_template is not None:
    messages = [{"role": "user", "content": prompt}]
    prompt = tokenizer.apply_chat_template(
        messages, add_generation_prompt=True
    )

response = generate(model, tokenizer, prompt=prompt, verbose=True)

Inference: M3 Ultra
==========
 world - 简书
hello world
hello world
hello world
推荐阅读 更多精彩内容
Android - 收藏集
Android 自定义View的各种姿势1 Activity的显示之ViewRootImpl详解 Activity...
passiontim 阅读 163,326 评论 24 赞 695
Android - 收藏集
用两张图告诉你，为什么你的 App 会卡顿? - Android - 掘金 Cover 有什么料？ 从这篇文章中你...
hw1212 阅读 10,519 评论 2 赞 52
Android - 收藏集
用两张图告诉你，为什么你的 App 会卡顿? - Android - 掘金 从这篇文章中你可以看到，Android 的...
passiontim 阅读 13,542 评论 2 赞 54
Android - 收藏集
用两张图告诉你，为
==========
Prompt: 2 tokens, 5.448 tokens-per-sec
Generation: 256 tokens, 42.701 tokens-per-sec
Peak memory: 49.838 GB