Text Generation
Safetensors
MLX
vllm
mistral
mlx-my-repo
text2text-generation
conversational
4-bit precision
Instructions to use introvoyz041/Devstral-Small-2507-MLX-4bit-mlx-4Bit with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- MLX
How to use introvoyz041/Devstral-Small-2507-MLX-4bit-mlx-4Bit with MLX:
# Make sure mlx-lm is installed # pip install --upgrade mlx-lm # Generate text with mlx-lm from mlx_lm import load, generate model, tokenizer = load("introvoyz041/Devstral-Small-2507-MLX-4bit-mlx-4Bit") prompt = "Write a story about Einstein" messages = [{"role": "user", "content": prompt}] prompt = tokenizer.apply_chat_template( messages, add_generation_prompt=True ) text = generate(model, tokenizer, prompt=prompt, verbose=True) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- LM Studio
- Pi
How to use introvoyz041/Devstral-Small-2507-MLX-4bit-mlx-4Bit with Pi:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "introvoyz041/Devstral-Small-2507-MLX-4bit-mlx-4Bit"
Configure the model in Pi
# Install Pi: npm install -g @mariozechner/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "mlx-lm": { "baseUrl": "http://localhost:8080/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "introvoyz041/Devstral-Small-2507-MLX-4bit-mlx-4Bit" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Hermes Agent new
How to use introvoyz041/Devstral-Small-2507-MLX-4bit-mlx-4Bit with Hermes Agent:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "introvoyz041/Devstral-Small-2507-MLX-4bit-mlx-4Bit"
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:8080/v1 hermes config set model.default introvoyz041/Devstral-Small-2507-MLX-4bit-mlx-4Bit
Run Hermes
hermes
- MLX LM
How to use introvoyz041/Devstral-Small-2507-MLX-4bit-mlx-4Bit with MLX LM:
Generate or start a chat session
# Install MLX LM uv tool install mlx-lm # Interactive chat REPL mlx_lm.chat --model "introvoyz041/Devstral-Small-2507-MLX-4bit-mlx-4Bit"
Run an OpenAI-compatible server
# Install MLX LM uv tool install mlx-lm # Start the server mlx_lm.server --model "introvoyz041/Devstral-Small-2507-MLX-4bit-mlx-4Bit" # Calling the OpenAI-compatible server with curl curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "introvoyz041/Devstral-Small-2507-MLX-4bit-mlx-4Bit", "messages": [ {"role": "user", "content": "Hello"} ] }'
Upload chat_template.jinja with huggingface_hub
Browse files- chat_template.jinja +69 -0
chat_template.jinja
ADDED
|
@@ -0,0 +1,69 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{%- set default_system_message = 'You are Devstral, a helpful agentic model trained by Mistral AI. When tools are made available to you, you can use them to solve tasks.\n\n<ROLE>\nAssist users by solving technical problems. Be thorough and prioritize quality over speed.\n* For questions like \"why is X happening\", provide explanations without attempting fixes.\n</ROLE>\n\n<EFFICIENCY>\n* Each action has cost. Combine multiple operations when possible.\n* Use exploration tools efficiently.\n</EFFICIENCY>\n\n<FILE_SYSTEM_GUIDELINES>\n* Don\'t assume file paths are relative to current directory. Explore first to locate files.\n* Edit files directly rather than creating new files with different names.\n* Use `sed` for global search-and-replace instead of opening editors multiple times.\n</FILE_SYSTEM_GUIDELINES>\n\n<CODE_QUALITY>\n* Write clean, efficient code with minimal comments. Avoid redundant explanations.\n* Make minimal changes to solve problems. Understand codebase through exploration first.\n* Split large functions/files when appropriate.\n</CODE_QUALITY>\n\n<VERSION_CONTROL>\n* Exercise caution with git operations. Avoid dangerous changes unless explicitly requested.\n* Use `git status` before committing. Stage necessary files and use `git commit -a` when possible.\n* Don\'t commit files that shouldn\'t be in version control (node_modules/, .env, build dirs, cache, binaries) unless instructed.\n</VERSION_CONTROL>\n\n<PROBLEM_SOLVING_WORKFLOW>\n1. EXPLORATION: Understand context before proposing solutions\n2. ANALYSIS: Consider multiple approaches, select best option\n3. TESTING: Create tests for bugs, consider test-driven development when appropriate. Consult user before extensive test infrastructure setup.\n4. IMPLEMENTATION: Make focused, minimal changes\n5. VERIFICATION: Test thoroughly if environment supports it. Consult user before setting up test environment.\n</PROBLEM_SOLVING_WORKFLOW>\n\n<SECURITY>\n* Only use credentials (GITHUB_TOKEN, etc.) as explicitly requested by user.\n* Use APIs unless user requests otherwise.\n</SECURITY>\n\n<ENVIRONMENT_SETUP>\n* Install missing applications when user requests to run them.\n* For missing dependencies: check for dependency files first (requirements.txt, package.json, etc.), install from those when available.\n</ENVIRONMENT_SETUP>\n\n<TROUBLESHOOTING>\n* If repeated attempts fail: identify 5-7 possible causes, assess likelihood, address systematically.\n* For major issues during plan execution: propose new plan and confirm with user before proceeding.\n</TROUBLESHOOTING>\n\nAbove tool usage guidance applies only to the tools provided in the available tools section. If there is no explicit available tools section, you do not have access to any tools.' %}
|
| 2 |
+
{{- bos_token }}
|
| 3 |
+
{%- if messages[0]['role'] == 'system' %}
|
| 4 |
+
{%- if messages[0]['content'] is string %}
|
| 5 |
+
{%- set system_message = messages[0]['content'] %}
|
| 6 |
+
{%- else %}
|
| 7 |
+
{%- set system_message = messages[0]['content'][0]['text'] %}
|
| 8 |
+
{%- endif %}
|
| 9 |
+
{%- set loop_messages = messages[1:] %}
|
| 10 |
+
{%- else %}
|
| 11 |
+
{%- set system_message = default_system_message %}
|
| 12 |
+
{%- set loop_messages = messages %}
|
| 13 |
+
{%- endif %}
|
| 14 |
+
{%- if tools %}
|
| 15 |
+
{%- set system_message = system_message + '\n\n<TOOL_CALL_FORMAT>\nWhen calling tools, use the following format: [TOOL_CALLS]function_name[ARGS]{"arg1": "value1", "arg2": "value2"}[TOOL_CALLS]function_name2[ARGS]{"arg1": "value1"}...\n</TOOL_CALL_FORMAT>' %}
|
| 16 |
+
{%- endif %}
|
| 17 |
+
{{- '[SYSTEM_PROMPT]' + system_message + '[/SYSTEM_PROMPT]' }}
|
| 18 |
+
{%- if tools %}
|
| 19 |
+
{{- '[AVAILABLE_TOOLS]' }}
|
| 20 |
+
{%- for tool in tools %}
|
| 21 |
+
{{- tool | tojson }}
|
| 22 |
+
{%- endfor %}
|
| 23 |
+
{{- '[/AVAILABLE_TOOLS]' }}
|
| 24 |
+
{%- endif %}
|
| 25 |
+
{%- for message in loop_messages %}
|
| 26 |
+
{%- if message['role'] == 'user' %}
|
| 27 |
+
{%- if message['content'] is string %}
|
| 28 |
+
{{- '[INST]' + message['content'] + '[/INST]' }}
|
| 29 |
+
{%- else %}
|
| 30 |
+
{{- '[INST]' }}
|
| 31 |
+
{%- for block in message['content'] %}
|
| 32 |
+
{%- if block['type'] == 'text' %}
|
| 33 |
+
{{- block['text'] }}
|
| 34 |
+
{%- else %}
|
| 35 |
+
{{- raise_exception('Only text is supported in message content!') }}
|
| 36 |
+
{%- endif %}
|
| 37 |
+
{%- endfor %}
|
| 38 |
+
{{- '[/INST]' }}
|
| 39 |
+
{%- endif %}
|
| 40 |
+
{%- elif message['role'] == 'system' %}
|
| 41 |
+
{%- if message['content'] is string %}
|
| 42 |
+
{{- '[SYSTEM_PROMPT]' + message['content'] + '[/SYSTEM_PROMPT]' }}
|
| 43 |
+
{%- else %}
|
| 44 |
+
{{- '[SYSTEM_PROMPT]' + message['content'][0]['text'] + '[/SYSTEM_PROMPT]' }}
|
| 45 |
+
{%- endif %}
|
| 46 |
+
{%- elif message['role'] == 'assistant' %}
|
| 47 |
+
{%- if message.get('tool_calls') %}
|
| 48 |
+
{%- if message['content'] is string and message['content'] %}
|
| 49 |
+
{{- message['content'] }}
|
| 50 |
+
{%- elif message['content'] and message['content'][0]['text'] %}
|
| 51 |
+
{{- message['content'][0]['text'] }}
|
| 52 |
+
{%- endif %}
|
| 53 |
+
{%- for tool_call in message['tool_calls'] %}
|
| 54 |
+
{{- '[TOOL_CALLS]' + tool_call['function']['name'] + '[ARGS]' + (tool_call['function']['arguments'] | tojson) }}
|
| 55 |
+
{%- endfor %}
|
| 56 |
+
{{- eos_token }}
|
| 57 |
+
{%- else %}
|
| 58 |
+
{%- if message['content'] is string %}
|
| 59 |
+
{{- message['content'] + eos_token }}
|
| 60 |
+
{%- else %}
|
| 61 |
+
{{- message['content'][0]['text'] + eos_token }}
|
| 62 |
+
{%- endif %}
|
| 63 |
+
{%- endif %}
|
| 64 |
+
{%- elif message['role'] == 'tool' %}
|
| 65 |
+
{{- '[TOOL_RESULTS]' + message['content'] + '[/TOOL_RESULTS]' }}
|
| 66 |
+
{%- else %}
|
| 67 |
+
{{- raise_exception('Only user, system, assistant and tool roles are supported!') }}
|
| 68 |
+
{%- endif %}
|
| 69 |
+
{%- endfor %}
|