Instructions to use prefeitura-rio/Rio-3.5-Open-397B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use prefeitura-rio/Rio-3.5-Open-397B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="prefeitura-rio/Rio-3.5-Open-397B")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("prefeitura-rio/Rio-3.5-Open-397B")
model = AutoModelForMultimodalLM.from_pretrained("prefeitura-rio/Rio-3.5-Open-397B", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use prefeitura-rio/Rio-3.5-Open-397B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "prefeitura-rio/Rio-3.5-Open-397B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "prefeitura-rio/Rio-3.5-Open-397B",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/prefeitura-rio/Rio-3.5-Open-397B

SGLang

How to use prefeitura-rio/Rio-3.5-Open-397B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "prefeitura-rio/Rio-3.5-Open-397B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "prefeitura-rio/Rio-3.5-Open-397B",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "prefeitura-rio/Rio-3.5-Open-397B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "prefeitura-rio/Rio-3.5-Open-397B",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use prefeitura-rio/Rio-3.5-Open-397B with Docker Model Runner:
```
docker model run hf.co/prefeitura-rio/Rio-3.5-Open-397B
```

Despite the way the project was released, this is a good model.

#25

by tarruda - opened Jun 16

Discussion

tarruda

Jun 16

Merging is a completely valid method of producing new models and this one produced a good result.

This is NOT N2 despite the model presenting itself as such. By the N2 devs own post, this model is approximately 57% N2 and 43% Qwen.

My suggested next steps:

Admit the mistakes so everyone can move on.
Re-run the benchmarks on the base merge.
Publish the model with correct ancestry information.

heltonleal

Jun 16

Reinforcing the message, I totally agree.

There's way too much bad PR going on vs what the actual situation is. There's value on what has been done and I would be exciting to see more value coming in the future. Admit the mistake, embrace it, turn it into a learning opportunity.

amarosnithe

Jun 16

Merging is a completely valid method of producing new models and this one produced a good result.

This is NOT N2 despite the model presenting itself as such. By the N2 devs own post, this model is approximately 57% N2 and 43% Qwen.

Reinforcing the message, I totally agree.

There's way too much bad PR going on vs what the actual situation is. There's value on what has been done and I would be exciting to see more value coming in the future. Admit the mistake, embrace it, turn it into a learning opportunity.

Yes, Merging is a real training method and nobody claimed otherwise, the problem is the fact that this ISN'T a good model because it never existed in the first place. These guys made a half-assed job at copying the weights of NEX and Qwen when they claimed this was a "model trained from scratch", then removed the weights and this description when caught red-handed, admitted that it was a merge, but even then its false because what they published was in fact a cheap copy (to the point they even had to HARD-CODE a prompt telling the model that it was RIo-3.5, and not Nex N2 Pro, again, please check up Nex's analysis and dossier proving this). Not to mention the fact they're STILL lying and giving BS excuses on why they haven't published the final weights yet (which they can't publish because they don't have them, since they never existed).

So the bad PR is completely justified (they lied about this constantly so far), and they should be held accountable for thinking we wouldn't notice this.

Alegoria36

Jun 17

This comment has been hidden (marked as Abuse)

amarosnithe

Jun 17

This comment has been hidden (marked as Abuse)

Nerdsking

Jun 18

"AIHNNNN .... THE MODEL IS GUD.... MERGE IS VALID...WE MONKEY MERGED 2 MODELS AND GOT A WORSE ONE " huahauahauhauhauahuahauauhauauha

no-mad

Jun 18

It's perfectly fine to disagree and debate, but please do this in a respectful and civilized way, otherwise we'll have to lock the discussion and/or potentially take further action on some of the participants. Also see our content policy as a reminder: https://huggingface.co/content-guidelines

biologin

Jun 18

It is okay to merge such models, sure. It is not okay to use R$ 500k public money (release was from an IT municipality) and call it brand new when all you really changed was some prompts and some mixing of models.

fpaboim

Jun 18

It's perfectly fine to disagree and debate, but please do this in a respectful and civilized way, otherwise we'll have to lock the discussion and/or potentially take further action on some of the participants. Also see our content policy as a reminder: https://huggingface.co/content-guidelines

Is there policy on fraudulently claiming to have finentuned a large model while releasing merged weights from other companies without disclosing the origin? Do any of the liceses of said models require attribution? Mind you this fraud was commited with brazilian taxpayer money so you can imagine why this causes such widespread anger, moreso coming from the government this reflects badly on all brazilians given the degenerate level of institutional incompetence and probably fraud we are witnessing.

baikw

Jun 20

Don't you find it laughable to play the impartial mediator here?
It started with outright deception—no mention of the "merge" at all. Then, when people found out, they kept making excuses, claiming it was an upload error. And even now, they still haven't uploaded the "actual" model. Are they using ADSL or something?
I'm no lawyer—I don't know if they're even allowed to keep quiet about it—but morally, they just keep getting it wrong.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment