Love the license, confused by some of the decisions.
Thanks for the great Apache-2.0 release. I think anything released with permissive licensing contributes to collective intelligence and so I also believe that this release will mitigate some of the harm that AI might impose upon us.
That being said, I just wanted to add a piece of feedback about the size discussion. First, an echo:
120b is unfortunately laughable to call small. It is a clear signal to the entire world that plebeians who cannot afford to have 100+ gigs of assignable memory in their systems are not even in the scope of open source AI in Mistral's eyes, which is a complete disaster for humanity, as we have been relying pretty heavily on Mistral for open source models that every day consumers can run on edge compute. I sincerely hope this was not the intention, but the signal is not a comforting one to see in our current times where the risk of the elite hoarding intelligence is so high.
That being said, even if that wasn't a component in this conversation, I'm kind of devastated to see this happen for another reason: the density of the model on the forward pass.
Just... I know that MoE models are all the rage. I see it. I get it. But I'm sorry to say, I am 100% sure that 6.6 billion parameters on the forward pass is nowhere near a replacement for a model that has 24 billion parameters of density. No way. Even if you argue that there isn't an overall disjointedness, a variability in the performance based on expert selection, a huge loss in overall critical/logical thinking and grounded real-world problem solving (think simplebench), there are still undeniable differences in the performance, feeling, evaluation, and overall technology between mistral small 3 and mistral small 4.
This is nowhere near as trainable downstream. That is a MASSIVE problem. It is not as easy to train a MoE model than a fully activated dense model.
There are people, obviously, who get lots of use out of OSS 120b, Qwen 3 Next or 30/35b3a, GLM 4.7 flash, etc etc. But these ultra sparse models are just plain different. They are made to excel in coding workflows and things that are highly technical and have easily identifiable, deterministic and quantitative "correct" responses. When we step back to needing a more nuanced, generally stable model for general knowledge, challenging tasks, and novel use cases, especially for finetuning, these sparse MoEs have proven brittle and ineffective for a massive amount of use cases time and time again. That would all be fine, because there are legitimately great reasons to go MoE and take those tradeoffs on, if it wasn't named Mistral Small.
However, here's the nail in the coffin: It would take willful ignorance not to see that Mistral Small 3 was hands down the most important truly open source LLM model to the community thus far in Mistral's catalogue, and probably even just the most important model period. The tunes. The releases. The absolutely staggering download counts.
The bottom line is this: 3090 and 4090 series cards comfortable can house at MAXIMUM about 32b parameters in weights in their 24gb of VRAM, and even a 5090 at FOUR THOUSAND DOLLARS where I live with 32gb cannot even come close to comfortably deploying this model. This model cannot even be deployed with two 5090s, which would be around a ten thousand dollar system. It can't even be deployed comfortably in a 128GB system like the Strix Halo system. You could argue that it could be put on regular RAM and ran through a beefy CPU for inference, or maybe run on a system with unified memory, but even the lowest spec Mac Studio with the 256gb you'd need to do that starts at $8499. So you're basically saying "unless you're someone with an upper workstation class/into server or enterprise class hardware, this model is not for you."
Ever scarier? You call it Small. Small! THAT'S what your new small is? What does that say about your intentions for the future? About if you're on the side of the elite few, or the many?
Worse STILL is that by doing so, you basically take the model that WAS for "us", and effectively sunset it by releasing this, which is sized medium AT BEST, while your large now is squarely outside of anything that could reasonably called "consumer hardware" in any way without getting laughed out of the room, and you traditionally have not open sourced your medium sized models. So it's a completely differently behaving performing technology that is NOT a simple swap in in many scenarios, has profoundly increased system requirements, barely increases performance across non-reasoning use cases in benchmarks? This is deeply unsettling. Sure, we have a 4.4x effective speedup or whatever, but even if it effectively has less compute needed on the forward pass, 4.4x speedup is only true on certain configurations, namely in ones that might be found in datacenters. I wonder what the performance per dollar of system over token processing speed a second is? I wonder what the actual speed is on a 4090 class system running small 3 vs a more expensive machine running on something like a unified ram setup or a CPU inference build might look like? Hopefully I'm just reading into this wrong and this was a simple oversight on the reality of computers people actually have access to. Thanks for all of the work you folks have done for collective intelligence in general thus far.
i ain't reading all that. im happy for you tho, or sorry that happened.
They just follow the trend of releasing phony "100b" like the other labs. All those models are effectively 20-40b dense performance. Sure, less compute but way more memory requirements. Never really found one of those I liked. They get slightly better at 10+A and 200-300b total; still full of small model smell.
The mistral large 3 at Q2 like deepseek was probably not that bad with proper sampling, devstral 123b ended up that way. I hated it on the API. Too bad that's where everyone got their impressions and there's like no decent support. Gonna be another sleeper like pixtral-large. This, not so much.
Apparently no copyrighted data in this train and very EU act compliant. Bleh.
I agree that the small model smell seems to begin to lessen at 10-12b on forward pass, and I also have not found any of the more sparse models super useful. I very much prefer using models that are at least 20B on the forward pass, but even then I find stability really isn't the same as a dense model with all parameters active on the forward pass.
Compliance or bias would be less of an issue if the base model could be augmented with post-training, but that'd be kind of insane to try and do on this big of a model, so I doubt we will see too much with it. It's just plain harder to get a good tune on a MoE model as well, from what I've heard from DavidAU, who has hit the previous Qwen 30b3a, and also from what I've heard from TheDrummer when working with GLM Air.
Most of the working MoE tunes in general were done by actual labs like wizard, etc.
"MoE bad"
There are new experimental backends in development which keep frequently-used experts in VRAM and infrequently-used experts in system RAM, giving the vast majority of VRAM-only performance without necessitating a billion dollar GPU. I don't know how long until these will reach the public but they do exist and they will make large sparse models even more attractive. It's still far cheaper to buy 128GB of system RAM than it is to buy a 96GB or even 48GB video card. I do agree however, especially in light of dense Qwen3.5-27B's impressive performance compared to their sparse 35B model, that extreme sparsity can be a step back. But if I can get 4+ prompts done with comparable quality in the time it would've taken to run 1 before, that's hard to ignore. MoE is still a relatively new thing, it will probably take a while to determine best practices. Something with more like 15B-30B active parameters but with really good MTP to keep speed high might be another path forward."Big model bad"
AFAIK there's nothing to say they won't soon release a "mini" model that's more in line with your size expectations, this is just what they're releasing today."Output quality bad"
Mistral 4 has to comply with new EU law which effectively mandates that any models produced in Europe must now use limited datasets which lobotomize models in order to protect consumers, thereby driving everyone to use Chinese and/or American models which don't protect consumers. It's a European mindset thing, like when they saved the planet by moving all their manufacturing to countries that generate power with coal and liked it so much they proceeded to switch from nuclear to coal themselves.
This nuance is the reason for the length of my original post, which it seems a bit like you didn't read (which I get). There, I already covered most of these points.
MoE models have many advantages and trade offs as I said. Particularly currently, pruning with things like REAP. There's also an argument that it is better for the earth as theoretically it consumes less power.
The problem is that it isn't Mistral small. Ministral 3 was 14b max and is already out, fairly recently. Sure, they may restructure the whole line. But think of it like this:
Imagine McDonalds decided to suddenly 4x, almost 5x the size of their small drinks. What might the message be? What might the media say? Public opinion?
This is even worse. Its like if sprite released sprite 2 and it was dark soda with 4x the sugar. Its fine, I don't hate sugar or large sodas, just that ain't sprite
As for the EU things those are larger political waves that we don't have control over, but I will say that synthetic data has often proven to have even better performance in some scenarios, so it's not necessarily worse.
I understand the nuance but I think you're getting hung up on the name. As I said, I have no inside knowledge on this but I think it would be premature to assume at this early point that there's no "mini" model coming. "Ministral" is a thing they've released before.
You don't have to fit a MoE model in VRAM to have solid token generation speeds, just the active layers, so I'm not really sure why that's a complaint here
Be happy that you have a decent computer next to you that you can run these models on. Mark my words, in 5-10 years there will be no computers on people's desks. Instead, everyone would be using their mobiles to connect to mesh networks that will feed endless streams of data from the cloud.
The entire existence will be moved to the cloud because nobody will be able to afford a desktop computer, and it will be a "new norm". Those who come after us will never know what they could've had, or how the life was before it. Just like everything else in today's world...nobody wants to remember the past and so history is bound to repeat itself.
Ministral exists which is more than likely in your side range. To my knowledge, they have no plans to discontinue it.
Mistral sometimes uses weights-available licenses, but they seemed to have cut down on that.
Mistral is still better here than some other labs that release 400B or even 1T behemoths without a version that's any smaller.
@mancub This really isn't the time or place. Cloud compute will undoubtedly become cheaper and cheaper over time but there will always be a place for local hardware unless Jensen brings out a way to overcome the speed of light. If western companies attempt to exploit their AI hegemony we will see China continue to do what China does best. There is no future where OpenAI or Anthropic or Google says "ha ha ha! we're jacking up prices 1000% and there's nothing you can do about it!" because AI servers are commodity hardware anyone can buy and there will always be someone willing to sell tokens for less. Chinese companies are also rapidly picking up their pace and may be able to directly compete with both Nvidia as well as the RAM cartel within a few years, if not in absolute performance then at least in price.
If anything, I would say the trend is toward local hardware, not cloud hardware. Local models are getting smaller and more intelligent; Qwen3.5-27B is better than any frontier model from a year ago and it's perhaps 1/20th the size or smaller. If not for the sudden unexpected RAM crunch you'd probably be able to run it (or something even better) locally on a phone in a few years. Even today I can run the surprisingly competent 9B model on my phone. There are constantly new innovations oriented toward improving model performance on smaller amounts of VRAM - just today I read about research Nvidia has been doing into new ways to compress KV cache. They describe it as being a similar principle to JPEG compression, presumably in comparison today's compression techniques which might be likened to GIF compression via color reduction with dithering or simply resizing a BMP if you can understand my meaning. This means less VRAM use and, critically, much faster performance on long-context tasks. And I myself just completed a fairly extensive benchmark study in which I've confirmed that Qwen 3.5-122B indeed performs identically (within 1%) on 4-bit AWQ/Autoround/GPTQ as on FP8. I haven't yet tested FP4 or a 4-bit GGUF, and I need to separately test all of the above on tool-calling, but it goes to show that models truly can be extremely quantized without obvious tradeoffs. Then there's REAP, a technique which has shown itself capable of shrinking a model by 25% or more (pre-quantization) with no apparent degradation in performance.
Additionally I remind you of what I mentioned above about new techniques which can offload most of a model to RAM (off the GPU) with far less performance cost than you'd otherwise see; one of these is called PowerInfer: https://github.com/Tiiny-AI/PowerInfer. You may notice the repo hasn't been touched in a long time; this is because they're preparing to sell a commercial mini AI computer which makes use of this technology. It remains to be seen if/when they will open-source the related code, but presumably they will need to if they want people to have confidence in their hardware product's longevity. This means there will - at some point - be no need to buy something like an RTX 6000 Pro or multiple 4090/5090s just for the sake of getting access to large pools of VRAM, you'll be able to get by with perhaps a 16GB card and 128GB of RAM. Despite the current inflated cost of RAM I assure you this will still be much cheaper.
Speaking of phones, since you say everyone will continue to be allowed to own these in your future nightmare dystopia, a premium 2025 phone or tablet is still more powerful than the average PC from 5 years ago, it fits in your pocket, it uses around 2-10 watts instead of 100-500 watts, and Samsung and Google are both at work on making them capable of desktop-like workflows when plugged into an external display. Another historical datapoint is that my first computer had 16MB (megabytes) of RAM and I paid more for that 16MB than 128GB costs today. "Boomer logic" maybe, but I mean to remind you of some perspective on how far we've come.
This is a temporary price spike. It will likely last 2-4 years but it is not the calamity some people are making it out to be. The media needs a catastrophe to report on and they will invent one if none exists. Electronics are going to be more expensive by 25-50% until probably 2030. There will be shortages. You will have to pay more for inferior products. But you don't need a new phone every year and there is no rational justification to buy a new GPU every generation. If you sat the last 6 months watching prices skyrocket, unwilling to pull the trigger on a RAM kit or GPU to tide you over (if you even truly needed such a thing and aren't just overdosing on FOMO), that's on you.
Mistral 4 being crippled by self-destructive EU law is not the beginning of a trend, it's a single datapoint.
Here are those test results I mentioned if anyone's curious - not directly relevant to Mistral 4 but further proof that nobody should hesitate to use 4-bit quantized models to make the most of their VRAM.
Note the chart Y-axis is zoomed in. The goal of the test was not to measure absolute performance of Qwen3.5-122B relative to other models but to determine the relative level of degradation caused by different quantization strategies for this individual model. Thinking was disabled on all tests. Virtually all scores were within margin of error. If I had to choose an INT4 quantization I would certainly go with autoround, it was the only model with any unusual high outliers in individual sub-tests and it was (at worst) average, while GPTQ had several low-score outliers. AWQ was average to good in general, as was FP8.
Be happy that you have a decent computer next to you that you can run these models on.
I cannot run this model
I have a feeling you are looking at this one sidedly, from a perspective of technology only. Or to sound cliche, you are not seeing the forest for the trees.
As I wrote above, those who do not remember the history are bound to repeat it. And one does not need to look far back...surely all of us here were around pre-C19 days and we can compare that (and prices and availability of items) to post-C19 present day (or the "new norm" as it was coined).
Everything needs a catalist and this specific "calamity" could be the one that takes things to the next level. The googles, openais and anthropics of the world are already laughing all the way to the bank. If you think that all of a sudden things will go back to what they used to be, out of kindness of their hearts, you are terribly mistaken.
I'd love to be wrong, for all of our sakes, but all things considered the cards are in my favour...
Competition will gradually bring prices back down. You point out the Covid situation - during lockdowns my 3080 Ti cost me something like $2000 CAD and I was pleased to pay such a reasonable price. I'm now planning to sell it on the secondary market for (optimistically) perhaps $500. During COVID the crappiest integrated-graphics laptop cost almost $1000 CAD because everyone needed one for remote work. Now - in the midst of this RAM crisis - you can still get a perfectly decent laptop with a modest dGPU for under $1000 CAD. In August I bought a really good laptop for just $1200 but during COVID its price would've been astronomical. Or how about toilet paper?
Governments will get involved if the situation does not improve, as has happened in the past when RAM manufacturers exhibited cartel behavior. Ready availability of cheap consumer electronics drives every modern developed country's economy. The US economy might be getting propped up by AI investments right now but the same isn't true of every other country on earth, including the countries where RAM is made. And if nothing else changes, China will force prices down by ramping up their own domestic production of RAM and undercutting the big players.
I don't mean to downplay the uniqueness of the current situation with AI and I suspect the situation will continue to get worse before it gets better. But for normal consumers who aren't nutjobs like us, preferring to run our own inferior local LLMs with expensive hardware instead of using far more capable online models that have quite cheap subscriptions which would fit our needs, a laptop's RAM costing $200 instead of $50 is not the end of the world.
In the absolute worst-case scenario where nothing else changes, the RAM going into AI servers will advance to a more efficient node and leave behind the old node for consumer RAM. The same thing that happens with CPUs and GPUs.