MLX/Mac Limited Usage
#21
by nhe-ai - opened
mlx-audio is able to infer random generations, which are great. However, is not possible to clone nor to produce long generations nor tweak any other parameter aside the temperature. Plus, sometimes the control tokens are just not respected, but I presume is more related to the model itself.
Any attempt to use text/audio references, it throws a dimensional mismatch:
...dac.py", line 203, in encode
- 2 * (z_q @ self.codebook.weight.T)
~~~~^~~~~~~~~~~~~~~~~~~~~~~~
ValueError: [matmul] Last dimension of first input with shape (1,62,64) must match second to last dimension of second input with shape (16,1024).
I tried patching the dac, and many other attempts without success.
By the way, is feasible to run the .safetensors in Apple Silicon in other ways? It’s possible to run effectively SGLang and the server or just not?
Extra note: the model is small, but for faster inference, before running .generate() I do the following:
import mlx.nn, mlx.core
mlx.nn.quantize(model, bits=8)
# mlx.core.eval(model.parameters())