Update README.md

README.md

@@ -1,35 +1,70 @@
 ---
-base_model: fancyfeast/llama-joycaption-beta-one-hf-llava
+base_model:
+- fancyfeast/llama-joycaption-beta-one-hf-llava
 tags:
 - captioning
 - mlx
-- mlx-my-repo
 pipeline_tag: image-text-to-text
-library_name: transformers
+library_name: mlx
 ---
 
-# n0kovo/llama-joycaption-beta-one-hf-llava-mlx-8Bit
+# Llama JoyCaption Beta One (MLX 8-bit)
 
-The Model [n0kovo/llama-joycaption-beta-one-hf-llava-mlx-8Bit](https://huggingface.co/n0kovo/llama-joycaption-beta-one-hf-llava-mlx-8Bit) was converted to MLX format from [fancyfeast/llama-joycaption-beta-one-hf-llava](https://huggingface.co/fancyfeast/llama-joycaption-beta-one-hf-llava) using mlx-lm version **0.28.3**.
+MLX port of [fancyfeast/llama-joycaption-beta-one-hf-llava](https://huggingface.co/fancyfeast/llama-joycaption-beta-one-hf-llava), quantized to 8-bit for efficient inference on Apple Silicon.
 
-## Use with mlx
+[JoyCaption](https://github.com/fpgaminer/joycaption) is a free, open, and uncensored image captioning VLM built on Llama 3.1 8B and SigLIP2, designed for generating descriptive captions to train diffusion models.
+
+## Model Details
+
+| | |
+|---|---|
+| Architecture | LLaVA (SigLIP2 vision encoder + Llama 3.1 8B) |
+| Quantization | 8-bit (`group_size=64`) |
+| Vision encoder | google/siglip2-so400m-patch14-384 |
+| Image resolution | 384x384 |
+| Total size | ~9.1 GB |
+
+## Usage with mlx-vlm
 
 ```bash
-pip install mlx-lm
+pip install mlx-vlm
 ```
 
 ```python
-from mlx_lm import load, generate
+import mlx.core as mx
+from mlx_vlm import load, generate
+from mlx_vlm.prompt_utils import apply_chat_template
+from mlx_vlm.utils import load_config
 
-model, tokenizer = load("n0kovo/llama-joycaption-beta-one-hf-llava-mlx-8Bit")
+MODEL = "n0kovo/llama-joycaption-beta-one-hf-llava-mlx-8Bit"
 
-prompt="hello"
+model, processor = load(MODEL)
+config = load_config(MODEL)
 
-if hasattr(tokenizer, "apply_chat_template") and tokenizer.chat_template is not None:
-    messages = [{"role": "user", "content": prompt}]
-    prompt = tokenizer.apply_chat_template(
-        messages, tokenize=False, add_generation_prompt=True
-    )
+prompt = apply_chat_template(
+    processor,
+    config,
+    "Write a long descriptive caption for this image in a formal tone.",
+    num_images=1,
+)
 
-response = generate(model, tokenizer, prompt=prompt, verbose=True)
+output = generate(
+    model,
+    processor,
+    prompt,
+    image="image.jpg",
+    max_tokens=512,
+    temperature=0.6,
+)
+print(output)
 ```
+
+## Conversion Notes
+
+- Language model weights quantized to 8-bit via `mlx-lm`
+- Vision encoder weights quantized to 8-bit where layer dimensions allow (`group_size=64`); 28 MLP layers with incompatible dimensions (4304, not divisible by 64) are kept in float16
+- Projector weights quantized to 8-bit
+
+## Credits
+
+- Original model by [fancyfeast](https://huggingface.co/fancyfeast) — [JoyCaption GitHub](https://github.com/fpgaminer/joycaption)