Update README.md

README.md

@@ -69,7 +69,7 @@ For testing and comparison, I use models produced by [Bartowski][btk] (see credi
 All experimental versions were generated using an appropriate imatrix created from datasets available at [eaddario/imatrix-calibration][ical]. In `llama.cpp`, an imatrix is a calibration file derived from running representative text through the model and collecting activation statistics. It is used to weight quantization error so that error in more “important” directions (as estimated from activations) is penalized more heavily.
 
 The process to generate these models is roughly as follows:
-1. Convert the original model's [safetensors][sfts] to [GGUF][ggf] F16*
+1. Convert the original model's [safetensors][sfts] to [GGUF][ggf] F16
 2. Estimate the [Perplexity][ppl] score for the F16 model (baseline) using the [wikitext-2-raw-v1][wki-dat] dataset, and save the [logits][lgt]
 3. Generate an [imatrix][imx-dat] from the most appropriate [calibration dataset][ical]
 4. Quantize the baseline model targeting a bpw average (e.g. `llama-quantize --target-bpw 4.5678 --state-file --imatrix imatrix.gguf baseline-model-F16.gguf 12`)
@@ -77,7 +77,20 @@ The process to generate these models is roughly as follows:
 6. Keep version with the best 𝜌PPL and μKLD scores
 7. Repeat until all desired quants are created
 
-*[BF16][bf16] would be preferred, but F16 performs better on Apple's GPUs
+### Misconceptions about BF16 to F16 Conversion
+A common concern when converting BFloat16 ([BF16][bf16]) models to Float16 (F16) is the potential for accuracy loss. Specifically:
+- Weight Clipping (Overflow): Clipping, or overflow, is often feared but only occurs if a model's weights exceed the range of ±65,503. This is a relatively rare issue in practice.
+- Subnormal Zeroing (Underflow): A more frequent occurrence is underflow, where weights smaller than approximately 5.96x10⁻⁸ are converted to zero.
+
+Crucially, when the F16 model is subsequently used for quantization, the resulting degradation in metrics like Perplexity ([PPL][ppl]) or Kullback–Leibler Divergence ([KLD][kld]) is minimal. Any variations are typically restricted to the hundreds or thousandths decimal places compared to the BF16 model.
+
+However, considering that weight clipping presents a more substantial risk to model integrity, every BF16 base model undergoes validation prior to the conversion process. Consequently, no models hosted in this repository exhibit performance degradation due to overflow clipping.
+
+While BF16 offers precision benefits, performance remains a key factor.
+- Conversion Speed: Tests, such as timing `convert_hf_to_gguf.py`, show a notable performance difference, with conversion to BF16 being 15–30% slower than to F16.
+- Inference Speed: A less pronounced but still present difference (3–6%) is observed during inference. Although native BF support has been introduced by many chip manufacturers, the slower performance **may** stem from the entire software and hardware stack (firmware, libraries, etc.) not being fully optimized yet.
+
+The choice to prioritize F16 over BF16 is driven by a focus on maximizing performance in specific deployment environments. My primary objective is not large-scale quantization production, a domain where others like [Bartowski][btk] and [Unsloth][ust] excel at, but rather optimizing inference performance for resource-constrained environments. Since BF16 support is not yet widespread in areas like mobile, edge, and embedded devices, using F16 ensures broader compatibility and easier optimization for these use cases.
 
 # Advantages and disadvantages of the global target bits‑per‑weight quantization process
 ### Advantages