NicoLab28 commited on
Commit
ef89d0f
·
verified ·
1 Parent(s): 6faa5bb

Point people arriving from older tutorials at the current files

Browse files
Files changed (1) hide show
  1. README.md +10 -0
README.md CHANGED
@@ -25,6 +25,16 @@ library_name: comfyui
25
  These files are useless on their own. They require the custom node:
26
  **[github.com/nicolab28/ComfyUI-ClipProj](https://github.com/nicolab28/ComfyUI-ClipProj)**
27
 
 
 
 
 
 
 
 
 
 
 
28
  ## Update to node 0.1.4, and re-download the `-mlp` matrices
29
 
30
  **The `-mlp` matrices are now fp16 and half the size.** The residual network was published in fp32 and the node forced fp32 on load regardless of the file, so storing it in half precision would have halved the download and saved nothing at all in VRAM. Node 0.1.4 keeps a residual in whatever precision it was saved in, converting its inputs and outputs around it instead. Measured: 240 MB on the card instead of 480 for the 4B, 288 instead of 576 for the 8B. The files here have been replaced under the same names — re-download them, and take 0.1.4 with them, because an older node will load them and cast them straight back up to fp32.
 
25
  These files are useless on their own. They require the custom node:
26
  **[github.com/nicolab28/ComfyUI-ClipProj](https://github.com/nicolab28/ComfyUI-ClipProj)**
27
 
28
+ ## Where this came from
29
+
30
+ I am not an ML researcher. I work in imaging, and programming is a tool and a hobby rather than my trade. This started as something to tinker with: I wanted to understand how a diffusion model actually uses its text encoder, and the only way I know how to understand something is to take it apart and see whether it still runs afterwards.
31
+
32
+ So the question was never "how do I save VRAM". It was "is this even possible at all". I expected it to fail. A linear map between two models that were never trained together, fitted in a single pass with no gradients and no learning rate, has no business producing usable video.
33
+
34
+ It did, and the first results were good enough that keeping them on my own disk seemed silly. That is the whole story, and it is why this is labelled a proof of concept rather than a tool: it was never designed as one.
35
+
36
+ It is also why there are so many measurements on the model card. Before showing this to anyone I had to convince myself I was not fooling myself, and most of what I tried along the way turned out to be wrong. Those attempts are written down as well, in [MEASUREMENTS.md](https://github.com/nicolab28/ComfyUI-ClipProj/blob/main/MEASUREMENTS.md) and [CALIBRATION.md](https://github.com/nicolab28/ComfyUI-ClipProj/blob/main/CALIBRATION.md).
37
+
38
  ## Update to node 0.1.4, and re-download the `-mlp` matrices
39
 
40
  **The `-mlp` matrices are now fp16 and half the size.** The residual network was published in fp32 and the node forced fp32 on load regardless of the file, so storing it in half precision would have halved the download and saved nothing at all in VRAM. Node 0.1.4 keeps a residual in whatever precision it was saved in, converting its inputs and outputs around it instead. Measured: 240 MB on the card instead of 480 for the 4B, 288 instead of 576 for the 8B. The files here have been replaced under the same names — re-download them, and take 0.1.4 with them, because an older node will load them and cast them straight back up to fp32.