vincentmin

/

llama-2-7b-reward-oasst1

like 1

It's the same as the standard Llama-7b. The answer depends on the precision in which you load you weights. If you load it in 4bit, which is how it was trained, that's about 4GB to load the model. You need some additional memory to do inference, so let's say at least 6GB as a minimum requirement.

Thanks! Does loading in 8bit increases the accuracy?

I haven't done a systematic analysis of the performance with 4bit and 8bit. Typically the performance increases with higher bits, but since this model was trained with 4bit precision, that may not be true in this case.