Continued Pretraining?
#1
by JAALBERTS - opened
I have tested the models on a domain-specific usecase. The PCA values were very unreliable unfortunately.
However, i do have access to a dataset of around 300 000 mid-high res images in our domain, would it be possible to perform continued pretraining on the model?, the paper suggests that the smaller ViT's have all been distilled, so i do not know if that will be a problem compared to training the 7B ViT. Also without the original training code performing gram anchoring for example seems very challenging.
For DinoV2 i was not able to find a lot of information either, anyone here have input/ideas?