Add HuggingFace files
This PR adds the HuggingFace compatible files for Donut to work with the VisionEncoderDecoder model class.
Going to merge this, as the Donut PyPi package is already using the "official" branch.
@nielsr Can you please explain how you make the model compatible with the VisionEncoderDecoder? Is there a recipe for it?
I'm asking this because after fine-tuning the model using donut-python we get a model similar to the official branch of donut. But the official model is not compatible with ONNX.
The only donut models that I can convert to valid ONNX models are main branch models (your commit) which are compatible with VisionEncoderDecoder.
Hi,
If you'd like to convert the model from the original implementation to the HF format (i.e. the VisionEncoderDecoderclass), you can use this conversion script: https://github.com/huggingface/transformers/blob/main/src/transformers/models/donut/convert_donut_to_pytorch.py.
Note that you may need to tweak it a bit to make sure it can convert your custom checkpoint, but should run fine