Abstract
Self-Distillation Fine-Tuning enables on-policy learning from demonstrations, reducing catastrophic forgetting and allowing continuous skill accumulation in foundation models.
Continual learning, enabling models to acquire new skills and knowledge without degrading existing capabilities, remains a fundamental challenge for foundation models. While on-policy reinforcement learning can reduce forgetting, it requires explicit reward functions that are often unavailable. Learning from expert demonstrations, the primary alternative, is dominated by supervised fine-tuning (SFT), which is inherently off-policy. We introduce Self-Distillation Fine-Tuning (SDFT), a simple method that enables on-policy learning directly from demonstrations. SDFT leverages in-context learning by using a demonstration-conditioned model as its own teacher, generating on-policy training signals that preserve prior capabilities while acquiring new skills. Across skill learning and knowledge acquisition tasks, SDFT consistently outperforms SFT, achieving higher new-task accuracy while substantially reducing catastrophic forgetting. In sequential learning experiments, SDFT enables a single model to accumulate multiple skills over time without performance regression, establishing on-policy distillation as a practical path to continual learning from demonstrations.
Community
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models (2026)
- Knowledge is Not Enough: Injecting RL Skills for Continual Adaptation (2026)
- Training-Trajectory-Aware Token Selection (2026)
- POPE: Learning to Reason on Hard Problems via Privileged On-Policy Exploration (2026)
- Entropy-Adaptive Fine-Tuning: Resolving Confident Conflicts to Mitigate Forgetting (2026)
- Trust-Region Adaptive Policy Optimization (2025)
- Stable On-Policy Distillation through Adaptive Target Reformulation (2026)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend
@WaltonFuture Their method corresponds to the "SFT from Teacher" baseline in our paper. We'll include a citation in the next version!
Great work! Has anyone applied SDFT to Vision-Language Models where the in-context demonstrations are images rather than text? The on-policy distillation approach seems promising for visual classification tasks where generalization to unseen object types matters, but I'm curious whether visual ICL at 14B scale provides a strong enough teacher signal.
Intuitively makes sense IMO
I'd love to see ablations for how sensitive is the conditioned teacher to the privileged context/info. But it is not trivial at all :( I feel like this is the tricky part though
Say, if I just give it up straight-up the final answer, the teacher logprobs perhaps might not be the desired (i.e. a math proof for which we know the outcome but yet the model doesn't know how to prove). In this case, ideally we would want a c so that it triggers a full correct reasoning in latent space. Perhaps gradually giving the model increasing hints until it solves the problem on its own (like in Leetcode) would be an interesting experiment to ablate
Get this paper in your agent:
hf papers read 2601.19897 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 25
Browse 25 models citing this paperDatasets citing this paper 1
Spaces citing this paper 0
No Space linking this paper