An end-to-end research project on correcting the systematic over-formatting habit of RLHF-trained language models.