Update README.md

README.md

@@ -1,3 +1,48 @@
 ---
 license: other
 ---
+### Introduction
+Basiclly an update to the old attempt of [vicuna-chinese-replication-beta](https://huggingface.co/keyfan/vicuna-chinese-replication-beta)
+* We adopted an curriculum-learning like approch, starting from simple QAs to reasoning-intensive coding & mathamatical problems. Coincidentally, [Ziya](https://huggingface.co/IDEA-CCNL/Ziya-LLaMA-13B-v1) adopted the same idea during SFT period.
+* The base model was changed from  [chinese-llama](https://huggingface.co/ziqingyang/chinese-llama-lora-13b) to [chinese-llama-plus](https://huggingface.co/ziqingyang/chinese-llama-plus-lora-13b). However, as observed by [BiLLa](https://github.com/Neutralzz/BiLLa), continue training on  Chinese-only corpus significantly increases its perplexity on English corpus, which in turns undermines its abilities in fields like mathematical calculation in our preliminary experiment. The subject of continuing-training is under-studied, while using bilingual corpus may be a better alternative as shown so far.
+* We changed to the Vicuna v1.1 conversative template and included more CoT training data.
+
+Again, this is for research purpose only. There's no guarantee for its performance. All credits to the original authors of LLaMA and Chinese-LLaMA.
+
+Compared with previous release, new model improves on coding and reasoning problem. However it still suffers from hallucinations and perform poorly on Chinese domain-specific problem, e.g. chinese literature and idioms.
+
+### Usage
+
+We use exactly the Vicuna template for training and inference. Sample code as below.
+
+```
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+checkpoint = "keyfan/vicuna-chinese-replication-v1.1"
+
+tokenizer = AutoTokenizer.from_pretrained(checkpoint, use_fast=False)
+model = AutoModelForCausalLM.from_pretrained(checkpoint).cuda()
+
+template = ("A chat between a curious human and an artificial intelligence assistant. "
+            "The assistant gives helpful, detailed, and polite answers to the human's questions. "
+            "USER: {}\nASSISTANT:")
+question = template.format("Who was the president of the United States in 1955?")
+inputs = tokenizer.encode(question, return_tensors="pt").cuda()
+outputs = model.generate(inputs, do_sample=True, temperature=0.2, max_new_tokens=512)
+print(tokenizer.decode(outputs[0]))
+```
+
+### Evaluation
+
+* Result on the [Chinese-LLaMA-Alpaca devset](https://github.com/ymcui/Chinese-LLaMA-Alpaca/tree/main/examples) compared with the result of Alpaca-Plus-13B. For simplity, we only sample one answer for each question without any cherry-picking. We used the template as provided in their repo. GPT-4 have strong bias for more detailed answers, so the score may not be consistent with human evaluation.
+
+| Model | Macro-Average | QA | OQA | REASONING | LITERATURE | ENTERTAINMENT | GENERATION | TRANSLATION | CODE | ETHICS | 
+| -  | -  | - | - | - | - | - | - | - | - | - |
+| Alpaca-Plus-13B | 77.3 | 70 | 74 | 70 | **80** | **77** | 82 | **89** | 64 | **90** |
+| ours | **82.4** | **81** | **87** | **88** | 73 | **78** | **85** | 83 | **83** | 84 |
+
+* Result on the newly released [C-Eval test set](https://cevalbenchmark.com/index.html#home) with 5-shot. We slightly modified [MOSS's code](https://github.com/SJTU-LIT/ceval/blob/main/code/evaluator_series/evaluators/moss.py)  from ceval codebase by moving the '答案：' (Answer: ) suffix from the end of question to the beginning of the chatbot response.
+
+| Average | Avg(Hard) | STEM | Social Science | Humanities | Others |
+| - | - | - | - | - | - |
+| 37.0 | 29.5 | 34.6 | 44.5 | 35.7 | 35.9 |