ConceptLM-Pythia-410M, pre-trained from scratch with 300B tokens from the Pile. 




|     Tasks      |Version|Filter|n-shot|    Metric     |   | Value |   |Stderr|
|----------------|------:|------|-----:|---------------|---|------:|---|------|
|arc_challenge   |      1|none  |     0|acc            |↑  | 0.2210|±  |0.0121|
|                |       |none  |     0|acc_norm       |↑  | 0.2645|±  |0.0129|
|arc_easy        |      1|none  |     0|acc            |↑  | 0.5202|±  |0.0103|
|                |       |none  |     0|acc_norm       |↑  | 0.4689|±  |0.0102|
|hellaswag       |      1|none  |     0|acc            |↑  | 0.3434|±  |0.0047|
|                |       |none  |     0|acc_norm       |↑  | 0.4101|±  |0.0049|
|lambada_openai  |      1|none  |     0|acc            |↑  | 0.5304|±  |0.0070|
|                |       |none  |     0|perplexity     |↓  | 9.8351|±  |0.2845|
|lambada_standard|      1|none  |     0|acc            |↑  | 0.3971|±  |0.0068|
|                |       |none  |     0|perplexity     |↓  |23.1996|±  |0.8277|
|piqa            |      1|none  |     0|acc            |↑  | 0.6659|±  |0.0110|
|                |       |none  |     0|acc_norm       |↑  | 0.6687|±  |0.0110|
|race            |      2|none  |     0|acc            |↑  | 0.3024|±  |0.0142|
|sciq            |      1|none  |     0|acc            |↑  | 0.8220|±  |0.0121|
|                |       |none  |     0|acc_norm       |↑  | 0.7650|±  |0.0134|
|wikitext        |      2|none  |     0|bits_per_byte  |↓  | 0.8012|±  |   N/A|
|                |       |none  |     0|byte_perplexity|↓  | 1.7425|±  |   N/A|
|                |       |none  |     0|word_perplexity|↓  |19.4855|±  |   N/A|
|winogrande      |      1|none  |     0|acc            |↑  | 0.5414|±  |0.0140|