MoWE Paper Releases associated with the MoWE paper written by Tavish, Vardhaman, Keshava, Dr. Deepan. Coded and trained with openlanguagemodel. FAIRC/mowe-mowe-768-19l 0.3B • Updated 23 days ago • 12 FAIRC/mowe-dense-768-19l 0.1B • Updated 23 days ago • 14 FAIRC/mowe-moe-768-19l 0.4B • Updated 23 days ago • 12
Token Averaging A comprehensive research framework for analyzing whether averaging adjacent tokens in a LLM can reduce the compute compared to standard model. FAIRC/token-averaging-matched_mean-avg_50m_k8 Updated 22 days ago • 16 FAIRC/token-averaging-model1_50m Updated 10 days ago • 36 FAIRC/token-averaging-avg_50m_k8_wexp Updated 22 days ago • 16 FAIRC/token-averaging-avg_50m_k8_learnable_pos Updated 22 days ago • 10
Token Averaging A comprehensive research framework for analyzing whether averaging adjacent tokens in a LLM can reduce the compute compared to standard model. FAIRC/token-averaging-matched_mean-avg_50m_k8 Updated 22 days ago • 16 FAIRC/token-averaging-model1_50m Updated 10 days ago • 36 FAIRC/token-averaging-avg_50m_k8_wexp Updated 22 days ago • 16 FAIRC/token-averaging-avg_50m_k8_learnable_pos Updated 22 days ago • 10
MoWE Paper Releases associated with the MoWE paper written by Tavish, Vardhaman, Keshava, Dr. Deepan. Coded and trained with openlanguagemodel. FAIRC/mowe-mowe-768-19l 0.3B • Updated 23 days ago • 12 FAIRC/mowe-dense-768-19l 0.1B • Updated 23 days ago • 14 FAIRC/mowe-moe-768-19l 0.4B • Updated 23 days ago • 12
