Cost + latency + hallucination evaluation for Japanese LLM selection
こんにちは! Hi Japanese LLM community 👋
Great work on this leaderboard. For Japanese enterprise deployments, cost and latency from Japanese data centers are often the deciding factor between a Japanese-specific model and a frontier model.
I built an open source LLM Evaluation Framework with 5 metrics:
→ 💰 Cost per 1K tokens — Japanese providers vs GPT-4o-mini vs Gemini Flash
→ ⚡ Latency p50/p95 — real-world API latency from Japan
→ 🔍 Hallucination Rate — especially important for Japanese legal, medical, and news AI
→ 🎯 Accuracy — MMLU + TruthfulQA + custom benchmark support for Japanese tasks
→ 🧠 Reasoning Quality — CoT depth scoring
Custom benchmark system supports adding Japanese-language evaluation tasks.
Live demo: https://huggingface.co/spaces/vigneshwar234/llm-eval-demo
GitHub: https://github.com/vignesh2027/LLM-Evaluation-Framework
Open source, free forever!
Dear @vigneshwar234
Thank you for your kind words and congratulations on your very interesting framework!
Yesterday, your suggestion was shared in the slack of LLM-jp. It will be review for the next research meeting by stakeholders.
I will keep you update! 😊
Best,
Akim
Hey ! I'd love to work with you remotely on a per-task basis if you have any opportunities available. I'm open to contributing to projects and would be happy to discuss how I can help. Please let me know if you're hiring or need any assistance. through contact -- https://www.linkedin.com/in/vigneshwar-l-td729994/
Hello @vigneshwar234
Thank you for your interest in LLM-jp. Unfortunately, it's a research initiative supported by the Japanese government via the National Institute of Informatics. Every year, the NII is recruiting but mostly at PhD level. Plus, you have to be fluent in Japanese (speak, read, write - mandatory), even for the open source projects.
Good luck for your research!
Akim