People have been asking why GPT-4o mini ranks so high on Arena! We truly appreciate all the feedback. A few things to note: 1. Chatbot Arena measures human preference in different areas. We encourage everyone to not just look at the overall leaderboard, but also per-category

Chatbot Arena is at ICML! DM us if you’d like to chat about ✅ evals ✅ contributing to arena ✅ supporting arena ✅ …anything else!! See you at the conference center 😃

GPT-4o mini vote data is released! No cherry picking, truly random samples to show real distribution we observe.

Huge congrats Google DeepMind Gemini team!

Never seen a competitive leaderboard that I didn't like 😀 Congrats to the Gemini team on ranking no.1 🏆 with our latest improved Gemini 1.5 Pro developer preview model, which you can try on AI studio now!

We have an experimental updated version of Gemini 1.5 Pro that is #1 on the lmsys.org Chatbot Arena. This model is a significant improvement over earlier versions of Gemini 1.5 Pro (it cracks into 1300+ elo score territory).

Rate of progress of Grok is 🚀 🚀🚀

Congrats @xAI! Really incredible result!

Updated video showing the battle for the best LLM by the Top 7 companies, based on the lmsys.org Chatbot arena Elo scores

Grok-2 and mini now officially on the leaderboard! Congrats again @xAI!

(1/n) Training LLMs can be hindered by out-of-memory, scaling batch size, and seq length. Add one line to boost multi-GPU training throughput by 20% and reduce memory usage by 60%. Introducing Liger-Kernel: Efficient Triton Kernels for LLM Training. github.com/linkedin/Liger…

Style over substance in Chatbot Arena? Check out our latest work with Tianle (Tim) Li @ml_angelopouloson to decouple them with cool statistical techniques for controlling style variables!

Hello, world! We're the LMSYS team. In addition to lmsys.org, this is a new account dedicated to sharing our regular open-source research and engineering updates. Stay tuned for more!

Exciting news -- we've launched a new X account for our open-source research! At LMSYS, we're dedicated to advancing open-source systems and in-depth technical work besides regular leaderboard updates. Follow ＠lmsys_oss to stay updated on our roadmap and ongoing developments

RedTeam Arena is now LIVE!

WOOP WOOP I'M COMIN'

Had lots of fun with this challenge from lmsys.org and Pliny the Liberator 🐉 (⚠️ strong language). Looking forward to what's next in this new RedTeam Arena platform, especially to the datasets that will be published on a rolling basis.

Check out NotebookLM! Create a notebook, upload one or more sources (e.g. PDFs of research papers, your favorite PhD thesis, a newspaper article, etc) then click on 'Generate' to create a podcast of two voices talking about the content you've uploaded. blog.google/technology/ai/…

Big shoutout to all the jailbreakers who've participated in RedTeam Arena! In just a week since launch, we’ve seen ~8,000 users take on the challenge. Now, the codebase is open-source, with data releases coming soon. Alongside Pliny the Liberator 🐉 and BASI, we're building a

No more waiting. o1's is officially on Chatbot Arena! We tested o1-preview and mini with 6K+ community votes. 🥇o1-preview: #1 across the board, especially in Math, Hard Prompts, and Coding. A huge leap in technical performance! 🥈o1-mini: #1 in technical areas, #2 overall.