I benchmarked Anthropic's new tool use beta API on the Berkeley function calling benchmark. Haiku beats GPT-4 Turbo in half of the scenarios. Results in 🧵 A huge thanks to Shishir Patil, Fanjia Yan, Tianjun Zhang, Joey Gonzalez & rest for providing this benchmark publicly.

Congrats NVIDIA on the exciting 340B model release! The model was tested under the codename "june-chatbot" and is now coming out of stealth with impressive performance, surpassing Llama-3-70b across hard benchmarks like Arena-Hard-Auto. The new best open model? Come play with

Here's the animated LMSYS arena tool I built for the talk using Claude 3.5 Sonnet and Artifacts - inspired by Peter Gostev's visualizations Peter's: linkedin.com/posts/peter-go… My tool: tools.simonwillison.net/arena-animated

.RunLLM's now on the Modin Project documentation site — thanks to the Modin team for their help! Let us know what you think: modin.readthedocs.io/en/stable/

Super fun to have the RunLLM team together for the last couple days! We might need a harder escape room next time 😎

Great post by Celebal Technologies on implementing RAFT on top of Databricks fine-tuning to out-perform RAG: celebaltech.com/blogs/enhancin… Incidentally the RAFT paper will be presented at Conference on Language Modeling so check it out if you're there.

Now this is fun research!

Torch.compile is awesome! No more headaches from hacking tricky low-level CUDA code for new attention variants. I can't wait to see them catch up with H100 and FP8 support as well.

🙏 Thank you NVIDIA for sponsoring vLLM development. The DGX H200 machine is marvelous! We plan to use the machine for benchmarking and performance enhancement 🏎️.

For the past year, Vikram Sreekanti and I have been writing about whatever we found interesting in our generative AI themed blog - Generating Conversations. However, when we look at which posts did the best, it is clear that our readers are most interested in deeper discussions on

We've been thinking a lot about how to price RunLLM recently — pricing is always hard, but AI adds some interesting wrinkles to the dynamic. We haven't completely figured it out yet, but we thought we'd share our thoughts about where things are headed: open.substack.com/pub/frontierai…

What is the right pricing model for AI? Should it be a monthly fee or a flat rate per token? Do you pay extra for more knowledge? Three years ago, I was focused on server-less computing for AI and how to allocate inference engines. At the time, consumption based pricing was

Want to answer NL questions over your data? Introducing Table Augmented Generation (TAG)! Joint work w/ the amazing Matei Zaharia Carlos Guestrin Joey Gonzalez Asim Biswal Sid Jha Amog Kamsetty Shu Liu 📚 Paper: arxiv.org/abs/2408.14717 🛠️ Code: github.com/tag-research/t… 🧵

Back in 2023, my students working on Gorilla project made the case for connecting LLMs to SaaS APIs. Today, everyone knows that models should be interacting with APIs. What people don't realize is that LLMs need to know more than how to call the API. They need to learn the

I am excited to announce the launch of the Video Arena. Our goal is to study video generation models and ultimately how humans prompt them. You can help us by watching entertaining generative AI videos constructed using the same prompt.

We finally have a benchmark that evaluates stateful multi-step and multi-turn function calling!