Today, we're releasing LAB-Bench, a set of >2000 evaluations for language models and agents on scientific research tasks in biology. Public models underperform PhD/postdoc-level humans on nearly all tasks. Claude 3.5 Sonnet is the clear frontrunner atm, but long way to go. 1/

I’m in Mexico City until Thursday. Let me know if you’re here too!

How well can LLM agents complete diverse tasks compared to skilled humans? Our preliminary results indicate that our baseline agents based on several public models (Claude 3.5 Sonnet and GPT-4o) complete a proportion of tasks similar to what humans can do in ~30 minutes. 🧵

On average, when agents can do a task, they do so at ~1/30th of the cost of the median hourly wage of a US bachelor’s degree holder. One example: our Claude 3.5 Sonnet agent fixed bugs in an ORM library at a cost of <$2, while the human baseline took >2 hours.

AI agents still aren't reliable yet, even for short tasks, but simple agents can occasionally complete tasks that take humans hours. Managing AI risks will become much more challenging when agents can autonomously execute on long-term projects.

Interesting

Just do things.

Can’t wait for the amount of resources Casey Handmer, PhD will put into antimatter production and storage after Terraform exits caseyhandmer.wordpress.com/2024/08/18/ant…

We’re open sourcing Vivaria, our platform for running LLM agent evaluations and conducting research in eliciting AI capabilities. 🪴

I wrote about the past and future of my podcast

1/ Can AI scaling continue through 2030? We examine whether constraints on power, chip manufacturing, training data, or data center latencies might hinder AI growth. Our analysis suggests that AI scaling can likely continue its current trend through 2030.

Dream job

Wow, diffusion models (used in AI image generation) are also game engines - a type of world simulation. By predicting the next frame of the classic shooter DOOM, you get a playable game at 20 fps without any underlying real game engine. This video is from the diffusion model.

Dwarkesh is going to be the Mr. Beast for intellectuals. Absolutely relentless on getting better.

We thought it would be helpful to have all of the similar themes/components from each of Deepmind's Frontier Safety Framework, OpenAI's Preparedness Framework, and Anthropic's Responsible Scaling Policy, in one place.

A few months ago I hosted a Q&A with Geoffrey Hinton in Berkeley: youtube.com/watch?v=PTF5Up…

He literally rewrote the form to make it easier for him to find you. You need to apply.

Help me build a media empire. New blog post: Why you should become my COO.

We ran o1-preview on our suite of ML R&D/SWE/general agency tasks, from Sep 3–9. 4 days of scaffolding iteration took it from well below GPT-4o to on par with the highest-scoring public model (3.5 Sonnet). We expect substantial performance gains from more elicitation/finetuning.