MMLU is the standard LM evaluation but model developers (i) use different prompting strategies and (ii) often do not release prompts. 3rd-party researchers often obtain lower scores 🤯

📢 HELM MMLU uses simple, standardized prompts, resulting in fair, reproducible comparisons of…

Excited to share that our paper introducing the REFORMS checklist is now out Science Advances!

In it, we:
- review common errors in ML for science
- create a checklist of 32 items applicable across disciplines
- provide in-depth guidelines for each item

science.org/doi/10.1126/sc…

I am really excited to be part of this project led by Sayash Kapoor and Arvind Narayanan to help improve practices in machine-learning based science. science.org/doi/10.1126/sc…

🇦🇹 I’m going to #iclr2024 in Vienna next week. Who all do I know that’ll be there?

Students’ papers there:
Katherine Tian: arxiv.org/abs/2311.08401
Charlotte Nicks: openreview.net/forum?id=4eJDM…
Eric: arxiv.org/abs/2310.12962
Parth Sarthi: arxiv.org/abs/2401.18059

A few questions.

1. I wonder if these licenses are exclusive? Can FT also license its data to other developers, or no?

2. How did they determine a value for their data? If terms are not disclosed for these deals, how can other news sites bargain/set an appropriate rate?

On tasks like coding we can keep increasing accuracy by indefinitely increasing inference compute, so leaderboards are meaningless. The HumanEval accuracy-cost Pareto curve is entirely zero-shot models + our dead simple baseline agents.
New research w Sayash Kapoor Benedikt Ströbl 🧵

Sara Hooker Research is social. It's important to talk to people, interact with other students, ask things, email researchers, go to conferences, talk about your work, and ask to talk about your work. It's intimidating, might feel awkward, but crucial.

Yes!!!😊

Jelani makes an excellent point

I would go further and extend this to the graduate level with fellowships from both government (e.g. NSF, NDSEG) and industry (e.g. Google, Meta).
Marginal benefit for students/faculty is often greater elsewhere.

- signed by Cornell/Stanford alum

in the coming weeks me and Zhengxuan Wu are giving in-person talks on ReFT at
- Demandbase (SF, 5/1)
- Stanford NLP Group Lunch (Stanford, 5/2)
- Amazon Web Services Generative AI (Santa Clara, 5/10)

belated but I'm excited to start a PhD at Machine Learning Dept. at Carnegie Mellon this fall as a NSF fellow!! I'm incredibly grateful to my mentors and advisors Aaron Roth, Kevin He, and Yi Xing for all their guidance and support along the way 😀

Has been super fun working on Llama 3! Really excited about the models yet to come!

ai.meta.com/blog/meta-llam…

Sometimes I randomly think about how so incredibly *lucky* we are that Alondra was appointed OSTP Director when she was - the AI Bill of Rights came at such the right time & continues to inform everything from the EO & OMB guidelines to state and municipal legislation. Like, wow.

Congratulations Dan Boneh of Stanford University, newly inducted #NASmember ! #NAS161 #computerscience

Earlier this month, we convened researchers and leaders in the EU, BRICS, Africa, global AI Safety Institutes & international organizations to navigate the complex policy ecosystem directing society's adoption of AI systems. Missed this event? Watch here: stanford.io/4bdPeOM

HELM Lite v1.2.0 is out!
Datasets: NarrativeQA, NaturalQA, OpenbookQA, MMLU, MATH, GSM8K, LegalBench, MedQA, WMT14
Results (we still need to add Claude 3, which requires more prompt finagling):
crfm.stanford.edu/helm/lite/v1.2…

There is a really nice community of researchers developing transformer alternatives. Want to highlight these impressive folks.
Simran Arora (Simran Arora), Chunting Zhou (@violet_zct), Dan Fu (@realDanFu), and Songlin Yang (@SonglinYang4)