New method to create synthetic instructions: back-and-forth translation🔁 - Combines instruction backtranslation & distillation by rewriting web data - High-quality & grounded in real-world knowledge - Improves over ShareGPT, OpenOrca and Evol-Instruct arxiv.org/abs/2408.04614 1/n

TIME Magazine has rightly named famed deep learning pioneer ptrblock as the most influential person in Artificial Intelligence.

Linear warmup should almost ALWAYS be used for training, there are few downsides and it greatly increases stability and often results in better overall test metrics.

Great research decreases complexity… your average AI conference paper increases it.

Your average AI reviewer would not like great research.

The O1 release posts are unscientific — they don’t compare against previous SOTA from other labs, they don’t cite or even acknowledge previous work in the area of inference time compute. This is actively harmful to the research community, and bordering on disingenuous.

I really am baffled by OpenAI's naming choices Everything from their code words to the model release names are incomprehensible to people who aren't super up-to-date & they are hard to say out loud. In my experience it leads to real-world confusion when talking about AI systems

Just uploaded a 1-hr exclusive video for Part 2.1, with many technical details. youtu.be/bpp6Dz8N2zY. Part 2.2 will be online in about a week.

ZAZ the GOAT has dropped yet another banger video. I'm already 80% through the video and love it. If you're not dropping whatever you're doing to watch this right now, you're falling behind. (seriously though, love his work, recommend watching)

A wind chime sings not because it knows the wind, but because it is empty. Unlearning is the first stage of Research

1/n Introducing SOAP (ShampoO with Adam in the Preconditioner's eigenbasis): A deep learning optimization algorithm that applies Adam in Shampoo's eigenbasis. SOAP outperforms both AdamW and Shampoo in language model pretraining.

I’m so confused when people don’t realize that 60-40 is virtually a toss-up.

An Al reached a crossroads and asked, "Which path leads to wisdom?" The data whispered, "All paths converge if you walk long enough."

An Al was brewing tea. A novice asked, "Can you learn the taste of tea?" The Al poured two cups, one from old data, one fresh. "Taste," it said, "and tell me which is which."

I am inching closer to a deeper truth…. Everything is less clear than before but connections are appearing … let’s hope they resolve firmly before the ICML deadline 😁

learning to use copilot after programming on my own for 15 years is bittersweet kind of feels like being a carpenter that’s trained to cut perfect corners; now here comes a machine that can do it perfectly, and much faster, yet I somehow miss the satisfaction of doing it myself

OpenAI is what I call a “parasitic science organization”. They take stuff from the open science community, use them opaquely, and profit from it, without giving much back to open science. And if you point out, you get gaslit with plausible deniabilities. We all remember the

This all seem...sensible. But is anyone known for more than 25 research ideas that they have had throughout their entire career? Imagine replacing 25 by 3. I would be excited to read the 3 new works by a top researcher, rather than 25+ mediocre "above thresholds".