Diffusion is the rising tide that eventually submerges all frequencies, high and low 🌊 Diffusion is the gradual decomposition into feature scales, fine and coarse 🗼 Diffusion is just spectral autoregression 🤷🌈

everyone working at the forefront of model training should internalize what this graph means for long term cluster scaling.

Here's a quick outline of the learning rate schedule for Llama 3.1 - this is likely the simplest and most powerful "shape" for a schedule when training models.

🤸🤸🤸

Weirdly this is an easier problem

Frantically refreshing Google's repos looking for `hidden_BLD`

"It's not that bad" is the best compliment I can imagine for our tool.

It's the first model that isn't just a funny little guy

hey look! Cartwheel is in times square. send me a picture if you happen to see it!!

Scale your layers to get transfer across model sizes

chat, what's the best way to process (using phi 3.5) 4 Trillion tokens?

There are really three classes of language models: 1. frontier reasoning models for solving coding problems 2. reliable specific models for product deployment 3. good enough and fast models for batch processing

sounds like a normal PhD experience to me

When training the SAEs for GemmaScope, the team had to store 17 Petabytes of activations. They found though, that if they used them within 100 days, it was cheaper than recomputing.

Deep learning is frustrating but still pretty neat

first excellent singing -> face model I've seen

chrome's tab grouping feature is actually so good

The irony is that the 80 hr / week hustle bros are winning right now cause Claude makes each marginal hour way more productive. Essentially mitigating tiredness.

Find Joakim from Cartwheel at the PyTorch conference, say hi!!

The only benchmark I trust - Meta's legal team