We have released our DCLM models on huggingface! To our knowledge these are by far the best performing truly open-source models (open data, open weight models, open training code) 1/5

On my way to Vienna! Happy to chat about ML, life, the universe, and everything 🌍. Also checkout the following link for an overview of Apple at #ICML2024 machinelearning.apple.com/updates/apple-…

Is bigger always better? 🐘 The idea that scaling more than any other ingredient has driven progress has become formalized as the “bitter lesson” Is Sutton right? 📜arxiv.org/abs/2407.05694…

Announcing the NeurIPS 2024 Workshops! Read our blog for details on our selection process this year - and the list of accepted workshops for 2024! blog.neurips.cc/2024/08/02/ann…

I am looking for a Ph.D. research intern focusing on personalization, preference-based learning, and LLMs to come work at Apple Machine Learning Research! You can apply via: jobs.apple.com/en-gb/details/… Please send an email to [email protected] after applying!

Do you have any cool ideas on how to intervene upon foundation model representations? Consider submitting to the 🍃MINT workshop! Deadline 13th September!!

Check out this simple tweak to AdamW that massively accelerates LLM training !!

Enjoy attention? Want to make it ~18% faster? Try out Sigmoid Attention. We replace the traditional softmax in attention with a sigmoid and a constant (not learned) scalar bias based on the sequence length. Paper: arxiv.org/abs/2409.04431 Code: github.com/apple/ml-sigmo… This was