The Alpaca moment of Large Multimodal Models! Can we build native LMMs just like Llama for simple multimodal generation? Introducing Anole: the first open-source, autoregressive native LMM for multimodal generation. Building on Chameleon by AI at Meta: github.com/GAIR-NLP/anole

More examples:

Folks at GAIR-NLP have successfully managed to fine-tune image and interleaved generation back into Chameleon! Turns out, it’s quite hard to disable image generation out of early fusion models. Love all the example generations! Hope to see many more lizards!

Within pre-training circles, character.ai was known to have the highest HFU numbers out of all pre-training teams during the A100 era because of this.

Excited to share the work of the team I've been mentoring at YerevaNN. They've built a large corpus covering 100M+ molecules, trained a few billion parameter language models, wrapped LMs into an optimization algorithm (augmented with genetic searches), and beaten all

1/n Introducing MoMa 🖼, our new sparse early-fusion architecture for mixed-modal language modeling that significantly boosts pre-training efficiency 🚀 (arxiv.org/pdf/2407.21770). MoMa employs a mixture-of-expert (MoE) framework with modality-specific expert groups. Given any

If you were interested in my cryptic posts on how to train Chameleon-like models up to 4x faster, check out our MoMa paper which covers a detailed overview of most of our architectural improvements. tl;dr adaptive compute in 3-dim, modality, width, depth.

With Chameleon we showed that early fusion mixed modal LLMs can deliver strong improvements over unimodal and late fusion alternatives, however with this paradigm shift how do we rethink our core model architecture to optimize for native multimodality and efficiency? We

this is one that i've been waiting for. I was hoping they were going to release a paper covering these approaches for this very reason.

xjdr holy shit armen's team cracked as fuck once again

Armen Aghajanyan Excited that we have deciphered most of the cryptic posts 🙌

New from AI at Meta: Efficient Early-Fusion Pre-training with Mixture of Modality-Aware Experts With a 1T token training budget, MoMa 1.4B achieves FLOPs savings of 3.7x 🚀 Authors Victoria X Lin, Akshat Shrivastava, and Liang Luo will be on alphaXiv this week to answer your questions.

Some cool things we did are investigate the "training-scaling-laws" behind early-fusion and late-fusion TLDR: Late-fusion gets good results fast, but we find that early-fusion models show more promise for learning modality connections and interleaved learning expanding on the

I've left FAIR/Meta, it's time to build.

Explains me getting this email five years after my last interview. Going to apply. If I get hired, shorting Meta stock.

early fusion omni models are the future. invest early.