Announcing Flash-Decoding 🚀

Flash-Decoding makes LLM decoding much faster, and in particular allows to scale to very long sequence lengths (64k+) without slowdown!

Read more on our blog: hubs.la/Q025Fldb0

Today we announced Flash-Decoding 🚀- a better way to run the LLM attention during decoding. It can be up to 60x faster than Flash-Attention v2, and enables scaling to sequence lengths of 64k+!
This is a common work with Tri Dao , Francisco Massa and Grigory Sizov
(1/n)

Announcing Flash-Decoding, to make long-context LLM inference up to 8x faster! Great collab with Daniel Haziza, Francisco Massa and Grigory Sizov.

Main idea: load the KV cache in parallel as fast as possible, then separately rescale to combine the results.
1/7

We just introduced PyTorch 2.0 at the #PyTorchConference , introducing torch.compile!

Available in the nightlies today, stable release Early March 2023.

Read the full post: bit.ly/3VNysOA

🧵below!

1/5

We're releasing an optimized implementation of GPT2/GPT3 with FlashAttention🚀!
This trains 3-5x faster than the Huggingface version, reaching up to 189 TFLOPs/sec per A100, 60.6% (model) FLOPs util of the theoretical maximum. 1/6
github.com/HazyResearch/f…

Glad to present our work with Francisco Massa and Alexandre Défossez « Hybrid Transformers for Music Source Separation » done at @MetaAI.
We achieve 9.20 dB of SDR on the MUSDB18 test set.
- paper: arxiv.org/abs/2211.08553
- code: github.com/facebookresear…
- audio: ai.honu.io/papers/htdemuc…
1/5

[🧠👨‍💻 ML Coding Series] Continuing on with the ML coding series! This might be the most thorough ML paper explanation in history - 1h and 45 minutes! 😅

Facebook (now @MetaAI) DETR explained

YT: youtu.be/xkuoZ50gK4Q

Nicolas Carion Francisco Massa Gabriel Synnaeve Alexander Kirillov Sergey Zagoruyko

1/

When using torchvision models, pretrained=True has traditionally returned the same pre-trained weights (from 2017) for reproducibility. What if it defaults to latest pre-trained weights with much better accuracies? We would love to hear your thoughts on github.com/pytorch/vision…

The latest release of TorchVision has a feature extraction utility. Super handy - check it out. Thanks Francisco Massa and team for helping put this together.

Looking forward to using this new functionality to allow more flexible feature extraction in timm -- especially for vision transformers and mlp models. Thanks Alexander Soare and Francisco Massa