This release of SWE-bench Verified feels to me like OpenAI will launch a new model that is highly capable in coding & math, a competitor to Sonnet 3.5. Only time can tell if 🍓/ Q* + larger model works like we think it will.

sus-column-r on the arena was Grok 2. Competitive with Claude and GPT! x.ai/blog/grok-2

Which are the best open SOTA datasets for finetuning and DPO that is competitive with instruct models from Meta and Mistral?

Real question: is OpenDevin or alternatives actually as good as the real Devin? Not just on the benchmark, but generally?

Imagine if Antrophic releases 3.5 Opus as the original 3.5 Sonnet and the new Sonnet 3.5 is just a quantized and aligned version of the original

lmsys continues to innovate in the hard field of measuring natural language in a fair manner. still, is the benchmark saturated or do we need GPT-5 / Claude 4 before we see a noticeable difference between the current gen and the next gen we want

It’s amazing to me that so many companies have been built on the Llama model series Llama also helps bring competition to OpenAI, Google, etc. and helps bring down prices and overall improves the ecosystem Llama is just a rerun of the Linux strategy

Best repositories for effective distillation from a larger to smaller model?

Sounds to me like we need better open weight coding models.

I wonder how they split this amongst the employees? I would like to see how they scale to such a large cluster, utilizing the whole thing efficiently

The more examples I see from Grok 2 and how it aces things, the more I am impressed because it was built extremely fast. xAI team really nailed this

If the $2k/month subscription from OpenAI is true, that means they generate 100x more tokens per token in the output. Imagine this on a 3T param model🤯 That would be similar to scaling Quiet-STaR with 96 thoughts and 48 ahead tokens. Looks like performance could keep improving.

Let's call this strawberry-mini. The method is not actively generating extra tokens, but it does seem to start by generating a rationale between <thinking> tags. Could be a useful technique!

An open invitation to the cracked Triton optimization engineers: Go optimize the new AWQ Triton kernel so that are fellow AMD users can benefit massively. Tricks to optimize can be found in this post: pytorch.org/blog/accelerat…

I'm quantizing mattshumer/Reflection-Llama-3.1-70B with the AWQ algorithm. Would be great if Matt Shumer can release the instruction dataset so we can use a better calibration dataset

AWQ Reflection Llama 3.1 70B released! I have not evaluated it but assume it works out of the box huggingface.co/casperhansen/r…