Glad to see MMMU being integrated into HELM. Gemini 1.5 Pro working (slightly) better than GPT-4V is aligned with our experience in using these models in various vision-language tasks

Alert 🚨🚨：
We have released our adversarial suffixes generators (AmpleGCG-series models) on Huggingface (huggingface.co/osunlp/AmpleGC…).
Our generated adversarial suffixes on AdvBench and MaliciousInstruct can be accessed via Google form (docs.google.com/forms/d/1P8hxs…).
In light of ethical…

The absolute number is meaningless when it comes to publications, but this is a really proud advisor moment: We OSU NLP Group started submitting to ML conferences last year. In the past cycle, we had 3 NeurIPS, 5 ICLR, and 4 ICML accepted. How lucky I am to work with so many amazing…

I will be attending ICLR2024 in Vienna next week and presenting our three accepted papers, including 'Knowledge Conflicts in the RAG Scenario', 'Efficient Instruction Tuning Dataset Construction', and 'A Complex Planning Benchmark for Language Agents'.

See the details 👇

TravelPlanner has been accepted to ICML 2024. See you in Vienna！

99% attack success rate on GPT-3.5, perhaps the best public jailbreaking method, check it out

I observe the same and thought a bit about that. I'm actually sympathetic to this 'abuse' of 'reasoning'.

Humans have multiple and relatively distinct (though still overlapping) cognitive faculties such as perceptual inferences, intuitive inferences, and reasoning (in the…

In case you can't read this blog post about startups and research on Medium, I've posted a version to the Perceiving Systems blog also: perceiving-systems.blog/en/post/ai-sta…

Quoting Yi Ma 'It is industry's job to find how to do better, but academia is to find out how to do it right.' While I think there're lots of good industry research doing things right, when it comes to reseach on agents, I do think academia has unique freedom to explore how…

We're having a big event on agents at CMU on May 2-3 (one week from now), all are welcome! cmu-agent-workshop.github.io

It will feature:
* Invited talks from Alane Suhr / suhr @ sigmoid . social Yu Su Xinyun Chen Maarten Sap (he/him) and Chris Paxton
* Posters of cutting edge research
* Seminars and hackathons

As always, a pleasure to read Yao Fu’s blog!

Llama3 was trained on 15 trillion tokens of public data. But where can you find such datasets and recipes??

Here comes the first release of 🍷Fineweb. A high quality large scale filtered web dataset out-performing all current datasets of its scale. We trained 200+ ablation…

Talking to many junior faculty members and students in AI lately. Many seem to be somewhat lost with all the seemingly fast progresses made by the industry. My suggestion to them is: It is industry's job to find how to do better, but academia is to find out how to do it right.

We evaluate Core on standard benchmarks for both text and multimodal, along with a blind third-party human evaluation.

These folks Tao Yu ✈️ ICLR 2024 Tianbao Xie✈️ICLR etc. are serious when it comes to agent benchmarks. Excited to have an agent benchmark with an OS simulator to play with!

In this issue: LLMs in the Imaginarium: Tool Learning through simulated trial and error; Benchmarking LLMs across languages, modalities, models and tasks; Training audio captioning models without audio. msft.it/6016cIfvY

I whole-heartedly second Nando de Freitas 🏳️‍🌈’s recommendation. This is THE book I recommend to my students OSU NLP Group interested in getting a first conceptual framework about what is intelligence and how it comes about from evolution.

🔍 In the past year, there has been a surge in the release of open-source LLMs, making them easily accessible and showing strong capabilities. However, the exploration of their trustworthiness remains much limited, compared to proprietary models. A natural question to ask is:…

Counter narratives directly challenge hate speech while limiting free speech infringement concerns. Automatic generation approaches are crucial due to the constant influx of toxic content online; yet, evaluating these systems is challenging. Can LLM-as-a-Judge strategies allow…