Registration is now open for an exciting workshop organized by Aditi Krishnapriyan and Jennifer Listgarten at the Simons Institute for the Theory of Computing June 10th-14th in Berkeley, AI≡Science: Strengthening the Bond Between the Sciences and Artificial Intelligence. simons.berkeley.edu/workshops/aisc…

Handling long context in LLMs is expensive, but can we cut the cost by learning them offline for a specific set/genre of documents?

Introducing LLoCO, our new technique that learns documents offline through context compression and in-domain finetuning using LoRA, which archives…

New paper from @Berkeley_AI on Autonomous Evaluation and Refinement of Digital Agents!

We show that VLM/LLM-based evaluators can significantly improve the performance of agents for web browsing and device control, advancing sotas by 29% to 75%.

arxiv.org/abs/2404.06474 [🧵]

📢Excited to release GoEx⚡️a runtime for LLM-generated actions like code, API calls, and more. Featuring 'post-facto validation' for assessing LLM actions after execution 🔍 Key to our approach is 'undo' 🔄 and 'damage confinement' abstractions to manage unintended actions &…

🚀 Introducing RouterBench, the first comprehensive benchmark for evaluating LLM routers! 🎉
A collaboration between Martian and Prof. Kurt Keutzer at UC Berkeley, we've created the first holistic framework to assess LLM routing systems. 🧵1/8

To read more:…

'Why don't we have better robots yet?': just posted on the TED Talks home page under Newest Talks (3rd row from the top) with links to PiE Robotics and Forbes article on art by Ben Wolff Benjamin Wolff @Berkeley_AI UC Berkeley Berkeley Engineering TED.com

New BAIR blog post on modeling extremely large images, by Ritwik Gupta 🇺🇦 Jitendra MALIK trevordarrell and more!: bair.berkeley.edu/blog/2024/03/2…

Access to *diverse* training data is a major bottleneck in robot learning. We're releasing DROID, a large-scale in-the-wild manipulation dataset. 76k trajectories, 500+ scenes, multi-view stereo, language annotations etc
Check it out & download today!

💻: droid-dataset.github.io

Humanoids 🤖 will do anything humans can do. But are state-of-the-art algorithms up to the challenge?

Introducing HumanoidBench, the first-of-its-kind simulated humanoid benchmark with 27 distinct whole-body tasks requiring intricate long-horizon planning and coordination.

🧵👇

U give me: a bunch of unlabeled data.

I give u: AI-generated labels.

Result: a massive, but biased, val set.

We use PPI to correct the bias, giving unbiased evaluations with better precision 🚀

arxiv.org/abs/2403.07008

Experiments on GPT-4 and ResNets, using lmsys.org :)

Looking to hire top AI talent?

We've compiled a list of the brilliant Berkeley AI Research Ph.D. Graduates of 2024 who are currently on the academic and industry job markets. (Thanks to our friends Stanford AI Lab for the idea!)

Check it out here:
bair.berkeley.edu/blog/2024/03/1…

We know LLMs hallucinate, but what governs what they dream up? Turns out it’s all about the “unfamiliar” examples they see during finetuning

Our new paper shows that manipulating the supervision on these special examples can steer how LLMs hallucinate

arxiv.org/abs/2403.05612
🧵

Do brain representations of language depend on whether the inputs are pixels or sounds?

Our Communications Biology paper studies this question from the perspective of language timescales. We find that representations are highly similar between modalities! rdcu.be/dACh5

1/8

🚀 Thrilled to share our CVPR 2024 paper: Self-correcting LLM-controlled Diffusion Models (SLD)!

SLD can automatically edit any image or fix text-to-image misalignments across any generative model like #DALLE3 and #SDXL - no extra training is needed.

youtube.com/watch?v=PxoOl9…

From your cell phone to your TV, images and videos are now captured in 4K resolution or better. Vision methods, however, opt to downsize or crop them, losing information. We introduce xT, our framework to model large images end-to-end on contemporary GPUs! ai-climate.berkeley.edu/xt-website/

Achieving bimanual dexterity with RL + Sim2Real!

toruowo.github.io/bimanual-twist/

TLDR - We train two robot hands to twist bottle lids using deep RL followed by sim-to-real. A single policy trained with simple simulated bottles can generalize to drastically different real-world objects.

I'm honored to be included in this amazing cohort of Schmidt Sciences AI2050 fellows.

Grateful to all the students and collaborators, whose efforts towards a comprehensive foundations of AI and ML that accounts for social and strategic considerations, this award also recognizes.

we cast real-world humanoid control as next token prediction; our approach enables joint training with youtube videos and walks in sf

Congratulations to Berkeley AI Research member Soufiane Hayou on receiving the Gradient AI Research Fellowship!

gradient.ai/blog/soufiane-…

Introducing Mirage: Zero-shot transfer of visuomotor policies to unseen robot embodiments 🤖

With Mirage, you can train a policy on one robot and deploy it on a different one that it has never seen, with no additional data or training! 🧵👇 (1/8)

🌐 robot-mirage.github.io