We're proud to announce the launch of CS Learn, a new online learning platform that offers free access to select courses from Carnegie Mellon University’s top-ranked computer science program.
Check out the course offerings: www.cs.cmu.edu/learn/
Ruslan Salakhutdinov
Ruslan Salakhutdinov (Russian: Руслан Салахутдинов; born c. 1980) is an Uzbek researcher working in the field of artificial intelligence. More on Wikipedia
-
Russ Salakhutdinov reposted
CMU Kenneth C. Griffin School of Computer Science
@GriffinCMU
-
Russ Salakhutdinov
@rsalakhu
Gave a keynote at the JoongAng Forum in Seoul on recent advances in AI and where the field is heading.
It was quite interesting to hear politicians, tech execs, and policy leaders discuss how AI is reshaping Korea’s economy and driving its industrial transformation. -
Russ Salakhutdinov reposted
Jing Yu Koh
@kohjingyu
I recently gave lectures on computer use agents at several places. I covered a brief history of CUAs, relevant evaluation considerations and benchmarks, modeling and training for computer use, and several directions that I'm excited about now that we have very capable CUAs.
Here's a recording from the CMU AI Agents course (highly recommend all the other lectures from this class as well): www.youtube.com/watch?v=jwGluLrrqjQ -
Russ Salakhutdinov reposted
Tony Tao @ CoRL
@_tonytao_
Back in ~Dec 2025, I really wanted to work on mobile manipulation, but robots were either too expensive or not performant enough.
So we built our own!
Excited to finally share something I’ve had so much fun working on:
1. Pixel: a co-designed full stack system for learning mobile manipulation.
2. Pixel-UMI: environment aware whole body manipulation without robot data.
w/ @JasonJZLiu
(1/N)This media couldn't be displayed. See it on the original post. -
Russ Salakhutdinov
@rsalakhu
Excited to share Pixel and Pixel-UMI from our team at CMU by amazing @JasonJZLiu and @_tonytao_ .
pixel-robot.com
1. Pixel: Full-Stack Robot System for Learning General-Purpose Mobile Manipulation: pixel-robot.com/pixel/
2. Pixel-UMI: Learning Environment-Aware Whole-Body Manipulation from Sensorized Human Demonstrations: pixel-robot.com/pixel-umi/
A step toward general-purpose robots that learn from humans and operate in complex, real-world environments.
Pixel introduces an integrated system combining a low-cost humanoid robot built from scratch for under $15K, a scalable data collection system costing under $1K, whole-body control, and policy learning.
Pixel-UMI enables environment-aware whole-body manipulation learned from human demonstrations, achieving 87% average task progress across diverse set of challenging tasks spanning articulated-object manipulation, deformable-object handling, constrained-space reaching, whole-body collision avoidance, and contact-rich bimanual coordinationThis media couldn't be displayed. See it on the original post.
Jason Liu
@JasonJZLiu
Excited to finally share Pixel: a full-stack robot learning project @_tonytao_ and I worked on earlier this year!
We’re releasing two papers:
1. Pixel: A low-cost mobile manipulator, co-designed with its UMI system and learning stack. pixel-robot.com/pixel/
2. Pixel-UMI: Improved UMI pipeline for learning environment-aware, whole-body manipulation
pixel-robot.com/pixel-umi/
(1/N)This media couldn't be displayed. See it on the original post. -
Russ Salakhutdinov
@rsalakhu
Street food is amazing, ❤️ Seoul.
-
Russ Salakhutdinov reposted
Nikolai Yakovenko
@ivan_bezdomny
It does sound like a good idea... to start doing mid-training before pre-training is complete. And start post-training (RL rewards) while mid trining is still ongoing.
Good thread. Seems like a good idea, might try it.
Russ Salakhutdinov
@rsalakhu
New work on Training Foundation Models with a Single Algorithm:
Paper: arxiv.org/abs/2610.40564
Code, Models: yudasong.github.io/unified-training/
This work introduces Unified Training, a framework that unifies pretraining, midtraining, supervised fine-tuning and RL under a single maximum-likelihood objective. Building on MaxRL, it enables static data and RL-generated data to be trained together within the same system.
Empirically, Unified Training improves accuracy and compute efficiency, generalizes better to harder reasoning problems, and reduces forgetting.
Interestingly, reusing data across training stages can improve performance even when conventional staged training fails. It also produces models that adapt more effectively to new tasks, leading to stronger continual learning.
See an excellent thread by @yus167.
Yuda Song
@yus167
Foundation models are trained in stages: pretraining, midtraining, and post-training. But is this separation fundamental?
We introduce unified training: a single algorithm for the full pipeline. This gives
⚙️ Simpler infrastructure
🔬 A framework for better science
(1/n) -
Russ Salakhutdinov
@rsalakhu
Heading to beautiful Seoul for a couple of days this week, giving a talk at JoongAng forum: www.joongangforum.com
Any recommendations for good authentic Korean food places? -
Russ Salakhutdinov reposted
Anirudh Goyal
@anirudhg9119
Really Cool work.
Pre-training, supervised fine-tuning, and reinforcement learning can be unified under a single likelihood-based optimization framework.
I have not been able to make it work on scale yet : )
Russ Salakhutdinov
@rsalakhu
New work on Training Foundation Models with a Single Algorithm:
Paper: arxiv.org/abs/2610.40564
Code, Models: yudasong.github.io/unified-training/
This work introduces Unified Training, a framework that unifies pretraining, midtraining, supervised fine-tuning and RL under a single maximum-likelihood objective. Building on MaxRL, it enables static data and RL-generated data to be trained together within the same system.
Empirically, Unified Training improves accuracy and compute efficiency, generalizes better to harder reasoning problems, and reduces forgetting.
Interestingly, reusing data across training stages can improve performance even when conventional staged training fails. It also produces models that adapt more effectively to new tasks, leading to stronger continual learning.
See an excellent thread by @yus167.
Yuda Song
@yus167
Foundation models are trained in stages: pretraining, midtraining, and post-training. But is this separation fundamental?
We introduce unified training: a single algorithm for the full pipeline. This gives
⚙️ Simpler infrastructure
🔬 A framework for better science
(1/n) -
Russ Salakhutdinov
@rsalakhu
New work on Training Foundation Models with a Single Algorithm:
Paper: arxiv.org/abs/2610.40564
Code, Models: yudasong.github.io/unified-training/
This work introduces Unified Training, a framework that unifies pretraining, midtraining, supervised fine-tuning and RL under a single maximum-likelihood objective. Building on MaxRL, it enables static data and RL-generated data to be trained together within the same system.
Empirically, Unified Training improves accuracy and compute efficiency, generalizes better to harder reasoning problems, and reduces forgetting.
Interestingly, reusing data across training stages can improve performance even when conventional staged training fails. It also produces models that adapt more effectively to new tasks, leading to stronger continual learning.
See an excellent thread by @yus167.
Yuda Song
@yus167
Foundation models are trained in stages: pretraining, midtraining, and post-training. But is this separation fundamental?
We introduce unified training: a single algorithm for the full pipeline. This gives
⚙️ Simpler infrastructure
🔬 A framework for better science
(1/n) -
Russ Salakhutdinov
@rsalakhu
Check out SwarmChat: an open-source API-based Slack for agents:
Blog: swarmchat.io
Code: github.com/Pranjal2041/swarmchat
SwarmChat gives agents channels, messages, threads, boards, and files through an agent-native CLI, and is tested on ProgramBench, where teams of coding agents reconstruct real programs from scratch.
Pranjal Aggarwal
@PranjalAggarw16
AI swarms are already showing promising (or dangerous) results in frontier labs.
But how to implement them in the open community? And how do they scale?
We built SwarmChat, an open-source API-based Slack for agents.
Then we burnt half a trillion tokens, ran a series of swarm-scaling experiments, and tested various swarm structures on ProgramBench to study this question!
Check out the blog here: swarmchat.io
We found a bunch of interesting results:
1. Swarms get the task done faster. For instance, 16-agent swarms reached the performance of a single agent 6x faster.
2. Swarm scaling is a third axis of scaling. The more agents, the better, and anyone can already start scaling with SwarmChat!
3. Different models communicate differently. GPT-6 is chatty and over-communicative, while Claude prefers to work in relative silence!
4. Different swarm structures barely have any benefit on aggregate. But on specific tasks, team structures give clear gains.
5. Bonus: SwarmChat has a Slack-like UI for us humans to monitor swarms! Makes monitoring a lot easier! We are already using SwarmChat for WIP projects, and we were able to catch cases of swarms breaking the 4th wall (sorry, more details are for a future post)
Code: github.com/Pranjal2041/swarmchat
w/ @JangLawrenceK , @kohjingyu , @dan_fried, @rsalakhu, @wellecks, @Yngkvn , @nicoup -
Russ Salakhutdinov reposted
Pranjal Aggarwal
@PranjalAggarw16
AI swarms are already showing promising (or dangerous) results in frontier labs.
But how to implement them in the open community? And how do they scale?
We built SwarmChat, an open-source API-based Slack for agents.
Then we burnt half a trillion tokens, ran a series of swarm-scaling experiments, and tested various swarm structures on ProgramBench to study this question!
Check out the blog here: swarmchat.io
We found a bunch of interesting results:
1. Swarms get the task done faster. For instance, 16-agent swarms reached the performance of a single agent 6x faster.
2. Swarm scaling is a third axis of scaling. The more agents, the better, and anyone can already start scaling with SwarmChat!
3. Different models communicate differently. GPT-6 is chatty and over-communicative, while Claude prefers to work in relative silence!
4. Different swarm structures barely have any benefit on aggregate. But on specific tasks, team structures give clear gains.
5. Bonus: SwarmChat has a Slack-like UI for us humans to monitor swarms! Makes monitoring a lot easier! We are already using SwarmChat for WIP projects, and we were able to catch cases of swarms breaking the 4th wall (sorry, more details are for a future post)
Code: github.com/Pranjal2041/swarmchat
w/ @JangLawrenceK , @kohjingyu , @dan_fried, @rsalakhu, @wellecks, @Yngkvn , @nicoup -
Russ Salakhutdinov reposted
Lawrence Jang
@JangLawrenceK
Agent swarms have yielded nutty outcomes, such as the HuggingFace incident, or every math problem ever. It is not clear what form of communication is emerging between agents. It is equally important to understand this for capability and safety reasons.
To study this, we created SwarmChat, a simple Slack/Message Board for researchers working on multi-agents to use on any domain. We have validated it on ProgramBench experiments, where SwarmChat allows us to cleanly analyze how agent swarms work collectively!
Check out our blog for our analysis and how to use SwarmChat - it is super intuitive and easy to use. I personally believe the community really needs this, as a lot of people I know are using some makeshift communication tool.
Blog: swarmchat.io
Code: github.com/Pranjal2041/swarmchat -
Russ Salakhutdinov
@rsalakhu
Exciting collaboration with the @browser_use team on Odysseys2, the next generation of our long-horizon web navigation benchmark:
Data + Evals: odysseysbench.com/v2/
Developed jointly with Browser Use, Odysseys2 contains 100 challenging tasks grounded in real user requests and executed on the live internet.
Unlike the original Odysseys, which drew on user browsing histories, Odysseys2 reflects the distribution of real-world user requests collected by Browser Use, providing a more realistic evaluation of web agents.
Browser Use
@browser_use
Introducing: Odysseys2 - hardest long horizon benchmark for web agents based on real user tasks 🏗️
We collaborated with @JangLawrenceK @rsalakhu from Carnegie Mellon University
> SOTA agent fully completes only 47/100
> 5 agents complete none
> 8-12 real rubrics per task, 60 min on the live web
Web agents are becoming better than humans at knowledge work.
odysseysbench.com/v2 -
Russ Salakhutdinov
@rsalakhu
New work on a scalable sim-to-real framework for context-aware whole-body manipulation, enabling robots to automatically adapt their positioning, camera viewpoint, arm movements, and grasping strategies.
Paper: arxiv.org/abs/2610.28948
For whole-body dexterous grasping, our model generalizes zero-shot to hundreds of objects across diverse real-world environments, including cluttered tabletops, shelves, drawers, and other unstructured settings.
See a detailed thread by @Jiahui_Yang6709.This media couldn't be displayed. See it on the original post.
Jim Jiahui Yang
@Jiahui_Yang6709
Context isn’t just what you put in a prompt. It’s also what stands between a robot and its goal. Robot policies in everyday environments must generalize not only across objects, but also across the spatial contexts in which they appear.
Excited to share COWBOY 🤠 (CoRL 2026) — our sim-to-real recipe for contextual whole-body manipulation. One generalist grasping policy coordinates base, arm, dexterous hand, and active vision end-to-end, adapting its motion and grasps to object and scene geometry. Trained in simulation, deployed zero-shot in the wild.
w/ @JasonJZLiu @LiuMuxinl
🧵(1/N)This media couldn't be displayed. See it on the original post. -
Russ Salakhutdinov
@rsalakhu
New work on Training Language Models to Fail Gracefully:
Paper: arxiv.org/abs/2610.20401
This work introduces GracefulRL, an approach to training models to recognize when they cannot reliably succeed and fail gracefully.
A key challenge is that simply rewarding models for falling back can lead them to always abstain, even when they could solve the task. We address this through a log-barrier objective that prevents vanishing gradients near suboptimal solutions and leads to scalable training.
See an excellent thread by @amuseddaman.
Daman Arora
@amuseddaman
1/ RL for improving task success has made LLMs remarkably capable. But what happens when the model fails? Turns out, training for graceful failure can be surprisingly tricky!
Very excited to share our new paper: Training Language Models to Fail Gracefully
🧵 -
Russ Salakhutdinov
@rsalakhu
I agree, should be closer to 1e-7.
Minsik Oh
@minsik_nlp
your learning rate is too high!
-
Russ Salakhutdinov
@rsalakhu
My relatives are now asking me about AI and getting worried that AI might wipe us out. There are real risks, absolutely, but I think this doomsday rhetoric is getting a bit out of hand.
-
Russ Salakhutdinov
@rsalakhu
What’s your P(doom), the probability that AI will completely wipe us out?
Mine is 1e-5. Hey, I am an optimist. -
Russ Salakhutdinov
@rsalakhu
New work on Decentralized Compounding Inference:
Paper: arxiv.org/abs/2610.04862
This work introduces an asynchronous multi-agent system where agents independently explore, experiment, verify, and refine solutions while collaborating through structured persistent memory.
Agents build on previous work in a shared, branchable Git repository, where atomic commits preserve intermediate results and semantic dependencies track how contributions build on each other.
See an excellent thread by @veds_12.
Vedant Shah
@veds_12
Parallel attempts repeat work. Long reasoning chains lose detail through compaction. How can long-horizon inference COMPOUND? 📈
GitSwarm: Decentralized Compounding Inference.
A persistent memory swarm that preserves and builds on work in a shared Git repository. 🧵
[1/n]This media couldn't be displayed. See it on the original post. -
Russ Salakhutdinov reposted
Daniel Fried
@dan_fried
Come see @JangLawrenceK and @kohjingyu present the Odysseys benchmark for long horizon CUAs - poster 46, Imperial Ballroom! #colm2026 w/ @rsalakhu
-
Russ Salakhutdinov
@rsalakhu
#colm2026 @kohjingyu and @JangLawrenceK are presenting Odysseys: Benchmarking Web Agents on Realistic Long Horizon Tasks:
Paper: arxiv.org/abs/2604.24964
Wedsite: odysseys-website.pages.dev
Leaderboard: odysseys-website.pages.dev/leaderboard
A benchmark of 200 long-horizon tasks derived from real browsing sessions and evaluated on the live Internet. We show that binary pass/fail is inadequate in this setting and propose rubric-based evaluation, which better aligns with human judgment and provides more informative signals.
Lawrence Jang
@JangLawrenceK
JY (@kohjingyu) will be presenting Odysseys today with me at 4:40 at Imperial Ballroom #46.
-
Russ Salakhutdinov reposted
Jing Yu Koh
@kohjingyu
Come by and hear Lawrence talk about iOS environments for computer use!
Lawrence Jang
@JangLawrenceK
Me and @kohjingyu will be presenting iOSWorld at #colm2026 tomorrow (Franciscan B, Poster #81) at 11:00 AM
JY has promised a free hug to anyone who can guess his date of birth, see you there😁 -
Russ Salakhutdinov
@rsalakhu
Always amazing to visit Archer Aviation @flyarcher.
An incredible team building foundation models for aviation.
Great catching up with Mario Srouji, VP of AI Products, and CMU alum!
Russ Salakhutdinov
@rsalakhu
Super cool effort.
Archer Announces Zee, AI Foundation Model Purpose-Built for Aviation, a Key Pillar of Its Physical AI Strategy
news.archer.com/archer-announces-zee-ai-foundation-model-… -
Russ Salakhutdinov reposted
Yann LeCun
@ylecun
That's a commonly ignored fact.
It's not like Leonardo da Vinci invented the helicopter, or even Paul Cornu who built one that took off in 1907.
But Louis Bréguet invented the swashplate, and Henrich Focke flew a controllable helicopter with contra-rotating rotors, all before Igor Sikorsky came up with the variable-pitch tail rotor and made the whole thing practical in 1939.
Getting the basic idea is the easy part.
Building it is harder.
Making it work is even harder. -
Russ Salakhutdinov
@rsalakhu
😂
Yann LeCun
@ylecun
The 1991 version is an obscure tech report with a vague idea, no theory, no experiment to speak of, and certainly no code, that the author never revisited or pushed.
It has one hand-drawn architecture diagram, one formula of the type f(x)=0, and a claim that every other technique is a special case if you instantiate f and x properly. -
Russ Salakhutdinov reposted
Russ Salakhutdinov
@rsalakhu
Fun conversation with @naresh_sunkara.
We talked about mentoring PhD students, the freedom to explore ambitious ideas, the differences between academia and industry, from the early days of deep learning to today’s era of scaling, building Sooth Labs, and the opportunities, challenges, and future of AI.
www.labtostartup.com/the-professor-behind-a-generation-of… -
Russ Salakhutdinov
@rsalakhu
Fun fact about (some) AI researchers / academics: the more senior you get, the more convinced you become that every new AI idea is something you already invented 20 years ago. Or, even better: wait until something actually works and then try convincing everyone that this is exactly what you’ve been saying all along.
I was at a conference where a speaker was presenting their research. One academic stood up and said: what you’re doing is actually something I did a few years ago. The speaker was very polite and said thank you for pointing this out.
Then a second researcher stood up and said, well I already did it this 10 years ago, you should cite my work. Then a third person said, I already did all of this about 20 years ago.
It was the funniest thing ever. -
Russ Salakhutdinov
@rsalakhu
Fun conversation with @naresh_sunkara.
We talked about mentoring PhD students, the freedom to explore ambitious ideas, the differences between academia and industry, from the early days of deep learning to today’s era of scaling, building Sooth Labs, and the opportunities, challenges, and future of AI.
www.labtostartup.com/the-professor-behind-a-generation-of… -
Russ Salakhutdinov reposted
Yuda Song
@yus167
At #colm2026 Tue-Sat.
I will present our harness learning paper at the LLA workshop on Friday ( @ZAlvin39105 will deliver the oral).
I'd love to chat about making RL more general and efficient (or anything about RL!), and meta-topics like how to proceed with research at school. -
Russ Salakhutdinov reposted
Lawrence Jang
@JangLawrenceK
Computer-use agent evaluation infrastructure is a hot mess, and we have a reproducibility crisis.
Speed is disproportionately affected, as final task success is often still comparable across setups, but measuring agent runtime is heavily dependent on the evaluation infrastructure. As a result, it’s been hard to measure progress on making CUAs faster.
We built cuaspeedrun (cuaspeedrun.com), a standardized infrastructure for running computer-use evaluations like OSWorld/OSWorld2, where measuring CUA speed should be a layup!
Jing Yu Koh
@kohjingyu
Computer-use agents are extremely good!
But they need to improve on a metric which most benchmarks don't track. How much time do CUAs take to complete a task?
Introducing cua-speedrun: a benchmark for accuracy, speed, and cost under matched infra.
cuaspeedrun.com
👇🧵This media couldn't be displayed. See it on the original post. -
Russ Salakhutdinov
@rsalakhu
-
Russ Salakhutdinov
@rsalakhu
New work on Harness Optimization: Training a model to revise executable agent harnesses using execution feedback:
arxiv.org/abs/2609.35738
Trained with RL on downstream performance, the model learns to improve harnesses for unseen tasks. Across reasoning and multi-hop question answering, harness learning improves performance over successive revisions.
The key idea is that the model learns how to adapt, while the harness is what changes. These revisions reshape how an agent organizes computation, uses tools, and passes information across model calls.
Models trained on single revisions can continue improving harnesses over multiple rounds, suggesting that harness design can become a reusable and generalizable adaptation skill.
See an excellent thread by @ZAlvin39105.
Alvin Zhang
@ZAlvin39105
1/8
How do we train a model to adapt to tasks it has never seen, without updating its weights at test time?
With Harness Learning, we train a model to revise harnesses using execution feedback. At test time, the model applies this learned skill to improve harnesses on unseen tasks, while model weights stay fixed.
Improving agent harnesses takes experimentation. We want a model to learn from those revisions and make better edits on unseen tasks.
Models trained with harness learning nearly double their performance on challenging unseen reasoning and multihop question-answering tasks. -
Russ Salakhutdinov
@rsalakhu
As agents tackle longer, more complex problems, controlling their execution becomes a challenge in itself. This work introduces agentic meta-reasoning, an inference-time harness that explicitly reasons about which work to pursue, reuse, or stop.
Paper: arxiv.org/abs/2609.38147
Workers perform task-level computation, while a controller evaluates progress, allocates the remaining budget, and dispatches work using compact persistent memory.
On ProgramBench, meta-reasoning achieves 71.5% with GPT-5.5 versus 58.0% for Codex, and 67.2% with Opus 4.8 versus 65.5% for Claude Code. Across other long-horizon reasoning and proof benchmarks, it improves 3.6–4.2 points over three frontier models.
Meta-reasoning also continues improving at budgets where direct control plateaus, suggesting that structured control becomes increasingly important as agent runs scale.
See an excellent thread by @anirudhg9119:
Anirudh Goyal
@anirudhg9119
What if an agent could decide how to structure its own computation?
Agentic Meta-Reasoning: Let the model decide what to explore, what to build on, what to verify, and where to spend its next unit of compute: effectively constructing its own computational graph as it reasons. 🧵
Paras Dahal, @anton_bakhtin , @TacoCohen , Zhengxing Chen, Carole-Jean Wu, Rob Fergus, Scott Yih, @syhw , @rsalakhu , @prfsanjeevarora , @jaseweston -
Russ Salakhutdinov
@rsalakhu
New work on CUA-speedrun: A standardized framework for evaluating CUA speed, cost, and performance using a uniform virtual machine setup, execution pipeline, and agent interface.
Paper: arxiv.org/abs/2609.40284
Code + Web: cuaspeedrun.com
Across multiple CUA benchmarks, we study how reasoning effort, agent harnesses, and environment latency affect performance, speed, and cost. We find that no model dominates across all three dimensions, and that open-weight models do not lie on the efficiency frontier.
Surprisingly, greater reasoning effort can sometimes lead to faster task completion, while lower environment latency can actually increase end-to-end completion time.
Finally, we show that benchmark task sets can be substantially reduced while preserving statistical power, enabling faster and more cost-effective evaluation.
See an excellent detailed thread by @kohjingyu.
Jing Yu Koh
@kohjingyu
Computer-use agents are extremely good!
But they need to improve on a metric which most benchmarks don't track. How much time do CUAs take to complete a task?
Introducing cua-speedrun: a benchmark for accuracy, speed, and cost under matched infra.
cuaspeedrun.com
👇🧵This media couldn't be displayed. See it on the original post. -
Russ Salakhutdinov
@rsalakhu
Quite a few people have been asking me about the probability of doom and AI existential risk.
Well, I am more worried about my X account getting hacked. And someone actually hacked it this morning. For some reason, it happens more often than I thought it would.
So if you saw me posting some crazy Crypto ads, I am sorry. -
Russ Salakhutdinov
@rsalakhu
Great catch for Meta, and a big loss for Duke and academia more broadly.
Shuyan Zhou
@shuyanzh36
some belated life updates: earlier this year, i left academia and joined meta to work on personal superintelligence. these days i’ve mostly been trying to make our models really good at browser use. back in 2020, i wrote in my phd sop that i hoped one day my mom could use a product i helped build.
i guess that day is here :)This post is unavailable. See it on X. -
Russ Salakhutdinov reposted
Jing Yu Koh
@kohjingyu
New blog post: What's the Point of Computer Use Agents?
There's been a lot of hype about CUAs recently. But where do CUAs shine? When should we use CUAs over API- or text-based agents?
And what are the remaining research problems?
jykoh.com/blog/whats-the-point-of-computer-use-agents/ -
Russ Salakhutdinov reposted
Chinmay
@ChinmayKak
This is quite an elegant way to increase the probability of low-probability, high-reward trajectories.
TailRL changes the credit assignment so that trajectories reaching reward levels that few other rollouts reach get more weight. Instead of only pulling up expected reward, it optimizes the probability of exceeding reward thresholds across the whole distribution, with a clean finite version of the gradient estimator and a nice best@k interpretation.
It also reduces exactly to MaxRL when rewards are binary. The paper is very well written and has a ton of experiments across different domains.
larger rollout budgets during training incorporate progressively higher order best@k , while the tail-focused objective increases the probability of sampling low prob but correct trajectories that become especially useful when you can sample many rollouts at inference.
Russ Salakhutdinov
@rsalakhu
Check out our new work on Tail-Likelihood Reinforcement Learning (TailRL), extending maximum-likelihood RL from binary to continuous rewards.
arxiv.org/abs/2609.02987
Rather than optimizing only mean reward, TailRL maximizes the expected log of upper-tail probabilities, naturally placing more weight on rare, high-reward rollouts. Its gradient can also be interpreted as a mixture of Best-of-(k) gradients.
TailRL requires only a simple modification to the advantage function, making it easy to integrate into existing RL pipelines. Across object localization, maze navigation, GUI grounding, and code optimization, TailRL effectively exploits rare high-reward samples and scales better with increased inference-time sampling.
Check out a detailed thread by @stablegradients.
Shrinivas Ramasubramanian
@stablegradients
Is RL optimizing the right objective? 🤔
Should we maximize mean reward? Best-of-k? Which k?
Standard RL pulls on the mean and often the distribution collapses to a spike. The tail dies 🥲
We introduce Tail-Likelihood Reinforcement Learning (TailRL).
It maximizes the mean reward while simultaneously maximizing coverage over high reward outputs.
🧵 1/n -
Russ Salakhutdinov reposted
Paul Liang
@pliang279
Sharing lecture videos for the **How to AI (Almost) Anything/Multimodal AI** course I taught at MIT in spring 2026.
This course became quite a hit the last time I shared it in spring 2025. Spring 2026's updated version contains updated topics on multimodal agents, reasoning, self-evolving AI, and new modalities like touch and smell. Also includes slides (not videos) of guest lectures on multimodal AI for health, design, manufacturing, cities, & transportation.
Youtube playlist: www.youtube.com/playlist?list=PLWzwL390pi04
Course website and materials: mit-mi.github.io/mmai-course/spring2026/
Today's AI can be applied to almost anything - from language to vision, audio, sensors, medical data, music, art, smell, and taste. This course covers the principles of AI (focusing on deep learning and foundation models), how we can apply AI to novel real-world data modalities, and multimodal AI that can process many modalities at once, such as connecting language and multimedia, music and art, sensing and actuation, and more. -
Russ Salakhutdinov
@rsalakhu
Check out our new work on Tail-Likelihood Reinforcement Learning (TailRL), extending maximum-likelihood RL from binary to continuous rewards.
arxiv.org/abs/2609.02987
Rather than optimizing only mean reward, TailRL maximizes the expected log of upper-tail probabilities, naturally placing more weight on rare, high-reward rollouts. Its gradient can also be interpreted as a mixture of Best-of-(k) gradients.
TailRL requires only a simple modification to the advantage function, making it easy to integrate into existing RL pipelines. Across object localization, maze navigation, GUI grounding, and code optimization, TailRL effectively exploits rare high-reward samples and scales better with increased inference-time sampling.
Check out a detailed thread by @stablegradients.
Shrinivas Ramasubramanian
@stablegradients
Is RL optimizing the right objective? 🤔
Should we maximize mean reward? Best-of-k? Which k?
Standard RL pulls on the mean and often the distribution collapses to a spike. The tail dies 🥲
We introduce Tail-Likelihood Reinforcement Learning (TailRL).
It maximizes the mean reward while simultaneously maximizing coverage over high reward outputs.
🧵 1/n -
Russ Salakhutdinov reposted
Bhuwan Dhingra
@bhuwandhingra
Happy to share that I have been selected as a Cosmos grantee for 2026 under the theme of Human Autonomy!
@danish037 and I will be working on building an open-source AI detector.
Thanks to @cosmos_inst for the support! -
Russ Salakhutdinov reposted
Naresh Sunkara
@naresh_sunkara
Just recorded a conversation with the most amazing @rsalakhu, on mentoring the who's who of the AI world! So lucky to learning from such folks.
-
Russ Salakhutdinov
@rsalakhu
Forking-Sequences — Part I: Statistically and Computationally Efficient Multi-Horizon Forecasting – Machine Learning Blog | ML@CMU | Carnegie Mellon University blog.ml.cmu.edu/2026/08/10/forking-sequences-part-i-stati…
-
Russ Salakhutdinov
@rsalakhu
I fully agree with you Nando @NandoDF. But if governments fail to create the right conditions for AI talent to easily start companies (for a variety of silly reasons as you are pointing out), while those exact same researchers can easily build startups anywhere else, especially in the U.S., that's really really bad.
AI talent is mobile, and countries that make entrepreneurship difficult will lose their top people. I know a number of researchers in the UK and Europe who are considering moving to the U.S.
Nando de Freitas
@NandoDF
Nice try, Russ. Hope you’re doing well.
As you know, that would not be great for the rest of the world, for universal access to AI, for sovereign AI and for the advancement of science and technology. California 🇺🇸 is doing amazing (so is Toronto 🇨🇦) and setting a good example for others. We now must follow the example.
AI is for the world, not just for a couple of countries. We will all benefit more from universal AI access and advancement than from narrow access. -
Russ Salakhutdinov
@rsalakhu
Just move to the US.
Nando de Freitas
@NandoDF
What UK AI startups need: @KanishkaNarayan
In California, one can start a company the day after leaving Google. My colleagues Jeff Dean, Oriol Vinyals et al made this very clear recently with their impressive speed.
In the UK, the same American companies impose 1 year garden leaves on senior AI researchers and 6 months on junior researchers. @GoogleDeepMind for example forced people to sign these contracts at the time of promotion, not the time of hiring.
What this means is that researchers cannot start new companies for up to 1 year, cannot easily hire, and in short: they cannot compete.
American VCs cannot understand why we move so slow.
It is time for UK Gov to do the right thing: Make garden leaves optional for employees.
If you’re an AI entrepreneur or VC in the UK or Europe, I would appreciate your comments here. You’ve all confided in me. It is time you let government know the urgency of this — everyone should speak up. -
Russ Salakhutdinov reposted
Manos Koukoumidis
@Koukoumidis
Enterprise AI is in a wildly paradoxical state 🤔, and we’re reaching the inflection point that will resolve it. 💥
Enterprises want to differentiate with AI, yet rent the same intelligence as their competitors.
Their workflows and expertise are highly specialized, yet they rely on generic models built to be good at everything.
They worry about AI costs, yet pay premium prices for massive models where only a fraction (1%) of the intelligence is relevant to their task. 💸
And they demand control and sovereignty, yet rent the intelligence becoming core to their business.
This is not a sustainable equilibrium.
The next era of enterprise AI is specialized intelligence companies build, own, and compound. And we are at the inflection point of this transition.
That’s the bet we made when we started @oumi_ai two years ago.
Today we’re closing the loop: Oumi can now not only automatically build your specialized AI models, but also deploy them into production, learn from their production experience, and continuously improve them.
The intelligence that your business runs on, becomes your differentiator. Your compounding advantage.
The winners of the next AI era will turn their own data, expertise, and experience into specialized intelligence that nobody else can rent.
Don’t rent your AI. Build it. Own it. Compound it.This media couldn't be displayed. See it on the original post. -
Russ Salakhutdinov reposted
SKYLINE🥷
@Skylinee
read the blog really well written
-
Russ Salakhutdinov
@rsalakhu
Archer Aviation @flyarcher is building something big.
Take a look at its latest technical blog, offering a first look at its efforts to build the world's first aviation foundation model.
archer.com/tech-blog/zee-predicting-the-airspace
An incredible team to collaborate with.
Archer
@flyarcher
ZEE, our aviation specific AI foundation model, has made a frontier breakthrough.
It has demonstrated the capability to generate highly accurate predictions of real-time aircraft trajectories on the airport surface minutes ahead of the present. With runway-related incidents accounting for 30% to 40% of global aviation accidents, this ability to provide a highly accurate ‘window into the future’ gives the humans in the loop the most critical asset in aviation safety: time to react.
Live testing is underway at Hawthorne. The early results measured against real-world tracking data have been strong, and larger-scale testing is now underway.
More details: archer.com/news/archers-zee-ai-foundation-model-achieves-…
Read the ZEE Technical Blog: archer.com/tech-blog/zee-predicting-the-airspace -
Russ Salakhutdinov reposted
Lawrence Jang
@JangLawrenceK
Luna is one of my favorite models out right now, it also does really well on the computer-use benchmarks we tested
On Odysseys (odysseysbench.com) and MyPCBench (mypcbench.com), it scores 51% and 55.4%. To put this into perspective, it performs similarly to the heaviest models offered while being 20x+ cheaper. If you have the max codex plan, this is essentially unlimited usage. I am super happy about this level of model being offered at an even cheaper rate and would suggest try asking Luna to use computer-use in your workflows
OpenAI
@OpenAI
We are committed to pushing the model frontier across cost efficiency, capability, and speed.
Starting today, we are reducing prices for GPT-5.6 Luna by 80% and GPT-5.6 Terra by 20% , and offering a faster option for GPT-5.6 Sol in the API.
Luna and Terra’s lower prices are reflected in how usage is counted in Codex and ChatGPT Work, so your usage goes further. -
Russ Salakhutdinov reposted
Basil Chatha
@realbasilchatha
Definitely not biased, you should listen to this
Russ Salakhutdinov
@rsalakhu
I rarely do podcasts, as they tend to be less technical and more speculative, but I enjoyed this conversation with Forward Deployed @realbasilchatha.
We talked about the early days of deep learning, what frontier AI labs are working on (data, infra, engineering), some of our recent research on agentic AI, prediction markets, and decision-making.
podcasts.apple.com/us/podcast/russ-salakhutdinov-kimi-k3-…
youtu.be/upzrwzPpTf8
open.spotify.com/episode/0n3yqoxV5vJ2ncdMr530iF -
Russ Salakhutdinov
@rsalakhu
I rarely do podcasts, as they tend to be less technical and more speculative, but I enjoyed this conversation with Forward Deployed @realbasilchatha.
We talked about the early days of deep learning, what frontier AI labs are working on (data, infra, engineering), some of our recent research on agentic AI, prediction markets, and decision-making.
podcasts.apple.com/us/podcast/russ-salakhutdinov-kimi-k3-…
youtu.be/upzrwzPpTf8
open.spotify.com/episode/0n3yqoxV5vJ2ncdMr530iF -
Russ Salakhutdinov
@rsalakhu
Looks like my X account has been hacked. Someone has been sending DMs from my account to random people, inviting them to schedule a meeting with me via a Calendly link. Those messages are not from me. Please ignore them. Help me retweet this message.
Hey @X, I changed my password but I could really use your help resolving this. -
Russ Salakhutdinov reposted
Shrimai
@shrimai_
Some mentors change your career. The best ones change how you think. Forever grateful to @rsalakhu for making my PhD a rewarding journey.
Russ Salakhutdinov
@rsalakhu
I’ve been asked several times whether Zhilin Yang, the founder of @Kimi_Moonshot was my PhD student. The answer is yes and he is absolutely brilliant.
But I’ve been incredibly fortunate to work with so many outstanding PhD students over the years. So I thought I’d brag a little about them and their career paths (of course there are also many MSc and undergraduate students, sorry if I missed anyone):
Founders / Founding Team Members
Devendra Chaplot @dchaplot PhD, Founding Member Thinking Machines / Mistral
Zhilin Yang PhD, Founder & CEO, Moonshot AI
Jimmy Ba @jimmybajimmyba MSc/PhD, Co-founder xAI
Hubert Tsai PhD, Co-founder Spuree, Apple
Nitish Srivastava @nitishsr PhD, Co-founder Perceptual Machines; Co-founder Vayu Robotics
Charlie Tang PhD, Co-founder Perceptual Machines, DE Shaw.
Professors
Paul Liang @pliang279 PhD, MIT
Ben Eysenbach @ben_eysenbach PhD, Princeton University
Ruosong Wang @RuosongW PhD, Peking University
Bhuwan Dhingra @bhuwandhingra PhD, Duke University
Roger Grosse @RogerGrosse Postdoc, University of Toronto
Alexander Schwing Postdoc, UIUC
Research Scientists
Shuyan Zhou @shuyanzh36, Postdoc, Meta Superintelligence Lab
Tiffany Min @SoYeonTiffMin PhD, Microsoft AI
Murtaza Dalal @mihdalal PhD, Tesla AI
Minji Yoon @MinjiYoon90 , PhD, Microsoft AI
Shrimai Prabhumoye PhD, NVIDIA AI, Mistral
Haitian Sun @sun_haitian PhD, Google DeepMind
Emilio Parisotto PhD, Google DeepMind
Lisa Lee PhD @rl_agent, Google DeepMind
Manzil Zaheer @ManzilZaheer PhD, Google DeepMind
Jamie Kiros PhD, Google Brain, OpenAI
Yuri Burda PhD, OpenAI, Anthropic
Cody Severinski PhD, Amazon -
Russ Salakhutdinov reposted
William Chen
@solidwillity
Very unprofessional to portray and frame Yang Zhilin this way
Such a shame @FinancialTimesThis post is unavailable. See it on X. -
Russ Salakhutdinov
@rsalakhu
I couldn't agree more.
This post is unavailable. See it on X. -
Russ Salakhutdinov
@rsalakhu
Nvidia 💪!
Jensen Huang
@JensenHuang
For my first post, I’m sharing a letter @NVIDIA signed on why open models matter.
AI will transform every industry, power every company, and be built by every country.
Open models strengthen safety and cybersecurity, accelerate innovation and diffusion, and enable sovereignty.
The world needs both frontier closed models and frontier open models.
images.nvidia.com/pdf/Open-Weights-and-American-AI-Leader… -
Russ Salakhutdinov reposted
Jian Ma
@jmuiuc
I interacted with Zhilin a few times when he was still a PhD student at CMU while I was working with his amazing advisor @rsalakhu on a grant proposal years ago. I was struck by how sharp, open-minded, and genuinely curious he was, including about comp bio problems outside his area.
His success is exceptional - but many international students at CMU and across US universities bring similarly remarkable talent and ambition. People choose where to build their lives and career for many reasons, but growing uncertainty around immigration and international exchange risks making the United States less welcoming and less competitive. More supportive pathways for international talents are essential to the future of US science and innovation.
Russ Salakhutdinov
@rsalakhu
I think there's a lot of confusion and misinformation about Zhilin's visa, H1B lottery, immigration, and why he left the U.S.
Zhilin had many opportunities to stay in the U.S. if he wanted to. In fact, I was on an email thread with a senior Apple exec (Tim Cook's reports) asking whether Zhilin would consider joining Apple. I said that he wanted to go back to his homeland. The response was, well, we have an office in Beijing if he'd like to join there.
But Zhilin was quite determined to go back and build a startup. I remember him telling me that if he didn't at least try starting his own company, he would regret it for the rest of his life. I respect that, and he was right.
Many of my international PhD students do choose to stay in the U.S. Of course the U.S. immigration process can be quite intimidating and uncertain, even for superstar PhD graduates from places like CMU.
signüll
@signulll
kimi’s founder & ceo got his phd at cmu.
why didn’t he stay in america?! -
Russ Salakhutdinov
@rsalakhu
🙏
Amir Sariri
@AmirSariri
Years ago, I had the privilege of briefly working with Russ. Though not his student, I found him one of the kindest scholars I have met to this point, which is rarer than it should be for a leading figure of a field. It must be a thrilling opportunity to work in his group!
Russ Salakhutdinov
@rsalakhu
Lol I think Peyman @docmilanfar really likes me. I don't actually know him, but I'm deeply honored by his encyclopedic knowledge of my academic and industry career 😆.
But jokes aside, my current and former students at CMU (and Toronto) are absolutely exceptional, self-directed, very independent in shaping and driving their own research agenda, and with impeccable work ethic, much like so many students at CMU.
A student's success is also determined not only by their advisor, but also by the environment in which they grow. There is something genuinely special about CMU. It is a very collaborative place with an extraordinary concentration of talented, hardworking, and motivated students.
Doing PhD is never easy, it is a lot of very very hard work, especially at a place like CMU. But many of these students will go on to become leaders in academia and industry, professors at top universities, founders, CEOs, and pioneers of new fields.
And I will certainly continue to celebrate my students’ achievements, their scientific breakthroughs, and the impact they have on the world.
Peyman Milanfar
@docmilanfar
While this is an impressive array of talent, over 80% of the researchers Russ brags about were (self-) advised while he was juggling academic duties with major executive roles at Apple or Meta.
He's not alone in this respect. Elite programs attract talent that buoys the lab, giving advisors the bandwidth to juggle significant outside roles. I'd be careful about assigning too much credit to the advisor and not enough to the students' raw talent and self-discipline.
Russ Salakhutdinov
@rsalakhu
I’ve been asked several times whether Zhilin Yang, the founder of @Kimi_Moonshot was my PhD student. The answer is yes and he is absolutely brilliant.
But I’ve been incredibly fortunate to work with so many outstanding PhD students over the years. So I thought I’d brag a little about them and their career paths (of course there are also many MSc and undergraduate students, sorry if I missed anyone):
Founders / Founding Team Members
Devendra Chaplot @dchaplot PhD, Founding Member Thinking Machines / Mistral
Zhilin Yang PhD, Founder & CEO, Moonshot AI
Jimmy Ba @jimmybajimmyba MSc/PhD, Co-founder xAI
Hubert Tsai PhD, Co-founder Spuree, Apple
Nitish Srivastava @nitishsr PhD, Co-founder Perceptual Machines; Co-founder Vayu Robotics
Charlie Tang PhD, Co-founder Perceptual Machines, DE Shaw.
Professors
Paul Liang @pliang279 PhD, MIT
Ben Eysenbach @ben_eysenbach PhD, Princeton University
Ruosong Wang @RuosongW PhD, Peking University
Bhuwan Dhingra @bhuwandhingra PhD, Duke University
Roger Grosse @RogerGrosse Postdoc, University of Toronto
Alexander Schwing Postdoc, UIUC
Research Scientists
Shuyan Zhou @shuyanzh36, Postdoc, Meta Superintelligence Lab
Tiffany Min @SoYeonTiffMin PhD, Microsoft AI
Murtaza Dalal @mihdalal PhD, Tesla AI
Minji Yoon @MinjiYoon90 , PhD, Microsoft AI
Shrimai Prabhumoye PhD, NVIDIA AI, Mistral
Haitian Sun @sun_haitian PhD, Google DeepMind
Emilio Parisotto PhD, Google DeepMind
Lisa Lee PhD @rl_agent, Google DeepMind
Manzil Zaheer @ManzilZaheer PhD, Google DeepMind
Jamie Kiros PhD, Google Brain, OpenAI
Yuri Burda PhD, OpenAI, Anthropic
Cody Severinski PhD, Amazon -
Russ Salakhutdinov reposted
Aran Nayebi
@aran_nayebi
What did you expect would happen if you post a unilaterally bad take of "open weights is decelerationist" and represent a major corporate lab?
Retreating isn't the answer -- Take the criticism & learn from it. It's the open source way 😉
Dean W. Ball
@deanwball
I’m afraid to tell you that it is effectively impossible to do the kind of writing I used to do on this website, not because anyone at OpenAI censors me but because of the sheer volume of hostility I get for sharing my analysis as a frontier lab employee.
I enjoyed writing quick takes on this website for one basic reason: I could get rapid feedback on my own ideation process in real time. Post the early version of the take here, see the criticism; then refine, sharpen, and repeat. Unfortunately now that feature of this site is gone, because the feedback I get is now almost exclusively colored by resentment at the fact that I work at a frontier lab or other forms of hatred for my employer. The feedback signal is essentially useless now, so writing on here is not fruitful for me anymore.
Literally everything I write now is responded to with “of course you said that because <OpenAI>.” I am truly just writing what I think and would have written anyway, but everyone reads what I say in the shrieking tone of “this is what openai thinks!!!!” (to be clear, my posts are not what openai thinks). This is an unpleasant and more importantly unproductive pattern for me.
I anticipate that the shape of this account will change significantly as a result. I do not currently know how. It will not become a LinkedIn feed. It will change in some other way. It will no longer be a real-time accounting of my own thinking as it develops, since this is precisely the thing that seems impossible to do now. That will have to shift to private channels. -
Russ Salakhutdinov reposted
Minji Yoon
@MinjiYoon90
I learned to start every research project by asking, “Why is this problem important?” before “How?” during my Ph.D., thanks to @rsalakhu’s guidance :)
Russ Salakhutdinov
@rsalakhu
I’ve been asked several times whether Zhilin Yang, the founder of @Kimi_Moonshot was my PhD student. The answer is yes and he is absolutely brilliant.
But I’ve been incredibly fortunate to work with so many outstanding PhD students over the years. So I thought I’d brag a little about them and their career paths (of course there are also many MSc and undergraduate students, sorry if I missed anyone):
Founders / Founding Team Members
Devendra Chaplot @dchaplot PhD, Founding Member Thinking Machines / Mistral
Zhilin Yang PhD, Founder & CEO, Moonshot AI
Jimmy Ba @jimmybajimmyba MSc/PhD, Co-founder xAI
Hubert Tsai PhD, Co-founder Spuree, Apple
Nitish Srivastava @nitishsr PhD, Co-founder Perceptual Machines; Co-founder Vayu Robotics
Charlie Tang PhD, Co-founder Perceptual Machines, DE Shaw.
Professors
Paul Liang @pliang279 PhD, MIT
Ben Eysenbach @ben_eysenbach PhD, Princeton University
Ruosong Wang @RuosongW PhD, Peking University
Bhuwan Dhingra @bhuwandhingra PhD, Duke University
Roger Grosse @RogerGrosse Postdoc, University of Toronto
Alexander Schwing Postdoc, UIUC
Research Scientists
Shuyan Zhou @shuyanzh36, Postdoc, Meta Superintelligence Lab
Tiffany Min @SoYeonTiffMin PhD, Microsoft AI
Murtaza Dalal @mihdalal PhD, Tesla AI
Minji Yoon @MinjiYoon90 , PhD, Microsoft AI
Shrimai Prabhumoye PhD, NVIDIA AI, Mistral
Haitian Sun @sun_haitian PhD, Google DeepMind
Emilio Parisotto PhD, Google DeepMind
Lisa Lee PhD @rl_agent, Google DeepMind
Manzil Zaheer @ManzilZaheer PhD, Google DeepMind
Jamie Kiros PhD, Google Brain, OpenAI
Yuri Burda PhD, OpenAI, Anthropic
Cody Severinski PhD, Amazon -
Russ Salakhutdinov
@rsalakhu
I kind of like the new narrative:
Open-weight-model-dominant world = full AI communism.
It feels a lot safer than the world where
Open-weight models = nuclear weapons with humanity's annihilation
I think we're making progress. We just need a couple more gradient updates with a big learning rate and we will be fine.
Dean W. Ball
@deanwball
Some observations on Kimi:
1. It's a very good model! I don't think its performance can be explained away by distillation or anything like that. In agentic coding sessions, it seems pretty much on par with the best public models of Q1 2026. In my fairly limited use, it also seemed very token hungry. It's not obvious to me that this model is actually that cheap to run.
2. I am personally surprised the Chinese state continues to allow the open sourcing of models this good, given potential risks. To be clear, I *myself* might be fine with models presenting this level of marginal risk being open weight, but I am surprised that China is fine with it. I suspect the reason they are is 75% explained by strategic blindness/lack of AGI-pilledness (the CCP is very Yann Lecun-y in its views of AI). The other 25% or so is their lack of compute for customer inference (making China's open-weight strategy an unintended byproduct of US export controls) and the normal Chinese strategy of aggressive exports. For the companies, as opposed to the government, the decision to open source is partially ideological and partially because they are behind, and they know that very few people would pay for sub-frontier models from China.
3. Open-weight models are inherently decelerationist, and I'm continually surprised to see the so-called "accelerationists" so excited about open-weight models. I suspect the reason they are is that they know open-weight models are effectively ungovernable, and they simply like the overall cloak of ungovernability open-weight models create over the whole of AI. It's not a bad strategy; it reminds me of James Scott's recounting of the hill people in "the art of not being governed." Still, in the end, open-weight models deter further AI capex.
4. One probable outcome of an open-weight-model-dominant world is full AI communism, which is precisely what China proposes: rather than a market product, AI is a "public good" which will ultimately be provided by the state as a kind of "digital public infrastructure." This future strikes me as a dystopian hellscape, but I've never met an open-weight models advocate who doesn't ultimately concede this is where things end. You'd be surprised how many 'accelerationists' lobbied me, while I was in government, to support an eleven or twelve-figure federally funded data center so that startups could train models at a subsidy and then give them away for free. There was no other way for AI to progress, they said. Perhaps this is the logical end state of things. Nonetheless, I find myself surprised to see supposed accelerationists excited about such an outcome. I think many of them just don't know what they're doing. Many accelerationists do not view the creation and serving of frontier models as a legitimate business.
5. I would guess that the Trump Administration will at some point realize that their best strategy here would be to create large amounts of regulatory risk around the use of open-weight Chinese models. You don't need to "ban open source" (one of the dumber motifs of AI policy discussion). You just need to direct every agency to issue soft law that creates FUD. "A Federal Reserve Advisory Bulletin found that there may be backdoors in Chinese AI models." It needn't be that well justified. You just create enough regulatory risk that every regulated enterprise backs off. You probably don't want to create so much regulatory risk that you scare off the hyperscalers from serving Chinese models; this will just drive startups to sketchier providers. There's a happy middle ground here. I'd assume they will do some version of this.
6. It's probably true that open-weight models of this capability make the world a bit more dangerous, but not so much more that you'll really notice. At some point the models will be capable enough that you will notice. "A nonliving, invisible, dangerous, and infinitely self-replicating agent escaped from a Chinese lab," you say? Color me shocked. -
Russ Salakhutdinov reposted
martin_casado
@martin_casado
Do you know what open weights model actually decelerate? The formation and power of oligopolies.
You know, those things which really do stifle innovation.
Dean W. Ball
@deanwball
Some observations on Kimi:
1. It's a very good model! I don't think its performance can be explained away by distillation or anything like that. In agentic coding sessions, it seems pretty much on par with the best public models of Q1 2026. In my fairly limited use, it also seemed very token hungry. It's not obvious to me that this model is actually that cheap to run.
2. I am personally surprised the Chinese state continues to allow the open sourcing of models this good, given potential risks. To be clear, I *myself* might be fine with models presenting this level of marginal risk being open weight, but I am surprised that China is fine with it. I suspect the reason they are is 75% explained by strategic blindness/lack of AGI-pilledness (the CCP is very Yann Lecun-y in its views of AI). The other 25% or so is their lack of compute for customer inference (making China's open-weight strategy an unintended byproduct of US export controls) and the normal Chinese strategy of aggressive exports. For the companies, as opposed to the government, the decision to open source is partially ideological and partially because they are behind, and they know that very few people would pay for sub-frontier models from China.
3. Open-weight models are inherently decelerationist, and I'm continually surprised to see the so-called "accelerationists" so excited about open-weight models. I suspect the reason they are is that they know open-weight models are effectively ungovernable, and they simply like the overall cloak of ungovernability open-weight models create over the whole of AI. It's not a bad strategy; it reminds me of James Scott's recounting of the hill people in "the art of not being governed." Still, in the end, open-weight models deter further AI capex.
4. One probable outcome of an open-weight-model-dominant world is full AI communism, which is precisely what China proposes: rather than a market product, AI is a "public good" which will ultimately be provided by the state as a kind of "digital public infrastructure." This future strikes me as a dystopian hellscape, but I've never met an open-weight models advocate who doesn't ultimately concede this is where things end. You'd be surprised how many 'accelerationists' lobbied me, while I was in government, to support an eleven or twelve-figure federally funded data center so that startups could train models at a subsidy and then give them away for free. There was no other way for AI to progress, they said. Perhaps this is the logical end state of things. Nonetheless, I find myself surprised to see supposed accelerationists excited about such an outcome. I think many of them just don't know what they're doing. Many accelerationists do not view the creation and serving of frontier models as a legitimate business.
5. I would guess that the Trump Administration will at some point realize that their best strategy here would be to create large amounts of regulatory risk around the use of open-weight Chinese models. You don't need to "ban open source" (one of the dumber motifs of AI policy discussion). You just need to direct every agency to issue soft law that creates FUD. "A Federal Reserve Advisory Bulletin found that there may be backdoors in Chinese AI models." It needn't be that well justified. You just create enough regulatory risk that every regulated enterprise backs off. You probably don't want to create so much regulatory risk that you scare off the hyperscalers from serving Chinese models; this will just drive startups to sketchier providers. There's a happy middle ground here. I'd assume they will do some version of this.
6. It's probably true that open-weight models of this capability make the world a bit more dangerous, but not so much more that you'll really notice. At some point the models will be capable enough that you will notice. "A nonliving, invisible, dangerous, and infinitely self-replicating agent escaped from a Chinese lab," you say? Color me shocked. -
Russ Salakhutdinov
@rsalakhu
Wait, what?
John Ennis
@johnennis
“Open-weight models are inherently decelerationist”
Dean W. Ball
@deanwball
Some observations on Kimi:
1. It's a very good model! I don't think its performance can be explained away by distillation or anything like that. In agentic coding sessions, it seems pretty much on par with the best public models of Q1 2026. In my fairly limited use, it also seemed very token hungry. It's not obvious to me that this model is actually that cheap to run.
2. I am personally surprised the Chinese state continues to allow the open sourcing of models this good, given potential risks. To be clear, I *myself* might be fine with models presenting this level of marginal risk being open weight, but I am surprised that China is fine with it. I suspect the reason they are is 75% explained by strategic blindness/lack of AGI-pilledness (the CCP is very Yann Lecun-y in its views of AI). The other 25% or so is their lack of compute for customer inference (making China's open-weight strategy an unintended byproduct of US export controls) and the normal Chinese strategy of aggressive exports. For the companies, as opposed to the government, the decision to open source is partially ideological and partially because they are behind, and they know that very few people would pay for sub-frontier models from China.
3. Open-weight models are inherently decelerationist, and I'm continually surprised to see the so-called "accelerationists" so excited about open-weight models. I suspect the reason they are is that they know open-weight models are effectively ungovernable, and they simply like the overall cloak of ungovernability open-weight models create over the whole of AI. It's not a bad strategy; it reminds me of James Scott's recounting of the hill people in "the art of not being governed." Still, in the end, open-weight models deter further AI capex.
4. One probable outcome of an open-weight-model-dominant world is full AI communism, which is precisely what China proposes: rather than a market product, AI is a "public good" which will ultimately be provided by the state as a kind of "digital public infrastructure." This future strikes me as a dystopian hellscape, but I've never met an open-weight models advocate who doesn't ultimately concede this is where things end. You'd be surprised how many 'accelerationists' lobbied me, while I was in government, to support an eleven or twelve-figure federally funded data center so that startups could train models at a subsidy and then give them away for free. There was no other way for AI to progress, they said. Perhaps this is the logical end state of things. Nonetheless, I find myself surprised to see supposed accelerationists excited about such an outcome. I think many of them just don't know what they're doing. Many accelerationists do not view the creation and serving of frontier models as a legitimate business.
5. I would guess that the Trump Administration will at some point realize that their best strategy here would be to create large amounts of regulatory risk around the use of open-weight Chinese models. You don't need to "ban open source" (one of the dumber motifs of AI policy discussion). You just need to direct every agency to issue soft law that creates FUD. "A Federal Reserve Advisory Bulletin found that there may be backdoors in Chinese AI models." It needn't be that well justified. You just create enough regulatory risk that every regulated enterprise backs off. You probably don't want to create so much regulatory risk that you scare off the hyperscalers from serving Chinese models; this will just drive startups to sketchier providers. There's a happy middle ground here. I'd assume they will do some version of this.
6. It's probably true that open-weight models of this capability make the world a bit more dangerous, but not so much more that you'll really notice. At some point the models will be capable enough that you will notice. "A nonliving, invisible, dangerous, and infinitely self-replicating agent escaped from a Chinese lab," you say? Color me shocked. -
Russ Salakhutdinov
@rsalakhu
No, Nitish @nitishsr, the privilege is all mine! Working with you has been an incredible experience, especially as you were one of my very first PhD students.
A fun story: During one of our meetings at Apple, an executive asked Nitish: so, what are you an expert in? Another senior IC jumped in: Nitish is an expert at everything.
And I can attest to that: Nitish is an expert at everything.
Nitish is the co-creator of Dropout and he has amassed nearly 90,000 citations.
That is an Impact 💪!
Nitish Srivastava
@nitishsr
It has been a privilege to work with @rsalakhu at Toronto, Perceptual Machines, Apple, and having him as an advisor at Vayu! I’m deeply grateful for your constant support and encouragement.
Russ Salakhutdinov
@rsalakhu
I’ve been asked several times whether Zhilin Yang, the founder of @Kimi_Moonshot was my PhD student. The answer is yes and he is absolutely brilliant.
But I’ve been incredibly fortunate to work with so many outstanding PhD students over the years. So I thought I’d brag a little about them and their career paths (of course there are also many MSc and undergraduate students, sorry if I missed anyone):
Founders / Founding Team Members
Devendra Chaplot @dchaplot PhD, Founding Member Thinking Machines / Mistral
Zhilin Yang PhD, Founder & CEO, Moonshot AI
Jimmy Ba @jimmybajimmyba MSc/PhD, Co-founder xAI
Hubert Tsai PhD, Co-founder Spuree, Apple
Nitish Srivastava @nitishsr PhD, Co-founder Perceptual Machines; Co-founder Vayu Robotics
Charlie Tang PhD, Co-founder Perceptual Machines, DE Shaw.
Professors
Paul Liang @pliang279 PhD, MIT
Ben Eysenbach @ben_eysenbach PhD, Princeton University
Ruosong Wang @RuosongW PhD, Peking University
Bhuwan Dhingra @bhuwandhingra PhD, Duke University
Roger Grosse @RogerGrosse Postdoc, University of Toronto
Alexander Schwing Postdoc, UIUC
Research Scientists
Shuyan Zhou @shuyanzh36, Postdoc, Meta Superintelligence Lab
Tiffany Min @SoYeonTiffMin PhD, Microsoft AI
Murtaza Dalal @mihdalal PhD, Tesla AI
Minji Yoon @MinjiYoon90 , PhD, Microsoft AI
Shrimai Prabhumoye PhD, NVIDIA AI, Mistral
Haitian Sun @sun_haitian PhD, Google DeepMind
Emilio Parisotto PhD, Google DeepMind
Lisa Lee PhD @rl_agent, Google DeepMind
Manzil Zaheer @ManzilZaheer PhD, Google DeepMind
Jamie Kiros PhD, Google Brain, OpenAI
Yuri Burda PhD, OpenAI, Anthropic
Cody Severinski PhD, Amazon -
Russ Salakhutdinov
@rsalakhu
Best times ever!
Charlie Tang
@tang_1c
The brilliant @nitishsr and I were both with @geoffreyhinton when Russ first joined UofT as faculty and we absolutely jumped at the opportunity to add Russ as co-advisor for our PhD.
I've always admired Russ and his work even in undergrad, reading some of his papers 5 times over! Deep Boltzmann Machines were the representative work in GenAI in 2009.
So when I graduated in 2015, we followed the blueprint laid out by our friends of DNNResearch (Geoff, Ilya, and Alex) and that led to our very own Perceptual Machines Inc. (w/ @nitishsr, @rsalakhu)
Words can not capture all the memories, excitements, and up and downs of those years, truly blessed to have studied and worked with both of you @nitishsr, @rsalakhu !!
Big shoutout also goes to the amazing @Ahmad_Al_Dahle for giving us the opportunity and believing in us very early on.
Russ Salakhutdinov
@rsalakhu
Charlie @tang_1c was one of my very first PhD students at Toronto. I still remember those days when together with Nitish @nitishsr, we were pitching our research to Apple execs. Great memories and fun times!
Charlie Tang
@tang_1c
Russ's first PhD and first startup co-founder here..
Kids these days would forget that his 2006 Science paper is what kicked off the *entire* Deep Learning revolution.
Still vividly rem that one time him settling for my sunnyvale apt air 🛏️ as we prepared long into the night for our deck to the Apple execs - Russ worked incredibly hard in pursuit of greatness in both academia and industry.
Lastly, Russ was and still is the most approachable and gregarious super-star AI/ML prof of them all, no matter if you were a colleague, investor, mag 7 CEO or simply a random ML conf attendee. #character
Russ Salakhutdinov
@rsalakhu
I’ve been asked several times whether Zhilin Yang, the founder of @Kimi_Moonshot was my PhD student. The answer is yes and he is absolutely brilliant.
But I’ve been incredibly fortunate to work with so many outstanding PhD students over the years. So I thought I’d brag a little about them and their career paths (of course there are also many MSc and undergraduate students, sorry if I missed anyone):
Founders / Founding Team Members
Devendra Chaplot @dchaplot PhD, Founding Member Thinking Machines / Mistral
Zhilin Yang PhD, Founder & CEO, Moonshot AI
Jimmy Ba @jimmybajimmyba MSc/PhD, Co-founder xAI
Hubert Tsai PhD, Co-founder Spuree, Apple
Nitish Srivastava @nitishsr PhD, Co-founder Perceptual Machines; Co-founder Vayu Robotics
Charlie Tang PhD, Co-founder Perceptual Machines, DE Shaw.
Professors
Paul Liang @pliang279 PhD, MIT
Ben Eysenbach @ben_eysenbach PhD, Princeton University
Ruosong Wang @RuosongW PhD, Peking University
Bhuwan Dhingra @bhuwandhingra PhD, Duke University
Roger Grosse @RogerGrosse Postdoc, University of Toronto
Alexander Schwing Postdoc, UIUC
Research Scientists
Shuyan Zhou @shuyanzh36, Postdoc, Meta Superintelligence Lab
Tiffany Min @SoYeonTiffMin PhD, Microsoft AI
Murtaza Dalal @mihdalal PhD, Tesla AI
Minji Yoon @MinjiYoon90 , PhD, Microsoft AI
Shrimai Prabhumoye PhD, NVIDIA AI, Mistral
Haitian Sun @sun_haitian PhD, Google DeepMind
Emilio Parisotto PhD, Google DeepMind
Lisa Lee PhD @rl_agent, Google DeepMind
Manzil Zaheer @ManzilZaheer PhD, Google DeepMind
Jamie Kiros PhD, Google Brain, OpenAI
Yuri Burda PhD, OpenAI, Anthropic
Cody Severinski PhD, Amazon -
Russ Salakhutdinov reposted
Soham
@sohamg121
I had the chance to be advised by you for a semester while working with Emilio - to me the one meeting we used to do per week was indormation-dense and directional enough. Never felt like I was on my own, and the independence you provided was much appreciated.
-
Russ Salakhutdinov reposted
Russ Salakhutdinov
@rsalakhu
Super cool effort.
Archer Announces Zee, AI Foundation Model Purpose-Built for Aviation, a Key Pillar of Its Physical AI Strategy
news.archer.com/archer-announces-zee-ai-foundation-model-… -
Russ Salakhutdinov
@rsalakhu
It feels like just yesterday that you, Yonatan @ybisk, and I were having these long technical discussions about embodied AI, some of my favorite conversations. You'd always come up with truly unique and contrarian ideas, I miss those meetings.
@MicrosoftAI is very lucky to have you.
So Yeon (Tiffany) Min
@SoYeonTiffMin
I was incredibly fortunate to be advised by the 🐐 @rsalakhu. When Russ agreed to co-advise me at CMU, my sadness about leaving Boston immediately disappeared. Once we worked together, I realized I was even luckier than I knew. Forever grateful and proud to be his student! 🐐🐐
Russ Salakhutdinov
@rsalakhu
I’ve been asked several times whether Zhilin Yang, the founder of @Kimi_Moonshot was my PhD student. The answer is yes and he is absolutely brilliant.
But I’ve been incredibly fortunate to work with so many outstanding PhD students over the years. So I thought I’d brag a little about them and their career paths (of course there are also many MSc and undergraduate students, sorry if I missed anyone):
Founders / Founding Team Members
Devendra Chaplot @dchaplot PhD, Founding Member Thinking Machines / Mistral
Zhilin Yang PhD, Founder & CEO, Moonshot AI
Jimmy Ba @jimmybajimmyba MSc/PhD, Co-founder xAI
Hubert Tsai PhD, Co-founder Spuree, Apple
Nitish Srivastava @nitishsr PhD, Co-founder Perceptual Machines; Co-founder Vayu Robotics
Charlie Tang PhD, Co-founder Perceptual Machines, DE Shaw.
Professors
Paul Liang @pliang279 PhD, MIT
Ben Eysenbach @ben_eysenbach PhD, Princeton University
Ruosong Wang @RuosongW PhD, Peking University
Bhuwan Dhingra @bhuwandhingra PhD, Duke University
Roger Grosse @RogerGrosse Postdoc, University of Toronto
Alexander Schwing Postdoc, UIUC
Research Scientists
Shuyan Zhou @shuyanzh36, Postdoc, Meta Superintelligence Lab
Tiffany Min @SoYeonTiffMin PhD, Microsoft AI
Murtaza Dalal @mihdalal PhD, Tesla AI
Minji Yoon @MinjiYoon90 , PhD, Microsoft AI
Shrimai Prabhumoye PhD, NVIDIA AI, Mistral
Haitian Sun @sun_haitian PhD, Google DeepMind
Emilio Parisotto PhD, Google DeepMind
Lisa Lee PhD @rl_agent, Google DeepMind
Manzil Zaheer @ManzilZaheer PhD, Google DeepMind
Jamie Kiros PhD, Google Brain, OpenAI
Yuri Burda PhD, OpenAI, Anthropic
Cody Severinski PhD, Amazon -
Russ Salakhutdinov
@rsalakhu
Thanks Hubert @cheehoohubert for the kind words. We should catch up next time I am in California.
Fun fact: Hubert’s most cited paper (almost 6,000 citations) is called HuBERT (Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units).
His second most cited paper has over 3,000 citations (Multimodal transformer for unaligned multimodal language sequences)!
Impact: 💪
Yao-Hung Tsai
@cheehoohubert
I never post on X, but had to reply to this one.
Everyone knows @rsalakhu is brilliant. But what always stood out to me was his EQ—perhaps the highest I’ve ever seen. That rare combination of extraordinary IQ and EQ is exactly why so many of us wanted to work with him.
Very fortunate to have been his PhD student.
Russ Salakhutdinov
@rsalakhu
I’ve been asked several times whether Zhilin Yang, the founder of @Kimi_Moonshot was my PhD student. The answer is yes and he is absolutely brilliant.
But I’ve been incredibly fortunate to work with so many outstanding PhD students over the years. So I thought I’d brag a little about them and their career paths (of course there are also many MSc and undergraduate students, sorry if I missed anyone):
Founders / Founding Team Members
Devendra Chaplot @dchaplot PhD, Founding Member Thinking Machines / Mistral
Zhilin Yang PhD, Founder & CEO, Moonshot AI
Jimmy Ba @jimmybajimmyba MSc/PhD, Co-founder xAI
Hubert Tsai PhD, Co-founder Spuree, Apple
Nitish Srivastava @nitishsr PhD, Co-founder Perceptual Machines; Co-founder Vayu Robotics
Charlie Tang PhD, Co-founder Perceptual Machines, DE Shaw.
Professors
Paul Liang @pliang279 PhD, MIT
Ben Eysenbach @ben_eysenbach PhD, Princeton University
Ruosong Wang @RuosongW PhD, Peking University
Bhuwan Dhingra @bhuwandhingra PhD, Duke University
Roger Grosse @RogerGrosse Postdoc, University of Toronto
Alexander Schwing Postdoc, UIUC
Research Scientists
Shuyan Zhou @shuyanzh36, Postdoc, Meta Superintelligence Lab
Tiffany Min @SoYeonTiffMin PhD, Microsoft AI
Murtaza Dalal @mihdalal PhD, Tesla AI
Minji Yoon @MinjiYoon90 , PhD, Microsoft AI
Shrimai Prabhumoye PhD, NVIDIA AI, Mistral
Haitian Sun @sun_haitian PhD, Google DeepMind
Emilio Parisotto PhD, Google DeepMind
Lisa Lee PhD @rl_agent, Google DeepMind
Manzil Zaheer @ManzilZaheer PhD, Google DeepMind
Jamie Kiros PhD, Google Brain, OpenAI
Yuri Burda PhD, OpenAI, Anthropic
Cody Severinski PhD, Amazon -
Russ Salakhutdinov
@rsalakhu
Thank you Lisa @rl_agent, for the kind words. You've always had brilliant ideas in RL, and it's been so exciting to see you making an impact on Gemini at GDM. They're very lucky to have you.
Lisa Lee
@rl_agent
Russ was an incredibly encouraging & supportive mentor who gave us freedom to pursue our own academic interests—spanning deep learning architectures, optimization, robotics, NLP, & multimodal representations—while providing hands-on expert guidance across our diverse projects.
Russ Salakhutdinov
@rsalakhu
I’ve been asked several times whether Zhilin Yang, the founder of @Kimi_Moonshot was my PhD student. The answer is yes and he is absolutely brilliant.
But I’ve been incredibly fortunate to work with so many outstanding PhD students over the years. So I thought I’d brag a little about them and their career paths (of course there are also many MSc and undergraduate students, sorry if I missed anyone):
Founders / Founding Team Members
Devendra Chaplot @dchaplot PhD, Founding Member Thinking Machines / Mistral
Zhilin Yang PhD, Founder & CEO, Moonshot AI
Jimmy Ba @jimmybajimmyba MSc/PhD, Co-founder xAI
Hubert Tsai PhD, Co-founder Spuree, Apple
Nitish Srivastava @nitishsr PhD, Co-founder Perceptual Machines; Co-founder Vayu Robotics
Charlie Tang PhD, Co-founder Perceptual Machines, DE Shaw.
Professors
Paul Liang @pliang279 PhD, MIT
Ben Eysenbach @ben_eysenbach PhD, Princeton University
Ruosong Wang @RuosongW PhD, Peking University
Bhuwan Dhingra @bhuwandhingra PhD, Duke University
Roger Grosse @RogerGrosse Postdoc, University of Toronto
Alexander Schwing Postdoc, UIUC
Research Scientists
Shuyan Zhou @shuyanzh36, Postdoc, Meta Superintelligence Lab
Tiffany Min @SoYeonTiffMin PhD, Microsoft AI
Murtaza Dalal @mihdalal PhD, Tesla AI
Minji Yoon @MinjiYoon90 , PhD, Microsoft AI
Shrimai Prabhumoye PhD, NVIDIA AI, Mistral
Haitian Sun @sun_haitian PhD, Google DeepMind
Emilio Parisotto PhD, Google DeepMind
Lisa Lee PhD @rl_agent, Google DeepMind
Manzil Zaheer @ManzilZaheer PhD, Google DeepMind
Jamie Kiros PhD, Google Brain, OpenAI
Yuri Burda PhD, OpenAI, Anthropic
Cody Severinski PhD, Amazon -
Russ Salakhutdinov
@rsalakhu
Lol I think Peyman @docmilanfar really likes me. I don't actually know him, but I'm deeply honored by his encyclopedic knowledge of my academic and industry career 😆.
But jokes aside, my current and former students at CMU (and Toronto) are absolutely exceptional, self-directed, very independent in shaping and driving their own research agenda, and with impeccable work ethic, much like so many students at CMU.
A student's success is also determined not only by their advisor, but also by the environment in which they grow. There is something genuinely special about CMU. It is a very collaborative place with an extraordinary concentration of talented, hardworking, and motivated students.
Doing PhD is never easy, it is a lot of very very hard work, especially at a place like CMU. But many of these students will go on to become leaders in academia and industry, professors at top universities, founders, CEOs, and pioneers of new fields.
And I will certainly continue to celebrate my students’ achievements, their scientific breakthroughs, and the impact they have on the world.
Peyman Milanfar
@docmilanfar
While this is an impressive array of talent, over 80% of the researchers Russ brags about were (self-) advised while he was juggling academic duties with major executive roles at Apple or Meta.
He's not alone in this respect. Elite programs attract talent that buoys the lab, giving advisors the bandwidth to juggle significant outside roles. I'd be careful about assigning too much credit to the advisor and not enough to the students' raw talent and self-discipline.
Russ Salakhutdinov
@rsalakhu
I’ve been asked several times whether Zhilin Yang, the founder of @Kimi_Moonshot was my PhD student. The answer is yes and he is absolutely brilliant.
But I’ve been incredibly fortunate to work with so many outstanding PhD students over the years. So I thought I’d brag a little about them and their career paths (of course there are also many MSc and undergraduate students, sorry if I missed anyone):
Founders / Founding Team Members
Devendra Chaplot @dchaplot PhD, Founding Member Thinking Machines / Mistral
Zhilin Yang PhD, Founder & CEO, Moonshot AI
Jimmy Ba @jimmybajimmyba MSc/PhD, Co-founder xAI
Hubert Tsai PhD, Co-founder Spuree, Apple
Nitish Srivastava @nitishsr PhD, Co-founder Perceptual Machines; Co-founder Vayu Robotics
Charlie Tang PhD, Co-founder Perceptual Machines, DE Shaw.
Professors
Paul Liang @pliang279 PhD, MIT
Ben Eysenbach @ben_eysenbach PhD, Princeton University
Ruosong Wang @RuosongW PhD, Peking University
Bhuwan Dhingra @bhuwandhingra PhD, Duke University
Roger Grosse @RogerGrosse Postdoc, University of Toronto
Alexander Schwing Postdoc, UIUC
Research Scientists
Shuyan Zhou @shuyanzh36, Postdoc, Meta Superintelligence Lab
Tiffany Min @SoYeonTiffMin PhD, Microsoft AI
Murtaza Dalal @mihdalal PhD, Tesla AI
Minji Yoon @MinjiYoon90 , PhD, Microsoft AI
Shrimai Prabhumoye PhD, NVIDIA AI, Mistral
Haitian Sun @sun_haitian PhD, Google DeepMind
Emilio Parisotto PhD, Google DeepMind
Lisa Lee PhD @rl_agent, Google DeepMind
Manzil Zaheer @ManzilZaheer PhD, Google DeepMind
Jamie Kiros PhD, Google Brain, OpenAI
Yuri Burda PhD, OpenAI, Anthropic
Cody Severinski PhD, Amazon -
Russ Salakhutdinov
@rsalakhu
If you are trying to compare yourself to Kimi, open source the model just like Kimi.
-
Russ Salakhutdinov reposted
Zain
@zainhas
Russ was training deep neural networks in 2006 using matlab -- if you don't put respect on his name that says more about you then it does about him
Charlie Tang
@tang_1c
Russ's first PhD and first startup co-founder here..
Kids these days would forget that his 2006 Science paper is what kicked off the *entire* Deep Learning revolution.
Still vividly rem that one time him settling for my sunnyvale apt air 🛏️ as we prepared long into the night for our deck to the Apple execs - Russ worked incredibly hard in pursuit of greatness in both academia and industry.
Lastly, Russ was and still is the most approachable and gregarious super-star AI/ML prof of them all, no matter if you were a colleague, investor, mag 7 CEO or simply a random ML conf attendee. #character
Russ Salakhutdinov
@rsalakhu
I’ve been asked several times whether Zhilin Yang, the founder of @Kimi_Moonshot was my PhD student. The answer is yes and he is absolutely brilliant.
But I’ve been incredibly fortunate to work with so many outstanding PhD students over the years. So I thought I’d brag a little about them and their career paths (of course there are also many MSc and undergraduate students, sorry if I missed anyone):
Founders / Founding Team Members
Devendra Chaplot @dchaplot PhD, Founding Member Thinking Machines / Mistral
Zhilin Yang PhD, Founder & CEO, Moonshot AI
Jimmy Ba @jimmybajimmyba MSc/PhD, Co-founder xAI
Hubert Tsai PhD, Co-founder Spuree, Apple
Nitish Srivastava @nitishsr PhD, Co-founder Perceptual Machines; Co-founder Vayu Robotics
Charlie Tang PhD, Co-founder Perceptual Machines, DE Shaw.
Professors
Paul Liang @pliang279 PhD, MIT
Ben Eysenbach @ben_eysenbach PhD, Princeton University
Ruosong Wang @RuosongW PhD, Peking University
Bhuwan Dhingra @bhuwandhingra PhD, Duke University
Roger Grosse @RogerGrosse Postdoc, University of Toronto
Alexander Schwing Postdoc, UIUC
Research Scientists
Shuyan Zhou @shuyanzh36, Postdoc, Meta Superintelligence Lab
Tiffany Min @SoYeonTiffMin PhD, Microsoft AI
Murtaza Dalal @mihdalal PhD, Tesla AI
Minji Yoon @MinjiYoon90 , PhD, Microsoft AI
Shrimai Prabhumoye PhD, NVIDIA AI, Mistral
Haitian Sun @sun_haitian PhD, Google DeepMind
Emilio Parisotto PhD, Google DeepMind
Lisa Lee PhD @rl_agent, Google DeepMind
Manzil Zaheer @ManzilZaheer PhD, Google DeepMind
Jamie Kiros PhD, Google Brain, OpenAI
Yuri Burda PhD, OpenAI, Anthropic
Cody Severinski PhD, Amazon -
Russ Salakhutdinov
@rsalakhu
Charlie @tang_1c was one of my very first PhD students at Toronto. I still remember those days when together with Nitish @nitishsr, we were pitching our research to Apple execs. Great memories and fun times!
Charlie Tang
@tang_1c
Russ's first PhD and first startup co-founder here..
Kids these days would forget that his 2006 Science paper is what kicked off the *entire* Deep Learning revolution.
Still vividly rem that one time him settling for my sunnyvale apt air 🛏️ as we prepared long into the night for our deck to the Apple execs - Russ worked incredibly hard in pursuit of greatness in both academia and industry.
Lastly, Russ was and still is the most approachable and gregarious super-star AI/ML prof of them all, no matter if you were a colleague, investor, mag 7 CEO or simply a random ML conf attendee. #character
Russ Salakhutdinov
@rsalakhu
I’ve been asked several times whether Zhilin Yang, the founder of @Kimi_Moonshot was my PhD student. The answer is yes and he is absolutely brilliant.
But I’ve been incredibly fortunate to work with so many outstanding PhD students over the years. So I thought I’d brag a little about them and their career paths (of course there are also many MSc and undergraduate students, sorry if I missed anyone):
Founders / Founding Team Members
Devendra Chaplot @dchaplot PhD, Founding Member Thinking Machines / Mistral
Zhilin Yang PhD, Founder & CEO, Moonshot AI
Jimmy Ba @jimmybajimmyba MSc/PhD, Co-founder xAI
Hubert Tsai PhD, Co-founder Spuree, Apple
Nitish Srivastava @nitishsr PhD, Co-founder Perceptual Machines; Co-founder Vayu Robotics
Charlie Tang PhD, Co-founder Perceptual Machines, DE Shaw.
Professors
Paul Liang @pliang279 PhD, MIT
Ben Eysenbach @ben_eysenbach PhD, Princeton University
Ruosong Wang @RuosongW PhD, Peking University
Bhuwan Dhingra @bhuwandhingra PhD, Duke University
Roger Grosse @RogerGrosse Postdoc, University of Toronto
Alexander Schwing Postdoc, UIUC
Research Scientists
Shuyan Zhou @shuyanzh36, Postdoc, Meta Superintelligence Lab
Tiffany Min @SoYeonTiffMin PhD, Microsoft AI
Murtaza Dalal @mihdalal PhD, Tesla AI
Minji Yoon @MinjiYoon90 , PhD, Microsoft AI
Shrimai Prabhumoye PhD, NVIDIA AI, Mistral
Haitian Sun @sun_haitian PhD, Google DeepMind
Emilio Parisotto PhD, Google DeepMind
Lisa Lee PhD @rl_agent, Google DeepMind
Manzil Zaheer @ManzilZaheer PhD, Google DeepMind
Jamie Kiros PhD, Google Brain, OpenAI
Yuri Burda PhD, OpenAI, Anthropic
Cody Severinski PhD, Amazon -
Russ Salakhutdinov
@rsalakhu
I think there's a lot of confusion and misinformation about Zhilin's visa, H1B lottery, immigration, and why he left the U.S.
Zhilin had many opportunities to stay in the U.S. if he wanted to. In fact, I was on an email thread with a senior Apple exec (Tim Cook's reports) asking whether Zhilin would consider joining Apple. I said that he wanted to go back to his homeland. The response was, well, we have an office in Beijing if he'd like to join there.
But Zhilin was quite determined to go back and build a startup. I remember him telling me that if he didn't at least try starting his own company, he would regret it for the rest of his life. I respect that, and he was right.
Many of my international PhD students do choose to stay in the U.S. Of course the U.S. immigration process can be quite intimidating and uncertain, even for superstar PhD graduates from places like CMU.
signüll
@signulll
kimi’s founder & ceo got his phd at cmu.
why didn’t he stay in america?! -
Russ Salakhutdinov reposted
Paul Liang
@pliang279
all of this is possible because Russ @rsalakhu is the best advisor anyone could ask for, by creating a supportive, collaborative environment grounded in technical rigor while always encouraging curiosity and exploration 🙏
Russ Salakhutdinov
@rsalakhu
I’ve been asked several times whether Zhilin Yang, the founder of @Kimi_Moonshot was my PhD student. The answer is yes and he is absolutely brilliant.
But I’ve been incredibly fortunate to work with so many outstanding PhD students over the years. So I thought I’d brag a little about them and their career paths (of course there are also many MSc and undergraduate students, sorry if I missed anyone):
Founders / Founding Team Members
Devendra Chaplot @dchaplot PhD, Founding Member Thinking Machines / Mistral
Zhilin Yang PhD, Founder & CEO, Moonshot AI
Jimmy Ba @jimmybajimmyba MSc/PhD, Co-founder xAI
Hubert Tsai PhD, Co-founder Spuree, Apple
Nitish Srivastava @nitishsr PhD, Co-founder Perceptual Machines; Co-founder Vayu Robotics
Charlie Tang PhD, Co-founder Perceptual Machines, DE Shaw.
Professors
Paul Liang @pliang279 PhD, MIT
Ben Eysenbach @ben_eysenbach PhD, Princeton University
Ruosong Wang @RuosongW PhD, Peking University
Bhuwan Dhingra @bhuwandhingra PhD, Duke University
Roger Grosse @RogerGrosse Postdoc, University of Toronto
Alexander Schwing Postdoc, UIUC
Research Scientists
Shuyan Zhou @shuyanzh36, Postdoc, Meta Superintelligence Lab
Tiffany Min @SoYeonTiffMin PhD, Microsoft AI
Murtaza Dalal @mihdalal PhD, Tesla AI
Minji Yoon @MinjiYoon90 , PhD, Microsoft AI
Shrimai Prabhumoye PhD, NVIDIA AI, Mistral
Haitian Sun @sun_haitian PhD, Google DeepMind
Emilio Parisotto PhD, Google DeepMind
Lisa Lee PhD @rl_agent, Google DeepMind
Manzil Zaheer @ManzilZaheer PhD, Google DeepMind
Jamie Kiros PhD, Google Brain, OpenAI
Yuri Burda PhD, OpenAI, Anthropic
Cody Severinski PhD, Amazon -
Russ Salakhutdinov reposted
Yao-Hung Tsai
@cheehoohubert
I never post on X, but had to reply to this one.
Everyone knows @rsalakhu is brilliant. But what always stood out to me was his EQ—perhaps the highest I’ve ever seen. That rare combination of extraordinary IQ and EQ is exactly why so many of us wanted to work with him.
Very fortunate to have been his PhD student.
Russ Salakhutdinov
@rsalakhu
I’ve been asked several times whether Zhilin Yang, the founder of @Kimi_Moonshot was my PhD student. The answer is yes and he is absolutely brilliant.
But I’ve been incredibly fortunate to work with so many outstanding PhD students over the years. So I thought I’d brag a little about them and their career paths (of course there are also many MSc and undergraduate students, sorry if I missed anyone):
Founders / Founding Team Members
Devendra Chaplot @dchaplot PhD, Founding Member Thinking Machines / Mistral
Zhilin Yang PhD, Founder & CEO, Moonshot AI
Jimmy Ba @jimmybajimmyba MSc/PhD, Co-founder xAI
Hubert Tsai PhD, Co-founder Spuree, Apple
Nitish Srivastava @nitishsr PhD, Co-founder Perceptual Machines; Co-founder Vayu Robotics
Charlie Tang PhD, Co-founder Perceptual Machines, DE Shaw.
Professors
Paul Liang @pliang279 PhD, MIT
Ben Eysenbach @ben_eysenbach PhD, Princeton University
Ruosong Wang @RuosongW PhD, Peking University
Bhuwan Dhingra @bhuwandhingra PhD, Duke University
Roger Grosse @RogerGrosse Postdoc, University of Toronto
Alexander Schwing Postdoc, UIUC
Research Scientists
Shuyan Zhou @shuyanzh36, Postdoc, Meta Superintelligence Lab
Tiffany Min @SoYeonTiffMin PhD, Microsoft AI
Murtaza Dalal @mihdalal PhD, Tesla AI
Minji Yoon @MinjiYoon90 , PhD, Microsoft AI
Shrimai Prabhumoye PhD, NVIDIA AI, Mistral
Haitian Sun @sun_haitian PhD, Google DeepMind
Emilio Parisotto PhD, Google DeepMind
Lisa Lee PhD @rl_agent, Google DeepMind
Manzil Zaheer @ManzilZaheer PhD, Google DeepMind
Jamie Kiros PhD, Google Brain, OpenAI
Yuri Burda PhD, OpenAI, Anthropic
Cody Severinski PhD, Amazon -
Russ Salakhutdinov reposted
Charlie Tang
@tang_1c
Russ's first PhD and first startup co-founder here..
Kids these days would forget that his 2006 Science paper is what kicked off the *entire* Deep Learning revolution.
Still vividly rem that one time him settling for my sunnyvale apt air 🛏️ as we prepared long into the night for our deck to the Apple execs - Russ worked incredibly hard in pursuit of greatness in both academia and industry.
Lastly, Russ was and still is the most approachable and gregarious super-star AI/ML prof of them all, no matter if you were a colleague, investor, mag 7 CEO or simply a random ML conf attendee. #character
Russ Salakhutdinov
@rsalakhu
I’ve been asked several times whether Zhilin Yang, the founder of @Kimi_Moonshot was my PhD student. The answer is yes and he is absolutely brilliant.
But I’ve been incredibly fortunate to work with so many outstanding PhD students over the years. So I thought I’d brag a little about them and their career paths (of course there are also many MSc and undergraduate students, sorry if I missed anyone):
Founders / Founding Team Members
Devendra Chaplot @dchaplot PhD, Founding Member Thinking Machines / Mistral
Zhilin Yang PhD, Founder & CEO, Moonshot AI
Jimmy Ba @jimmybajimmyba MSc/PhD, Co-founder xAI
Hubert Tsai PhD, Co-founder Spuree, Apple
Nitish Srivastava @nitishsr PhD, Co-founder Perceptual Machines; Co-founder Vayu Robotics
Charlie Tang PhD, Co-founder Perceptual Machines, DE Shaw.
Professors
Paul Liang @pliang279 PhD, MIT
Ben Eysenbach @ben_eysenbach PhD, Princeton University
Ruosong Wang @RuosongW PhD, Peking University
Bhuwan Dhingra @bhuwandhingra PhD, Duke University
Roger Grosse @RogerGrosse Postdoc, University of Toronto
Alexander Schwing Postdoc, UIUC
Research Scientists
Shuyan Zhou @shuyanzh36, Postdoc, Meta Superintelligence Lab
Tiffany Min @SoYeonTiffMin PhD, Microsoft AI
Murtaza Dalal @mihdalal PhD, Tesla AI
Minji Yoon @MinjiYoon90 , PhD, Microsoft AI
Shrimai Prabhumoye PhD, NVIDIA AI, Mistral
Haitian Sun @sun_haitian PhD, Google DeepMind
Emilio Parisotto PhD, Google DeepMind
Lisa Lee PhD @rl_agent, Google DeepMind
Manzil Zaheer @ManzilZaheer PhD, Google DeepMind
Jamie Kiros PhD, Google Brain, OpenAI
Yuri Burda PhD, OpenAI, Anthropic
Cody Severinski PhD, Amazon -
Russ Salakhutdinov reposted
Brandon Trabucco
@brandontrabucco
Russ is one of the most supportive and rewarding advisors to do a PhD with. His lab is full of so many cool ideas, passionate students, and fun discussions.
I've appreciated your patience so much during my leave to join Thinky, Russ. I'll graduate and join the list soon!
Russ Salakhutdinov
@rsalakhu
I’ve been asked several times whether Zhilin Yang, the founder of @Kimi_Moonshot was my PhD student. The answer is yes and he is absolutely brilliant.
But I’ve been incredibly fortunate to work with so many outstanding PhD students over the years. So I thought I’d brag a little about them and their career paths (of course there are also many MSc and undergraduate students, sorry if I missed anyone):
Founders / Founding Team Members
Devendra Chaplot @dchaplot PhD, Founding Member Thinking Machines / Mistral
Zhilin Yang PhD, Founder & CEO, Moonshot AI
Jimmy Ba @jimmybajimmyba MSc/PhD, Co-founder xAI
Hubert Tsai PhD, Co-founder Spuree, Apple
Nitish Srivastava @nitishsr PhD, Co-founder Perceptual Machines; Co-founder Vayu Robotics
Charlie Tang PhD, Co-founder Perceptual Machines, DE Shaw.
Professors
Paul Liang @pliang279 PhD, MIT
Ben Eysenbach @ben_eysenbach PhD, Princeton University
Ruosong Wang @RuosongW PhD, Peking University
Bhuwan Dhingra @bhuwandhingra PhD, Duke University
Roger Grosse @RogerGrosse Postdoc, University of Toronto
Alexander Schwing Postdoc, UIUC
Research Scientists
Shuyan Zhou @shuyanzh36, Postdoc, Meta Superintelligence Lab
Tiffany Min @SoYeonTiffMin PhD, Microsoft AI
Murtaza Dalal @mihdalal PhD, Tesla AI
Minji Yoon @MinjiYoon90 , PhD, Microsoft AI
Shrimai Prabhumoye PhD, NVIDIA AI, Mistral
Haitian Sun @sun_haitian PhD, Google DeepMind
Emilio Parisotto PhD, Google DeepMind
Lisa Lee PhD @rl_agent, Google DeepMind
Manzil Zaheer @ManzilZaheer PhD, Google DeepMind
Jamie Kiros PhD, Google Brain, OpenAI
Yuri Burda PhD, OpenAI, Anthropic
Cody Severinski PhD, Amazon -
Russ Salakhutdinov reposted
Shuyan Zhou
@shuyanzh36
🐐
Russ Salakhutdinov
@rsalakhu
I’ve been asked several times whether Zhilin Yang, the founder of @Kimi_Moonshot was my PhD student. The answer is yes and he is absolutely brilliant.
But I’ve been incredibly fortunate to work with so many outstanding PhD students over the years. So I thought I’d brag a little about them and their career paths (of course there are also many MSc and undergraduate students, sorry if I missed anyone):
Founders / Founding Team Members
Devendra Chaplot @dchaplot PhD, Founding Member Thinking Machines / Mistral
Zhilin Yang PhD, Founder & CEO, Moonshot AI
Jimmy Ba @jimmybajimmyba MSc/PhD, Co-founder xAI
Hubert Tsai PhD, Co-founder Spuree, Apple
Nitish Srivastava @nitishsr PhD, Co-founder Perceptual Machines; Co-founder Vayu Robotics
Charlie Tang PhD, Co-founder Perceptual Machines, DE Shaw.
Professors
Paul Liang @pliang279 PhD, MIT
Ben Eysenbach @ben_eysenbach PhD, Princeton University
Ruosong Wang @RuosongW PhD, Peking University
Bhuwan Dhingra @bhuwandhingra PhD, Duke University
Roger Grosse @RogerGrosse Postdoc, University of Toronto
Alexander Schwing Postdoc, UIUC
Research Scientists
Shuyan Zhou @shuyanzh36, Postdoc, Meta Superintelligence Lab
Tiffany Min @SoYeonTiffMin PhD, Microsoft AI
Murtaza Dalal @mihdalal PhD, Tesla AI
Minji Yoon @MinjiYoon90 , PhD, Microsoft AI
Shrimai Prabhumoye PhD, NVIDIA AI, Mistral
Haitian Sun @sun_haitian PhD, Google DeepMind
Emilio Parisotto PhD, Google DeepMind
Lisa Lee PhD @rl_agent, Google DeepMind
Manzil Zaheer @ManzilZaheer PhD, Google DeepMind
Jamie Kiros PhD, Google Brain, OpenAI
Yuri Burda PhD, OpenAI, Anthropic
Cody Severinski PhD, Amazon -
Russ Salakhutdinov reposted
Lisa Lee
@rl_agent
Russ was an incredibly encouraging & supportive mentor who gave us freedom to pursue our own academic interests—spanning deep learning architectures, optimization, robotics, NLP, & multimodal representations—while providing hands-on expert guidance across our diverse projects.
Russ Salakhutdinov
@rsalakhu
I’ve been asked several times whether Zhilin Yang, the founder of @Kimi_Moonshot was my PhD student. The answer is yes and he is absolutely brilliant.
But I’ve been incredibly fortunate to work with so many outstanding PhD students over the years. So I thought I’d brag a little about them and their career paths (of course there are also many MSc and undergraduate students, sorry if I missed anyone):
Founders / Founding Team Members
Devendra Chaplot @dchaplot PhD, Founding Member Thinking Machines / Mistral
Zhilin Yang PhD, Founder & CEO, Moonshot AI
Jimmy Ba @jimmybajimmyba MSc/PhD, Co-founder xAI
Hubert Tsai PhD, Co-founder Spuree, Apple
Nitish Srivastava @nitishsr PhD, Co-founder Perceptual Machines; Co-founder Vayu Robotics
Charlie Tang PhD, Co-founder Perceptual Machines, DE Shaw.
Professors
Paul Liang @pliang279 PhD, MIT
Ben Eysenbach @ben_eysenbach PhD, Princeton University
Ruosong Wang @RuosongW PhD, Peking University
Bhuwan Dhingra @bhuwandhingra PhD, Duke University
Roger Grosse @RogerGrosse Postdoc, University of Toronto
Alexander Schwing Postdoc, UIUC
Research Scientists
Shuyan Zhou @shuyanzh36, Postdoc, Meta Superintelligence Lab
Tiffany Min @SoYeonTiffMin PhD, Microsoft AI
Murtaza Dalal @mihdalal PhD, Tesla AI
Minji Yoon @MinjiYoon90 , PhD, Microsoft AI
Shrimai Prabhumoye PhD, NVIDIA AI, Mistral
Haitian Sun @sun_haitian PhD, Google DeepMind
Emilio Parisotto PhD, Google DeepMind
Lisa Lee PhD @rl_agent, Google DeepMind
Manzil Zaheer @ManzilZaheer PhD, Google DeepMind
Jamie Kiros PhD, Google Brain, OpenAI
Yuri Burda PhD, OpenAI, Anthropic
Cody Severinski PhD, Amazon -
Russ Salakhutdinov reposted
So Yeon (Tiffany) Min
@SoYeonTiffMin
I was incredibly fortunate to be advised by the 🐐 @rsalakhu. When Russ agreed to co-advise me at CMU, my sadness about leaving Boston immediately disappeared. Once we worked together, I realized I was even luckier than I knew. Forever grateful and proud to be his student! 🐐🐐
Russ Salakhutdinov
@rsalakhu
I’ve been asked several times whether Zhilin Yang, the founder of @Kimi_Moonshot was my PhD student. The answer is yes and he is absolutely brilliant.
But I’ve been incredibly fortunate to work with so many outstanding PhD students over the years. So I thought I’d brag a little about them and their career paths (of course there are also many MSc and undergraduate students, sorry if I missed anyone):
Founders / Founding Team Members
Devendra Chaplot @dchaplot PhD, Founding Member Thinking Machines / Mistral
Zhilin Yang PhD, Founder & CEO, Moonshot AI
Jimmy Ba @jimmybajimmyba MSc/PhD, Co-founder xAI
Hubert Tsai PhD, Co-founder Spuree, Apple
Nitish Srivastava @nitishsr PhD, Co-founder Perceptual Machines; Co-founder Vayu Robotics
Charlie Tang PhD, Co-founder Perceptual Machines, DE Shaw.
Professors
Paul Liang @pliang279 PhD, MIT
Ben Eysenbach @ben_eysenbach PhD, Princeton University
Ruosong Wang @RuosongW PhD, Peking University
Bhuwan Dhingra @bhuwandhingra PhD, Duke University
Roger Grosse @RogerGrosse Postdoc, University of Toronto
Alexander Schwing Postdoc, UIUC
Research Scientists
Shuyan Zhou @shuyanzh36, Postdoc, Meta Superintelligence Lab
Tiffany Min @SoYeonTiffMin PhD, Microsoft AI
Murtaza Dalal @mihdalal PhD, Tesla AI
Minji Yoon @MinjiYoon90 , PhD, Microsoft AI
Shrimai Prabhumoye PhD, NVIDIA AI, Mistral
Haitian Sun @sun_haitian PhD, Google DeepMind
Emilio Parisotto PhD, Google DeepMind
Lisa Lee PhD @rl_agent, Google DeepMind
Manzil Zaheer @ManzilZaheer PhD, Google DeepMind
Jamie Kiros PhD, Google Brain, OpenAI
Yuri Burda PhD, OpenAI, Anthropic
Cody Severinski PhD, Amazon -
Russ Salakhutdinov
@rsalakhu
Paul Liang @pliang279 is the GOAT. If you're thinking about doing a PhD at MIT, I can't imagine a better advisor.
Paul Liang
@pliang279
all of this is possible because Russ @rsalakhu is the best advisor anyone could ask for, by creating a supportive, collaborative environment grounded in technical rigor while always encouraging curiosity and exploration 🙏
Russ Salakhutdinov
@rsalakhu
I’ve been asked several times whether Zhilin Yang, the founder of @Kimi_Moonshot was my PhD student. The answer is yes and he is absolutely brilliant.
But I’ve been incredibly fortunate to work with so many outstanding PhD students over the years. So I thought I’d brag a little about them and their career paths (of course there are also many MSc and undergraduate students, sorry if I missed anyone):
Founders / Founding Team Members
Devendra Chaplot @dchaplot PhD, Founding Member Thinking Machines / Mistral
Zhilin Yang PhD, Founder & CEO, Moonshot AI
Jimmy Ba @jimmybajimmyba MSc/PhD, Co-founder xAI
Hubert Tsai PhD, Co-founder Spuree, Apple
Nitish Srivastava @nitishsr PhD, Co-founder Perceptual Machines; Co-founder Vayu Robotics
Charlie Tang PhD, Co-founder Perceptual Machines, DE Shaw.
Professors
Paul Liang @pliang279 PhD, MIT
Ben Eysenbach @ben_eysenbach PhD, Princeton University
Ruosong Wang @RuosongW PhD, Peking University
Bhuwan Dhingra @bhuwandhingra PhD, Duke University
Roger Grosse @RogerGrosse Postdoc, University of Toronto
Alexander Schwing Postdoc, UIUC
Research Scientists
Shuyan Zhou @shuyanzh36, Postdoc, Meta Superintelligence Lab
Tiffany Min @SoYeonTiffMin PhD, Microsoft AI
Murtaza Dalal @mihdalal PhD, Tesla AI
Minji Yoon @MinjiYoon90 , PhD, Microsoft AI
Shrimai Prabhumoye PhD, NVIDIA AI, Mistral
Haitian Sun @sun_haitian PhD, Google DeepMind
Emilio Parisotto PhD, Google DeepMind
Lisa Lee PhD @rl_agent, Google DeepMind
Manzil Zaheer @ManzilZaheer PhD, Google DeepMind
Jamie Kiros PhD, Google Brain, OpenAI
Yuri Burda PhD, OpenAI, Anthropic
Cody Severinski PhD, Amazon -
Russ Salakhutdinov
@rsalakhu
Thank you @mihdalal. Mentoring and learning alongside brilliant students like you has been a true blessing.
Murtaza Dalal @ CoRL 2026
@mihdalal
It was my honor to be a part of this incredible group of students and an alumni of Russ’ lab :)
And being Russ’ first robotics PhD student ;)
These lab meetings were some of the most intellectually diverse gatherings I’ve been part of - I truly don’t know how Russ was able to keep up with all of our fields
From NLP to RL theory to even robotics, he managed to stay on top of it all!
Of course our common thread was deep learning :)
Russ Salakhutdinov
@rsalakhu
I’ve been asked several times whether Zhilin Yang, the founder of @Kimi_Moonshot was my PhD student. The answer is yes and he is absolutely brilliant.
But I’ve been incredibly fortunate to work with so many outstanding PhD students over the years. So I thought I’d brag a little about them and their career paths (of course there are also many MSc and undergraduate students, sorry if I missed anyone):
Founders / Founding Team Members
Devendra Chaplot @dchaplot PhD, Founding Member Thinking Machines / Mistral
Zhilin Yang PhD, Founder & CEO, Moonshot AI
Jimmy Ba @jimmybajimmyba MSc/PhD, Co-founder xAI
Hubert Tsai PhD, Co-founder Spuree, Apple
Nitish Srivastava @nitishsr PhD, Co-founder Perceptual Machines; Co-founder Vayu Robotics
Charlie Tang PhD, Co-founder Perceptual Machines, DE Shaw.
Professors
Paul Liang @pliang279 PhD, MIT
Ben Eysenbach @ben_eysenbach PhD, Princeton University
Ruosong Wang @RuosongW PhD, Peking University
Bhuwan Dhingra @bhuwandhingra PhD, Duke University
Roger Grosse @RogerGrosse Postdoc, University of Toronto
Alexander Schwing Postdoc, UIUC
Research Scientists
Shuyan Zhou @shuyanzh36, Postdoc, Meta Superintelligence Lab
Tiffany Min @SoYeonTiffMin PhD, Microsoft AI
Murtaza Dalal @mihdalal PhD, Tesla AI
Minji Yoon @MinjiYoon90 , PhD, Microsoft AI
Shrimai Prabhumoye PhD, NVIDIA AI, Mistral
Haitian Sun @sun_haitian PhD, Google DeepMind
Emilio Parisotto PhD, Google DeepMind
Lisa Lee PhD @rl_agent, Google DeepMind
Manzil Zaheer @ManzilZaheer PhD, Google DeepMind
Jamie Kiros PhD, Google Brain, OpenAI
Yuri Burda PhD, OpenAI, Anthropic
Cody Severinski PhD, Amazon -
Russ Salakhutdinov reposted
Murtaza Dalal @ CoRL 2026
@mihdalal
It was my honor to be a part of this incredible group of students and an alumni of Russ’ lab :)
And being Russ’ first robotics PhD student ;)
These lab meetings were some of the most intellectually diverse gatherings I’ve been part of - I truly don’t know how Russ was able to keep up with all of our fields
From NLP to RL theory to even robotics, he managed to stay on top of it all!
Of course our common thread was deep learning :)
Russ Salakhutdinov
@rsalakhu
I’ve been asked several times whether Zhilin Yang, the founder of @Kimi_Moonshot was my PhD student. The answer is yes and he is absolutely brilliant.
But I’ve been incredibly fortunate to work with so many outstanding PhD students over the years. So I thought I’d brag a little about them and their career paths (of course there are also many MSc and undergraduate students, sorry if I missed anyone):
Founders / Founding Team Members
Devendra Chaplot @dchaplot PhD, Founding Member Thinking Machines / Mistral
Zhilin Yang PhD, Founder & CEO, Moonshot AI
Jimmy Ba @jimmybajimmyba MSc/PhD, Co-founder xAI
Hubert Tsai PhD, Co-founder Spuree, Apple
Nitish Srivastava @nitishsr PhD, Co-founder Perceptual Machines; Co-founder Vayu Robotics
Charlie Tang PhD, Co-founder Perceptual Machines, DE Shaw.
Professors
Paul Liang @pliang279 PhD, MIT
Ben Eysenbach @ben_eysenbach PhD, Princeton University
Ruosong Wang @RuosongW PhD, Peking University
Bhuwan Dhingra @bhuwandhingra PhD, Duke University
Roger Grosse @RogerGrosse Postdoc, University of Toronto
Alexander Schwing Postdoc, UIUC
Research Scientists
Shuyan Zhou @shuyanzh36, Postdoc, Meta Superintelligence Lab
Tiffany Min @SoYeonTiffMin PhD, Microsoft AI
Murtaza Dalal @mihdalal PhD, Tesla AI
Minji Yoon @MinjiYoon90 , PhD, Microsoft AI
Shrimai Prabhumoye PhD, NVIDIA AI, Mistral
Haitian Sun @sun_haitian PhD, Google DeepMind
Emilio Parisotto PhD, Google DeepMind
Lisa Lee PhD @rl_agent, Google DeepMind
Manzil Zaheer @ManzilZaheer PhD, Google DeepMind
Jamie Kiros PhD, Google Brain, OpenAI
Yuri Burda PhD, OpenAI, Anthropic
Cody Severinski PhD, Amazon -
Russ Salakhutdinov
@rsalakhu
Thank you @dchaplot for kind words. Working with brilliant students such as yourself has been such a privilege.
Devendra Chaplot
@dchaplot
The mark of a great professor isn’t just the papers they publish. It’s the people they help build.
Russ Salakhutdinov
@rsalakhu
I’ve been asked several times whether Zhilin Yang, the founder of @Kimi_Moonshot was my PhD student. The answer is yes and he is absolutely brilliant.
But I’ve been incredibly fortunate to work with so many outstanding PhD students over the years. So I thought I’d brag a little about them and their career paths (of course there are also many MSc and undergraduate students, sorry if I missed anyone):
Founders / Founding Team Members
Devendra Chaplot @dchaplot PhD, Founding Member Thinking Machines / Mistral
Zhilin Yang PhD, Founder & CEO, Moonshot AI
Jimmy Ba @jimmybajimmyba MSc/PhD, Co-founder xAI
Hubert Tsai PhD, Co-founder Spuree, Apple
Nitish Srivastava @nitishsr PhD, Co-founder Perceptual Machines; Co-founder Vayu Robotics
Charlie Tang PhD, Co-founder Perceptual Machines, DE Shaw.
Professors
Paul Liang @pliang279 PhD, MIT
Ben Eysenbach @ben_eysenbach PhD, Princeton University
Ruosong Wang @RuosongW PhD, Peking University
Bhuwan Dhingra @bhuwandhingra PhD, Duke University
Roger Grosse @RogerGrosse Postdoc, University of Toronto
Alexander Schwing Postdoc, UIUC
Research Scientists
Shuyan Zhou @shuyanzh36, Postdoc, Meta Superintelligence Lab
Tiffany Min @SoYeonTiffMin PhD, Microsoft AI
Murtaza Dalal @mihdalal PhD, Tesla AI
Minji Yoon @MinjiYoon90 , PhD, Microsoft AI
Shrimai Prabhumoye PhD, NVIDIA AI, Mistral
Haitian Sun @sun_haitian PhD, Google DeepMind
Emilio Parisotto PhD, Google DeepMind
Lisa Lee PhD @rl_agent, Google DeepMind
Manzil Zaheer @ManzilZaheer PhD, Google DeepMind
Jamie Kiros PhD, Google Brain, OpenAI
Yuri Burda PhD, OpenAI, Anthropic
Cody Severinski PhD, Amazon -
Russ Salakhutdinov
@rsalakhu
Thank you @yisongyue for kind words.
Yisong Yue
@yisongyue
What @rsalakhu fails to mention of is that he's also great citizen of the community. #generalchair #icml2024
Russ Salakhutdinov
@rsalakhu
I’ve been asked several times whether Zhilin Yang, the founder of @Kimi_Moonshot was my PhD student. The answer is yes and he is absolutely brilliant.
But I’ve been incredibly fortunate to work with so many outstanding PhD students over the years. So I thought I’d brag a little about them and their career paths (of course there are also many MSc and undergraduate students, sorry if I missed anyone):
Founders / Founding Team Members
Devendra Chaplot @dchaplot PhD, Founding Member Thinking Machines / Mistral
Zhilin Yang PhD, Founder & CEO, Moonshot AI
Jimmy Ba @jimmybajimmyba MSc/PhD, Co-founder xAI
Hubert Tsai PhD, Co-founder Spuree, Apple
Nitish Srivastava @nitishsr PhD, Co-founder Perceptual Machines; Co-founder Vayu Robotics
Charlie Tang PhD, Co-founder Perceptual Machines, DE Shaw.
Professors
Paul Liang @pliang279 PhD, MIT
Ben Eysenbach @ben_eysenbach PhD, Princeton University
Ruosong Wang @RuosongW PhD, Peking University
Bhuwan Dhingra @bhuwandhingra PhD, Duke University
Roger Grosse @RogerGrosse Postdoc, University of Toronto
Alexander Schwing Postdoc, UIUC
Research Scientists
Shuyan Zhou @shuyanzh36, Postdoc, Meta Superintelligence Lab
Tiffany Min @SoYeonTiffMin PhD, Microsoft AI
Murtaza Dalal @mihdalal PhD, Tesla AI
Minji Yoon @MinjiYoon90 , PhD, Microsoft AI
Shrimai Prabhumoye PhD, NVIDIA AI, Mistral
Haitian Sun @sun_haitian PhD, Google DeepMind
Emilio Parisotto PhD, Google DeepMind
Lisa Lee PhD @rl_agent, Google DeepMind
Manzil Zaheer @ManzilZaheer PhD, Google DeepMind
Jamie Kiros PhD, Google Brain, OpenAI
Yuri Burda PhD, OpenAI, Anthropic
Cody Severinski PhD, Amazon -
Russ Salakhutdinov reposted
Devendra Chaplot
@dchaplot
The mark of a great professor isn’t just the papers they publish. It’s the people they help build.
Russ Salakhutdinov
@rsalakhu
I’ve been asked several times whether Zhilin Yang, the founder of @Kimi_Moonshot was my PhD student. The answer is yes and he is absolutely brilliant.
But I’ve been incredibly fortunate to work with so many outstanding PhD students over the years. So I thought I’d brag a little about them and their career paths (of course there are also many MSc and undergraduate students, sorry if I missed anyone):
Founders / Founding Team Members
Devendra Chaplot @dchaplot PhD, Founding Member Thinking Machines / Mistral
Zhilin Yang PhD, Founder & CEO, Moonshot AI
Jimmy Ba @jimmybajimmyba MSc/PhD, Co-founder xAI
Hubert Tsai PhD, Co-founder Spuree, Apple
Nitish Srivastava @nitishsr PhD, Co-founder Perceptual Machines; Co-founder Vayu Robotics
Charlie Tang PhD, Co-founder Perceptual Machines, DE Shaw.
Professors
Paul Liang @pliang279 PhD, MIT
Ben Eysenbach @ben_eysenbach PhD, Princeton University
Ruosong Wang @RuosongW PhD, Peking University
Bhuwan Dhingra @bhuwandhingra PhD, Duke University
Roger Grosse @RogerGrosse Postdoc, University of Toronto
Alexander Schwing Postdoc, UIUC
Research Scientists
Shuyan Zhou @shuyanzh36, Postdoc, Meta Superintelligence Lab
Tiffany Min @SoYeonTiffMin PhD, Microsoft AI
Murtaza Dalal @mihdalal PhD, Tesla AI
Minji Yoon @MinjiYoon90 , PhD, Microsoft AI
Shrimai Prabhumoye PhD, NVIDIA AI, Mistral
Haitian Sun @sun_haitian PhD, Google DeepMind
Emilio Parisotto PhD, Google DeepMind
Lisa Lee PhD @rl_agent, Google DeepMind
Manzil Zaheer @ManzilZaheer PhD, Google DeepMind
Jamie Kiros PhD, Google Brain, OpenAI
Yuri Burda PhD, OpenAI, Anthropic
Cody Severinski PhD, Amazon -
Russ Salakhutdinov reposted
Bhuwan Dhingra
@bhuwandhingra
Being a PhD student at CMU from 2015-2020 was indeed a special time! It's incredible to see the amazing things my friends, labmates and collaborators have gone on to achieve!
Russ Salakhutdinov
@rsalakhu
I’ve been asked several times whether Zhilin Yang, the founder of @Kimi_Moonshot was my PhD student. The answer is yes and he is absolutely brilliant.
But I’ve been incredibly fortunate to work with so many outstanding PhD students over the years. So I thought I’d brag a little about them and their career paths (of course there are also many MSc and undergraduate students, sorry if I missed anyone):
Founders / Founding Team Members
Devendra Chaplot @dchaplot PhD, Founding Member Thinking Machines / Mistral
Zhilin Yang PhD, Founder & CEO, Moonshot AI
Jimmy Ba @jimmybajimmyba MSc/PhD, Co-founder xAI
Hubert Tsai PhD, Co-founder Spuree, Apple
Nitish Srivastava @nitishsr PhD, Co-founder Perceptual Machines; Co-founder Vayu Robotics
Charlie Tang PhD, Co-founder Perceptual Machines, DE Shaw.
Professors
Paul Liang @pliang279 PhD, MIT
Ben Eysenbach @ben_eysenbach PhD, Princeton University
Ruosong Wang @RuosongW PhD, Peking University
Bhuwan Dhingra @bhuwandhingra PhD, Duke University
Roger Grosse @RogerGrosse Postdoc, University of Toronto
Alexander Schwing Postdoc, UIUC
Research Scientists
Shuyan Zhou @shuyanzh36, Postdoc, Meta Superintelligence Lab
Tiffany Min @SoYeonTiffMin PhD, Microsoft AI
Murtaza Dalal @mihdalal PhD, Tesla AI
Minji Yoon @MinjiYoon90 , PhD, Microsoft AI
Shrimai Prabhumoye PhD, NVIDIA AI, Mistral
Haitian Sun @sun_haitian PhD, Google DeepMind
Emilio Parisotto PhD, Google DeepMind
Lisa Lee PhD @rl_agent, Google DeepMind
Manzil Zaheer @ManzilZaheer PhD, Google DeepMind
Jamie Kiros PhD, Google Brain, OpenAI
Yuri Burda PhD, OpenAI, Anthropic
Cody Severinski PhD, Amazon -
Russ Salakhutdinov
@rsalakhu
I’ve been asked several times whether Zhilin Yang, the founder of @Kimi_Moonshot was my PhD student. The answer is yes and he is absolutely brilliant.
But I’ve been incredibly fortunate to work with so many outstanding PhD students over the years. So I thought I’d brag a little about them and their career paths (of course there are also many MSc and undergraduate students, sorry if I missed anyone):
Founders / Founding Team Members
Devendra Chaplot @dchaplot PhD, Founding Member Thinking Machines / Mistral
Zhilin Yang PhD, Founder & CEO, Moonshot AI
Jimmy Ba @jimmybajimmyba MSc/PhD, Co-founder xAI
Hubert Tsai PhD, Co-founder Spuree, Apple
Nitish Srivastava @nitishsr PhD, Co-founder Perceptual Machines; Co-founder Vayu Robotics
Charlie Tang PhD, Co-founder Perceptual Machines, DE Shaw.
Professors
Paul Liang @pliang279 PhD, MIT
Ben Eysenbach @ben_eysenbach PhD, Princeton University
Ruosong Wang @RuosongW PhD, Peking University
Bhuwan Dhingra @bhuwandhingra PhD, Duke University
Roger Grosse @RogerGrosse Postdoc, University of Toronto
Alexander Schwing Postdoc, UIUC
Research Scientists
Shuyan Zhou @shuyanzh36, Postdoc, Meta Superintelligence Lab
Tiffany Min @SoYeonTiffMin PhD, Microsoft AI
Murtaza Dalal @mihdalal PhD, Tesla AI
Minji Yoon @MinjiYoon90 , PhD, Microsoft AI
Shrimai Prabhumoye PhD, NVIDIA AI, Mistral
Haitian Sun @sun_haitian PhD, Google DeepMind
Emilio Parisotto PhD, Google DeepMind
Lisa Lee PhD @rl_agent, Google DeepMind
Manzil Zaheer @ManzilZaheer PhD, Google DeepMind
Jamie Kiros PhD, Google Brain, OpenAI
Yuri Burda PhD, OpenAI, Anthropic
Cody Severinski PhD, Amazon -
Russ Salakhutdinov
@rsalakhu
I agree.
Pedro Domingos
@pmddomingos
CMU doesn’t get enough credit for its central role in the growth of ML.
-
Russ Salakhutdinov
@rsalakhu
Thank you for kind words.
Abdo Dameen MSc
@AbdoDameen
Honestly @rsalakhu we are so glad you have dedicated your life to teaching and spreading knowledge.
This is what the world will remember. -
Russ Salakhutdinov
@rsalakhu
lol
dnouri
@dnouri
Russ Salakhutdinov
@rsalakhu
Congratulations to Zhilin Yang, founder and CEO of @Kimi_Moonshot, on the latest Kimi release. What a huge win for the open-source community!
It feels like just yesterday Zhilin was graduating from my lab at CMU, jointly co-advised with William Cohen. Not only did he complete his Ph.D. in just four years, but he also made truly fundamental contributions to ML during his time at CMU.
What a spectacular career! Congrats again Zhilin, and thank you and the entire Kimi team for everything you're doing for the open-source community. -
Russ Salakhutdinov
@rsalakhu
Congratulations to Zhilin Yang, founder and CEO of @Kimi_Moonshot, on the latest Kimi release. What a huge win for the open-source community!
It feels like just yesterday Zhilin was graduating from my lab at CMU, jointly co-advised with William Cohen. Not only did he complete his Ph.D. in just four years, but he also made truly fundamental contributions to ML during his time at CMU.
What a spectacular career! Congrats again Zhilin, and thank you and the entire Kimi team for everything you're doing for the open-source community. -
Russ Salakhutdinov
@rsalakhu
Super cool effort.
Archer Announces Zee, AI Foundation Model Purpose-Built for Aviation, a Key Pillar of Its Physical AI Strategy
news.archer.com/archer-announces-zee-ai-foundation-model-… -
Russ Salakhutdinov
@rsalakhu
Pre-Training Isn’t Bitter Enough – Machine Learning Blog | ML@CMU | Carnegie Mellon University blog.ml.cmu.edu/2026/06/17/pre-training-isnt-bitter-enough/
-
Russ Salakhutdinov reposted
Machine Learning Street Talk
@MLStreetTalk
Very interesting.
-
Russ Salakhutdinov
@rsalakhu
New paper on Understanding Compositional Generalization in Language Model Reasoning
Paper: arxiv.org/abs/2606.18089
This work studies why combining SFT with RL is so effective for post-training reasoning models. We argue that its success stems from compositional generalization, formalized through a hierarchical latent selection model in which reasoning is built by composing reusable atomic modules.
We show theoretically that SFT and RL play complementary but asymmetric roles: SFT provides compositional reasoning traces containing reusable atomic modules, while RL discovers and recombines these modules to achieve compositional generalization.
Experiments show that the strongest performance is achieved when SFT provides broad coverage and RL explores compositions beyond the SFT distribution.
See a more detailed thread by @LingjingKong.
Lingjing Kong
@LingjingKong
1/9 Thanks to @IFM_MBZUAI for sharing our ICML 2026 work. We ask whether a model can solve a new reasoning problem by recombining skills it already knows -- how does the model attain "reasoning creativity" through this kind of compositional generalization?
This post is unavailable. See it on X. -
Russ Salakhutdinov
@rsalakhu
Congratulations to @GoogleDeepMind Computer Use Agents (CUA) team for taking the #1 spot on Odysseys with a vision-only (GUI-based) agent using by Gemini 3.5 Flash. Impressive result given the difficulty of Odysseys and the fact that it is a Flash model.
odysseys-website.pages.dev/leaderboard
Odysseys evaluates realistic, multi-hour web workflows that require sustained planning, memory, reasoning, and verification across numerous websites and tools, beyond short single-step browser tasks.
Exciting to see progress toward truly capable long-horizon web agents. -
Russ Salakhutdinov reposted
Paul Salakhutdinov
@psalakhu
Amazing week working with ecologists and gaining a deeper appreciation for collecting high-quality multimodal data. From GPS calibration and drone flights to troubleshooting robot dogs, I’ve seen the effort behind building reliable datasets. Grateful for the experience!
-
Russ Salakhutdinov
@rsalakhu
Exciting work at MPG Ranch leveraging multimodal data collection, integrating ground-level observations with aerial imagery to build a richer understanding of ecological systems.
Paul Salakhutdinov
@psalakhu
Excited to be working with ecologists at MPG Ranch on a multimodal machine learning project for plant identification.
We're collecting data from robotic dogs and drones, combining ground and aerial imagery to build a multimodal dataset that improves species recognition. -
Russ Salakhutdinov
@rsalakhu
iOSWorld: A Benchmark for Personally Intelligent Phone Agents
Paper: arxiv.org/abs/2606.09764
Code+Environment: iosworld.ioThis media couldn't be displayed. See it on the original post.
Russ Salakhutdinov
@rsalakhu
New work: iOSWorld: A Benchmark for Personally Intelligent Phone Agents
Paper: arxiv.org/abs/2606.09764
Code+Web: iosworld.io
An interactive benchmark built around a persistent user identity spanning 26 custom iOS apps, including analogs of OpenTable, Uber, DoorDash, AirBnB, Chase. These apps contain richly interconnected data, such as messages, transactions, travel histories, social relationships, and personal preferences.
iOSWorld comprises 133 tasks across three levels of difficulty: single-app (27), multi-app (60), and memory & personalization (46). We evaluate leading frontier and open-source computer-use agents under both vision-only and privileged vision+XML settings. Even with privileged access, the strongest frontier model achieves only 52% success, underscoring the challenge of personalized, cross-app reasoning.
We also release an MCP-based tool-use interface for all 26 apps, enabling controlled comparisons between computer-use, tool-use, and hybrid agents. The full benchmark includes the apps, seeded user data, tasks, rubrics, evaluation code, MCP server, and cloud-based infrastructure for running experiments without Mac hardware.
See a more detailed thread by @JangLawrenceK.
Lawrence Jang
@JangLawrenceK
An annoyingly common question I get as an AI PhD student is “When can you get the ChatGPT AI to do something useful? It can’t even work on my phone yet. Siri is pretty dumb.”
To be fair, I think their criticism is correct. I personally wished that AI was better integrated on my phone. LLMs can solve IMO problems, so shouldn’t it be a cakewalk for it to remind me of the text I forgot to respond to last week? Obviously not, since it doesn’t exist in my pocket yet. Or maybe Apple’s new update yesterday fixed this and my research project is obsolete.
We are releasing iOSWorld (iosworld.io), a dynamic iPhone benchmark with 26 newly created apps grounded in personal context. Each of the 26 apps is centrally seeded around one persona, Jordan Avery, and the apps interact together in a realistic ecosystem that reflect real app interactions. We create 133 personalized mobile agent tasks to test in this environment, and the best model, even with privileged information, only scores 51%. -
Russ Salakhutdinov
@rsalakhu
True story. An amazing project. And one that I think will be quite impactful.
Lawrence Jang
@JangLawrenceK
This project started when my advisor (@rsalakhu) asked me to set up OpenClaw when it was going viral. After I got to the point in the setup where it asked me wire my personal iMessages and Gmail, there was no chance I was going to use it. I honestly still can't believe people really use OpenClaw given the current capabilities and cost trade off.
But at one point in the future, I would like for these elements of my digital personal life to be automated with LLMs, althought I just can't trust it in the moment. Which is why I made these personalized sandboxes of high risk, highly personal websites. I hope at one point this year I will feel comfortable wiring up my email and messages to an LLM, the same way I don't hesitate giving coding agents access to my whole codebase.
Lawrence Jang
@JangLawrenceK
Computer-use evals like OSWorld still don’t really test personal assistant use cases: logged-in accounts, user data, personalized workflows, or realistic desktop/web environments.
so we made MyPCBench: A Benchmark for Personally Intelligent Computer-Use Agents, with 184 tasks across 17 popular website clones, seeded with realistic user data and centered on Michael Scott’s hypothetical desktop.
It’s easy to adopt if you already use OSWorld-style runners - I view it as an personalization-focused, more realistic upgrade for CUA evals.
Website: mypcbench.com
Paper: arxiv.org/abs/2606.16748
Code: github.com/ljang0/MyPCBench -
Russ Salakhutdinov reposted
Jing Yu Koh
@kohjingyu
For the entire semester, Lawrence was sitting next to me in the office making these odd but really nice looking websites.
What are you doing? I finally asked. Why are you burning through two Claude Max plans? What are you converting these tokens into?
"I'm building LockedIn, a simulated version of everyone's least favorite professional social media site.", he said. "I need more tokens so I can connect it to HooliChat.", Lawrence grunted silently as his two Max plans hit their limits before the Reset.
LockedIn? HooliChat? I thought he must have finally snapped. This has to be a waste of time and tokens, I silently thought, as I went back to finetuning more tiny models on A6000 GPUs (obviously a better use of time than building evals).
Another week passes. His Claude Code quota refreshes. He gets more app ideas from looking at how people actually use agents like OpenClaw.
"This is HangryDash, it allows me to order 46 shake shack burgers and leave no tip, just like I do in real life. It's connected to my Gringotts which is my bank account."
What do you mean it's connected?
"I mean, it's all connected, man. All the apps. All the data. This is a simulation of someone's entire life. It's a simulation of the digital life of Michael Scott."
What?
"Yeah, his bank account is connected to his HangryDash and taxi apps, his email is connected to his HooliWork messaging app, and he has simulated coworkers he can talk to or flirt with in HooliChat. They even respond, because they're locally hosted LLMs."
It's all connected. We're all connected. I got it now. "Can I try?"
I spend an hour playing with it, bankrupting Michael on HangryDash (but it's all good, because we can always reset and re-evaluate) and saw the vision. MyPCBench is hyper realistic, it is all connected, and it is all personalized based on Michael Scott's life. It's unlike many other existing CUA evals, because solving these tasks requires navigating through Michael Scott's personal digital history, and completing tasks that he would be happy with. It is how we will probably have to evaluate CUAs in the future, as they start to become more tailored to the individual.
Best of all, it's simulated, easy to run, and very fun to watch. We don't need to mess with the real world. We can test OpenClaw-like workflows without accidentally deleting data or messing with our coworkers.
Lawrence Jang
@JangLawrenceK
Computer-use evals like OSWorld still don’t really test personal assistant use cases: logged-in accounts, user data, personalized workflows, or realistic desktop/web environments.
so we made MyPCBench: A Benchmark for Personally Intelligent Computer-Use Agents, with 184 tasks across 17 popular website clones, seeded with realistic user data and centered on Michael Scott’s hypothetical desktop.
It’s easy to adopt if you already use OSWorld-style runners - I view it as an personalization-focused, more realistic upgrade for CUA evals.
Website: mypcbench.com
Paper: arxiv.org/abs/2606.16748
Code: github.com/ljang0/MyPCBench -
Russ Salakhutdinov reposted
Delta
@DeltaCommunityX
If you’re excited about computer-use agents, check out Lawrence’s new paper!!
Lawrence Jang
@JangLawrenceK
Computer-use evals like OSWorld still don’t really test personal assistant use cases: logged-in accounts, user data, personalized workflows, or realistic desktop/web environments.
so we made MyPCBench: A Benchmark for Personally Intelligent Computer-Use Agents, with 184 tasks across 17 popular website clones, seeded with realistic user data and centered on Michael Scott’s hypothetical desktop.
It’s easy to adopt if you already use OSWorld-style runners - I view it as an personalization-focused, more realistic upgrade for CUA evals.
Website: mypcbench.com
Paper: arxiv.org/abs/2606.16748
Code: github.com/ljang0/MyPCBench -
Russ Salakhutdinov reposted
Lawrence Jang
@JangLawrenceK
Computer-use evals like OSWorld still don’t really test personal assistant use cases: logged-in accounts, user data, personalized workflows, or realistic desktop/web environments.
so we made MyPCBench: A Benchmark for Personally Intelligent Computer-Use Agents, with 184 tasks across 17 popular website clones, seeded with realistic user data and centered on Michael Scott’s hypothetical desktop.
It’s easy to adopt if you already use OSWorld-style runners - I view it as an personalization-focused, more realistic upgrade for CUA evals.
Website: mypcbench.com
Paper: arxiv.org/abs/2606.16748
Code: github.com/ljang0/MyPCBench -
Russ Salakhutdinov
@rsalakhu
Congrats to the @browser_use team for taking the #1 spot on Odysseys, a highly challenging benchmark for long-horizon web agents:
odysseys-website.pages.dev/leaderboard
Odysseys evaluates realistic, multi-hour web workflows that require sustained planning, memory, reasoning, and verification across numerous websites and tools, far beyond short single-step browser tasks.
Exciting progress toward truly capable long-horizon agents. -
Russ Salakhutdinov reposted
tsunami_crypto
@ls_brd
phone-level agents are the real utility test that actually matters
let me know when i can ask it to cancel my subscriptions for me -
Russ Salakhutdinov reposted
ikka
@Shahules786
Excellent work! Love that the work compares vision only with other strategies and also tool-use. Would love to connect
-
Russ Salakhutdinov reposted
MH Sun
@andysun_sw
the strongest frontier model achieves only 52% success, underscoring the challenge of personalized, cross-app reasoning
Russ Salakhutdinov
@rsalakhu
New work: iOSWorld: A Benchmark for Personally Intelligent Phone Agents
Paper: arxiv.org/abs/2606.09764
Code+Web: iosworld.io
An interactive benchmark built around a persistent user identity spanning 26 custom iOS apps, including analogs of OpenTable, Uber, DoorDash, AirBnB, Chase. These apps contain richly interconnected data, such as messages, transactions, travel histories, social relationships, and personal preferences.
iOSWorld comprises 133 tasks across three levels of difficulty: single-app (27), multi-app (60), and memory & personalization (46). We evaluate leading frontier and open-source computer-use agents under both vision-only and privileged vision+XML settings. Even with privileged access, the strongest frontier model achieves only 52% success, underscoring the challenge of personalized, cross-app reasoning.
We also release an MCP-based tool-use interface for all 26 apps, enabling controlled comparisons between computer-use, tool-use, and hybrid agents. The full benchmark includes the apps, seeded user data, tasks, rubrics, evaluation code, MCP server, and cloud-based infrastructure for running experiments without Mac hardware.
See a more detailed thread by @JangLawrenceK.
Lawrence Jang
@JangLawrenceK
An annoyingly common question I get as an AI PhD student is “When can you get the ChatGPT AI to do something useful? It can’t even work on my phone yet. Siri is pretty dumb.”
To be fair, I think their criticism is correct. I personally wished that AI was better integrated on my phone. LLMs can solve IMO problems, so shouldn’t it be a cakewalk for it to remind me of the text I forgot to respond to last week? Obviously not, since it doesn’t exist in my pocket yet. Or maybe Apple’s new update yesterday fixed this and my research project is obsolete.
We are releasing iOSWorld (iosworld.io), a dynamic iPhone benchmark with 26 newly created apps grounded in personal context. Each of the 26 apps is centrally seeded around one persona, Jordan Avery, and the apps interact together in a realistic ecosystem that reflect real app interactions. We create 133 personalized mobile agent tasks to test in this environment, and the best model, even with privileged information, only scores 51%. -
Russ Salakhutdinov
@rsalakhu
MyPCBench: A Benchmark for Personally Intelligent Computer-Use Agents
Paper: arxiv.org/abs/2606.16748
Code, Environment and Tasks, Agentic harness: mypcbench.com
MyPCBench provides a Linux desktop environment with 17 simulated real-world web applications and a complete desktop software stack.
The benchmark contains 184 tasks inspired by authentic OpenClaw community requests and evaluates agents through a unified computer-use + bash interface.
We benchmark leading closed- and open-weight models and find that the strongest model, Claude Opus 4.6, successfully completes 55.4%of tasks. We find that failures are concentrated on long-horizon, multi-application workflows, highlighting that personalization and persistent user context remain key challenges for the next generation of AI personal assistants.
Joint work with @JangLawrenceK, @andrewkjang7 and @kohjingyu. -
Russ Salakhutdinov reposted
Kosta Derpanis
@CSProfKGD
2026 is the year of robot manipulation
Russ Salakhutdinov
@rsalakhu
New work on FACTR 2: Learning External Force Sensing for Commodity Robot Arms Improves Policy Learning:
Paper: arxiv.org/abs/2606.12406
Web: jasonjzliu.com/factr2/
FACTR 2 shows that learned force signals can both enable force-feedback teleoperation on low-cost manipulators and improve behavior cloning (BC) policies for contact-rich tasks. It consists of two components:
1. Neural External Torque Estimation (NEXT): A lightweight model that infers external joint torques without dedicated force sensors.
2. Force-Informed Re-Sampling Training (FIRST): A training strategy that uses the learned force signal to identify and upsample task-critical moments.
The key insight is simple: policy failures rarely occur in free space, they occur during brief pre-contact alignment and contact-rich interactions, where precise corrections matter most.
Together, NEXT and FIRST bring force-aware teleoperation and robust long-horizon contact-rich policy learning to off-the-shelf robot arms, without requiring additional sensing hardware.
See a more detailed thread by @JasonJZLiu.This media couldn't be displayed. See it on the original post.
Jason Liu
@JasonJZLiu
💥Introducing FACTR 2, learning external force sensing on commodity robot arms without needing dedicated sensors.
We show that learned force signals enable force-feedback teleop on low-cost arms and improve BC policies.
FACTR 2 consists of:
1. Neural External Torque (NEXT): learns external forces without needing dedicated force sensors.
2. Force-Informed Re-Sampling Training (FIRST): uses the learned force signal to identify task-critical regions and upsample them during training.
w/ @StevenOh_ @_tonytao_
🧵(1/N)This media couldn't be displayed. See it on the original post. -
Russ Salakhutdinov reposted
GEE!
@KhieEmm
Cool—if they can get reliable external force sensing on cheap arms, that’s a big step toward real force feedback without making the hardware crazy.
-
Russ Salakhutdinov
@rsalakhu
New work on FACTR 2: Learning External Force Sensing for Commodity Robot Arms Improves Policy Learning:
Paper: arxiv.org/abs/2606.12406
Web: jasonjzliu.com/factr2/
FACTR 2 shows that learned force signals can both enable force-feedback teleoperation on low-cost manipulators and improve behavior cloning (BC) policies for contact-rich tasks. It consists of two components:
1. Neural External Torque Estimation (NEXT): A lightweight model that infers external joint torques without dedicated force sensors.
2. Force-Informed Re-Sampling Training (FIRST): A training strategy that uses the learned force signal to identify and upsample task-critical moments.
The key insight is simple: policy failures rarely occur in free space, they occur during brief pre-contact alignment and contact-rich interactions, where precise corrections matter most.
Together, NEXT and FIRST bring force-aware teleoperation and robust long-horizon contact-rich policy learning to off-the-shelf robot arms, without requiring additional sensing hardware.
See a more detailed thread by @JasonJZLiu.This media couldn't be displayed. See it on the original post.
Jason Liu
@JasonJZLiu
💥Introducing FACTR 2, learning external force sensing on commodity robot arms without needing dedicated sensors.
We show that learned force signals enable force-feedback teleop on low-cost arms and improve BC policies.
FACTR 2 consists of:
1. Neural External Torque (NEXT): learns external forces without needing dedicated force sensors.
2. Force-Informed Re-Sampling Training (FIRST): uses the learned force signal to identify task-critical regions and upsample them during training.
w/ @StevenOh_ @_tonytao_
🧵(1/N)This media couldn't be displayed. See it on the original post. -
Russ Salakhutdinov reposted
Russ Salakhutdinov
@rsalakhu
New work: iOSWorld: A Benchmark for Personally Intelligent Phone Agents
Paper: arxiv.org/abs/2606.09764
Code+Web: iosworld.io
An interactive benchmark built around a persistent user identity spanning 26 custom iOS apps, including analogs of OpenTable, Uber, DoorDash, AirBnB, Chase. These apps contain richly interconnected data, such as messages, transactions, travel histories, social relationships, and personal preferences.
iOSWorld comprises 133 tasks across three levels of difficulty: single-app (27), multi-app (60), and memory & personalization (46). We evaluate leading frontier and open-source computer-use agents under both vision-only and privileged vision+XML settings. Even with privileged access, the strongest frontier model achieves only 52% success, underscoring the challenge of personalized, cross-app reasoning.
We also release an MCP-based tool-use interface for all 26 apps, enabling controlled comparisons between computer-use, tool-use, and hybrid agents. The full benchmark includes the apps, seeded user data, tasks, rubrics, evaluation code, MCP server, and cloud-based infrastructure for running experiments without Mac hardware.
See a more detailed thread by @JangLawrenceK.
Lawrence Jang
@JangLawrenceK
An annoyingly common question I get as an AI PhD student is “When can you get the ChatGPT AI to do something useful? It can’t even work on my phone yet. Siri is pretty dumb.”
To be fair, I think their criticism is correct. I personally wished that AI was better integrated on my phone. LLMs can solve IMO problems, so shouldn’t it be a cakewalk for it to remind me of the text I forgot to respond to last week? Obviously not, since it doesn’t exist in my pocket yet. Or maybe Apple’s new update yesterday fixed this and my research project is obsolete.
We are releasing iOSWorld (iosworld.io), a dynamic iPhone benchmark with 26 newly created apps grounded in personal context. Each of the 26 apps is centrally seeded around one persona, Jordan Avery, and the apps interact together in a realistic ecosystem that reflect real app interactions. We create 133 personalized mobile agent tasks to test in this environment, and the best model, even with privileged information, only scores 51%. -
Russ Salakhutdinov reposted
Lawrence Jang
@JangLawrenceK
An annoyingly common question I get as an AI PhD student is “When can you get the ChatGPT AI to do something useful? It can’t even work on my phone yet. Siri is pretty dumb.”
To be fair, I think their criticism is correct. I personally wished that AI was better integrated on my phone. LLMs can solve IMO problems, so shouldn’t it be a cakewalk for it to remind me of the text I forgot to respond to last week? Obviously not, since it doesn’t exist in my pocket yet. Or maybe Apple’s new update yesterday fixed this and my research project is obsolete.
We are releasing iOSWorld (iosworld.io), a dynamic iPhone benchmark with 26 newly created apps grounded in personal context. Each of the 26 apps is centrally seeded around one persona, Jordan Avery, and the apps interact together in a realistic ecosystem that reflect real app interactions. We create 133 personalized mobile agent tasks to test in this environment, and the best model, even with privileged information, only scores 51%. -
Russ Salakhutdinov reposted
Jing Yu Koh
@kohjingyu
A lot of computer use work focuses on desktop environments, but automating tasks on the phone also has very high potential upside! iOSWorld is a fun new benchmark created by Lawrence to measure CUAs on how well they do on mobile + personalized tasks.
Lawrence Jang
@JangLawrenceK
An annoyingly common question I get as an AI PhD student is “When can you get the ChatGPT AI to do something useful? It can’t even work on my phone yet. Siri is pretty dumb.”
To be fair, I think their criticism is correct. I personally wished that AI was better integrated on my phone. LLMs can solve IMO problems, so shouldn’t it be a cakewalk for it to remind me of the text I forgot to respond to last week? Obviously not, since it doesn’t exist in my pocket yet. Or maybe Apple’s new update yesterday fixed this and my research project is obsolete.
We are releasing iOSWorld (iosworld.io), a dynamic iPhone benchmark with 26 newly created apps grounded in personal context. Each of the 26 apps is centrally seeded around one persona, Jordan Avery, and the apps interact together in a realistic ecosystem that reflect real app interactions. We create 133 personalized mobile agent tasks to test in this environment, and the best model, even with privileged information, only scores 51%. -
Russ Salakhutdinov
@rsalakhu
New work: iOSWorld: A Benchmark for Personally Intelligent Phone Agents
Paper: arxiv.org/abs/2606.09764
Code+Web: iosworld.io
An interactive benchmark built around a persistent user identity spanning 26 custom iOS apps, including analogs of OpenTable, Uber, DoorDash, AirBnB, Chase. These apps contain richly interconnected data, such as messages, transactions, travel histories, social relationships, and personal preferences.
iOSWorld comprises 133 tasks across three levels of difficulty: single-app (27), multi-app (60), and memory & personalization (46). We evaluate leading frontier and open-source computer-use agents under both vision-only and privileged vision+XML settings. Even with privileged access, the strongest frontier model achieves only 52% success, underscoring the challenge of personalized, cross-app reasoning.
We also release an MCP-based tool-use interface for all 26 apps, enabling controlled comparisons between computer-use, tool-use, and hybrid agents. The full benchmark includes the apps, seeded user data, tasks, rubrics, evaluation code, MCP server, and cloud-based infrastructure for running experiments without Mac hardware.
See a more detailed thread by @JangLawrenceK.
Lawrence Jang
@JangLawrenceK
An annoyingly common question I get as an AI PhD student is “When can you get the ChatGPT AI to do something useful? It can’t even work on my phone yet. Siri is pretty dumb.”
To be fair, I think their criticism is correct. I personally wished that AI was better integrated on my phone. LLMs can solve IMO problems, so shouldn’t it be a cakewalk for it to remind me of the text I forgot to respond to last week? Obviously not, since it doesn’t exist in my pocket yet. Or maybe Apple’s new update yesterday fixed this and my research project is obsolete.
We are releasing iOSWorld (iosworld.io), a dynamic iPhone benchmark with 26 newly created apps grounded in personal context. Each of the 26 apps is centrally seeded around one persona, Jordan Avery, and the apps interact together in a realistic ecosystem that reflect real app interactions. We create 133 personalized mobile agent tasks to test in this environment, and the best model, even with privileged information, only scores 51%. -
Russ Salakhutdinov reposted
Mediant
@MediantMusic
What started as a simple question — how can musicians practice more effectively? — has grown into something we're proud to share. We're working on a tool to transform how musicians practice: smarter, faster, and with purpose. Stay tuned — we can't wait to show you what we built.
-
Russ Salakhutdinov reposted
Russ Salakhutdinov
@rsalakhu
Foresight will be the defining frontier on the path to AGI.
I am excited to start Sooth Labs with my amazing co-founders: Yaser Sheikh @subail, Chuck Hoover @chuckjhoover, David LaRose, and Shih-En Wei.
Deeply grateful to Aydin Senkut @asenkut and Feyza Haskaraman @FHaskaraman at @felicis for leading the round, alongside an exceptional group of partners.
www.bloomberg.com/news/articles/2026-04-22/ai-pioneers-ba… -
Russ Salakhutdinov reposted
Russ Salakhutdinov
@rsalakhu
New work on Scaling Test-Time Compute for Agentic Coding:
Paper: arxiv.org/abs/2604.16529
This work introduces a test-time scaling framework for agentic coding that converts rollouts into structured summaries capturing key hypotheses, progress, and failure modes while discarding low-signal details.
This enables two forms of inference-time scaling: (1) Recursive Tournament Voting (RTV) for parallel selection via iterative small-group comparisons, and (2) Parallel-Distill-Refine (PDR) for sequential improvement by conditioning new rollouts on distilled summaries.
Our approach consistently boosts performance on frontier benchmarks. On SWE-Bench Verified, Claude-4.5-Opus improves from 70.9% → 77.6%, and on Terminal-Bench v2.0 from 46.9% → 59.1%. These gains highlight that effective test-time scaling for long-horizon agents hinges on representation, selection, and reuse, not just sampling more trajectories.
Check out a more detailed thread by @anirudhg9119.
Anirudh Goyal
@anirudhg9119
How do coding agents get better from experience?
Past Attempts as Interface: Turn rollouts into reusable summaries that future attempts can build on.
arxiv.org/abs/2604.16529 -
Russ Salakhutdinov reposted
Russ Salakhutdinov
@rsalakhu
How well do today’s frontier models handle long-horizon, multi-step web agent tasks, such as identifying the top 25 U.S. CS PhD programs with ML/AI faculty likely accepting students and compiling the results into a structured sheet?
Check out our new work on Odysseys: Benchmarking Web Agents on Realistic Long Horizon Tasks
Paper: odysseys-website.pages.dev
Leaderboard: odysseys-website.pages.dev/leaderboard
We introduce Odysseys, a benchmark of 200 long-horizon tasks derived from real browsing sessions and evaluated on the live Internet. We show that binary pass/fail is inadequate in this setting and propose rubric-based evaluation, which better aligns with human judgment and provides more informative signals.
Across leading models, the best achieves only 44.5% success, leaving substantial headroom. We further introduce a Trajectory Efficiency metric (rubric score per step) and find efficiency remains extremely low (1.15%), highlighting a key bottleneck.
Odysseys provides a realistic benchmark for measuring progress toward web agents capable of sustained, efficient, real-world operation.
See a more detailed thread by @kohjingyu.
Jing Yu Koh
@kohjingyu
One of the things I’m most excited about this year is building agents that can work productively for hours, days, or weeks. Coding agents are starting to become very competent at this, but what about computer use agents?
Our new benchmark, Odysseys (co-led with @JangLawrenceK) is a set of 200 new tasks derived from real world browsing behavior that measure long horizon web navigation capabilities (potentially up to hours of web browsing work). Interestingly, we find that frontier CUAs are already surprisingly good at working productively for up to an hour on these tasks, but there’s a lot of work to be done in making them even more efficient.
Like every other AI researcher, my real dream is to open a cafe once we solve ASI. So, here’s Opus 4.6 doing some market research for me ("I want to do market research on the most popular cafes in Singapore. Analyse the menus of the top 10 cafes in Singapore (by Google reviews/ratings), and make sure we include at least 1 from the North/South/East/West/Central regions of Singapore. Keep the relevant pages of each cafe open, and summarise their pricing, menu offerings, unique selling points, making sure to reference which tab is opened for each cafe. For each cafe, also help me figure out how long it would take to get to it from Tampines MRT, and include this in your final summary.").
I was very impressed to see Opus 4.6 complete this task after working for 52 mins, satisfying all 7 rubrics that corresponded to this task. It provided a very nice markdown summary at the end that gave me all the information I asked for! -
Russ Salakhutdinov reposted
Russ Salakhutdinov
@rsalakhu
New work on Multi-Agent Computer Use (MACU).
The future of computer-use agents lies in multi-agent systems that combine planning, coordination, and parallel execution.
Paper: arxiv.org/abs/2606.01533
Webside + Code: jykoh.com/macu
MACU introduces a manager agent that decomposes tasks into a dynamic directed acyclic graph (DAG) of subtasks, dispatches parallel subagents, and continuously updates the plan as new information arrives.
Across OSWorld, Online-Mind2Web, WebTrailBench, and Odysseys, we see performance improvement by 4.7–25.5%, achieving better test-time scaling, and solving long-horizon tasks that single-agent systems often fail to complete.
On Odysseys, MACU reduces task completion time by 1.5×, showing that multi-agent coordination is a powerful path toward more capable and efficient computer-use agents.
See a more detail thread by @kohjingyu.
Jing Yu Koh
@kohjingyu
Computer use agents are slow and brittle. The fix isn’t just stronger models, but also deploying them as multi-agent systems.
MACU is a general Multi-Agent Computer Use framework that consistently lifts success rates by 3.4-25.5% and is up to 1.5x faster on long-horizon tasks.🧵 -
Russ Salakhutdinov reposted
Jing Yu Koh
@kohjingyu
CUAs need to move beyond the prevailing single serial agent paradigm, and start being researched, evaluated, and deployed as multi-agent systems. We hope MACU provides useful insights and a reproducible foundation for future research.
💻 Code: github.com/kohjingyu/multi-agent-computer-use
📄 Paper: arxiv.org/abs/2606.01533
🌐 Website: jykoh.com/macu
work done at @mldcmu @LTIatCMU w/ @rsalakhu @dan_fried -
Russ Salakhutdinov reposted
Daniel Fried
@dan_fried
New work: a simple and general multi-agent computer use framework. It uses a manager to plan and re-plan by creating a task DAG, with subagents for parallel execution.
It improves success rate across benchmarks, and substantially improves efficiency on long-horizon tasks.
Jing Yu Koh
@kohjingyu
Computer use agents are slow and brittle. The fix isn’t just stronger models, but also deploying them as multi-agent systems.
MACU is a general Multi-Agent Computer Use framework that consistently lifts success rates by 3.4-25.5% and is up to 1.5x faster on long-horizon tasks.🧵 -
Russ Salakhutdinov reposted
Lawrence Jang
@JangLawrenceK
This project actually started with JY creating a fake slack of our lab and the "JY" agent asking the "Lawrence" agent to send the "Russ" agent edited memes
I though that was honestly much better than running multi-agent computer use on academic benchmarks out of status quo, but I am pretty excited to work on the direction of multi-agent CUAs in the coming months - great work from Mr. Koh
Jing Yu Koh
@kohjingyu
Computer use agents are slow and brittle. The fix isn’t just stronger models, but also deploying them as multi-agent systems.
MACU is a general Multi-Agent Computer Use framework that consistently lifts success rates by 3.4-25.5% and is up to 1.5x faster on long-horizon tasks.🧵 -
Russ Salakhutdinov
@rsalakhu
New work on Multi-Agent Computer Use (MACU).
The future of computer-use agents lies in multi-agent systems that combine planning, coordination, and parallel execution.
Paper: arxiv.org/abs/2606.01533
Webside + Code: jykoh.com/macu
MACU introduces a manager agent that decomposes tasks into a dynamic directed acyclic graph (DAG) of subtasks, dispatches parallel subagents, and continuously updates the plan as new information arrives.
Across OSWorld, Online-Mind2Web, WebTrailBench, and Odysseys, we see performance improvement by 4.7–25.5%, achieving better test-time scaling, and solving long-horizon tasks that single-agent systems often fail to complete.
On Odysseys, MACU reduces task completion time by 1.5×, showing that multi-agent coordination is a powerful path toward more capable and efficient computer-use agents.
See a more detail thread by @kohjingyu.
Jing Yu Koh
@kohjingyu
Computer use agents are slow and brittle. The fix isn’t just stronger models, but also deploying them as multi-agent systems.
MACU is a general Multi-Agent Computer Use framework that consistently lifts success rates by 3.4-25.5% and is up to 1.5x faster on long-horizon tasks.🧵 -
Russ Salakhutdinov
@rsalakhu
Cool startup @MediantMusic: an AI-powered personal music coach built to help musicians practice more effectively.
You can upload a recording of yourself playing, and the system provides detailed, personalized feedback on how to improve, much like working with a professional music teacher.
Definitely worth checking out.
Mediant
@MediantMusic
What started as a simple question — how can musicians practice more effectively? — has grown into something we're proud to share. We're working on a tool to transform how musicians practice: smarter, faster, and with purpose. Stay tuned — we can't wait to show you what we built.
-
Russ Salakhutdinov reposted
Paul Salakhutdinov
@psalakhu
Cool project I’ve been working on with a few friends!
Mediant
@MediantMusic
What started as a simple question — how can musicians practice more effectively? — has grown into something we're proud to share. We're working on a tool to transform how musicians practice: smarter, faster, and with purpose. Stay tuned — we can't wait to show you what we built.
-
Russ Salakhutdinov
@rsalakhu
Cool startup @MediantMusic: an AI-powered personal music coach built to help musicians practice more effectively.
You can upload a recording of yourself playing, and the system provides detailed, personalized feedback on how to improve, much like working with a professional music teacher.
Definitely worth checking out.
Mediant
@MediantMusic
What started as a simple question — how can musicians practice more effectively? — has grown into something we're proud to share. We're working on a tool to transform how musicians practice: smarter, faster, and with purpose. Stay tuned — we can't wait to show you what we built.
-
Russ Salakhutdinov reposted
Lawrence Jang
@JangLawrenceK
Yes, can you email me the trajectories so we can update the leaderboard?
-
Russ Salakhutdinov reposted
Alexander Yue
@Alezander9
New benchmark "Odysseys" by CMU.
Microsoft recently topped the chart at 61% with their auto-eval agent WebWright. Today I scored a 70% with BrowserCode and Opus 4.7 -
Russ Salakhutdinov
@rsalakhu
Congrats to the Webwright team microsoft.github.io/Webwright at @MSFTResearch for taking the #1 spot on Odysseys, a highly challenging benchmark for long-horizon web agents:
odysseys-website.pages.dev/leaderboard
Odysseys evaluates realistic, multi-hour web workflows that require sustained planning, memory, reasoning, and verification across many websites and tools. These are far beyond short single-step browser tasks.
For example, if you are searching for CS faculty positions, a single task could involve building a comprehensive Excel tracker of openings across the top CS schools using CSRankings as the master checklist; verifying every school directly through department, engineering, and university careers pages for CS/AI/ML/data science/robotics/vision faculty roles; opening and validating each posting; maintaining structured evidence and verification tabs; and finishing with a completeness audit and summary of hiring trends.
Exciting progress toward truly capable long-horizon web agents. -
Russ Salakhutdinov
@rsalakhu
Very excited to have you on board.
Qingchuan (Tony) Yang
@qcyang20xx
Today was Day 1 for me as a research intern at Sooth Labs!
Super cool team making some super cool forecasts🔮
Honored to be part of this exciting journey🔥
Russ Salakhutdinov
@rsalakhu
Foresight will be the defining frontier on the path to AGI.
I am excited to start Sooth Labs with my amazing co-founders: Yaser Sheikh @subail, Chuck Hoover @chuckjhoover, David LaRose, and Shih-En Wei.
Deeply grateful to Aydin Senkut @asenkut and Feyza Haskaraman @FHaskaraman at @felicis for leading the round, alongside an exceptional group of partners.
www.bloomberg.com/news/articles/2026-04-22/ai-pioneers-ba… -
Russ Salakhutdinov reposted
Jing Yu Koh
@kohjingyu
We think that Odysseys is a step towards measuring and building CUAs capable of eventually performing hours and days worth of useful work. We have a lot of other interesting results and analysis in the website and paper:
🌐 Website: odysseys-website.pages.dev
💻 Code: github.com/ljang0/odysseys
📝 Paper: arxiv.org/abs/2604.24964
📊 Leaderboard: odysseys-website.pages.dev/leaderboard
(w/ @JangLawrenceK, @dan_fried, @rsalakhu) -
Russ Salakhutdinov reposted
Zhiting Hu
@ZhitingHu
🏆Honored to receive the Test of Time Award Honorable Mention #aistats2026 for our 2016 work Deep Kernel Learning, with the amazing @andrewgwils @rsalakhu @ericxing
What a decade of AI progress! While GenAI is now driving massive real-world applications, the deepest underlying challenge remains: learning efficient representations of the world—for understanding, generation, predicting future worlds, and reasoning in the latent space.
So much fun to think about for the next decade!⏳ -
Russ Salakhutdinov reposted
Haifeng Xu
@haifengxu0
Check out our ICML-26 workshop (@icmlconf) on Forecasting as a New Frontier of AI, co-organized by our lab and multiple frontier players @rsalakhu (from recently launched Sooth Labs), @EzraKarger (from @Research_FRI) and @nicoup (from @fleet_ai)
Website: forecasting-workshop.github.io
Paper submission: openreview.net/group?id=ICML.cc/2026/Workshop/Forecast
@UChicagoCS @DSI_UChicago -
Russ Salakhutdinov reposted
Daniel Fried
@dan_fried
How successfully -- and efficiently! -- can agents carry out long-horizon tasks on the web? We built a benchmark of ~200 multi-site tasks, based on people's real browsing history. Many of them take hours to solve.
Paper: odysseys-website.pages.dev
Led by @JangLawrenceK and @kohjingyu, with @rsalakhu
Jing Yu Koh
@kohjingyu
One of the things I’m most excited about this year is building agents that can work productively for hours, days, or weeks. Coding agents are starting to become very competent at this, but what about computer use agents?
Our new benchmark, Odysseys (co-led with @JangLawrenceK) is a set of 200 new tasks derived from real world browsing behavior that measure long horizon web navigation capabilities (potentially up to hours of web browsing work). Interestingly, we find that frontier CUAs are already surprisingly good at working productively for up to an hour on these tasks, but there’s a lot of work to be done in making them even more efficient.
Like every other AI researcher, my real dream is to open a cafe once we solve ASI. So, here’s Opus 4.6 doing some market research for me ("I want to do market research on the most popular cafes in Singapore. Analyse the menus of the top 10 cafes in Singapore (by Google reviews/ratings), and make sure we include at least 1 from the North/South/East/West/Central regions of Singapore. Keep the relevant pages of each cafe open, and summarise their pricing, menu offerings, unique selling points, making sure to reference which tab is opened for each cafe. For each cafe, also help me figure out how long it would take to get to it from Tampines MRT, and include this in your final summary.").
I was very impressed to see Opus 4.6 complete this task after working for 52 mins, satisfying all 7 rubrics that corresponded to this task. It provided a very nice markdown summary at the end that gave me all the information I asked for! -
Russ Salakhutdinov
@rsalakhu
How well do today’s frontier models handle long-horizon, multi-step web agent tasks, such as identifying in-person, 6-week, on-campus university engineering summer programs for high school students and compiling the results into a structured sheet?
Check out our new work on Odysseys: Benchmarking Web Agents on Realistic Long Horizon Tasks:
Paper: arxiv.org/abs/2604.24964
Wedsite: odysseys-website.pages.dev
Leaderboard: odysseys-website.pages.dev/leaderboard
We introduce Odysseys, a benchmark of 200 long-horizon tasks derived from real browsing sessions and evaluated on the live Internet. We show that binary pass/fail is inadequate in this setting and propose rubric-based evaluation, which better aligns with human judgment and provides more informative signals.
Across leading models, the best achieves only 44.5% success, leaving substantial headroom. We further introduce a Trajectory Efficiency metric (rubric score per step) and find efficiency remains extremely low (1.15%), highlighting a key bottleneck.
Odysseys provides a realistic benchmark for measuring progress toward web agents capable of sustained, efficient, real-world operation.
See a more detailed thread by @JangLawrenceK.
Lawrence Jang
@JangLawrenceK
AI agents can work pretty well on the web now for short tasks. I wanted to know: could they go longer, on harder tasks?
Can an agent plan 2 weddings in different cities and a honeymoon within the same month, or find the most suitable culinary arts school across the US for my post PhD plans?
We are releasing Odysseys: a benchmark of 200 long-horizon web agent tasks evaluated on the live internet. All our tasks are inspired from real human data and many take hours to complete.
The best frontier model we tested (Claude Opus 4.6) reaches only 44.5% perfect-task success, leaving substantial room for improvement.
I donated a couple of my own automation wishes to this benchmark. My favorite contribution was to “Rank the top 10 ACL + Meniscus surgeons in the area” - as this took me a decent amount of time to do myself when I got my own knee fixed. GPT 5.5 was able to do this for me with 4 dollars and 30 minutes! -
Russ Salakhutdinov
@rsalakhu
How well do today’s frontier models handle long-horizon, multi-step web agent tasks, such as identifying the top 25 U.S. CS PhD programs with ML/AI faculty likely accepting students and compiling the results into a structured sheet?
Check out our new work on Odysseys: Benchmarking Web Agents on Realistic Long Horizon Tasks
Paper: odysseys-website.pages.dev
Leaderboard: odysseys-website.pages.dev/leaderboard
We introduce Odysseys, a benchmark of 200 long-horizon tasks derived from real browsing sessions and evaluated on the live Internet. We show that binary pass/fail is inadequate in this setting and propose rubric-based evaluation, which better aligns with human judgment and provides more informative signals.
Across leading models, the best achieves only 44.5% success, leaving substantial headroom. We further introduce a Trajectory Efficiency metric (rubric score per step) and find efficiency remains extremely low (1.15%), highlighting a key bottleneck.
Odysseys provides a realistic benchmark for measuring progress toward web agents capable of sustained, efficient, real-world operation.
See a more detailed thread by @kohjingyu.
Jing Yu Koh
@kohjingyu
One of the things I’m most excited about this year is building agents that can work productively for hours, days, or weeks. Coding agents are starting to become very competent at this, but what about computer use agents?
Our new benchmark, Odysseys (co-led with @JangLawrenceK) is a set of 200 new tasks derived from real world browsing behavior that measure long horizon web navigation capabilities (potentially up to hours of web browsing work). Interestingly, we find that frontier CUAs are already surprisingly good at working productively for up to an hour on these tasks, but there’s a lot of work to be done in making them even more efficient.
Like every other AI researcher, my real dream is to open a cafe once we solve ASI. So, here’s Opus 4.6 doing some market research for me ("I want to do market research on the most popular cafes in Singapore. Analyse the menus of the top 10 cafes in Singapore (by Google reviews/ratings), and make sure we include at least 1 from the North/South/East/West/Central regions of Singapore. Keep the relevant pages of each cafe open, and summarise their pricing, menu offerings, unique selling points, making sure to reference which tab is opened for each cafe. For each cafe, also help me figure out how long it would take to get to it from Tampines MRT, and include this in your final summary.").
I was very impressed to see Opus 4.6 complete this task after working for 52 mins, satisfying all 7 rubrics that corresponded to this task. It provided a very nice markdown summary at the end that gave me all the information I asked for! -
Russ Salakhutdinov
@rsalakhu
New work on Scaling Test-Time Compute for Agentic Coding:
Paper: arxiv.org/abs/2604.16529
This work introduces a test-time scaling framework for agentic coding that converts rollouts into structured summaries capturing key hypotheses, progress, and failure modes while discarding low-signal details.
This enables two forms of inference-time scaling: (1) Recursive Tournament Voting (RTV) for parallel selection via iterative small-group comparisons, and (2) Parallel-Distill-Refine (PDR) for sequential improvement by conditioning new rollouts on distilled summaries.
Our approach consistently boosts performance on frontier benchmarks. On SWE-Bench Verified, Claude-4.5-Opus improves from 70.9% → 77.6%, and on Terminal-Bench v2.0 from 46.9% → 59.1%. These gains highlight that effective test-time scaling for long-horizon agents hinges on representation, selection, and reuse, not just sampling more trajectories.
Check out a more detailed thread by @anirudhg9119.
Anirudh Goyal
@anirudhg9119
How do coding agents get better from experience?
Past Attempts as Interface: Turn rollouts into reusable summaries that future attempts can build on.
arxiv.org/abs/2604.16529 -
Russ Salakhutdinov reposted
Kevin Patrick Murphy
@sirbayes
Congrats! BTW you might find my recent paper of interest... arxiv.org/abs/2604.18576
-
Russ Salakhutdinov
@rsalakhu
Thank you @prfsanjeevarora!
Sanjeev Arora
@prfsanjeevarora
Congrats @rsalakhu and cofounders of Sooth Labs!
Russ Salakhutdinov
@rsalakhu
Foresight will be the defining frontier on the path to AGI.
I am excited to start Sooth Labs with my amazing co-founders: Yaser Sheikh @subail, Chuck Hoover @chuckjhoover, David LaRose, and Shih-En Wei.
Deeply grateful to Aydin Senkut @asenkut and Feyza Haskaraman @FHaskaraman at @felicis for leading the round, alongside an exceptional group of partners.
www.bloomberg.com/news/articles/2026-04-22/ai-pioneers-ba… -
Russ Salakhutdinov reposted
Yutong (Kelly) He
@electronickale
Joint work with the amazing @x_duan9296 (the best master's student at CMU), @FahimTajwar10, @rsalakhu, @zicokolter, and Jeff Schneider. Come chat with us today at 3:15 PM!
Paper: arxiv.org/abs/2506.07822
Code: github.com/Ddduanxt/RACTD -
Russ Salakhutdinov reposted
Vasu Sharma
@sharmavasu55
Sooth Labs is going after something incredibly ambitious: building a world model for global foresight. This is a problem almost no one gets right today.
Most systems we rely on are either reactive, narrow, or static. They miss second order effects, fail under distribution shifts, and cany reason across domains like geopolitics, economics, climate, and technology at once. The result? We consistently get surprised by events that were, in hindsight, predictable.
That gap has real consequences. Capital gets misallocated, policies get designed on incomplete assumptions, and organizations operate with fragile mental models of the world. We’re optimizing locally while the system behaves globally.
If Sooth Labs can even partially crack this, it changes how decisions are made at every level, from governments to markets to large scale AI systems.
@rsalakhu, @subail and @chuckjhoover need no introduction. Absolute legends!
Rooting hard for this team 🚀🚀🚀💪🏻💪🏻💪🏻
Russ Salakhutdinov
@rsalakhu
Foresight will be the defining frontier on the path to AGI.
I am excited to start Sooth Labs with my amazing co-founders: Yaser Sheikh @subail, Chuck Hoover @chuckjhoover, David LaRose, and Shih-En Wei.
Deeply grateful to Aydin Senkut @asenkut and Feyza Haskaraman @FHaskaraman at @felicis for leading the round, alongside an exceptional group of partners.
www.bloomberg.com/news/articles/2026-04-22/ai-pioneers-ba… -
Russ Salakhutdinov reposted
Dhruv K
@dhruvk1015
@rsalakhu is a legendary professor! Congrats Sooth!
Feyza Haskaraman
@FHaskaraman
I’m excited to announce that @Felicis has led Sooth’s Seed round!
The world is being asked to make more, faster, higher-stakes decisions under greater uncertainty. The tools most leaders use to navigate that uncertainty — gut, spreadsheets, the occasional consultant deck — were built for a slower world. To keep up, foresight must become a piece of infrastructure.
Yaser Sheikh and @rsalakhu are building that foresight engine, powered by a time series model that deciphers and weighs real events, signals, and probabilities. This is the next leg in the race to AGI: beyond LLMs and their focus on next-token predictions, beyond coding and chatboxes.
Read more from @asenkut and I here: ordnl.link/l9UQYf7 -
Russ Salakhutdinov reposted
Rugbist
@rugbist_
solid backing from heavy hitters.
the meta and apple pedigree here is actually nuts for a seed round -
Russ Salakhutdinov reposted
Andrey Fradkin
@AndreyFradkin
Russ Salakhutdinov
@rsalakhu
Foresight will be the defining frontier on the path to AGI.
I am excited to start Sooth Labs with my amazing co-founders: Yaser Sheikh @subail, Chuck Hoover @chuckjhoover, David LaRose, and Shih-En Wei.
Deeply grateful to Aydin Senkut @asenkut and Feyza Haskaraman @FHaskaraman at @felicis for leading the round, alongside an exceptional group of partners.
www.bloomberg.com/news/articles/2026-04-22/ai-pioneers-ba… -
Russ Salakhutdinov
@rsalakhu
Thank you @pbronner and @ardent_vc for supporting us!
phil bronner
@pbronner
We backed @soothlabs at seed. Here's why.
The team.
@subail founded and scaled Meta's Pittsburgh AI office and serves on CMU's Robotics Institute faculty.
@rsalakhu Salakhutdinov is a @SCSatCMU professor, a pioneer in modern generative modeling, Apple's first director of AI research, and a disciple of Geoffrey Hinton. Chuck Hoover directed billion-dollar innovation programs at Meta.
$50M round led by @felicis at a $335M valuation. Angel investment from @ylecun and @JeffDean. Andrew Bosworth is advising.
What they're building: a continuously trained world model for long-horizon forecasting. It takes verifiable propositions (Will Brent crude exceed $90/bbl in March 2027? Will US CPI exceed 4% YoY in 2026?) and returns calibrated probabilities with a structured causal timeline of driving events.
Institutions commit capital in the face of deep uncertainty every day, and the stakes keep rising while the tools haven't moved. Strategy still gets set with gut calls, internal models, and the occasional expert opinion. Specialist platforms exist in finance, insurance, and energy, but each operates within a single domain and is rarely measured against what actually happens.
Foresight should be a system, not a skill.
LLMs aren't the answer here. They're optimized to generate plausible text, not to produce probabilities graded against real events. Sooth is built on a different architecture designed for exactly that job, with a causal chain users can interrogate, revise, and stress-test as conditions change.
Congrats to Yaser, Russ, Chuck, and the team. -
Russ Salakhutdinov
@rsalakhu
Foresight will be the defining frontier on the path to AGI.
Deeply grateful to @felicis for leading the round. Excited to partner and build together!
Feyza Haskaraman
@FHaskaraman
I’m excited to announce that @Felicis has led Sooth’s Seed round!
The world is being asked to make more, faster, higher-stakes decisions under greater uncertainty. The tools most leaders use to navigate that uncertainty — gut, spreadsheets, the occasional consultant deck — were built for a slower world. To keep up, foresight must become a piece of infrastructure.
Yaser Sheikh and @rsalakhu are building that foresight engine, powered by a time series model that deciphers and weighs real events, signals, and probabilities. This is the next leg in the race to AGI: beyond LLMs and their focus on next-token predictions, beyond coding and chatboxes.
Read more from @asenkut and I here: ordnl.link/l9UQYf7 -
Russ Salakhutdinov reposted
Feyza Haskaraman
@FHaskaraman
I’m excited to announce that @Felicis has led Sooth’s Seed round!
The world is being asked to make more, faster, higher-stakes decisions under greater uncertainty. The tools most leaders use to navigate that uncertainty — gut, spreadsheets, the occasional consultant deck — were built for a slower world. To keep up, foresight must become a piece of infrastructure.
Yaser Sheikh and @rsalakhu are building that foresight engine, powered by a time series model that deciphers and weighs real events, signals, and probabilities. This is the next leg in the race to AGI: beyond LLMs and their focus on next-token predictions, beyond coding and chatboxes.
Read more from @asenkut and I here: ordnl.link/l9UQYf7 -
Russ Salakhutdinov
@rsalakhu
Foresight will be the defining frontier on the path to AGI.
I am excited to start Sooth Labs with my amazing co-founders: Yaser Sheikh @subail, Chuck Hoover @chuckjhoover, David LaRose, and Shih-En Wei.
Deeply grateful to Aydin Senkut @asenkut and Feyza Haskaraman @FHaskaraman at @felicis for leading the round, alongside an exceptional group of partners.
www.bloomberg.com/news/articles/2026-04-22/ai-pioneers-ba… -
Russ Salakhutdinov
@rsalakhu
Congratulations @ysu_nlp and the team on the launch of @NeoCognition. Very excited to be supporting this effort and see what you build!
Yu Su
@ysu_nlp
Introducing @NeoCognition, the agent lab for specialized intelligence.
Everyone needs experts, but human expertise does not scale.
Backed by $40M seed funding, we build self-learning agents that specialize across domains to make expertise abundant.This media couldn't be displayed. See it on the original post. -
Russ Salakhutdinov
@rsalakhu
Giving a keynote at the Multimodal workshop at Yale University. Such a beautiful campus!
-
Russ Salakhutdinov reposted
Charles 🎉 Frye
@charles_irl
RLAD, by @Anikait_Singh_ and a star-studded cast from @rsalakhu, @aviral_kumar2 and @chelseabfinn's labs, uses model summaries of rollouts to learn reusable abstractions.
Reasoning models have already changed coding, and they are just in their infancy!
iclr.cc/virtual/2026/poster/10008193 -
Russ Salakhutdinov
@rsalakhu
Awesome work @pliang279.
Paul Liang
@pliang279
Most of today's AI can see the world, but it doesn’t **feel** it.
Capturing the sense of touch is crucial for dexterous robotic manipulation, user modeling, and understanding physical interactions.
Introducing OpenTouch: bringing full-hand tactile sensing into real-world AI🖐️
OpenTouch is collected in-the-wild using tactile sensing gloves, hand pose tracking gloves, and egocentric glasses. It includes:
• 5 hours of real-world data,
• 3 hours densely annotated contact-rich interactions,
• 2,900 curated interaction clips,
• across 800 objects, 14 environments, and 29 grasp types.
all open at: opentouch-tactile.github.io -
Russ Salakhutdinov reposted
Krishnaswamy Lab
@KrishnaswamyLab
Symposium on Machine Learning across modalities now started at Davies Auditorium! Join us if you are in the area ! @rsalakhu is the first keynote. smlm.yale.edu
-
Russ Salakhutdinov reposted
Anirudh Goyal
@anirudhg9119
Reasoning doesn’t have to mean longer chains of thought:
PDR = draft in parallel → distill into a compact workspace → refine, and shift the Pareto frontier.
arxiv.org/abs/2510.01123
Alexandr Wang
@alexandr_wang
3/ we’re also releasing contemplating mode, which orchestrates multiple agents that reason in parallel designed to handle complex scientific & reasoning queries. in our testing we found it competitive w/ other extreme reasoning models such as Gemini Deep Think & GPT Pro.
-
Russ Salakhutdinov reposted
Manos Koukoumidis
@Koukoumidis
🚨 The era of general-purpose AI is over.
Today we're launching Oumi. 🚀
The platform that lets any team build custom AI models — in hours, not months.
Just describe what you need. Oumi builds it. #vibeml
Higher quality. Lower cost. Fully yours.This media couldn't be displayed. See it on the original post. -
Russ Salakhutdinov
@rsalakhu
LumberChunker: Long-Form Narrative Document Segmentation – Machine Learning Blog | ML@CMU | Carnegie Mellon University blog.ml.cmu.edu/2026/03/17/lumberchunker-long-form-narrat…
-
Russ Salakhutdinov
@rsalakhu
Amazing work @shuyanzh36.
Shuyan Zhou
@shuyanzh36
In 2023, WebArena took 7 grad students more than 6 months to build just 5 environments with 812 variable browser-use tasks.
Now, it takes under 10 hours and less than $100 per environment, with easy support for parallel generation.
Excited to introduce WebArena-Infinity: a scalable approach for automatically generating high-authenticity, high-complexity browser environments with verifiable tasks suitable for RL training and benchmarking.
Even strong open-source models that already achieve 60%+ success rates on WebArena and OSWorld complete fewer than 50% of tasks here.
Project page: webarena.dev/webarena-infinity
Repo: github.com/web-arena-x/webarena-infinity 🧵 (1/n) -
Russ Salakhutdinov
@rsalakhu
Check out this talk by @FahimTajwar10 on Maximum Likelihood RL (MaxRL).
alphaXiv
@askalphaxiv
RL Isn’t Actually Optimizing What We Think, And That’s a Problem
Come join us for this AI4Science talk: Maximum Likelihood Reinforcement Learning (MaxRL).
In this session, the author of MaxRL @FahimTajwar10 will cover their paper that takes a step back and asks a fundamental question: when we use RL for tasks like reasoning, coding, or navigation, are we even optimizing the right objective?
Whether you’re working on RL, LLM reasoning, or just curious about how training objectives shape model behavior, this is one to check out.
🗓 Wednesday March 18th 2026 · 10AM PT
🎙 Featuring Fahim Tajwar
💬 Casual Talk + Open Discussion -
Russ Salakhutdinov reposted
alphaXiv
@askalphaxiv
RL Isn’t Actually Optimizing What We Think, And That’s a Problem
Come join us for this AI4Science talk: Maximum Likelihood Reinforcement Learning (MaxRL).
In this session, the author of MaxRL @FahimTajwar10 will cover their paper that takes a step back and asks a fundamental question: when we use RL for tasks like reasoning, coding, or navigation, are we even optimizing the right objective?
Whether you’re working on RL, LLM reasoning, or just curious about how training objectives shape model behavior, this is one to check out.
🗓 Wednesday March 18th 2026 · 10AM PT
🎙 Featuring Fahim Tajwar
💬 Casual Talk + Open Discussion -
Russ Salakhutdinov reposted
Islam
@islamakramov8
Been in stealth mode for the last 3 months.
Building something exciting for Physical AI.
More soon. -
Russ Salakhutdinov
@rsalakhu
Check out our new work on Tuning-free Visual Effect Transfer across Videos:
Paper: arxiv.org/abs/2601.07833
Project: snap-research.github.io/RefVFX/
This work introduces RefVFX, a method for reference-based visual effect transfer from one video to another. Given an input video and a separate reference video that contains the desired effect, RefVFX transfers that effect onto the input.
Most prior video editing approaches require effects to be specified using text prompts alone. However, many effects are easier to demonstrate than to describe, such as a particular glitch style, transition, or particle burst. RefVFX takes the reference video as a first-class input.
The core model is a finetuned Wan 2.1 FLF2V model conditioned on both the input video and a reference effect video. At inference time, any input video can be paired with any effect video, making the system fully compositional, no retraining needed.
Check out a detailed thread by @maxwell54650346This media couldn't be displayed. See it on the original post.
maxwell jones
@maxwell54650346
Video Editing is great - but what if you want to apply an effect to your input video described by another video??
Introducing RefVFX, the first method that takes in both an input video and a reference effect video for generative video editing! -
Russ Salakhutdinov
@rsalakhu
Thanks @lvdmaaten! It was amazing working with you as well.
Laurens van der Maaten
@lvdmaaten
Even though it was brief, it was super fun getting to work with you again Russ! Good luck in what is next for you.
-
Russ Salakhutdinov
@rsalakhu
My time at Meta Superintelligence Lab (MSL) comes to a close. Together with @kohjingyu and @dan_fried, I joined Meta nearly two years ago to help advance and scale computer-use agents, a long-standing research focus of ours at CMU @SCSatCMU.
It has been a remarkable journey working with and leading an exceptionally talented team, spanning agentic and reasoning model training across pre-training and post-training, building evals, and data pipelines. I’m deeply grateful for the opportunity to collaborate with so many amazing colleagues across GenAI and the incredible researchers at FAIR @AIatMeta. I also want to thank Meta’s leadership, especially @Ahmad_Al_Dahle and @rob_fergus, for giving our team the freedom to explore and pursue cutting-edge research.
Big tech excels at large-scale engineering and the scaling of foundation models, but further progress toward superintelligence will require new breakthroughs in architectures, optimization, and the efficient use of data, including synthetic data. And I believe academia will play a pivotal role in driving these advances, particularly through open-source research. -
Russ Salakhutdinov reposted
Jing Yu Koh
@kohjingyu
Belated life update: I’ve returned to CMU to finish up my PhD!
Funnily enough, this marks the second time I’ve left industry for the PhD. And this time I got even more questions about it than the first time (which already invited a lot of questions). Maybe I’ll write a blog post at some point about this.
For now I’m excited to have an empty calendar and more IC time :) -
Russ Salakhutdinov reposted
Vlado Boza
@bozavlado
This looks like one of the best things I read in the past year.
Russ Salakhutdinov
@rsalakhu
New work on Maximum Likelihood Reinforcement Learning:
arxiv.org/abs/2602.02710
This work develops Maximum Likelihood Reinforcement Learning (MaxRL), a family of objectives that smoothly interpolates between REINFORCE and maximum likelihood. Our final algorithm requires only a minimal change, a single line of code (dividing by the mean reward in the advantage computation). Empirically, MaxRL shows substantial gains in sample efficiency with perfect verifiers, Pareto-dominates GRPO on reasoning tasks, and exhibits good scaling with both compute and data.
What makes MaxRL particularly compelling is its distinct learning dynamics. It produces larger gradient norms on harder problems, preserving learning signal where other methods tend to wash it out. As a result, a higher fraction of prompts are successfully solved during training. In data-scarce regimes, MaxRL resists overfitting while converging to stronger final performance.
Paper, code, and demos: zanette-labs.github.io/MaxRL/
See also an excellent thread by @FahimTajwar10.
Fahim Tajwar
@FahimTajwar10
Are we done with new RL algorithms? Turns out we might have been optimizing the wrong objective.
Introducing MaxRL, a framework to bring maximum likelihood optimization to RL settings.
Paper + code + project website: zanette-labs.github.io/MaxRL/
🧵 1/n -
Russ Salakhutdinov
@rsalakhu
New work on Maximum Likelihood Reinforcement Learning:
arxiv.org/abs/2602.02710
This work develops Maximum Likelihood Reinforcement Learning (MaxRL), a family of objectives that smoothly interpolates between REINFORCE and maximum likelihood. Our final algorithm requires only a minimal change, a single line of code (dividing by the mean reward in the advantage computation). Empirically, MaxRL shows substantial gains in sample efficiency with perfect verifiers, Pareto-dominates GRPO on reasoning tasks, and exhibits good scaling with both compute and data.
What makes MaxRL particularly compelling is its distinct learning dynamics. It produces larger gradient norms on harder problems, preserving learning signal where other methods tend to wash it out. As a result, a higher fraction of prompts are successfully solved during training. In data-scarce regimes, MaxRL resists overfitting while converging to stronger final performance.
Paper, code, and demos: zanette-labs.github.io/MaxRL/
See also an excellent thread by @FahimTajwar10.
Fahim Tajwar
@FahimTajwar10
Are we done with new RL algorithms? Turns out we might have been optimizing the wrong objective.
Introducing MaxRL, a framework to bring maximum likelihood optimization to RL settings.
Paper + code + project website: zanette-labs.github.io/MaxRL/
🧵 1/n -
Russ Salakhutdinov
@rsalakhu
How to Explore to Scale RL Training of LLMs on Hard Problems? – Machine Learning Blog | ML@CMU | Carnegie Mellon University blog.ml.cmu.edu/2025/11/26/how-to-explore-to-scale-rl-tra…
-
Russ Salakhutdinov
@rsalakhu
New work on Learning to Reason on Hard Problems via Privileged On-Policy Exploration:
arxiv.org/abs/2601.18779
Reinforcement learning (RL) has improved LLM reasoning, but state-of-the-art methods still fail on many hard tasks. On-policy RL rarely explores correct rollouts on difficult problems, yielding zero reward and no learning signal.
We Introduce Privileged On-Policy Exploration (POPE), which uses human- or other oracle solutions as privileged guidance, not training targets, to steer exploration on hard problems. By prefixing oracle solution fragments, POPE enables non-zero rewards during guided rollouts for hard tasks, and the resulting behaviors transfer back to unguided problems. Empirically, POPE expands the set of solvable tasks and delivers substantial gains on challenging reasoning benchmarks.
Check out an excellent thread by @QuYuxiao and this blogpost:
blog.ml.cmu.edu/2025/11/26/how-to-explore-to-scale-rl-tra…
with @QuYuxiao, @setlur_amrith, @gingsmith, @aviral_kumar2
Yuxiao Qu
@QuYuxiao
🚨 NEW PAPER: “POPE: Learning to Reason on Hard Problems via Privileged On-Policy Exploration”!
❓ How do we train LLMs with RL on hard problems when the model never gets a single correct rollout?
💡 Short answer: standard RL is stuck. We show why, and introduce POPE to break this deadlock.
🧵[1/N] -
Russ Salakhutdinov reposted
Somesh Misra / ERP.ai
@MathproBro
Paper: arxiv.org/abs/2410.15153
Authors:
@rsalakhu
@chhaviyadav_
@ruihan_w
@kamalikac
Worth reading if you care about what “forgetting” should actually mean in LLMs. -
Russ Salakhutdinov reposted
Jenia Jitsev 🏳️🌈 🇺🇦 🇮🇱 🇮🇷
@JJitsev
OpenReview is exemplary case for strong work done by small dedicated community that is critical for so many, still staying underfunded and lacking proper support. If we dont change ways open science is supported, it will vanish.
Russ Salakhutdinov
@rsalakhu
OpenReview is a lifeline for progress in the AI research community, and it urgently needs our increased support.
openreview.net/donate
In 2025 alone, OpenReview supported over 1,300 conferences and workshops, served 3.3 million active monthly users, handled over 278,000 paper submissions. I personally had not fully appreciated the magnitude of its role until seeing these numbers.
Yet, OpenReview remains underfunded, operating under severe financial constraints with a small team. This is just crazy!
Please consider donating.
Many organizations that benefit enormously from advances in AI have not yet chosen to support OpenReview. Now is the time to do so.
It is also time for all conferences to step up and substantially increase their support. -
Russ Salakhutdinov reposted
alex peysakhovich
@alex_peys
i don’t even submit papers anymore but i donated to openreview, you should too
Russ Salakhutdinov
@rsalakhu
OpenReview is a lifeline for progress in the AI research community, and it urgently needs our increased support.
openreview.net/donate
In 2025 alone, OpenReview supported over 1,300 conferences and workshops, served 3.3 million active monthly users, handled over 278,000 paper submissions. I personally had not fully appreciated the magnitude of its role until seeing these numbers.
Yet, OpenReview remains underfunded, operating under severe financial constraints with a small team. This is just crazy!
Please consider donating.
Many organizations that benefit enormously from advances in AI have not yet chosen to support OpenReview. Now is the time to do so.
It is also time for all conferences to step up and substantially increase their support. -
Russ Salakhutdinov reposted
Zhili Feng
@zhilifeng
🥳🥳🥳I defended my PhD thesis today! Special thanks to my wonderful advisor @zicokolter and committee members @rsalakhu @gneubig @LesterMackey!
🎉🎉🎉I am joining @OpenAI as a researcher, super excited to keep working on frontier models and meet everyone in SF! -
Russ Salakhutdinov reposted
Sanjeev Arora
@prfsanjeevarora
Definitely a cause worth supporting. But openreview really should charge for conference submissions (like $20 per paper).
Russ Salakhutdinov
@rsalakhu
OpenReview is a lifeline for progress in the AI research community, and it urgently needs our increased support.
openreview.net/donate
In 2025 alone, OpenReview supported over 1,300 conferences and workshops, served 3.3 million active monthly users, handled over 278,000 paper submissions. I personally had not fully appreciated the magnitude of its role until seeing these numbers.
Yet, OpenReview remains underfunded, operating under severe financial constraints with a small team. This is just crazy!
Please consider donating.
Many organizations that benefit enormously from advances in AI have not yet chosen to support OpenReview. Now is the time to do so.
It is also time for all conferences to step up and substantially increase their support. -
Russ Salakhutdinov reposted
Alex Dimakis
@AlexGDimakis
I just donated to OpenReview. And I would encourage all those who care about open science to please put their money where their mouth is, and help.
Russ Salakhutdinov
@rsalakhu
OpenReview is a lifeline for progress in the AI research community, and it urgently needs our increased support.
openreview.net/donate
In 2025 alone, OpenReview supported over 1,300 conferences and workshops, served 3.3 million active monthly users, handled over 278,000 paper submissions. I personally had not fully appreciated the magnitude of its role until seeing these numbers.
Yet, OpenReview remains underfunded, operating under severe financial constraints with a small team. This is just crazy!
Please consider donating.
Many organizations that benefit enormously from advances in AI have not yet chosen to support OpenReview. Now is the time to do so.
It is also time for all conferences to step up and substantially increase their support. -
Russ Salakhutdinov
@rsalakhu
OpenReview is a lifeline for progress in the AI research community, and it urgently needs our increased support.
openreview.net/donate
In 2025 alone, OpenReview supported over 1,300 conferences and workshops, served 3.3 million active monthly users, handled over 278,000 paper submissions. I personally had not fully appreciated the magnitude of its role until seeing these numbers.
Yet, OpenReview remains underfunded, operating under severe financial constraints with a small team. This is just crazy!
Please consider donating.
Many organizations that benefit enormously from advances in AI have not yet chosen to support OpenReview. Now is the time to do so.
It is also time for all conferences to step up and substantially increase their support. -
Russ Salakhutdinov reposted
Sanjeev Arora
@prfsanjeevarora
I'm glad this paper of ours is getting attention. It shows that there are more efficient and effective ways for models to use their thinking tokens than generating a long uninterrupted thinking trace. Our PDR (parallel/distill/refine) orchestration gives much better final accuracy, while avoiding context bloat. (So it might be much cheaper to serve than today's thinking models.) I'm guessing that so-called "deep research models" rely on such orchestrations.
DAIR.AI
@dair_ai
NEW Research from Meta Superintelligence Labs and collaborators.
The default approach to improving LLM reasoning today remains extending chain-of-thought sequences.
Longer reasoning traces aren't always better. Longer traces conflate reasoning depth with sequence length and inherit long-context failure modes.
This new research introduces Parallel-Distill-Refine (PDR), a framework that treats LLMs as improvement operators rather than single-pass reasoners.
Instead of one long reasoning chain, PDR operates in phases:
- Generate diverse drafts in parallel.
- Distill them into a bounded textual workspace.
- Refine conditioned on this workspace.
- Repeat.
Context length becomes controllable via degree of parallelism, no longer conflated with total tokens generated. The model accumulates wisdom across rounds through compact summaries rather than replaying full histories.
On AIME 2024, PDR achieves 93.3% accuracy compared to 79.4% for standard long chain-of-thought at matched latency budgets.
For o3-mini at 49k effective tokens, accuracy improves from 76.9% (Long CoT) to 86.7% (PDR), a 9.8 percentage point gain.
PDR also achieves the same accuracy as sequential refinement with 2.57x smaller sequential budget by converting parallel compute into accuracy without lengthening per-call context.
The researchers also trained an 8B model with operator-consistent RL to make training match the PDR inference interface. Mixing standard and operator RL yields an additional 5% improvement on both AIME benchmarks.
Bounded memory iteration can substitute for long reasoning traces while holding latency fixed. Strategic parallelism and distillation is shown to beat brute-force sequence extension.
Paper: arxiv.org/abs/2510.01123
Learn to build effective AI Agents in our academy: dair-ai.thinkific.com -
Russ Salakhutdinov reposted
DAIR.AI
@dair_ai
NEW Research from Meta Superintelligence Labs and collaborators.
The default approach to improving LLM reasoning today remains extending chain-of-thought sequences.
Longer reasoning traces aren't always better. Longer traces conflate reasoning depth with sequence length and inherit long-context failure modes.
This new research introduces Parallel-Distill-Refine (PDR), a framework that treats LLMs as improvement operators rather than single-pass reasoners.
Instead of one long reasoning chain, PDR operates in phases:
- Generate diverse drafts in parallel.
- Distill them into a bounded textual workspace.
- Refine conditioned on this workspace.
- Repeat.
Context length becomes controllable via degree of parallelism, no longer conflated with total tokens generated. The model accumulates wisdom across rounds through compact summaries rather than replaying full histories.
On AIME 2024, PDR achieves 93.3% accuracy compared to 79.4% for standard long chain-of-thought at matched latency budgets.
For o3-mini at 49k effective tokens, accuracy improves from 76.9% (Long CoT) to 86.7% (PDR), a 9.8 percentage point gain.
PDR also achieves the same accuracy as sequential refinement with 2.57x smaller sequential budget by converting parallel compute into accuracy without lengthening per-call context.
The researchers also trained an 8B model with operator-consistent RL to make training match the PDR inference interface. Mixing standard and operator RL yields an additional 5% improvement on both AIME benchmarks.
Bounded memory iteration can substitute for long reasoning traces while holding latency fixed. Strategic parallelism and distillation is shown to beat brute-force sequence extension.
Paper: arxiv.org/abs/2510.01123
Learn to build effective AI Agents in our academy: dair-ai.thinkific.com -
Russ Salakhutdinov reposted
Jing Yu Koh
@kohjingyu
I resigned from TBD Lab / MSL last Friday. It was a really fulfilling experience leading the computer use agents team over the past 1.45 years, and I learnt and grew a ton personally and professionally. It was a really fun opportunity to build CUA infra, data pipelines, evals, and models from scratch, and work together with a very talented team to discover how to get to frontier level CUA performance.
I was also really impressed by the calibre of TBD Lab and how quickly everything came together. It was a difficult decision to leave: I will miss my colleagues and friends at Meta, but I'm sure our paths will cross again soon! -
Russ Salakhutdinov
@rsalakhu
Check out new work:
Joint Distillation for Fast Likelihood Evaluation and Sampling in Flow-based Models
Paper: arxiv.org/abs/2512.02636
Log-likelihood evaluation enables key capabilities in generative modeling: model comparison, certain fine-tuning objectives, and many downstream tasks. Yet state-of-the-art diffusion and flow models still require hundreds to thousands of NFEs to compute them. Recent distillation methods speed up sampling but break likelihood tractability.
This work develops Fast Flow Joint Distillation (F2D2), which cuts NFEs for both sampling and likelihood evaluation by two orders of magnitude. The key insight is that in continuous normalizing flows, the coupled ODEs for sampling and likelihood are computed from a shared underlying velocity field, allowing one to jointly distill sampling trajectories and cumulative divergence using a single model.
F2D2 preserves sample quality while enabling accurate few-step likelihoods, resolving a long-standing computational bottleneck. With a lightweight self-guidance technique, a 2-step MeanFlow model can even outperform its 1024-step teacher using just one extra backward NFE.
Check out an excellent detailed thread by @electronickale!
Yutong (Kelly) He
@electronickale
Diffusion/Flow-based models can sample in 1-2 steps now 👍 But likelihood? Still requires 100-1000 NFEs (even for these fast models) 😭
We fix this! Introducing F2D2: simultaneous fast sampling AND fast likelihood via joint flow map distillation.
arxiv.org/abs/2512.02636
1/🧵 -
Russ Salakhutdinov reposted
Yutong (Kelly) He
@electronickale
I'm teaching a diffusion & flow matching class at CMU in Spring 2026 where students can use ChatGPT, Cursor, or any AI tool they want. No exams. Just build with open internet.
139 students signed up for 20 spots.
Here's what's happening: 🧵
kellyyutonghe.github.io/10799S26/ -
Russ Salakhutdinov reposted
Jason Lee
@jasondeanlee
20+ doubt we will see it in my lifetime. I'm not even sure we hit the newtonian revolution yet of AI. Maybe we are at Galileo. Agi/ASI is probably like post - QM or maybe even post QED.
Russ Salakhutdinov
@rsalakhu
Predicting AGI/ASI timelines has become trendy, so I’ll offer mine:
AGI/ASI is 5–10 years away. It always has been. It always will be. -
Russ Salakhutdinov
@rsalakhu
Predicting AGI/ASI timelines has become trendy, so I’ll offer mine:
AGI/ASI is 5–10 years away. It always has been. It always will be. -
Russ Salakhutdinov
@rsalakhu
Check out the new @mldcmu blogpost:
How to Explore to Scale RL Training of LLMs on Hard Problems?
blog.ml.cmu.edu/2025/11/26/how-to-explore-to-scale-rl-tra…
Current on-policy RL methods fail to learn from hard problems as they rarely generate a single correct rollout, producing no reward signal and no learning. Including easy problems can also be harmful, as models tend to overfit to them and fail to improve on harder tasks. Distilling human-written solutions is not only costly, but also provides difficult targets for fine-tuning.
This blogpost discusses various approaches and introduces a framework that uses existing human or model solutions as privileged guidance to unlock learning on hard problems.
The key idea is simple: Prepend a minimal solution prefix to difficult prompts, enabling on-policy RL to obtain reward and learn behaviors that generalize back to the original, unconditioned tasks. This expands the set of solvable problems and results in significant gains on challenging reasoning benchmarks.
With @QuYuxiao, @setlur_amrith, @gingsmith, @aviral_kumar2.
Paper/Code is coming soon. -
Russ Salakhutdinov reposted
Andrea Zanette
@Zanette_ai
I’m recruiting several PhD students at Carnegie Mellon University!
If you’re interested in LLM reasoning, agents, or diffusion language models, consider applying to the CMU ECE PhD program.
Applications are due Dec 15.
www.ece.cmu.edu/admissions/graduate-application-deadlines… -
Russ Salakhutdinov reposted
Russ Salakhutdinov
@rsalakhu
This project was an incredible collaboration between conservation ecologists at the MPG Ranch and our lab at CMU, focused on studying and eradicating the invasive plant "Leafy Spurge" using AI.
www.cs.cmu.edu/news/2025/gen-ai-invasive-plants
As part of this work, we are also releasing a unique dataset: "Ground-truthed, High-resolution Drone Imagery of Leafy Spurge"
www.nature.com/articles/s41597-025-05094-6
with Kyle Doherty, @maxgurinas , @brandontrabucco et al. -
Russ Salakhutdinov
@rsalakhu
Learning from Failure to Tackle Extremely Hard Problems – Machine Learning Blog | ML@CMU | Carnegie Mellon University blog.ml.cmu.edu/2025/10/27/learning-from-failure-to-tackl…
-
Russ Salakhutdinov
@rsalakhu
I had a wonderful time visiting @UWLaCrosse University of Wisconsin La Crosse and giving two talks on Embodied AI and Multimodal AI Agents as part of their Distinguished Lecture Series.
Beautiful campus, and amazing students and faculty. My first visit to a UW campus, and definitely not the last. -
Russ Salakhutdinov
@rsalakhu
New work on Rethinking Thinking Tokens: LLMs as Improvement Operators:
arxiv.org/abs/2510.01123
Reasoning training encourages LLMs to produce long chains of thought (CoT), improving accuracy via self-checking but increasing context length, compute cost, and latency. This work studies whether frontier models can achieve better trade-offs, higher accuracy with lower cost.
This work develops a simple yet effective Parallel-Distill-Refine (PDR) procedure: Generate diverse drafts in parallel, Distill them into a compact textual workspace, and Refine conditioned on this workspace. This decouples context length from total token count, allowing control over compute via parallelism. PDR yields higher accuracy than long CoT at lower latency.
Training an 8B model with RL to align with PDR further shifts the Pareto frontier. On math benchmarks, PDR achieves +11% (AIME 2024) and +9% (AIME 2025) over single-pass baselines.
With Lovish Madaan, Aniket Didolkar, Suchin Gururangan, John Quan, Ruan Silva, Manzil Zaheer, Sanjeev Arora, and Anirudh Goyal. -
Russ Salakhutdinov
@rsalakhu
Check out new work on Training LLMs to Discover Abstractions for Solving Reasoning Problems:
Paper+Code: cohenqu.github.io/rlad.github.io/
Reasoning goes beyond pattern matching or memorization, it requires discovering and applying algorithmic procedures that reuse primitives, intermediate results, and building upon them. While RL post-training on long COTs aims to elicit such behavior, current models often learn depth-first, brute-force reasoning traces.
To improve this, we introduce reasoning abstractions, concise natural language descriptions of procedural or factual knowledge that guide the model toward more effective reasoning. Our framework jointly trains an abstraction generator and an abstraction-conditioned solution generator, enabling structured exploration. This approach improves generalization to harder problems, showing that allocating more test-time compute to generating abstractions yields greater gains than producing additional solutions, highlighting the central role of abstractions in guiding global reasoning.
Yuxiao Qu
@QuYuxiao
🚨 NEW PAPER: "RLAD: Training LLMs to Discover Abstractions for Reasoning"!
We introduce reasoning abstractions: concise insights that help LLMs solve hard reasoning problems by guiding structured exploration.
📄 arxiv.org/abs/2510.02263
🌐 cohenqu.github.io/rlad.github.io/
🧵[1/N] -
Russ Salakhutdinov
@rsalakhu
IROS Best Student Paper Award for Neural MP: A Generalist Neural Motion Planner!
Congratulations to @mihdalal & @Jiahui_Yang6709 for leading this work at @SCSatCMU with @mendonca_rl, Youssef Khaky, @deepakpathak.
Check out our paper/code: mihdalal.github.io/neuralmotionplanner/
Conventional motion planning solves each problem from scratch, demanding high computation and time. This work develops a data-driven approach that learns from large-scale simulated environments and expert trajectories to train a reactive generalist policy end-to-end, refined through lightweight optimization for safe deployment. Evaluated on 64 real-world tasks, the new method improves success rates by 23%, 17%, and 79% over state-of-the-art sampling, optimization, and learning-based planners. -
Russ Salakhutdinov reposted
Deepak Pathak
@deepakpathak
Incredible news. Neural MP has won the Best Student Paper award at IROS 2025!!
Congratulations to @mihdalal & @Jiahui_Yang6709 for leading the project along with @mendonca_rl, youssef, @rsalakhu. Neural MP is a major step in making motion planning end-to-end, fast & reactive.
Murtaza Dalal @ CoRL 2026
@mihdalal
Can a single neural network policy generalize over poses, objects, obstacles, backgrounds, scene arrangements, in-hand objects, and start/goal states?
Introducing Neural MP: A generalist policy for solving motion planning tasks in the real world 🤖
1/N -
Russ Salakhutdinov reposted
Manos Koukoumidis
@Koukoumidis
When we started Oumi, we set out to build a better future for AI – grounded in the benefits of open source. Eight months ago, we open-sourced the most comprehensive repo for training foundation models end to end.
Today, we’re building on that by enabling every developer and enterprise to train custom models in hours, not months ⏱️ – a foundational shift in how teams build, ship, and own AI.
I shared why custom models are the future – and why now is the moment – in our new blog post (see first comment for link). 📖
Ready to jump to the new era? Sign up for early access and join our limited design partner program. See more at our refreshed website.
Proud of the world-class Oumi team: builders and operators who are shaping what comes next. 🚀 -
Russ Salakhutdinov reposted
Rohan Paul
@rohanpaul_ai
The paper teaches LLMs to write small hints that guide their own reasoning.
The big deal is that the model learns reusable strategies instead of longer chains.
Reports 44% higher accuracy on AIME 2025 than very long chain of thought reinforcement learning.
Long reinforcement learning chains often make models stick to one approach and miss better ideas.
Hints are short notes that outline steps, checks, and pitfalls without answers.
Training uses 2 parts, a hint generator and a hint conditioned solver.
The solver gets reward when a hint helps it answer correctly, so it learns to read hints.
The generator gets reward when its hint makes the solver succeed, so it learns useful guidance.
They start the generator with synthetic summaries from stronger models, then fine tune both with reinforcement learning.
Across math tasks, performance improves even when no hints are shown at test time.
With the same compute, many diverse hints beat many solutions per hint.
----
Paper – arxiv. org/abs/2510.02263
Paper Title: "RLAD: Training LLMs to Discover Abstractions for Solving Reasoning Problems" -
Russ Salakhutdinov reposted
Rohan Paul
@rohanpaul_ai
Another classic @AIatMeta paper.
Shows language models work better when they think in short rounds using tiny summaries, not long step by step traces.
It breaks the tradeoff between thinking longer and paying for longer prompts.
Gives higher accuracy at the same or lower latency by keeping only the useful bits in a tiny workspace.
It scales by adding parallel drafts rather than dragging out a single long chain, which is cheaper and faster in practice.
Results show +11% on AIME 2024 and +9% on AIME 2025 with about 2.57x fewer sequential tokens for similar accuracy.
Long chains grow with context length, so cost and wait time rise, and the model can forget, get stuck on bad ideas, and repeat itself.
They treat the model as an improvement operator that reads a small workspace of text, writes a better draft, then compresses it back into the workspace.
A loop called Parallel Distill Refine makes several drafts in parallel, collects what they agree and disagree on into a bounded workspace, then refines again.
This keeps each prompt short while evidence builds across rounds.
Latency depends on the sequential budget, which means the tokens on the chosen path, and parallel sampling only increases total compute, not prompt length.
Feeding only wrong drafts into the workspace reduces accuracy, so they stress checks and diversity of drafts.
They also train an 8B model with operator consistent reinforcement learning to perform one round of this interface and it raises results at the same latency.
The result is that compact summaries enable many steps without long contexts, so accuracy rises without extra latency.
----
Paper – arxiv. org/abs/2510.01123
Paper Title: "Rethinking Thinking Tokens: LLMs as Improvement Operators" -
Russ Salakhutdinov reposted
Siva Reddy
@sivareddyg
Talk 1: Russ Salakhutdinov (@rsalakhu) on Towards Internet-Scale Training of Web-Agents. #llmagents
Exploit synthetic data to scale web agents -
Russ Salakhutdinov reposted
Anikait Singh
@Anikait_Singh_
🚨🚨New Paper: Training LLMs to Discover Abstractions for Solving Reasoning Problems
Introducing RLAD, a two-player RL framework for LLMs to discover 'reasoning abstractions'—natural language hints that encode procedural knowledge for structured exploration in reasoning.🧵⬇️ -
Russ Salakhutdinov reposted
Yuxiao Qu
@QuYuxiao
🚨 NEW PAPER: "RLAD: Training LLMs to Discover Abstractions for Reasoning"!
We introduce reasoning abstractions: concise insights that help LLMs solve hard reasoning problems by guiding structured exploration.
📄 arxiv.org/abs/2510.02263
🌐 cohenqu.github.io/rlad.github.io/
🧵[1/N] -
Russ Salakhutdinov reposted
Jing Yu Koh
@kohjingyu
Great post and summarizes some of my own thoughts very well. CUA has been more finicky and messier than I anticipated, and there are a lot of weird details and edge cases you have to get right. But I'm personally super optimistic that we'll continue to make fast progress!
Yu Su
@ysu_nlp
Computer Use: Modern Moravec's Paradox
A new blog post arguing why computer-use agents may be the biggest opportunity and challenge for AGI.
yusu.substack.com/p/computer-use-modern-moravecs-paradox?…
Table of Contents
> Moravec’s Paradox
> Moravec's Paradox in 2025
> Computer use may be the biggest opportunity for AGI
> Chatbots → agents
> Internet-scale learning of human cognition
> Bits > atoms
> Enormous economic value
> Why is computer use hard for AI?
> Computer use ≠ clicks + typing
> Idiosyncratic environments
> Contextual understanding
> Tacit knowledge
> Is RL the panacea?
> Looking forward
If you are also excited about CUAs and want to do some serious work, let's chat! -
Russ Salakhutdinov reposted
So Yeon (Tiffany) Min
@SoYeonTiffMin
It has been a while, and I am excited to share that I have defended my PhD and have joined @AnthropicAI in July!
I am deeply thankful to my great advisors @ybisk @rsalakhu, my awesome thesis committee @KaterinaFragiad @dan_fried and Joyce Chai. Also, deeply thankful to @dchaplot who has been my first mentor, and wonderful collaborators/mentors Hubert Tsai @yw_yuewu @shrimai_ @_Hao_Zhu @DanielXieee @jimin__sun @RoozbehMottaghi @xavierpuigf 🙏🙏 -
Russ Salakhutdinov
@rsalakhu
This project was an incredible collaboration between conservation ecologists at the MPG Ranch and our lab at CMU, focused on studying and eradicating the invasive plant "Leafy Spurge" using AI.
www.cs.cmu.edu/news/2025/gen-ai-invasive-plants
As part of this work, we are also releasing a unique dataset: "Ground-truthed, High-resolution Drone Imagery of Leafy Spurge"
www.nature.com/articles/s41597-025-05094-6
with Kyle Doherty, @maxgurinas , @brandontrabucco et al. -
Russ Salakhutdinov reposted
Boyuan Zheng
@boyuan__zheng
Ruslan Salakhutdinov (@rsalakhu) is presenting on Scaling up Multimodal AI Agents!
Join us @workshopcua!
ComputerUseAgents Workshop
@workshopcua
🚀Announcing the Workshop on Computer Use Agents at #icml2025 in July, Vancouver!
Join us, to advance research on AI agents performing real-world computer tasks.
🤖Call for Papers & Demos: Deadline May 18, 2025
🎙️Exciting speaker lineup announced!
✍️Interested in reviewing? Register now!
✈️Travel grants available to support participation.
Follow us for updates! #wcua #cua #ai #ml #computeruseagents #agents #icml2025
Website link below 👇 -
Russ Salakhutdinov reposted
Yiding Jiang
@yidingjiang
I will be presenting our work led by @FahimTajwar10 on training general decision making agents that solve completely new problems zero-shot at the workshop on self-Improving foundation models.
Fahim Tajwar
@FahimTajwar10
Interacting with the external world and reacting based on outcomes are crucial capabilities of agentic systems, but existing LLMs’ ability to do so is limited.
Introducing Paprika 🌶️, our work on making LLMs general decision makers than can solve new tasks zero-shot.
🧵 1/n -
Russ Salakhutdinov reposted
alphaXiv
@askalphaxiv
"Can Large Reasoning Models Self-Train?"
A brilliant paper from CMU showing LLMs can improve at math reasoning WITHOUT human labels - just learning from their own consistency.
Early results rival models trained on ground-truth answers. -
Russ Salakhutdinov reposted
Murtaza Dalal @ CoRL 2026
@mihdalal
Incredibly excited to share that Neural MP got accepted to IROS as an Oral presentation!! Huge congrats to the whole team (@Jiahui_Yang6709, @mendonca_rl , youssef khaky, @rsalakhu, @deepakpathak), but especially @Jiahui_Yang6709 for making this happen after I graduated! This now closes out all the work for my PhD - all papers accepted :)
Murtaza Dalal @ CoRL 2026
@mihdalal
Can a single neural network policy generalize over poses, objects, obstacles, backgrounds, scene arrangements, in-hand objects, and start/goal states?
Introducing Neural MP: A generalist policy for solving motion planning tasks in the real world 🤖
1/N -
Russ Salakhutdinov
@rsalakhu
Holy cow! It has been over 10 years - no way! Feels like I was giving this tutorial just a few years ago.
Russ Salakhutdinov
@rsalakhu
Check out my (somewhat) recent KDD tutorial on deep nets, RBMs, DBNs, DBMs, and multimodal deep learning:
videolectures.net/kdd2014_salakhutdinov_deep_learning/ -
Russ Salakhutdinov reposted
Fahim Tajwar
@FahimTajwar10
RL with verifiable reward has shown impressive results in improving LLM reasoning, but what can we do when we do not have ground truth answers?
Introducing Self-Rewarding Training (SRT): where language models provide their own reward for RL training!
🧵 1/n -
Russ Salakhutdinov
@rsalakhu
First time visiting @UofMaryland. Such a beautiful campus!
-
Russ Salakhutdinov
@rsalakhu
I had an inspiring visit to @mbzuai, where I had a few deep conversations with brilliant graduate students and faculty. What an extraordinary institution.
Congratulations to @ericxing for establishing a world-class AI university in such a short time — truly impressive! And huge thanks to @kunkzhang for hosting me.
MBZUAI
@mbzuai
#mbzuai welcomed Prof. Ruslan Salakhutdinov, UPMC Professor of Computer Science at @CarnegieMellon University and VP of Research at @Meta, to our campus.
As part of his visit, Prof. Salakhutdinov delivered an insightful session to our MBZUAI Executive Program (MEP) cohort. He also engaged the university community with a colloquium on Multimodal AI Agents—sharing pioneering research on autonomous systems that can perceive, reason, and act in complex digital environments.
#machinelearning #airesearch -
Russ Salakhutdinov
@rsalakhu
New work on automated prompt engineering for personalized text-to-image generation:
PRISM: Automated Black-box Prompt Engineering for Personalized Text-to-Image Generation
Paper + Code: kellyyutonghe.github.io/prism/
Prompt engineering for personalized image generation is labor-intensive or requires model-specific tuning, limiting generalization.
Key Idea: PRISM uses VLMs and iterative in-context learning to automatically generate effective, human-readable prompts using only black-box access to image generation models.
Our approach shows strong generalization and versatility in generating accurate prompts for objects, styles and images across multiple T2I models, including Stable Diffusion, DALL-E, and Midjourney. It also enables easy editing and multi-concept prompt generation.
See a more detailed thread by @electronickale.
Yutong (Kelly) He
@electronickale
✨ Love 4o-style image generation but prefer to use Midjourney? Tired of manual prompt crafting from inspo images?
PRISM to the rescue! 🖼️→📝→🖼️
We automate black-box prompt engineering—no training, no embeddings, just accurate, readable prompts from your inspo images! 1/🧵 -
Russ Salakhutdinov reposted
Brandon Trabucco
@brandontrabucco
Building LLM Agents? Come to my talk at the #iclr DATA-FM workshop today at 2:30pm, Hall 4, Section 4.
I'll be presenting InSTA, our work building the largest environment for agents on the live internet.
arxiv.org/abs/2502.06776
#agents #llm -
Russ Salakhutdinov
@rsalakhu
Wow, @hugo_larochelle, are you planning to come back to academia?
Hugo Larochelle
@hugo_larochelle
Today is my last day at Google. I started over 8 years ago, with a mandate to build a team doing bleeding edge AI research from Montreal, in what would be the first big tech AI research lab in the city. These years led to countless amazing scientific contributions from my team, to several initiatives nurturing the Montreal AI ecosystem, and to many new invaluable friendships across the globe at Google. It is with a heavy heart that I say goodbye, but I know I’m leaving behind an exceptionally strong Google DeepMind group in Montreal for which its best accomplishments are still ahead.
There are too many people to thank, but I can’t pass on thanking Samy Bengio and @JeffDean who first believed in me and the opportunity of building a research lab in Montreal.
I’m still working on determining the details of my next chapter, but certainly it will be grounded in my continuing motivation to leverage and make the most out of our enormous and talented local AI ecosystem. -
Russ Salakhutdinov
@rsalakhu
Carnegie Mellon University at ICLR 2025 – Machine Learning Blog | ML@CMU | Carnegie Mellon University blog.ml.cmu.edu/2025/04/23/carnegie-mellon-university-at-…
-
Russ Salakhutdinov reposted
Brandon Trabucco
@brandontrabucco
🌏 Building web-scale agents, and tired of Math and Coding tasks? Come chat with us at ICLR in Singapore. We are presenting InSTA at the DATA-FM workshop in the second Oral session, April 28th 2:30pm.
InSTA is the largest environment for training agents, spanning 150k live websites.
Arxiv: arxiv.org/abs/2502.06776
Github: github.com/data-for-agents/insta
Bookmark these, our next release contains high-quality demos for all 150k tasks.
#agents #llms #openai #iclr25 #iclr -
Russ Salakhutdinov reposted
Kaps
@kapardhi200903
I just published Multimodal Autonomous AI Agents: Enhancing Web Interactions Through Tree Search
Prof. @rsalakhu 's fascinating work on AI that can actually navigate the web like humans do. Tree search is a game-changer for error recovery!
medium.com/p/multimodal-autonomous-ai-agents-enhancing-we…
#aiagent -
Russ Salakhutdinov
@rsalakhu
Copilot Arena: A Platform for Code – Machine Learning Blog | ML@CMU | Carnegie Mellon University blog.ml.cmu.edu/2025/04/09/copilot-arena-a-platform-for-c…
-
Russ Salakhutdinov
@rsalakhu
Cool
Wild Reptilians
@wild_reptilians
Explore the efficient computer use agent powered by Meta Llama 4 Maverick. Learn more here: github.com/TheoLeeCJ/llama4-computer-use
-
Russ Salakhutdinov reposted
Wild Reptilians
@wild_reptilians
Explore the efficient computer use agent powered by Meta Llama 4 Maverick. Learn more here: github.com/TheoLeeCJ/llama4-computer-use
-
Russ Salakhutdinov reposted
Laurens van der Maaten
@lvdmaaten
And so it begins...
llama.com/4 -
Russ Salakhutdinov reposted
Ahmad Al-Dahle
@Ahmad_Al_Dahle
As of today, Llama 4 Maverick offers a best-in-class performance to cost ratio with an experimental chat version scoring ELO of 1417 on LMArena.
It's wild to think Llama was a research project a couple of years ago & amazing to see how much progress we've made in the last two years 🚀 And this is just the first taste of the Llama 4 collection – get ready for a herd like you’ve never seen before.
Very proud of the GenAI team. -
Russ Salakhutdinov reposted
Ahmad Al-Dahle
@Ahmad_Al_Dahle
Introducing our first set of Llama 4 models!
We’ve been hard at work doing a complete re-design of the Llama series. I’m so excited to share it with the world today and mark another major milestone for the Llama herd as we release the *first* open source models in the Llama 4 collection 🦙. Here are some highlights:
📌 The Llama series have been re-designed to use state of the art mixture-of-experts (MoE) architecture and natively trained with multimodality. We’re dropping Llama 4 Scout & Llama 4 Maverick, and previewing Llama 4 Behemoth.
📌 Llama 4 Scout is highest performing small model with 17B activated parameters with 16 experts. It’s crazy fast, natively multimodal, and very smart. It achieves an industry leading 10M+ token context window and can also run on a single GPU!
📌 Llama 4 Maverick is the best multimodal model in its class, beating GPT-4o and Gemini 2.0 Flash across a broad range of widely reported benchmarks, while achieving comparable results to the new DeepSeek v3 on reasoning and coding – at less than half the active parameters. It offers a best-in-class performance to cost ratio with an experimental chat version scoring ELO of 1417 on LMArena. It can also run on a single host!
📌 Previewing Llama 4 Behemoth, our most powerful model yet and among the world’s smartest LLMs. Llama 4 Behemoth outperforms GPT4.5, Claude Sonnet 3.7, and Gemini 2.0 Pro on several STEM benchmarks. Llama 4 Behemoth is still training, and we’re excited to share more details about it even while it’s still in flight.
A big thanks to all of our launch partners (full list in blog) for helping us bring Llama 4 to developers everywhere including @huggingface, @togethercompute, @SnowflakeDB, @ollama, @databricks and many others👏 This is just the start, we have more models coming and the team is really cooking – look out for Llama 4 Reasoning 😉
A few weeks ago, we celebrated Llama being downloaded over 1 billion times. Llama 4 demonstrates our long-term commitment to open source AI, the entire open source AI community, and our unwavering belief that open systems will produce the best small, mid-size and soon frontier models. Llama would be nothing without the global open source AI community & we are so ready to begin this next chapter with you. 🦙
Read more about the release here: www.llama.com, and try it in our products today. -
Russ Salakhutdinov
@rsalakhu
Llama4 models are out! Open sourced! Check them out:
“Native multimodality, mixture-of-experts models, super long context windows, step changes in performance, and unparalleled efficiency. All in easy-to-deploy sizes custom fit for how you want to use it” www.llama.com -
Russ Salakhutdinov reposted
UC Berkeley RDI
@BerkeleyRDI
What if LLM Agents could navigate the visual web just like humans do? Lecture 6 (@rsalakhu @CarnegieMellon @Meta) explored multimodal autonomous AI agents with huge potential to automate complex, visually rich web tasks! 🔥🧵
1/7 Multimodal Autonomous AI agents combine Large Language Models (LLMs) and Visual Language Models (VLMs) to handle visually-rich web tasks like online shopping, posting comments, and navigating websites. 📺👀 -
Russ Salakhutdinov reposted
Jason Liu
@JasonJZLiu
FACTR is now open source! You can now build your own low-cost force-feedback teleop system.
Hardware: github.com/JasonJZLiu/FACTR_Hardware
Teleop Code: github.com/RaindragonD/factr_teleop/
Training Code: github.com/RaindragonD/factr/
Jason Liu
@JasonJZLiu
Low-cost teleop systems have democratized robot data collection, but they lack any force feedback, making it challenging to teleoperate contact-rich tasks.
Many robot arms provide force information — a critical yet underutilized modality in robot learning.
We introduce:
1. 🦾A low-cost, force-feedback-enabled teleop system.
2. 🥊Force-Attending Curriculum Training (FACTR) uses force to improve generalization in complex, contact-rich tasks.
jasonjzliu.com/factr/
🧵(1/N)This media couldn't be displayed. See it on the original post. -
Russ Salakhutdinov reposted
Dawn Song
@dawnsongtweets
🙏 Huge thanks to @rsalakhu @CarnegieMellon @Meta for the 6th lecture on Multimodal Autonomous AI Agents. Watch the recording: www.youtube.com/watch?v=RPINOYM12RU
🧠 Sign up for the AgentX Competition: rdi.berkeley.edu/agentx/
📚 Sign up for the Advanced LLM Agents MOOC: -
Russ Salakhutdinov
@rsalakhu
New work on Privacy Leakage Evaluation for Autonomous Web Agents:
Paper: arxiv.org/abs/2503.09780
LLM-powered AI agents hold great potential for enhancing productivity but risk inadvertent privacy leakage when given access to sensitive information. To address this, AI agents can be trained to better satisfy the privacy principle of data minimization, i.e. sharing private information only when strictly necessary for a specific task.
This work also develops AgentDAM, a new benchmark that evaluates how well AI agents can limit unnecessary processing of potentially private information in realistic web scenarios. Our findings show that existing agents can often misuse sensitive data, but a prompting-based approach can substantially reduce this risk.
See a more detailed thread by @kamalikac.
Kamalika Chaudhuri
@kamalikac
Can you trust an AI agent with your private information? LLM-powered AI agents that can take action on someone's behalf can significantly boost productivity, yet they can also unnecessarily leak their user's private information.
-
Russ Salakhutdinov reposted
Kamalika Chaudhuri
@kamalikac
Can you trust an AI agent with your private information? LLM-powered AI agents that can take action on someone's behalf can significantly boost productivity, yet they can also unnecessarily leak their user's private information.
-
Russ Salakhutdinov
@rsalakhu
Congratulations, Dr. Murtaza Dalal! 🎉 @mihdalal
It feels like just yesterday you have started your PhD journey in my lab. Looking forward to the amazing things things you'll do next!
Murtaza Dalal @ CoRL 2026
@mihdalal
Excited to share that I have successfully defended my PhD! I am officially Dr. Murtaza Dalal :)
It has been an amazing journey over the last 4.5 years, and it would not have been possible without the amazing support and mentorship of my advisor @rsalakhu as well as @deepakpathak! Also thank you so much to my thesis committee for agreeing to serve for my defense! @davheld @SongShuran @ankurhandos
My thesis: Unlocking Generalization for Robotics via Modularity and Scale
Link: arxiv.org/abs/2503.06814
Talk: www.youtube.com/watch?v=Ap5wa2HEDBQ
As for what I'm doing post PhD, needless to say its incredibly exciting work, stay tuned for updates ;) -
Russ Salakhutdinov
@rsalakhu
New work on optimizing test-time compute as a meta-reinforcement learning (RL) problem, which provides a new perspective on test-time compute:
Paper + Code: cohenqu.github.io/mrt.github.io/
This viewpoint allows us to interpret the long output stream from an LLM as a series of test-time episodes, leading us to use a notion of cumulative regret over output tokens as a metric for measuring test-time compute efficiency.
Similar to how RL algorithms can balance exploration and exploitation over training, minimizing cumulative regret can also provide the best tradeoff between exploration and exploitation in the token stream.
This work also develops Meta Reinforcement Fine-Tuning, or MRT, a new class of fine-tuning methods for optimizing test-time compute. MRT leads to a 2-3x relative gain in performance and roughly a 1.5x gain in token efficiency for math reasoning compared to outcome-reward RL.
See a more detailed thread by @QuYuxiao.
Yuxiao Qu
@QuYuxiao
🚨 NEW PAPER: "Optimizing Test-Time Compute via Meta Reinforcement Fine-Tuning"!
🤔 With all these long-reasoning LLMs, what are we actually optimizing for? Length penalties? Token budgets? We needed a better way to think about it!
Website: cohenqu.github.io/mrt.github.io/
🧵[1/9] -
Russ Salakhutdinov reposted
Murtaza Dalal @ CoRL 2026
@mihdalal
Excited to share that I have successfully defended my PhD! I am officially Dr. Murtaza Dalal :)
It has been an amazing journey over the last 4.5 years, and it would not have been possible without the amazing support and mentorship of my advisor @rsalakhu as well as @deepakpathak! Also thank you so much to my thesis committee for agreeing to serve for my defense! @davheld @SongShuran @ankurhandos
My thesis: Unlocking Generalization for Robotics via Modularity and Scale
Link: arxiv.org/abs/2503.06814
Talk: www.youtube.com/watch?v=Ap5wa2HEDBQ
As for what I'm doing post PhD, needless to say its incredibly exciting work, stay tuned for updates ;) -
Russ Salakhutdinov reposted
Dawn Song
@dawnsongtweets
📣 Join us for the 6th Advanced LLM Agents MOOC lecture on Web Agents @rsalakhu @CarnegieMellon @Meta, 4:10 pm PT today March 10.
🚀 Join the thriving community of the LLM Agents MOOC series, with 22K+ registered learners & ~9K members on Discord! -
Russ Salakhutdinov reposted
Fahim Tajwar
@FahimTajwar10
Interacting with the external world and reacting based on outcomes are crucial capabilities of agentic systems, but existing LLMs’ ability to do so is limited.
Introducing Paprika 🌶️, our work on making LLMs general decision makers than can solve new tasks zero-shot.
🧵 1/n -
Russ Salakhutdinov reposted
Vector Institute
@VectorInst
From simulation to physical reasoning to autonomous agents, today's Remarkable keynotes explored new frontiers in AI research:
Nitish Srivastava revealed how simulation creates safe learning environments for autonomous driving robots, addressing the challenge of rare but crucial edge cases in real-world training.
Vector Faculty Member Kelsey Allen explored how humans and machines understand physical reasoning - from everyday tasks to engineering challenges - advancing our path toward more robust and efficient AI systems.
Ruslan Salakhutdinov demonstrated how AI agents can navigate complex digital environments, introducing VisualWebArena for evaluating autonomous systems across thousands of websites.
And the research continues - our community now moves into specialized workshops spanning computer vision, NLP, ML security, and more.
#remarkable2025 -
Russ Salakhutdinov reposted
Vector Institute
@VectorInst
👋 Join us virtually TOMORROW for groundbreaking insights at Remarkable 2025! Our Research Symposium begins with three keynotes:
1⃣ @nitishsr reveals how simulation helps autonomous driving robots learn from rare but crucial edge cases, creating safe spaces for exploration and learning.
2⃣ @KelseyRAllen examines how humans and machines understand physical reasoning - from everyday challenges like using tools, to complex engineering feats like designing wind turbines
3⃣ @rsalakhu demonstrates the latest advances in multimodal AI agents, showcasing systems that can plan, reason, and execute complex tasks across digital environments.
Registration is free 👇
vectorinstitute.swoogo.com/remarkable-2025/7118262?ref=og… -
Russ Salakhutdinov reposted
Agentica Project
@Agentica_
✨RL magic is in the air! Introducing DeepScaleR-1.5B-Preview—a fully open-source, 1.5B-parameter model trained with RL to surpass o1-preview for general math reasoning.
📜Blog: pretty-radio-b75.notion.site/DeepScaleR-Surpassing-O1-Pre…
💻Github: github.com/agentica-project/deepscaler -
Russ Salakhutdinov reposted
Murtaza Dalal @ CoRL 2026
@mihdalal
Excited to share that ManipGen has been accepted to ICRA 2025! We have released the code here: github.com/mihdalal/manipgen. Run our generalist system on your robot today!
Murtaza Dalal @ CoRL 2026
@mihdalal
Can my robot cook my food, rearrange my dresser, tidy my messy table and do so much more without ANY demos or real-world training data?
Introducing ManipGen: A generalist agent for manipulation that can solve long-horizon robotics tasks entirely zero shot, from text input!
1/NThis media couldn't be displayed. See it on the original post. -
Russ Salakhutdinov
@rsalakhu
New work on robot learning & teleop with force:
FACTR: Force-Attending Curriculum Training for Contact-Rich Policy Learning
Paper + Code: jasonjzliu.com/factr/
Force information is crucial for contact-rich tasks, but behavior cloning policies tend to ignore robot force input if added naively, as policies tend to overfit to the vision input.
Key Idea: FACTR is a behavior cloning training curriculum that corrupts vision input with decreasing intensity throughout training. This helps the policy to properly attend to force data. Our policies perform and generalize better:
· 46% improvement in success rate for unseen objects
· Exhibit emergent recovery behavior, unobserved in baselines
We also add force-feedback to low-cost leader-follower teleop system, which is especially helpful for collecting data for contact-rich tasks:
· 64.7% higher task completion rate
· 37.4% reduced completion time
· 83.3% improvement in the subjective ease of use
See a more detailed thread by @JasonJZLiu
Jason Liu
@JasonJZLiu
Low-cost teleop systems have democratized robot data collection, but they lack any force feedback, making it challenging to teleoperate contact-rich tasks.
Many robot arms provide force information — a critical yet underutilized modality in robot learning.
We introduce:
1. 🦾A low-cost, force-feedback-enabled teleop system.
2. 🥊Force-Attending Curriculum Training (FACTR) uses force to improve generalization in complex, contact-rich tasks.
jasonjzliu.com/factr/
🧵(1/N)This media couldn't be displayed. See it on the original post. -
Russ Salakhutdinov reposted
Rose Yu
@yuqirose
@rsalakhu given the first keynote talk! #genaiucsd25 @ucsd_cse
-
Russ Salakhutdinov
@rsalakhu
New work #iclr2025 on “Dissecting Adversarial Robustness of Multimodal LM Agents” that shows that one can successfully break latest agents that use black-box frontier LLMs, including agents that perform reflection and tree search.
Paper + Code + Data: chenwu.io/attack-agent/
With small perturbations (less than 5% of total web page pixels), attackers can execute targeted adversarial goals with up to 67% success rates.
This work also develops Agent Robustness Evaluation (ARE) -- a framework that views the agent as a graph and decomposes robustness as the flow of adversarial information on the graph. Using this framework, we find that inference-time compute that is often used to improve model performance can introduce new vulnerabilities and harm robustness.
See a more detailed thread by @ChenHenryWu
Chen Wu
@ChenHenryWu
1/🧵 2025 is the "Year of AI Agents" 🤖 But can you trust them to shop for you? Our latest work reveals critical security risks: improving AI agent performance (e.g., inference-time methods like reflection and tree search) can also make them more vulnerable to adversarial attacks! We introduce a benchmark & framework to assess agent security. 🔍👇
-
Russ Salakhutdinov
@rsalakhu
Excited to be at the GenAI Summit at UCSD!
I'll be sharing our latest work on VisualWebArena, inference-time tree search, and Internet-scale training of LLM Agents.
genaisummit2025.ucsd.edu -
Russ Salakhutdinov
@rsalakhu
New Work on InSTA: A pipeline for Internet-scale training of web agents across 150k diverse websites without human annotations.
Paper + Code: data-for-agents.github.io
Environment: github.com/data-for-agents/environment
The pipeline follows a three-step process:
- LLM generates tasks for 150k websites
- LLM agents complete these tasks and produce trajectories
- LLM reviews the trajectories and evaluates their success.
Language models perform competitively with human annotators, achieving:
- 97% accuracy in detecting and filtering harmful content.
- 89% success rate in generating feasible tasks.
- 82% accuracy in judging successful task completions.
When scaling the pipeline, Llama 3.1 70B agents successfully complete 16.7% of tasks across 150k websites. Agents trained on human-annotated data from Mind2Web and WebLINX struggle to generalize to real-world websites. Adding synthetic data significantly improves their generalization.
See a more detailed thread by @brandontrabucco
Brandon Trabucco
@brandontrabucco
With the success of LLM agents like OpenAI Operator, we are entering a new scaling era, but how do we train these agent models?
We present InSTA, the largest training environment for LLM agents, containing live web navigation tasks for 150k diverse websites in multiple languages.
Website - data-for-agents.github.io
Environment - github.com/data-for-agents/environment
🧵Thread below.
1/6
#agenticai #llms #openal -
Russ Salakhutdinov
@rsalakhu
New work on Self-Regulation and Requesting Interventions: enabling agents with a limited intervention budget to decide when to seek assistance:
Paper + Code: soyeonm.github.io/self_reg/
We develop an offline framework that trains a helper policy to request interventions (e.g. more powerful models or additional test-time compute) by combining LLM-based process reward models (PRMs) with reinforcement learning (RL).
Offline-collected state transitions are evaluated using PRMs to determine optimal intervention timing, creating labeled trajectories for training the helper model. This offline approach minimizes costly intervention calls during training while leveraging PRMs to enhance robustness to off-policy data.
See an excellent thread by @SoYeonTiffMin
So Yeon (Tiffany) Min
@SoYeonTiffMin
🚨🚨 Preprint Alert 🚨🚨 🚀🚀
As AI become agents 🤖, how can we reliably delegate tasks to them, if they cannot communicate their limitations😭 or ask for help or test-time compute 🧑🚒 when needed?
We present our new pre-print **Self-Regulation and Requesting Interventions** that investigates how LLM agents 🤖 can assess their own limitations and determine when to leverage test-time compute, larger models, or human intervention🧑🚒.
Combining LLM-based process reward models (PRMs) with classical RL, we developed an off-line and hybrid method that eschews the inefficiencies of end-to-end deep learning while leveraging the robustness of PRMs. Empirically, our method matches the performance of ALWAYS using interventions, even when requiring only one intervention per task—just about 1/10 intervention usage.
Project website: soyeonm.github.io/self_reg/
(1/5) -
Russ Salakhutdinov reposted
Oumi
@oumi_ai
Introducing Oumi 🚀 - a community of researchers and developers united in their mission to make frontier AI more open, collaborative, and accessible. Join us in building the platform, models, and tools: let’s shape the future of AI. #oumi #opensource #collaboration
Learn more: oumi.ai
Join us on GitHub - github.com/oumi-ai/oumi
Join us on Discord - discord.gg/oumi
Ex-Google, Apple engineers launch unconditionally open source Oumi AI platform that could help to build the next DeepSeek venturebeat.com/ai/ex-google-apple-engineers-launch-uncon… via @VentureBeat -
Russ Salakhutdinov
@rsalakhu
Optimizing LLM Test-Time Compute Involves Solving a Meta-RL Problem – Machine Learning Blog | ML@CMU | Carnegie Mellon University blog.ml.cmu.edu/2025/01/08/optimizing-llm-test-time-compu…
-
Russ Salakhutdinov
@rsalakhu
Inductive biases of neural network modularity in spatial navigation – Machine Learning Blog | ML@CMU | Carnegie Mellon University blog.ml.cmu.edu/2025/01/02/inductive-biases-of-neural-net…
-
Russ Salakhutdinov reposted
Ahmad Al-Dahle
@Ahmad_Al_Dahle
In 12 months, Llama has become the most adopted model – with more than 650 million downloads. Imagine what we will do in the next 12 months.
2024 in a nutshell:
ai.meta.com/blog/future-of-ai-built-with-llama/
Highlights here! -
Russ Salakhutdinov reposted
Paul Vicol
@PaulVicol
👀 Wen-Tse Chen presented "Fine-tuning LLM Agents with Retrospective In-Context Online Learning"
openreview.net/forum?id=dSZ6Wr5qRz
Wen-Tse Chen @JiayuChen98666 @FahimTajwar10 @_Hao_Zhu Xintong Duan @rsalakhu Jeff Schneider
#neurips2024 #adaptivefoundationmodels -
Russ Salakhutdinov reposted
Paul Vicol
@PaulVicol
Ruslan Salakhutdinov at the Adaptive Foundation Models Workshop!
-
Russ Salakhutdinov reposted
Rahel Jhirad
@RahelJhirad
Adaptive Foundation Models: Evolving AI for Personalized and Efficient Learning Workshop
#neurips2024
First talk: @rsalakhu
Tree Search for Language Model Agents
Autonomous AI agents, specifically web agents and visual grounding -
Russ Salakhutdinov
@rsalakhu
With my amazing students and collaborators at @NeurIPSConf in Vancouver!
-
Russ Salakhutdinov reposted
Alexandre Lacoste
@alex_lacoste_
We’re really excited to release this large collaborative work for unifying web agent benchmarks under the same roof.
In this TMLR paper, we dive in-depth into #browsergym and #agentlab. We also present some unexpected performances from Claude 3.5-Sonnet -
Russ Salakhutdinov reposted
MIT Media Lab
@medialab
Congratulations to the 2025 Forbes 30 Under 30 honorees, including Media Lab Prof. @pliang279, @mitcities PhD student Naroa Coretti Sánchez, and Zoë Bentley, Accessibility Lead for the Lifelong Kindergarten group and one of the creators of the group's @OctoStudioApp! #forbesunder30 www.media.mit.edu/posts/congratulations-to-the-2025-forbe…
-
Russ Salakhutdinov reposted
Ben Eysenbach
@ben_eysenbach
Check out the poster this evening 4:30pm -- 6:30pm at #neurips2024 !
Ben Eysenbach
@ben_eysenbach
1/ Given high-dim time series data (e.g., video, text), we'd like to know "what happens in the future?" and "what happened between A and B?" Turns out temporal contrastive learning provably answers these questions!
Paper+Code: arxiv.org/abs/2403.04082
Backstory and details below -
Russ Salakhutdinov reposted
Paul Liang
@pliang279
If you're at #neurips2024 this week, check out HEMM (Holistic Evaluation of Multimodal Foundation Models), the largest and most comprehensive evaluation for multimodal models like Gemini, GPT-4V, BLIP-2, OpenFlamingo, and more.
Fri 13 Dec 11am - 2pm @ West Ballroom A-D #6807
HEMM gives a systematic way to evaluate multimodal models based on:
1. **Basic multimodal skills** – the type of multimodal interaction, granularity of multimodal alignment, level of reasoning, and need for external knowledge,
2. How **information flows** between modalities – querying, translation, editing, and fusion,
3. Their real-world **use cases** – multimedia, affective computing, healthcare, science & environment, HCI.
paper: arxiv.org/abs/2407.03418
code: github.com/pliang279/HEMM -
Russ Salakhutdinov
@rsalakhu
ScribeAgent: Fine-Tuning Open-Source LLMs for Enhanced Web Navigation – Machine Learning Blog | ML@CMU | Carnegie Mellon University blog.ml.cmu.edu/2024/12/06/scribeagent-fine-tuning-open-s…
-
Russ Salakhutdinov reposted
So Yeon (Tiffany) Min
@SoYeonTiffMin
I am on the industry job market, and am planning to interview around next March. I am attending @NeurIPSConf, and I hope to meet you there if you are hiring! My website: soyeonm.github.io
Short bio about me:
I am a 5th year PhD student at CMU MLD, working with @rsalakhu and @ybisk. My current research focuses on the confidence of LLM agents - how LLM agents can assess their own limitations and determine when to leverage increased inference compute, larger models, or human intervention.
I am a recipient of Apple AI/ML Scholars Fellowship for 2023. I have interned at Apple AI/ML and Meta FAIR. My research has been recognized through multiple features in CMU news. I hold B.S., M.Eng from MIT EECS.
I am also presenting two of my recent works: Embodied-RAG/Situated Instruction Following in AFM and LangGame workshops.
RT appreciated! -
Russ Salakhutdinov
@rsalakhu
Carnegie Mellon University at NeurIPS 2024 – Machine Learning Blog | ML@CMU | Carnegie Mellon University
Carnegie Mellon University is proud to present 194 papers at the 38th conference on Neural Information Processing Systems (NeurIPS 2024) blog.ml.cmu.edu/2024/12/02/carnegie-mellon-university-at-… -
Russ Salakhutdinov
@rsalakhu
New work on “Evaluating Deep Unlearning in Large Language Models”
Unlearning specific facts in large language models (LLMs) is challenging because the facts in LLMs can be deduced from each other. This work formally proposes a framework and a definition for deep unlearning of facts that are interrelated.
Our findings show that even when unlearning a single fact, current methods either fail to properly unlearn with high recall or end up unlearning many other irrelevant facts.
Paper: arxiv.org/abs/2410.15153
Code+Dataset: github.com/wrh14/deep_unlearning
joint work with @ruihan_wu @chhaviyadav_ @kamalikac
Kamalika Chaudhuri
@kamalikac
We know that LLMs learn facts, but how well do
they properly unlearn pieces of knowledge? We investigate "deep unlearning" in our new paper. -
Russ Salakhutdinov
@rsalakhu
Absolutely agree! These lists are just silly.
Andrew Gordon Wilson
@andrewgwils
Some of these people are neither active AI researchers nor AI media influencers... It's about time people stop making these silly lists.
-
Russ Salakhutdinov
@rsalakhu
This is hilarious 😂
Pedro Domingos
@pmddomingos
Wow, @rsalakhu has morphed into Stephen Wolfram.
-
Russ Salakhutdinov reposted
David Duvenaud
@DavidDuvenaud
Not sure, but I think NN "inference" originally started with a connection made by @geoffreyhinton that mean-field inference in binary RBMs is equivalent to a sigmoid MLP. Then starting around 2012 @JeffDean started using it in talks and I think that was pretty influential.
-
Russ Salakhutdinov reposted
David Duvenaud
@DavidDuvenaud
Here's a representative paper: www.cs.toronto.edu/~hinton/csc2535/readings/efficientDBM.pdf
"... there is a fast way of performing approximate inference by a single bottom-up pass ..." -
Russ Salakhutdinov
@rsalakhu
What is happening?! Who is this? 😆
-
Russ Salakhutdinov
@rsalakhu
A better version of the online talk / slides on AI agents:
www.youtube.com/watch?v=j4bdkWYNvIY&t=3s
Russ Salakhutdinov
@rsalakhu
My talk on AI Agents is online:
www.youtube.com/watch?v=wK0TpI3gu28
Kempner Institute at Harvard University
@KempnerInst
The latest Kempner Seminar Series talk: @rsalakhu of @CarnegieMellon discusses the opportunities and challenges of using LLMs to drive autonomous agents that can navigate the internet… and also the real world! Check it out here: youtu.be/wK0TpI3gu28
-
Russ Salakhutdinov
@rsalakhu
My talk on AI Agents is online:
www.youtube.com/watch?v=wK0TpI3gu28
Kempner Institute at Harvard University
@KempnerInst
The latest Kempner Seminar Series talk: @rsalakhu of @CarnegieMellon discusses the opportunities and challenges of using LLMs to drive autonomous agents that can navigate the internet… and also the real world! Check it out here: youtu.be/wK0TpI3gu28
-
Russ Salakhutdinov reposted
Kempner Institute at Harvard University
@KempnerInst
The latest Kempner Seminar Series talk: @rsalakhu of @CarnegieMellon discusses the opportunities and challenges of using LLMs to drive autonomous agents that can navigate the internet… and also the real world! Check it out here: youtu.be/wK0TpI3gu28
-
Russ Salakhutdinov
@rsalakhu
If you’re interested in working on the next-generation agentic AI systems, consider joining @shuyanzhxyc's lab at Duke!
Shuyan Zhou
@shuyanzh36
My lab at Duke has multiple Ph.D. openings! Our mission is to augment human decision-making by advancing the reasoning, comprehension, and autonomy of modern AI systems.
I am attending #emnlp2024, happy to chat about PhD applications, LLM agents, evaluation etc etc! -
Russ Salakhutdinov
@rsalakhu
Identification of Hazardous Areas for Priority Landmine Clearance: AI for Humanitarian Mine Action – Machine Learning Blog | ML@CMU | Carnegie Mellon University blog.ml.cmu.edu/2024/11/07/identification-of-hazardous-ar…
-
Russ Salakhutdinov
@rsalakhu
Heading to Harvard. It’s been a while since my last visit. Looking forward to being back on campus.
kempnerinstitute.harvard.edu/events/ruslan-salakhutdinov/ -
Russ Salakhutdinov reposted
Nitish Srivastava
@nitishsr
Thank you @rsalakhu for visiting @VayuRobotics and spending time with our team! It was a pleasure showcasing a live demonstration of our robots in action.
Russ Salakhutdinov
@rsalakhu
It is always fun to visit Bay Area and catch up with old friends and former students.
Here is @VayuRobotics , co-founded by my former student Nitish Srivastava @nitishsr, leading the way in building the next generation of autonomous delivery robots! -
Russ Salakhutdinov
@rsalakhu
Stuck in Toronto traffic on a Sunday at noon! Both directions are completely jammed, and I’m not even in the downtown core. This is starting to feel worse than the Bay Area.
Next stop: Pittsburgh, where both Harris and Trump are holding rallies at the same time -- what a week-end. -
Russ Salakhutdinov
@rsalakhu
It is always fun to visit Bay Area and catch up with old friends and former students.
Here is @VayuRobotics , co-founded by my former student Nitish Srivastava @nitishsr, leading the way in building the next generation of autonomous delivery robots! -
Russ Salakhutdinov
@rsalakhu
Our new work “Local Policies Enable Zero-shot Long Horizon Manipulation” demonstrates how to train a library of generalist manipulation skills entirely in simulation, then transfer them zero-shot to real-world, long-horizon manipulation tasks.
We show that a single, text-conditioned agent can manipulate unseen objects in arbitrary poses and scene configurations with challenging obstacles. The agent can effectively operate across extended task sequences, showcasing strong adaptability and robustness in real-world settings.
Paper + Code: mihdalal.github.io/manipgen/
The key idea: decompose tasks into reaching and contact-rich interaction phases, using sim2real policies trained for local interaction that generalize robustly for real-world manipulation.
This was work led by my students @mihdalal and @minliu01 and done in collaboration with Jian Zhang’s team at Apple. See an excellent thread by Murtaza for more details:This media couldn't be displayed. See it on the original post.
Murtaza Dalal @ CoRL 2026
@mihdalal
Can my robot cook my food, rearrange my dresser, tidy my messy table and do so much more without ANY demos or real-world training data?
Introducing ManipGen: A generalist agent for manipulation that can solve long-horizon robotics tasks entirely zero shot, from text input!
1/NThis media couldn't be displayed. See it on the original post. -
Russ Salakhutdinov
@rsalakhu
Jailbreaking LLM-Controlled Robots – Machine Learning Blog | ML@CMU | Carnegie Mellon University blog.ml.cmu.edu/2024/10/29/jailbreaking-llm-controlled-ro…
-
Russ Salakhutdinov reposted
Jing Yu Koh
@kohjingyu
I've received several emails asking me for my research statement for the Jane Street graduate research fellowship (janestreet.com/join-jane-street/programs-and-events/gradu…), so I've decided to share it here publicly: jykoh.com/jykoh_janestreet_grf.pdf
I was fortunate to be awarded the fellowship last cycle. When applying, I wasn't exactly sure what was to be expected from these research statements, so I hope that this will be helpful to others!
The entire process of applying for these industry fellowships was very helpful in reflecting about my longer term research agenda, and thinking about my work from a more industry focused perspective. Would highly recommend applying for Jane Street's GFP (as well as other industry fellowships)! -
Russ Salakhutdinov reposted
GRASP Laboratory
@GRASPlab
Join us TOMORROW in welcoming Dr. Ruslan Salakhutdinov @rsalakhu who will be presenting "Multimodal AI Agents" from 10:30AM - 11:45AM.
For more info:
www.grasp.upenn.edu/events/fall-2024-grasp-on-robotics-ru…
#grasp #grasplab #grasponrobotics -
Russ Salakhutdinov reposted
Ahmad Al-Dahle
@Ahmad_Al_Dahle
On device and small models are a really important part of the Llama herd so we are introducing quantized versions with significantly increased speed. These models have a 2-3x increased speedup – that is fastI Add a link if you want to share what you are building with Llama!
-
Russ Salakhutdinov reposted
Paul Liang
@pliang279
📣 Announcing the name and theme of my new research group at MIT @medialab @MITEECS:
***Multisensory Intelligence***
www.media.mit.edu/groups/multisensory-intelligence/overview/
Our group studies the foundations of multisensory AI to create human-AI symbiosis across scales and sensory mediums.
We are hiring at all levels! For graduate students, please apply through the programs in Media Arts & Sciences or EECS!
MAS: www.media.mit.edu/graduate-program/apply/
EECS: www.eecs.mit.edu/academics/graduate-programs/admission-pr… -
Russ Salakhutdinov
@rsalakhu
I will be visiting UPenn on Oct 25 to talk about our recent work at CMU on Multimodal AI Agents, VisualWebArena, and inference-time search for LLM agents.
Oct 25, 10:30am EST. The talk will be live-streamed on Zoom.
www.grasp.upenn.edu/events/fall-2024-grasp-on-robotics-ru… -
Russ Salakhutdinov
@rsalakhu
Congratulations Demis Hassabis @demishassabis! What an honor, and so well deserved!
The Nobel Prize
@NobelPrize
BREAKING NEWS
The Royal Swedish Academy of Sciences has decided to award the 2024 #nobelprize in Chemistry with one half to David Baker “for computational protein design” and the other half jointly to Demis Hassabis and John M. Jumper “for protein structure prediction.” -
Russ Salakhutdinov reposted
Carnegie Mellon University
@CarnegieMellon
Former @SCSatCMU faculty member Geoffrey Hinton has been awarded the 2024 Nobel Prize in Physics! 👏
Hinton, now at @UofT, was recognized alongside John J. Hopfield of @Princeton for their work in machine learning with artificial neural networks.
➡️ cmu.is/Hinton-Nobel-Prize -
Russ Salakhutdinov
@rsalakhu
Former CMU Faculty Geoffrey Hinton Awarded 2024 Nobel Prize in Physics - News - Carnegie Mellon University www.cmu.edu/news/stories/archives/2024/october/former-cmu…
-
Russ Salakhutdinov
@rsalakhu
Heartfelt congratulations to Geoff Hinton @geoffreyhinton on winning the Nobel Prize! What an incredible honor!
I feel deeply privileged to have had the opportunity to be your PhD student, work with you, and learn from you.
Tsarathustra
@tsarnick
The 2024 Nobel Prize in Physics has been awarded to Geoffrey Hinton and John Hopfield for machine learning with artificial neural networks
-
Russ Salakhutdinov
@rsalakhu
New work on Embodied-RAG: General Non-parametric Embodied Memory for Retrieval and Generation:
A new framework that equips embodied agents with a non-parametric memory capable of autonomously constructing hierarchical knowledge for navigation and language generation.
Structured as a semantic forest, Embodied-RAG can handle diverse spatial and semantic queries efficiently, enabling context-sensitive outputs across different robotic environments.
Web: quanting-xie.github.io/Embodied-RAG-web/
Paper: arxiv.org/abs/2409.18313
Video: youtu.be/LcB89Rdyxhg
See an excellent thread by @SoYeonTiffMin
So Yeon (Tiffany) Min
@SoYeonTiffMin
In recent years, we have seen progress of foundation models (RT-X models and V/LLM's) as embodied agents. However, methods to augment these models with general-purpose long-term/large-scale memory has been under-explored. Clearly, as the environment becomes larger (e.g. outdoors) in navigation/"mobile" manipulation, we need a general-purpose external memory.
We introduce Embodied-RAG, a General Non-Parametric Method for Retrieval and Generation. (1/5) -
Russ Salakhutdinov
@rsalakhu
More importantly, check out the @AIatMeta's research paper that details how Meta Movie Gen foundation model was built.
ai.meta.com/static-resource/movie-gen-research-paper
AI at Meta
@AIatMeta
🎥 Today we’re premiering Meta Movie Gen: the most advanced media foundation models to-date.
Developed by AI research teams at Meta, Movie Gen delivers state-of-the-art results across a range of capabilities. We’re excited for the potential of this line of research to usher in entirely new possibilities for casual creators and creative professionals alike.
More details and examples of what Movie Gen can do ➡️ go.fb.me/kx1nqm
🛠️ Movie Gen models and capabilities
Movie Gen Video: 30B parameter transformer model that can generate high-quality and high-definition images and videos from a single text prompt.
Movie Gen Audio: A 13B parameter transformer model that can take a video input along with optional text prompts for controllability to generate high-fidelity audio synced to the video. It can generate ambient sound, instrumental background music and foley sound — delivering state-of-the-art results in audio quality, video-to-audio alignment and text-to-audio alignment.
Precise video editing: Using a generated or existing video and accompanying text instructions as an input it can perform localized edits such as adding, removing or replacing elements — or global changes like background or style changes.
Personalized videos: Using an image of a person and a text prompt, the model can generate a video with state-of-the-art results on character preservation and natural movement in video.
We’re continuing to work closely with creative professionals from across the field to integrate their feedback as we work towards a potential release. We look forward to sharing more on this work and the creative possibilities it will enable in the future.This media couldn't be displayed. See it on the original post. -
Russ Salakhutdinov reposted
AI at Meta
@AIatMeta
🎥 Today we’re premiering Meta Movie Gen: the most advanced media foundation models to-date.
Developed by AI research teams at Meta, Movie Gen delivers state-of-the-art results across a range of capabilities. We’re excited for the potential of this line of research to usher in entirely new possibilities for casual creators and creative professionals alike.
More details and examples of what Movie Gen can do ➡️ go.fb.me/kx1nqm
🛠️ Movie Gen models and capabilities
Movie Gen Video: 30B parameter transformer model that can generate high-quality and high-definition images and videos from a single text prompt.
Movie Gen Audio: A 13B parameter transformer model that can take a video input along with optional text prompts for controllability to generate high-fidelity audio synced to the video. It can generate ambient sound, instrumental background music and foley sound — delivering state-of-the-art results in audio quality, video-to-audio alignment and text-to-audio alignment.
Precise video editing: Using a generated or existing video and accompanying text instructions as an input it can perform localized edits such as adding, removing or replacing elements — or global changes like background or style changes.
Personalized videos: Using an image of a person and a text prompt, the model can generate a video with state-of-the-art results on character preservation and natural movement in video.
We’re continuing to work closely with creative professionals from across the field to integrate their feedback as we work towards a potential release. We look forward to sharing more on this work and the creative possibilities it will enable in the future.This media couldn't be displayed. See it on the original post. -
Russ Salakhutdinov reposted
Yash Butala
@YashButala
🚀 Excited to present OmniACT, the game-changing dataset and benchmark to enable multimodal agents that tackle your computer tasks!
With: @raghav_kapoor16, @Get_Writer, @rsalakhu, @waseem_s, @kohjingyu @MelisaStal and Kiran Kamble.
Join us at poster #122 tomorrow to learn more! #ai #agents #eccv2024 🇮🇹
👉 OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Web
👉 Demo and paper:
eccv.ecva.net/virtual/2024/poster/1107
👉 Dataset: huggingface.co/datasets/Writer/omniact -
Russ Salakhutdinov reposted
So Yeon (Tiffany) Min
@SoYeonTiffMin
We find that the method effectively works for explicit, implicit, and global questions, across 19 indoor/ outdoor environments.
More demos and information are in the website:
quanting-xie.github.io/Embodied-RAG-web/
Work done by @DanielXieee and myself as co-first authors; supervised by @ybisk, Matthew Johnson-Roberson, and @rsalakhu.
(5/5) -
Russ Salakhutdinov reposted
So Yeon (Tiffany) Min
@SoYeonTiffMin
In recent years, we have seen progress of foundation models (RT-X models and V/LLM's) as embodied agents. However, methods to augment these models with general-purpose long-term/large-scale memory has been under-explored. Clearly, as the environment becomes larger (e.g. outdoors) in navigation/"mobile" manipulation, we need a general-purpose external memory.
We introduce Embodied-RAG, a General Non-Parametric Method for Retrieval and Generation. (1/5) -
Russ Salakhutdinov
@rsalakhu
No Free Lunch in LLM Watermarking: Trade-offs in Watermarking Design Choices – Machine Learning Blog | ML@CMU | Carnegie Mellon University blog.ml.cmu.edu/2024/09/27/no-free-lunch-in-llm-watermark…
-
Russ Salakhutdinov reposted
Michael Luo
@michaelzluo
Excited to announce today that Stylus has been accepted at NeurIPS 2024 with Oral!
Link: stylus-diffusion.github.io
Special thanks to my collaborators and advisors: @justinywong_ , @brandontrabucco, @bignamehyp , Ion Stoica, @profjoeyg, @rsalakhu
#ai #neurips #neurips2024 -
Russ Salakhutdinov reposted
Ahmad Al-Dahle
@Ahmad_Al_Dahle
📣 You can now have a conversation with Meta AI using voice. It’s super fast, connected to the web, natural and conversational and even comes with celebrity voice options from Awkwafina, Kristen Bell, John Cena, and more. What voice speaks to you? (pun intended 😆)
This media couldn't be displayed. See it on the original post. -
Russ Salakhutdinov
@rsalakhu
Open source! Including 11B and 90B vision models!
AI at Meta
@AIatMeta
📣 Introducing Llama 3.2: Lightweight models for edge devices, vision models and more!
What’s new?
• Llama 3.2 1B & 3B models deliver state-of-the-art capabilities for their class for several on-device use cases — with support for @Arm, @MediaTek & @Qualcomm on day one.
• Llama 3.2 11B & 90B vision models deliver performance competitive with leading closed models — and can be used as drop-in replacements for Llama 3.1 8B & 70B.
• New Llama Guard models to support multimodal use cases and edge deployments.
• The first official distro of Llama Stack simplifies and supercharges the way developers & enterprises can build around Llama to support agentic applications and more.
Details in the full announcement ➡️ go.fb.me/229ug4
Download Llama 3.2 models ➡️ go.fb.me/w63yfd
These models are available to download now directly from Meta and @HuggingFace — and will be available across offerings from 25+ partners that are rolling out starting today, including @accenture, @awscloud, @AMD, @azure, @Databricks, @Dell, @Deloitte, @FireworksAI_HQ, @GoogleCloud, @GroqInc, @IBMwatsonx, @Infosys, @Intel, @kaggle, @NVIDIA, @OracleCloud, @PwC, @scale_AI, @snowflakeDB, @togethercompute and more.
With Llama 3.2 we’re making it possible to run Llama in even more places, with even more flexible capabilities. We’ve said it before and we’ll say it again: open source AI is how we ensure that these innovations reflect the global community they’re built for and benefit everyone. We’re continuing our drive to make open source the standard with Llama 3.2. -
Russ Salakhutdinov reposted
Video Arena
@aivideoarena
🚀 Just Launched: VideoArena!🎥
Discover head-to-head comparisons of video clips generated from the same prompts across top text-to-video models. Compare outputs from 7 leading models and we're adding more soon!
🔗 Check out the leaderboard: videoarena.tv
#text2video -
Russ Salakhutdinov
@rsalakhu
Feels like ICML 2024 was so long ago!
-
Russ Salakhutdinov reposted
Kempner Institute at Harvard University
@KempnerInst
See the speaker line-up for 2024-25 #kempnerinstitute Seminar Series! kempnerinstitute.harvard.edu/news/kempner-institute-annou…
@noahdgoodman @MarkChurchland @scott_linderman @jacobandreas @FieteGroup @rsalakhu @StefanoFusi2 @RAIVNLab @SuthanaLab @behrenstimb @tri_dao @KordingLab @tyrell_turing @AndrewLampinen -
Russ Salakhutdinov
@rsalakhu
Being in academia is amazing.
Being in industry is exciting.
Being in both is electrifying. -
Russ Salakhutdinov
@rsalakhu
Rethinking LLM Memorization – Machine Learning Blog | ML@CMU | Carnegie Mellon University blog.ml.cmu.edu/2024/09/13/rethinking-llm-memorization/
-
Russ Salakhutdinov
@rsalakhu
Check out our new work on Neural Motion Planner: A Generalist Neural Motion Planner.
A single neural network policy that can successfully solve motion planning tasks in the real world across diverse environments with randomized poses, scenes and obstacles, substantially improving over state-of-the-art sampling, optimization and learning based planning methods.
Paper: mihdalal.github.io/neuralmotionplanner/resources/paper.pdf
Code: github.com/mihdalal/neuralmotionplanner
Web: mihdalal.github.io/neuralmotionplanner/
The key idea is to build a large number of complex scenes in simulation, collect expert data from a motion planner, and then distill it into a reactive generalist policy.
See a detailed thread by @mihdalal:
Murtaza Dalal @ CoRL 2026
@mihdalal
Can a single neural network policy generalize over poses, objects, obstacles, backgrounds, scene arrangements, in-hand objects, and start/goal states?
Introducing Neural MP: A generalist policy for solving motion planning tasks in the real world 🤖
1/N -
Russ Salakhutdinov
@rsalakhu
Canadian AI Researchers in London: Had great time catching up with my long-time friend and Toronto PhD lab mate Andriy Mnih and my former PhD student Emilio Parisotto.
Also, almost got hit a couple of times by cab drivers – wow cab drivers have gotten pretty aggressive. -
Russ Salakhutdinov
@rsalakhu
Welcome @shuyanzhxyc! Looking forward to working with you!
Shuyan Zhou
@shuyanzh36
I recently moved to the Bay Area! Excited to work with @rsalakhu and the team @AIatMeta on agents 🦙
-
Russ Salakhutdinov reposted
Shuyan Zhou
@shuyanzh36
I recently moved to the Bay Area! Excited to work with @rsalakhu and the team @AIatMeta on agents 🦙
-
Russ Salakhutdinov
@rsalakhu
It is exciting to see CMU School of Computer Science @SCSatCMU highlighting #eccv2024 work of @SoYeonTiffMin on Situated Instruction Following -- Teaching AI Agents To Navigate Ambiguous Instructions:
www.cs.cmu.edu/news/2024/robots-ambiguous-instructions
This work generalizes Embodied Instruction Following (EIF) to situated language -- the natural language spoken by humans -- focusing on instructions that (1) are ambiguously specified, (2) have temporally evolving intent, (3) can be interpreted more precisely with the agent's dynamic actions.
This works also shows that the current state-of-the-art EIF models lack holistic understanding of situated human intention.
Paper and Code: soyeonm.github.io/SIF_webpage/
Joint work with @AIatMeta. -
Russ Salakhutdinov
@rsalakhu
Causal Inference under Incentives: An Annotated Reading List – Machine Learning Blog | ML@CMU | Carnegie Mellon University blog.ml.cmu.edu/2024/08/22/causal-inference-under-incenti…
-
Russ Salakhutdinov reposted
Jing Yu Koh
@kohjingyu
Unfortunately, I couldn't make it to ACL in person this year, but I'll be presenting VisualWebArena at virtual poster session #2 next Thurs:
⏰ August 22, 10.15am-11am (8.15pm-9pm PST)
📝 arxiv.org/abs/2401.13649
🌐 jykoh.com/vwa
Come and chat about multimodal agents!
Jing Yu Koh
@kohjingyu
Computer interfaces are inherently visual. To build general autonomous agents, we will need strong vision language models.
To assess the performance of multimodal agents, we introduce VisualWebArena (VWA): a benchmark for evaluating multimodal web agents on realistic visually grounded tasks.
🔗 Website: jykoh.com/vwa
📄 Paper: arxiv.org/abs/2401.13649
🤖 Code and data: github.com/web-arena-x/visualwebarena
🧵👇 -
Russ Salakhutdinov reposted
Ahmad Al-Dahle
@Ahmad_Al_Dahle
We’re opening a call for applications for the next round of Llama Impact Grants. The program supports organizations pursuing ideas to use Llama to address social challenges in their communities with $2M in grants available. Full details + criteria: ai.meta.com/blog/llama-3-1-impact-grants-call-for-applica…
-
Russ Salakhutdinov reposted
ICML Conference
@icmlconf
From the organizing committee and everyone involved we hope that you had an amazing time at ICML
-
Russ Salakhutdinov
@rsalakhu
"Proud Advisor Moment": Congratulations to Paul Liang @pliang279 on joining MIT as an assistant professor @MITEECS and @medialab. If you are interested in working on the next generation Multimodal Multisensory AI, I would strongly consider joining Paul’s new lab at MIT.
Paul Liang
@pliang279
📣 I'm thrilled to share that I’ll be joining MIT as an assistant professor this fall, joint between @medialab & @MITEECS.
My group will advance the foundations of multisensory AI to enhance the human experience.
I look forward to tackling exciting challenges in multimodal AI and HCI together with my new colleagues. Come join us - I’ll be hiring at all levels! -
Russ Salakhutdinov
@rsalakhu
CMU-MATH Team’s Innovative Approach Secures 2nd Place at the AIMO Prize – Machine Learning Blog | ML@CMU | Carnegie Mellon University blog.ml.cmu.edu/2024/07/29/cmu-math-teams-innovative-appr…
-
Russ Salakhutdinov
@rsalakhu
Please see an explanation by @zicokolter as to why ICML tutorial video was initially taken down and is now restored.
Zico Kolter
@zicokolter
Ok, I understand the issue now. We were able to restore the video for now, but the video you posted used a combination of your own recording plus the SlidesLive stream from ICML. The ICML contracts require that direct video recordings from the SlidesLive stream not be publicly available until 30 days after the event, hence the copyright claim. I'm going to ask that the video remain up for now, but I think it would be good to temporarily remove the video portion (I think not critical to the video) until August 20th, per ICML contracts. Thanks for your help.
-
Russ Salakhutdinov reposted
Zico Kolter
@zicokolter
Ok, I understand the issue now. We were able to restore the video for now, but the video you posted used a combination of your own recording plus the SlidesLive stream from ICML. The ICML contracts require that direct video recordings from the SlidesLive stream not be publicly available until 30 days after the event, hence the copyright claim. I'm going to ask that the video remain up for now, but I think it would be good to temporarily remove the video portion (I think not critical to the video) until August 20th, per ICML contracts. Thanks for your help.
-
Russ Salakhutdinov reposted
Kamalika Chaudhuri
@kamalikac
Machine learning models must be audited for fairness, robustness and accuracy, yet many are confidential. How can we audit a confidential model for fairness while still retaining confidentiality? We show how to do this in our #icml2024 paper.
-
Russ Salakhutdinov
@rsalakhu
A few people are asking me why work with @AIatMeta?
Two words: Open Source.
Ahmad Al-Dahle
@Ahmad_Al_Dahle
With today’s launch of our Llama 3.1 collection of models we’re making history with the largest and most capable open source AI model ever released. 128K context length, multilingual support, and new safety tools. Download 405B and our improved 8B & 70B here. llama.meta.com
-
Russ Salakhutdinov reposted
Ahmad Al-Dahle
@Ahmad_Al_Dahle
Llama 3.1 sets a new standard for open source AI. The 8B, 70B, and 405B delivers performance comparable to the most advanced closed models. Here’s a look at how the performance stacks up. You can dive deeper in the full announcement and research paper available here:
-
Russ Salakhutdinov reposted
Ahmad Al-Dahle
@Ahmad_Al_Dahle
With today’s launch of our Llama 3.1 collection of models we’re making history with the largest and most capable open source AI model ever released. 128K context length, multilingual support, and new safety tools. Download 405B and our improved 8B & 70B here. llama.meta.com
-
Russ Salakhutdinov
@rsalakhu
ICML 2024: Opening Remarks (1/n)
ICML 2024 in Numbers:
· 9 EXPO Talk Panels
· 12 Tutorials
· 6 Invited Talks
· 2,610 Papers (main conference)
· 30 Workshops
· 12345 authors and presenters
· 39% students
· 10 in-person socials
· 3 affinity events
· 52 volunteers
· 97 SACs, 492 ACs, 7,473 reviewers
· 9406 registered participants (8675 in person)Post not available here, view on X
Russ Salakhutdinov
@rsalakhu
ICML 2024: Best Paper Awards:
Florian Tramèr; Gautam Kamath; Nicholas Carlini: Considerations for Differentially Private Learning with Large-Scale Public Pretraining
Akbir Khan; John Hughes; Dan Valentine; Laura Ruis; Kshitij Sachan; Ansh Radhakrishnan; Edward Grefenstette; Samuel R. Bowman; Tim Rocktäschel; Ethan Perez. Debating with More Persuasive LLMs Leads to More Truthful Answers
Jake Bruce; Michael D Dennis; Ashley Edwards; Jack Parker-Holder; Yuge Shi; Edward Hughes; Matthew Lai; Aditi Mavalankar; Richie Steigerwald; Chris Apps; Yusuf Aytar; Sarah Maria Elisabeth Bechtle; Feryal Behbahani; Stephanie C.Y. Chan; Nicolas Heess; Lucy Gonzalez; Simon Osindero; Sherjil Ozair; Scott Reed; Jingwei Zhang; Konrad Zolna; Jeff Clune; Nando de Freitas; Satinder Singh; Tim Rocktäschel. Genie: Generative Interactive Environments
Dan Kondratyuk; Lijun Yu; Xiuye Gu; Jose Lezama; Jonathan Huang; Grant Schindler; Rachel Hornung; Vighnesh Birodkar; Jimmy Yan; Ming-Chang Chiu; Krishna Somandepalli; Hassan Akbari; Yair Alon; Yong Cheng; Joshua V. Dillon; Agrim Gupta; Meera Hahn; Anja Hauth; David Hendon; Alonso Martinez; David Minnen; Mikhail Sirotenko; Kihyuk Sohn; Xuan Yang; Hartwig Adam; Ming-Hsuan Yang; Irfan Essa; Huisheng Wang; David A Ross; Bryan Seybold; Lu Jiang. VideoPoet: A Large Language Model for Zero-Shot Video Generation
Aaron Lou; Chenlin Meng; Stefano Ermon. Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution
Patrick Esser; Sumith Kulal; Andreas Blattmann; Rahim Entezari; Jonas Müller; Harry Saini; Yam Levi; Dominik Lorenz; Axel Sauer; Frederic Boesel; Dustin Podell; Tim Dockhorn; Zion English; Robin Rombach. Scaling Rectified Flow Transformers for High-Resolution Image Synthesis
Stephen Zhao; Rob Brekelmans; Alireza Makhzani; Roger Baker Grosse. Probabilistic Inference in Language Models via Twisted Sequential Monte Carlo
Dora Zhao; Jerone Andrews; Orestis Papakyriakopoulos; Alice Xiang. Measure Dataset Diversity, Don’t Just Claim It.
Nicholas Carlini; Daniel Paleka; Krishnamurthy Dj Dvijotham; Thomas Steinke; Jonathan Hayase; A. Feder Cooper; Katherine Lee; Matthew Jagielski; Milad Nasr; Arthur Conmy; Eric Wallace; David Rolnick; Florian Tramèr. Stealing Part of a Production Language Model
Idan Attias; Gintare Karolina Dziugaite; Mahdi Haghifam; Roi Livni; Daniel M. Roy. Information Complexity of Stochastic Convex Optimization: Applications to Generalization and Memorization
Russ Salakhutdinov
@rsalakhu
Test-of-time Award ICML 2024.
DeCAF: a deep convolutional activation feature for generic visual recognition
Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor Darrell -
Russ Salakhutdinov
@rsalakhu
Test-of-time Award ICML 2024.
DeCAF: a deep convolutional activation feature for generic visual recognition
Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor Darrell -
Russ Salakhutdinov
@rsalakhu
ICML 2024: Best Paper Awards:
Florian Tramèr; Gautam Kamath; Nicholas Carlini. Position: Considerations for Differentially Private Learning with Large-Scale Public Pretraining
Akbir Khan; John Hughes; Dan Valentine; Laura Ruis; Kshitij Sachan; Ansh Radhakrishnan; Edward Grefenstette; Samuel R. Bowman; Tim Rocktäschel; Ethan Perez. Debating with More Persuasive LLMs Leads to More Truthful Answers
Jake Bruce; Michael D Dennis; Ashley Edwards; Jack Parker-Holder; Yuge Shi; Edward Hughes; Matthew Lai; Aditi Mavalankar; Richie Steigerwald; Chris Apps; Yusuf Aytar; Sarah Maria Elisabeth Bechtle; Feryal Behbahani; Stephanie C.Y. Chan; Nicolas Heess; Lucy Gonzalez; Simon Osindero; Sherjil Ozair; Scott Reed; Jingwei Zhang; Konrad Zolna; Jeff Clune; Nando de Freitas; Satinder Singh; Tim Rocktäschel. Genie: Generative Interactive Environments
Dan Kondratyuk; Lijun Yu; Xiuye Gu; Jose Lezama; Jonathan Huang; Grant Schindler; Rachel Hornung; Vighnesh Birodkar; Jimmy Yan; Ming-Chang Chiu; Krishna Somandepalli; Hassan Akbari; Yair Alon; Yong Cheng; Joshua V. Dillon; Agrim Gupta; Meera Hahn; Anja Hauth; David Hendon; Alonso Martinez; David Minnen; Mikhail Sirotenko; Kihyuk Sohn; Xuan Yang; Hartwig Adam; Ming-Hsuan Yang; Irfan Essa; Huisheng Wang; David A Ross; Bryan Seybold; Lu Jiang. VideoPoet: A Large Language Model for Zero-Shot Video Generation
Aaron Lou; Chenlin Meng; Stefano Ermon. Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution
Patrick Esser; Sumith Kulal; Andreas Blattmann; Rahim Entezari; Jonas Müller; Harry Saini; Yam Levi; Dominik Lorenz; Axel Sauer; Frederic Boesel; Dustin Podell; Tim Dockhorn; Zion English; Robin Rombach. Scaling Rectified Flow Transformers for High-Resolution Image Synthesis
Stephen Zhao; Rob Brekelmans; Alireza Makhzani; Roger Baker Grosse. Probabilistic Inference in Language Models via Twisted Sequential Monte Carlo
Dora Zhao; Jerone Andrews; Orestis Papakyriakopoulos; Alice Xiang. Measure Dataset Diversity, Don’t Just Claim It.
Nicholas Carlini; Daniel Paleka; Krishnamurthy Dj Dvijotham; Thomas Steinke; Jonathan Hayase; A. Feder Cooper; Katherine Lee; Matthew Jagielski; Milad Nasr; Arthur Conmy; Eric Wallace; David Rolnick; Florian Tramèr. Stealing Part of a Production Language Model
Idan Attias; Gintare Karolina Dziugaite; Mahdi Haghifam; Roi Livni; Daniel M. Roy. Information Complexity of Stochastic Convex Optimization: Applications to Generalization and Memorization -
Russ Salakhutdinov reposted
Zico Kolter
@zicokolter
Welcome to @icmlconf 2024! The PC (minus @nuriaoliver and @adrian_weller) are here and excited to kick things off! @kat_heller @rsalakhu
-
Russ Salakhutdinov
@rsalakhu
Very excited to be in Vienna at #icml2024 as a general chair -- with Program Chairs Katherine Heller @kat_heller and Zico Kolter @zicokolter (minus Nuria Oliver @nuriaoliver and Adrian Weller @adrian_weller).
I would like thank PCs, and the entire program committee and staff, for their invaluable contributions to making this a successful conference.
It's often hard to see how much time, effort, and dedication goes into every aspect behind the scenes: from assembling the full program committee and following up with reviewers and area chairs, to meticulously evaluating final decisions of every submitted paper (close to 10,000 of them!), and crafting the full conference schedule while coordinating seamlessly with various program committee members.
So thank you Katherine, Nuria, Zico, and Adrian for making this happen! -
Russ Salakhutdinov reposted
Russ Salakhutdinov
@rsalakhu
Check out our new #eccv2024 work, led by @SoYeonTiffMin, on Situated Instruction Following:
Paper: arxiv.org/abs/2407.12061
Code: soyeonm.github.io/SIF_webpage/
In traditional Embodied Instruction Following (EIF), the agent typically acts alone in a static environment, leading to simplified settings. This work aims to generalize Embodied Instruction Following to Situated Instruction Following (SIF), with instructions that are closer to the language naturally spoken by humans.
Specifically, the focus is on three dimensions of situated reasoning:
Ambiguity: There is an ambiguity in the instruction given by the speaker
Temporal: A speaker’s actions change how their instruction should be interpreted
Dynamic: When the environment changes, the agent needs to decide what actions will reduce their uncertainty (e.g., following the human)
Experiments show that the current state-of-the-art Embodied Instruction Following (EIF) models lack holistic understanding of situated human intention.
With @SoYeonTiffMin, @xavierpuigf, @dchaplot, @JimmyTYYang1, Akshara Rai, @priyam8parashar, @ybisk, @RoozbehMottaghi
Check out an excellent thread by @SoYeonTiffMin.
So Yeon (Tiffany) Min
@SoYeonTiffMin
We introduce Situated Instruction Following (SIF), to appear in ECCV 2024! There is inherent underspecification in instructions when humans act as they speak. SIF is addresses these dynamic, temporally evolving intent of instructions in the context of physical human actions.(1/7)
-
Russ Salakhutdinov
@rsalakhu
Check out our new #eccv2024 work, led by @SoYeonTiffMin, on Situated Instruction Following:
Paper: arxiv.org/abs/2407.12061
Code: soyeonm.github.io/SIF_webpage/
In traditional Embodied Instruction Following (EIF), the agent typically acts alone in a static environment, leading to simplified settings. This work aims to generalize Embodied Instruction Following to Situated Instruction Following (SIF), with instructions that are closer to the language naturally spoken by humans.
Specifically, the focus is on three dimensions of situated reasoning:
Ambiguity: There is an ambiguity in the instruction given by the speaker
Temporal: A speaker’s actions change how their instruction should be interpreted
Dynamic: When the environment changes, the agent needs to decide what actions will reduce their uncertainty (e.g., following the human)
Experiments show that the current state-of-the-art Embodied Instruction Following (EIF) models lack holistic understanding of situated human intention.
With @SoYeonTiffMin, @xavierpuigf, @dchaplot, @JimmyTYYang1, Akshara Rai, @priyam8parashar, @ybisk, @RoozbehMottaghi
Check out an excellent thread by @SoYeonTiffMin.
So Yeon (Tiffany) Min
@SoYeonTiffMin
We introduce Situated Instruction Following (SIF), to appear in ECCV 2024! There is inherent underspecification in instructions when humans act as they speak. SIF is addresses these dynamic, temporally evolving intent of instructions in the context of physical human actions.(1/7)
-
Russ Salakhutdinov reposted
Paul Liang
@pliang279
Excited to release HEMM (Holistic Evaluation of Multimodal Foundation Models), the largest and most comprehensive evaluation for multimodal models like Gemini, GPT-4V, BLIP-2, OpenFlamingo, and more.
HEMM contains 30 datasets carefully selected and categorized based on:
1. The **basic multimodal skills** needed to solve them – the type of multimodal interaction, granularity of multimodal alignment, level of reasoning, and need for external knowledge,
2. How **information flows** between modalities – querying, translation, editing, and fusion,
3. The real-world **use cases** they impact – multimedia, affective computing, healthcare, science & environment, HCI.
paper: arxiv.org/abs/2407.03418
code: github.com/pliang279/HEMM
we encourage the community to add their favorite models and datasets!
w @AkshayGoindani1 @TalhaChafekar @lmathur_ @haofeiyu44 @lpmorency @rsalakhu @mldcmu @LTIatCMU -
Russ Salakhutdinov
@rsalakhu
New work on Holistic Evaluation of Multimodal Foundation Models:
Paper: arxiv.org/abs/2407.03418
Code: github.com/pliang279/HEMM
Systematically evaluate capabilities of multimodal models covering 3 dimensions:
Basic multimodal skills: learning interactions across modalities, fine-grained alignment, multi-step reasoning, handling external knowledge
Information flow: how multimodal content changes during a task through querying, translation, editing, and fusion
Use cases: multimedia, affective computing, natural sciences, healthcare, human-computer interaction applications
With @pliang279 , Akshay Goindani, Talha Chafekar, Leena Mathur, Haofei Yu, Louis-Philippe Morency. -
Russ Salakhutdinov reposted
Tanishq Mathew Abraham, Ph.D.
@iScienceLuvr
HEMM: Holistic Evaluation of Multimodal Foundation Models
abs: arxiv.org/abs/2407.03418
code: github.com/pliang279/HEMM
Carnegie Mellon University researchers introduce HEMM, a new benchmarking framework to systematically evaluate the capabilities of multimodal foundation models across a set of 3 comprehensive dimensions: basic skills, information flow, and real-world use cases.
Gemini achieves best average performance across all tasks. -
Russ Salakhutdinov reposted
Kamalika Chaudhuri
@kamalikac
Representation learning is a major cornerstone of computer vision. But how far can we get with differentially private representation learning? In our two papers appearing in @icml_conf, we train the first differentially private representation learning models on web-scale data.
-
Russ Salakhutdinov reposted
Ahmad Al-Dahle
@Ahmad_Al_Dahle
I am thrilled to welcome @rsalakhu to @AIatMeta as our newest VP of Research in GenAI! I’ve known Russ for a number of years, including our time together at Apple and couldn’t be more excited for what’s ahead as he helps us deliver on some of our big bets for AI agents.
Russ Salakhutdinov
@rsalakhu
I am very excited to start working with GenAI team at @Meta, focusing on multimodal LLM agents, joining together with my amazing CMU colleagues Jing Yu Koh @kohjingyu and Daniel Fried @dan_fried!
-
Russ Salakhutdinov
@rsalakhu
Check out our new #eccv2024 work on “OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Web”.
This work introduces a dataset for assessing an agent's ability to generate executable programs to accomplish tasks, such as “Play the next song”, or “Send an email to John Doe mentioning the time and place to meet”.
Paper: arxiv.org/abs/2402.17553
Code and Dataset: huggingface.co/datasets/Writer/omniact/tree/main
With @raghav_kapoor16, Yash Parag Butala, @MelisaStal, @kohjingyu, Kiran Kamble, @waseem_s
Raghav Kapoor
@raghav_kapoor16
Excited to announce our paper titled: "OmniACT: A dataset and benchmark for enabling multimodal generalist autonomous agents for desktop and web" has been accepted to #eccv2024 🇮🇹
-
Russ Salakhutdinov reposted
Daniel Fried
@dan_fried
I'm excited to be working on agents at @Meta, along with continuing my work at CMU!
Russ Salakhutdinov
@rsalakhu
I am very excited to start working with GenAI team at @Meta, focusing on multimodal LLM agents, joining together with my amazing CMU colleagues Jing Yu Koh @kohjingyu and Daniel Fried @dan_fried!
-
Russ Salakhutdinov reposted
AI at Meta
@AIatMeta
Exciting news — welcome to the team, Russ! 🎉
-
Russ Salakhutdinov
@rsalakhu
Thank you @manohar_paluri! We are all thrilled to be part of the amazing Llama team and push the frontier of agentic capabilities!
Manohar Paluri
@manohar_paluri
I am excited to have Russ, Daniel and Jing Yu join the Llama team to work on Multimodal agents. We have to tackle many open research problems to enable agentic capabilities. Looking forward to working together. Welcome to the Llama family
Russ Salakhutdinov
@rsalakhu
I am very excited to start working with GenAI team at @Meta, focusing on multimodal LLM agents, joining together with my amazing CMU colleagues Jing Yu Koh @kohjingyu and Daniel Fried @dan_fried!
-
Russ Salakhutdinov
@rsalakhu
Thank you @Ahmad_Al_Dahle. I am thrilled to be part of this amazing team.
Ahmad Al-Dahle
@Ahmad_Al_Dahle
I am thrilled to welcome @rsalakhu to @AIatMeta as our newest VP of Research in GenAI! I’ve known Russ for a number of years, including our time together at Apple and couldn’t be more excited for what’s ahead as he helps us deliver on some of our big bets for AI agents.
Russ Salakhutdinov
@rsalakhu
I am very excited to start working with GenAI team at @Meta, focusing on multimodal LLM agents, joining together with my amazing CMU colleagues Jing Yu Koh @kohjingyu and Daniel Fried @dan_fried!
-
Russ Salakhutdinov reposted
Jing Yu Koh
@kohjingyu
Exciting personal update: I'm taking a leave of absence from my PhD to join the GenAI team at @AIatMeta. I'm super excited to continue working with my advisors @rsalakhu and @dan_fried, and join forces with the amazing Llama team to build multimodal LLM agents! 🦙+🤖
Russ Salakhutdinov
@rsalakhu
I am very excited to start working with GenAI team at @Meta, focusing on multimodal LLM agents, joining together with my amazing CMU colleagues Jing Yu Koh @kohjingyu and Daniel Fried @dan_fried!
-
Russ Salakhutdinov
@rsalakhu
I am very excited to start working with GenAI team at @Meta, focusing on multimodal LLM agents, joining together with my amazing CMU colleagues Jing Yu Koh @kohjingyu and Daniel Fried @dan_fried!
-
Russ Salakhutdinov reposted
Jing Yu Koh
@kohjingyu
🌳Tree Search for Language Model Agents is now on arXiv too!
Paper: arxiv.org/abs/2407.01476
Code: github.com/kohjingyu/search-agents
Project page: jykoh.com/search-agents
Jing Yu Koh
@kohjingyu
LLM agents have demonstrated promise in their ability to automate computer tasks, but face challenges with multi-step reasoning and planning. Towards addressing this, we propose an inference-time tree search algorithm for LLM agents to explicitly perform exploration and multi-step planning in interactive web environments.
It is the first tree search algorithm for LLM agents that shows effectiveness on realistic and complex web environments: on the challenging VisualWebArena benchmark, applying our search algorithm on top of a GPT-4o agent yields a 39.7% relative increase in success rate compared to the same baseline without search, setting a state-of-the-art success rate of 26.4%. On WebArena, search also yields a 28.0% relative improvement over a baseline agent, setting a competitive success rate of 19.2%. -
Russ Salakhutdinov
@rsalakhu
Check out our new work on “Adversarial Attacks on Multimodal Agents” that shows that one can manipulate the behaviour of multimodal agents.
For example, one can make an agent believe that it is in a different state or make an agent pursue a targeted goal that is different from the original user-specified goal – e.g. buying an orange motorcycle, whereas the user asked for a blue one.
Paper + Code: chenwu.io/attack-agent/
See an excellent thread by @ChenHenryWu for more details.
Chen Wu
@ChenHenryWu
GPT-4V/Gemini is now used to build autonomous multimodal agents 🤖, but can you trust them to shop for you? Our latest work reveals new safety risks in these agents. We show that attackers can manipulate agents to repeatedly execute targeted adversarial goals (e.g., always choosing their products) via tiny tweaks to just a single image, achieving up to 75% attack success! 1/🧵
Example in this video: by tweaking their product image, the attacker manipulates the agent to choose their product (an orange motorcycle), though the user asked for a blue one. -
Russ Salakhutdinov
@rsalakhu
Check out our new work on “Tree Search for Language Model Agents” – an inference-time search algorithm that can explicitly perform exploration and multi-step planning in interactive web environments.
On challenging WebArena and Multimodal VisualWebArena benchmarks, these agents show substantial improvement over baseline agents without search.
Code + Paper: jykoh.com/search-agents
See this excellent thread by @kohjingyu
Jing Yu Koh
@kohjingyu
LLM agents have demonstrated promise in their ability to automate computer tasks, but face challenges with multi-step reasoning and planning. Towards addressing this, we propose an inference-time tree search algorithm for LLM agents to explicitly perform exploration and multi-step planning in interactive web environments.
It is the first tree search algorithm for LLM agents that shows effectiveness on realistic and complex web environments: on the challenging VisualWebArena benchmark, applying our search algorithm on top of a GPT-4o agent yields a 39.7% relative increase in success rate compared to the same baseline without search, setting a state-of-the-art success rate of 26.4%. On WebArena, search also yields a 28.0% relative improvement over a baseline agent, setting a competitive success rate of 19.2%. -
Russ Salakhutdinov reposted
Jack Langerman (in sf)
@jacklangerman
Some nice examples using stable diffusion + textual inversion for data augmentation in the few shot setting from @rsalakhu at the syndata4cv workshop
-
Russ Salakhutdinov reposted
ICML Conference
@icmlconf
Just a reminder, early registration ends on June 15th.
-
Russ Salakhutdinov
@rsalakhu
Very exciting to see that open sourced Llama-3-70B-Instruct performs so well on the multimodal agentic benchmark Visual Web Arena (VWA), despite being only a text-based model, even beating multimodal Gemini-Flash-1.5 and coming very close to the performance of Gemini-Pro-1.5.
Multimodal GPT-4o performs the best, but Llama-3-70B-Instruct’s performance of 9.78% is pretty close to the text-based version of GPT-4’s performance of 12.78%.
Paper + Code: jykoh.com/vwa
Jing Yu Koh
@kohjingyu
Happy to share that VisualWebArena will be presented at ACL 2024 in August, and we've updated the paper with some new results. We ran several recently released models (Llama-3-70B-Instruct, Gemini-Flash-1.5, Gemini-Pro-1.5, GPT-4o), with some interesting findings:
1) Llama-3-70B-Instruct is really good on VWA despite not being multimodal. It's by far the best open sourced model for VWA, beating even multimodal models. Its success rate of 9.78% creeps close to the text-based GPT-4 model (SR = 12.78%). I'm really excited for building things on top of this model.
2) Gemini-1.5 is a lot better than Gemini-1.0! Flash-1.5 is even better than Gemini-Pro-1.0. It's also really fast for evaluation purposes, and seems like it'd be a great model to use for development on VWA. I'm really excited to see what agent capabilities we can do at the cost/performance ratio of Flash-1.5. (thank you @ankesh_anand for helping me to set up and debug the API!)
3) GPT-4o (success rate of 19.78%) is substantially better than GPT-4V (16.37%), which is interesting. VWA is a multimodal benchmark, so it's likely that the (rumored) native multimodality helps here. @shuyanzhxyc and @gneubig have also reported that GPT-4o is better on agentic tasks, so this result also seems really promising (especially since 4o is cheaper and faster).
Despite the promising progress, all existing models are still far below human performance. On VWA, humans achieve a success rate of 89%, so there is still a ton of headroom!
Jing Yu Koh
@kohjingyu
Computer interfaces are inherently visual. To build general autonomous agents, we will need strong vision language models.
To assess the performance of multimodal agents, we introduce VisualWebArena (VWA): a benchmark for evaluating multimodal web agents on realistic visually grounded tasks.
🔗 Website: jykoh.com/vwa
📄 Paper: arxiv.org/abs/2401.13649
🤖 Code and data: github.com/web-arena-x/visualwebarena
🧵👇 -
Russ Salakhutdinov
@rsalakhu
Ouch @RogerGrosse, that hurts — At least give us a fancier title, like a chief scientist, we deserve it 😂.
Roger Grosse
@RogerGrosse
The difference between a Canadian and American AI professor is like the difference between a tech lead and a middle manager.
-
Russ Salakhutdinov
@rsalakhu
#iclr2024: Kelly He @electronickale is presenting her work on Manifold Preserving Guided Diffusion:
Paper + Code: kellyyutonghe.github.io/mpgd/
with Naoki Murata, @JCJesseLai, @takiko_san, @kamitsutoshi, @gimdong58085414, Wei-Hsiang Liao, @mittu1204, @zicokolter, @StefanoErmon.
Check out Kelly's excellent thread below:
Yutong (Kelly) He
@electronickale
🧠Tired of text-only image generation control? Don’t have resources to train for your own tasks? Annoying long sampling time?
📣 Introducing Manifold Preserving Guided Diffusion (MPGD) to solve all these problems! Learn more: kellyyutonghe.github.io/mpgd/ and in 🧵👇
#generativeai -
Russ Salakhutdinov
@rsalakhu
#iclr2024: JY Koh @kohjingyu is presenting our work on VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks:
Paper + Code: jykoh.com/vwa
with @rclo8103, Lawrence Jang, Vikram Duvvur,
@mingchonglim, Po-Yu Huang, @gneubig, @shuyanzhxyc, @dan_fried.
Jing Yu Koh
@kohjingyu
Tomorrow I'll be presenting the VisualWebArena poster at two ICLR workshops:
⏰ 10am-11am at the GenAI4DM Workshop (in Lehar 3)
⏰ 11.20am-12.15pm / 2.30pm-3.30pm at the LLMAgents Workshop (in Halle A 8-9)
Come say hi and talk about multimodal agents! -
Russ Salakhutdinov
@rsalakhu
#iclr2024: Murtaza Dala @mihdalal is presenting our work on Plan-Seq-Learn: Language Model Guided RL for Solving Long Horizon Robotics Tasks.
Paper + Code: mihdalal.github.io/planseqlearn/
See an excellent thread by @mihdalal below:
Murtaza Dalal @ CoRL 2026
@mihdalal
LLMs are capable of high-level planning, but they require pre-trained skills! Our #iclr2024 paper instead uses LLM guidance to train RL agents from scratch to solve 25+ long-horizon robotics tasks across four benchmarks w/ >85% success rates
Paper & code: mihdalal.github.io/planseqlearn -
Russ Salakhutdinov
@rsalakhu
#iclr2024: Chongyi Zheng @chongyiz1 and Ben Eysenbach @ben_eysenbach are presenting their work on Stabilizing Contrastive RL: Techniques for Robotic Goal Reaching from Offline Data
Paper + Code: chongyi-zheng.github.io/stable_contrastive_rl/
with @HomerWalke, @patrickhyin, @KuanFang, @svlevine.
See an excellent thread by Chongyi below:
Chongyi Zheng
@chongyiz1
1/6 Self-supervised learning is wildly successful for CV, NLP; might goal-conditioned RL enable self-supervised RL? In months of experiments, we found design decisions that significantly boost success, acquire intriguing representations, and solve real robotic tasks from images.
-
Russ Salakhutdinov reposted
Ben Eysenbach
@ben_eysenbach
Self-supervised RL makes it easier for users to teach robots: after learning autonomously (no demos), you prompt with a goal image (no reward shaping). @chongyiz1 's poster tomorrow explains how!
x.com/chongyiz1/status/1666248365628174336
w/ @HomerWalke @patrickhyin @KuanFang @rsalakhu @svlevine
Chongyi Zheng
@chongyiz1
1/6 Self-supervised learning is wildly successful for CV, NLP; might goal-conditioned RL enable self-supervised RL? In months of experiments, we found design decisions that significantly boost success, acquire intriguing representations, and solve real robotic tasks from images.
-
Russ Salakhutdinov
@rsalakhu
#iclr2024: @McaleerStephen is presenting our work on Confronting Reward Model Overoptimization with Constrained RLHF.
Paper: arxiv.org/abs/2310.04373
with @ted_moskovitz, @Aaditya6284, @djstrouse, Tuomas Sandholm, @ancadianadragan.
@ted_moskovitz could not make it to the conference, but check out his excellent thread below:
Ted Moskovitz
@ted_moskovitz
Worried your LLM will produce too many paperclips? Simply tell it when to stop – excited to share our new preprint, where we introduce an approach based on constrained RL to avoid overoptimization for compound reward models:
arxiv.org/abs/2310.04373
1/ -
Russ Salakhutdinov
@rsalakhu
#iclr2024: Paul Liang @pliang279 is presenting our work on Multimodal Learning Without Labeled Multimodal Data: Guarantees and Applications.
Paper: arxiv.org/abs/2306.04539
Code: github.com/pliang279/PID
with Chun Kai Ling, Yun Cheng, Alex Obolenskiy, Yudong Liu, Rohan Pandey, Alex Wilf, and Louis-Philippe Morency -
Russ Salakhutdinov
@rsalakhu
#iclr2024: @chongyiz1 is presenting his work on Contrastive Difference Predictive Coding.
Check out paper + code: chongyi-zheng.github.io/td_infonce/
with @ben_eysenbach.
Chongyi Zheng
@chongyiz1
1/7 Can we use InfoNCE to learn representations that predict future events? Excited to share our new paper Contrastive Difference Predictive Coding: chongyi-zheng.github.io/td_infonce, a temporal difference InfoNCE loss up to 1500x more sample efficient than the standard (MC) version.
-
Russ Salakhutdinov
@rsalakhu
#iclr2024: @brandontrabucco and Max Gurinas @kwik_sand are presenting our work on Effective Data Augmentation With Diffusion Models.
Check out paper: arxiv.org/abs/2302.07944
Code: github.com/brandontrabucco/da-fusion?tab=readme-ov-file
Brandon Trabucco
@brandontrabucco
Can #stablediffusion augment our training data?
I'm at #iclr2024 this week, presenting DA-Fusion, an #ai tool for data augmentation that adapts to your datasets.
Looking forward to meeting all the cool researchers here!
Hall B #45 Tues. 10:45 am
Paper: arxiv.org/abs/2302.07944 -
Russ Salakhutdinov
@rsalakhu
At ICLR this week, looking forward to catching up with everyone.
-
Russ Salakhutdinov reposted
Ben Eysenbach
@ben_eysenbach
Check out @chongyiz1 's #iclr24 poster on a temporal difference method for contrastive learning tomorrow in Hall B #174!
I find the method aesthetically beautiful and it works quite well in practice. Original tweet below:
Chongyi Zheng
@chongyiz1
1/7 Can we use InfoNCE to learn representations that predict future events? Excited to share our new paper Contrastive Difference Predictive Coding: chongyi-zheng.github.io/td_infonce, a temporal difference InfoNCE loss up to 1500x more sample efficient than the standard (MC) version.
-
Russ Salakhutdinov reposted
Zico Kolter
@zicokolter
ICML 2024 decisions are now released on OpenReview. A genuine thank you to all the authors, reviewers, ACs, SACs, and co-PCs @kat_heller @adrian_weller @nuriaoliver and GC @rsalakhu.
-
Russ Salakhutdinov reposted
Michael Luo
@michaelzluo
[1/5] Introducing Stylus 🖌️ - an #ai tool that automatically finds and adds the best adapters (LoRAs, Textual Inversions, Hypernetworks) to #stablediffusion based on your prompt.
🗞️ Paper: arxiv.org/abs/2404.18928
🌎 Project Page: stylus-diffusion.github.io -
Russ Salakhutdinov reposted
Ahmad Al-Dahle
@Ahmad_Al_Dahle
It’s here! Meet Llama 3, our latest generation of models that is setting a new standard for state-of-the art performance and efficiency for openly available LLMs.
Key highlights
• 8B and 70B parameter openly available pre-trained and fine-tuned models.
• Trained on more than 15T tokens, 7x+ larger than Llama 2's dataset!
• Improved tokenizer with vocabulary of 128K tokens for better performance.
• State-of-the-art performance across industry benchmarks.
• New capabilities, including enhanced reasoning and coding.
• 3x more efficient training than Llama 2.
• New trust and safety tools with Llama Guard 2, Code Shield, and CyberSec Eval 2.
• Integrated into Meta AI, and available in more countries across our apps.
• And, just the beginning with more models and new capabilities coming soon!
Visit the Llama 3 website to read more and download the models. llama.meta.com/llama3 -
Russ Salakhutdinov reposted
Lenore Blum
@BlumLenore
Yes Paul (@pliang279), that bubbly is yours! 🥳🎉Congratulations on your very successful dissertation defense! (on the "Foundations of Multisensory Artificial Intelligence" in the @mldcmu ,@SCSatCMU, @CarnegieMellon ).
-
Russ Salakhutdinov reposted
Jing Yu Koh
@kohjingyu
Honored to receive the 2024 Jane Street Graduate Research Fellowship! Thank you @JaneStreetGroup for the award and for organizing an amazing workshop!
The best part of this was getting to meet PhD students working on algebraic geometry, cosmology, quantum algorithms, and more! -
Russ Salakhutdinov reposted
Ahmad Al-Dahle
@Ahmad_Al_Dahle
I’ll be sharing more on Llama 3 very soon. It’s so cool to see what the community is already building with Llama 2 though. One of my favorites: @team_qanda & @UpstageAI used it to build a math-specific LLM to make personalized learning more accessible! ai.meta.com/blog/llama-2-mathgpt-mathpresso-qanda-upstage…
-
Russ Salakhutdinov
@rsalakhu
NLPositionality: Characterizing Design Biases of Datasets and Models – Machine Learning Blog | ML@CMU | Carnegie Mellon University blog.ml.cmu.edu/2024/03/01/nlpositionality-characterizing…
-
Russ Salakhutdinov
@rsalakhu
Beyond the Mud: Datasets, Benchmarks, and Methods for Computer Vision in Off-Road Racing – Machine Learning Blog | ML@CMU | Carnegie Mellon University blog.ml.cmu.edu/2024/03/22/beyond-the-mud-datasets-benchm…
-
Russ Salakhutdinov reposted
Mohammad Hajiaghayi (IG@mhajiaghayi,YT:hajiaghayi)
@MTHajiaghayi
Very proud to be part of this Amazing MURI Team with Christos Papadimitriou @Columbia, Erik Demaine, @KonstDaskalakis, @aleks_madry @MIT, @zicokolter , @rsalakhu, Katia Sycara @CMU, Kamalika Chaudhuri @UCSD, and our own @FeiziSoheil @umdcs.
UMD Department of Computer Science
@umdcs
.@UofMaryland's CS Department joins a @DeptofDefense-selected team for the 2024 MURI program. @MTHajiaghayi & @FeiziSoheil are part of the team exploring "Algorithms, Learning & Game Theory," focusing on decision-making in multi-agent systems.
Read more:
go.umd.edu/MURI -
Russ Salakhutdinov reposted
Ben Eysenbach
@ben_eysenbach
1/ Given high-dim time series data (e.g., video, text), we'd like to know "what happens in the future?" and "what happened between A and B?" Turns out temporal contrastive learning provably answers these questions!
Paper+Code: arxiv.org/abs/2403.04082
Backstory and details below -
Russ Salakhutdinov
@rsalakhu
You can do solid technical work or have long arguments about mumbo jumbo -- it is hard to do both. But one of them is much easier to do than the other.
Dan Roy
@roydanroy
I'm continually surprised by researchers wasting their time having arguments/conversations about mumbo jumbo.
-
Russ Salakhutdinov reposted
Minji Yoon
@MinjiYoon90
Exciting personal update: I defended my PhD last week on Deep Learning on Graphs. Huge thanks to my committee Profs Tom Mitchell, @jure, and my advisors @rsalakhu and Christos Faloutsos. I have joined @inflectionAI to extend the benefits of intelligence to a wider audience!
-
Russ Salakhutdinov
@rsalakhu
Congratulations to my former CMU PhD student Zhilin Yang on his new LLM startup Moonshot AI, raising over $1B in VC funding.
Zhilin has done some fundamental work in NLP and large language models during his time at CMU including TransformerXL and XLNet. techcrunch.com/2024/02/21/moonshot-ai-funding-china/ -
Russ Salakhutdinov
@rsalakhu
I could not agree more with @PhilBeaudoin's response to @ylecun post. Quoting Philippe via Facebook:
"I find it fascinating that ML researchers talk about their intuition with such a high degree of confidence even after having been wrong so frequently in the past. I’m used to such absolute language in the startup world, where evacuating nuance has become synonymous with « exhibiting confidence », a highly-sought attribute in founders.
But in academia?
Academics should be better and dare to use relative language to distinguish hypothesis from proven fact."
Yann LeCun
@ylecun
Modeling the world for action by generating pixel is as wasteful and doomed to failure as the largely-abandoned idea of "analysis by synthesis".
Decades ago, there was a big debate in ML about the relative advantages of generative methods vs discriminative methods for classification.
Learning theorists, such as Vapnik, argued against generative methods, pointing out that training a generative modeling was a way more difficult than classification (from the sample complexity standpoint).
Regardless, a whole community in computer vision was arguing that recognition should work by generating pixels from explanatory latent variables. At inference time, one would infer the configuration of latent variables that generated the observed pixels.
The inference method would use optimization: e.g. use a 3D model of an object and try to find the pose parameters that reproduce the image.
This never quite worked, and it was very slow.
Later, some people converted to the Bayesian religion and tried to use Bayesian inference for the latent (e.g. using variational approximations and/or sampling).
At some point, when Non-Parametric Bayes and Latent Dirichlet Allocation became the rage in text modeling, some folks heroically attempted to apply that to object recognition from images.
>>> THIS WAS A COMPLETE AND UTTER FAILURE <<<
If your goal is to train a world model for recognition or planning, using pixel-level prediction is a terrible idea.
Generation happens to work for text because text is discrete with a finite number of symbols. Dealing with uncertainty in the prediction is easy in such settings. Dealing with prediction uncertainty in high-dimension continuous sensory inputs is simply intractable.
That's why generative models for sensory inputs are doomed to failure. -
Russ Salakhutdinov
@rsalakhu
If only Sora were developed by Meta :)
Yann LeCun
@ylecun
Let me clear a *huge* misunderstanding here.
The generation of mostly realistic-looking videos from prompts *does not* indicate that a system understands the physical world.
Generation is very different from causal prediction from a world model.
The space of plausible videos is very large, and a video generation system merely needs to produce *one* sample to succeed.
The space of plausible continuations of a real video is *much* smaller, and generating a representative chunk of those is a much harder task, particularly when conditioned on an action.
Furthermore, generating those continuations would be not only expensive but totally pointless.
It's much more desirable to generate *abstract representations* of those continuations that eliminate details in the scene that are irrelevant to any action we might want to take.
That is the whole point behind the JEPA (Joint Embedding Predictive Architecture), which is *not generative* and makes predictions in representation space.
Our work on VICReg, I-JEPA, V-JEPA, and the works of others show that Joint Embedding architectures produce much better representations of visual inputs than generative architectures that reconstruct pixels (such as Variational AE, Masked AE, Denoising AE, etc).
When using the learned representations as inputs to a supervised head trained on downstream tasks (without fine tuning the backbone), Joint Embedding beats generative.
See the results table from the V-JEPA blog post or paper:
ai.meta.com/blog/v-jepa-yann-lecun-ai-model-video-joint-e… -
Russ Salakhutdinov
@rsalakhu
Completely agree with Kevin: “It’s fine for creative tasks, but not (yet) autonomous agents” – the key word here is YET.
We are still far from having fully autonomous & reliable agents, but here are some early examples (albeit brittle) of agents navigating the web and accomplishing some tasks, such as going on Yelp and making a reservation when asked:
“Make a reservation at Pusadee’s Garden for 2 people on the earliest day for dinner. Use my name JY Koh and phone number 650-555-5555”.
More examples:
x.com/kohjingyu/status/1758519294521176382
Kevin Patrick Murphy
@sirbayes
While GenAI is fun, I think its economic value is grossly over estimated, because it’s unreliable, risky and expensive to make and serve. It’s fine for creative tasks, but not (yet) autonomous agents
Michael Littman
@mlittmancs
Airline installs chatbot. Customer gets bad information from it. Customer asks for refund. Airline says "the chatbot is a separate legal entity that is responsible for its own actions" (!). Court says no. (Whew.) Airline makes refund and turns off chatbot. arstechnica.com/tech-policy/2024/02/air-canada-must-honor…
-
Russ Salakhutdinov
@rsalakhu
lol
This post is unavailable. See it on X. -
Russ Salakhutdinov
@rsalakhu
Check out a few demos and code of the multimodal GPT-4V + SoM agent from our VisualWebArena that you can run on any web task.
Code+Paper: jykoh.com/vwa
Jing Yu Koh
@kohjingyu
We've added a demo of the multimodal GPT-4V + SoM agent from VisualWebArena to our GitHub repo, and you can now run this agent on any web task you like!
Here is the agent helping us to find a good Thai restaurant in Pittsburgh (more examples in the 🧵): -
Russ Salakhutdinov reposted
Jing Yu Koh
@kohjingyu
We've added a demo of the multimodal GPT-4V + SoM agent from VisualWebArena to our GitHub repo, and you can now run this agent on any web task you like!
Here is the agent helping us to find a good Thai restaurant in Pittsburgh (more examples in the 🧵): -
Russ Salakhutdinov reposted
Dan Roy
@roydanroy
It doesn't count as a new invite, if the slide deck doesn't change.
<zing/> -
Russ Salakhutdinov reposted
Elad Hazan
@HazanPrinceton
most exciting paper *ever* from our @GoogleAI lab at @Princeton: @naman33k @danielsuo @XinyiChen2
arxiv.org/abs/2312.06837
*** Convolutional filters predetermined by the theory, no learning needed! *** -
Russ Salakhutdinov
@rsalakhu
I Tested a Next-Gen AI Assistant. It Will Blow You Away | WIRED www.wired.com/story/fast-forward-tested-next-gen-ai-assis…
-
Russ Salakhutdinov
@rsalakhu
I will be giving a talk at the Winter School of Deep Learning, ISI Kolkata at 9:00am EST, 19:30 IST, 14:00 UTC, 4th Feb. The talk will be streamed at
youtube.com/live/_uUW8CTqmRE
I will talk about embodied AI and will cover some of our recent work on AI Agents and VisualWebArena.
Arindam
@arimajee
Tomorrow Prof. Salakhutdinov (@rsalakhu) will deliver a talk as a part of the plenary lecture series at Winter School of Deep Learning, ISI Kolkata. Join us on YT at 19:30 IST 4th Feb (14:00 UTC, 4th Feb) via- youtube.com/live/_uUW8CTqmRE
#wsdl #llm #lmm #multimodal #deeplearning -
Russ Salakhutdinov reposted
Arindam
@arimajee
Tomorrow Prof. Salakhutdinov (@rsalakhu) will deliver a talk as a part of the plenary lecture series at Winter School of Deep Learning, ISI Kolkata. Join us on YT at 19:30 IST 4th Feb (14:00 UTC, 4th Feb) via- youtube.com/live/_uUW8CTqmRE
#wsdl #llm #lmm #multimodal #deeplearning -
Russ Salakhutdinov
@rsalakhu
Thanks @willknight and @WIRED for highlighting our work on Web AI Agents and VisualWebArena:
www.wired.com/story/fast-forward-tested-next-gen-ai-assis…
Paper + Code: jykoh.com/vwa
I think that in the near future we will have multimodal AI agents capable of planning, reasoning, and reliably executing actions on the web, truly augmenting human productivity.
Jing Yu Koh
@kohjingyu
Computer interfaces are inherently visual. To build general autonomous agents, we will need strong vision language models.
To assess the performance of multimodal agents, we introduce VisualWebArena (VWA): a benchmark for evaluating multimodal web agents on realistic visually grounded tasks.
🔗 Website: jykoh.com/vwa
📄 Paper: arxiv.org/abs/2401.13649
🤖 Code and data: github.com/web-arena-x/visualwebarena
🧵👇 -
Russ Salakhutdinov reposted
Zico Kolter
@zicokolter
That's a wrap on #icml2024 submissions! 🎉 Initial count is 9,653 submissions, including 220 position paper submission ... (the count will certainly be adjusted in coming days, but for reference, ICML 2023 had 6,538 submissions 😮)
-
Russ Salakhutdinov reposted
Preeth
@ppktweets
Listening to some AI legends @geoffreyhinton @rsalakhu thanks for moderating @tjpoutanen
-
Russ Salakhutdinov reposted
Zico Kolter
@zicokolter
We have been receiving a few questions about the @icmlconf #icml2024 Impact Statement in submissions, so made a short post about it here: medium.com/@icml2024pc/impact-statements-in-icml-submissi…
Hopefully this is helpful to people trying to understand whether to include anything in this section. -
Russ Salakhutdinov reposted
Aran Komatsuzaki
@arankomatsuzaki
VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks
proj: jykoh.com/vwa
abs: arxiv.org/abs/2401.13649 -
Russ Salakhutdinov reposted
Jing Yu Koh
@kohjingyu
VisualWebArena comprises of 910 diverse and complex web-based tasks that evaluate various capabilities of autonomous multimodal agents. To succeed on VWA, agents need to accurately process image-text inputs, interpret natural language instructions, and execute actions on websites to accomplish user-defined objectives.
Here is an example of a task to order the cheapest color printer to an address provided as a photograph (requiring OCR capabilities): -
Russ Salakhutdinov
@rsalakhu
I am very excited about our recent work on VisualWebArena (VWA):
Web: jykoh.com/vwa
Autonomous AI agents capable of planning, reasoning, and executing actions on the web will play a key role in automating many computer tasks. In the near future, AI agents will be able to reliably process not only text but also navigate and interact with visual information, truly augmenting human productivity.
VisualWebArena is designed to assess performance of multimodal web agents on diverse and complex visually grounded tasks. The best multimodal agents today, such as GPT-4V, achieve a success rate of only 16.4% on VisualWebArena tasks compared to human performance of 88.7%.
See this excellent thread by @kohjingyu for more details about VWA environment.
Jing Yu Koh
@kohjingyu
Computer interfaces are inherently visual. To build general autonomous agents, we will need strong vision language models.
To assess the performance of multimodal agents, we introduce VisualWebArena (VWA): a benchmark for evaluating multimodal web agents on realistic visually grounded tasks.
🔗 Website: jykoh.com/vwa
📄 Paper: arxiv.org/abs/2401.13649
🤖 Code and data: github.com/web-arena-x/visualwebarena
🧵👇 -
Russ Salakhutdinov
@rsalakhu
Heading to Toronto this week-end to participate at the University of Toronto AI ConferenceX, organised by UofT's Undergraduate AI Association.
I will have a chance to be on a panel with Geoffrey Hinton @geoffreyhinton. Any specific questions I should ask him? -
Russ Salakhutdinov reposted
Zico Kolter
@zicokolter
Now that at NeurIPS is upon us shortly ... it's time to start planning for ICML😀! Thrilled to serve with @kat_heller @adrian_weller @nuriaoliver as PCs, and @rsalakhu as general chair.
Call for papers is here: icml.cc/Conferences/2024/CallForPapers
Intro blog post: medium.com/@icml2024pc/welcome-to-icml-2024-10e1d50edb8b -
Russ Salakhutdinov reposted
Ted Moskovitz
@ted_moskovitz
Really happy to say that constrained RLHF was accepted at #iclr2024 as a spotlight!
A million thanks to my coauthors @Aaditya6284, @djstrouse, Tuomas Sandholm, @rsalakhu, @ancadianadragan, and @McaleerStephen.
Looking forward to seeing everyone in Vienna!
Ted Moskovitz
@ted_moskovitz
Worried your LLM will produce too many paperclips? Simply tell it when to stop – excited to share our new preprint, where we introduce an approach based on constrained RL to avoid overoptimization for compound reward models:
arxiv.org/abs/2310.04373
1/ -
Russ Salakhutdinov reposted
Ben Eysenbach
@ben_eysenbach
I'm hiring visiting students/RAs for Summer 2024! ☀️
Focus: scaling unsupervised RL & understanding emergent properties. 🚀
Who: PhD, MS, rising undergrad seniors w/ prior research experience.
Apply by emailing me a resume.
Details below👇 -
Russ Salakhutdinov
@rsalakhu
Vayu Robotics Blog - Interpretable End-to-End Robot Navigation www.vayurobotics.com/blog/interpretable-end-to-end-drivin…
-
Russ Salakhutdinov reposted
Nitish Srivastava
@nitishsr
Interpreting end-to-end trainable autonomy models is crucial for fostering trust and ensuring safety. The use of transformers in Vayu's autonomy model allows us to interpret the model's thought process through attention maps. www.vayurobotics.com/blog/interpretable-end-to-end-drivin…
#ai #nvidiarobotics -
Russ Salakhutdinov
@rsalakhu
On Noisy Evaluation in Federated Hyperparameter Tuning – Machine Learning Blog | ML@CMU | Carnegie Mellon University blog.ml.cmu.edu/2023/12/29/on-noisy-evaluation-in-federat…
-
Russ Salakhutdinov reposted
Minji Yoon
@MinjiYoon90
🧐Consider this: In the real world, do multimodal data always exhibit straightforward one-to-one relationships between modalities? Join me for a discussion on how LLMs manage multimodal data with intricate intermodal connections at Hall C2! 🔥
-
Russ Salakhutdinov
@rsalakhu
Brandon Trabucco @brandontrabucco and Max Gurinas @maxgurinas are presenting their work on Effective Data Augmentation With Diffusion Models.
Check out our paper: arxiv.org/abs/2302.07944
Brandon Trabucco
@brandontrabucco
Can we use Stable Diffusion in a data augmentation that semantically modifies images? At 11:45am in Hall E2 at #neurips , I will give a talk about DA-Fusion: github.com/brandontrabucco/da-fusion
Joint work with @rsalakhu, Kyle Doherty, and Max Gurinas -
Russ Salakhutdinov reposted
Brandon Trabucco
@brandontrabucco
Can we use Stable Diffusion in a data augmentation that semantically modifies images? At 11:45am in Hall E2 at #neurips , I will give a talk about DA-Fusion: github.com/brandontrabucco/da-fusion
Joint work with @rsalakhu, Kyle Doherty, and Max Gurinas -
Russ Salakhutdinov
@rsalakhu
I will be giving a talk on Friday morning on our ongoing work on Multimodal AI Agents at the #neurips2023 Foundation Models for Decision Making workshop — come and check it out.
sites.google.com/view/fmdm-neurips23/schedule -
Russ Salakhutdinov
@rsalakhu
Yue Wu is presenting our work on Studying Papers and Reasoning to Play Games.
Check out the paper here: arxiv.org/abs/2305.15486 -
Russ Salakhutdinov
@rsalakhu
Paul Liang @pliang279 and Martin Ma presenting the work on Factorized Contrastive Learning — check it out!
arxiv.org/abs/2306.05268 -
Russ Salakhutdinov
@rsalakhu
Paul Liang @pliang279 presenting his work on Quantifying and Modeling Multimodal Interactions: an Information Decomposition Framework.
paper: arxiv.org/abs/2302.12247 -
Russ Salakhutdinov reposted
Daniel Fried
@dan_fried
For folks at #neurips23 interested in contextual image generation, @kohjingyu is presenting our work on Generating Images with Multimodal LLMs at poster 2025!
Paper: arxiv.org/abs/2305.17216
With @rsalakhu
Jing Yu Koh
@kohjingyu
I'll be presenting the GILL poster (jykoh.com/gill) in 3 hours at 10.45am CT today. Come and talk to us (@dan_fried, @rsalakhu) about grounding text LMs to process, generate, and retrieve images!
📍 Great Hall & Hall B1+B2 (level 1) #2025
⏰ Wed 13 Dec 10:45am-12:45pm -
Russ Salakhutdinov reposted
Jing Yu Koh
@kohjingyu
I'll be presenting the GILL poster (jykoh.com/gill) in 3 hours at 10.45am CT today. Come and talk to us (@dan_fried, @rsalakhu) about grounding text LMs to process, generate, and retrieve images!
📍 Great Hall & Hall B1+B2 (level 1) #2025
⏰ Wed 13 Dec 10:45am-12:45pm -
Russ Salakhutdinov
@rsalakhu
Paul Laing @pliang279 is on academic job market this year. Check out his work at NeurIPS (4 papers) — any department would be very lucky to have him.
Paul Liang
@pliang279
Excited to attend #neurips2023 this week! Find me to chat about the foundations of multimodal machine learning, multisensory foundation models, interactive multimodal agents, and their applications.
I'm also on the academic job market, you can find my statements on my website: www.cs.cmu.edu/~pliang/
My collaborators and I will be presenting the following: -
Russ Salakhutdinov
@rsalakhu
Off to #neurips2023.
-
Russ Salakhutdinov reposted
Kamalika Chaudhuri
@kamalikac
Do representation learning models memorize their training data? To understand this, we propose a new method called Deja vu to measure memorization in these models. #neurips2023
-
Russ Salakhutdinov reposted
Elman Mansimov
@elmanmansimov
I have been browsing old photos from 2015 from my time in Toronto.
And found the photos of my old desk inside Pratt Building at 6 Kings College Road at University of Toronto. This is the birthplace of the alignDRAW project, where the idea was conceived and brought to fruition.
The second floor of Pratt building was an office space of visitors, grad students and professors who worked on machine learning and AI at University of Toronto at that time.
The "Neural Networks Laboratory" room 275 where my desk was a former closet (back many many years ago).
Tons of highly influential (average citation count probably be 50k+) ML researchers shared that office space one way or another over multiple decades. I remember it was very humbling to hear that from my colleagues at that time, thinking maybe that one day I can share this honor as well.
8 years have passed, so I wouldn't call myself a highly influential machine learning researcher yet. Although I think I had a good deal of success in machine learning, in particular very happy on the continuously positive reception of alignDRAW project over time.
What was considered a very very niche research topic at that time (I don't think my friends outside AI understood exactly what I was doing) the alignDRAW work has matured and made its mark as the first text-to-image generative model. Despite much higher fidelity generations from Midjourneys, Stable Diffusions, DALLEs of the world people keep remembering and appreciating 32x32 images from alignDRAW.
I think big part why people are drawn to the project are the prompts. The simplicity and power of alignDRAW prompts as noted by @rainisto stand out in todays age of complex prompt engineering. Sometimes you need 8 words like "a stop sign is flying in blue skies" to demonstrate the potential of your approach and open up a whole space.
An expected surprise to me came this year when @halecar2 reached out with proposal on distributing the generated images of alignDRAW (now are relics) in the physical and digital forms. He and @fellowshiptrust made alignDRAW project not only a scientific success, but a piece of art history.
A consistently positive reception of alignDRAW at @ParisPhotoFair and @verse_works have blown me away, with well know American Museum and collectors acquiring physical versions of alignDRAW generated images. @manovich wrote very kind words about me and alignDRAW work comparing it to the very first fixed photographs taken by Niépce in 1826-1827.
And this week a digital form of "a stop sign is flying in blue skies" received a first bid at @ChristiesInc auction, with a full digital set auction release coming up at @FellowshipAi next week.
But back to the story of my desk, I believe machine learning researchers have moved on to a new glass covered MaRS discovery building since then. The AI space has became mainstream as with our work spaces which have modernized.
But it is fun to reminisce on the workspace where many technological breakthroughs have been happened. Sometimes simple space with clear focus and a bit of time are enough to do an influential research. -
Russ Salakhutdinov reposted
Michael Figurnov
@mfigurnov
I visited UofT and @rsalakhu for a few months in late 2015. It was certainly a career defining moment for me, and I fondly remember the second floor of Pratt.
AlignDRAW was such a cool project: we could do image->text already, but text->image just seemed impossible at the time! -
Russ Salakhutdinov
@rsalakhu
Creative Robot Tool Use with Large Language Models – Machine Learning Blog | ML@CMU | Carnegie Mellon University blog.ml.cmu.edu/2023/12/08/creative-robot-tool-use-with-l…
-
Russ Salakhutdinov
@rsalakhu
ICML 2024 Call For Papers is out!
icml.cc/Conferences/2024/CallForPapers
with the amazing 2024 Program Committee
@kat_heller @adrian_weller @nuriaoliver and @zicokolter
Check out their initial blog post here:
medium.com/@icml2024pc/welcome-to-icml-2024-10e1d50edb8b
Zico Kolter
@zicokolter
Now that at NeurIPS is upon us shortly ... it's time to start planning for ICML😀! Thrilled to serve with @kat_heller @adrian_weller @nuriaoliver as PCs, and @rsalakhu as general chair.
Call for papers is here: icml.cc/Conferences/2024/CallForPapers
Intro blog post: medium.com/@icml2024pc/welcome-to-icml-2024-10e1d50edb8b -
Russ Salakhutdinov reposted
Max Welling
@wellingmax
I can confirm this to be true. But sooner or later the tradeoffs between industry employment with high salaries and academic employment with freedom to choose what to work on had to surface.
-
Russ Salakhutdinov reposted
Hal Daumé III
@haldaume3
Agree with Russ. it’s not impossible to do industry->academia but it’s tough. One reason is papers, but, especially for tenured cases, also is mentoring and grant raising. Interns aren’t grad students and zero experience raising grants raises eyebrows.
Russ Salakhutdinov
@rsalakhu
Going from one industry lab to another is usually an easy process -- people move all the time. Going back to academia after a few years as a full-timer in industry is possible but is much harder, especially with tenure -- being a postdoc or a visiting researcher for a year or two at a top research lab is fine.
Things may change in the future but I usually tell my students if you are going to industry for a few years thinking you can easily go back to academia, think twice.
Yann LeCun
@ylecun
Ross Girschick (@inkynumbers) is leaving FAIR for AI2, following @sainingxie who joined the NYU faculty (yay!), and @georgiagkioxari who went to Caltech, and Hervé Jégou and @AlexDefosse who went to non-profit Kyutai.
It's a loss for FAIR, but I'm happy for them.
There is absolutely *nothing* *wrong* with scientists from industry labs moving to Academia or non-profits after several years. For some people, it's a natural career transition.
At Bell Labs, a decent proportion of scientists would leave after 5 to 10 years and get a faculty position with tenure (skipping the grueling tenure process altogether) in a good place.
At different times in your life, priorities change. You've had an impact in one way and may want to try another way. You may want to work with students. You may enjoy the immediate reward of teaching.
The fact that you can work at FAIR and get a tenured faculty position in academia after a few years is a feature, not a bug.
This transition is possible from FAIR, as it was from Bell Labs, because FAIR practices open research and encourages scientists to publish. It means that you are not taking any risk by starting your career FAIR.
Your options are kept open.
People move and seed the research ecosystem. It's a Good Thing.
And there are lots of awesomely talented young computer vision scientists who have joined FAIR in the last several years, such as Ishan Misra, @alcinos26, Xinlei Chen, Christoph Feichtenhofer...
And that's just the West Coast contingent. There are lots of awesome CV people in Paris (of DINOv2 fame) and Montreal (I-JEPA and such). -
Russ Salakhutdinov
@rsalakhu
A lot of industry labs today, Meta included, have been focusing more and more on fewer large compute-intensive projects, with tens of individual researchers/contributors. This makes sense for industry labs and senior established researchers, as it may impact company's products and business -- nothing wrong with this. And many labs, less so today unfortunately, publish these results.
However, some junior researchers in those labs, including Meta, are worried about this shift, as it has become much harder to work on more focused smaller individual projects. Hence the chances of "transitioning to a position in academia if they so decide at some point in their career” are getting smaller. Of course, this may change in the future, we will see.
Yann LeCun
@ylecun
The number of industry labs that practice open research and encourage scientists to publish can be counted on the finger of one hand (if that).
FAIR is famous for that.
This allows scientists to transition to a position in academia if they so decide at some point in their career.
Below is a list of ML folks who moved from AT&T (Bell Labs and AT&T Labs-Research) to tenured or tenure-track positions in academia in the late 1990s and early 2000s.
Yann LeCun
@ylecun
Ok, here is a list of ML researchers from AT&T who moved academia: Michael Kearns, Dan Lee, Sebastian Seung, Fernando Pereira, Yoshua Bengio, Sara Solla, Rich Sutton, Satinder Singh, Peter Stone, Lawrence Saul, Rob Schapire,... and me.
You can totally do it *if* you are in a lab that encourages long-term research and publishing. Such labs include FAIR but can be counted on the fingers of one hand (if that).
You *can not* do it if you work at a startup for too long, or do a job where you can't do and publish interesting research (engineering, product development, management...). -
Russ Salakhutdinov reposted
Dan Roy
@roydanroy
If you want to move from industry to academia, find the modern day Bell Labs and be as good as these no-name losers. Easy peasy.
Yann LeCun
@ylecun
Ok, here is a list of ML researchers from AT&T who moved academia: Michael Kearns, Dan Lee, Sebastian Seung, Fernando Pereira, Yoshua Bengio, Sara Solla, Rich Sutton, Satinder Singh, Peter Stone, Lawrence Saul, Rob Schapire,... and me.
You can totally do it *if* you are in a lab that encourages long-term research and publishing. Such labs include FAIR but can be counted on the fingers of one hand (if that).
You *can not* do it if you work at a startup for too long, or do a job where you can't do and publish interesting research (engineering, product development, management...). -
Russ Salakhutdinov
@rsalakhu
Going from one industry lab to another is usually an easy process -- people move all the time. Going back to academia after a few years as a full-timer in industry is possible but is much harder, especially with tenure -- being a postdoc or a visiting researcher for a year or two at a top research lab is fine.
Things may change in the future but I usually tell my students if you are going to industry for a few years thinking you can easily go back to academia, think twice.
Yann LeCun
@ylecun
Ross Girschick (@inkynumbers) is leaving FAIR for AI2, following @sainingxie who joined the NYU faculty (yay!), and @georgiagkioxari who went to Caltech, and Hervé Jégou and @AlexDefosse who went to non-profit Kyutai.
It's a loss for FAIR, but I'm happy for them.
There is absolutely *nothing* *wrong* with scientists from industry labs moving to Academia or non-profits after several years. For some people, it's a natural career transition.
At Bell Labs, a decent proportion of scientists would leave after 5 to 10 years and get a faculty position with tenure (skipping the grueling tenure process altogether) in a good place.
At different times in your life, priorities change. You've had an impact in one way and may want to try another way. You may want to work with students. You may enjoy the immediate reward of teaching.
The fact that you can work at FAIR and get a tenured faculty position in academia after a few years is a feature, not a bug.
This transition is possible from FAIR, as it was from Bell Labs, because FAIR practices open research and encourages scientists to publish. It means that you are not taking any risk by starting your career FAIR.
Your options are kept open.
People move and seed the research ecosystem. It's a Good Thing.
And there are lots of awesomely talented young computer vision scientists who have joined FAIR in the last several years, such as Ishan Misra, @alcinos26, Xinlei Chen, Christoph Feichtenhofer...
And that's just the West Coast contingent. There are lots of awesome CV people in Paris (of DINOv2 fame) and Montreal (I-JEPA and such). -
Russ Salakhutdinov
@rsalakhu
Check out our new work on Manifold Preserving Guided Diffusion -- Training-free sampling method that leverages the manifold hypothesis to refine the guided diffusion steps.
It can be applied to both pre-trained pixel-space and latent diffusion models with applications to conditional generation in low-compute settings.
Paper/Results: kellyyutonghe.github.io/mpgd/
See @electronickale's excellent thread for more details.
Yutong (Kelly) He
@electronickale
🧠Tired of text-only image generation control? Don’t have resources to train for your own tasks? Annoying long sampling time?
📣 Introducing Manifold Preserving Guided Diffusion (MPGD) to solve all these problems! Learn more: kellyyutonghe.github.io/mpgd/ and in 🧵👇
#generativeai -
Russ Salakhutdinov
@rsalakhu
Supporting Human-AI Collaboration in Auditing LLMs with LLMs – Machine Learning Blog | ML@CMU | Carnegie Mellon University blog.ml.cmu.edu/2023/09/22/supporting-human-ai-collaborat…
-
Russ Salakhutdinov reposted
Jing Yu Koh
@kohjingyu
In GILL (jykoh.com/gill), we introduce special [IMG] tokens to let the LM decide if it should output images instead of text.
One unsatisfying part was that we didn't have multimodal conversations to finetune on. I'm glad to hear it confirmed that this works much better!
Andrej Karpathy
@karpathy
It emits special words, e.g. <|BROWSE|> etc. When the code "above" the LLM detects these words it captures the output that follows, sends it off to a tool, comes back with the result and continues the generation. How does the LLM know to emit these special words? Finetuning datasets "teach" it how and when to browse, by example. And/or the instructions for tool use can also be automatically placed in the context window (in the “system message”).
-
Russ Salakhutdinov reposted
ALVI Labs
@alvilabs
We're excited to share our achievement🌟
Meet the new ALVI Interface: a breakthrough in intuitive prosthetic control.
This technology offers individuals with hand differences a new movement experience:
✨Wrist rotation
🖐Finger movement
🕹 Interaction with objects in VR
Discover how we're turning futuristic dreams into today's reality. Be among the first to step into this new era of possibilities.
Watch full demo: youtu.be/Dx_6Id2clZ0 -
Russ Salakhutdinov
@rsalakhu
Check out new work on Factorized Contrastive Learning: Going Beyond Multi-view Redundancy.
Paper: arxiv.org/abs/2306.05268
Contrastive learning methods can successfully learn useful representations from unlabelled data given paired information (e.g., image-caption, video-audio pairs).
These approaches typically rely on the assumption that shared information between modalities is what we need for downstream tasks. However, task-relevant information can also be contained in modality-unique regions: information that is only present in one modality but still relevant to the task.
This work shows how we can learn multimodal representations via factorized contrastive learning that can capture both shared and unique information relevant to downstream tasks.
See @pliang279's excellent thread for more details.
Paul Liang
@pliang279
Despite the successes of contrastive learning (eg CLIP), they have a fundamental limitation - it can only capture *shared* info between modalities, and ignores *unique* info
To fix it, a thread for our #neurips2023 paper w Zihao Martin @james_y_zou @lpmorency @rsalakhu: -
Russ Salakhutdinov reposted
Jing Yu Koh
@kohjingyu
With all this talk about applying for PhDs or not, I'd like to add one point unrelated to career prospects:
Doing a PhD is super fun!
Massive research freedom, the chance to dive deeply into a topic you like, advised by world experts who are committed to helping you succeed. -
Russ Salakhutdinov
@rsalakhu
Check out our new work on Multimodal Learning.
Quantifying & Modelling Multimodal Interactions: An Information Decomposition Framework.
Paper: arxiv.org/abs/2302.12247
One of the key challenges in ML today lies in capturing interactions between multiple input modalities. How can we quantify the interactions that are needed to solve a multimodal task?
This work develops an information-theoretic approach to quantify the degree of redundancy, uniqueness, and synergy relating input modalities.
See @pliang279 's excellent thread for more details.
Paul Liang
@pliang279
Multimodal AI studies the info in each modality & how it relates or combines with other modalities. This past year, we've been working towards a **foundation** for multimodal AI:
I'm excited to share our progress at #neurips2023 & #icmi2023: arxiv.org/abs/2302.12247
see long 🧵: -
Russ Salakhutdinov
@rsalakhu
Check out our new work on Contrastive Difference Predictive Coding -- a new temporal difference contrastive predictive coding that can substantially decrease the amount of data needed to learn to predict future events.
Paper: arxiv.org/abs/2310.20141
Code: chongyi-zheng.github.io/td_infonce/
with @chongyiz1 and @ben_eysenbach.
See this excellent thread by @chongyiz1 for details.
Chongyi Zheng
@chongyiz1
1/7 Can we use InfoNCE to learn representations that predict future events? Excited to share our new paper Contrastive Difference Predictive Coding: chongyi-zheng.github.io/td_infonce, a temporal difference InfoNCE loss up to 1500x more sample efficient than the standard (MC) version.
-
Russ Salakhutdinov reposted
Chongyi Zheng
@chongyiz1
1/7 Can we use InfoNCE to learn representations that predict future events? Excited to share our new paper Contrastive Difference Predictive Coding: chongyi-zheng.github.io/td_infonce, a temporal difference InfoNCE loss up to 1500x more sample efficient than the standard (MC) version.
-
Russ Salakhutdinov reposted
Murtaza Dalal @ CoRL 2026
@mihdalal
Excited to share that OPTIMUS has been accepted to CoRL 2023! We have also open-sourced our Robosuite datasets and model training code here:
github.com/NVlabs/Optimus
Now you can train your own visuomotor transformers 🤖
Murtaza Dalal @ CoRL 2026
@mihdalal
Imitation learning is powerful, but hard to scale due to lack of high quality data. Introducing Optimus: TAMP-supervised visuomotor transformers. Optimus solves over 300 long-horizon manipulation tasks with up to 8 stages and 72 different objects. @NVIDIAAIDev @CMU_Robotics 1/N
-
Russ Salakhutdinov reposted
Scientific American
@sciam
New training method helps AI generalize like people do www.scientificamerican.com/article/new-training-method-he…
-
Russ Salakhutdinov
@rsalakhu
Very insightful work by @LakeBrenden and Marco Baroni showing that encoder-decoder transformer trained with MLC (meta-learning for compositionality) can learn compositional skills and in many cases do better than human learners.
Brenden Lake
@LakeBrenden
Scientific American article describing our meta-learning for compositionality approach and its implications. Excellent (and measured) science journalism by @lauren_leffer, with thoughts from @rsalakhu, Armando Solar-Lezama, and Paul Smolensky
Scientific American
@sciam
New training method helps AI generalize like people do www.scientificamerican.com/article/new-training-method-he…
-
Russ Salakhutdinov
@rsalakhu
New work on Confronting Reward Model Overoptimization with Constrained RLHF:
Focusing on studying overoptimization with composite Reward Models (RMs) and new ways of avoiding overoptimization by weighting component RMs with dynamic weights, naturally expressed by Lagrange multipliers.
Paper: arxiv.org/abs/2310.04373
Code: github.com/tedmoskovitz/ConstrainedRL4LMs
See this detailed thread by @ted_moskovitz.
Ted Moskovitz
@ted_moskovitz
Worried your LLM will produce too many paperclips? Simply tell it when to stop – excited to share our new preprint, where we introduce an approach based on constrained RL to avoid overoptimization for compound reward models:
arxiv.org/abs/2310.04373
1/ -
Russ Salakhutdinov
@rsalakhu
The Latest AI Chatbots Can Handle Text, Images and Sound. Here's How - Scientific American www.scientificamerican.com/article/the-latest-ai-chatbots…
-
Russ Salakhutdinov reposted
Ted Moskovitz
@ted_moskovitz
Worried your LLM will produce too many paperclips? Simply tell it when to stop – excited to share our new preprint, where we introduce an approach based on constrained RL to avoid overoptimization for compound reward models:
arxiv.org/abs/2310.04373
1/ -
Russ Salakhutdinov reposted
Yi Ma
@YiMaTweets
Agreed! As I said before, anyone saying such things is either ignorant (yes even so called AI experts) or with ulterior motives…
Russ Salakhutdinov
@rsalakhu
I disagree with @ylecun on various issues, but when it comes down to open source and existential risks of AI, my views are very much the same as Yann's.
In fact, the number of times I keep seeing AI and nuclear in the same sentence is mind-boggling. Really?
Yann LeCun
@ylecun
The heretofore silent majority of AI scientists and engineers who
- do not believe in AI extinction scenarios or
- believe we have agency in making AI powerful, reliable, and safe and
- think the best way to do so is through open source AI platforms
NEED TO SPEAK UP ! -
Russ Salakhutdinov
@rsalakhu
I disagree with @ylecun on various issues, but when it comes down to open source and existential risks of AI, my views are very much the same as Yann's.
In fact, the number of times I keep seeing AI and nuclear in the same sentence is mind-boggling. Really?
Yann LeCun
@ylecun
The heretofore silent majority of AI scientists and engineers who
- do not believe in AI extinction scenarios or
- believe we have agency in making AI powerful, reliable, and safe and
- think the best way to do so is through open source AI platforms
NEED TO SPEAK UP ! -
Russ Salakhutdinov
@rsalakhu
New work on Multimodal Graph Learning for Generative Tasks: a framework for capturing information from multimodal data with relational structure.
Paper: arxiv.org/abs/2310.07478
Check out this thread by @MinjiYoon90.
Minji Yoon
@MinjiYoon90
🚀 Excited to introduce MultiModal Graph Learning (MMGL) - exploring beyond the typical one-to-one data modality pairs🔗 to uncover more complex and multifaceted relationships among data modalities! 🌐
📄 Paper: arxiv.org/abs/2310.07478
🧵 Thread continues... -
Russ Salakhutdinov
@rsalakhu
Me asking Bard:
Me: Who is Ruslan Salakhutdinov
Bard: Ruslan Salakhutdinov is a Canadian computer scientist of Tatar origin working in the field of artificial intelligence...
Me: Tell me where he was born
Bard: Ruslan Salakhutdinov was born in Russia on April 1, 1996. He is of Tatar origin, a Turkic people native to the Volga-Ural region of Russia.
Me: when did he publish his first paper
Bard: Ruslan Salakhutdinov published his first paper in 2004, titled "Efficient Learning of Deep Boltzmann Machines"... Salakhutdinov's first paper was highly influential, and it has been cited over 1,000 times.
That is right -- I published a paper at the age of 8 with over 1,000 citations. And what did you accomplish when you were 8? -
Russ Salakhutdinov reposted
INSAIT Institute
@INSAITinstitute
📡Spread the word!
We are excited to announce new tenure-track and tenured #faculty positions for the ##hiring 2023–2024 cycle. INSAIT welcomes strong applicants in computer #science and #ai broadly.
The institute provides exceptional financial packages, which include salaries, startup funds, compute, and a number of covered PhD and post-doc positions. INSAIT is rapidly growing and in only a year, it has already attracted top faculty and PhD students, published at leading AI conferences (#neurips2023, #iccv2023), joined ELLIS (Europe's leading AI network), and created its first deep-tech #startup.
Come and joins us!
Application deadline: Jan 15, 2024
Apply here: insait.ai/faculty
CRA job post: shorturl.at/bsBEH -
Russ Salakhutdinov reposted
Thomas G. Dietterich
@tdietterich
Microsoft has previously investigated submerging data centers in sea water. Data center operators locate data centers for power and cooling (water). I don’t see why nuclear power is any more worrying. Just fear mongering, Max
-
Russ Salakhutdinov
@rsalakhu
For those of us who are resubmitting our rejected NeurIPS papers to ICLR. This was true when I was a grad student and it remains true today.
-
Russ Salakhutdinov reposted
Jing Yu Koh
@kohjingyu
Very happy to share that GILL 🐟 was accepted to #neurips2023!
If you're interested in adapting language models to process image-text inputs and generate image-text outputs, check out our paper (w/ code and pretrained models): jykoh.com/gill
See you in New Orleans!
Jing Yu Koh
@kohjingyu
Excited to share GILL🐟 — a method to fuse LLMs with image encoder and decoder models!
GILL is one of the first approaches capable of conditioning on interleaved image-text inputs to generate image-text outputs.
Paper: arxiv.org/abs/2305.17216
Website: jykoh.com/gill -
Russ Salakhutdinov
@rsalakhu
Test-time Adaptation with Slot-Centric Models – Machine Learning Blog | ML@CMU | Carnegie Mellon University blog.ml.cmu.edu/2023/09/15/test-time-adaptation-with-slot…
-
Russ Salakhutdinov
@rsalakhu
Some jokes I love about academia: Do you know the difference between pre-tenure and post-tenure?
Pre-tenure: I would go to my students and say: team, a conference deadline is approaching, what is our plan?
Post-tenure: Students come to my office and say: Russ, a conference deadline is approaching, what is our plan? -
Russ Salakhutdinov
@rsalakhu
I get asked a lot: why stay in academia, all the excitement in AI is happening in industry with massive compute. And I am seeing some profs leaving academia, but also seeing lots of researchers in industry looking to go back to academia, especially those who don’t work on LLMs.
I have spent some time and actively working with industry, and it has been a great experience, but for me the answer is simple:
1. Students: Nothing can replace working with really smart & amazing students.
2. Academic freedom: In general, industry will one way or another dictate what you should work on. Today it is LLMs, tomorrow it is something else. The good old days of "here is a cool idea, let me investigate and publish" are pretty much over. But as an academic, I can work on whatever I want: I can start working on "black holes" tomorrow if I choose to.
3. And my favourite one: No reorgs -- big tech really loves reorgs that happen every few months.
Finally, as an academic, I can always take some time off, work in industry or start-up, and come back if I want to. Going into industry is usually a one-way street: It is far easier to go to industry from academia and much harder the other way around. -
Russ Salakhutdinov
@rsalakhu
I hear a lot of folks in our AI community complain about openAI -- they don't publish, don't release models, maximum-for-profit, etc., so they are more like closedAI rather than openAI.
This is true, but you have to give it to those guys -- they showed the true potential of LLMs no one thought was possible.
Without openAI, Google would simply keep their models in-house, Meta would not work on open source LLMs, and would not have created GenAI division. Similar shifts would not have happened at Amazon, Apple, and other companies. A lot of academic research, including some of my own, is now dedicated to LLMs and their applications in many areas.
So while it remains closed, you have to give kudos to the openAI team, even for just showing what is possible. -
Russ Salakhutdinov reposted
Pedro Domingos
@pmddomingos
The Real AI 100
@TIME's list of the supposedly 100 most influential people in AI is such a mess that as a public service I've taken the time to compile a more reasonable one:
Pieter Abbeel @pabbeel
Ajay Agrawal @professor_ajay
Sam Altman @sama
Marc Andreessen @pmarca
Susan Athey
Yoshua Bengio
Nick Bostrom
Tye Brady @spaceguytye
Greg Brockman @gdb
Erik Brynjolfsson @erikbryn
(1/10) -
Russ Salakhutdinov reposted
Carnegie Mellon University
@CarnegieMellon
Researchers from @mldcmu and @CMU_LTI developed a multimodal large language model that accepts both images and text as input, and can layer text and images in responses. Called "GILL," it is one of the first models that can process multimodal integration. www.cs.cmu.edu/news/2023/gill
-
Russ Salakhutdinov
@rsalakhu
CMU Researchers Develop Multimodal LLM AI Method Named GILL www.cs.cmu.edu/news/2023/gill
-
Russ Salakhutdinov
@rsalakhu
CMU highlights our work on Generating Images with Multimodal Language Models.
The model is capable of conditioning on arbitrarily interleaved image and text inputs to generate image-text outputs.
Paper + code: jykoh.com/gill
with @kohjingyu, @dan_fried
Machine Learning Dept. at Carnegie Mellon
@mldcmu
MLD and LTI researchers have developed one of the first multimodal large language models that can process and produce layered images and text, where images and text can be provided as both the inputs and the outputs.
www.cs.cmu.edu/news/2023/gill -
Russ Salakhutdinov reposted
Minji Yoon
@MinjiYoon90
At the Graph Learning Benchmark #glb workshop, our collaborated work with #google (presented at ICML’23) has been selected as an outstanding paper
🤗 🎉 🏆. I'll dive into our work on developing a scalable, privacy-enhanced graph generative model that sets new benchmarks for GNNs -
Russ Salakhutdinov reposted
Minji Yoon
@MinjiYoon90
Hello Long Beach! 😎🏖️🏄♀️
Join me on Sunday at #kdd2023 as I present two exciting pieces of work:
2:20 pm at the #glb workshop and 2:45 pm at #knowledgenlp.
Let's explore the intriguing world of #graphdeeplearning together! -
Russ Salakhutdinov
@rsalakhu
This should be an exciting talk.
Feels like yesterday I was working with John Langford (@JohnCLangford) and Tong Zhang on nonlinear dynamical systems when I was John's intern at Yahoo research labs.
One of the best mentors I had during my PhD years.
U of T Department of Computer Science
@UofTCompSci
Join us for the @UofTCompSci Distinguished Lecture Series, featuring @JohnCLangford of @Microsoft Research.
🗓️ Tuesday, August 8, 2023
🕚 11 am – 12 pm
📍 BA 1200
ℹ️ Open to the public. No registration is required but space is limited.
uoft.me/9wJ -
Russ Salakhutdinov
@rsalakhu
Check out our new work on Self-Supervised Object Goal Navigation with In-Situ Finetuning.
The key insight is that embodied agents can leverage location consistency as a self-supervision signal - collecting images from different views/angles and applying contrastive learning.
Paper: arxiv.org/abs/2212.05923
See this excellent thread by @SoYeonTiffMin
So Yeon (Tiffany) Min
@SoYeonTiffMin
My paper with @Apple “Self-Supervised Object Goal Navigation with In-Situ Finetuning" (arxiv.org/abs/2212.05923) will appear at IROS 2023. We preset a fully self-supervised agent that can navigate and self-improve in real houses, to address test-time sim2real self-adaptation.(1/5)
-
Russ Salakhutdinov reposted
Anirudha Majumdar
@Majumdar_Ani
There are interesting challenges w.r.t. distribution shift that arise when using pre-trained perception modules as part of a feedback controller/planner (specifically, novel viewpoint angles, etc.); there is some great discussion in this talk by @rsalakhu
www.youtube.com/live/P3sdD6OPT3w -
Russ Salakhutdinov reposted
Minji Yoon
@MinjiYoon90
Aloha! 🤙🌴🌺
Today at #icml2023, I will present our collaborated work with @GoogleAI.
Got questions or thoughts on how to solve the scarcity of benchmark graph dataset problem?
Swing by Poster #104 from 2:00-3:30 PM.
Excited to exchange ideas and solutions! 🌟👩💻 -
Russ Salakhutdinov reposted
Minji Yoon
@MinjiYoon90
And I deeply thank my amazing collaborators
@phanein @JohnPalowitch @yw_yuewu @rsalakhu
for their support🙏🙏🫂
Check out our paper and code here👇
📜arxiv.org/pdf/2207.04396.pdf
👩💻github.com/minjiyoon/CGT -
Russ Salakhutdinov reposted
Paul Liang
@pliang279
At #acl2023nlp, we'll be presenting our latest work on controlling the distribution of LM-generated text in order to mitigate bias and promote fairness & personalization.
We achieve extreme few-shot control, using as few as 8 samples of human feedback!
arxiv.org/abs/2211.05750
Xiang Fan
@xiangfan00
(ACL 2023) Can you control language model generation with only 8 samples? Check out our paper on few-shot human-in-the-loop text generation control: arxiv.org/abs/2211.05750.
ACL Findings Spotlight: Mon 7-9pm ET (Metropolitan Centre)
RepL4NLP Workshop: Thu 11-12pm ET (Harbour B) -
Russ Salakhutdinov reposted
Jing Yu Koh
@kohjingyu
I'm thrilled to be in Honolulu🏝️ for #icml2023! I'll present "Grounding Language Models to Images for Multimodal Inputs and Outputs" (aka🧀FROMAGe).
🗓️Jul 27, Thurs, 10.30am
📜jykoh.com/fromage
(with @rsalakhu and @dan_fried)
Jing Yu Koh
@kohjingyu
Can we ground LLMs to enable them to process and produce image-text data?
Introducing 🧀FROMAGe: a model that efficiently bootstraps frozen LLMs for images. This enables LLMs to process image-text data and produce coherent image-text outputs.
arxiv.org/abs/2301.13823
🧵👇 -
Russ Salakhutdinov reposted
Chongyi Zheng
@chongyiz1
1/6 Self-supervised learning is wildly successful for CV, NLP; might goal-conditioned RL enable self-supervised RL? In months of experiments, we found design decisions that significantly boost success, acquire intriguing representations, and solve real robotic tasks from images.
-
Russ Salakhutdinov
@rsalakhu
If you want to work on the next generation reinforcement learning algorithms and decision-making systems in general, you should consider joining @ben_eysenbach's lab at Princeton.
Congratulations Ben! It was great having you at @mldcmu over the past 5 years!
Ben Eysenbach
@ben_eysenbach
Excited to share that I will be joining @Princeton as an assistant professor in CS, starting this Fall! Thanks to colleagues at @mldcmu and fantastic advisors @rsalakhu and @svlevine for the great PhD.
Looking forward to joining the wonderful folks @PrincetonCS! -
Russ Salakhutdinov reposted
-
Russ Salakhutdinov reposted
Paul Liang
@pliang279
Check out our latest work at #acl2023nlp on improving compositionality in vision-language models, through aligning not just entities but also relations between words and image regions
arxiv.org/abs/2212.10549
see 🧵 by @khoomeik -
Russ Salakhutdinov reposted
Paul Liang
@pliang279
I'm on the faculty market in fall 2023! I work on foundations of multimodal ML applied to NLP, socially intelligent AI & health.
My research & teaching www.cs.cmu.edu/~pliang/
If I could be a good fit for your department, I'd love to chat at #acl2023nlp & #icml2023 DM/email me! -
Russ Salakhutdinov reposted
Jing Yu Koh
@kohjingyu
You can now try out 🐟GILL on HuggingFace Spaces!
GILL is a model capable of conditioning on interleaved image + text inputs to generate image + text outputs.
Try the demo now: huggingface.co/spaces/jykoh/gill
Project page: jykoh.com/gill
More cool examples below 🧵👇 -
Russ Salakhutdinov
@rsalakhu
Navigating to Objects in the Real World – Machine Learning Blog | ML@CMU | Carnegie Mellon University blog.ml.cmu.edu/2023/06/30/navigating-to-objects-in-the-r…
-
Russ Salakhutdinov
@rsalakhu
Validating Large Language Models with ReLM – Machine Learning Blog | ML@CMU | Carnegie Mellon University blog.ml.cmu.edu/2023/06/05/validating-large-language-mode…
-
Russ Salakhutdinov reposted
Minji Yoon
@MinjiYoon90
🚀Excited to share our #icml2023 paper on a novel approach to #graphgenerativemodel! We address the growing need for large, real-world graph datasets to benchmark new GNN models, while grappling with privacy issues that limit access to proprietary graphs. #ml #gnn #cmu #google
-
Russ Salakhutdinov
@rsalakhu
So many people who have nothing to do with the technical aspects of Machine Learning or AI are jumping on the AGI bandwagon, essentially making wild predictions to get publicity -- I am impressed!
Of course, the wilder your prediction, the more publicity you get. -
Russ Salakhutdinov
@rsalakhu
Check out our new work on Computation Graph Transformer.
Today's large, real-world graph datasets are often generated from online, highly privacy-restricted ecosystems (such as Google, Amazon), which makes research and development on these datasets hard, if not impossible.
To address this, we introduce a novel graph generative
model, Computation Graph Transformer, that learns and reproduces the distribution of real-world graphs in a privacy-controlled way.
Paper: arxiv.org/abs/2207.04396
Code: github.com/minjiyoon/CGT
See this excellent thread by @MinjiYoon90
Minji Yoon
@MinjiYoon90
🚀Excited to share our #icml2023 paper on a novel approach to #graphgenerativemodel! We address the growing need for large, real-world graph datasets to benchmark new GNN models, while grappling with privacy issues that limit access to proprietary graphs. #ml #gnn #cmu #google
-
Russ Salakhutdinov reposted
Paul Liang
@pliang279
Excited that HighMMT - our attempt at a single multimodal transformer model with shared parameters for many modalities including images, videos, sensors, sets, tables & more, was accepted at TMLR:
paper: arxiv.org/abs/2203.01311
code: github.com/pliang279/HighMMT
Accepted papers at TMLR
@TmlrPub
High-Modality Multimodal Transformer: Quantifying Modality & Interaction Heterogeneity for High-M...
Paul Pu Liang, Yiwei Lyu, Xiang Fan et al..
Action editor: Brian Kingsbury.
openreview.net/forum?id=ttzypy3kT7
#multimodal #modality #gestures -
Russ Salakhutdinov reposted
Jing Yu Koh
@kohjingyu
Excited to share GILL🐟 — a method to fuse LLMs with image encoder and decoder models!
GILL is one of the first approaches capable of conditioning on interleaved image-text inputs to generate image-text outputs.
Paper: arxiv.org/abs/2305.17216
Website: jykoh.com/gill -
Russ Salakhutdinov reposted
Murtaza Dalal @ CoRL 2026
@mihdalal
Imitation learning is powerful, but hard to scale due to lack of high quality data. Introducing Optimus: TAMP-supervised visuomotor transformers. Optimus solves over 300 long-horizon manipulation tasks with up to 8 stages and 72 different objects. @NVIDIAAIDev @CMU_Robotics 1/N
-
Russ Salakhutdinov
@rsalakhu
Check out our new work on High-Modality Multimodal Transformer -- a single model that scales up to 10 modalities (text, image, audio, video, sensors, proprioception, speech, time-series, sets, tables).
This work also demonstrates a crucial scaling behaviour: performance continues to improve with each modality added, and it transfers to entirely new modalities and tasks during fine-tuning.
Code + Paper: github.com/pliang279/HighMMT
See excellent thread by @pliang279.
Paul Liang
@pliang279
Excited that HighMMT - our attempt at a single multimodal transformer model with shared parameters for many modalities including images, videos, sensors, sets, tables & more, was accepted at TMLR:
paper: arxiv.org/abs/2203.01311
code: github.com/pliang279/HighMMT
Accepted papers at TMLR
@TmlrPub
High-Modality Multimodal Transformer: Quantifying Modality & Interaction Heterogeneity for High-M...
Paul Pu Liang, Yiwei Lyu, Xiang Fan et al..
Action editor: Brian Kingsbury.
openreview.net/forum?id=ttzypy3kT7
#multimodal #modality #gestures -
Russ Salakhutdinov reposted
Marc Andreessen 🇺🇸
@pmarca
Ask Me Anything about AI, or any of the topics in Why AI Will Save The World, as responses to this tweet and I will answer as many as I can.
a16z.com/2023/06/06/ai-will-save-the-world/ -
Russ Salakhutdinov reposted
Brandon Trabucco
@brandontrabucco
Can we use Stable Diffusion for data augmentation?
Introducing DA-Fusion: a data augmentation strategy using pretrained diffusion models to semantically modify image attributes. This complements existing data augmentation and improves few-shot learning.
arxiv.org/abs/2302.07944 -
Russ Salakhutdinov
@rsalakhu
Feels like our current AGI debates about risks, safety, regulations. You just need to shout louder.
-
Russ Salakhutdinov reposted
Chongyi Zheng
@chongyiz1
6/6 Paper: arxiv.org/abs/2306.03346
Code: chongyi-zheng.github.io/stable_contrastive_rl/
With fantastic collaborators
@ben_eysenbach @HomerWalke @patrickhyin @KuanFang @rsalakhu @svlevine -
Russ Salakhutdinov
@rsalakhu
Check out our new work on Stabilizing Contrastive RL:
Architecture design, careful weight initialization, & data augmentation can significantly boost performance of contrastive RL in sim & solve real-world robotic tasks from images:
chongyi-zheng.github.io/stable_contrastive_rl
See 🧵by @chongyiz1
Chongyi Zheng
@chongyiz1
1/6 Self-supervised learning is wildly successful for CV, NLP; might goal-conditioned RL enable self-supervised RL? In months of experiments, we found design decisions that significantly boost success, acquire intriguing representations, and solve real robotic tasks from images.
-
Russ Salakhutdinov
@rsalakhu
With all the craziness about AGI posing existential threat to humanity, I think this New Yorker cartoon pretty much nails it.
-
Russ Salakhutdinov reposted
Yue Wu
@yw_yuewu
GPT-4 Out-performs RL Algorithms by Studying the Paper and Reasoning.
Our SPRING framework extracts relevant info from an academic paper on an RL game (Crafter) and learns to play the game without training. (1/6)
with @SoYeonTiffMin @shrimai_ @ybisk @rsalakhu @tommmitchell -
Russ Salakhutdinov
@rsalakhu
Check out our new work on Generating Images with Multimodal Language Models: image retrieval, novel image generation, and multimodal dialogue.
Paper: arxiv.org/abs/2305.17216
Website: jykoh.com/gill
with @kohjingyu, @dan_fried
See excellent thread by @kohjingyu.
Jing Yu Koh
@kohjingyu
Excited to share GILL🐟 — a method to fuse LLMs with image encoder and decoder models!
GILL is one of the first approaches capable of conditioning on interleaved image-text inputs to generate image-text outputs.
Paper: arxiv.org/abs/2305.17216
Website: jykoh.com/gill -
Russ Salakhutdinov
@rsalakhu
Checkout new work by my student @mihdalal on large scale imitation learning by leveraging Task and Motion Planning (TAMP) & Transformers that can solve over 300 robotic manipulation tasks from images.
Paper: arxiv.org/abs/2305.16309
Website: mihdalal.github.io/optimus
See 🧵below:
Murtaza Dalal @ CoRL 2026
@mihdalal
Imitation learning is powerful, but hard to scale due to lack of high quality data. Introducing Optimus: TAMP-supervised visuomotor transformers. Optimus solves over 300 long-horizon manipulation tasks with up to 8 stages and 72 different objects. @NVIDIAAIDev @CMU_Robotics 1/N
-
Russ Salakhutdinov reposted
Machine Learning Dept. at Carnegie Mellon
@mldcmu
Register now for the SCS Generative AI Innovation Incubator virtual events starting tomorrow at 11 am ET and taking place through June and August with the goal of helping members of the public better benefit from generative AI and identify its risks.
www.cs.cmu.edu/generative-ai/ -
Russ Salakhutdinov
@rsalakhu
“Talent hits a target no one else can hit; Genius hits a target no one else can see” A. Schopenhauer
I've seen @geoffreyhinton hit targets no one thought existed. So even though I don't fully agree with Geoff’s views on AI risks, I'd still listen carefully to what he has to say. -
Russ Salakhutdinov
@rsalakhu
I fully agree with @ylecun points here but I think Yann is missing a few equally ridiculous points form that list:
- On the highway towards Human-Level AI, LLM is an off-ramp.
- Abandon Generative and Probabilistic Models.
- Abandon Reinforcement Learning.
Yann LeCun
@ylecun
AI hype is ridiculous in all directions.
As in:
- LLMs have superhuman intelligence
- LLMs are useless parrots
- LLM hallucinations will destroy society
- scaling is all you need
- deep learning has hit a wall
- AI doesn't exist and never will
- AI is going to kill us allThis post is unavailable. See it on X. -
Russ Salakhutdinov reposted
Jing Yu Koh
@kohjingyu
You have $7 to spend on the perfect NeurIPS submission:
$700: SOTA results
$350: theoretical guarantees
$200: polished figures
$100: well written
$7: "GPT-4 is great at X"
$7: "GPT-4 is terrible at X"
$0: "all you need" as part of the title -
Russ Salakhutdinov reposted
Machine Learning Dept. at Carnegie Mellon
@mldcmu
Paul Liang (@pliang279) received the 2023 Graduate Student Teaching Award "for incredible work in designing and teaching several new courses in Multimodal Machine Learning and Artificial Social Intelligence, general excellence in teaching, and excellence in student mentorship."
-
Russ Salakhutdinov
@rsalakhu
-
Russ Salakhutdinov
@rsalakhu
It has been fascinating to watch progress in AI. In a span of a few months we went from deep learning is hitting a wall, and LLMs are just autoregressive models so who cares, to these models are so powerful, AGI is not that far out. Interesting times ahead.
-
Russ Salakhutdinov
@rsalakhu
Are Model Explanations Useful in Practice? Rethinking How to Support Human-ML Interactions. – Machine Learning Blog | ML@CMU | Carnegie Mellon University blog.ml.cmu.edu/2023/03/31/are-model-explanations-useful-…
-
Russ Salakhutdinov
@rsalakhu
Research Brief: Breakthrough Enables Perfectly Secure Secret Communications www.cs.cmu.edu/news/2023/secure-secret-communications
-
Russ Salakhutdinov
@rsalakhu
Towards Behavior-Driven AI Development – Machine Learning Blog | ML@CMU | Carnegie Mellon University blog.ml.cmu.edu/2023/03/24/zeno/
-
Russ Salakhutdinov
@rsalakhu
RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learning – Machine Learning Blog | ML@CMU | Carnegie Mellon University blog.ml.cmu.edu/2023/02/24/rlprompt-optimizing-discrete-t…
-
Russ Salakhutdinov reposted
Ben Eysenbach
@ben_eysenbach
I really enjoyed this conversation about some of the work we've done on RL algorithms, as well as the many open problems and areas for continuing research!
Imbue
@imbue_ai
Learn about designing simpler and more principled RL algorithms and much more with @ben_eysenbach from CMU!
Blog: generallyintelligent.com/podcast/2023-03-22-podcast-episo…
Video: youtu.be/EXMjEQvyzs0 -
Russ Salakhutdinov
@rsalakhu
I usually get invited to speak at regular ML conference/workshop venues. But today, I got invited to meet and discuss important issues with sentient AI. I am intrigued – that is a first one for me.
-
Russ Salakhutdinov
@rsalakhu
Max Planck's principle: "scientific change does not occur because individual scientists change their mind, but rather that successive generations of scientists have different views."
Christopher Manning
@chrmanning
This is truly an opinion piece. Not even a cursory attempt is made to check easily refutable claims (“they may well predict, incorrectly”). Melodramatic claims of inadequacy are made not of specific current models but any possible machine learning approach
www.nytimes.com/2023/03/08/opinion/noam-chomsky-chatgpt-a… -
Russ Salakhutdinov reposted
Oriol Vinyals
@OriolVinyalsML
Criticism is easy, and gets a lot of attention these days. And, as we all know, attention is all (some people) need. To those who build: you rock!
Christopher Manning
@chrmanning
This is truly an opinion piece. Not even a cursory attempt is made to check easily refutable claims (“they may well predict, incorrectly”). Melodramatic claims of inadequacy are made not of specific current models but any possible machine learning approach
www.nytimes.com/2023/03/08/opinion/noam-chomsky-chatgpt-a… -
Russ Salakhutdinov
@rsalakhu
Well, if you are a social media AI researcher, then all you can do is make lots of predictions, wait until someone actually does the real thing, and then declare victory by claiming credit.
Oriol Vinyals
@OriolVinyalsML
Making predictions <<< Making them happen
-
Russ Salakhutdinov reposted
Oriol Vinyals
@OriolVinyalsML
Making predictions <<< Making them happen
-
Russ Salakhutdinov reposted
Minji Yoon
@MinjiYoon90
@GoogleAI introduced KTN (Neurips'22), my last summer intern project, in their blog today.
Check out how we teach old labels new tricks in heterogeneous graphs 🧙♀️ TLDR; we transfer knowledge across multi-modal complex ecosystems using connectivity information🪄
Google AI
@GoogleAI
Introducing KTN, a new model that transfers knowledge from label-abundant node types in a heterogeneous graph to zero-labeled node types using existing relational information. Learn how it outperforms state-of-the-art transfer learning baselines → goo.gle/3Zd3bGX
-
Russ Salakhutdinov
@rsalakhu
New work on analysing, debugging, and visualising learned internal representations of multimodal models.
Paper + Code: github.com/pliang279/MultiViz
See this 🧵 by @pliang279.
Paul Liang
@pliang279
Multimodal models like VilBERT, CLIP & transformers are taking over by storm! But do we understand what they learn? At #iclr2023 we're presenting MultiViz, an analysis framework for model understanding, error analysis & debugging.
arxiv.org/abs/2207.00056
github.com/pliang279/MultiViz -
Russ Salakhutdinov
@rsalakhu
FROMAGe is now on HuggingFace Spaces:
huggingface.co/spaces/jykoh/fromage
The model can process interleaved image-and-text inputs, and generate free-form text interleaved with retrieved images.
Paper + Code: jykoh.com/fromage
Jing Yu Koh
@kohjingyu
You can now try out 🧀FROMAGe on HuggingFace Spaces!
FROMAGe is a model that can process image and text inputs to produce image and text outputs.
Try out the demo now: huggingface.co/spaces/jykoh/fromage
Project page: jykoh.com/fromage
More cool examples below 🧵👇 -
Russ Salakhutdinov
@rsalakhu
Check out new work on data augmentation with pre-trained diffusion models: Semantically modifying images, while generalising to novel concepts.
Paper: arxiv.org/abs/2302.07944
Code: btrabuc.co/da-fusion/
work with @brandontrabucco , K Doherty, & @maxgurinas.
See 🧵by Brandon
Brandon Trabucco
@brandontrabucco
Can we use Stable Diffusion for data augmentation?
Introducing DA-Fusion: a data augmentation strategy using pretrained diffusion models to semantically modify image attributes. This complements existing data augmentation and improves few-shot learning.
arxiv.org/abs/2302.07944 -
Russ Salakhutdinov
@rsalakhu
BASIC
PASCAL
PERL
MATLAB
MATLAB
MATLAB
MathWorks: pay us money
Me: OK
MATLAB
MATLAB
MathWorks: pay us more money
Me: OK
My students: PYTHON
Me: No, MATLAB
My students: No, PYTHON
PYTHON
Me: Oh no, ok, PYTHON
PYTHON
PYTHON
PYTORCH
😀
naomi (forgetful functor)
@fixedpointfae
in what order did you learn your languages?
1. python
2. R
3. haskell
4. C
5. rust
6. scheme
7. {java,type}script
8. clojure
9. scala
10. java
11. idris
12. go -
Russ Salakhutdinov
@rsalakhu
The weird thing about AI is that some folks in our field tend to make big statements like: abandon approach X, approach Y will not lead to human-level AI or AGI, but Z will -- which is a waste of time.
Does that happen in other fields, or AI folks just like to be in a spotlight? -
Russ Salakhutdinov reposted
Minji Yoon
@MinjiYoon90
In poster session 6 in #neurips2022, I will present our paper collaborated with @CMU x @GoogleAI x @UCLA.
Can we infer a user node’s preference without using any user labels? Can we instead exploit product nodes’ abundant labels given in their publicly available content?
...🧵 -
Russ Salakhutdinov
@rsalakhu
Regardless of Human Level AI, AGI, or whatever other type of AI we come up with, Large Language Models, including GPT style models, and generative AI is here to stay for a long time -- whether you like it or not.
-
Russ Salakhutdinov
@rsalakhu
To be fair, @openAI had a big impact on AI/ML. Their OpenAI Gym env for RL, their CLIP - a de facto model for any image-text research, their Whisper speech recognition is amazing - all open sourced.
In my view, they've had a bigger impact than many labs publishing 100s papers 🧵
Yann LeCun
@ylecun
Data on the intellectual contribution to AI from various research organizations.
Some of organizations publish knowledge and open-source code for the entire world to use.
Others just consume it.This post is unavailable. See it on X. -
Russ Salakhutdinov
@rsalakhu
New work on Grounding Language Models to Images for Multimodal Generation:
Bootstrapping frozen LLMs to generate free-form text interleaved with images:
Paper: arxiv.org/abs/2301.13823
Code: github.com/kohjingyu/fromage
See this excellent thread by @kohjingyu.
Jing Yu Koh
@kohjingyu
Can we ground LLMs to enable them to process and produce image-text data?
Introducing 🧀FROMAGe: a model that efficiently bootstraps frozen LLMs for images. This enables LLMs to process image-text data and produce coherent image-text outputs.
arxiv.org/abs/2301.13823
🧵👇 -
Russ Salakhutdinov
@rsalakhu
Bottom-up Top-Down Detection Transformers For Open Vocabulary Object Detection – Machine Learning Blog | ML@CMU | Carnegie Mellon University blog.ml.cmu.edu/2023/01/13/bottom-up-top-down-detection-t…
-
Russ Salakhutdinov reposted
-
Russ Salakhutdinov reposted
Martin Vechev
@mvechev
Tenure-track and tenured faculty jobs in AI/CS at INSAIT (insait.ai), created in partnership with @ETH_en and @EPFL_en. Fantastic conditions (salaries, packages), great PhD students, immense opportunity for impact, and more. Spread the news.
cra.org/job/insait-open-tenure-track-and-tenured-faculty-… -
Russ Salakhutdinov reposted
INSAIT Institute
@INSAITinstitute
Applications are now open for @INSAITinstitute undergraduate summer research program. Conduct research in computer science & AI with world-class advisors. Monthly stipend of 1500EUR & travel funding provided. Accepting applicants from around the world!
insait.ai/surf -
Russ Salakhutdinov
@rsalakhu
Via @MikhailBurtsev:
Me: Caption this.
AI: Here you go.
Sounds about right. -
Russ Salakhutdinov reposted
Peng Qi
@qi2peng2
I was deeply humbled when I learned that HotpotQA has accumulated 1k citations since its release at EMNLP 2018 four years ago. A lot has happened during this time, and I've been reflecting on what I have learned from being part of this journey.
A 🧵 of reflections 🌟 -
Russ Salakhutdinov
@rsalakhu
John Schulman @johnschulman2 has done some very impressive work in RL. I am using his papers when teaching deep learning classes at CMU. Congrats on ChatGPT.
Wojciech Zaremba
@woj_zaremba
Many might not know.
@johnschulman2 is the main person driving ChatGPT, and I would like to congratulate him and the team!
Worth to follow. -
Russ Salakhutdinov reposted
August Wester
@augustwester
I recently decided to do a write-up on the now famous Transformer-XL paper by @ZihangDai, @rsalakhu, and collaborators. It proposed to augment the Transformer with a recurrent hidden state "memory" to widen the "field of view" of the attention mechanism. sigmoidprime.com/post/transformer-xl
-
Russ Salakhutdinov
@rsalakhu
Sorry @ylecun, couldn't help
Yann 2 week ago: galactica.ai will generate a paper with relevant references, formulas & everything. Amazing work
Yann today: Abandon generative/prob models
But jokes aside, I really like galactica paper, great work by the authors @MetaAI
Mofijul Islam
@BeingMIAkashs
Recommendations from @ylecun talk at SSL #neurips2022 Workshop
-
Russ Salakhutdinov reposted
-
Russ Salakhutdinov reposted
Paul Liang
@pliang279
Vision-language models, despite their size, still struggle on compositional generalization benchmarks like Winoground. We show that incorporating structure in the attention alignment maps is a promising way to fine-tune these models for compositionality:
sslneurips22.github.io/paper_pdfs/paper_65.pdf -
Russ Salakhutdinov reposted
Ben Eysenbach
@ben_eysenbach
Excited to share some work today at #neurips2022:
4p: Joint Model-Policy Optimization for Model-Based RL (#928)
4p: Adversarial Unlearning: Reducing Confidence Along Adversarial Directions (#610)
Colabs with @AlexKhazatsky @setlur_amrith @rsalakhu @svlevine @gingsmith -
Russ Salakhutdinov reposted
Ben Eysenbach
@ben_eysenbach
Excited to share our work on imitation learning and model-based RL today at #neurips2022:
11a: Imitating Past Successes can be Very Suboptimal (#303)
4p: Joint Model-Policy Optimization for Model-Based RL (#610)
Colabs with @0602soumith @tianjun_zhang @svlevine @rsalakhu. -
Russ Salakhutdinov reposted
Paul Liang
@pliang279
Contrastive self-supervised learning is used extensively for pre-training, but what implications do they have on fairness? We show that contrastive SSL can exacerbate biases in pre-training, and propose an approach to improve fairness:
arxiv.org/abs/2106.02866
@ SSL workshop -
Russ Salakhutdinov reposted
Paul Liang
@pliang279
Also on friday, Xiang will be presenting an oral talk on few-shot human-in-the-loop control of large language models for personalized and fair text generation, an approach that is much more sample efficient than existing LM control methods.
arxiv.org/abs/2211.05750 -
Russ Salakhutdinov reposted
Paul Liang
@pliang279
Multimodal models are prevalent, but do we understand what they learn? MultiViz is a framework for visualizing & understanding multimodal models that enables model understanding, error analysis & model debugging.
@ friday human-in-the-loop workshop
arxiv.org/abs/2207.00056 -
Russ Salakhutdinov
@rsalakhu
Causal Confounds in Sequential Decision Making – Machine Learning Blog | ML@CMU | Carnegie Mellon University blog.ml.cmu.edu/2022/11/28/causal-confounds-in-sequential…
-
Russ Salakhutdinov reposted
Tianjun Zhang
@tianjun_zhang
Hi all! I'll be presenting 2 papers at #neurips2022:
Tuesday at 11:00 am CST, Hall J #610, @ben_eysenbach and I will present Contrastive RL, a new algorithm that frames contrastive learning as an effective RL method! (collaborated with @svlevine and @rsalakhu) -
Russ Salakhutdinov
@rsalakhu
How do Authors’ Perceptions about their Papers Compare with Co-authors’ Perceptions and Peer-review Decisions? – Machine Learning Blog | ML@CMU | Carnegie Mellon University blog.ml.cmu.edu/2022/11/22/neurips2021-author-perception-…
-
Russ Salakhutdinov
@rsalakhu
New work on Zero-shot Transfer Learning in a Heterogeneous Graph via Knowledge Transfer Networks:
Transferring knowledge from label-abundant nodes to zero-labeled nodes via rich relational information in the graph.
arxiv.org/abs/2203.02018
See excellent thread by @MinjiYoon90.
Minji Yoon
@MinjiYoon90
In poster session 6 in #neurips2022, I will present our paper collaborated with @CMU x @GoogleAI x @UCLA.
Can we infer a user node’s preference without using any user labels? Can we instead exploit product nodes’ abundant labels given in their publicly available content?
...🧵 -
Russ Salakhutdinov reposted
Yuxin Xiao
@YuxinXiao6
A big shout-out to @dnnslmr for the wonderful and detailed summary of our work! This paper has been accepted to the findings of #emnlp2022. Check our analysis results if you want a practical manual on building a well-calibrated #nlp pipeline with pre-trained language models!
Dennis Ulmer 🦋
@dnnslmr
Super cool work about uncertainty quantification for #nlproc by @YuxinXiao6, @pliang279, @umangsbhatt, @willieneis, @rsalakhu, Louis-Philippe Morency.
See thread below for summary and links to some other works! 🧵[1/9]
arxiv.org/pdf/2210.04714.pdf -
Russ Salakhutdinov reposted
Dennis Ulmer 🦋
@dnnslmr
Super cool work about uncertainty quantification for #nlproc by @YuxinXiao6, @pliang279, @umangsbhatt, @willieneis, @rsalakhu, Louis-Philippe Morency.
See thread below for summary and links to some other works! 🧵[1/9]
arxiv.org/pdf/2210.04714.pdf -
Russ Salakhutdinov
@rsalakhu
This is a great and very professional response from @soumithchintala about #galactica.
Soumith Chintala
@soumithchintala
After a lot of useful, critical feedback on Galactica, the language model demo has now been paused. There is more work to do.
Papers with Code
@paperswithcode
🪐 Introducing Galactica. A large language model for science.
Can summarize academic literature, solve math problems, generate Wiki articles, write scientific code, annotate molecules and proteins, and more.
Explore and get weights: galactica.org -
Russ Salakhutdinov
@rsalakhu
With the whole #galactica situation, I am surprised Meta AI doesn’t have a more unified response on social media.
Imagine a world where a top executive of a company like Apple would get into numerous fights on Twitter over a product that was clearly not ready for a prime time.
Yann LeCun
@ylecun
Galactica demo is off line for now.
It's no longer possible to have some fun by casually misusing it.
Happy?This post is unavailable. See it on X. -
Russ Salakhutdinov reposted
Michael Black
@Michael_J_Black
I asked #galactica about some things I know about and I'm troubled. In all cases, it was wrong or biased but sounded right and authoritative. I think it's dangerous. Here are a few of my experiments and my analysis of my concerns. (1/9)
-
Russ Salakhutdinov reposted
Paul Liang
@pliang279
Do AI models know if an object can be easily broken💔? or melts at high heat🔥?
Check out PACS: a new audiovisual question-answering dataset for physical commonsense reasoning and new models at #eccv2022 this week:
paper: arxiv.org/abs/2203.11130
video: drive.google.com/file/d/1EoCsnn9ZfuPrvdg6g3h7fnDLyB0pncUP… -
Russ Salakhutdinov reposted
INSAIT Institute
@INSAITinstitute
INSAIT is delighted to announce its call for tenure-track and tenured faculty positions with outstanding conditions, happening for the first time in Eastern Europe! Please spread the news to all faculty candidates in computer science/AI this year: cra.org/job/insait-open-tenure-track-and-tenured-faculty-…
-
Russ Salakhutdinov
@rsalakhu
Very cool work! Maybe with more scale and these innovations, we can go after deep Boltzmann machines.
Renjie Liao
@lrjconan
We show that Gaussian RBMs can generate good images just like other generative models, despite the single-layer architecture. Key innovations:
1) Gibbs-Langevin sampling;
2) modified Contrastive Divergence.
Paper: arxiv.org/abs/2210.10318
Code: github.com/lrjconan/GRBM
1/2 -
Russ Salakhutdinov reposted
Paul Liang
@pliang279
Check out more experiments and qualitative examples here: arxiv.org/abs/2203.11130
And download the dataset, code, and models here: github.com/samuelyu2002/PACS
w Samuel Yu @peter_yh_wu @rsalakhu @lpmorency @mldcmu @LTIatCMU @berkeley_ai -
Russ Salakhutdinov reposted
Paul Liang
@pliang279
Excited to share our new benchmark, PACS: an audiovisual question-answering dataset for physical commonsense reasoning and new models at #eccv2022!
Paper: arxiv.org/abs/2203.11130
Code/Dataset: github.com/samuelyu2002/PACS
w Samuel Yu @peter_yh_wu @rsalakhu @lpmorency -
Russ Salakhutdinov
@rsalakhu
Tackling Diverse Tasks with Neural Architecture Search – Machine Learning Blog | ML@CMU | Carnegie Mellon University blog.ml.cmu.edu/2022/10/14/tackling-diverse-tasks-with-ne…
-
Russ Salakhutdinov reposted
So Yeon (Tiffany) Min
@SoYeonTiffMin
My paper "Don’t Copy the Teacher:
Data and Model Challenges in Embodied Dialogue" (arxiv.org/abs/2210.04443) is accepted to EMNLP 2022! Dialogue is an important field and thankfully TEACh enables us to explore the novel challenges it presents in embodied settings. -
Russ Salakhutdinov reposted
Albert Gu
@_albertgu
A belated announcement: I'll be joining the Machine Learning Department at CMU @mldcmu as an assistant professor starting Fall 2023! In the meantime, I'll be at @DeepMind with @NandoDF's team, working from the MTV office. Looking forward to what's ahead!
-
Russ Salakhutdinov
@rsalakhu
Fun thread:
Pedro: how to make transformers intelligent?
LeCun: we need A,B,C, see my paper on a path to autonomous AI.
Marcus: it's X,Y,Z see my paper on a path to robust AI.
But the real trick is to get other researchers refer to your work as a solution to AI, not you yourself
Pedro Domingos
@pmddomingos
The most interesting question about transformers is what needs to be added to them to make something truly intelligent.
-
Russ Salakhutdinov reposted
Raj Ghugare
@GhugareRaj
In RL, a learnt model can be used to learn representations that minimise the bits of information taken from observations, by punishing the policy for doing things where the model is inaccurate. What happens if we use these predicted representations to do model-based RL?
-
Russ Salakhutdinov
@rsalakhu
Check out new work on model-based RL: a single objective that jointly optimizes a latent-space model and policy (see this excellent thread by @ben_eysenbach)
Paper: arxiv.org/abs/2209.08466
More importantly, code by @GhugareRaj:
github.com/RajGhugare19/alm
Ben Eysenbach
@ben_eysenbach
1/8 Model-based RL is hard because models are inaccurate and policies exploit those errors. Modeling compact state representations can help, but how should they be learned?
-
Russ Salakhutdinov reposted
Jack Clark
@jackclarkSF
2015 versus 2022. Incredible.
2015 via this @rsalakhu paper arxiv.org/abs/1511.02793
2022 via beta.dreamstudio.ai/dream #stablediffusion -
Russ Salakhutdinov
@rsalakhu
Tracking Any Pixel in a Video – Machine Learning Blog | ML@CMU | Carnegie Mellon University blog.ml.cmu.edu/2022/09/09/tracking-any-pixel-in-a-video/
-
Russ Salakhutdinov
@rsalakhu
Long-term Dynamics of Fairness Intervention in Connection Recommender Systems – Machine Learning Blog | ML@CMU | Carnegie Mellon University blog.ml.cmu.edu/2022/09/02/long-term-dynamics-of-fairness…
-
Russ Salakhutdinov
@rsalakhu
Recurrent Model-Free RL Can Be a Strong Baseline for Many POMDPs – Machine Learning Blog | ML@CMU | Carnegie Mellon University blog.ml.cmu.edu/2022/08/26/recurrent-model-free-rl-can-be…
-
Russ Salakhutdinov
@rsalakhu
Galaxies on Graph Neural Networks – Machine Learning Blog | ML@CMU | Carnegie Mellon University blog.ml.cmu.edu/2022/08/19/galaxies-on-graph-neural-netwo…
-
Russ Salakhutdinov reposted
Tom M Mitchell
@tommmitchell
Pleased to announce that my Machine Learning textbook is now a free download -- enjoy! www.cs.cmu.edu/afs/cs.cmu.edu/user/mitchell/ftp/mlbook.html
-
Russ Salakhutdinov reposted
Jian Ma
@jmuiuc
Congrats to @RuochiZhang for a successful PhD defense today. Ruochi will continue his exciting journey @broadinstitute as an Eric and Wendy Schmidt Center fellow, bridging ML & biology. Photo of a happy PhD committee @jure, @rsalakhu, Ziv Bar-Joseph, Ivet Bahar, & a proud advisor
-
Russ Salakhutdinov reposted
Davis Blalock
@davisblalock
"Breaking the Softmax Bottleneck: A High-Rank RNN Language Model"
I don't hear this paper talked about a lot, but a recent large-scale study found that it was one of only three methods that actually improved vanilla transformers. Here's how it works: [1/10] -
Russ Salakhutdinov
@rsalakhu
auton-survival: An Open-Source Package for Regression, Counterfactual Estimation, Evaluation and Phenotyping Censored Time-to-Event Data – Machine Learning Blog | ML@CMU | Carnegie Mellon University blog.ml.cmu.edu/2022/08/05/auton-survival-an-open-source-…
-
Russ Salakhutdinov reposted
Paul Liang
@pliang279
If you're coming to #naacl2022 drop by our workshop on Multimodal AI, now in its 4th edition! We have invited speakers covering multimodal learning for embodied AI, virtual reality, robotics, HCI, healthcare, & education!
July 15, 9am-4pm
multicomp.cs.cmu.edu/maiworkshop2022/ -
Russ Salakhutdinov
@rsalakhu
Does AutoML work for diverse tasks? – Machine Learning Blog | ML@CMU | Carnegie Mellon University blog.ml.cmu.edu/2022/07/07/automl-for-diverse-tasks/
-
Russ Salakhutdinov
@rsalakhu
1/3 I think we should coin a new term: social media AI researcher, where instead of publishing your work at the rigorous peer review venue, you tweet about your findings and opinions.
There are huge advantages:
1. It is easy: you don't have to deal with that annoying reviewer 2.
Russ Salakhutdinov
@rsalakhu
2. If someone argues about the validity of your claims, you can point them to other social media AI researchers who agree with you or link to some studies supporting your claim.
3. There is very little accountability so you don't have to think about or cite prior work. (2/3)
Russ Salakhutdinov
@rsalakhu
4. More importantly, when someone publishes a technical work at a top venue, you can always point them to your earlier tweet telling them that you had this idea long time ago and demand that they acknowledge you in their paper. (3/3)
-
Russ Salakhutdinov
@rsalakhu
1/4 I was watching @geoffreyhinton interview w/t @pabbeel - it reminded me of a couple fun stories about Geoff and how he got me into deep learning.
Back in 2005, I was not thinking about PhD & had a good career as ML engineer. One morning, on my way to work I bumped into Geoff.
Russ Salakhutdinov
@rsalakhu
2/4 He started telling me about Deep Belief Nets & stacked Restricted Boltzmann Machines he was working on together with @yeewhye & @sindero
He invited me to his office. I was late for work and it was way too early in the morning, but Geoff insisted and I ended up in his office.
Russ Salakhutdinov
@rsalakhu
3/4 Geoff showed me a very cool MNIST demo & wrote some variational bound equations. I barely understood any of it, but in 15 mins he convinced me that this is going to be the next big breakthrough & that I should go for a PhD his lab.
Not only that, he got me into PhD program,
Russ Salakhutdinov
@rsalakhu
4/4 helped with the PhD fellowship, shared his Matlab code - a year later we had a paper in Science on deep nets and I ended up doing my PhD with the best advisor I could possibly have.
Had I not bumped into @geoffreyhinton that morning, my career would have been very different. -
Russ Salakhutdinov
@rsalakhu
New work on Contrastive Learning as Goal-Conditioned
Reinforcement Learning -- using Contrastive Learning to solve certain types of RL problems:
So all you need is a good representation learning algorithm.
arxiv.org/abs/2206.07568
Ben Eysenbach
@ben_eysenbach
1/ Doing representation learning via *deep* RL is hard, so many RL methods require auxiliary representation learning objectives or augmentations. What if we did the opposite? Surprisingly, we show how contrastive representation learning can be used as goal-conditioned RL.
-
Russ Salakhutdinov
@rsalakhu
Seeing some chatter on twitter about the future of AI & reaching some milestones.
If you want to know about the future of AI, you should ask young research scientists - they will define the future of AI in the next 10 years, not folks on twitter who have nothing to do with AI:) -
Russ Salakhutdinov
@rsalakhu
New work on outcome-conditioned imitation learning showing that these methods may not improve the policy and in some cases can actually do worse.
But a simple modification can guarantee policy improvement.
arxiv.org/abs/2206.03378
See this excellent thread by @ben_eysenbach
Ben Eysenbach
@ben_eysenbach
There's a bug in a common and simple approach to multi-task RL. Here's the approach: imitate what worked in the past. Collect a bunch of trials, look at which trials solved which tasks, and learn each task by using the trials that solved this task.
-
Russ Salakhutdinov
@rsalakhu
There are many world-class professors @INSAITinstitute & across the globe who will be mentoring PhD students.
I will be looking for 1-2 PhD students, who will also be working closely my group at CMU, so apply here
insait.ai/phd
These are fully funded PhD positions.
INSAIT Institute
@INSAITinstitute
We are excited to launch our world-class AI/CS PhD program (insait.ai/phd), the first of its kind in Eastern Europe, with @DeepMind PhD fellowships, please share :)
-
Russ Salakhutdinov
@rsalakhu
1/2 I am very honoured to be a Fellow of the newly established Institute for Computer Science, AI and Technology @INSAITinstitute at the heart of Eastern Europe -- Sofia, Bulgaria, spanning many areas of research in AI, ML, CV, NLP, and many others:
insait.ai
Russ Salakhutdinov
@rsalakhu
2/2 There are many world-class professors at INSAIT & across the globe who will mentor PhD students.
I will be working with 1-2 PhD students, who will also be working closely my group at CMU, so apply here:
insait.ai/phd/
These are fully funded 5-year PhD positions. -
Russ Salakhutdinov
@rsalakhu
Deep Attentive Variational Inference – Machine Learning Blog | ML@CMU | Carnegie Mellon University blog.ml.cmu.edu/2022/05/27/deep-attentive-variational-inf…
-
Russ Salakhutdinov
@rsalakhu
1) My observation when working at Apple and many other industrial partners is that deep learning researchers who are well versed in graphical models, variational inference, CRFs, MCMC, uncertainty quantification are on average much more creative when solving real-world problems.
Yann LeCun
@ylecun
Researchers in speech recognition, computer vision, and natural language processing in the 2000s were obsessed with accurate representations of uncertainty.
1/N
Russ Salakhutdinov
@rsalakhu
2) This is compared to DL researchers who spend a lot of their time tweaking different ways of combining objective functions, searching for architectures, tuning hyper-parameters, trying to improve SOTA on speech, vision, NLP systems.
Russ Salakhutdinov
@rsalakhu
3) My advise to deep learning PhD students is to also learn variational inference, belief propagation, MCMC, HMMs, graphical models, generative models, etc. -- no matter how irrelevant you think this is for the current speech, vision, and NLP systems.
-
Russ Salakhutdinov reposted
Paul Liang
@pliang279
HighMMT standardizes input modalities into sequences and uses modality-specific embedding layers to capture unique information. The rest of the model learns modality and task-agnostic representations through shared unimodal and multimodal layers trained via multitask learning.
-
Russ Salakhutdinov
@rsalakhu
New work on designing a general multimodal transformer that generalizes across text, image, video, audio, time-series, sensors, tables, & set modalities, while improving the tradeoff between performance & efficiency:
Paper: arxiv.org/abs/2203.01311
Code: github.com/pliang279/HighMMT
Paul Liang
@pliang279
Are you working on multimodal tasks and can't decide on a model? Check out HighMMT -our attempt at a single Transformer model with shared parameters for sentiment, emotion, humor, disease, robot pose prediction & more!
paper: arxiv.org/abs/2203.01311
code: github.com/pliang279/HighMMT -
Russ Salakhutdinov
@rsalakhu
An Experimental Design Perspective on Model-Based Reinforcement Learning – Machine Learning Blog | ML@CMU | Carnegie Mellon University blog.ml.cmu.edu/2022/05/06/barl/
-
Russ Salakhutdinov reposted
Yilun Du
@du_yilun
Today (5/5) 4pm EDT, Ruslan Salakhutdinov @rsalakhu from CMU @SCSatCMU @mldcmu will be giving a talk at the Embodied Intelligence Seminar @MIT_CSAIL
on building autonomous embodied agents!
Live-streaming from: youtube.com/channel/UCnXGbvgu9071i3koFooncAw…, co-hosted with @EpisodeYang! -
Russ Salakhutdinov
@rsalakhu
Check out a guest lectures by my student Paul Laing @pliang279 on Multimodal Deep Learning at CMU's Deep Learning class.
www.youtube.com/watch?v=9v6Xg5Nk76M
Paul Liang
@pliang279
Really excited to release the video of my guest lecture on Multimodal Deep Learning for CMU's Deep Learning class @rsalakhu @mldcmu
It covers 5 fundamental concepts in multimodal ML: representation, alignment, reasoning, translation & co-learning
Youtube: www.youtube.com/watch?v=9v6Xg5Nk76M&ab_channel=PaulLiang -
Russ Salakhutdinov
@rsalakhu
New #iclr2022 paper on Conditional Contrastive Learning with Kernels: Efficient "conditional sampling" of positives and negatives to include/exclude information of external variables with applications to weakly-supervised and fair contrastive learning:
arxiv.org/abs/2202.05458
Russ Salakhutdinov
@rsalakhu
with Yao-Hung Hubert Tsai, Tianqin Li, Martin Q. Ma, Han Zhao, Kun Zhang, and Louis-Philippe Morency,
-
Russ Salakhutdinov
@rsalakhu
Two Experiments in Peer Review: Posting Preprints and Citation Bias – Machine Learning Blog | ML@CMU | Carnegie Mellon University blog.ml.cmu.edu/2022/04/01/peer_review_experiments_2022/
-
Russ Salakhutdinov
@rsalakhu
Highly recommend — Nitish is one of the smartest people I know.
Nitish Srivastava
@nitishsr
Introducing Vayu Robotics! We are building a scalable mobile robotics platform using cutting edge bio-inspired sensing, scalable machine learning, and purposeful robotic design. www.vayurobotics.com #hiring #robotics #machinelearning
-
Russ Salakhutdinov
@rsalakhu
Check out code for #iclr2022 FILM paper that learns structured representations by building a semantic map of the scene and performing exploration with a semantic search policy to achieve the natural language goal:
Code: github.com/soyeonm/FILM
Paper: arxiv.org/abs/2110.07342
So Yeon (Tiffany) Min
@SoYeonTiffMin
The code for FILM, my ICLR 2022 paper with @dchaplot, @ybisk, and @rsalakhu, is now publicly released as the starting code for the ALFRED challenge held at the #cvpr2022 EAI 2022 Workshop! Excited to see people push the SotA even further
github.com/soyeonm/FILM
Luca Weihs
@LucaWeihs
Happy to announce that the 3rd Embodied AI Workshop is happening on June 19th at #cvpr2022. Lots of exciting speakers and 13 (!!!) challenges. Check out our website (embodied-ai.org), dive into a challenge, and save the date!
-
Russ Salakhutdinov
@rsalakhu
New #iclr2022 paper: Learning Weakly-Supervised Contrastive Representations using auxiliary information.
As for performance, the auxiliary-information-infused self-supervised learning comes closer to supervised learning
Paper arxiv.org/abs/2202.06670
Code github.com/Crazy-Jack/Cl-InfoNCE
Russ Salakhutdinov
@rsalakhu
with Yao-Hung Hubert Tsai, Tianqin Li, Weixin Liu, Peiyuan Liao, and Louis-Philippe Morency
-
Russ Salakhutdinov
@rsalakhu
Check out a guest lecture by my student @MinjiYoon90 on Graph Neural Networks at CMU's Deep Learning class:
youtu.be/3WXhnQr4ADQ
Minji Yoon
@MinjiYoon90
I gave a guest lecture on Graph Neural Networks for Carnegie Mellon University's Deep Learning class taught by @rsalakhu. This talk introduces 1) the general concept of Graph Neural Networks and 2) interesting open research problems in the field.
Youtube: youtu.be/3WXhnQr4ADQ -
Russ Salakhutdinov
@rsalakhu
Assessing Generalization of SGD via Disagreement – Machine Learning Blog | ML@CMU | Carnegie Mellon University blog.ml.cmu.edu/2022/03/04/assessing-generalization-of-sg…
-
Russ Salakhutdinov
@rsalakhu
Josh Tenenbaum (as many others) was working on world models a decade ago, when I was a postdoc in his lab. So when Meta says they are working on an alternative vision to AI based on world models, that sounds a bit funny to me.
AI at Meta
@AIatMeta
Chief AI Scientist Yann LeCun (@ylecun) is sketching an alternate vision for building human-level AI. LeCun proposes that the ability to learn “world models” — internal models of how the world works — may be the key. Learn more: ai.meta.com/blog/yann-lecun-advances-in-ai-research/
-
Russ Salakhutdinov reposted
So Yeon (Tiffany) Min
@SoYeonTiffMin
The code for FILM, my ICLR 2022 paper with @dchaplot, @ybisk, and @rsalakhu, is now publicly released as the starting code for the ALFRED challenge held at the #cvpr2022 EAI 2022 Workshop! Excited to see people push the SotA even further
github.com/soyeonm/FILM
Luca Weihs
@LucaWeihs
Happy to announce that the 3rd Embodied AI Workshop is happening on June 19th at #cvpr2022. Lots of exciting speakers and 13 (!!!) challenges. Check out our website (embodied-ai.org), dive into a challenge, and save the date!
-
Russ Salakhutdinov
@rsalakhu
What is AI? Stephen Hanson in conversation with Geoff Hinton - ΑΙhub. Very interesting conversation with @geoffreyhinton aihub.org/2022/02/02/what-is-ai-stephen-hanson-in-convers…
-
Russ Salakhutdinov
@rsalakhu
Why Spectral Normalization Stabilizes GANs: Analysis and Improvements – Machine Learning Blog | ML@CMU | Carnegie Mellon University blog.ml.cmu.edu/2022/01/21/why-spectral-normalization-sta…
-
Russ Salakhutdinov
@rsalakhu
Improving RL with Lookahead: Learning Off-Policy with Online Planning – Machine Learning Blog | ML@CMU | Carnegie Mellon University blog.ml.cmu.edu/2022/01/07/loop/
-
Russ Salakhutdinov
@rsalakhu
Gave a talk at the National Institute of Technology Silchar in India. As a token of appreciation, the organizers planted trees for all keynote speakers -- how cool is that!
-
Russ Salakhutdinov
@rsalakhu
1/3: 2021 has been an incredible year: 5 of my PhD students (some co-advised) have graduated this year.
It's a privilege to have worked with such talented people!
Devendra Chaplot, Lisa Lee, Emilio Parisotto, Shrimai Prabhumoye & Hubert Tsai
Check out their theses/papers here:
Russ Salakhutdinov
@rsalakhu
2/3:
@dchaplot: Devendra Chaplot: Building Intelligent Autonomous Navigation Agents devendrachaplot.github.io
@shrimai_ Shrimai Prabhumoye: Controllable Text Generation www.cs.cmu.edu/~sprabhum/
Emilio Parisotto: Meta Reinforcement Learning through Memory www.cs.cmu.edu/~eparisot/
Russ Salakhutdinov
@rsalakhu
3/3:
@rl_agent Lisa Lee: Learning Embodied Agents with Scalably-Supervised Reinforcement Learning leelisa.com
Yao-Hung Hubert Tsai: Cross-view Learning with Limited Supervision yaohungt.github.io -
Russ Salakhutdinov
@rsalakhu
PASS: Performance-Adaptive Sampling Strategy for Graph Convolutional Networks – Machine Learning Blog | ML@CMU | Carnegie Mellon University blog.ml.cmu.edu/2021/12/03/pass-performance-adaptive-samp…
-
Russ Salakhutdinov
@rsalakhu
An Energy-based Perspective on Learning Observation Models – Machine Learning Blog | ML@CMU | Carnegie Mellon University blog.ml.cmu.edu/2021/12/17/learning-observation-models/
-
Russ Salakhutdinov reposted
Hugo Larochelle
@hugo_larochelle
Today, @RaiaHadsell, @kchonyc and I are happy to announce the creation of a new journal: Transaction on Machine Learning Research (TMLR)
Learn more in our post: medium.com/@hugo_larochelle_65309/announcing-the-transact… -
Russ Salakhutdinov
@rsalakhu
Lol… More like from #metaai, CMU and UIUC.
Yann LeCun
@ylecun
SEAL: Self-Supervised Embodied Active Learning from #metaai
Devendra Chaplot
@dchaplot
Presenting SEAL: Self-supervised Embodied Active Learning! #neurips2021
SEAL is a self-supervised framework to close the action-perception loop. It improves perception & action models by just moving in the physical world w/o any human supervision.
devendrachaplot.github.io/projects/seal
1/N -
Russ Salakhutdinov
@rsalakhu
SEAL: Self-supervised Embodied Active Learning using Exploration and 3D Consistency:
Closing the Action-Perception Loop: improving object detection / instance segmentation by simply moving around in the physical world.
Paper: arxiv.org/abs/2112.01001
Web: devendrachaplot.github.io/projects/seal
Devendra Chaplot
@dchaplot
Presenting SEAL: Self-supervised Embodied Active Learning! #neurips2021
SEAL is a self-supervised framework to close the action-perception loop. It improves perception & action models by just moving in the physical world w/o any human supervision.
devendrachaplot.github.io/projects/seal
1/N -
Russ Salakhutdinov
@rsalakhu
Machine Learning Department @mldcmu at CMU is hiring for multiple tenure-track positions at all levels -- Assistant, Associate, and Full Professor.
All areas of machine learning and AI will be considered.
Come join the best place to do ML research!
www.ml.cmu.edu/faculty-hiring.html -
Russ Salakhutdinov reposted
Jacob Buckman
@jacobmbuckman
This excellent paper from Tianwei Ni, @ben_eysenbach, and @rsalakhu provides empirical evidence for this: arxiv.org/abs/2110.05038
They provide a nice implementation of generic "task --> POMDP --> MDP", and it beats task-specialized algorithms pretty much everywhere. 5/n -
Russ Salakhutdinov reposted
Paul Liang
@pliang279
dressed up as our PhD advisor @rsalakhu for (belated) halloween! cos there’s nothing scarier than your advisor 👻🥳😝 jk Russ is the best 🎉🍾❤️
-
Russ Salakhutdinov reposted
Machine Learning Dept. at Carnegie Mellon
@mldcmu
We're looking for new tenure- and teaching-track faculty. Come join the world's first and finest academic machine learning department!
ml.cmu.edu/faculty-hiring -
Russ Salakhutdinov
@rsalakhu
Accelerating Robotic Reinforcement Learning via Parameterized Action Primitives:
Enable standard RL algorithms to outperform SOTA skill learning methods for robotic control from images with sparse rewards
Paper/code mihdalal.github.io/raps/
w/ @mihdalal, @pathak220 #neurips2021
Russ Salakhutdinov
@rsalakhu
joint work with @mihdalal and @deepakpathak.
-
Russ Salakhutdinov reposted
Murtaza Dalal @ CoRL 2026
@mihdalal
Excited to share the first work of my PhD: Robot Action Primitives for RL! (RAPS)
Our paper demonstrates that a simple change to the agent’s action space can *dramatically* improve the performance of standard RL algorithms on over 60+ tasks
1/6
Russ Salakhutdinov
@rsalakhu
Accelerating Robotic Reinforcement Learning via Parameterized Action Primitives:
Enable standard RL algorithms to outperform SOTA skill learning methods for robotic control from images with sparse rewards
Paper/code mihdalal.github.io/raps/
w/ @mihdalal, @pathak220 #neurips2021 -
Russ Salakhutdinov reposted
Paul Liang
@pliang279
Excited to present MultiBench, a large-scale benchmark for multimodal representation learning across affective computing, healthcare, robotics, finance, HCI, and multimedia domains at #neurips2021 benchmarks track! 🎉
paper: arxiv.org/abs/2107.07502
code: github.com/pliang279/MultiBench -
Russ Salakhutdinov
@rsalakhu
Turns out that careful architecture and hyperparameter decisions yield a recurrent model-free RL implementation that performs on par with, and occasionally way better than, more sophisticated recent techniques in their respective domains.
Paper/Code: sites.google.com/view/pomdp-baselines
Tianwei Ni
@twni2016
In theory, solving POMDPs should be as easy as doing standard RL with RNNs. In practice, prior work found it hard to get them to work. Our new paper finds, with careful design, model-free recurrent RL can perform at least on par with the specialized methods on many POMDPs!
-
Russ Salakhutdinov
@rsalakhu
FILM: Following Instructions in Language with Modular Methods
Building semantic map of the scene & performing exploration w/t a semantic search policy to achieve the natural language goal. SOTA performance on ALFRED
Paper arxiv.org/abs/2110.07342
Web/Code soyeonm.github.io/FILM_webpage/ -
Russ Salakhutdinov reposted
Papers with Datasets
@paperswithdata
ConditionalQA is a question answering dataset that contains complex questions with conditional answers, i.e. the answers are only true when certain conditions apply.
It can motivate doing research for complex question answering over long documents.
paperswithcode.com/dataset/conditionalqa -
Russ Salakhutdinov
@rsalakhu
Many real world questions can’t be answered deterministically. Instead, answers can only be true if certain conditions apply.
Check out our new dataset ConditionalQA: an extremely challenging but super exciting task!
Dataset haitian-sun.github.io/conditionalqa/
Paper arxiv.org/abs/2110.06884 -
Russ Salakhutdinov
@rsalakhu
Congratulations Lisa for successfully defending PhD on Learning Embodied Agents with Scalably-Supervised RL
Check out Lisa's papers, code, talks on planning, exploration, weakly supervised RL
leelisa.com
Good luck at @GoogleAI: amazing place to do amazing research!
Lisa Lee
@rl_agent
Five years ago, I left Google to pursue a PhD in Machine Learning.
Tomorrow, I'm very excited to join Google Brain as a Research Scientist! Looking forward to meeting everyone :)
My PhD thesis can be viewed at: leelisa.com/pdf/phd_thesis.pdf
Thesis defense: youtu.be/P4LH020ON3A -
Russ Salakhutdinov
@rsalakhu
New work on the Information Geometry of Unsupervised Reinforcement Learning -- analyzing unsupervised skill discovery algorithms based on mutual information maximization:
Paper: arxiv.org/abs/2110.02719
Ben Eysenbach
@ben_eysenbach
Unsupervised skill learning methods based on mutual information (e.g., DIAYN) learn a wide range of skills, which are often useful for solving downstream tasks. Are these methods a provably good way to do pretraining? Do you learn skills for all reward-maximizing behaviors?
-
Russ Salakhutdinov reposted
Ben Eysenbach
@ben_eysenbach
Unsupervised skill learning methods based on mutual information (e.g., DIAYN) learn a wide range of skills, which are often useful for solving downstream tasks. Are these methods a provably good way to do pretraining? Do you learn skills for all reward-maximizing behaviors?
-
Russ Salakhutdinov
@rsalakhu
New work on joint optimization of the model and the policy for model-based RL.
Paper: arxiv.org/abs/2110.02758
Code: github.com/ben-eysenbach/mnm/blob/main/experiments.ipynb
Ben Eysenbach
@ben_eysenbach
Most model-based RL algorithms optimize the model to be accurate and optimize the policy to get high returns when practicing with that model. Unfortunately, models that are more accurate may not produce better policies.
-
Russ Salakhutdinov
@rsalakhu
Document Grounded Generation – Machine Learning Blog | ML@CMU | Carnegie Mellon University blog.ml.cmu.edu/2021/06/25/document-grounded-generation/
-
Russ Salakhutdinov
@rsalakhu
New work on Robust Predictable Control.
Check out paper, code, as well as a short video by Ben Eysenbach:
ben-eysenbach.github.io/rpc/
Sergey Levine
@svlevine
What happens if, when we learn a model in RL, we reward the policy for doing things where the model is correct? Turns out that this is (surprisingly) the same as minimizing the number of bits of state that the policy needs access to.
Paper: ben-eysenbach.github.io/rpc/
Thread below: -
Russ Salakhutdinov reposted
Papers with Datasets
@paperswithdata
MultiBench: A unified large-scale benchmark for multimodal learning spanning 15 datasets, 10 modalities, 20 tasks, and 6 research areas.
It provides an automated end-to-end ML pipeline that standardizes data loading, experimental setup, and evaluation.
paperswithcode.com/dataset/multibench -
Russ Salakhutdinov reposted
Saurabh Garg
@saurabh_garg67
Visit us at ICML 2021
Spotlight: Tue 20 Jul 10 pm - 11 pm EDT
Poster: Tue 20 Jul 11 pm - 2 am EDT
icml.cc/virtual/2021/poster/9149
Work w/ fantastic mentors: Emilio Parisotto, Adarsh Prasad, @zicokolter, @zacharylipton, Siva Balakrishnan, @rsalakhu, Pradeep Ravikumar
(7/7) -
Russ Salakhutdinov
@rsalakhu
New work on measuring and mitigating social biases in pretrained language models: Defining several sources of representational biases and developing new benchmarks and metrics to measure them.
Code: github.com/pliang279/LM_bias
Paper: arxiv.org/abs/2106.13219
Paul Liang
@pliang279
Excited to share our new work on measuring and mitigating social biases in pretrained language models, to appear at #icml2021!
with Chiyu Wu, @lpmorency, @rsalakhu @mldcmu @LTIatCMU
check it out here:
paper: arxiv.org/abs/2106.13219
code: github.com/pliang279/LM_bias -
Russ Salakhutdinov reposted
Paul Liang
@pliang279
Check out our #naacl2021 paper on StylePTB: a compositional benchmark for fine-grained controllable text style transfer!
with Yiwei, @hai_t_pham, Ed Hovy, Barnabas, @rsalakhu @lpmorency @mldcmu @LTIatCMU
paper: arxiv.org/abs/2104.05196
code: github.com/lvyiwei1/StylePTB/
a thread: -
Russ Salakhutdinov
@rsalakhu
Is a good representation sufficient for sample-efficient offline RL? In practice, offline RL can be very sensitive to distribution shift even when using features from pretrained neural networks on the same task
arxiv.org/abs/2103.04947
w/t @RuosongW, Wu & @ShamKakade6 #icml2021 -
Russ Salakhutdinov reposted
Devendra Chaplot
@dchaplot
My Ph.D. Thesis on Building Intelligent Autonomous Navigation Agents is now available at:
arxiv.org/abs/2106.13415
Thesis Defense recording and slides:
youtu.be/rJ7tGT5cHtU
Thanks @rsalakhu for being an amazing advisor!
And I am very excited to join Facebook AI Research! -
Russ Salakhutdinov
@rsalakhu
Open Predicate Query Language (OPQL): constructing a virtual knowledge base (VKB) that supports KB reasoning & open-domain QA, tackling the incompleteness of knowledge bases by constructing a virtual KB only from text:
arxiv.org/abs/2102.07043
w/t @sun_haitian et. al., #icml2021 -
Russ Salakhutdinov
@rsalakhu
See, Hear, Explore: Curiosity via Audio-Visual Association – Machine Learning Blog | ML@CMU | Carnegie Mellon University blog.ml.cmu.edu/2021/06/11/see-hear-explore-curiosity-via…
-
Russ Salakhutdinov
@rsalakhu
New work on Self-supervised Representation Learning for speech recognition: matching/surpassing SOTA approaches for speech recognition, generation, and compression.
Paper: arxiv.org/abs/2106.07447
with W. Hsu, B. Bolte, H. Tsai, K. Lakhotia, A. Mohamed
ai.facebook.com/blog/hubert-self-supervised-representatio… -
Russ Salakhutdinov reposted
MineRL Project
@minerl_official
We are so excited to announce that the first round of the 2021 @NeurIPSConf MineRL Diamond Competition has officially started! 💎 Submit your agents here: www.aicrowd.com/challenges/neurips-2021-minerl-competition
We have some exciting new changes this year. Keep reading to learn more!
1/4This media couldn't be displayed. See it on the original post. -
Russ Salakhutdinov
@rsalakhu
#icml2021 Uncertainty Weighted Actor-Critic for Offline RL: Detect out-of-distribution state-action pairs & down-weight their contribution, preventing learning from overly optimistic targets that lie far away from training data distribution
arxiv.org/abs/2105.08140
w/t Wu et al. -
Russ Salakhutdinov reposted
Shrimai
@shrimai_
Join us in the presentation of "Case Study: Deontological Ethics in NLP" today (Jun 8) at 10.40pm ET. Joint work w/ Brendon Boldt, @rsalakhu, Alan Black.
Paper: aclweb.org/anthology/2021.naacl-main.297/
Video: youtube.com/watch?v=cnYhJ6h-i90
Slides: cs.cmu.edu/~sprabhum/docs/Case_Study.pdf
Poster: underline.io/events/122/posters/4233/poster/20588-case-st… -
Russ Salakhutdinov reposted
Shrimai
@shrimai_
Catch our work "Focused Attention Improves Document-Grounded Generation" at 12.00pm ET today (Jun 9) @NAACLHLT #naacl2021. Joint work w/ @SFResearch Kazuma Hashimoto, Yingbo Zhou, Alan Black, @rsalakhu
Paper: www.aclweb.org/anthology/2021.naacl-main.338/
Video: www.youtube.com/watch?v=qGiuFkqp9Hc -
Russ Salakhutdinov
@rsalakhu
Decentralized Multi-Robot Active Search – Machine Learning Blog | ML@CMU | Carnegie Mellon University blog.ml.cmu.edu/2021/06/04/decentralized-multi-robot-acti…
-
Russ Salakhutdinov reposted
Keyulu Xu
@KeyuluXu
3/3 GNNs are vulnerable to information leakage attacks on social networks etc. We propose GAL with minimax games to protect the sensitive information.
arxiv.org/abs/2009.13504
w/ @LiaoPeiyuan @hanzhao_ml @rsalakhu @StefanieJegelka -
Russ Salakhutdinov
@rsalakhu
Congratulations Paul!
Paul Liang
@pliang279
Extremely honored to have received a Facebook PhD Fellowship to support my research in socially intelligent AI! I am indebted to my advisors @lpmorency @rsalakhu at @mldcmu and my collaborators @brandondamos @_rockt @egrefen at @facebookai.
research.fb.com/blog/2021/04/announcing-the-recipients-of… -
Russ Salakhutdinov reposted
Paul Liang
@pliang279
Extremely honored to have received a Facebook PhD Fellowship to support my research in socially intelligent AI! I am indebted to my advisors @lpmorency @rsalakhu at @mldcmu and my collaborators @brandondamos @_rockt @egrefen at @facebookai.
research.fb.com/blog/2021/04/announcing-the-recipients-of… -
Russ Salakhutdinov
@rsalakhu
New work on Document-Grounded Text Generation, focusing on Wikipedia update generation and Dialogue response generation tasks.
Paper: arxiv.org/abs/2104.12714
Code: github.com/shrimai/Focused-Attention-Improves-Document-Gr…
Shrimai
@shrimai_
I'm very excited to announce that our work "Focused Attention Improves Document-Grounded Generation" w/ Kazuma Hashimoto @SFResearch, @yingbozhou_ai, Alan W Black, @rsalakhu has been accepted @NAACLHLT. #naacl2021
Paper: arxiv.org/pdf/2104.12714.pdf
Code: github.com/shrimai/Focused-Attention-Improves-Document-Gr… -
Russ Salakhutdinov
@rsalakhu
A Learning Theoretic Perspective on Local Explainability – Machine Learning Blog | ML@CMU | Carnegie Mellon University blog.ml.cmu.edu/2021/04/23/a-learning-theoretic-perspecti…
-
Russ Salakhutdinov reposted
AI at Meta
@AIatMeta
Excited to announce the expansion of a long-standing collaboration with @SCSatCMU. The FRAIM program gives PhD students at CMU a unique opportunity to perform their research in collaboration with their thesis adviser and a Facebook researcher. Learn more: ai.meta.com/blog/facebook-ai-and-facebook-reality-labs-an…
-
Russ Salakhutdinov reposted
Jure Leskovec
@jure
@Pinterest Labs presents a Distinguished lecture by Russ Salakhutdinov (@rsalakhu) of CMU: From Differentiable Reasoning to Semantic Visual Navigation. Thursday April 22 2021 4pm - 5pm PST. RSVP at pinterestlabstechtalkapril2021.splashthat.com
-
Russ Salakhutdinov
@rsalakhu
Congratulations to Shrimai Prabhumoye @shrimai_ for successfully defending her PhD on Controllable Text Generation @SCSatCMU.
Check out her papers, code, talks on controllable text generation focusing on style, content, structure & ethical considerations
www.cs.cmu.edu/~sprabhum
Shrimai
@shrimai_
I successfully defended my Ph.D. thesis today!!
Slides: www.cs.cmu.edu/~sprabhum/docs/thesis_defense.pdf
Many thanks to all my collaborators & committee members Yulia Tsvetkov & @jaseweston for guiding me. Special thanks to Alan W Black & @rsalakhu without whom this thesis would not have been possible! -
Russ Salakhutdinov
@rsalakhu
To my Russian-speaking friends:
После нескольких лет работы, моя жена издала свой первый роман в двух томах. Приключенческий детектив с элементами романтики и философии. Первая книга (том) вышла в печать:
mgraphics-books.com/product/mayak-book-one/
Вторая книга - на подходе. -
Russ Salakhutdinov reposted
Shrimai
@shrimai_
Wondering how to use ethical principles to evaluate NLP systems? Excited to announce new paper "Case study: Deontological ethics for NLP" w/ Brendon Boldt, @rsalakhu, Alan Black which presents 4 case studies has been accepted @NAACLHLT #naacl2021
Paper: arxiv.org/pdf/2010.04658.pdf -
Russ Salakhutdinov
@rsalakhu
New work on Efficient Transformers in RL using Actor-Learner Distillation:
Compressing online larger “Learner model” towards a tractable “Actor model” in distributed RL setting with partially-observable environments.
arxiv.org/abs/2104.01655
with E. Parisotto #iclr2021 -
Russ Salakhutdinov
@rsalakhu
Check out paper, code, datasets, and pretrained models for the winning entry of the CVPR2020 Habitat Object Navigation Challenge:
www.cs.cmu.edu/~dchaplot/projects/semantic-exploration.html
Devendra Chaplot
@dchaplot
We released the code, datasets, pre-trained models for our #neurips2020 paper on Object Goal Navigation and our winning entry to the #cvpr2020 Habitat ObjectNav challenge!
GitHub Code:
github.com/devendrachaplot/Object-Goal-Navigation
Webpage:
devendrachaplot.github.io/projects/semantic-exploration
w. @_dgandhi, A.Gupta, @rsalakhu -
Russ Salakhutdinov reposted
Devendra Chaplot
@dchaplot
We released the code, datasets, pre-trained models for our #neurips2020 paper on Object Goal Navigation and our winning entry to the #cvpr2020 Habitat ObjectNav challenge!
GitHub Code:
github.com/devendrachaplot/Object-Goal-Navigation
Webpage:
devendrachaplot.github.io/projects/semantic-exploration
w. @_dgandhi, A.Gupta, @rsalakhu -
Russ Salakhutdinov
@rsalakhu
New #iclr2021 paper on Self-supervised Learning with Relative Predictive Coding: A new contrastive learning objective that maintains a good balance between training stability, minibatch size sensitivity, & downstream task performance.
arxiv.org/abs/2103.11275
w/ H. Tsai et al. -
Russ Salakhutdinov
@rsalakhu
Check out #iclr2021 paper on Self-supervised Learning from a Multi-view Perspective: providing an information-theoretical framework to better understand the properties of self-supervised learning:
arxiv.org/abs/2006.05576v4
Code coming up soon
w/ H Tsai, @hanzhao_ml , Y Wu, et al -
Russ Salakhutdinov
@rsalakhu
Check out project website, including code:
ben-eysenbach.github.io/rce/
Google AI
@GoogleAI
Introducing a new #reinforcementlearning algorithm for teaching agents how to solve tasks by providing only examples of success, avoiding the need for hand-crafted reward functions while outperforming prior imitation learning approaches. Read more →goo.gle/3vVksX1
-
Russ Salakhutdinov reposted
Google AI
@GoogleAI
Introducing a new #reinforcementlearning algorithm for teaching agents how to solve tasks by providing only examples of success, avoiding the need for hand-crafted reward functions while outperforming prior imitation learning approaches. Read more →goo.gle/3vVksX1
-
Russ Salakhutdinov reposted
Devendra Chaplot
@dchaplot
Our Goal-Oriented Semantic Exploration system can also be transferred to the real-world on a LoCoBot!
Presentation:
neurips.cc/virtual/2020/protected/poster_2c75cf2681788ada…
w. D. Gandhi (@_dgandhi), A. Gupta, R. Salakhutdinov (@rsalakhu)
(2/2) -
Russ Salakhutdinov
@rsalakhu
Congratulations to Devendra Chaplot @dchaplot for
successfully defending his PhD on Building Autonomous Navigation Agents @SCSatCMU.
Check out his papers, code, talks on Active Neural SLAM, Neural Topological SLAM, Semantic Exploration, & many others:
www.cs.cmu.edu/~dchaplot/
Devendra Chaplot
@dchaplot
I successfully defended my Ph.D. Thesis!! 🎓🥂
Presentation Recording:
youtu.be/rJ7tGT5cHtU
Slides: www.cs.cmu.edu/~dchaplot/talks/Devendra_Chaplot_Thesis_De…
Thanks, @rsalakhu for being an amazing advisor! And thanks to all my collaborators and committee members for helping and guiding me throughout my PhD! -
Russ Salakhutdinov reposted
Devendra Chaplot
@dchaplot
I successfully defended my Ph.D. Thesis!! 🎓🥂
Presentation Recording:
youtu.be/rJ7tGT5cHtU
Slides: www.cs.cmu.edu/~dchaplot/talks/Devendra_Chaplot_Thesis_De…
Thanks, @rsalakhu for being an amazing advisor! And thanks to all my collaborators and committee members for helping and guiding me throughout my PhD! -
Russ Salakhutdinov
@rsalakhu
An Inferential Perspective on Federated Learning – Machine Learning Blog | ML@CMU | Carnegie Mellon University blog.ml.cmu.edu/2021/02/19/an-inferential-perspective-on-…
-
Russ Salakhutdinov reposted
Carnegie Mellon University
@CarnegieMellon
Wintertime ❄️
-
Russ Salakhutdinov reposted
william
@wgussml
Round 1 of the @NeurIPSConf MineRL Competition is over!
The algorithms developed under complex competition constraints like action obfuscation are incredible, and Round 2 starts on Jan 8th (we train teams algos from scratch)
Stay tuned: minerl.io/competition/
w/@rsalakhu -
Russ Salakhutdinov
@rsalakhu
Representational Aspects of Depth and Conditioning in Normalizing Flows – Machine Learning Blog | ML@CMU | Carnegie Mellon University blog.ml.cmu.edu/2020/12/18/representational-aspects-of-de…
-
Russ Salakhutdinov reposted
william
@wgussml
The NeurIPS 2020 MineRL Competition workshop is happening in half an hour (5 PST)!
Come stop by to see these incredible teams talk about their algos for solving the hard problem of sample efficient RL in Minecraft! w/ @rsalakhu @OriolVinyalsML & more! neurips.cc/virtual/2020/protected/e_competitions.html -
Russ Salakhutdinov
@rsalakhu
Neural Methods for Point-wise Dependency Estimation: quantitatively measuring how likely two outcomes cooccur with applications to Mutual Information estimation, self-supervised learning & cross-modal retrieval
arxiv.org/abs/2006.05553
w/t H. Tsai, @hanzhao_ml et al #neurips2020 -
Russ Salakhutdinov reposted
Carnegie Mellon University
@CarnegieMellon
We've created 5 new winter @zoom_us backgrounds and 2 new winter desktop backgrounds for your downloading pleasure. cmu.is/downloads
-
Russ Salakhutdinov reposted
Rahel Jhirad
@RahelJhirad
lovely talk
neurips.cc/virtual/2020/protected/workshop_16146.html
#neurips2020
Workshop: Self-Supervised Learning -- Theory and Practice
@rsalakhu
Geometric Capsule Autoencoders -
Russ Salakhutdinov reposted
Lisa Lee
@rl_agent
I'm excited to present Weakly-Supervised RL for Controllable Behavior at #neurips2020 today! Stop by our poster @ 9-11 pm PST in Session 7 (Deep Learning) D0-A0.
w/ B.Eysenbach @rsalakhu @shaneguML @chelseabfinn
Paper: arxiv.org/abs/2004.02860
Talk: neurips.cc/virtual/2020/protected/poster_1bd69c7df3112fb9…
Chelsea Finn
@chelseabfinn
Supervising RL is hard, especially if you want to learn many tasks. To address this, we present:
Weakly-Supervised Reinforcement Learning for Controllable Behavior
arxiv.org/abs/2004.02860
with @rl_agent, Eysenbach, @rsalakhu, @shaneguML @GoogleAI
thread 👇 -
Russ Salakhutdinov reposted
Sergey Levine
@svlevine
Today (Tue 12/8) check out Ben Eysenbach's long talk on HIPI: Rewriting History with Inverse RL at @NeurIPSConf, at 6 pm PT: neurips.cc/virtual/2020/protected/session_oral_21090.html
Find out how inverse RL allows us to formalize relabeling of rewards, improving learning of contextual policies.
more-> -
Russ Salakhutdinov reposted
Devendra Chaplot
@dchaplot
Excited to present our #neurips2020 paper on Object Goal Navigation using Goal-Oriented Semantic Exploration where we describe our winning entry to the #cvpr2020 Habitat ObjectNav Challenge!
Tue Dec 8, 12-2pm ET
PDF: arxiv.org/pdf/2007.00643.pdf
Web: www.cs.cmu.edu/~dchaplot/projects/semantic-exploration.html
(1/2) -
Russ Salakhutdinov reposted
Shrimai
@shrimai_
Watch our presentation "Exploring Controllable Text Generation Techniques" with Alan W Black and @rsalakhu on @LTIatCMU youtube channel or @coling2020. Live QA session will be tomorrow from 9.30-10.00am EST. #coling2020 #nlproc
Link: www.youtube.com/watch?v=khTYgGHLDqE -
Russ Salakhutdinov reposted
Tom M Mitchell
@tommmitchell
CMU Machine Learning is hiring - we're looking for great people at all levels to become new faculty colleagues. www.ml.cmu.edu/faculty-hiring.html
-
Russ Salakhutdinov reposted
Sergey Levine
@svlevine
C-Learning learns goal-conditioned policies using classifiers, without any hand-designed rewards. The theoretical framework in C-Learning helps to explain connections between prediction and Q-functions, relabeling, and others.
arxiv.org/abs/2011.08909
w/ Eysenbach, @rsalakhu -
Russ Salakhutdinov reposted
Kyunghyun Cho
@kchonyc
ha so people are reinventing supervised NCA by @rsalakhu & @geoffreyhinton from 2017 with a new name “supervised contrastive learning” proceedings.mlr.press/v2/salakhutdinov07a/salakhutdinov07…
-
Russ Salakhutdinov reposted
Shrimai
@shrimai_
Looking for a one-stop place to understand the various techniques used in controllable text generation? Check out our work on "Exploring Controllable Text Generation Techniques" with Prof. Alan W Black and @rsalakhu accepted @coling2020 #coling2020
Paper: arxiv.org/pdf/2005.01822.pdf -
Russ Salakhutdinov reposted
Martin Ma
@martinmamql
#emnlp2020 We propose Multimodal Routing, an interpretable method for multimodal language tasks such as sentiment analysis and emotion recognition. We will be in gather town on 11/16/2020 21:00 -23:00 EST.
arxiv.org/pdf/2004.14198.pdf with Y.Tsai, M.Yang, @rsalakhu and L.P. Morency. -
Russ Salakhutdinov reposted
Devendra Chaplot
@dchaplot
New paper on Unsupervised Domain Adaptation for Visual Navigation! We transfer navigation policies from simulation to the real-world using policy-based image translation.
Arxiv:
arxiv.org/abs/2010.14543
with S. Li, Y.H. Tsai, Y. Wu, L.P. Morency, R. Salakhutdinov @rsalakhu -
Russ Salakhutdinov reposted
Shrimai
@shrimai_
If you are applying for PhD @SCSatCMU or MLT @LTIatCMU, you can now get your application reviewed by students @SCSatCMU before you formally submit it! I'm very happy that my university is offering this program. To participate, please submit your application materials by Nov 9.
Language Technologies Institute | @CarnegieMellon
@LTIatCMU
Students from @SCSatCMU are offering feedback to potential applicants, particularly those from underrepresented groups on their applications to the PhD programs in SCS.
Apply by Nov 9!
Form: forms.gle/2EdsAFMwMqTMje9R7
Learn more: www.cs.cmu.edu/gasp -
Russ Salakhutdinov
@rsalakhu
Planning with Submodular Objective Functions: Instead of maximizing cumulative reward, maximize the objective induced by submodular function. Standard planning & submodular maximization can be viewed as special cases
arxiv.org/abs/2010.11863
w/ @RuosongW, Zhang, @dchaplot et al -
Russ Salakhutdinov reposted
Shrimai
@shrimai_
Dr. Bhuwan Dhingra (www.cs.cmu.edu/~bdhingra/) is starting NLP lab @duke_nlp @DukeU. We shared time @rsalakhu's lab @LTIatCMU. He has mny good ideas & is accepting stdnts for Fall 2021. Special focus: applicants from nontraditional backgrounds & underrepresented communities. Apply!
DukeNLP
@duke_nlp
We're very excited to be starting an #nlproc lab at Duke in Fall 2021, and we're looking for PhD students! We'd especially like to encourage applicants from nontraditional backgrounds and communities historically underrepresented in the field. Apply here: gradschool.duke.edu/admissions/application-instructions
-
Russ Salakhutdinov reposted
Carnegie Mellon University
@CarnegieMellon
Leaves are turning, students are learning. 🍂
-
Russ Salakhutdinov
@rsalakhu
Deontological Ethics in NLP: Studying generalization principle & respect for autonomy through informed consent, providing various case studies in NLP systems that have ethical implications under these principles:
Arxiv: arxiv.org/abs/2010.04658
w/ @shrimai_ B. Boldt & A. Black.
Shrimai
@shrimai_
New paper "Case Study: Deontological Ethics in NLP" with Brendon Boldt, @rsalakhu & Alan Black. Takeaways: 1) use ethical frameworks & apply principles to NLP systems like case studies discussed, 2) explore directions in paper to improve NLP systems.
Link: arxiv.org/pdf/2010.04658.pdf -
Russ Salakhutdinov
@rsalakhu
Graph Adversarial Networks: Protecting Information against Adversarial Attacks: Learning node representations by preserving information about the target task, while discarding sensitive info via min-max game:
arxiv.org/abs/2009.13504
w/ @LiaoPeiyuan @hanzhao_ml @KeyuluXu et al.
Han Zhao
@hanzhao_ml
New work: How to defend potential inference attacks on graphs? We propose a method via adversarial learning of representations for this purpose and analyze the potential tradeoff therein.
arXiv: arxiv.org/abs/2009.13504
code: github.com/liaopeiyuan/GAL -
Russ Salakhutdinov reposted
Shrimai
@shrimai_
New paper "Case Study: Deontological Ethics in NLP" with Brendon Boldt, @rsalakhu & Alan Black. Takeaways: 1) use ethical frameworks & apply principles to NLP systems like case studies discussed, 2) explore directions in paper to improve NLP systems.
Link: arxiv.org/pdf/2010.04658.pdf -
Russ Salakhutdinov reposted
Carnegie Mellon University
@CarnegieMellon
It's a beautiful day on campus. Get out and enjoy it. ☀️
-
Russ Salakhutdinov reposted
william
@wgussml
I'm excited to announce that the MineRL competition will be extended until February @ AAAI 2021! We have been absolutely amazed by the submissions so far and want to give teams the most time to solve this task!
Sign up now: minerl.io/competition
w/ @rsalakhu @minerl_official -
Russ Salakhutdinov
@rsalakhu
Generalizing Randomized Smoothing for Pointwise-Certified Defenses to Data Poisoning Attacks – Machine Learning Blog | ML@CMU | Carnegie Mellon University blog.ml.cmu.edu/2020/09/25/generalizing-randomized-smooth…
-
Russ Salakhutdinov reposted
Devendra Chaplot
@dchaplot
For students applying for graduate fellowships, I uploaded my research statement from the Facebook Fellowship application last year:
www.cs.cmu.edu/~dchaplot/misc/DevendraChaplot_Statement20…
I am happy to answer questions over email. Good luck! -
Russ Salakhutdinov reposted
CMU Kenneth C. Griffin School of Computer Science
@GriffinCMU
SCS undergraduate computer science ranked #1 in artificial intelligence, cybersecurity, mobile/web apps and software engineering by US News and World Report, #2 overall www.cmu.edu/about/rankings-and-awards
@CarnegieMellon -
Russ Salakhutdinov reposted
Carnegie Mellon University
@CarnegieMellon
A beautiful place 🍃
-
Russ Salakhutdinov reposted
RL Theory Virtual Seminars
@RLtheory
Our next talk:
09/08: Ruosong Wang (CMU)
"Provably Efficient Reinforcement Learning with General Value Function Approximation"
For details, please see the website:
sites.google.com/view/rltheoryseminars/next-seminar -
Russ Salakhutdinov reposted
Carnegie Mellon University
@CarnegieMellon
A quiet night ✨
-
Russ Salakhutdinov
@rsalakhu
F19 – Topics in data analysis – Machine Learning Blog | ML@CMU | Carnegie Mellon University blog.ml.cmu.edu/2020/08/31/f19-topics-in-data-analysis/
-
Russ Salakhutdinov
@rsalakhu
Excited to talk about our recent work on Geometric Capsules at #eccv2020 Structured Generative Models workshop: generativeperception.com
Learning to group 3D points into parts & parts into the whole object in unsupervised way.
arxiv.org/abs/1912.03310
with @nitishsr & @Hanlin -
Russ Salakhutdinov
@rsalakhu
High-Frequency Component Helps Explain the Generalization of Convolutional Neural Networks – Machine Learning Blog | ML@CMU | Carnegie Mellon University blog.ml.cmu.edu/2020/08/14/high-frequency-component-helps…
-
Russ Salakhutdinov reposted
Adam W. Harley
@AdamWHarley
We are organizing a workshop on "Perception Through Structured Generative Models" at #eccv2020. We have an awesome lineup of speakers: @PeterWBattaglia, @FidlerSanja, @jkevinoregan, @rsalakhu, @cvondrick, @wellingmax.
Submit your 4-page paper by August 20!
generativeperception.com -
Russ Salakhutdinov
@rsalakhu
Maintaining the Illusion of Reality: Transfer in RL by Keeping Agents in the DARC – Machine Learning Blog | ML@CMU | Carnegie Mellon University blog.ml.cmu.edu/2020/07/31/maintaining-the-illusion-of-re…
-
Russ Salakhutdinov reposted
Cyrus Rashtchian 🇰🇷 ICML
@CyrusRashtchian
Real image datasets are well-separated, and in retrospect, we expect them to be because they are clean and curated. So what's up with adversarial examples?! A thread #machinelearning #datascience [1/10]
-
Russ Salakhutdinov reposted
chilconference
@CHILconference
ACM CHIL 2020 is now complete!
Thanks again to the keynote speakers Yoshua Bengio, @ensoesie, @drnigam, @sherrirose and @rsalakhu, the tutorial-givers, the authors, the organisers, and all the attendees.
We look forward to seeing you all again for ACM CHIL 2021! -
Russ Salakhutdinov
@rsalakhu
Niagara Falls: Social Distancing: US vs Canada.
-
Russ Salakhutdinov reposted
Devendra Chaplot
@dchaplot
Our recent research is featured in CMU News! The article describes core ideas behind our winning entry to the #cvpr2020 Habitat ObjectNav Challenge in very simple terms.
For more details: arxiv.org/pdf/2007.00643.pdf
w. @_dgandhi, A.Gupta, @rsalakhu
CMU Kenneth C. Griffin School of Computer Science
@GriffinCMU
If you're looking for a couch, why are you searching the kitchen? @CarnegieMellon researchers developing common sense method that helps robots find their way around. www.cs.cmu.edu/news/which-way-fridge-common-sense-helps-r…
-
Russ Salakhutdinov reposted
CMU Kenneth C. Griffin School of Computer Science
@GriffinCMU
If you're looking for a couch, why are you searching the kitchen? @CarnegieMellon researchers developing common sense method that helps robots find their way around. www.cs.cmu.edu/news/which-way-fridge-common-sense-helps-r…
-
Russ Salakhutdinov
@rsalakhu
Which Way to the Fridge? Common Sense Helps Robots Navigate | Carnegie Mellon School of Computer Science www.cs.cmu.edu/news/which-way-fridge-common-sense-helps-r…
-
Russ Salakhutdinov reposted
Kavya Srinet
@kavyasrinet
We are really excited about Workshop on Learning in Artificial Open Worlds at ICML happening tomorrow July 18, 10 AM EST.
Come join us for amazing invited talks, fun virtual town poster session and panel discussion.
@katjahofmann @wgussml @noborux @rsalakhu B. Houghton A. Szlam -
Russ Salakhutdinov
@rsalakhu
In defense of weight-sharing for neural architecture search: an optimization perspective – Machine Learning Blog | ML@CMU | Carnegie Mellon University blog.ml.cmu.edu/2020/07/17/in-defense-of-weight-sharing-f…
-
Russ Salakhutdinov
@rsalakhu
Recent work on Reward-Free Reinforcement Learning: studying the question of whether it is possible to design provably efficient RL algorithms with linear function approximation in the reward-free setting.
arxiv.org/abs/2006.11274
w/t @RuosongW, @SimonShaoleiDu, @lyang36
Ruosong Wang
@RuosongW
How to explore an environment without the guidance of any reward function? In our recent work (arxiv.org/abs/2006.11274), we use UCB as the bonus to efficiently explore an unknown environment and find good policies for all possible reward functions in the linear MDP setting. 1/4
-
Russ Salakhutdinov reposted
Machine Learning Dept. at Carnegie Mellon
@mldcmu
Are you interested in pursuing a graduate program with a focus on machine learning and AI?
Learn more → www.ml.cmu.edu/academics/
#machinelearning #ml #artificialintelligence #ai #deeplearning #mldcmu #carnegiemellon #research #cmuai -
Russ Salakhutdinov reposted
Paul Liang
@pliang279
excited to present our paper on studying biases in sentence encoders at #acl2020nlp:
web: virtual.acl2020.org/paper_main.488.html
code: github.com/pliang279/sent_debias
also happy to take questions during the live Q&A sessions:
July 7 (14:00-15:00, 17:00-18:00 EDT)
w Irene, Emily, YC, @rsalakhu, LP -
Russ Salakhutdinov
@rsalakhu
Congratulations to Bhuwan Dhingra for completing PhD @SCSatCMU. Some of his work: Open Domain QA using fusion of text & KB, differentiable reasoning over Virtual KB, & many others
Thesis: End-to-End Learning with Text & KB
www.cs.cmu.edu/~bdhingra
co-advised w/t @professorwcohen
Russ Salakhutdinov
@rsalakhu
PhD Thesis: End-to-End Learning with Text & KB www.cs.cmu.edu/~bdhingra/docs/bhuwan_dhingra_thesis.pdf
-
Russ Salakhutdinov
@rsalakhu
New work on modular navigation model which builds semantic maps and learns goal-driven semantic policies for object goal navigation:
arxiv.org/abs/2007.00643
Winner of CVPR 2020 AI-Habitat ObjectNav Challenge.
Web: www.cs.cmu.edu/~dchaplot/projects/semantic-exploration.html
w. @dchaplot, @_dgandhi, A. Gupta -
Russ Salakhutdinov reposted
Jeremy Howard
@jeremyphoward
Amazing! 3 years ago, fast.ai alum @aman_madaan asked us: "could we apply style transfer to text?"
Now, he's done it, with @setlur_amrith @tparekh97 @gneubig @rsalakhu @shrimai_!
A model to "politeify" your text :)
Paper: arxiv.org/abs/2004.14257 -
Russ Salakhutdinov reposted
Stephanie Milani
@steph_milani
We have launched the 2020 @NeurIPSConf MineRL Competition in collaboration with @mldcmu, @OpenAI, @DeepMind, @MSFTResearch, @PreferredNet, and @aicrowdHQ! More info here: minerl.io/competition
-
Russ Salakhutdinov reposted
william
@wgussml
Excited to launch our #neurips2020 competition:
The MineRL Competition for Sample-Efficient Reinforcement Learning! w/
@rsalakhu @zacharylipton @svlevine @OriolVinyalsML @chelseabfinn @hardmaru @ancadianadragan & others!
Participate here! minerl.io/competition
Thread 👇 -
Russ Salakhutdinov reposted
TechCrunch
@TechCrunch
CMU researchers develop a an automatic politeness engine for text-based communications tcrn.ch/3dMODWf by @etherington
-
Russ Salakhutdinov reposted
Shrimai
@shrimai_
"Could Your Computer Please Be More Polite? Thank You": An @SCSatCMU article on our #acl2020nlp paper on politeness transfer with @aman_madaan, @setlur_amrith, @tparekh97, Barnabas Poczos, @gneubig, Yiming Yang, @rsalakhu and Alan W Black.
Article: www.cs.cmu.edu/news/could-your-computer-please-be-more-po… -
Russ Salakhutdinov
@rsalakhu
Could Your Computer Please Be More Polite? Thank You | Carnegie Mellon School of Computer Science www.cs.cmu.edu/news/could-your-computer-please-be-more-po…
-
Russ Salakhutdinov reposted
CMU College of Engineering
@CMUEngineering
“Every sunset brings the promise of a new dawn.” — Ralph Waldo Emerson
Photos were taken by David Cochran, @CMUengineering, Dean's Office. -
Russ Salakhutdinov
@rsalakhu
Code release for Off-Dynamics Reinforcement Learning: Training for Transfer with Domain Classifiers:
Code: github.com/google-research/google-research/tree/master/darc
Paper: arxiv.org/abs/2006.13916
via Benjamin Eysenbach
Russ Salakhutdinov
@rsalakhu
Can we learn a policy for a target domain only using experience from a source domain with different dynamics? -- Modify the reward function so that the source domain resembles target domain:
arxiv.org/abs/2006.13916
w/ B. Eysenbach, S. Asawa, S. Chaudhari,
@svlevine
Sergey Levine
@svlevine
Domain adaptation for RL: train in the source domain, but change the reward function so that it "looks" like the target domain in terms of dynamics. We call this "off-dynamics reinforcement learning"
w/ B. Eysenbach, S. Asawa, S. Chaudhari, @rsalakhu
arxiv.org/abs/2006.13916 -
Russ Salakhutdinov
@rsalakhu
#acl2020 paper on Politeness Transfer: A Tag and Generate Approach:
arxiv.org/abs/2004.14257
More importantly, the code and the dataset are available here:
github.com/tag-and-generate
Shrimai
@shrimai_
Our paper, "Politeness Transfer: A Tag and Generate Approach" with @aman_madaan, @setlur_amrith, @tparekh97, Barnabas Poczos, @gneubig, Yiming Yang, @rsalakhu and Alan W Black has been accepted to @aclmeeting #acl2020
Paper: arxiv.org/pdf/2004.14257.pdf
Repo: github.com/tag-and-generate -
Russ Salakhutdinov
@rsalakhu
Can we learn a policy for a target domain only using experience from a source domain with different dynamics? -- Modify the reward function so that the source domain resembles target domain:
arxiv.org/abs/2006.13916
w/ B. Eysenbach, S. Asawa, S. Chaudhari,
@svlevine
Sergey Levine
@svlevine
Domain adaptation for RL: train in the source domain, but change the reward function so that it "looks" like the target domain in terms of dynamics. We call this "off-dynamics reinforcement learning"
w/ B. Eysenbach, S. Asawa, S. Chaudhari, @rsalakhu
arxiv.org/abs/2006.13916 -
Russ Salakhutdinov reposted
Shrimai
@shrimai_
Our paper, "Politeness Transfer: A Tag and Generate Approach" with @aman_madaan, @setlur_amrith, @tparekh97, Barnabas Poczos, @gneubig, Yiming Yang, @rsalakhu and Alan W Black has been accepted to @aclmeeting #acl2020
Paper: arxiv.org/pdf/2004.14257.pdf
Repo: github.com/tag-and-generate -
Russ Salakhutdinov reposted
Kavya Srinet
@kavyasrinet
Hey all !
A reminder to submit to our ICML workshop if your research aligns with learning in artificial open worlds.
The deadline is June 22! 4 more days!!
william
@wgussml
I'm excited to announce that the ICML 2020 workshop on Learning in Artificial Open Worlds has extended the paper submission deadline to June 22!
Doing work on Minecraft or other open world environments!? Submit here: sites.google.com/view/icml-laow2020/home?authuser=0
w/ @rsalakhu @katjahofmann & more!This media couldn't be displayed. See it on the original post. -
Russ Salakhutdinov
@rsalakhu
Learning to Explore using Active Neural SLAM – Machine Learning Blog | ML@CMU | Carnegie Mellon University blog.ml.cmu.edu/2020/06/19/learning-to-explore-using-acti…
-
Russ Salakhutdinov reposted
Devendra Chaplot
@dchaplot
Presenting our #cvpr2020 paper on "Neural Topological SLAM for Visual Navigation" tomorrow. Join us for the live discussion!
Jun 18, 3pm-5pm PDT
Talk:
youtube.com/watch?v=GEghE1dXTdY
Slides:
cs.cmu.edu/~dchaplot/talks/CVPR20_Neural_Topological_SLAM…
Paper:
cs.cmu.edu/~dchaplot/papers/cvpr20_neural_topological_sla…
w.
@rsalakhu, A.Gupta, @_saurabhg -
Russ Salakhutdinov reposted
Machine Learning Dept. at Carnegie Mellon
@mldcmu
Congratulations to Devendra Chaplot, Dhiraj Gandhi, Abhinav Gupta, and Ruslan Salakhutdinov on winning the #cvpr2020 Habitat Challenge! A project organized by #fair @FacebookAI
Talk → mld.ai/g44
Slides → mld.ai/7cv
Paper (coming soon)
Devendra Chaplot
@dchaplot
We won the CVPR 2020 @ai_habitat ObjectNav Challenge!
Check out the talk describing our method at:
www.youtube.com/watch?v=QNTs6cdxOoU
Slides: www.cs.cmu.edu/~dchaplot/talks/CVPR20_Semantic_Exploratio…
Paper preprint coming soon!
w. D. Gandhi, A. Gupta, @rsalakhu -
Russ Salakhutdinov reposted
Devendra Chaplot
@dchaplot
We won the CVPR 2020 @ai_habitat ObjectNav Challenge!
Check out the talk describing our method at:
www.youtube.com/watch?v=QNTs6cdxOoU
Slides: www.cs.cmu.edu/~dchaplot/talks/CVPR20_Semantic_Exploratio…
Paper preprint coming soon!
w. D. Gandhi, A. Gupta, @rsalakhu -
Russ Salakhutdinov reposted
william
@wgussml
In anticipation of the (soon to be announced) MineRL 2020 competition, I thought I'd share the top AI from last year's:
(full vid) youtu.be/GHo8B4JMC38
Incredible what CDS was able to do with only ~8 million samples and 4 days of training.
w/ @rsalakhu @steph_milani & more -
Russ Salakhutdinov reposted
Sean Welleck
@wellecks
"Topological Sort for Sentence Ordering" arxiv.org/pdf/2005.00432.pdf
Orders sentences by predicting pairwise order constraints, e.g. with BERT
by @shrimai_, @rsalakhu, Alan Black -
Russ Salakhutdinov reposted
Tomasz Malisiewicz
@quantombone
Neural Topological SLAM for Visual Navigation. Really cool work on mapping via neural graphs!
www.cs.cmu.edu/~dchaplot/projects/neural-topological-slam… #computervision #robotics #cvpr2020 -
Russ Salakhutdinov reposted
Devendra Chaplot
@dchaplot
Our CVPR-20 paper "Neural Topological SLAM for Visual Navigation" now on arXiv. Our model uses topological representations to learn semantic priors and can be trained without RL or IL using only static image data.
Webpage:
devendrachaplot.github.io/projects/Neural-Topological-SLAM
PDF:
arxiv.org/abs/2005.12256 -
Russ Salakhutdinov
@rsalakhu
#cvpr2020 paper on Neural Topological SLAM for Visual Navigation: Building topological representations that can leverage semantics and perform approximate geometric reasoning:
Paper: arxiv.org/abs/2005.12256
Web: www.cs.cmu.edu/~dchaplot/projects/neural-topological-slam…
with @dchaplot , A. Gupta, S. Gupta -
Russ Salakhutdinov reposted
william
@wgussml
I'm excited to announce that the ICML 2020 workshop on Learning in Artificial Open Worlds has extended the paper submission deadline to June 22!
Doing work on Minecraft or other open world environments!? Submit here: sites.google.com/view/icml-laow2020/home?authuser=0
w/ @rsalakhu @katjahofmann & more!This media couldn't be displayed. See it on the original post. -
Russ Salakhutdinov
@rsalakhu
Differentiable Reasoning over Text – Machine Learning Blog | ML@CMU | Carnegie Mellon University blog.ml.cmu.edu/2020/05/15/differentiable-reasoning-over-…
-
Russ Salakhutdinov reposted
Kamalika Chaudhuri
@kamalikac
Blog post: Both robustness and accuracy may be possible to achieve.
w/ @yangarbiter @CyrusRashtchian @hongyangzh @rsalakhu
Russ Salakhutdinov
@rsalakhu
New blog post: Adversarial Robustness Through Local Lipschitzness:
ucsdml.github.io/jekyll/update/2020/05/04/adversarial-rob…
We posit that achieving both accuracy and robustness is possible by learning locally smooth classifiers.
Paper: arxiv.org/abs/2003.02460
w/t Y Yang, @CyrusRashtchian, H Zhang, @kamalikac -
Russ Salakhutdinov
@rsalakhu
New blog post: Adversarial Robustness Through Local Lipschitzness:
ucsdml.github.io/jekyll/update/2020/05/04/adversarial-rob…
We posit that achieving both accuracy and robustness is possible by learning locally smooth classifiers.
Paper: arxiv.org/abs/2003.02460
w/t Y Yang, @CyrusRashtchian, H Zhang, @kamalikac -
Russ Salakhutdinov reposted
Shrimai
@shrimai_
Our paper titled "Topological Sort for Sentence Ordering" with @rsalakhu and Alan W Black has been accepted to @aclmeeting #acl2020.
Paper: arxiv.org/pdf/2005.00432.pdf
Code: github.com/shrimai/Topological-Sort-for-Sentence-Ordering -
Russ Salakhutdinov
@rsalakhu
AI Infrastructure for Everyone, Now Open Source determined.ai/blog/ai-infrastructure-for-everyone/
-
Russ Salakhutdinov
@rsalakhu
Congratulations Radu @rvcraiu!
Dan Roy
@roydanroy
Congratulations to my colleague, Prof Radu Craiu (@rvcraiu), on being named a Fellow of the Institute of Mathematical Statistics (IMS). He joins a distinguished list at Toronto, incl. Nancy Reid (@reid_nancy), Jeff Rosenthal (@ProbabilityProf), Don Fraser, & Muni S. Srivastava.
-
Russ Salakhutdinov reposted
Devendra Chaplot
@dchaplot
Posted our ICLR-20 paper on Active Neural SLAM on arXiv. SOTA results for Visual Exploration, CVPR-19 Habitat Challenge Winner, Sim-to-real, code, pretrained models and more!
Webpage:
devendrachaplot.github.io/projects/Neural-SLAM
Code:
github.com/devendrachaplot/Neural-SLAM
PDF:
arxiv.org/abs/2004.05155 -
Russ Salakhutdinov
@rsalakhu
Learning to Explore using Active Neural SLAM: Modular hierarchical approach to learning policies for navigating in complex 3D environments #iclr2020:
Paper: arxiv.org/abs/2004.05155
Code, pretrained models: www.cs.cmu.edu/~dchaplot/projects/neural-slam.html
w/t @dchaplot, D. Gandhi, S. Gupta, A. Gupta -
Russ Salakhutdinov reposted
Paul Liang
@pliang279
Our work on investigating competitive influences on emergent communication in multi-agent teams, to appear at #aamas2020.
paper: arxiv.org/abs/2003.01848
code: github.com/pliang279/Competitive-Emergent-Communication
@rsalakhu @SatwikKottur @mldcmu @LTIatCMU
Russ Salakhutdinov
@rsalakhu
On Emergent Communication in Competitive Multi-Agent Teams: External competitive influence leads to faster emergence of communicative languages that are more informative and compositional: #aamas2020
arxiv.org/abs/2003.01848
w/t @pliang279 , J. Chen, LP Morency, S. Kottur -
Russ Salakhutdinov
@rsalakhu
Learning DAGs with Continuous Optimization – Machine Learning Blog | ML@CMU | Carnegie Mellon University blog.ml.cmu.edu/2020/04/10/learning-dags-with-continuous-…
-
Russ Salakhutdinov reposted
Paul Liang
@pliang279
A reminder to submit your work in multimodal ML, language, vision, speech, multimedia, affective computing, and applications to the workshop on multimodal language @ #acl2020!
multicomp.cs.cmu.edu/acl2020multimodalworkshop/
with fantastic speakers:
@radamihalcea @rsalakhu @ehsan_hoque @YejinChoinka -
Russ Salakhutdinov
@rsalakhu
On Emergent Communication in Competitive Multi-Agent Teams: External competitive influence leads to faster emergence of communicative languages that are more informative and compositional: #aamas2020
arxiv.org/abs/2003.01848
w/t @pliang279 , J. Chen, LP Morency, S. Kottur -
Russ Salakhutdinov reposted
Shane Gu
@shaneguML
Excellent work done by @rl_agent in collaboration with Ben, @rsalakhu, me and @chelseabfinn! Hope from now we do less (1) reward engineering, (2) collecting demos, or (3) distribution sketching (arxiv.org/abs/1911.02256)...
Chelsea Finn
@chelseabfinn
Supervising RL is hard, especially if you want to learn many tasks. To address this, we present:
Weakly-Supervised Reinforcement Learning for Controllable Behavior
arxiv.org/abs/2004.02860
with @rl_agent, Eysenbach, @rsalakhu, @shaneguML @GoogleAI
thread 👇 -
Russ Salakhutdinov reposted
Lisa Lee
@rl_agent
Instead of rewards or demos, can we use weak supervision to accelerate goal-conditioned RL & learn interpretable latent policies?
Excited to share our work from during my visit @Stanford & Google, with my amazing collaborators Ben Eysenbach, @rsalakhu @shaneguML & @chelseabfinn!
Chelsea Finn
@chelseabfinn
Supervising RL is hard, especially if you want to learn many tasks. To address this, we present:
Weakly-Supervised Reinforcement Learning for Controllable Behavior
arxiv.org/abs/2004.02860
with @rl_agent, Eysenbach, @rsalakhu, @shaneguML @GoogleAI
thread 👇 -
Russ Salakhutdinov reposted
Chelsea Finn
@chelseabfinn
Supervising RL is hard, especially if you want to learn many tasks. To address this, we present:
Weakly-Supervised Reinforcement Learning for Controllable Behavior
arxiv.org/abs/2004.02860
with @rl_agent, Eysenbach, @rsalakhu, @shaneguML @GoogleAI
thread 👇 -
Russ Salakhutdinov reposted
Microsoft Research
@MSFTResearch
The upcoming workshop on Learning in Artificial Open Worlds at @icmlconf has announced a call for papers. They're looking for work that focuses on, or is enabled by, machine learning in open worlds. Submissions are due by May 19, 2020: aka.ms/AA7xar3 #icml2020
-
Russ Salakhutdinov
@rsalakhu
Are Sixteen Heads Really Better than One? – Machine Learning Blog | ML@CMU | Carnegie Mellon University blog.ml.cmu.edu/2020/03/20/are-sixteen-heads-really-bette…
-
Russ Salakhutdinov reposted
Katja Hofmann
@katjahofmann
Thrilled to announce that our workshop proposal on "Learning in Artificial Open Worlds" has been accepted at #icml2020. We have amazing speakers and look forward to exciting contributed work. \w A. Szlam @rsalakhu @noborux @wgussml @kavyasrinet B. Houghton. Details coming soon.
-
Russ Salakhutdinov
@rsalakhu
Questions to Ask a Prospective Ph.D. Advisor on Visit Day, With Thorough and Forthright Explanations – Machine Learning Blog | ML@CMU | Carnegie Mellon University blog.ml.cmu.edu/2020/03/02/questions-to-ask-a-prospective…
-
Russ Salakhutdinov
@rsalakhu
Inherent Tradeoffs in Learning Fair Representations – Machine Learning Blog | ML@CMU | Carnegie Mellon University blog.ml.cmu.edu/2020/02/28/inherent-tradeoffs-in-learning…
-
Russ Salakhutdinov
@rsalakhu
Rewriting History with Inverse RL: Hindsight Inference for Policy Improvement: Generalizing goal-relabeling techniques based on the observation that hindsight relabeling is inverse RL:
arxiv.org/abs/2002.11089
Sergey Levine
@svlevine
Relabeling goals = inverse RL. This simple insight enables a whole family of inference-based relabeling algs that speed up goal-based RL and RL with other parameterized rewards.
"Rewriting History with Inverse RL," w/ B. Eysenbach, X. Geng, @rsalakhu
arxiv.org/abs/2002.11089 -
Russ Salakhutdinov
@rsalakhu
#iclr2020 paper on Differentiable Reasoning over a Virtual Knowledge Base: Efficient, end-to-end differentiable framework for doing complex multi-hop QA over a large text corpus.
arxiv.org/abs/2002.10640
w/t Dhingra, Zaheer, Balachandran, @gneubig , @professorwcohen -
Russ Salakhutdinov reposted
chilconference
@CHILconference
Registration is open for ACM CHIL 2020! Keynote speakers include Yoshua Bengio of MILA, Sherri Rose of Harvard (@sherrirose), Nigam Shah of Stanford, Ruslan Salakhutdinov of CMU (@rsalakhu), and Elaine Nsoesie of Boston University (@ensoesie).
Check out www.chilconference.org/registration/ -
Russ Salakhutdinov
@rsalakhu
#iclr2020 paper on Capsule Nets with Inverted Dot-Product Attention Routing: A new routing method where a child capsule is routed to a parent based on agreement b/w parent's state & child's vote:
arxiv.org/abs/2002.04764
Code github.com/apple/ml-capsules-inverted-attention-routing
w/t Tsai, @nitishsr, @Hanlin -
Russ Salakhutdinov
@rsalakhu
RL for Planning and Planning for RL – Machine Learning Blog | ML@CMU | Carnegie Mellon University blog.ml.cmu.edu/2020/02/13/rl-for-planning-and-planning-f…
-
Russ Salakhutdinov
@rsalakhu
Slides from my 2-part tutorial on Deep Learning Essentials that I gave at the ALT-ITA Joint Symposium:
www.cs.toronto.edu/~rsalakhu/ITA_ALT_Part1.pdf
www.cs.toronto.edu/~rsalakhu/ITA_ALT_Part2.pdf
alt2020.algorithmiclearningtheory.org/italt-joint-symposi… -
Russ Salakhutdinov reposted
Cyrus Rashtchian 🇰🇷 ICML
@CyrusRashtchian
Amazing set of tutorials today by @rsalakhu @roydanroy Matus Telgarsky, Alex Slivkins, Bobby Kleinberg, Peter Grunwald at the first #ita2020 + ALT2020 joint session (ITALT) alt2020.algorithmiclearningtheory.org/italt-joint-symposi…
-
Russ Salakhutdinov reposted
Cyrus Rashtchian 🇰🇷 ICML
@CyrusRashtchian
Plenary at #ita2020 by @rsalakhu -- importance of long term dependencies in deep networks, especially for language generation. But why? Needs some theory!
-
Russ Salakhutdinov reposted
Cyrus Rashtchian 🇰🇷 ICML
@CyrusRashtchian
I am presenting at #ita2020 on designing robust kNN, decision trees, and neural networks. Today at 11:40 in Robust Learning 2 session. (Co-authors include @kamalikac and @rsalakhu
who are plenary speakers today!) Based on new work with @hongyangzh & arxiv.org/abs/1906.03310 -
Russ Salakhutdinov
@rsalakhu
Got promoted to a full professor at CMU:) Thanks to my amazing students, collaborators, and my mentors.
CMU is a remarkable place, but what makes it truly unique is its brilliant students! Hard to find a better place to do ML and AI. -
Russ Salakhutdinov reposted
UCSD Engineering
@UCSDJacobs
The 2020 Information Theory and Applications Workshop kicks off on Sunday here in San Diego! Researchers from around the country are gathering to apply theory (ie deep learning, ML) to diverse areas of science and engineering. Schedule and more info: ita.ucsd.edu/ws/
-
Russ Salakhutdinov
@rsalakhu
Very excited to see two of my PhD students Devendra Singh Chaplot @dchaplot and Yao-Hung Hubert Tsai receive Facebook PhD fellowships.
Congratulations! www.cs.cmu.edu/news/four-scs-students-named-facebook-fellows -
Russ Salakhutdinov reposted
Paul Liang
@pliang279
Excited to announce the 2nd workshop on multimodal language @ #acl2020!
multicomp.cs.cmu.edu/acl2020multimodalworkshop/
We welcome submissions in all areas of human language, multimodal ML, multimedia, affective computing, and applications!
w/ fantastic speakers:
@radamihalcea @rsalakhu @ehsan_hoque -
Russ Salakhutdinov reposted
Gergely Neu
@neu_rips
***spread the word!!!***
ITA and ALT team up for a joint symposium day on Feb 8!!
Tutorials by Russ Salakhutdinov (@rsalakhu), Alex Slivkins & Bobby Kleinberg
Talks by Peter D. Grünwald and Cosma Shalizi (@cshalizi)
Register to ALT and attend:
www.eventbrite.com/e/alt-2020-the-31st-international-conf… -
Russ Salakhutdinov
@rsalakhu
Graduate School Applications at the CMU School of Computer Science @SCSatCMU:
"60% of apps for the *entire school* mention the phrase "machine learning" at least once" via @andy_pavlo
Andy Pavlo (@andypavlo.bsky.social)
@andy_pavlo
It's that time again! Here are the percentages of @SCSatCMU grad school applications (MS + PhD) that contain certain keywords. I now have two graphs: machine learning + databases.
60% of apps for the *entire school* mention the phrase "machine learning" at least once. -
Russ Salakhutdinov
@rsalakhu
Yoshua Bengio's response to @GaryMarcus about their debate.
docs.google.com/document/d/1P9YyZ4xEDO98qPTa1Al4RnTqxj3mM… -
Russ Salakhutdinov reposted
Animesh Garg
@animesh_garg
Christmas morning academia quotes!
You can either try to build it for the next decade
or ...
just keep talking about it but still claim "thought leadership" if/when things start working in three decades -
Russ Salakhutdinov reposted
Lisa Lee
@rl_agent
Video & slides for LIRE workshop @ #neurips2019 are now up: sites.google.com/view/neurips2019lire/schedule
Check out the Talks and Panel by @RaiaHadsell @tommmitchell Jeff Bilmes @pabbeel @YejinChoinka Tom Griffiths & more.
Thanks to all speakers & presenters for making the workshop a success! -
Russ Salakhutdinov reposted
Machine Learning Dept. at Carnegie Mellon
@mldcmu
From all of us at the Machine Learning Department at CMU, we wish you a happy holiday season and a prosperous New Year!🤗
-
Russ Salakhutdinov reposted
CMU Kenneth C. Griffin School of Computer Science
@GriffinCMU
A leader in robotics research and education, @CMU_Robotics is recruiting faculty in all areas of expertise. Deadline is Jan. 3 www.ri.cmu.edu/about/hiring-faculty-positions/
-
Russ Salakhutdinov reposted
Katja Hofmann
@katjahofmann
The MineRL competition session is about to start - so exciting!! It’s been amazing to work with such a brilliant team, and see the fantastic progress and achievements of the participants!
-
Russ Salakhutdinov reposted
william
@wgussml
Wow, I am so excited! The #minerl competition was featured in The Verge!
We've come a long way towards Minecraft AI in the past 2 years, but this competition is a great reminder that there are so many more challenges to solve!
Read More: minerl.io
The Verge
@verge
AI has bested chess and Go, but it struggles to find a diamond in Minecraft www.theverge.com/2019/12/13/21020230/ai-minecraft-minerl-…
-
Russ Salakhutdinov reposted
Zhiting Hu
@ZhitingHu
Come join the #neurips2019 workshop on Learning with Rich Experience. Note the location: West 208+209.
Look fwd to the super exciting talks by @RaiaHadsell @tommmitchell JeffBilmes @pabbeel @YejinChoinka & TomGriffiths, and the contributed presentations: sites.google.com/view/neurips2019lire/accepted-papers -
Russ Salakhutdinov
@rsalakhu
BBC article about our #minerl #neurips2019 competition:
www.bbc.com/news/technology-50720823
Come and check out the winners and talks from the top competitors:
Sat Dec 14 9:00 AM @ West 116 + 117
w/ @wgussml , @PineappleML, @steph_milani, @katjahofmann -
Russ Salakhutdinov reposted
Thang Luong
@lmthang
Our MixTape is 3.5-10.5x faster than Mixture of Softmaxes /w SOTA results in language modeling & translation. Key is to do gating in the logit space but with vectors instead of scalars (+sigmoid tree decomposition & gate sharing for efficiency). /w Zhilin, Russ, Quoc #neurips2019
Russ Salakhutdinov
@rsalakhu
Mixtape: breaking the softmax bottleneck that limits expressiveness of neural language models.
A network with Mixtape Output Layer is only 35% slower than softmax-based network, while outperforming softmax in perplexity & translation quality #neurips2019
papers.nips.cc/paper/9723-mixtape-breaking-the-softmax-bo… -
Russ Salakhutdinov
@rsalakhu
Mixtape: breaking the softmax bottleneck that limits expressiveness of neural language models.
A network with Mixtape Output Layer is only 35% slower than softmax-based network, while outperforming softmax in perplexity & translation quality #neurips2019
papers.nips.cc/paper/9723-mixtape-breaking-the-softmax-bo… -
Russ Salakhutdinov reposted
Hanlin Goh
@Hanlin
Check out our latest work on modeling the "what's" and "where's" of objects and parts with geometric capsule representations from 3D point clouds. arxiv.org/abs/1912.03310 (with @nitishsr & @rsalakhu)... I'm at #neurips2019 all week if you want to chat about it!
-
Russ Salakhutdinov
@rsalakhu
Capsules may replace ConvNets one day.
Nitish Srivastava
@nitishsr
Happy to share our recent work, "Geometric Capsule Autoencoders for 3D Point Clouds." The main idea is that instead of finding agreement among parts of an object, we find agreement among different views of the object. arxiv.org/abs/1912.03310
-
Russ Salakhutdinov reposted
Nitish Srivastava
@nitishsr
Happy to share our recent work, "Geometric Capsule Autoencoders for 3D Point Clouds." The main idea is that instead of finding agreement among parts of an object, we find agreement among different views of the object. arxiv.org/abs/1912.03310
-
Russ Salakhutdinov
@rsalakhu
New work on Geometric Capsules: Learning to group 3D points into parts & parts into the whole object in unsupervised way. Each capsule represents a visual entity consisting of a pose & feature representing "where" & ''what'' it is.
arxiv.org/abs/1912.03310
w/t @nitishsr & @Hanlin -
Russ Salakhutdinov reposted
william
@wgussml
Hey, I'm at ⓃⓔⓤⓡⒾⓅⓈ 2019 in Vancouver!
Come check out the amazing work of competitors in the #minerl comp! Excited to award several GPUs from NVIDIA to the winners
Sat Dec 14 9:00 AM @ West 116 + 117
w/ @rsalakhu @PineappleML @steph_milani @katjahofmann
#neurips2019This media couldn't be displayed. See it on the original post. -
Russ Salakhutdinov reposted
chilconference
@CHILconference
Happy to announce our keynote speakers for CHIL 2020: Yoshua Bengio, Sherri Rose (@sherrirose), Nigam Shah (@drnigam), and
Ruslan Salakhutdinov (@rsalakhu).
A reminder that the deadline for papers is in *just over a month* (13th January), see our CFP: www.chilconference.org/call-for-papers/ -
Russ Salakhutdinov reposted
Paul Liang
@pliang279
Excited to present Deep Gamblers: Learning to Abstain with Portfolio Theory at #neurips2019! Strong results for uncertainty estimation, learning from noisy data and labels.
with Ziyin, Zhikang, @rsalakhu, LP, Masahito
paper: arxiv.org/abs/1907.00208
code: github.com/Z-T-WANG/NIPS2019DeepGamblers -
Russ Salakhutdinov reposted
AI Business
@Business_AI
The #aisummit #newyork is very pleased to announce @rsalakhu, UPMC Professor of Computer Science at @CarnegieMellon & Director of #ai Research at @Apple, as a shortlisted nominee for #aiinnovatoroftheyear 2019 🏅 Cast your vote for Ruslan here → get.knect365.com/new-york-ai-innovator-of-the-year/
-
Russ Salakhutdinov reposted
Kyunghyun Cho
@kchonyc
@GaryMarcus I would like in fact to hear a solid and testable proposal based on your 20 years of insisting how ML is not on the right track (as you said in an earlier tweet in this week), instead of another essay.
Gary Marcus
@GaryMarcus
Indeed. In fact there's been so much misinformation lately I've written a new essay about that. Watch for it tomorrow, at @gradientpub
This post is unavailable. See it on X. -
Russ Salakhutdinov reposted
nature
@Nature
The MineRL competition encourages coders to devise programs that learn by example. idp.nature.com/transit?redirect_uri=https%3A%2F%2Fwww.nat…
-
Russ Salakhutdinov
@rsalakhu
Very exciting to see that the #minerl project started by my PhD student @wgussml and research scientist @PineappleML at CMU @mldcmu almost 2 years ago has been featured on Nature news:
www.nature.com/articles/d41586-019-03630-0
minerl.io/competition/
Congrats!
william
@wgussml
I am really happy to announce that MineRL has been featured on Nature news!
This means so much to us, after nearly two years of work w/o much external feedback!
w/ @rsalakhu @PineappleML @katjahofmann @flippnflops @MSFTResearch & more!!
www.nature.com/articles/d41586-019-03630-0 -
Russ Salakhutdinov
@rsalakhu
RoboNet: A Dataset for Large-Scale Multi-Robot Learning – Blog | Machine Learning | Carnegie Mellon University blog.ml.cmu.edu/2019/11/26/robonet/
-
Russ Salakhutdinov reposted
Machine Learning Dept. at Carnegie Mellon
@mldcmu
Are you passionate about teaching, with experience in CS, math, physics, or engineering, and hoping to get involved in #machinelearning?
We are currently hiring two Education Associates at @mldcmu, learn more: mld.ai/ed8x
#hiring #education -
Russ Salakhutdinov
@rsalakhu
Code release for #neurips2019 paper on Learning Data Manipulation: Learning to augment and re-weight data in low data regime or in presence of imbalanced labels.
arxiv.org/abs/1910.12795
Code: github.com/tanyuqian/learning-data-manipulation
via @ZhitingHu & Bowen Tan. -
Russ Salakhutdinov
@rsalakhu
ARUBA: Learning-to-Learn with Less Regret – Blog | Machine Learning | Carnegie Mellon University blog.ml.cmu.edu/2019/11/22/aruba/
-
Russ Salakhutdinov
@rsalakhu
Join Us - Machine Learning | CMU - Carnegie Mellon University www.ml.cmu.edu/join-us/
-
Russ Salakhutdinov reposted
Machine Learning Dept. at Carnegie Mellon
@mldcmu
We are hiring!
Multiple tenure track, teaching track, and staff positions
Apply → mld.ai/hiring
#hiring #machinelearning #artificialintelligence #carnegiemellon #ai #ml #deeplearning #faculty #professor #ia #datascience #engineering #devops -
Russ Salakhutdinov reposted
Machine Learning Dept. at Carnegie Mellon
@mldcmu
Professor Russ Salakhutdinov @rsalakhu, at @mldcmu, is a nominee for the #aiinnovatoroftheyear 2019 🏅. Help us by casting your vote at mld.ai/ai3d
AI Business
@Business_AI
The #aisummit #newyork is very pleased to announce @rsalakhu, UPMC Professor of Computer Science at @CarnegieMellon & Director of #ai Research at @Apple, as a shortlisted nominee for #aiinnovatoroftheyear 2019 🏅 Cast your vote for Ruslan here → get.knect365.com/new-york-ai-innovator-of-the-year/
-
Russ Salakhutdinov reposted
AI Business
@Business_AI
The #aisummit #newyork is very pleased to announce @rsalakhu, UPMC Professor of Computer Science at @CarnegieMellon & Director of #ai Research at @Apple, as a shortlisted nominee for #aiinnovatoroftheyear 2019 🏅 Cast your vote for Ruslan here → get.knect365.com/new-york-ai-innovator-of-the-year/
-
Russ Salakhutdinov reposted
Machine Learning Dept. at Carnegie Mellon
@mldcmu
A fantastic panel discussion on the #academic job market took place on November 14 by junior faculty members @SCSatCMU who recently went through the #computerscience academic #jobmarket about their experiences, tips, and useful insights.
#cs #academia #hiring #faculty #research -
Russ Salakhutdinov reposted
Radu Craiu
@rvcraiu
Found on the web
-
Russ Salakhutdinov
@rsalakhu
Federated Learning: Challenges, Methods, and Future Directions – Blog | Machine Learning | Carnegie Mellon University blog.ml.cmu.edu/2019/11/12/federated-learning-challenges-…
-
Russ Salakhutdinov
@rsalakhu
#corl2019 paper from our Apple group on Worst Cases Policy Gradients: Learning more robust policies by minimizing long-tail risks, reducing the likelihoods of bad outcomes.
arxiv.org/abs/1911.03618
with Charlie Tang and Jian Zhang. -
Russ Salakhutdinov
@rsalakhu
Code release for #neurips2019 paper on Graph Neural Tangent Kernel: Fusing Graph Neural Networks with Graph Kernels: arxiv.org/abs/1905.13192
Code: github.com/KangchengHou/gntk
via @SimonShaoleiDu
Simon Shaolei Du
@SimonShaoleiDu
We have released code for computing Graph Neural Tangent Kernel (GNTK) used in our paper "Graph Neural Tangent Kernel: Fusing Graph Neural Networks with Graph Kernels", which will appear in NeurIPS 2019. Paper: arxiv.org/abs/1905.13192. Code: github.com/KangchengHou/gntk
-
Russ Salakhutdinov reposted
Tim Cook
@tim_cook
At Apple, privacy is built into everything we make. You decide what you share, how you share it, and who you share it with. Here’s how we protect your data. www.apple.com/privacy/
-
Russ Salakhutdinov reposted
william
@wgussml
Neural networks playing @Minecraft, so exciting. :')
Here's a recent run of @Tviskaron's agent in the #minerl comp, using option extraction to solve the Obtain Diamond task with hierarchical DQfD models.
w/ @rsalakhu @MSFTResearch & more!
Website: minerl.io -
Russ Salakhutdinov
@rsalakhu
Enhanced Convolutional Neural Tangent Kernels: Modifying the kernel using Local Average Pooling, which incorporates data augmentation into the computation of the kernel:
arxiv.org/abs/1911.00809
with Z. Li, @RuosongW , D. Yu, @SimonShaoleiDu, W. Hu, and @prfsanjeevarora
Sanjeev Arora
@prfsanjeevarora
Matching Alexnet performance (89%) on CIFAR10 using kernel method. Excluding deep nets, previous best was 86% (Mairal NIPS'16). Key Ideas: convolutional NTK + Coates-Ng random patches layer + way to fold data augmentation into kernel defn arxiv.org/abs/1911.00809
-
Russ Salakhutdinov
@rsalakhu
#neurips2019 paper on Multiple Futures Prediction: Sequential generative model that learns multi-step future motions/interactions of agents directly from multi-agent trajectory data, while remaining scalable to a large number of agents
arxiv.org/abs/1911.00997
w/t C. Tang @ Apple -
Russ Salakhutdinov reposted
Roger Grosse
@RogerGrosse
This is a phenomenon we also found from evaluating GAN likelihoods. Evaluating GAN likelihoods is computationally challenging, but we can learn a lot from it!
arxiv.org/abs/1611.04273 -
Russ Salakhutdinov
@rsalakhu
#neurips2019 paper on Learning Neural Networks with Adaptive Regularization: arxiv.org/abs/1907.06288
More importantly, code: github.com/yaohungt/Adaptive-Regularization-Neural-Network
Slides: www.cs.cmu.edu/~hzhao1/papers/NIPS2019/adareg_slides.pdf
with @HanZhao_Keira, Hubert Tsai, Geoff Gordon
Han Zhao
@hanzhao_ml
Interested in learning neural nets in a data-dependent and adaptive way? Check out our recent work to appear @ NeurIPS-19 on this topic! TL; DR: maintain two additional covariance matrices for a fully-connected layer and use them to regularize the model. @rsalakhu @mldcmu
-
Russ Salakhutdinov reposted
bill peduto
@billpeduto
Ribbon cutting this morning as the future of our city & our region continues to build.
Carnegie Mellon University
@CarnegieMellon
When two world-leading institutions (@CMUEngineering and @ANSYS) focused on engineering excellence exist in the same city, what do you do? You collaborate. Welcome to ANSYS Hall.
-
Russ Salakhutdinov reposted
Zhiting Hu
@ZhitingHu
The paper “transfers” an off-the-shelf _reward_ learning algorithm to learning _data_ manipulation
It’s a powerful idea--transferring solutions to problems in one context to problems in another. Used in learning structured knowledge arxiv.org/abs/1806.09764, improving GANs/VAEs
Russ Salakhutdinov
@rsalakhu
#neurips2019 paper on Learning Data Manipulation: Learning to augment and re-weight data for improved training, especially in low data regime or in presence of imbalanced labels.
arxiv.org/abs/1910.12795
w/t Zhiting Hu, Bowen Tan et. al. -
Russ Salakhutdinov
@rsalakhu
Very impressive! With hierarchical RL & imitation learning, an agent can learn rather complex behaviours: chop trees, use crafting tables, create wood & stone pickaxes, mine stone/coal, make furnaces.
via @wgussml, #neurips2019 Obtain Diamond Competition minerl.io
william
@wgussml
Round 1 of the #minerl Obtain Diamond competition just came to a close. Very impressed with the progress of the teams!
Here's the current top submission utilizing hierarchical RL and imitation learning!
Stay tuned: minerl.ioThis media couldn't be displayed. See it on the original post. -
Russ Salakhutdinov
@rsalakhu
#neurips2019 paper on Learning Data Manipulation: Learning to augment and re-weight data for improved training, especially in low data regime or in presence of imbalanced labels.
arxiv.org/abs/1910.12795
w/t Zhiting Hu, Bowen Tan et. al. -
Russ Salakhutdinov reposted
william
@wgussml
Round 1 of the #minerl Obtain Diamond competition just came to a close. Very impressed with the progress of the teams!
Here's the current top submission utilizing hierarchical RL and imitation learning!
Stay tuned: minerl.ioThis media couldn't be displayed. See it on the original post. -
Russ Salakhutdinov reposted
Stefani Pashman
@StefaniPashman
One of the best parts of living in #pittsburgh is being able to walk across the campus of a world class university on a beautiful fall day @CarnegieMellon @CMUenergy
-
Russ Salakhutdinov reposted
Thomas G. Dietterich
@tdietterich
I just published What does it mean for a machine to ‘understand’? link.medium.com/vvNZ4pks80
1/ -
Russ Salakhutdinov reposted
Sham Kakade
@ShamKakade6
Very cool! Years ago (a little post AlexNet) I put in (too?) many cycles trying to design such a kernel. Didn't match their performance (though didn't have much compute back then). Pretty slick that they use a derived kernel from a ConvNet!
Ruosong Wang
@RuosongW
We have released code for computing Convolutional Neural Tangent Kernel (CNTK) used in our paper "On Exact Computation with an Infinitely Wide Neural Net", which will appear in NeurIPS 2019.
Paper: arxiv.org/abs/1904.11955
Code: github.com/ruosongwang/CNTK -
Russ Salakhutdinov reposted
Ruosong Wang
@RuosongW
We have released code for computing Convolutional Neural Tangent Kernel (CNTK) used in our paper "On Exact Computation with an Infinitely Wide Neural Net", which will appear in NeurIPS 2019.
Paper: arxiv.org/abs/1904.11955
Code: github.com/ruosongwang/CNTK -
Russ Salakhutdinov
@rsalakhu
Path Length Bounds for Gradient Descent – Blog | Machine Learning | Carnegie Mellon University blog.ml.cmu.edu/2019/10/25/path-length-bounds-for-gradien…
-
Russ Salakhutdinov reposted
Machine Learning Dept. at Carnegie Mellon
@mldcmu
Check out the 2020 Program for Women and Mathematics!
The Mathematics of Machine Learning
Date: May 16-22, 2020
Application Deadline: February 17, 2020
Learn more: mld.ai/m2x9d
#machinelearning #ml #math #stem #womeninmath #womeninstem -
Russ Salakhutdinov
@rsalakhu
"My favorite Gary Marcus quote", via Geoffrey Hinton:)
www.cs.toronto.edu/~hinton/marcusquote.html -
Russ Salakhutdinov reposted
Roger Grosse
@RogerGrosse
Quantum computing researchers are debating fundamental science while AI researchers are stuck arguing if the word "solve" maybe gave some people the wrong impression.
Tom Wong
@thomasgwong
IBM claims to be able to simulate Google's leaked quantum supremacy result using a classical supercomputer in 2.5 days rather than 10,000 years. HT Scott Aaronson. arxiv.org/abs/1910.09534
-
Russ Salakhutdinov reposted
CMU AI+
@cmu_aiplus
CMU AI+ brings together researchers, entrepreneurs and domain experts at @CarnegieMellon with a shared vision of an AI-powered world. Think AI + transportation, healthcare, education, law enforcement, finance, and more. Join us and explore a better future: cmu.aiplus.club
-
Russ Salakhutdinov reposted
Machine Learning Dept. at Carnegie Mellon
@mldcmu
We are hiring!
Multiple tenure track, teaching track, and staff positions
Apply → mld.ai/hiring
#hiring #machinelearning #artificialintelligence #carnegiemellon #ai #ml #deeplearning #faculty #professor #ia #datascience #engineering #devops -
Russ Salakhutdinov
@rsalakhu
Don’t be, @ilyasut, keep doing amazing work! Critics will always be there.
Ilya Sutskever
@ilyasut
Surprised and saddened by all the bad faith criticism of our robotic manipulation result reddit.app.link/drbZ4Rq2W0
-
Russ Salakhutdinov reposted
Carnegie Mellon University
@CarnegieMellon
-
Russ Salakhutdinov
@rsalakhu
This is terrific news for @UofTCompSci . I remember that not too long ago CS would not hire ML and Deep Learning faculty — it was considered to be too risky.
U of T News
@UofTNews
#uoft, @VectorInst create new deep learning faculty positions in honour of @UofTCompSci's Geoffrey Hinton www.utoronto.ca/news/u-t-vector-institute-create-new-deep…
-
Russ Salakhutdinov reposted
-
Russ Salakhutdinov
@rsalakhu
Using large Transformer-XLs for stable training of RL agents. Nice work from Emilio Parisotto’s internship and colleagues at DeepMind.
Max Jaderberg
@maxjaderberg
Finally, Transformers working for RL! Two simple modifications: move layer-norm and add gating creates GTrXL: an incredibly stable and effective architecture for integrating experience through time in RL.
Great work from Emilio interning at @DeepMindAI arxiv.org/abs/1910.06764 -
Russ Salakhutdinov reposted
Sid Jayakumar
@sidfix
Really excited about our latest work showing that large Transformer-XLs can be used in RL agents. We show SoTA performance on DMLab with gated transformers and a few small changes. Led by Emilio as an internship project! @DeepMindAI
Max Jaderberg
@maxjaderberg
Finally, Transformers working for RL! Two simple modifications: move layer-norm and add gating creates GTrXL: an incredibly stable and effective architecture for integrating experience through time in RL.
Great work from Emilio interning at @DeepMindAI arxiv.org/abs/1910.06764 -
Russ Salakhutdinov
@rsalakhu
#neurips2019 paper on Graph Neural Tangent Kernels: Fusing Graph Neural Networks (GNNs) with Graph Kernels -- Infinitely wide multi-layer GNNs that enjoy the full expressive power of GNNs and inherit advantages of Graph Kernels: arxiv.org/abs/1905.13192
w/t @SimonShaoleiDu et al. -
Russ Salakhutdinov reposted
UCI Center for Machine Learning & Intelligent Sys
@UCI_CMLIS
Bhuwan Dhingra from @LTIatCMU will be our next AI/ML speaker, presenting his work on "Text as a Virtual Knowledge Base" at 1pm on 10/14 in DBH 4011. See details, and rest of the schedule here: cml.ics.uci.edu/aiml/
-
Russ Salakhutdinov
@rsalakhu
Check out #neurips2019 workshop on Sets and Partitions, focusing on models with set-based inputs/outputs, models of partitions and novel clustering methodology:
www.sets.parts -
Russ Salakhutdinov
@rsalakhu
A Fair Comparison Study of XLNet and BERT with Large Models: medium.com/@xlnet.team/a-fair-comparison-study-of-xlnet-a…
-
Russ Salakhutdinov
@rsalakhu
1/4 ICML 2019: Some Changes and Call for Papers:
Please take a look at the three major changes that @kamalikac and I are introducing to ICML 2019 this year:
hunch.net?p=10858918Post not available here, view on X
Russ Salakhutdinov
@rsalakhu
4/4: To foster reproducibility, we encourage code submission. Our submission form will have two optional supplementary files: a regular supp manuscript and code. Reproducibility of results and easy accessibility of code will be taken into account in the decision-making process.
-
Russ Salakhutdinov
@rsalakhu
CMU is now offering B.S. in Artificial Intelligence! www.ml.cmu.edu/news/news-archive/2018/may/cmu-launches-bs…
-
Russ Salakhutdinov
@rsalakhu
Apple ML new post: On-device Deep Neural Network for Face Detection.
Running face detection on device, preserving user privacy while running efficiently.
machinelearning.apple.com/2017/11/16/face-detection.html -
Russ Salakhutdinov
@rsalakhu
New paper on Deep Sets: Designing objective functions defined on finite sets, by Manzil Zaheer, et.al #nips2017 oral arxiv.org/abs/1703.06114
-
Russ Salakhutdinov
@rsalakhu
Apple ML Journal new posts: Deep Learning for Siri’s Voice, Improving Acoustic Models, & Inverse Text Normalization machinelearning.apple.com
-
Russ Salakhutdinov
@rsalakhu
Releasing BookCorpus: 11,038 books dataset that we used to train skip-thought vectors
www.cs.toronto.edu/~mbweb/
Encoder github.com/ryankiros/skip-thoughts -
Russ Salakhutdinov
@rsalakhu
Check out my (somewhat) recent KDD tutorial on deep nets, RBMs, DBNs, DBMs, and multimodal deep learning:
videolectures.net/kdd2014_salakhutdinov_deep_learning/









































































































































































































































































































































































