Standard PG analysis requires "all-iterate coverage" ie bounded π*/π_t ∀𝐭. What if we only assume it at 𝐭=𝟎 (eg base model has nontrivial success rate on all prompts)?
Ans: 𝐞𝐱𝐩𝐨𝐧𝐞𝐧𝐭𝐢𝐚𝐥 complexity due to "coverage collapse". arxiv.org/abs/2611.14467
Sham Kakade
Sham Machandranath Kakade is an American computer scientist. He is a Gordon McKay Professor in Computer Science at Harvard University, with a joint appointment in the Department of Statistics. Kakade is a co-director of the Kempner Institute for the Study of Natural and Artificial Intelligence. He co-founded the Algorithmic Foundations of Data Science Institute. More on Wikipedia
-
Sham Kakade reposted
Nan Jiang
@nanjiang_cs
-
Sham Kakade reposted
Kempner Institute at Harvard University
@KempnerInst
This month, #kempnerinstitute bids farewell to research fellows @cogscikid @t_andy_keller and @GabrielPoesia. Learn more about their innovative work and next steps: kempnerinstitute.harvard.edu/news/kempner-institute-celeb… @UMichCSE
@CSHL @percepta @VanderbiltU -
Sham Kakade reposted
Kempner Institute at Harvard University
@KempnerInst
The application is now open for our #kempnerinstitute Research #fellowship! Postdocs studying the foundations of #intelligence or applications of #ai are encouraged to apply. Learn more and apply by Oct. 1: kempnerinstitute.harvard.edu/kempner-institute-research-f… #llms #neuroai #ml
@ShamKakade6 @blsabatini -
Sham Kakade reposted
Rachit Bansal
@rach_it_
announcing the NeurIPS 2026 workshop on *Transitioning from Pre-training to Post-training*!🍓
come bring your theory, empirics, & methods to enable better foundations, mechanics, & evaluation for LLM training!
submit your work by 29th August!🗓️
pretrain2posttrain.github.io
🧵 -
Sham Kakade reposted
Simo Ryu
@cloneofsimo
Prediction: this will be looked back as one of the most important paper in 2026
Alex Meterez
@alexmeterez
1/8 How well does the simplest model in optimization theory, the quadratic, actually describe LLM pretraining? Surprisingly well. We linearize a 150M model at checkpoints throughout training and the local Taylor expansion tracks the true loss for up to 10% of the training budget.
-
Sham Kakade reposted
Alex Meterez
@alexmeterez
1/8 How well does the simplest model in optimization theory, the quadratic, actually describe LLM pretraining? Surprisingly well. We linearize a 150M model at checkpoints throughout training and the local Taylor expansion tracks the true loss for up to 10% of the training budget.
-
Sham Kakade reposted
Kamalika Chaudhuri
@kamalikac
After a long hiatus, I have a blog post! Here I talk about how modern AI systems have a four-layered stack. Well-developed theory exists for the first two but not the rest. kamalikachaudhuri.substack.com/p/the-four-layers-in-moder…
-
Sham Kakade
@ShamKakade6
Next talk tomo #icml2026: How far does Taylor's Thm take us in deep learning?
@alex_damian_ and I take the local quadratic approx seriously, asking how well it actually holds during LLM pretraining.
Surprisingly, on 150M models, it tracks the true loss deep into training ↓
Sham Kakade
@ShamKakade6
Bonus teaser: we compute the full spectral density of 150M-param models via Lanczos quadrature. Come by to hear about the structure there!
9am Friday at the HILD workshop.
bonus: @alexmeterez also presents at HILD, going deeper into Wed's keynote material. -
Sham Kakade reposted
Mary Letey
@maryiletey
Transformers are sequence models, but much of in-context learning theory considers prompts of independent examples / tokens.
We asked: what changes when ICL prompts are actually sequential?
I’ll be presenting this at #icml2026 HiLD Workshop! -
Sham Kakade reposted
Hanlin Zhang
@_hanlin_zhang_
Presenting our work at the #icml26 Oral session:
Prescriptive Scaling Reveals the Evolution of Language Model Capabilities
We study a practical question for language models:
given a pre-training compute budget, what downstream performance can we reliably expect after post-training?
📍 #icml Oral Hall C
🕧 Thu, Jul 9, 4:30 PM–4:45 PM Local Time -
Sham Kakade
@ShamKakade6
It was a pleasure to speak at #icml2026. Thank you!
ICML Conference
@icmlconf
@ShamKakade6 gave an incredible talk this morning to a packed house! #icml2026
-
Sham Kakade reposted
Tessa
@tessa1157
How can we pretrain language models that continue to learn? In our #icml2026 paper, we find that weight decay shapes this model ability. For more details, stop by our poster (Poster Session 2) or check out the full paper! @sbordt @_hanlin_zhang_ @ShamKakade6 @Schmidt_Center
Tessa
@tessa1157
⭐ How can we set up LLM pretraining to improve the model’s ability to learn new data upon further training? In this new preprint, we find that weight decay during pretraining helps!
Preprint: arxiv.org/abs/2602.11137
Thread below🧵⬇️ -
Sham Kakade reposted
Kempner Institute at Harvard University
@KempnerInst
Congratulations to #kempnerinstitute researchers Binxu Wang, Cengiz Pehlevan, and Jacob A. Zavatone-Veth, whose paper received an #icml2026 Outstanding Paper Honorable Mention!
Read more about the award-winning paper here:
kempnerinstitute.harvard.edu/news/kempner-institute-resea…
@WangBinxu, @CPehlevan #ai -
Sham Kakade
@ShamKakade6
How far can quadratics take us in LLM pretraining? Surprisingly far.
Excited to give a plenary at #icml2026 on how a simple, solvable quadratic model predicts real pretraining — critical batch size, LR decay, batch-size scaling, and what momentum does (and doesn't) do.
Kempner Institute at Harvard University
@KempnerInst
#kempnerinstitute Co-Director Sham Kakade’s upcoming #icml2026 plenary (7/8 @ 8:30am KST) will spotlight a broad science-of-pretraining effort—showcasing how a remarkably simple model can accurately forecast pretraining outcomes. Learn more: kempnerinstitute.harvard.edu/news/kempner-institute-co-di… @shamkakade6
-
Sham Kakade
@ShamKakade6
Momentum accelerates training, but are the savings compute or serial runtime? New work: we prove compute-efficiency (CE: total compute to reach a target loss, not serial steps) lower bounds for stochastic Heavy Ball (HB) and Accelerated SGD (ASGD) [Kidambi et al., 2018]
Post not available here, view on X
Sham Kakade
@ShamKakade6
Work done with amazing collaborators: @depen_morwani, @alexmeterez, @pranavn1008 .
Preprint:
arxiv.org/abs/2606.19179
Sham Kakade
@ShamKakade6
Tagging some folks:
@QuanquanGu, @jainprateek_, @PNetrapalli, @aaron_defazio, @BachFrancis, @konstmish, @ddrusvyat, @KempnerInst, @_arohan_, @jxbz -
Sham Kakade
@ShamKakade6
1/8 Introducing Recurrent Transformer (RT). At 300M params, RT improves validation CE over standard Transformers. The best RT model is only 6 layers, but wider at 2048 — beating deeper 12- and 24-layer Transformers by trading depth for width.
Post not available here, view on X
Sham Kakade
@ShamKakade6
6/8 RT’s depth→width tradeoff gives a systems benefit at inference: when depth drops more than width grows, serial depth and total KV-cache footprint both shrink. Wider layers are also easier to tensor-parallelize, shifting cost from “sequential layers” to “parallel width.”
Sham Kakade
@ShamKakade6
7/8 Paper: arxiv.org/abs/2604.21215
With amazing collaborators: @costinoncescu, @depen_morwani, Samy Jelassi, @alexmeterez, @MujinKwun.
Shoutout to @costinoncescu, who heroed the implementation — see the Fig. 4 scaling!
Sham Kakade
@ShamKakade6
8/8 Tagging folks who may be interested: @Kimi_Moonshot, @tri_dao, @_albertgu, @SonglinYang4, @srush_nlp, @ZeyuanAllenZhu, @HannaHajishirzi, @simran_s_arora, @lambdaviking, @DimitrisPapail
-
Sham Kakade
@ShamKakade6
1/ Au revoir, RLVR. New work: EBFT (Energy-Based Fine-Tuning), a post-training method that directly optimizes the long-horizon behavior of model generations, addressing SFT’s deployment-time error amplification without relying on sparse, task-specific rewards.
Post not available here, view on X
Sham Kakade
@ShamKakade6
7/ Tagging folks who might be interested: @ahatamiz1, @YejinChoinka, @natolambert, @_lewtun, @aviral_kumar2, @tengyuma, @Zanette_ai, @johnschulman2, @agarwl_, @ericzelikman
Sham Kakade
@ShamKakade6
8/ links:
Blog post: kempnerinstitute.harvard.edu/research/deeper-learning/ene…
Paper: arxiv.org/abs/2603.12248
Code: github.com/sjelassi/ebft_openrlhf
Website: energy-based-fine-tuning.github.io -
Sham Kakade reposted
Thomas Fel
@thomas_fel_
Bidding farewell to @KempnerInst! Leaving with great memories of fun moments and a truly supportive, wonderful community 🙏
If you want great research freedom surrounded by brilliant people, apply for the Kempner Fellowship. Cannot recommend it enough!
Next stop: SF! 🌉 -
Sham Kakade reposted
Tessa
@tessa1157
⭐ How can we set up LLM pretraining to improve the model’s ability to learn new data upon further training? In this new preprint, we find that weight decay during pretraining helps!
Preprint: arxiv.org/abs/2602.11137
Thread below🧵⬇️ -
Sham Kakade reposted
Rachit Bansal
@rach_it_
In standard LLM training, RL comes last. In our new work, we question this paradigm.
So, when does an LLM become capable of learning via RL?
Short answer: Much earlier than you expect!
Blogpost: rl-excursions.github.io
w/ @clara_mohri @sunnytqin @elmelis @ShamKakade6
🧵(1/n) -
Sham Kakade reposted
Jaeyeon (Jay) Kim
@Jaeyeon_Kim_0
🚨🚨🚨 Now you can stop training your masked diffusion models ''for the worst''.
We propose 🐆PUMA🐆--Progressive UnMAsking, a simple modification of the forward masking process that speeds up the masked diffusion training. -
Sham Kakade reposted
Dmitriy Drusvyatskiy
@ddrusvyat
I'll be in the Boston area next week: Wednesday (Feb 18) visiting @ShamKakade6 at Harvard and on Friday (Feb 20) speaking in the Statistics and DS seminar at MIT. Happy to meet and chat with those in the area. DM to set something up.
-
Sham Kakade
@ShamKakade6
1/8 Most LLM schedulers aren’t “anytime” because cosine LR decay depends on knowing the training horizon. We show a simple horizon-free alternative—constant or 1/sqrt(t) learning rates + weight averaging (EMA/SWA)—can match cosine’s performance across checkpoints.
Post not available here, view on X
Sham Kakade
@ShamKakade6
7/8 w/ amazing collaborators: @alexmeterez, @pranavn1008, @depen_morwani, @CPehlevan, @KempnerInst.
Paper: www.arxiv.org/abs/2602.03702
Kempner blogpost: shorturl.at/1gpDX
Sham Kakade
@ShamKakade6
8/8 Tagging some people who might be interested: @jxbz, @_arohan_, @zacharynado, @cloneofsimo, @aaron_defazio, @konstmish, @KwangjunA, @PierreAblin, @vfleaking
-
Sham Kakade reposted
Ben Edelman
@EdelmanBen
People sometimes ask me how to leverage a technical background to jump into U.S. AI policy. As of this week my answer is straightforward: apply to join us at CAISI! We're a startup within government, and we're doing a hiring surge.
-
Sham Kakade reposted
Greta Tuckute
@GretaTuckute
How do LLMs process syntax? Do different syntactic phenomena recruit the same model units, or do they recruit distinct model components? And do different languages rely on similar units to process the same syntactic phenomenon?
Check out our new preprint (to appear at ACL 2026)! -
Sham Kakade reposted
Qianqian Wang
@QianqianWang5
I'm recruiting multiple PhD students this cycle to join me at Harvard University and the Kempner Institute! My interests span vision and intelligence, including 3D/4D, active perception, memory, representation learning, and anything you're excited to explore!
Deadline: Dec 15th. -
Sham Kakade reposted
Alex Meterez
@alexmeterez
Will be at NeurIPS 25 presenting this work at OPT-ML this weekend! Reach out to discuss any optimization or LLM training related topics!
Sham Kakade
@ShamKakade6
1/6 Infinite-dim SGD in linear regression is the strawman model for studying scaling laws, critical batch sizes, and LR schedules. We revisit (and simplify) its analysis using just linear algebra, making it easier to derive and reason about. No PSD operators. No tensor calculus.
-
Sham Kakade reposted
Zhenting Qi
@ZhentingQi
[NeurIPS ’25 Oral] EvoLM: In Search of Lost Language Model Training Dynamics
We trained and evaluated 100+ LMs from scratch—spanning pretraining → continued pretraining → SFT → RL—to probe some of the most mysterious and underexplored dynamics in LM development.
Our investigations include:
- Does RLVR really improve base-model capabilities?
- How does mid-training bridge pre- and post-training?
- How do we allocate data between SFT and RL (and the tradeoffs)?
- When can excessive pretraining hurt post-training performance?
- When might smaller models outperform larger models?
- …and much more!
📄 Paper: arxiv.org/pdf/2506.16029
🧪 Project page: zhentingqi.github.io/internal/projects/EvoLM/
💻 Code: github.com/zhentingqi/evolm
📌 Poster: Exhibit Hall C,D,E Poster Location: #5319
🎤 Oral session: Oral 5A Language Model 3
Huge thanks again to my amazing collaborators and advisors @du_yilun @hima_lakkaraju @ShamKakade6 @ericxing @FanNie1208 @james_y_zou @AlexAlahi @_hanlin_zhang_
---
PS: I’ll be at NeurIPS in San Diego (12/1–12/7) — happy to chat about Reasoning, Reliability, and Compositionality😄 -
Sham Kakade reposted
Michael Albergo
@msalbergo
⭐️New Paper⭐️ just in time for NeurIPS. Presenting: Test-time scaling of diffusions with flow maps.
project: flow-map-trajectory-tilting.github.io
arXiv: arxiv.org/abs/2511.22688
Want to properly adapt your diffusion at inference time with a reward? You should do everything at the level of the flow map.
With the amazing @amsabour, @cdomingoenrich, @nmboffi, @FidlerSanja , @karsten_kreis, and @evdende2. -
Sham Kakade reposted
Kempner Institute at Harvard University
@KempnerInst
🎉Huge congratulations to #kempnerinstitute community members @chooi_jeq, @EmmaFinn314, and Helen He on being named 2026 Rhodes Scholars!
We're so proud to see their talent and leadership recognized on the world stage.🌍🎓
Read more➡️ kempnerinstitute.harvard.edu/news/three-kempner-undergrad…
#rhodesscholar -
Sham Kakade reposted
Kempner Institute at Harvard University
@KempnerInst
In the latest #kempnerinstitute Seminar Series talk, @kamalikac from FAIR @Meta discussed the ways in which fundamental challenges of trustworthy AI currently manifest within LLMs.
Watch the talk: youtu.be/T2Cyv5YpNCs
#ai #llms -
Sham Kakade reposted
Jaeyeon (Jay) Kim
@Jaeyeon_Kim_0
🚨🚨🚨 Now your Masked Diffusion Model can self-correct! We propose PRISM, a plug-and-play approach fine-tuning method that adds self-correction ability to any pretrained MDM! (1/N)
-
Sham Kakade
@ShamKakade6
Looking forward to working with you and CZI!
Alex Rives
@alexrives
Today CZI is announcing an unprecedented new scientific initiative to build the future of AI-powered biology. I am joining CZI to lead this initiative as Head of Science, and the EvolutionaryScale team is joining forces with Biohub.
This is the first large scale scientific effort to combine frontier AI and frontier biology.
I feel an incredible sense of optimism in this moment. There is a path to build predictive models of life that can fundamentally accelerate science, and unlock a new understanding of disease.
biohub.org/blog/frontier-ai-biology-initiative/ -
Sham Kakade reposted
Soumith Chintala
@soumithchintala
Leaving Meta and PyTorch
I'm stepping down from PyTorch and leaving Meta on November 17th.
tl;dr: Didn't want to be doing PyTorch forever, seemed like the perfect time to transition right after I got back from a long leave and the project built itself around me.
Eleven years at Meta. Nearly all my professional life. Making many friends for life. Almost eight years leading PyTorch, taking it from nothing to 90%+ adoption in AI. Walking away from this was one of the hardest things I've ever done. But I'm leaving with a full heart.
PyTorch handles exascale training now. It powers foundation models that are redefining intelligence. It's in production at virtually every major AI company. It's taught in classrooms from MIT to rural India. The tools I dreamed about making accessible? They are. The barrier to entry I wanted to lower? It's almost gone.
To be clear, there’s so much more to do. As long as AI evolves at a breakneck pace, PyTorch will continue to play catch up. Obsessing over the yet-to-come sometimes makes us forget how much we’ve already done.
To everyone who built this with me—who believed research should be joyful, that tools should be elegant, that open source changes everything—thank you. This wasn't my journey. It was ours.
What's next for me? Something small. Something new. Something I don't fully understand yet. Something uncomfortable. I could have moved to something else inside Meta. But I needed to know what's out there. I needed to do something small again. I couldn't live with the counterfactual regret of never trying something outside Meta.
It's very hard to leave. I probably have one of the AI industry’s most leveraged seats, I lead the software layer that powers the entire AI industry. Every major AI company and hardware vendor are on a speed dial. This kind of power is really hard to give up. But curiosity ultimately won out in my head.
Keep making AI delicious and accessible. I'll be watching. Probably filing issues. Definitely staying involved.
Is PyTorch going to be okay?
I don't want to be doing PyTorch forever. I don't want to be like Guido or Linus— bound to a single thing for decades. Last November, coinciding with the birth of my daughter, I started planning my exit with Aparna. My goal was to leave PyTorch in a good and stable place.
By this August, during the second half of my parental leave, I knew: Edward, Suo, Alban, Greg, John, Joe and Jana were ready. The team faced hard people, product, technical and organizational problems and didn’t feel the need to lean back on me to solve these for them (unlike in the past). The product story they crafted for the PyTorch Conference was coherent—really coherent. The things I'd flagged red were turning healthy. The project didn't need me anymore. Unlike 2020-2022 (when I stepped down to go do robotics and came back when Lin, Dima and Dwarak left), I have strong confidence that this time PyTorch is truly resilient. The most aligned culture carriers of PyTorch – Greg, Alban, Ed, Jason and Joe are at the decision table now, and people with strong value alignment – Suo, John and Jana have joined them at the table. And there’s a long list of equally value-aligned people willing to sit at the table should any of these people leave. There are many little things that make up my confidence on the people – John worked on Julia and open-source for a very long time (in fact we hacked a Torch.jl in 2015), Suo has been the strongest systems builder and strategic partner I’ve had for the past two years, and Jana worked on resilient core systems for a very long time, I’ve had long technical and organizational discussions with her over the past few months that give me confidence. And the product lineup and execution in 2025 should be sufficient evidence for any remaining doubt.
I’m confident that this band of PyTorchers are going to do exceptionally well. PyTorch might change in flavor because I no longer impose my own taste from the top, but I’m confident that the values are going to stay intact and the product is going to be awesome.
My time at Meta
The early years of FAIR were absolutely magical. I was part of a small family of absolutely brilliant people building state-of-the-art AI out in the open. From working on GANs with Emily Denton, Rob Fergus, Leon Bottou, Martin Arjovsky and the (now legendary) Alec Radford to building Starcraft bots with Gabriel Synnaeve, to building the first FAIR Cluster with Howard Mansell, to working on object detection with Adam Lerer and Piotr Dollar, to building PyTorch. It was more fun than I can describe in words. 2015 and 2016 were probably the most productive and professionally enjoyable years of my life. I’ll probably romanticize this period of my life forever.
When I joined FAIR, I had massive impostor syndrome, and the first 3 months were very very difficult. I can’t credit Andrew Tulloch enough for being the most thoughtful, kind and welcoming mentor, without whom I wouldn’t have made it. I’m so damn bullish for Meta just from the fact that he’s back.
---
My time on PyTorch was special.
I loved every part of building it—designing it, managing it, being the PM, TL, comms lead, doc engineer, release engineer, squashing bugs, growth hacking, turning it into a coherent product with hundreds of people, transitioning it to industry stakeholdership – the whole nine yards.
To the core PyTorch team at Meta: the engineers, researchers, open-source maintainers, docs writers, CI infrastructure folks, hardware partners, the community builders. To the hundreds more inside and outside Meta—thank you. You turned a library into a movement.
There are too many people to credit and thank, but I can't not mention Adam Paszke, Sam Gross, Greg Chanan, Joe Spisak, Alban Desmaison, Edward Yang, Richard Zou, Tongzhou Wang, Francisco Massa, Luca Antiga, Andreas Köpf, Zach DeVito, Zeming Lin, Adam Lerer, Howard Mansell and Natalia Gimelshein. And Schrep. They made the launch happen. And so many more people became centrally important later: Lu Fang, Xiaodong Wang, Junjie Bai, Nikita Shulga, Horace He, Mark Saroufim, Jason Ansel, Dmytro Dzhulgakov, Yangqing Jia, Geeta Chauhan, Will Constable, Briah Hirsh, Jane Xu, Mario Lezcano, Piotr Balecki, Yinghai Lu, Less Wright, Andrew Tulloch, Bruce Lin, Woo Kim, Helen Suk, Chris Gottbrath, Peng Wu, Joe Isaacson, Eli Uriegas, Tristan Rice, Yanan Cao, Elias Ellison, Animesh Jain, Peter Noordhuis, Tianyu Liu, Yifu Wang, Lin Qiao and hundreds more. It’s criminal of me to not take the space to list out everyone else I should be mentioning here. PyTorch is nothing without its people ❤️.
The most joyful moments of building PyTorch was meeting users eager to share their happiness, love and feedback. I remember a grad student coming to me at Neurips 2017, in a slurring emotional voice he said he’d been trying to make progress on his research for 3 years but within 3 months of using PyTorch he made so much progress that he was ready to graduate. That moment made it tangible that what we do matters, a lot, to a lot of people, even if you don't constantly hear from them. I do miss the intimacy of the PyTorch community, with a 300 person conference that felt like an extended family gathering, but I feel that’s a small price to pay considering the scale of impact PyTorch is truly having today – yes the Conference is now 3,000 people where market-moving deals get brokered, but it’s helping orders of magnitude more people to do their best AI work. I miss the intimacy, but I'm proud of that growth.
---
To Mark Zuckerberg and Mike Schroepfer, who believed that open-sourcing is fundamentally important and is a sound business strategy. This is so hard to understand for most people within the course of business, but we’ve run lock-step on this strategy without ever having to discuss it. Without you two, neither FAIR nor PyTorch would’ve happened. And those mean so much to me.
To Yann LeCun and Rob Fergus, for building the magical early FAIR that I so revere.
To Aparna Ramani, a leader that I find so rare at Meta in her ability to hold a really high bar for the org, technically brilliant with the span to discuss deep infra systems and industry-strategy within the same conversation and for being an absolute execution-machine! I’ve learned so much from you.
To Santosh, Kaushik, Delia, Oldham and Ben for being so welcoming to Infra. For someone coming over from FAIR with a wildly different culture, you all made me feel at home and made me part of the family, and thank you for that.
To all my managers who've championed me through the PSC video game – Serkan, Howard, Jerome, Abhijit, Yoram, Joelle, Aparna and Damien – I owe you a lifetime of drinks.
---
Signing off for now.
—Soumith -
Sham Kakade reposted
Kempner Institute at Harvard University
@KempnerInst
Congratulations to #kempnerinstitute community members @msalbergo and @mweber_PU— recipients of @schmidtsciences AI2050 Fellowships! 🎉
Discover their innovative research shaping the future of AI 👉 kempnerinstitute.harvard.edu/news/schmidt-sciences-awards…
#ai @hseas -
Sham Kakade reposted
Jason Lee
@jasondeanlee
Haozhe's paper is worth a read, really nice use of fixed point theorems. The new one about 1 to 1 seems almost immediate though Just from reading the thread only I would guess the proof is as follows: Say your input space is discrete in j \in [n] and represented by x_j. The embedding is E: [n] \to R^d . "For almost all E", E x_j are distinct. Then a transformer f is composed of building blocks that are analytic, composition of analytic is analytic and also preserved under many algebraic operations (o-minimal stuff) . Analytic functions are either the zero function, or crosses zero on a measure zero set (they can't have positive measure set f(set) =0) . Thus so long as f \neq 0 uniformly, this should be injective (not bijective necessarily its not onto).
Haozhe Jiang
@erichzjiang
(1/7) Glad to see that people are following up on our work studying topological properties of modern neural network architectures. It was cool to see that widely used neural architectures can almost always generate any output given appropriate inputs, a.k.a. are surjective.
-
Sham Kakade reposted
Stefano Ermon
@StefanoErmon
Tired of chasing references across dozens of papers? This monograph distills it all: the principles, intuition, and math behind diffusion models. Thrilled to share!
Chieh-Hsin (Jesse) Lai
@JCJesseLai
Tired to go back to the original papers again and again? Our monograph: a systematic and fundamental recipe you can rely on!
📘 We’re excited to release 《The Principles of Diffusion Models》— with @DrYangSong, @gimdong58085414, @mittu1204, and @StefanoErmon.
It traces the core ideas that shaped diffusion modeling and explains how today’s models work, why they work, and where they’re heading.
🧵You’ll find the link and a few highlights in the thread.
We’d love to hear your thoughts and join some discussions!
⚡ Stay tuned for our markdown version, where you can drop your comments! -
Sham Kakade reposted
Joseph Bejjani
@jbejjani2022
Hot take: Ecology is the most important and underexplored policy learning paradigm in ML.
1/n -
Sham Kakade reposted
Blake Bordelon ☕️🧪👨💻
@blake__bordelon
Applying to do a postdoc or PhD in theoretical ML or neuroscience this year? Consider joining my group (starting next Fall) at UT Austin!
POD Postdoc: oden.utexas.edu/programs-and-awards/odonnell-jr-postdoc-f… CSEM PhD: oden.utexas.edu/academics/prospective-students/how-to-apply/ -
Sham Kakade
@ShamKakade6
(1/9) Diagonal preconditioners such as Adam typically use empirical gradient information rather than true second-order curvature. Is this merely a computational compromise or can it be advantageous? Our work confirms the latter: Adam can outperform Gauss-Newton in certain cases.
Sham Kakade
@ShamKakade6
(2/9) We consider 3 diagonal preconditioners: Adam, Gauss-Newton with power -1, and additionally Gauss-Newton with power -1/2 given that many empirical-gradient-based methods (e.g. Adam, Shampoo, SOAP) effectively take the square root of the second-order estimate.
Post not available here, view on X
Sham Kakade
@ShamKakade6
(8/9) Joint work with amazing collaborators: @BingbinL, @rach_it_, @depen_morwani, @vyasnikhil96, @elmelis. Paper: arxiv.org/abs/2510.13680
Sham Kakade
@ShamKakade6
(9/9) Tagging some people who might be interested: @runame_, @LinYorker, @_arohan_, @aaron_defazio, @zacharynado, @jxbz
-
Sham Kakade reposted
Andreas Kirsch 🇺🇦
@BlackHC
Very interesting! At first I thought this was related to SGLD but it's not. Note that it looks more efficient vs steps because it keeps doubling the batch size. Required FLOPs stay the same. It provides a wall time speedup only if you have additional parallel compute available
Sham Kakade
@ShamKakade6
1/6 Introducing Seesaw: a principled batch size scheduling algo. Seesaw achieves theoretically optimal serial run time given a fixed compute budget and also matches the performance of cosine annealing at fixed batch size.
-
Sham Kakade
@ShamKakade6
1/6 Introducing Seesaw: a principled batch size scheduling algo. Seesaw achieves theoretically optimal serial run time given a fixed compute budget and also matches the performance of cosine annealing at fixed batch size.
Post not available here, view on X
Sham Kakade
@ShamKakade6
5/6 At Chinchilla scale, Seesaw reduces the serial run time of training by roughly 36% across several model scales (150M, 300M, 600M), approaching the max achievable theoretical reduction implied by our analysis.
Sham Kakade
@ShamKakade6
6/6 For more details, check out the paper: arxiv.org/abs/2510.14717.
Work done with great collaborators: @alexmeterez, @depen_morwani, @uuujingfeng, @costinoncescu, @CPehlevan
Tagging folks who might be interested: @jxbz , @zacharynado, @cloneofsimo, @aaron_defazio, @konstmish, @KwangjunA, @PierreAblin -
Sham Kakade reposted
Aayush Karan
@aakaran31
We found a new way to get language models to reason. 🤯
No RL, no training, no verifiers, no prompting. ❌
With better sampling, base models can achieve single-shot reasoning on par with (or better than!) GRPO while avoiding its characteristic loss in generation diversity. -
Sham Kakade reposted
Mujin Kwun
@MujinKwun
LOTION is the balm
Sham Kakade
@ShamKakade6
1/9 Introducing LOTION (Low-precision optimization via stochastic-noise smoothing), a principled alternative to Quantization-Aware Training (QAT) that explicitly smooths the quantized loss surface while preserving all global minima of the true quantized loss. Details below:
-
Sham Kakade
@ShamKakade6
1/9 Introducing LOTION (Low-precision optimization via stochastic-noise smoothing), a principled alternative to Quantization-Aware Training (QAT) that explicitly smooths the quantized loss surface while preserving all global minima of the true quantized loss. Details below:
Post not available here, view on X
Sham Kakade
@ShamKakade6
8/9 With amazing collaborators: @MujinKwun @depen_morwani @Huangyu58589918 Stephanie Gil @nikhil_anand91
Paper: arxiv.org/abs/2510.08757
Blog post: kempnerinstitute.harvard.edu/research/deeper-learning/lot…
Sham Kakade
@ShamKakade6
9/9 Tagging people who might be interested:
@DAlistarh @songhan_mit @chrismdesa @Tim_Dettmers @wellingmax @tsengalb99 -
Sham Kakade
@ShamKakade6
1/8 Second Order Optimizers like SOAP and Muon have shown impressive performance on LLM optimization. But are we fully utilizing the potential of second order information? New work: we show that a full second order optimizer is much better than existing optimizers in terms of iteration complexity (~5x over SOAP and ~15x over Muon).
Post not available here, view on X
Sham Kakade
@ShamKakade6
7/8 We hope that this work motivates people to continue pushing the limits of second order optimizers. Work done with amazing collaborators: Natalie Abreu, @vyasnikhil96, @depen_morwani .
Check out the manuscript here - arxiv.org/abs/2510.09378
Sham Kakade
@ShamKakade6
8/8 Tagging some people who might be interested: @jxbz , @_arohan_, @zacharynado, @cloneofsimo, @aaron_defazio, @kellerjordan0, @leloykun
-
Sham Kakade reposted
Jaeyeon (Jay) Kim
@Jaeyeon_Kim_0
We introduce a new ''rule'' for understanding diffusion models: Selective Underfitting.
It explains:
🚨 How diffusion models generalize beyond training data
🚨 Why popular training recipes (e.g., DiT, REPA) are effective and scale well
Co-led with @kiwhansong0!
(1/n) -
Sham Kakade reposted
Jingfeng Wu
@uuujingfeng
sharing a new paper w Peter Bartlett, @jasondeanlee, @ShamKakade6, Bin Yu
ppl talking about implicit regularization, but how good is it? We show its surprisingly effective, that GD dominates ridge for all linear regression, w/ more cool stuff on GD vs SGD
arxiv.org/abs/2509.17251 -
Sham Kakade reposted
Jaeyeon (Jay) Kim
@Jaeyeon_Kim_0
Announcing Flexible Masked Diffusion Models (FlexMDMs)—a new diffusion language model for flexible-length sequences.
🚨 Solves MDMs' fixed-length issue + retrains any-order sampling
🚨 <1000 GPU-hrs to fine-tune LLaDA-8B into FlexMDM (GSM8K 58→67%, HumanEval-infill: 52→65%) -
Sham Kakade reposted
Richard Sutton
@RichardSSutton
My acceptance speech at the Turing award ceremony:
Good evening ladies and gentlemen.
The main idea of reinforcement learning is that a machine might discover what to do on its own, without being told, from its own experience, by trial and error. As far as I know, the first person to propose this was Alan Turing in 1947, which makes it particularly gratifying and humbling to receive this award in his name for reviving this essential but still nascent idea.
I have three people that I would like to particularly thank.
First, Andy Barto. As my PhD supervisor he taught me my whole approach to science, and in particular instilled in me an appreciation of scholarship and craft, and of the great breath of prior work.
Second, I would like to thank Oliver Selfridge, my other main mentor; sadly, now deceased. Oliver taught me how keeping ideas simple can be the boldest of all ambitions.
Third, I want to thank Martha Steenstrup, my life partner and intellectual sparring partner. She keeps me honest and grounded.
Finally, I also want to thank the University of Alberta, which has been an ideal environment for me and for reinforcement learning research these past 22 years.
These three people and my university have reinforced in me the ambition to have ideas that matter, without getting too full of myself about it. They taught me that the quest for better ideas is serious, but is best approached playfully, with humility, kindness, and optimism. For this I am eternally grateful.
I would also like to thank all of you for being here and for celebrating the pursuit of intellectual excellence.
Thank you very much. -
Sham Kakade reposted
Lin Yang
@lyang36
Our IMO gold medal-winning AI pipeline is now model-agnostic. 🥇
What worked for Gemini 2.5 Pro now gets the same 5/6 score with GPT-5 & Grok4. This confirms the power of our verification-and-refinement pipeline to improve base model capabilities.
The new code & results are live on GitHub[github.com/lyang36/IMO25]! Paper update coming soon.
Huge thanks to @xai for the Grok4 API credits! #ai #llm #imo #matholympiad #opensource” -
Sham Kakade reposted
Jaeyeon (Jay) Kim
@Jaeyeon_Kim_0
🚨New video! I talked about my research — 1️⃣ My work at Seoul National University 2️⃣ Our Outstanding Paper Award at ICML 2025
Also check out another podcast with @du_yilun — an amazing researcher & AP at Harvard CS! Harvard CS is growing fast 🫡
Delta
@DeltaCommunityX
Huge thanks to Jaeyeon Kim for coming on the Delta Podcast! Check out the podcast episode here: www.youtube.com/watch?v=nP9xR3xUjNk
-
Sham Kakade reposted
Qianqian Wang
@QianqianWang5
📢Thrilled to share that I'll be joining Harvard and the Kempner Institute as an Assistant Professor starting Fall 2026!
I'll be recruiting students this year for the Fall 2026 admissions cycle. Hope you apply!
Kempner Institute at Harvard University
@KempnerInst
We are thrilled to share the appointment of @QianqianWang5 as an #kempnerinstitute Investigator! She will bring her expertise in computer vision to @Harvard.
Read the announcement: kempnerinstitute.harvard.edu/news/computer-scientist-qian…
@hseas #ai #computervision -
Sham Kakade reposted
Kempner Institute at Harvard University
@KempnerInst
Applications are now open for our #kempnerinstitute Research Fellowships!
Postdocs studying the foundations of #intelligence or applications of #ai are encouraged to apply.
Learn more and apply by Oct. 1:
kempnerinstitute.harvard.edu/kempner-institute-research-f…
#llms #neuroai #ml @ShamKakade6 @blsabatini -
Sham Kakade reposted
Kempner Institute at Harvard University
@KempnerInst
We are thrilled to share the appointment of @QianqianWang5 as an #kempnerinstitute Investigator! She will bring her expertise in computer vision to @Harvard.
Read the announcement: kempnerinstitute.harvard.edu/news/computer-scientist-qian…
@hseas #ai #computervision -
Sham Kakade reposted
Blake Bordelon ☕️🧪👨💻
@blake__bordelon
Excited to announce that I will be joining @UTAustin with a joint position between @OdenInstitute for Computational Science and dept of Neuroscience in FL 2026! I plan on recruiting PhD students and postdocs interested in mathematics of neural computation (more details to come).
-
Sham Kakade reposted
-
Sham Kakade reposted
Kempner Institute at Harvard University
@KempnerInst
Thrilled to announce the 2025 recipients of #kempnerinstitute Research Fellowships: Elom Amemastro, Ruojin Cai, David Clark, Alexandru Damian, William Dorrell, Mark Goldstein, Richard Hakim, Hadas Orgad, Gizem Ozdil, Gabriel Poesia, & Greta Tuckute!
kempnerinstitute.harvard.edu/news/announcing-2025-kempner… -
Sham Kakade reposted
Noah Golowich
@GolowichNoah
I'll be attending ICML this week; come stop by our poster on length generalization in LLMs on Tuesday morning (poster session 1 west)! Paper link: openreview.net/forum?id=S9LkB0UBKb
-
Sham Kakade reposted
Kempner Institute at Harvard University
@KempnerInst
A team from #kempnerinstitute, @hseas & @UTCompSci has won a best paper award at #icml2025 for work unlocking the potential of masked diffusion models.
Congrats to @Jaeyeon_Kim_0,
@shahkulin98, Vasilis Kontonis, @ShamKakade6 and
@sitanch.
kempnerinstitute.harvard.edu/news/kempner-institute-resea… #ai -
Sham Kakade reposted
Kulin Shah
@shahkulin98
Thrilled to share that our work received the Outstanding Paper Award at ICML!
I will be giving the oral presentation on Tuesday at 4:15 PM.
@Jaeyeon_Kim_0 and I both will be at the poster session shortly after the oral presentation. Please attend if possible!
Sitan Chen
@sitanch
Excited about this new work where we dig into the role of token order in masked diffusions!
MDMs train on some horribly hard tasks, but careful planning at inference can sidestep the hardest ones, dramatically improving over vanilla MDM sampling (e.g. 7%->90% acc on Sudoku) 1/ -
Sham Kakade reposted
Gabriel Poesia
@GabrielPoesia
Thrilled to join the UMich faculty in 2026!
I'll also be recruiting PhD students this upcoming cycle. If you're interested in AI and formal reasoning, consider applying!
Computer Science and Engineering at Michigan
@UMichCSE
We’re happy to announce that @GabrielPoesia will be joining our faculty as an assistant professor in Fall 2026. Welcome to CSE!
▶️Learn more about Gabriel here: gpoesia.com
#umichcse #goblue -
Sham Kakade reposted
Hanlin Zhang
@_hanlin_zhang_
[1/n] Discussions about LM reasoning and post-training have gained momentum. We identify several missing pieces:
✏️Post-training based on off-the-shelf base models without transparent pre-training data components and scale.
✏️Intermediate checkpoints with incomplete learning rate decay.
✏️Scaling studies that evaluate solely upstream cloze task performance, without extending to downstream problem-solving capabilities.
✏️Analysis that does not consider OOD generalization.
We seek to bridge these gaps by introducing EvoLM, a more comprehensive, end-to-end study (by @ZhentingQi) of language model training dynamics.🧵 -
Sham Kakade
@ShamKakade6
I took this class. Good times! Thank you @ZoubinGhahrama1 and @geoffreyhinton!! Yeah, no backprop. I view this as more the “modeling phase” of deep learning vs “scale”. I’m going with the ideas are still relevant for AI4science.
Zoubin Ghahramani
@ZoubinGhahrama1
This is the syllabus of the course @geoffreyhinton and I taught in 1998 at the Gatsby Unit (just after it was founded). Notice anything?
-
Sham Kakade reposted
Chloe Su ✈️ NeurIPS 2026
@Huangyu58589918
What precision should we use to train large AI models effectively? Our latest research probes the subtle nature of training instabilities under low precision formats like MXFP8 and ways to mitigate them. Thread 🧵👇
-
Sham Kakade
@ShamKakade6
1/6 Infinite-dim SGD in linear regression is the strawman model for studying scaling laws, critical batch sizes, and LR schedules. We revisit (and simplify) its analysis using just linear algebra, making it easier to derive and reason about. No PSD operators. No tensor calculus.
Post not available here, view on X
Sham Kakade
@ShamKakade6
3/6 We show these diagonal terms evolve independently of the off-diagonals --- reducing the problem to tracking a vector recurrence.
This allows us to recover the same bias-variance decomposition as Zou et al. (2021), but with simpler tools.
Sham Kakade
@ShamKakade6
4/6 We provide risk bounds for both tail averaging and minibatching, recovering the same sharp bounds with simpler derivations. We hope this simplification enables future work on SGD in more realistic regimes: e.g. novel LR schedules and batch scalings.
Sham Kakade
@ShamKakade6
5/6 Work done with amazing collaborators: @alexmeterez, @depen_morwani, @costinoncescu, @uuujingfeng, @CPehlevan. The manuscript: arxiv.org/abs/2506.15535
Sham Kakade
@ShamKakade6
6/6 Tagging some relevant folks: @QuanquanGu, @jainprateek_, @PNetrapalli, @aaron_defazio, @BachFrancis, @konstmish
-
Sham Kakade reposted
Hanlin Zhang
@_hanlin_zhang_
[1/n] New work [JSKZ25] w/ @JikaiJin2002, @syrgkanis, @ShamKakade6.
We introduce new formulations and tools for evaluating language model capabilities, which help explain recent observations of post-training behaviors of Qwen-series models — there is a sensitive causal link from instruction-following capabilities to math reasoning ones, which can lead to performance inflation.
See more details 🧵: hanlin-zhang.com/causal-capabilities -
Sham Kakade reposted
Aayush Karan
@aakaran31
Steering diffusion models with external rewards has recently led to exciting results, but what happens when the reward is inherently difficult?
Introducing ReGuidance: a simple algorithm to (provably!) boost your favorite guidance method on hard problems! 🚀🚀🚀
A thread: (1/n) -
Sham Kakade reposted
Kempner Institute at Harvard University
@KempnerInst
NEW: @du_yilun of @GoogleDeepMind & incoming #kempnerinstitute faculty explains how optimizing energy functions can help solve challenging navigation & reasoning problems. Watch the talk:
www.youtube.com/watch?v=UKbLBOuHQ-0
#neuroai2025 #ml #neuroscience #neuroai -
Sham Kakade reposted
Antonio Orvieto
@orvieto_antonio
We have a new SSM theory paper, just accepted to COLT, revisiting recall properties of linear RNNs.
It's surprising how much one can delve into, and how beautiful it can become.
With (and only thanks to) the amazing Alexandre and @BachFrancis
arxiv.org/pdf/2502.09287 -
Sham Kakade reposted
Kempner Institute at Harvard University
@KempnerInst
New in the Deeper Learning blog: Kempner researchers show how VLMs speak the same semantic language across images and text.
kempnerinstitute.harvard.edu/research/deeper-learning/int…
by @isabelpapad, @Huangyu58589918, @Napoolar, Stephanie Gil, and @ShamKakade6
#ai #ml #vlms #saes -
Sham Kakade reposted
Depen Morwani
@depen_morwani
Excited to attend #iclr25 this week. My DMs are open, feel free to drop a message to talk about anything related to optimization of deep networks.
Presenting multiple works related to second order optimization, critical batch size and diagonal preconditioning. Details below. -
Sham Kakade reposted
Rosie Zhao
@rosieyzh
Excited to be attending 🇸🇬#iclr2025! My DMs are open, please reach out to chat about LLM reasoning/optimization/training dynamics!
Will be presenting a study on diagonal preconditioning optimizers for LLM pretraining (arxiv.org/abs/2407.07972) and SOAP (arxiv.org/abs/2409.11321) -
Sham Kakade reposted
Eran Malach
@EranMalach
How does RL improve performance on math reasoning? Studying RL from pretrained models is hard, as behavior depends on choice of base model. 🚨 In our new work, we train models *from scratch* to study the effect of the data mix on the behavior of RL. arxiv.org/abs/2504.07912
-
Sham Kakade reposted
Naomi Saphra
@nsaphra
Life update: I'm starting as faculty at Boston University in 2026! BU has SCHEMES for LM interpretability & analysis, so I couldn't be more pumped to join a burgeoning supergroup w/ @najoungkim @amuuueller. Looking for my first students, so apply and reach out!
-
Sham Kakade reposted
Richard Sutton
@RichardSSutton
awards.acm.org/about/2024-turing
Machines that learn from experience were explored by Alan Turing almost eighty years ago, which makes it particularly gratifying and humbling to receive an award in his name for reviving this essential but still nascent idea. -
Sham Kakade
@ShamKakade6
Amazing and well deserved!! Their work was an inspiration to me and why I moved to RL.
Association for Computing Machinery
@TheOfficialACM
Meet the recipients of the 2024 ACM A.M. Turing Award, Andrew G. Barto and Richard S. Sutton! They are recognized for developing the conceptual and algorithmic foundations of reinforcement learning. Please join us in congratulating the two recipients! www.acm.org/media-center/2025/march/turing-award-2024
-
Sham Kakade reposted
Kempner Institute at Harvard University
@KempnerInst
Calling all #ai, #ml and #compneuro researchers! The Kempner Institute's Frontiers in #neuroai symposium is a great new opportunity to showcase your work. Submit an abstract by March 16! Details: kempnerinstitute.harvard.edu/frontiers-in-neuroai/abstracts/ @ShamKakade6 @blsabatini
-
Sham Kakade reposted
Naomi Saphra
@nsaphra
Ever looked at LLM skill emergence and thought 70B parameters was a magic number? Our new paper shows sudden breakthroughs are samples from bimodal performance distributions across seeds. Observed accuracy jumps abruptly while the underlying accuracy DISTRIBUTION changes slowly!
-
Sham Kakade reposted
Sitan Chen
@sitanch
Excited about this new work where we dig into the role of token order in masked diffusions!
MDMs train on some horribly hard tasks, but careful planning at inference can sidestep the hardest ones, dramatically improving over vanilla MDM sampling (e.g. 7%->90% acc on Sudoku) 1/ -
Sham Kakade reposted
Kempner Institute at Harvard University
@KempnerInst
NEW blog post: Do modern #llms capture the conceptual diversity of human populations? #kempnerinstitute researchers find #alignment reduces conceptual diversity of language models. Read more: kempnerinstitute.harvard.edu/research/deeper-learning/ali…
@soniakmurthy @tomerullman @_jennhu -
Sham Kakade reposted
Nikhil Vyas
@vyasnikhil96
Combining SOAP and Muon: nikhilvyas.github.io/SOAP_Muon.pdf and some rough thoughts on interesting future directions.
-
Sham Kakade
@ShamKakade6
1/n In new work, we draw connections between accelerated SGD and various recent optimizers including AdEMAMix, Schedule-Free optimizers and MARS, and use it to design ‘Simplified-AdEMAMix’ which matches performance of AdEMAMix without any extra momentum buffer.
Post not available here, view on X
Sham Kakade
@ShamKakade6
7/n For more details, see arxiv.org/abs/2502.02431. Joint work with @depen_morwani, @vyasnikhil96, @_hanlin_zhang_
Sham Kakade
@ShamKakade6
8/n Tagging some relevant people @MatPagliardini @GrangierDavid @PierreAblin @aaron_defazio @QuanquanGu @mattkdouglas @Ar_Douillard
-
Sham Kakade reposted
Jacob Austin
@jacobaustin132
Making LLMs run efficiently can feel scary, but scaling isn’t magic, it’s math! We wanted to demystify the “systems view” of LLMs and wrote a little textbook called “How To Scale Your Model” which we’re releasing today. 1/n
-
Sham Kakade reposted
Kempner Institute at Harvard University
@KempnerInst
We are thrilled to share the appointment of @s_y_chung as an #kempnerinstitute Investigator, bringing her expertise in using #ai to understand #brain structure and function to @Harvard. Read the announcement: kempnerinstitute.harvard.edu/news/leading-neuroscientist-… @hseas @harvardphysics
-
Sham Kakade reposted
Kempner Institute at Harvard University
@KempnerInst
A huge congratulations to @KanakaRajanPhD of the #kempnerinstitute & @harvardmed who was awarded the Presidential Early Career Award for Scientists and Engineers by the @WhiteHouse! Read more and see the full list of honorees: www.whitehouse.gov/ostp/news-updates/2025/01/14/president…
-
Sham Kakade
@ShamKakade6
Think they used MatFormers in these embeddings!
Voyage AI by MongoDB
@VoyageAI
📢 Announcing the new SOTA voyage-3-large embedding model!
• 9.74% over OpenAI and +20.71% over Cohere
• flexible dim. (256-2048) and quantizations (float, int8, binary)
• 8.56% over OpenAI with 1/24x storage cost
• 1.16% over OpenAI with 1/192x storage cost ($10K → $52) -
Sham Kakade reposted
Kempner Institute at Harvard University
@KempnerInst
NEW in the #kempnerinstitute blog: learn about ProCyon, a multimodal foundation model to model, generate & predict protein phenotypes. Led by @oq_35, Robert Calef, @YepHuang, @valegiunca w/ @marinkazitnik @HarvardDBMI. Read it here:
kempnerinstitute.harvard.edu/research/deeper-learning/pro… -
Sham Kakade reposted
Andreas Kirsch 🇺🇦
@BlackHC
And finally, in "CoLoR-Filter: Conditional Loss Reduction Filtering for Targeted Language Model Pre-training" also at @NeurIPSConf 2024, we show that you can compute RhoLoss once between a downstream reference model and a pretrained model to use it for data filtering (instead of active selection) and then pre-train models using the filtered datasets (without any additional constraints!)
This work was done by @brandfonbrener *, @_hanlin_zhang_, me, @schwarzjn_ and @ShamKakade6
Come to the poster session Fri 13 Dec 4:30 p.m. PST — 7:30 p.m. PST (West Ballroom A-D #6706)!
arxiv.org/abs/2406.10670 -
Sham Kakade reposted
Kempner Institute at Harvard University
@KempnerInst
College grads interested in intelligence research: the application for #kempnerinstitute's post-bac program w/ the @HarvardGSAS Office for Equity, Diversity, Inclusion & Belonging is now open! Apply by 2/1/2025. kempnerinstitute.harvard.edu/education/post-baccalaureate… @EllaBatty @grez72 @ShamKakade6 @blsabatini
-
Sham Kakade reposted
Kempner Institute at Harvard University
@KempnerInst
NEW in the #kempnerinstitute blog: A method to predict how #llms scale w/ compute across different datasets from @brandfonbrener, @nikhil_anand91, @vyasnikhil96, @eranmalach, & @ShamKakade6. Read it here: kempnerinstitute.harvard.edu/research/deeper-learning/los…
-
Sham Kakade reposted
David Brandfonbrener
@brandfonbrener
Sunday at the OPT-ML workshop
Sham Kakade
@ShamKakade6
Which optimizer is opt? Our new work compares SGD, Adam, Adafactor (+ momentum), Lion, and, simply, SignSGD on LLM training wrt performance _and_ hyperparameter stability. tldr: Use anything but SGD, the rest are nearly identical:
-
Sham Kakade reposted
Yuda Song
@yus167
LLM self-improvement has critical implications in synthetic data, post-training and test-time inference. To understand LLMs' true capability of self-improvement, we perform large-scale experiments with multiple families of LLMs, tasks and mechanisms. Here is what we found: (1/9)
-
Sham Kakade reposted
Akshara Prabhakar
@aksh_555
Have a task that can be decomposed into two tasks requiring different skills? BUT
- it is difficult to generate expert-curated training data?
- do not want to use RAG?
🚀 Introducing LoRA Soups: Merging LoRAs for Practical Skill Composition Tasks
🔗 arxiv.org/abs/2410.13025
1/n -
Sham Kakade reposted
Yilun Du
@du_yilun
How do we steer behavioral cloning policies to the intents of human users?
Introducing a training-free method that allows us to directly specify and steer policies to ad-hoc user intents at execution time!
Felix Wang
@felixwyw
Want your robot to clean the kitchen your way? 🧹✨
🔗yanweiw.github.io/itps/"
Introducing Inference-Time Policy Steering: a training-free method that lets you specify where and how to manipulate objects, so you can guide non-interactive policies to align with your preferences!This media couldn't be displayed. See it on the original post. -
Sham Kakade reposted
Kempner Institute at Harvard University
@KempnerInst
NEW: we have an exciting opportunity for a tenure-track professor at the #kempnerinstitute and @hseas. Read the full description & apply today: academicpositions.harvard.edu/postings/14362
#ml #ai -
Sham Kakade reposted
Depen Morwani
@depen_morwani
Very excited to speak at the Theory of Interpretable AI seminar coming Thursday at 10am EST. Please join to hear about the recent work on how margin maximization can explain some intriguing observations in mechanistic interpretability.
Suraj Srinivas
@Suuraj
📢Theory of Interpretable AI Seminar📢
On Thursday, @depen_morwani from Harvard will present work on how **margin maximization** can explain observed phenomena in mech. interpretability!
⏲️Thursday Dec 5th, 4pm CET / 10am EST
🌐tverven.github.io/tiai-seminar/
@ML_Theorist @tverven -
Sham Kakade reposted
Kempner Institute at Harvard University
@KempnerInst
A huge congratulations to two Kempner-affiliated undergrads for each winning a #rhodesscholarship:
@aneeshers (mentored by @cogscikid & @gershbrain)
and
@ayushnoori (advised by @marinkazitnik)
news.harvard.edu/gazette/story/2024/11/rhodes-scholars-sh… @hseas @harvardmed -
Sham Kakade
@ShamKakade6
(1/n) 💡How can we speed up the serial runtime of long pre-training runs? Enter Critical Batch Size (CBS): the tipping point where the gains of data parallelism balance with diminishing efficiency. Doubling batch size halves the optimization steps—until we hit CBS, beyond which returns diminish.
Post not available here, view on X
Sham Kakade
@ShamKakade6
(5/n) 🤝 Shoutout to some great collaborators:
@hanlin_zhang, @depen_morwani, @vyasnikhil96, @uuujingfeng, @difanzou, @udayaghai
#ai #ml #scalinglaws
Sham Kakade
@ShamKakade6
fixed tag: @_hanlin_zhang_
-
Sham Kakade reposted
David Brandfonbrener
@brandfonbrener
How does test loss change as we change the training data? And how does this interact with scaling laws?
We propose a methodology to approach these questions by showing that we can predict the performance across datasets and losses with simple shifted power law fits. -
Sham Kakade reposted
Kempner Institute at Harvard University
@KempnerInst
We're proud to share the #kempnerinstitute’s AI cluster has joined the ranks of the fastest green supercomputers in the world (32nd on Green500 and 85th in TOP500)! Read the full announcement: kempnerinstitute.harvard.edu/news/kempner-institutes-ai-c… #ml #computing #sc24
-
Sham Kakade reposted
Jenny Kaufmann
@JKaufmann23072
Are we headed for a world where every student has a 24/7 1-1 tutor?
My takes on the AI tutoring landscape right now, and what's coming: aiinterpreter.substack.com/p/the-coming-army-of-ai-tutors (tldr: "cheap + widely available" is a big deal even if the AIs aren't human-quality at everything.) -
Sham Kakade reposted
Charlie Blake
@thecharlieblake
Thanks to everyone who made our week in Boston so fruitful and shared their time and ideas
Particularly @jxbz, along with @jyo_pari & @HanGuo97 for inviting us to MIT and showing us the city, and @EranMalach & @ShamKakade6 for welcoming us to Harvard. We're most grateful 🙏 -
Sham Kakade reposted
Yilun Du
@du_yilun
Cool work showing how we integrate slow/fast learning systems to get video models with memory!
Yining Hong @CoRL
@yining_hong
🎬Meet SlowFast-VGen: an action-conditioned long video generation system that learns like a human brain!
🧠Slow learning builds the world model, while fast learning captures memories - enabling incredibly long, consistent videos that respond to your actions in real-time.
🔗slowfast-vgen.github.io
📑arxiv.org/pdf/2410.23277 -
Sham Kakade reposted
Kempner Institute at Harvard University
@KempnerInst
We're hiring! Our Associate Director of Educational Programs oversees all aspects of the #kempnerinstitute's fellowship and training programs for undergraduate, post-baccalaureate, and graduate students. Apply today: sjobs.brassring.com/TGnewUI/Search/home/HomeWithPreLoad?p… #ai #education @Harvard
-
Sham Kakade reposted
Ella Batty
@EllaBatty
Excited to share that I am transitioning to be a Senior Machine Learning Research Scientist at @KempnerInst! It’s been a privilege to help launch Kempner educational programs in my previous role, and I’m looking forward to diving back into research.
-
Sham Kakade reposted
Kempner Institute at Harvard University
@KempnerInst
We're thrilled to welcome our newest cohort of KURE undergrads who are working on projects focused on the foundations of intelligence. Meet the students and their mentors: kempnerinstitute.harvard.edu/news/kempner-institute-welco… #kempnerinstitute #ai @ShamKakade6 @blsabatini
-
Sham Kakade
@ShamKakade6
MoEs: are they parrots or can they reason? Learn more about when MoEs offer a ‘free lunch’ in our latest findings.
Eran Malach
@EranMalach
MoEs increase parameter count but not FLOPs. Do they offer "free lunch", improving performance without paying in compute?
Our answer: for memorization, MoEs give performance gains "for free", but have limited benefit for reasoning!
Arxiv: arxiv.org/pdf/2410.19034 🦜🦜🦜 -
Sham Kakade
@ShamKakade6
1/5⚡Introducing Flash Inference: an *exact* method cutting inference time for Long Convolution Sequence Models (LCSMs) to near-linear O(L log² L) complexity! Faster inference, same precision. Learn how we accelerate LCSM inference.
Sham Kakade
@ShamKakade6
2/5Flash Inference speeds up the convolution-based layers computation by up to 50× while yielding an end-to-end Hyena inference time improvement of up to 1.6×.
Sham Kakade
@ShamKakade6
3/5 How? By combining a custom tiling and some FFT magic to streamline memory movement and cut computational costs. The result? Faster inference with no accuracy compromises.
Sham Kakade
@ShamKakade6
4/5 Our approach works on any LCSM (Hyena, SGConv, H3, Ckconv). We identified key architectural traits that make this possible, from contribution-based layers to well-defined tiles.
Sham Kakade
@ShamKakade6
5/5 To see how parallelization and other optimizations boost performance, check out the paper here: arxiv.org/abs/2410.12982. Work with @costinoncescu, @sanketpurandare, and Stratos Idreos.
-
Sham Kakade
@ShamKakade6
I did my PhD there!
Zoubin Ghahramani
@ZoubinGhahrama1
I moved from Toronto to the London with @geoffreyhinton as one of the founding faculty of the @GatsbyUCL Unit. A few years afterwards, @demishassabis did his PostDoc there. Congrats to the Unit; what an amazing place! amp.theguardian.com/science/2024/oct/11/nobel-awards-high…
-
Sham Kakade reposted
Gokul Swamy
@g_k_swamy
Super excited to announce that REBEL has been accepted to #neurips2024 🚀! We're also unveiling a brand new suite of Llama-3-8B 🦙models fine-tuned with REBEL that hit GPT-4 level on AlpacaEval 2.0, while keeping strong performance on MMLU & GSM8K! Code, model, and paper ⬇️
-
Sham Kakade reposted
Keller Jordan
@kellerjordan0
NanoGPT speedrunning update: Using the SOAP optimizer (arxiv.org/abs/2409.11321), @vyasnikhil96 has achieved a new sample efficiency record of 3.28 Fineweb validation loss in 3.25B training tokens. The previous record was 3.67B tokens by my proposed optimizer.
-
Sham Kakade
@ShamKakade6
Fantastic session today on LLM evaluations with @JuliaANeagu and @DeannaLEmery! The case they presented on Wayfair’s use of generative AI, powered by @QuotientAI, was a standout example of creating real, privatizable value with AI. Grateful to have hosted the insightful discussion.
Julia Neagu
@juliaaneagu
Just taught a Harvard class on LLM evals with @DeannaLEmery! Thanks @ShamKakade6 for hosting us. Almost 3 hours of pure eval goodness.
-
Sham Kakade reposted
Jennifer Hu
@_jennhu
Many thanks @yoavartzi, and thank you to all the @COLM_conf organizers for a fantastic conference!!! 🥳🦙
I'm honored to receive a paper award with @mcxfrank at the first-ever COLM, and so excited to see more cogsci-related work in the COLMs to come! -
Sham Kakade reposted
Yilun Du
@du_yilun
Excited to share our new benchmark for detailed video captioning!
Existing video generative models are handicapped by the availability of detailed text descriptions. We curate a new video captioning benchmark 20x more detailed than prior datasets!
rese1f.github.io/aurora-web/
Wenhao Chai
@wenhaocha1
(1/n) 📸Want to train a better video generation model? You need a better video captioner first! Check out AuroraCap! Trained on 20M+ high-quality data, it’s more efficient at inference. #lmms #videogen
🔗Homepage: rese1f.github.io/aurora-web/ -
Sham Kakade reposted
Yilun Du
@du_yilun
Introducing Streaming Diffusion Policy (SDP)!
SDP allows you to generate actions using a single denoising step, with no distillation!
SDP keeps an action buffer of partially denoised future actions, which is refined over time.
streaming-diffusion-policy.github.io
Led by @sigmundhh -
Sham Kakade
@ShamKakade6
1/n Introducing SOAP (ShampoO with Adam in the Preconditioner's eigenbasis): A deep learning optimization algorithm that applies Adam in Shampoo's eigenbasis. SOAP outperforms both AdamW and Shampoo in language model pretraining.
Post not available here, view on X
Sham Kakade
@ShamKakade6
7/n tagging some people who might be interested: @_arohan_ @cloneofsimo @jxbz @runame_ @typedfemale @QuanquanGu @aaron_defazio @tfjgeorge @HessianFree @kellerjordan0 @LinYorker @SadhikaMalladi
Sham Kakade
@ShamKakade6
8/n Code at github.com/nikhilvyas/SOAP.
-
Sham Kakade reposted
Yohan J. John
@DrYohanJohn
This week I began a new role as Science Writer at the (very new!) Kempner Institute for the Study of Natural & Artificial Intelligence at Harvard. @KempnerInst
I'm also teaching intro to comp neuro at BU this semester, so it's going to be a busy+stimulating few months! -
Sham Kakade reposted
Kempner Institute at Harvard University
@KempnerInst
Introducing PINNACLE, a novel contextual #ai model for single-cell protein biology that supports a broad array of #biomedicalai tasks by tailoring outputs to individual cell type contexts. Read more: kempnerinstitute.harvard.edu/research/deeper-learning/con… @_michellemli @marinkazitnik #kempnerinstitute
-
Sham Kakade
@ShamKakade6
Please apply and spread the word! These positions are pretty great, and it's a wonderful community to study the most exciting AI and neuro questions!
Kempner Institute at Harvard University
@KempnerInst
The application is now open for our #kempnerinstitute Research #fellowship! Postdocs studying the foundations of #intelligence or applications of #ai are encouraged to apply. Learn more and apply by Oct. 1: kempnerinstitute.harvard.edu/kempner-institute-research-f… #llms #neuroai #ml @ShamKakade6 @blsabatini
-
Sham Kakade
@ShamKakade6
Thrilled to have you join and super excited for what’s next!!
Michael Albergo
@msalbergo
🌞 One final bit of news — excited to announce I’ll be starting as an assistant professor in applied mathematics @Harvard and @KempnerInst Investigator in 2026 :) Please reach out if you may be interested in working with me. Grateful for everyone that made this possible!
Kempner Institute at Harvard University
@KempnerInst
We are thrilled to announce the appointments of three new #kempnerinstitute Investigators who will play key roles in shaping & advancing our research program. Learn more about @xkianteb, @du_yilun and @msalbergo in our official announcement: kempnerinstitute.harvard.edu/news/kiante-brantley-yilun-d… @hseas #ai #ml
-
Sham Kakade
@ShamKakade6
We are thrilled to have this stellar cohort joining us!! Super excited for the new directions they are pursuing in AI.
Kempner Institute at Harvard University
@KempnerInst
We are thrilled to announce the appointments of three new #kempnerinstitute Investigators who will play key roles in shaping & advancing our research program. Learn more about @xkianteb, @du_yilun and @msalbergo in our official announcement: kempnerinstitute.harvard.edu/news/kiante-brantley-yilun-d… @hseas #ai #ml
-
Sham Kakade reposted
Kempner Institute at Harvard University
@KempnerInst
Be sure to check out our #kempnerinstitute posters, workshops & presentations this week at #icml2024!
@AdaFang_ @blake__bordelon @CPehlevan @SimmonsEdler @nsaphra @KanakaRajanPhD @brandfonbrener @rosieyzh @EranMalach @RyanPaulBadman1 @marinkazitnik @_hanlin_zhang_ @boazbaraktcs -
Sham Kakade reposted
Jennifer Hu
@_jennhu
Excited to share that I will join Johns Hopkins as an Assistant Professor of Cognitive Science in July 2025! I am starting the ✨Group for Language and Intelligence (GLINT)✨, which will study how language works in minds and machines: glintlab.org
Details below 👇 1/4 -
Sham Kakade
@ShamKakade6
Which optimizer is opt? Our new work compares SGD, Adam, Adafactor (+ momentum), Lion, and, simply, SignSGD on LLM training wrt performance _and_ hyperparameter stability. tldr: Use anything but SGD, the rest are nearly identical:
Sham Kakade
@ShamKakade6
The previous plot showed performance and stability wrt learning rate, showing that tuning is stable across orders of magnitude. More generally, practitioners also care about hyperparameter stability and tuning for other params. Again, all except SGD are similarly stable:
Sham Kakade
@ShamKakade6
So if the memory utilization of Adam is a concern, use Lion or Adafactor (with momentum). Importantly, our results suggest no performance hit.
Post not available here, view on X
Sham Kakade
@ShamKakade6
Work done with co-authors @rosieyzh @depen_morwani @vyasnikhil06 @brandfonbrener. See our paper and blog post for details: arxiv.org/abs/2407.07972 kempnerinstitute.harvard.edu/research/deeper-learning/any…
Sham Kakade
@ShamKakade6
also, see more details in the post from @rosieyzh:
Rosie Zhao
@rosieyzh
In our new work on evaluating optimizers for LLM training, we perform a series of experiments to investigate the role of adaptivity in optimizers like Adam in achieving good performance and stability. A thread: 🧵
-
Sham Kakade
@ShamKakade6
What's the opt optimizer? New work comparing (diagonally conditioned) first order methods.
Kempner Institute at Harvard University
@KempnerInst
NEW #kempnerinstitute blog: @rosieyzh, @depen_morwani, @brandfonbrener, @vyasnikhil96 & @ShamKakade6 study a variety of #llm training optimizers and find they are all fairly similar except for SGD, which is notably worse. Read more: kempnerinstitute.harvard.edu/research/deeper-learning/any… #ml #ai
-
Sham Kakade reposted
Eran Malach
@EranMalach
Can Transformers extrapolate from short training sequences to long ones? Our new work shows that they display surprising “length generalization” capabilities on many algorithmic tasks: addition, multiplication, and even in-context simulation of SGD! arxiv.org/abs/2407.03310
-
Sham Kakade reposted
Kempner Institute at Harvard University
@KempnerInst
We're thrilled to introduce the 2024 cohort of #kempnerinstitute Graduate Fellows! This year’s recipients include seven incoming and eight continuing graduate students enrolled across six @Harvard Ph.D. programs. Read more: kempnerinstitute.harvard.edu/news/kempner-institute-annou…
#ai #neuroai #ml -
Sham Kakade reposted
Kempner Institute at Harvard University
@KempnerInst
We are thrilled to announce the appointments of two new associate faculty members: @elmelis and @pjslade - pioneering research scientists who will join the #kempnerinstitute in July! Read more: kempnerinstitute.harvard.edu/news/kempner-institute-annou… #ai #ml #neuroai
-
Sham Kakade
@ShamKakade6
Why does Shampoo work well? Our new work sheds light on this, highlighting a wide misconception about the optimizer. We show the *square* of Shampoo's preconditioner is provably near to the optimal Kronecker approximation of the (Adagrad) Hessian. See:
arxiv.org/pdf/2406.17748
Sham Kakade
@ShamKakade6
In practice, the exponent in the preconditioner is often a hyperparameter, so the difference doesn't manifest. But this view significantly alters our understanding behind the workings of Shampoo. w/ the great team @depen_morwani @IShapira1 @vyasnikhil96 @EranMalach Lucas Janson
Sham Kakade
@ShamKakade6
A little more info from @vyasnikhil96:
Nikhil Vyas
@vyasnikhil96
1/n A technical thread on our results in arxiv.org/pdf/2406.17748 on connecting the Shampoo optimizer and Optimal Kronecker product approximation of the the Adagrad (or Hessian) preconditioner.
Sham Kakade
@ShamKakade6
Why does Shampoo work well? Our new work sheds light on this, highlighting a wide misconception about the optimizer. We show the *square* of Shampoo's preconditioner is provably near to the optimal Kronecker approximation of the (Adagrad) Hessian. See:
arxiv.org/pdf/2406.17748 -
Sham Kakade reposted
Aran Komatsuzaki
@arankomatsuzaki
Recite, Reconstruct, Recollect: Memorization in LMs as a Multifaceted Phenomenon
- breaks memorization down into a taxonomy
- uses it to construct a predictive model for memorization
- finds that different factors influence the likelihood of memorization differently -
Sham Kakade
@ShamKakade6
Welcome @du_yilun!! So excited to have you join! This is going to be epic :)
@KempnerInst
Yilun Du
@du_yilun
Super excited to be joining Harvard as an assistant professor in Fall 2025 in @KempnerInst and Computer Science!!
I will be hiring students this upcoming cycle -- come join our quest to build the next generation of AI systems! -
Sham Kakade
@ShamKakade6
Amazing to be connecting with the Harvard ugrads through this Kempner research program!
Kempner Institute at Harvard University
@KempnerInst
Our first cohort of KRANIUM @Harvard undergrads is here! Learn more about this intensive summer research program, the students, their mentors, and their projects: kempnerinstitute.harvard.edu/news/kempner-kicks-off-summe… #ai #neuroai #ml
-
Sham Kakade reposted
-
Sham Kakade reposted
Alex Dimakis
@AlexGDimakis
This paper seems very interesting: say you train an LLM to play chess using only transcripts of games of players up to 1000 elo. Is it possible that the model plays better than 1000 elo? (i.e. "transcends" the training data performance?). It seems you get something from nothing, and some information theory arguments that this should be impossible were discussed in conversations I had in the past. But this paper shows this can happen: training on 1000 elo game transcripts and getting an LLM that plays at 1500! Further the authors connect to a clean theoretical framework for why: it's ensembling weak learners, where you get "something from nothing" by averaging the independent mistakes of multiple models. The paper argued that you need enough data diversity and careful temperature sampling for the transcendence to occur. I had been thinking along the same lines but didn't think of using chess as a clean measurable way to scientifically measure this. Fantastic work that I'll read I'll more depth.
-
Sham Kakade reposted
Leo Dirac
@leopd
This is a clear nice counter-point to the IMO all-too-frequent negativity about current AI along the lines of "how could it ever ___ when it's just trained on ___?"
-
Sham Kakade
@ShamKakade6
CoLor-Filter: It's all about the data! Our new algorithm learns data mixtures for better pertaining performance. Check it out here:
David Brandfonbrener
@brandfonbrener
🌈 New paper! 🌈
“CoLoR-Filter: Conditional Loss Reduction Filtering for Targeted Language Model Pre-training”
We introduce a simple approach for data selection and see >11x gains in data efficiency on downstream target tasks when sub-selecting from C4 for 1.2B models. 🧵 -
Sham Kakade
@ShamKakade6
Can modern generative models "transcend" the data which they were trained on? Yes! We looked at this for chess. They can in fact beat the experts they were trained on! See the blogpost and preprint:
Kempner Institute at Harvard University
@KempnerInst
NEW blog post! Edwin Zhang and @EranMalach show how generative models in #chess can surpass the performance of their training data. Read more on our #kempnerinstitute blog and check out the preprint: kempnerinstitute.harvard.edu/research/deeper-learning/tra… #transcendence
-
Sham Kakade
@ShamKakade6
Excited to be working with this cohort!
Kempner Institute at Harvard University
@KempnerInst
We’re thrilled to announce #kempnerinstitute’s first cohort of post-bac scholars! These 4 recent college grads will undertake extensive mentored research & coursework with 2 years of funding & professional development. Read more: kempnerinstitute.harvard.edu/news/announcing-2024-kempner… @harvardGSAS #ai #neuroai
-
Sham Kakade
@ShamKakade6
A good overview of these limits! This def a nice way to try and handle hyper-parameter transfer!
Kempner Institute at Harvard University
@KempnerInst
NEW! Check out recent findings on width and depth limits in part 1 of a #kempnerinstitute two-part series from @ABAtanasov, @blake__bordelon & @CPehlevan. Read on: kempnerinstitute.harvard.edu/research/deeper-learning/inf… #neuralnetworks #ai
-
Sham Kakade reposted
Kempner Institute at Harvard University
@KempnerInst
We’re pleased to recognize the great work of our first cohort of KURE undergraduate students who just completed their research projects. Read more: kempnerinstitute.harvard.edu/news/kure-undergraduates-cel… #kempnerinstitute #ai #neuroai @Harvard @ShamKakade6 @blsabatini @EllaBatty
-
Sham Kakade reposted
Simons Foundation
@SimonsFdn
It is with great sadness that the Simons Foundation announces the death of its co-founder and chair emeritus, James Harris Simons. Jim was an award-winning mathematician, a legendary investor and a generous philanthropist. www.simonsfoundation.org/2024/05/10/simons-foundation-co-…
-
Sham Kakade reposted
Depen Morwani
@depen_morwani
Looking forward to presenting this work at #iclr2024 next week in Vienna
Stop by our spotlight poster on 7th (4:30-6:30 pm) for understanding how the principle of margin maximization alone can explain some recent intriguing observations by mechanistic interpretability community
Sham Kakade
@ShamKakade6
Can inductive biases explain mechanistic interpretability? Why do sinusoidal patterns emerge for NNs trained on modular addition? (e.g. @NeelNanda5)
New work pins this down! w/ @depen_morwani @EdelmanBen @rosieyzh @costinoncescu! @KempnerInst -
Sham Kakade reposted
Ilenna Jones, PhD @ ilennaj.bsky.social
@IlennaJ
Happy to help make this intro - I've been having a great time at the @KempnerInst so far! Every meeting has been fun and interesting!
The biggest stress I've had has been the freedom - which highlights my need to answer the question:
Who do I want to be going forward?
Kempner Institute at Harvard University
@KempnerInst
[2/2] Video feature: take a closer look into the #kempnerinstitute's mission and research - we're tackling some of the most compelling mysteries of our time. Watch now: www.youtube.com/watch?v=DHmYv7Y-jxE #ai #ml @Harvard @blsabatini @ShamKakade6
-
Sham Kakade reposted
Bernardo Sabatini
@blsabatini
Cool. The space is fantastic and it’s so exciting to see the community in action in there.
Kempner Institute at Harvard University
@KempnerInst
New from the @Harvard Gazette: #kempnerinstitute is a place for researchers to meet, and for #ai and #naturalintelligence to do the same. Read about our mission and see inside our state-of-the-art space: news.harvard.edu/gazette/story/2024/04/a-space-for-resear… @blsabatini @ShamKakade6 [1/2]
-
Sham Kakade reposted
Gustaf Ahdritz
@gahdritz
In new work (with Tian Qin, @vyasnikhil96, @boazbaraktcs, and @EdelmanBen), we show that LLMs can identify their own epistemic uncertainty in free-form text, suggesting new approaches for combating hallucinations. (1/6)
Paper: arxiv.org/abs/2402.03563
Blog: kempnerinstitute.harvard.edu/research/deeper-learning/dis… -
Sham Kakade reposted
Kempner Institute at Harvard University
@KempnerInst
New Deeper Learning blog post: a linear probe can unlock LM's metacognitive capability to distinguish tokens that are "knowable" from tokens where its predictions can't be improved. kempnerinstitute.harvard.edu/research/deeper-learning/dis…
@gahdritz, Tian Qin, @vyasnikhil96, @boazbaraktcs, @EdelmanBen
#ai #llm -
Sham Kakade reposted
Surbhi Goel
@SurbhiGoel_
New initiative from @let4all - Office Hours!
I (along with @thegautamkamath and Ellen Vitercik) will be holding bi-weekly office hours starting next week. Open to chat about research, navigating academic life, and anything else!
Learning Theory Alliance
@let4all
The Learning Theory Alliance is piloting a new office hours program! Drop in and chat about *anything* with experienced learning theorists, including @SurbhiGoel_ @thegautamkamath and Ellen Vitercik.
Starts tomorrow at 9 AM with Gautam!
More details in next tweet 👇 -
Sham Kakade reposted
Aditya Kusupati
@adityakusupati
Gecko elastic embeddings powered by Matryoshka🪆 announced today at @Google CloudNext today!
Available through Cloud (Vertex) and Google AI Studio.
developers.googleblog.com/2024/04/gemini-15-pro-in-public… -
Sham Kakade
@ShamKakade6
Can’t wait to work with this amazing new cohort!
Kempner Institute at Harvard University
@KempnerInst
Thrilled to announce the 2024 recipients of #kempnerinstitute Research Fellowships: Thomas Fel, Mikail Khona, Bingbin Liu, Isabel Papadimitriou, Noor Sajid, & Aaron Walsman! kempnerinstitute.harvard.edu
@Napoolar @KhonaMikail @BingbinL @isabelpapad @nsajidt @aaronwalsman -
Sham Kakade reposted
Ezra Edelman
@ezra_edelman
How do induction heads / in-context learning emerge? We study a new synthetic task to better understand how these circuits evolve (in stages!) over time.
w/ @EdelmanBen, @SurbhiGoel_, @EranMalach & Nikos Tsilivis
Blog: unprovenalgos.github.io/statistical-induction-heads
Paper: arxiv.org/abs/2402.11004 (1/7) -
Sham Kakade reposted
Kempner Institute at Harvard University
@KempnerInst
We are thrilled to congratulate the #kempnerinstitute's
@HSompolinsky for winning the prestigious
@BrainPrize, the world's largest #brainresearch prize - awarded to researchers with ground-breaking impact on #neuroscience! news.harvard.edu/gazette/story/2024/03/haim-sompolinsky-a…
@HarvardGSAS @harvardphysics -
Sham Kakade reposted
Marinka Zitnik
@marinkazitnik
📢 New @Harvard Seminar Series on Efficient ML! 👇
Thanks @schwarzjn_ for spearheading this effort!
@HarvardDBMI @KempnerInst @harvard_data @harvardmed @zakkohane @ShamKakade6
🎓 Check out a phenomenal speaker lineup with an opening by LLM superstars @sarahookr @cohere and @valentina__py @allen_ai
🧑🏫 Each session starts with a tutorial by a rising star researcher, followed by a talk on recent work and open questions by an established scientist in the field
Jonathan Richard Schwarz
@schwarzjn_
🚨 Excited to announce our new @Harvard seminar series on Efficient ML! Please join LLM superstars @sarahookr (@cohere) & @valentina__py (@allen_ai) for our kick-off meeting next Monday!
Open to everyone! 📅
🔗 sites.google.com/view/harvard-efficient-ml/
@harvard_data @harvardmed @KempnerInst -
Sham Kakade reposted
Kenneth Li
@ke_li_2021
We propose Q-probe, a simple technique that improves coding and alignment for LLM, without requiring fine-tuning!. The idea is to learn a "task vector" in the hidden space and use it to select from multiple candidate generations. arxiv.org/abs/2402.14688
-
Sham Kakade
@ShamKakade6
Big congrats tot he 2024 Sloan Research Fellows!
Sloan Foundation
@SloanFoundation
We have today announced the names of the 2024 Sloan Research Fellows! Congratulations to these 126 outstanding early-career researchers:
sloan.org/fellowships/2024-Fellows -
Sham Kakade reposted
Wilka Carvalho
@cogscikid
Excited to share our review on the successor representation and two important generalizations: successor features and successor models!
These provide answers spanning AI, cognitive science, and neuroscience for how agents can efficiently reuse their knowledge in new situations -
Sham Kakade reposted
Kempner Institute at Harvard University
@KempnerInst
Calling all @Harvard undergrads interested in the study of intelligence: our summer research program’s application deadline is Feb. 14! More details: kempnerinstitute.harvard.edu #kempnerinstitute
@EllaBatty @ShamKakade6 @blsabatini -
Sham Kakade
@ShamKakade6
Retweet after me...
Eran Malach
@EranMalach
Our recent work on the comparison between Transformers and State Space Models for sequence modeling now on arxiv! TLDR - we find a key disadvantage of SSMs compared to Transformers: they cannot copy from their input. 🧵
Arxiv: arxiv.org/abs/2402.01032
Blog: kempnerinstitute.harvard.edu -
Sham Kakade
@ShamKakade6
Repeat After Me:
Transformers are Better than State Space Models at Copying
Transformers are Better than State Space Models at Copying
Kempner Institute at Harvard University
@KempnerInst
Check out #kempnerinstitute’s newest blog post! Authors Samy Jelassi, @brandfonbrener, @ShamKakade6 @EranMalach show the improved efficiency of State Space Models sacrifices some core capabilities for modern LLMs. #machinelearning #ai kempnerinstitute.harvard.edu
-
Sham Kakade
@ShamKakade6
It's a really great codebase and excited for future collabs with @allen_ai!
Kempner Institute at Harvard University
@KempnerInst
So excited to collaborate with @allen_ai and its partners
@databricks @AMD @LUMIhpc on this groundbreaking work. Special thanks to @KempnerInst’s co-director @ShamKakade6 and engineering lead @maxshadx! -
Sham Kakade
@ShamKakade6
It was a great talk with an engaged audience!
Kempner Institute at Harvard University
@KempnerInst
Thanks to @OpenAI’s Noam Brown for joining the @KempnerInst Seminar Series to discuss CICERO, the first #ai agent to achieve human-level performance in the strategy game Diplomacy.
Next in the series : Rajesh Rao on Feb 16.
@polynoamial @ShamKakade6 @blsabatini @boazbaraktcs -
Sham Kakade reposted
Joelle Pineau
@joellepineau
We just shared a new larger and improved Code Llama 70B parameter model. With every release, we have better generation quality, but also a better sense of how to make these models safe.
AI at Meta
@AIatMeta
CodeLlama-70B-Instruct achieves 67.8 on HumanEval, making it one of the highest performing open models available today.
CodeLlama-70B is the most performant base for fine-tuning code generation models and we’re excited for the community to build on this work. -
Sham Kakade reposted
Prateek Jain
@jainprateek_
Pace of progress in AI is lightning!
@OpenAI released MRL style text embeddings, mirroring our NeurIPS '22 paper (w/ awesome folks from UW and Harvard).
However, as an advocate of open science, I am a bit disappointed with rebranding to "shortening embs" without ref to MRL 1/n
Aditya Kusupati
@adityakusupati
🤯WOW🪆Matryoshka Representation Learning enables "native support for shortening embs" &"very flexible usage"
Jokes aside, excited that @OpenAI serves MRL by default in v3 embedding API for retrieval & RAG!
Other models & services should catch-up soon😄
arxiv.org/abs/2205.13147 -
Sham Kakade reposted
Damek
@damekdavis
Plugging work done with @ShamKakade6, @jasondeanlee, and Dima Drusvyatskiy where we showed “tameness” is enough to guarantee criticality of limit points.
Gabriel Peyré
@gabrielpeyre
Gradient descent does not always converge. Trajectories can have infinite length. Picture & insights from @GuillaumeG_ epubs.siam.org/doi/abs/10.1137/040605266
-
Sham Kakade reposted
Kempner Institute at Harvard University
@KempnerInst
Calling all @Harvard undergrads interested in the study of intelligence: our spring research program’s application deadline is Jan. 28! More details: kempnerinstitute.harvard.edu #kempnerinstitute
@EllaBatty @ShamKakade6 @blsabatini -
Sham Kakade reposted
Quanquan Gu
@QuanquanGu
The aforementioned papers are authored by esteemed researchers like Peter Bartlett, @philipmlong, @lugosi_gabor @ShamKakade6, Nati Srebro, Misha Belkin, along with many amazing young researchers including @_YuanCao_ @_zxchen_ @difanzou @uuujingfeng @sfrei_ @niladrichat @galvardi @weihu_ @YiwenKou666 et al.
-
Sham Kakade
@ShamKakade6
Please spread the word!
Kempner Institute at Harvard University
@KempnerInst
Our post-bac fellowship application deadline is fast approaching! Read more about this program and apply today: kempnerinstitute.harvard.edu #kempnerinstitute
@EllaBatty @grez72 @ShamKakade6 @blsabatini @HarvardGSAS -
Sham Kakade reposted
Kempner Institute at Harvard University
@KempnerInst
If you’re a @Harvard undergrad interested in pursuing research related to the study of intelligence, consider applying to the #kempnerinstitute’s spring & summer programs! Application deadlines are approaching: kempnerinstitute.harvard.edu
@EllaBatty @ShamKakade6 @blsabatini -
Sham Kakade reposted
Bingbin Liu
@BingbinL
Please come to our poster at the MATH-AI workshop! 😊
(Room 217-219, 3-4pm)
Sebastien Bubeck
@SebastienBubeck
Relatedly, take a look at this work arxiv.org/abs/2312.09241 where we fine-tune phi-1.5 on math (achieving >GPT-3.5 on GSM8k). Work led by @BingbinL and her mentor at MSR @YiZhangZZZ (note: the data described there, TinyGSM, is different from the 1M exercises for the table above).
-
Sham Kakade reposted
Wilka Carvalho
@cogscikid
If you've been out of your undergrad for < 3 years, want to do an AI PhD and want more research experience, please consider this postbac program @Harvard
Housing + stipend provided
Can take Harvard classes
Get to work with world class researchers -
Sham Kakade reposted
Max Shad
@maxshadx
Discover exciting AI/ML research and engineering opportunities at @Kempnerinst at @Harvard, spanning data engineering, software engineering, model development, deployment, and distributed #gpu training.
#hiringnow #ml #ai #kempnerinstitute #gpu
Kempner Institute at Harvard University
@KempnerInst
The #kempnerinstitute is hiring for several #ai #ml engineering roles to help build cutting-edge, open-source AI models and data. See the opportunities here: kempnerinstitute.harvard.edu
#nowhiring #aiengineer #joinourteam -
Sham Kakade
@ShamKakade6
this work did change my world view of resource tradeoffs: how more compute makes up for less data. the frontier plots were quite compelling! checkout the poster for more info!
Ben Edelman
@EdelmanBen
Will deep learning improve with more data, a larger model, or training for longer?
"Any balanced combination of them" <– in our #neurips2023 spotlight, we reveal this through the lens of gradient-based feature learning in the presence of computational-statistical gaps. 1/5 -
Sham Kakade
@ShamKakade6
I found this quite thought provoking!
Bingbin Liu
@BingbinL
🧵What’s the simplest failure mode of Transformers? Our #neurips2023 spotlight paper identifies the “attention glitches” phenomenon, where Transformers intermittently fail to capture robust reasoning, due to undesirable architectural inductive biases.
Poster: Wed 5-7pm CST, #528 -
Sham Kakade reposted
Depen Morwani
@depen_morwani
Excited to share our recent work at #neurips2023 on the nature of Simplicity Bias (SB) in 1-Hidden Layer Neural Networks (NNs) with Jatin Batra @jainprateek_ @PNetrapalli. SB is known to be one of the reasons behind brittleness of neural networks towards distribution shift (1/5)
-
Sham Kakade
@ShamKakade6
we are growing! please apply to join a vibrant community and please spread the word.
Kempner Institute at Harvard University
@KempnerInst
The #kempnerinstitute is hiring scientists, researchers, and engineers to join our growing community! Check out our openings and apply today: kempnerinstitute.harvard.edu #scienceforsocialgood #openscience
@Harvard @ChanZuckerberg @ShamKakade6 @blsabatini -
Sham Kakade reposted
Aditya Kusupati
@adityakusupati
📢📢At the last minute, I decided to go on the job market this year!!!
Grateful for RTs & promotion at your univ.😇
CV & Statements: adityakusupati.com
Will be at #neurips2023! presenting AdANNS, Priming, Objaverse & MADLAD. DM if you are around, would love to catch up👋 -
Sham Kakade reposted
Aditya Kusupati
@adityakusupati
📢📢We are releasing MatFormer based OLMo (@allen_ai) checkpoints (up to 1.3B params & 160B tokens) and code to further open research powered by @KempnerInst cluster🔥🔥
github.com/RAIVNLab/MatFormer-OLMo
Checkout our new article summarizing MatFormer as part of the Deeper Learning blog!
Kempner Institute at Harvard University
@KempnerInst
In our latest Deeper Learning blog post, the authors introduce an algorithmic method to elastically deploy large models, the #matformer. Read more: kempnerinstitute.harvard.edu #kempnerinstitute @adityakusupati @snehaark @Devvrit_Khatri @Tim_Dettmers
-
Sham Kakade reposted
Kempner Institute at Harvard University
@KempnerInst
We’re looking forward to following along at #neurips2023! Congratulations to all of our #kempnerinstitute community members whose work will be highlighted at @NeurIPSconf in New Orleans.
@marinkazitnik @CPehlevan @talia_konkle @grez72 @boazbaraktcs @cogscikid -
Sham Kakade
@ShamKakade6
interested in elastic ML? check out our new blog post. this should help serving foundation models on more devices and in more settings.
Kempner Institute at Harvard University
@KempnerInst
In our latest Deeper Learning blog post, the authors introduce an algorithmic method to elastically deploy large models, the #matformer. Read more: kempnerinstitute.harvard.edu #kempnerinstitute @adityakusupati @snehaark @Devvrit_Khatri @Tim_Dettmers
-
Sham Kakade reposted
Hugh Zhang
@hughbzhang
I also have no idea what Q* is, but given speculation that it’s a method of self-learning and Monte-Carlo Tree Search (MCTS) in language models, I thought I’d share some recent work on an adjacent idea.
-
Sham Kakade reposted
Simon Shaolei Du
@SimonShaoleiDu
We proved Q* is hard 4 years ago 🤷♂️
Sham Kakade
@ShamKakade6
What actually constitutes a good representation for reinforcement learning? Lots of sufficient conditions. But what's necessary? New paper: arxiv.org/abs/1910.03016. Surprisingly, good value (or policy) based representations just don't cut it! w/ @SimonShaoleiDu @RuosongW @lyang36
-
Sham Kakade reposted
Edith Cohen
@inhaleopenair
Beautiful work explaining how and why NN learn modular addition.
Sham Kakade
@ShamKakade6
Can inductive biases explain mechanistic interpretability? Why do sinusoidal patterns emerge for NNs trained on modular addition? (e.g. @NeelNanda5)
New work pins this down! w/ @depen_morwani @EdelmanBen @rosieyzh @costinoncescu! @KempnerInst -
Sham Kakade
@ShamKakade6
I found this to be very informative for LLM training. the science was just super well done. highly recommended for anyone training transformer based LLMs.
Mitchell Wortsman
@Mitchnw
Sharing some highlights from our work on small-scale proxies for large-scale Transformer training instabilities: arxiv.org/abs/2309.14322
With fantastic collaborators @peterjliu, @Locchiu, @_katieeverett, many others (see final tweet!), @hoonkp, @jmgilmer, @skornblith!
(1/15) -
Sham Kakade
@ShamKakade6
and the second post in Deeper Learning...
Kempner Institute at Harvard University
@KempnerInst
In our newest Deeper Learning #kempnerinstitute blog, authors @EdelmanBen, @depen_morwani, @costinoncescu, and @rosieyzh explain mechanic interpretability results using known inductive biases. Read it here: kempnerinstitute.harvard.edu @ShamKakade6 @Harvard #ai #machinelearning
-
Sham Kakade
@ShamKakade6
A downright classic. And let us take a moment to ponder on the most didactic figure of all time, seen in this beautiful paper: :)
Michael Nielsen
@michael_nielsen
This is amazing, and very beautiful:
-
Sham Kakade
@ShamKakade6
Exciting! New RL Conference. Thanks to @yayitsamyzhang and others for their leadership!
Amy Zhang
@amyzhanginRL
Thrilled to announce the first annual Reinforcement Learning Conference @RL_Conference, which will be held at UMass Amherst August 9-12! RLC is the first strongly peer-reviewed RL venue with proceedings, and our call for papers is now available: rl-conference.cc.
-
Sham Kakade
@ShamKakade6
Can inductive biases explain mechanistic interpretability? Why do sinusoidal patterns emerge for NNs trained on modular addition? (e.g. @NeelNanda5)
New work pins this down! w/ @depen_morwani @EdelmanBen @rosieyzh @costinoncescu! @KempnerInst
Sham Kakade
@ShamKakade6
Turns out margin maximization, yes just margin maximization, implies this emergence. Some cool new mathematical techniques let us precisely derive the max margin… (yup, that observed margin of 1/(105√426) is indeed what we predict).
Sham Kakade
@ShamKakade6
Theory extends to general finite groups (e.g. @bilalchughtai_ et al.). Many open questions.
See paper: arxiv.org/abs/2311.07568
and blog post: kempnerinstitute.harvard.edu -
Sham Kakade
@ShamKakade6
a really excellent result. very intuitive! (also, Kempner's new blog: Deeper Learning)
Boaz Barak
@boazbaraktcs
1/5 New preprint w @_hanlin_zhang_, Edelman, Francanti, Venturi & Ateniese!
We prove mathematically & demonstrate empirically impossibility for strong watermarking of generative AI models.
What's strong watermarking? What assumptions? See blog and 🧵
www.harvard.edu/kempner-institute/2023/11/09/watermarking… -
Sham Kakade
@ShamKakade6
Super cool result on the impossibility of watermarking!
(+ Kempner's new blog: Deeper Learning)
Kempner Institute at Harvard University
@KempnerInst
Deeper Learning, our new #kempnerinstitute blog is live! Check it out: kempnerinstitute.harvard.edu
In our first post, Ben Edelman, @_hanlin_zhang_ & @boazbaraktcs show that robust #watermarking in #ai is impossible under natural assumptions. Read more: kempnerinstitute.harvard.edu/research/deeper-learning/wat… -
Sham Kakade reposted
Kempner Institute at Harvard University
@KempnerInst
Our new post-bac fellowship program, in partnership w/ @HarvardGSAS Office for Equity, Diversity, Inclusion & Belonging, prepares recent college grads for PhD programs related to the study of intelligence. Applications are due Feb. 1. Learn more: kempnerinstitute.harvard.edu
-
Sham Kakade reposted
Kempner Institute at Harvard University
@KempnerInst
The #kempnerinstitute is hiring! We’ve just posted two new roles for tenure-track positions: one in #computerscience and one in applied #math. We're also recruiting faculty in #psychology and #computational #neuroscience. Learn more & apply today: kempnerinstitute.harvard.edu
-
Sham Kakade reposted
Max Shad
@maxshadx
I'm honored to support pioneering efforts to understand intelligence at @KempnerInst and university-wide #rse initiatives, and ready to take on this new chapter alongside my dedicated colleagues and inspiring leaders, Elise Porter, @compchemguy , @ShamKakade6 , @blsabatini
Kempner Institute at Harvard University
@KempnerInst
We're pleased to announce @mahmoodmsh as #kempnerinstitute's new Director of Engineering! Read more about Dr. Shad's expertise and research computing plans at the Kempner: kempnerinstitute.harvard.edu #computation #softwareengineering #unlockingintelligence @compchemguy @Harvard
-
Sham Kakade reposted
Kempner Institute at Harvard University
@KempnerInst
We're pleased to announce @mahmoodmsh as #kempnerinstitute's new Director of Engineering! Read more about Dr. Shad's expertise and research computing plans at the Kempner: kempnerinstitute.harvard.edu #computation #softwareengineering #unlockingintelligence @compchemguy @Harvard
-
Sham Kakade reposted
Kempner Institute at Harvard University
@KempnerInst
Tune in TODAY at 3:30p EST to hear from #kempnerinstitute's @KanakaRajanPhD on @NYUDocs!
Kempner Institute at Harvard University
@KempnerInst
📢 Don't miss our own @KanakaRajanPhD live on @SIRIUSXM's @NYUDocs this Tuesday, Oct. 31, 3:30p EST! Tune in:
📻SiriusXM Channel 110
📻SiriusXM app
📻Streaming online: www.siriusxm.com/channels/doctor-radio -
Sham Kakade reposted
Kempner Institute at Harvard University
@KempnerInst
We are thrilled to announce #kempnerinstitute's new scientific advisory board, composed of visionaries in #ai, machine learning & computational biology: @miramurati, @jpineau1, @StephenQuake, Aviv Regev, @EeroSimoncelli, & @svoboda314. Learn more: kempnerinstitute.harvard.edu @Harvard
-
Sham Kakade reposted
Kempner Institute at Harvard University
@KempnerInst
The #kempnerinstitute is hosting a seminar series jointly organized by @boazbaraktcs & @Harvard ML Foundations. Recent speakers: @denny_zhou & Emmanuel Abbe. Coming up: Eric Shultz, @polynoamial, Tom Griffiths, @StefanoErmon, @neuro_kim & @ev_fedorenko. kempnerinstitute.harvard.edu
-
Sham Kakade reposted
biohub
@biohub
We’re thrilled to share that we’re launching a new @CZBiohub in New York!
Bringing together engineers + scientists at @Columbia, @RockefellerUniv and @Yale, #czbiohubny will engineer immune cells for earlier detection & treatment of diseaseThis media couldn't be displayed. See it on the original post. -
Sham Kakade reposted
Aditya Kusupati
@adityakusupati
Announcing MatFormer - a nested🪆(Matryoshka) Transformer that offers elasticity across deployment constraints.
MatFormer is an architecture that lets us use 100s of accurate smaller models that we never actually trained for!
arxiv.org/abs/2310.07707 1/9 -
Sham Kakade reposted
Sasha Rush
@srush_nlp
Introducing COLM (colmweb.org) the Conference on Language Modeling. A new research venue dedicated to the theory, practice, and applications of language models.
Submissions: March 15 (it's pronounced "collum" 🕊️) -
Sham Kakade reposted
Kempner Institute at Harvard University
@KempnerInst
The #computationalcluster at the #kempnerinstitute is getting a lot bigger! We're excited to announce the purchase of 384 H100-80 GB Nvidia #gpus to support our research in natural and artificial intelligence. Learn more: kempnerinstitute.harvard.edu #openscience #ai #neuroscience
-
Sham Kakade reposted
Boaz Barak
@boazbaraktcs
Boaz Barak
@boazbaraktcs
Applications are now open for postdoctoral research fellows at the Kempner Institute at Harvard. These are 3-year positions with independent funding and access to the amazing resources of the institute. Apply by October 9th 2023!
www.harvard.edu/kempner-institute/opportunities/the-kempn… -
Sham Kakade
@ShamKakade6
Also attended this powerful lecture by Loretta Lynch, the 1st Black woman to serve as US Attorney General. Grateful for all she has done. ❤️
Zed Zha, MD, FAAFP
@DrZedZha
Attended the Dr. Martin Luther King Jr. commemorative lecture by Loretta Lynch, the 1st Black woman to serve as US Attorney General, introduced by Pro. Claudine Gay, the 1st Black president of Harvard University. The message was clear: Never Lose Infinite Hope. INFINITE HOPE. ❤️
-
Sham Kakade reposted
Kempner Institute at Harvard University
@KempnerInst
Congratulations to @CPehlevan who has been awarded the prestigious @NSF Career Award! Dr. Pehlevan studies how networks of neurons compute and learn in both natural and artificial contexts. Read more: kempnerinstitute.harvard.edu #ai #neuroscience #kempnerinstitute
-
Sham Kakade reposted
Kempner Institute at Harvard University
@KempnerInst
We recently purchased ~400 H100 GPUs and are hiring scientists, researchers & engineers to join our growing community! Check out our openings and apply today: kempnerinstitute.harvard.edu #kempnerinstitute #unlockingintelligence #scienceforsocialgood #openscience
-
Sham Kakade reposted
Kempner Institute at Harvard University
@KempnerInst
Congratulations to @marinkazitnik of the #kempnerinstitute for her recent publications in @Nature and @NatMachIntell. Her work offers a framework for making sense of today’s vast #ai landscape & understanding its role in scientific discovery. Read more: kempnerinstitute.harvard.edu
-
Sham Kakade reposted
Kempner Institute at Harvard University
@KempnerInst
Thanks to all who attended our opening event yesterday! We're thrilled to welcome our community members and special guests like @Harvard's John Shaw, Vice Provost for Research and @DOC_DNA, Assoc Provost for Research. #kempnerinstitute's future is bright! #ai #neuroscience
-
Sham Kakade reposted
Boaz Barak
@boazbaraktcs
Reminder that the deadline for applying to Kempner postdoctoral research fellowship is getting closer.
Boaz Barak
@boazbaraktcs
Applications are now open for postdoctoral research fellows at the Kempner Institute at Harvard. These are 3-year positions with independent funding and access to the amazing resources of the institute. Apply by October 9th 2023!
www.harvard.edu/kempner-institute/opportunities/the-kempn… -
Sham Kakade
@ShamKakade6
I ❤️ efficiency. A wonderful example.
Crémieux
@cremieuxrecueil
The adoption of cellphones by Keralan fishermen is, I believe, the most stunning example of the contribution of information technology to market performance.
Take a look at this graph for background: in three different regions of Kerala, phones were adopted at different times.
Kozhikode got cellphones before Kannur, who got them before Kasaragod. Adoption by fishermen was fast when phones were finally introduced:
Now look at what happened to the price of fish after phones entered the scene:
The dispersion in prices virtually disappeared! The author of the study wrote about this that:
"Before any region had mobile phones, the degree of price dispersion across markets within a region on any given day is high, and there are many cases where the price is zero (i.e., waste). However, within a few weeks of mobile phones being introduced in Region I, there is a sharp and striking reduction in price dispersion. Prices across markets in the region rarely differ by more than a few rupees per kilogram on any day, compared to cases of as much as 10 Rs/kg prior to the introduction of mobile phones. In addition, the prices in the various markets rise and fall together and the week-to-week variability within each market is much smaller, since catchment zone-specific quantity shocks are now spread across markets via arbitrage. Further, there are no cases of waste in this region after phones are introduced.
"By contrast, price behavior in regions II and III appears largely unchanged after phones are introduced in region I. However, after mobile phones are introduced in region II, prices again become much less dispersed across markets on any given day, less variable within markets over time, and waste is ultimately eliminated, whereas region III again remains unchanged. Finally, the same pattern holds once region III adds phones."
Efficient information transmission enables efficient markets. It's amazing what technology can do.
Read the study here: www.jstor.org/stable/25098864 -
Sham Kakade reposted
Kempner Institute at Harvard University
@KempnerInst
The application for our #kempnerinstitute research fellowship is now open! Learn more, spread the word, and apply by Oct. 9: www.harvard.edu/kempner-institute/opportunities/the-kempn… #airesearch #neuroscience #researchfellowship @Harvard
-
Sham Kakade reposted
Kanaka Rajan
@KanakaRajanPhD
Thrilled to share that I'll be joining @harvardmed’s Dept. of Neurobiology & the @KempnerInst as a founding faculty member in September 2023: www.harvard.edu/kempner-institute/2023/09/05/computationa…
-
Sham Kakade reposted
Boaz Barak
@boazbaraktcs
I've said it before, @arxiv has done much more to advance science, and expand participation in it, than all the anonymity interventions ever will.
Any policy that obstructs arXiv is not just silly, but also counterproductive to both science progress and inclusiveness.
Naomi Saphra
@nsaphra
Just got a desk reject, post-rebuttals, for a paper being submitted to arxiv <30 min late for the anonymity deadline. I talk about how the ACL embargo policy hurts junior researchers and makes ACL venues less desirable for NLP work. I don’t talk about the pointless NOISE it adds.
-
Sham Kakade reposted
Naomi Saphra
@nsaphra
Just got a desk reject, post-rebuttals, for a paper being submitted to arxiv <30 min late for the anonymity deadline. I talk about how the ACL embargo policy hurts junior researchers and makes ACL venues less desirable for NLP work. I don’t talk about the pointless NOISE it adds.
-
Sham Kakade reposted
Boaz Barak
@boazbaraktcs
Applications are now open for postdoctoral research fellows at the Kempner Institute at Harvard. These are 3-year positions with independent funding and access to the amazing resources of the institute. Apply by October 9th 2023!
www.harvard.edu/kempner-institute/opportunities/the-kempn… -
Sham Kakade reposted
Noah Golowich
@GolowichNoah
I'll be at #icml this week, presenting "Hardness of Independent Learning and Sparse Equilibrium Computation in Markov Games" (arxiv.org/abs/2303.12287; joint work with @ShamKakade6 & @canondetortugas). Come check out our poster on Tuesday morning! (1/3)
-
Sham Kakade reposted
Jens Tuyls
@JensTuyls
Imitation learning is one of the most widely used methods in ML, but how does compute affect its performance?
We explore this question in the challenging game of NetHack and find our scaled-up agent to outperform prior SOTA by 2x!
arxiv.org/abs/2307.09423
[1/6] -
Sham Kakade reposted
Greg Yang
@TheGregYang
Finally launched x.ai!
The mathematics of deep learning is profound, beautiful, and unreasonably effective. Developing the "theory of everything" for large neural networks will be central to taking AI to the next level. Conversely, this AI will enable everyone to understand our mathematical universe in ways unimaginable before.
Math for AI and AI for math!
Any mathematician/theorist excited about this needs to DM me! -
Sham Kakade
@ShamKakade6
Dean Foster, @dhruvmadeka, and I have been excited about the application of AI to education! We collected our thoughts here - and we're curious what people think:
dhruvmadeka.com/post/llms_education/ -
Sham Kakade
@ShamKakade6
One MacAskill might have a discount factor of 1 but the other has a discount factor of 0!
youtu.be/GL0rbxB9Lqg?t=263 -
Sham Kakade reposted
Ai2
@allen_ai
AI2 is thrilled to announce that Ali Farhadi is our institute's new CEO! Learn more on the AI2 Blog:
blog.allenai.org/allen-institute-for-artificial-intellige… -
Sham Kakade reposted
Richard Ngo
@RichardMCNgo
The near-term AI application I'm most excited about is tutoring:
- 1:1 tutoring improves kids' learning massively
- even existing AIs are pretty good at this, and they'll get way better quickly
- forcing kids to sit and listen to a teacher for hours every day is a travesty -
Sham Kakade
@ShamKakade6
Congratulations!! Excited to see what comes next!
Jonathan Frankle
@jefrankle
I'm absolutely thrilled that @MosaicML has agreed to join @databricks as we continue on our journey to make the latest advances deep learning efficient and accessible for everyone. The best of MosaicML is yet to come 🎉🎉🎉
-
Sham Kakade reposted
Boaz Barak
@boazbaraktcs
Happy to join the Kempner as an associate faculty and honored to be in the company of @SusanMurphylab1 @gershbrain and @marinkazitnik
Harvard University
@Harvard
The Kempner Institute’s newest associate faculty members are Boaz Barak, Susan Murphy, Samuel Gershman, and Marinka Zitnik news.harvard.edu/gazette/story/newsplus/kempner-institute…
-
Sham Kakade reposted
Harvard University
@Harvard
The Kempner Institute’s newest associate faculty members are Boaz Barak, Susan Murphy, Samuel Gershman, and Marinka Zitnik news.harvard.edu/gazette/story/newsplus/kempner-institute…
-
Sham Kakade reposted
Boaz Barak
@boazbaraktcs
1/4 New paper on the (lack of) implicit bias for stochastic gradient descent in online learning. arxiv.org/abs/2306.08590 with @vyasnikhil96 @depen_morwani @rosieyzh @GalKaplun and @ShamKakade6
-
Sham Kakade reposted
Sebastien Bubeck
@SebastienBubeck
Terence Tao reflecting on GPT-4 in the AI Anthology coordinated by @erichorvitz :
"I expect, say, 2026-level AI, when used properly, will be a trustworthy co-author in mathematical research, and in many other fields as well."
Terry gets it.
unlocked.microsoft.com/ai-anthology/terence-tao/ -
Sham Kakade reposted
Seth Neel
@SethInternet
On the mitigation side @boazbaraktcs @ShamKakade6 have this interesting approach: arxiv.org/abs/2302.10870 essentially if you split your data and train 2 LLMs on disjoint data, if you force the models to only generate outputs that are likely under both models you get guarantees
-
Sham Kakade reposted
Prateek Jain
@jainprateek_
Exciting direction, which completes one part of the puzzle. Matryoshka representations (arxiv.org/abs/2205.13147) seemed powerful, but this work shows how it's power can be exploited for faster and adaptive semantic search.
Led by ever awesome @wregss @adityakusupati
Aditya Kusupati
@adityakusupati
Introducing💃AdANNS: A Framework for Adaptive Semantic Search🕺
TL;DR: Up to 90× faster nearest neighbor retrieval and 2× lower memory cost for web-scale search.
Applies to vector search at scale & improves all "retrieval" augmented models!
arxiv.org/abs/2305.19435
[1/8] -
Sham Kakade reposted
Aditya Kusupati
@adityakusupati
Introducing💃AdANNS: A Framework for Adaptive Semantic Search🕺
TL;DR: Up to 90× faster nearest neighbor retrieval and 2× lower memory cost for web-scale search.
Applies to vector search at scale & improves all "retrieval" augmented models!
arxiv.org/abs/2305.19435
[1/8] -
Sham Kakade reposted
Marc G. Bellemare
@marcgbellemare
Now that we’re all recovered from the NeurIPS deadline – look what I received in the mail last week…!
Publication day is just around the corner - May 30th.
A small 🧵: -
Sham Kakade reposted
CogSci Society
@cogsci_soc
Huge congratulations to Talia Konkle @talia_konkle, recipient of the inaugural Lila R. Gleitman Prize 🏆
She’s a Professor of Psychology at @Harvard with a Ph.D. in Brain and Cognitive Sciences from the @MIT
Learn about all her scientific achievements at cognitivesciencesociety.org/gleitman-prize/ -
Sham Kakade reposted
Ilenna Jones, PhD @ ilennaj.bsky.social
@IlennaJ
This fall 2023 I will join the first cohort of Research Fellows at the Harvard Kempner Institute for the Study of Natural and Artificial Intelligence!
This is an amazing next step after PhD and I look forward to what we can accomplish together!
Boaz Barak
@boazbaraktcs
Kempner Institute announces the first cohort of research fellows starting this fall! Looking forward to learning from and collaborating with @brandfonbrener, @cogscikid , @_jennhu, @IlennaJ , @WangBinxu , @nsaphra , Eran Malach, and @t_andy_keller.
www.harvard.edu/kempner-institute/2023/05/11/announcing-2… -
Sham Kakade reposted
Naomi Saphra
@nsaphra
you know this is going to be good if they convinced me to move to boston for it
Boaz Barak
@boazbaraktcs
Kempner Institute announces the first cohort of research fellows starting this fall! Looking forward to learning from and collaborating with @brandfonbrener, @cogscikid , @_jennhu, @IlennaJ , @WangBinxu , @nsaphra , Eran Malach, and @t_andy_keller.
www.harvard.edu/kempner-institute/2023/05/11/announcing-2… -
Sham Kakade reposted
Wilka Carvalho
@cogscikid
Honored to be a part of this very thoughtful cohort of Kempner Fellows at @Harvard this coming Fall
www.harvard.edu/kempner-institute/2023/05/11/announcing-2…
Very excited to learn from and collaborate with them to better understand intelligence in brains and machines! -
Sham Kakade reposted
Jennifer Hu
@_jennhu
Life update: I defended my thesis and will be joining the Harvard Kempner Institute as a Research Fellow 🙂 Thrilled to continue pursuing questions at the intersection of language, cognition, and AI (and to be sticking around Boston)!
Boaz Barak
@boazbaraktcs
Kempner Institute announces the first cohort of research fellows starting this fall! Looking forward to learning from and collaborating with @brandfonbrener, @cogscikid , @_jennhu, @IlennaJ , @WangBinxu , @nsaphra , Eran Malach, and @t_andy_keller.
www.harvard.edu/kempner-institute/2023/05/11/announcing-2… -
Sham Kakade reposted
Andy Keller
@t_andy_keller
Incredibly honored and excited to be a part of this first group of fellows at the Kempner Institute. Very much looking forward to what the next few years may bring. See you in Boston!
Boaz Barak
@boazbaraktcs
Kempner Institute announces the first cohort of research fellows starting this fall! Looking forward to learning from and collaborating with @brandfonbrener, @cogscikid , @_jennhu, @IlennaJ , @WangBinxu , @nsaphra , Eran Malach, and @t_andy_keller.
www.harvard.edu/kempner-institute/2023/05/11/announcing-2… -
Sham Kakade reposted
Binxu Wang 🐱
@WangBinxu
Thrilled to share that I'm going to join Kempner Institute as the first cohort of research fellows this fall! It's an incredible moment to start my next chapter in #neuroai research! Looking forward to collaborating & learning w/ these amazing colleagues!
kempnerinstitute.harvard.edu
Boaz Barak
@boazbaraktcs
Kempner Institute announces the first cohort of research fellows starting this fall! Looking forward to learning from and collaborating with @brandfonbrener, @cogscikid , @_jennhu, @IlennaJ , @WangBinxu , @nsaphra , Eran Malach, and @t_andy_keller.
www.harvard.edu/kempner-institute/2023/05/11/announcing-2… -
Sham Kakade
@ShamKakade6
So excited to work with this amazing new cohort!
Boaz Barak
@boazbaraktcs
Kempner Institute announces the first cohort of research fellows starting this fall! Looking forward to learning from and collaborating with @brandfonbrener, @cogscikid , @_jennhu, @IlennaJ , @WangBinxu , @nsaphra , Eran Malach, and @t_andy_keller.
www.harvard.edu/kempner-institute/2023/05/11/announcing-2… -
Sham Kakade reposted
Harvard SEAS
@hseas
Harvard researchers part of @NSF AI Institute for Societal Decision Making. $20M collaboration brings together AI researchers, social scientists to develop tools for societal challenges such as disaster management and public health. seas.harvard.edu/news/2023/05/harvard-researchers-part-ns…
-
Sham Kakade reposted
Jason Lee
@jasondeanlee
I want to remind everyone that disabilities may also be invisible. Your colleagues, group members, students, postdocs, may be going through this. I am not an eloquent person, so WE NEED TO PAY MORE ATTENTION TO THE DISABLED AND THEIR ACCOMMODATION
Maria Skoularidou (she/her)
@skoularidou
Being a #disabled junior researcher in #ai comes at a massive price; when your disabilities flare, you are on your own: there is neither medical insurance nor salary for you during this difficult time
This is a very important aspect that needs our attention
#academia #insecurity -
Sham Kakade
@ShamKakade6
Excited to be a part of this, with Aarti Singh who is spearheading the CMU effort!
U.S. National Science Foundation
@NSF
📢 Announcing seven new National Artificial Intelligence Research Institutes!
Discover the themes and the institutions that are helping advance foundational AI research to address national economic and societal priorities in the 🧵 ⬇️: www.nsf.gov/news/nsf-announces-7-new-national-artificial-… -
Sham Kakade reposted
Boaz Barak
@boazbaraktcs
Btw I don’t think the original writeup is public but the article links to a short video youtu.be/nQD6lDwFmCc explaining the (very pretty!) proof based on Johnson and Jackson’s slides
-
Sham Kakade reposted
The Harvard Crimson
@thecrimson
Harvard President Lawrence S. Bacow urged fellow university presidents to demand immigration reform during a Friday speech at the American Council on Education’s annual gathering.
@claireyuan33 and @Mherszenhorn report.
www.thecrimson.com/article/2023/4/17/bacow-ace-conference/ -
Sham Kakade reposted
Boaz Barak
@boazbaraktcs
Horace He @cHHillee kindly agreed to give a guest lecture today. Part of the pre-reading is his great blog horace.io/brrr_intro.html explaining what you need to do to get an N parameter (more accurately N FLOP) model to do inference in N/(312*10¹²) seconds.
-
Sham Kakade reposted
Boaz Barak
@boazbaraktcs
Kempner institute announces first cohort of associate faculty! Looking forward to (continuing to) interact with Demba, Talia, Haim and Cengiz!
www.harvard.edu/kempner-institute/2023/04/05/kempner-inst… -
Sham Kakade
@ShamKakade6
Only 2D? Amateur. Still has nothing on @adamfungi.
Adam Tauman Kalai
@adamfungi
Results of GPT-4's attempts at "What do you get when you cross X with Y" jokes.
-
Sham Kakade reposted
Chelsea Finn
@chelseabfinn
Recently gave a talk at Harvard @hseas on how neural nets make stuff up & what we should do about it:
youtu.be/bCyWCz4NbN4
I covered our work on
- DetectGPT
- OOD detection & selective classification
- robustness via DivDis
Thanks @boazbaraktcs @ShamKakade6 & co for hosting! -
Sham Kakade reposted
Aditya Kusupati
@adityakusupati
I did a thing (MRL & AdANNS -arxiv.org/abs/2205.13147, TBD); & @GoogleAI thought it was useful & interesting to be in the blog articles on Google Research, 2022 & beyond
Read the last para before conclusion.
@jainprateek_, @RAIVNLab, @ShamKakade6, @wregss, @uwcse, @BhattGantavya
Google AI
@GoogleAI
As #deeplearning models become more widely used, it is increasingly important that they be both robust and efficient. Today we summarize some of our many efforts to improve #ml efficiency through algorithms research. → goo.gle/3I6asCj
-
Sham Kakade reposted
Sheikh Abdur Raheem Ali
@Sheikheddy
Fav paper this week was probably arxiv.org/abs/2302.14753 by @_cyrilzhang, @ShamKakade6, Gaurav Mahajan, and Akshay Krishnamurthy.
-
Sham Kakade reposted
Chan Zuckerberg Initiative
@ChanZuckerberg
✨ Big News ✨We’re launching a new research hub in Chicago.
#czbiohubchi will bring together scientists + engineers at @UChicago, @NorthwesternU & @UofIllinois to build new tech + gain insights into how inflammation impacts our cells -
Sham Kakade reposted
Adam Tauman Kalai
@adamfungi
Breakthrough LLM copyright paper by @vyasnikhil96,
@ShamKakade6, @boazbaraktcs gives a way to train LLMs that don't infringe on copyright! Their brilliant trick: train 2 models on disjoint datasets; ignore generations that are too likely from 1 of them arxiv.org/abs/2302.10870 -
Sham Kakade reposted
Sebastian Raschka
@rasbt
Whether it's okay to train generative models on copyrighted material is still up for debate. However, beyond the fair-use discussion, there's this related issue whether the *generated outputs* violate copyrights. This is an issue because generative models can memorize & output copyrighted data.
Previous work showed that diffusion models memorize photographs ("Extracting Training Data from Diffusion Models", arxiv.org/abs/2301.13188) and LLMs memorize copyrighted book passages ("Quantifying Memorization Across Neural Language Models", arxiv.org/abs/2202.07646).
This morning, researchers shared an interesting information-theoretic approach In an attempt to address this issue: "Provable Copyright Protection for Generative Models" (arxiv.org/abs/2302.10870)
When I understand it correctly, the idea is to
1) train three models:
2a) m1 and m2 are trained on two non-overlapping partitions of the dataset containing copyrighted images; these models often generate copyrighted works (or images that are similar to copyrighted works)
2b) the third model m3 is trained on the whole dataset, also outputting copyrighted work
3) All three models generate data that have a similar distribution as the training data to some extend. Using the so-called CP-k algorithm, we combine these models to output data from the intersection of these distributions such that memorized images are unlikely to be generated.
Link to the paper for more details: arxiv.org/abs/2302.10870 -
Sham Kakade
@ShamKakade6
Very excited about this new work with
@vyasnikhil96 and @boazbaraktcs! Provable copyright protection for generative models? See:
Boaz Barak
@boazbaraktcs
1/5 In new paper with @vyasnikhil96 and @ShamKakade6 we give a way to certify that a generative model does not infringe on the copyright of data that was in its training set. See windowsontheory.org/2023/02/21/provable-copyright-protect… for blog, but TL;DR is...
-
Sham Kakade reposted
Sloan Foundation
@SloanFoundation
We are delighted to announce the winners of this year’s Sloan Research Fellowship! These outstanding researchers are shining examples of innovation and impact—and we are thrilled to support them. Meet the winners here: sloan.org/fellowships/2023-Fellows 🎉 #sloanfellow #stem #sciencetwitter
-
Sham Kakade reposted
Jonathan Frankle
@jefrankle
Just bumping this one more time: @MosaicML is hiring research interns this summer. If you want to study the science behind training large models and bring those capabilities within reach for everyone, apply now! boards.greenhouse.io/mosaicml/jobs/4170454004
Jonathan Frankle
@jefrankle
@MosaicML is hiring research interns this summer! Join an amazing team doing exciting work on efficient training of large models, including LLMs and more to come. To apply, visit www.mosaicml.com/careers. We'll interview in earnest in Jan! Major plus if you'll graduate soon.
-
Sham Kakade reposted
Boaz Barak
@boazbaraktcs
The Kempner Institute at @harvard is looking for a director of engineering, helping promote the research and education mission, as well as leading future engineering fellows. See sjobs.brassring.com/TGnewUI/Search/home/HomeWithPreLoad?p…
-
Sham Kakade reposted
Harvard University
@Harvard
Claudine Gay has been elected the 30th president of Harvard University hrvd.me/Gay2022t
-
Sham Kakade reposted
Boaz Barak
@boazbaraktcs
New position: 📢 Kempner Institute Fellows, loosely modeled around Harvard "Society of Fellows", a 3-year fellowship with attractive terms for people in AI / ML / Neuroscience / related fields.
academicpositions.harvard.edu/postings/11860
Boaz Barak
@boazbaraktcs
Reminder - our "opportunities" webpage contains links for several postdoc positions in machine learning at Harvard and MIT, not just in our group but also with collaborators / related groups.
mlfoundations.org -
Sham Kakade reposted
Bernardo Sabatini
@blsabatini
A great opportunity for those interesting in neurobiology and AI. This is the first joint search carried out by Harvard Department of Neurobiology and the new Kempner Institute for the study of natural and artificial intelligence. This is for a "dry lab"
academicpositions.harvard.edu/postings/11660 -
Sham Kakade reposted
Harvard Magazine
@HarvardMagazine
Harvard celebrated the launch of the Kempner Institute for the Study of Natural and Artificial Intelligence yesterday; Priscilla Chan ’07 and Mark Zuckerberg ’06, L.L.D. ’17 @ChanZuckerberg shared their vision for the new enterprise.
www.harvardmagazine.com/2022/09/kempner-institute-kickoff -
Sham Kakade reposted
Harvard Statistics Department
@HarvardStats
Learn about a brand new course in Reinforcement Learning for Harvard undergrads by reading this interview with Prof. Janson and Prof. Kakade: statistics.fas.harvard.edu/news/professors-janson-and-kak… @ShamKakade6
-
Sham Kakade reposted
Aditya Kusupati
@adityakusupati
Introducting🪆Matryoshka Representations for Adaptive Deployment🪆
TL;DR: up to 14× lower real-world classification & retrival costs at web-scale at no loss in accuracy & w/o any overhead across setups.
Paper: arxiv.org/abs/2205.13147
Code: github.com/RAIVNLab/MRL
[1/11] -
Sham Kakade reposted
Michael Littman
@mlittmancs
Starting a new "rotating" position shortly as Division Director of Information and Intelligent Systems at NSF (AI, HCI, ML, etc.). I'll be learning about what the research community needs, what NSF provides, and how to bring the two closer together!
content.govdelivery.com/accounts/USNSF/bulletins/318af01 -
Sham Kakade reposted
Jordan Ash
@jordan_t_ash
Life update: I was lucky to have a lot of nice options, but in the end I decided to take a senior researcher job @MSFTResearch NYC!
1/3 -
Sham Kakade reposted
Sloan Foundation
@SloanFoundation
Introducing… the winners of this year’s Sloan Research Fellowship! These extraordinary researchers represent some of the most exciting young minds working today—and we are thrilled to support them. Meet the winners here: sloan.org/fellowships/2022-Fellows #sloanfellow
-
Sham Kakade reposted
Jens Tuyls
@JensTuyls
How can RL agents deal with both sparse rewards and large, dynamic action spaces – a key challenge in text games?
Our method eXploit-Then-eXplore (XTX) tackles these challenges and achieves a more than 2x improvement on Zork!
arxiv.org/abs/2201.01251 #iclr2022 Spotlight
📜[1/5] -
Sham Kakade reposted
Jens Tuyls
@JensTuyls
See all of this and more in:
Multi-Stage Episodic Control for Strategic Exploration in Text Games
By @JensTuyls, @ShunyuYao12, @ShamKakade6, @karthik_r_n
Paper: arxiv.org/abs/2201.01251
Project page: sites.google.com/princeton.edu/xtx
Code: github.com/princeton-nlp/XTX -
Sham Kakade reposted
Boaz Barak
@boazbaraktcs
Happy and honored to receive the FOCS 2021 test of time award for my FOCS 2001 paper, and especially honored to be in the company of some amazing papers from 10, 20, and 30 years ago.
focs2021.cs.colorado.edu/test-of-time-awards/ -
Sham Kakade reposted
ACM India
@Indiaacm
Prateek Jain Chosen as Recipient of 2021 ACM India Early Career Researcher (ECR) Award
More information at: www.linkedin.com/feed/update/urn:li:activity:687389219106…
Please join us in congratulating Prateek Jain.
#acm #acmindia #ecr #researcher #ml #research -
Sham Kakade reposted
Behnam Neyshabur
@bneyshabur
Totally agree!
Anyone screening applications and any applicant thinking their CV is not representative of their skills/potentials, I think you might want to read the story of my own PhD application in this thread:
1/
Peyman Milanfar
@docmilanfar
Any document claiming an easy way to gauge grad school applicants needs to be challenged
To wit: While 2 or 3 Unis in Iran are far more selective than others, the # of outstanding candidates far exceeds their enrollment. The given ranking is thus opinion not fact & is misleading -
Sham Kakade reposted
Bernardo Sabatini
@blsabatini
It's an honor to be working with @cziscience, @ShamKakade6 and many others to build the Kempner Institute at @Harvard. It will push and blur the boundaries of and between computational neuroscience/AI/ML to learn how the brain computes, improve AI/ML and better lives of people.
-
Sham Kakade
@ShamKakade6
Grateful to Priscilla Chan/Mark Zuckerberg (@ChanZuckerberg Initiative) for generous gift. Kempner Natural & Artificial Intelligence Institute @Harvard. Excited to work w/@blsabatini+new colleagues to provide new educational and research opportunities. news.harvard.edu/gazette/story/2021/12/new-harvard-instit…
-
Sham Kakade reposted
Boaz Barak
@boazbaraktcs
1/14 More than 150 scientists & educators signed open letter raising alarm on efforts to water down K-12 math education
scottaaronson.blog?p=6146
Signers include Fields, Nobel & Turing laurates, and also founders of HS STEM educational initiatives (eg @adrian_mims, @minilek). -
Sham Kakade
@ShamKakade6
Congrats @timnitGebru for your efforts to provide a broader set of ideas to the community! Excited to see the work that comes out.
Distributed AI Research Institute is on Mastodon
@DAIRInstitute
We are @DAIRInstitute — an independent, community-rooted #ai research institute free from #bigtech's pervasive influence. Founded by @timnitGebru.
www.dair-institute.org -
Sham Kakade reposted
Derek Thompson
@DKThomp
A new paper studied the careers of 20,000 artists, scientists, and film directors to determine when they did their best work—their career "hot streak."
www.theatlantic.com/ideas/archive/2021/11/hot-streaks-in-…
The researchers' theory for how hot streaks happen boils down to two words: Explore & Exploit -
Sham Kakade reposted
Andrew Beam
@AndrewLBeam
New paper w/ @MarzyehGhassemi & @DrLukeOR where we argue against explainability for patient-level decision-making.
tl;dr: Don’t ask for explainability at the bedside, it isn’t made for that.
Paper: www.sciencedirect.com/science/article/pii/S2589750021002089 -
Sham Kakade reposted
Boaz Barak
@boazbaraktcs
1/20 A 🧵 on public key cryptography, and its interaction with quantum computing. Spurred by a discussion w/ @jfitzsimons, @mattyhoban, @dabacon, @rdviii but more general.
There is a fundamental gulf between public and private key encryption. -
Sham Kakade reposted
Shipra Agrawal
@ashipra
Presenting an online tutorial on bandits and reinforcement learning in OR, shortly (at 10:45 am EST) at INFORMS 20201. Log in if interested!
-
Sham Kakade reposted
Jordan Ash
@jordan_t_ash
New work! We take the maximum over an ensemble of regressors trained to predict Gaussian noise from policy-network features to approximate the LinUCB exploration bonus. Our algorithm can actually keep up with other deep RL intrinsic rewards!
arxiv.org/abs/2110.11202 -
Sham Kakade
@ShamKakade6
Solid move from DeepMind. Knowing Emo Todorov and his work, frankly surprised this one-man-show is only being purchased now. Hope Emo still stays in the driver's seat for MuJoCo going forward!
Google DeepMind
@GoogleDeepMind
We’ve acquired the MuJoCo physics simulator (mujoco.org) and are making it free for all, to support research everywhere. MuJoCo is a fast, powerful, easy-to-use, and soon to be open-source simulation tool, designed for robotics research: dpmd.ai/mujoco-blog
Sham Kakade
@ShamKakade6
Pinged Emo, and he'll have freedom to drive/add new things. So this looks like a win-win situation for the community...
-
Sham Kakade
@ShamKakade6
Thank you @SusanMurphylab1! I am thrilled that I can finally be your colleague ❤️
Susan Murphy lab
@SusanMurphylab1
I’m thrilled that @ShamKakade6 is joining the Harvard SEAS CS faculty!!! Welcome, Sham!
www.seas.harvard.edu/news/2021/10/seven-join-harvard-comp… -
Sham Kakade
@ShamKakade6
Super excited to join Harvard with a stellar group of new hires and looking forward to many new collabs with the terrific faculty there! Def sad to be leaving my wonderful UW and MSR colleagues and friends; rest assured, I'll keep up the collabs!
www.seas.harvard.edu/news/2021/10/seven-join-harvard-comp… -
Sham Kakade
@ShamKakade6
A huge congrats to MPI for hiring the terrific @mrtz as a director! Personally sad to have him across the pond, but excited to see what Moritz helps to build.
-
Sham Kakade reposted
Russ Salakhutdinov
@rsalakhu
Is a good representation sufficient for sample-efficient offline RL? In practice, offline RL can be very sensitive to distribution shift even when using features from pretrained neural networks on the same task
arxiv.org/abs/2103.04947
w/t @RuosongW, Wu & @ShamKakade6 #icml2021 -
Sham Kakade reposted
Rad Niazadeh
@rad_niazadeh
Come and join this fantastic annual event at Northwestern CS, specially if you are keen to watch super-polished talks by a line up of brilliant juniors in TCS:
theory.cs.northwestern.edu/events/2020-junior-theorists-w…
* Check out my junior co-author, Yiding Feng, who gives a talk on our recent paper on Friday! -
Sham Kakade reposted
Elad Hazan
@HazanPrinceton
Happy to announce our open-source differentiable control library w. many of our new algorithms for non-stochastic control!
github.com/google/deluca
Talk by my student Paula Gradu Friday 12:23 - 12:35 EST, poster: 16:30 - 17:30 EST -
Sham Kakade reposted
Bernhard Schölkopf
@bschoelkopf
Today at NeurIPS, we are holding a memorial session for Olivier Chapelle, who passed away earlier this year. Olivier was one of the leading machine learning researchers of his generation, a kind human being and a friend. nips.cc/virtual/2020/public/e_memorials.html
-
Sham Kakade reposted
Reinforcement Learning and Artificial Intelligence
@rlai_lab
#neurips 2020 has begun!
We start off with a tutorial on Policy Optimization
in RL (including interactive notebooks!) by RLAI faculty @white_martha, along with @ShamKakade6 and @le_roux_nicolas, from 12pm MST (Monday).
sites.google.com/ualberta.ca/rlandoptimization-neurips202… -
Sham Kakade
@ShamKakade6
Very cool! Getting RL to work with real sample size constraints is critical. Interesting to see how it was done here. Also, looks like the application with Loon is for social good! 👏
Marc G. Bellemare
@marcgbellemare
Our most recent work is out in Nature! We're reporting on (reinforcement) learning to navigate Loon stratospheric balloons and minimizing the sim2real gap. Results from a 39-day Pacific Ocean experiment show RL keeps its strong lead in real conditions. www.nature.com/articles/s41586-020-2939-8
-
Sham Kakade
@ShamKakade6
hmmm... well, it is true that a function that is a sum of degree 3 polynomials would still be a function that is a sum of degree 3 polynomials after any linear transformation of the input (but the # of terms in the sum might be much larger). this def would be neat to try!
Christian Szegedy
@ChrSzegedy
Yes, early versions and relative position encoding use fixed spatial features. Some encoding of position (learned or engineered is clearly necessary otherwise, it is just a BOW)
Rotating the sentence would be fun to try. IMO, unlikely to affect the performance significantly. -
Sham Kakade
@ShamKakade6
Congrats! A beautiful book indeed!
American Mathematical Society
@amermathsoc
Noga Alon @princeton and Joel Spencer @nyuniversity receive the 2021 Steele Prize for Mathematical Exposition for The Probabilistic Method @WileyGlobal . Now in its 4th ed, the text is invaluable for both the beginner and the experienced researcher. More...www.ams.org/news?news_id=6484
-
Sham Kakade
@ShamKakade6
Amazing and congrats! I have def been wondering if the inductive biases in DeepNets and in ML methods are well suited for certain scientific domains. This settles that for structure prediction! Hoping this can eventually help with drug discovery.
Google DeepMind
@GoogleDeepMind
In a major scientific breakthrough, the latest version of #alphafold has been recognised as a solution to one of biology's grand challenges - the “protein folding problem”. It was validated today at #casp14, the biennial Critical Assessment of protein Structure Prediction (1/3)
-
Sham Kakade reposted
Csaba Szepesvari
@CsabaSzepesvari
Our chance to stay positive during these dire times is to attend Simon's seminar tomorrow where I hope we learn that despite all other signs RL is not much harder than bandits. Long live RL, long live bandits!
RL Theory Virtual Seminars
@RLtheory
Our next talk:
11/24: Simon S. Du (University of Washington)
"Is Reinforcement Learning More Difficult Than Bandits? A Near-optimal Algorithm Escaping the Curse of Horizon"
For details, please see the website:
sites.google.com/view/rltheoryseminars/next-seminar -
Sham Kakade
@ShamKakade6
Congrats and well deserved!! 👏👏 It’s inspiring to have @madsjw as a leader in our community.
-
Sham Kakade reposted
Claire Vernade @claireve.bsky.app
@vernadec
Excited to announce our new mentoring program for women and non-binary folks in learning theory. Please consider signing up and forwarding to your students/friends/colleagues !
-
Sham Kakade reposted
Kamalika Chaudhuri
@kamalikac
And also UCSD PhD student @mary_ann_e and postdoc @ML_Theorist
UCSD Engineering
@UCSDJacobs
Excited to see that @ucsd_cse Prof. Kamalika Chaudhuri @kamalikac and Ph.D. student Jessica Sorrell @OptimistsInc who cofounded an online community for Women in #ml theory: www.wiml-t.org/home
-
Sham Kakade reposted
RadicalxChange
@RadxChange
We are thrilled to share with you the start of the new RxC FELLOWSHIP PROGRAM. Apply before December 13 if you are working on a project meant to strengthen civic society, rethink public goods, or create more equitable societies. www.radicalxchange.org/fellowship/ #rxcfellowship
-
Sham Kakade
@ShamKakade6
🇺🇸
-
Sham Kakade
@ShamKakade6
This should be good! @SurbhiGoel_ has done some exciting work in understanding neural nets, going beyond the "linear" NTK barrier. To make progress in deep learning theory, we def need to understand these beasts in the non-linear regime.
Boaz Barak
@boazbaraktcs
Looking forward to this Friday at 1pm when we'll hear from @SurbhiGoel_ about the computational complexity of learning neural networks over gaussian marginals.
We'll see some average-case hardness results as well as a poly-time algorithm for approximately learning ReLUs
Boaz Barak
@boazbaraktcs
1/3:
Excited for this year's theory of ML seminars mltheory.org! Surprisingly, managed to retain much of the interactive blackboard atmosphere over Zoom (2 hour slot helps: people don't give their "canned talks" and there is more room for discussions)
Lineup in thread: -
Sham Kakade reposted
Boaz Barak
@boazbaraktcs
1/3:
Excited for this year's theory of ML seminars mltheory.org! Surprisingly, managed to retain much of the interactive blackboard atmosphere over Zoom (2 hour slot helps: people don't give their "canned talks" and there is more room for discussions)
Lineup in thread: -
Sham Kakade reposted
Gautam Kamath
@thegautamkamath
Congratulations to Computer Science winners Yin Tat Lee and Aviad Rubinstein (@AviadRubinstein)!
Packard Foundation
@PackardFdn
We are excited to announce the 2020 #packardfellows in Science and Engineering! Over the next 5 years, these 20 @PackardFdn Fellows will engage in groundbreaking scientific research and reach new frontiers of discovery. Meet the Fellows: www.packard.org/insights/news/packard-fellowships-in-scie….
This media couldn't be displayed. See it on the original post. -
Sham Kakade reposted
Stanford University
@Stanford
The #nobelprize committee couldn't reach Paul Milgrom to share the news that he won, so his fellow winner and neighbor Robert Wilson knocked on his door in the middle of the night.
-
Sham Kakade
@ShamKakade6
Sweet. Def want to build up my intuition on Hermite polynomials! For Gaussians, I tend to think more in terms of Isserlis' theorem (aka Wick's theorem), often a more brute approach for dealing with higher moments.
Francis Bach
@BachFrancis
If you like Gaussian kernels and distributions, you will enjoy this month blog post on Hermite polynomials!
francisbach.com/hermite-polynomials/ -
Sham Kakade reposted
Brown University School of Public Health
@Brown_SPH
A pandemic, untamed: Time-lapse of how the coronavirus made its way across U.S. communities since March. Currently, there are twice as many daily COVID-19 cases as after Memorial Day. Cases are rising in 25+ states with the heartland most affected. With @HarvardEthics @MSFTIssues
-
Sham Kakade reposted
Yann LeCun
@ylecun
Code can now be attached to ArXiv papers through a partnership between ArXiv and FB's @paperswithcode
Papers with Code
@paperswithcode
🎉 Papers with Code partners with arXiv! Code links are now shown on arXiv articles, and authors can submit code through arXiv. Read more: medium.com/paperswithcode/papers-with-code-partners-with-…
-
Sham Kakade reposted
Karina Voggel ✨🔭🏃🏼♀️
@KarinaVoggel
This video is essentially why the Black Hole at the center of the Milky Way got the #nobelprize today: They tracked *individual* stars orbit around the black hole over time. Animation from here: astro.ucla.edu/~ghezgroup/gc/animations.html
-
Sham Kakade reposted
Mary L. Gray
@marylgray
Still can’t believe that I’m in this mix. Am proud, humbled, and honored beyond words (and a little freaked out)
-
Sham Kakade
@ShamKakade6
A nice note. Some cool tricks in these Bhatia matrix analyses books. If I understand correctly, Russo-Dye Thm lets u (exactly) compute the largest learning rate with a maximization problem using only vectors rather than matrices (still hitting it on the 4th-moment data tensor).
Yaroslav Bulatov
@yaroslavvb
What's the largest learning rate for which SGD converges? In deterministic case with Hessian H it is 2/||H||, from basic linear algebra. For SGD, an equivalent rate is 2/Tr(H), derivation from Russo-Dye theorem: yaroslavvb.com/convergence.pdf
Sham Kakade
@ShamKakade6
This is for SGD for least squares. Super basic problem. If I understand correctly, the _exact_ problem dependent largest learning rate (above which divergence occurs) is at 1/lambda_max( E[xx']^-1 E[||x||^2 x x']). This is pretty clean (and no SDP needed to compute it).
Sham Kakade
@ShamKakade6
Errata: my bad. 1. for the largest rate for non-divergence, still need an SDP (it's a spectral radius of a certain operator, defined in earlier works too). 2. the above is for monotone decrease in L2 distance to opt. 3. monotone decrease in squared error is an even lower rate.
-
Sham Kakade
@ShamKakade6
Nice to see John forging ahead with more rigorous evaluation metrics for audio-to-score alignment.
John Thickstun
@jwthickstun
Excited to share a deep-dive into evaluation methodology for audio-to-score alignment algorithms! As a side effect we created a new, high-quality dataset of ground truth alignments.
arxiv.org/abs/2009.14374 -
Sham Kakade reposted
Adam Wierman
@AdamWierman
An exciting new virtual seminar series on "control meets learning" is launching next week with a talk by @HazanPrinceton thanks to a few motivated @Caltech grads/postdocs/alums. sites.google.com/view/control-meets-learning
-
Sham Kakade reposted
Tim Roughgarden
@Tim_Roughgarden
Opening quip by @mkearnsupenn at his FOCS 1998 tutorial: "I've been tasked with the impossible: Convincing a room full of theorists that there's actually something interesting about *cough* AI." <insert times have changed cliché here>
-
Sham Kakade reposted
Alex Dimakis
@AlexGDimakis
www.thehindu.com/opinion/open-page/a-doyen-of-statistics-…
Of Cramer-Rao and Rao-Blackwell fame -
Sham Kakade reposted
Francis Bach
@BachFrancis
Integration by parts rules again this month! A single equation with diverse applications to randomized smoothing, differentiable perturbed optimizers, and score functions.
francisbach.com/integration-by-parts-randomized-smoothing… -
Sham Kakade reposted
RL Theory Virtual Seminars
@RLtheory
Our next talk:
09/08: Ruosong Wang (CMU)
"Provably Efficient Reinforcement Learning with General Value Function Approximation"
For details, please see the website:
sites.google.com/view/rltheoryseminars/next-seminar -
Sham Kakade reposted
Alex Dimakis
@AlexGDimakis
We're very excited that @UT Austin will lead an NSF national Institute on the Foundations of Machine Learning with @UW, @WichitaState and @MSFTResearch
Announcement: news.utexas.edu/2020/08/26/ut-austin-selected-as-home-of-… -
Sham Kakade
@ShamKakade6
Great to see some theory on self-supervised learning. Looking forward to reading this one!
Jason Lee
@jasondeanlee
Predicting What You Already Know Helps: Provable Self-Supervised Learning
We analyze how predicting parts of the input from other parts (missing patch, missing word, etc.) helps to learn a representation that linearly separates the downstream task.
arxiv.org/abs/2008.01064 1/2 -
Sham Kakade
@ShamKakade6
I feel like I should take the class after reading this 😂
Boaz Barak
@boazbaraktcs
GPT-3 on why Harvard students should take CS 182 this fall (bold text is prompt)
-
Sham Kakade reposted
Emma Brunskill
@EmmaBrunskill
Our ICML RL theory workshop is Friday! @ShamKakade6 will kick us off with our first invited talk. #icml2020
RL Theory Virtual Seminars
@RLtheory
A great venue for sharing your work on RL theory:
ICML 2020 workshop on "THEORETICAL FOUNDATIONS OF REINFORCEMENT LEARNING"
Submission deadline: June 9.
Organized by our friends @EmmaBrunskill, Thodoris Lykouris, Max Simchowitz, Wen Sun, Mengdi Wang
wensun.github.io/rl_theory_workshop_2020_ICML.github.io/ -
Sham Kakade
@ShamKakade6
Thank you so much to the awards committee! Also a huge thanks to the past and current ICML chairs and organizers for all their great work for our community! 👍 It is an honor to receive this 😀😀 with such wonderful co-authors: @arkrause, Matthias, and Niranjan!
Hal Daumé III
@haldaume3
We are very pleased to announce that the #icml2020 Test of Time award goes to
Gaussian Process Optimization in the Bandit Setting: No Regret and Experimental Design
by Niranjan Srinivas, Andreas Krause, Sham Kakade and Matthias Seeger
icml.cc/Conferences/2020/Awards
> -
Sham Kakade reposted
IEEE Information Theory Society
@ieee_itsoc
Alon Orlitsky Named 2021 Claude E. Shannon Award Winner www.itsoc.org/news-events/recent-news/alon-orlitsky-named…
-
Sham Kakade
@ShamKakade6
1/ David Blackwell. Leagues ahead of his time: "What is a good prediction strategy, and how well can you do?" While some things do not seem possible, "Looking for a p [probability] that does well against every x [an outcome] seems hopeless", Blackwell does give us a strategy:
Sham Kakade
@ShamKakade6
2/ In a seminal paper, "Minimax vs Bayes prediction" ('56) homes.cs.washington.edu/~sham/papers/Blackwell56.pdf, Blackwell shows we can predict well on any sequence, using randomization: "we can improve matters by allowing randomized predictions." These ideas permeate so much of learning theory today.
Sham Kakade
@ShamKakade6
3/ Due to Dean Foster, my own education of online learning and sequential prediction started through first understanding Blackwell's approachability, which is a wonderful way to grasp the foundations. I signed this: www.change.org/p/american-statistical-association-rename-…
-
Sham Kakade reposted
RL Theory Virtual Seminars
@RLtheory
We are obviously fans of the list of invited speakers: Shipra Agrawal (@ashipra), @ShamKakade6, Akshay Krishnamurthy, Gergely Neu (@neu_rips), @CsabaSzepesvari, Martha White
-
Sham Kakade reposted
Aravind Rajeswaran
@aravindr93
Algorithmic approach largely inspired by @ShamKakade6's metric E3 and our recent MBRL work in the online RL setting (sites.google.com/view/mbrl-game-arxiv. (5/n)
-
Sham Kakade reposted
Aravind Rajeswaran
@aravindr93
Can model-based RL be effective for offline RL? Absolutely! We develop a new framework and algorithm called MOReL that has both strong theoretical guarantees as well as SOTA results in widely studied benchmarks. (1/n)
arxiv.org/abs/2005.05951 -
Sham Kakade reposted
Simon Shaolei Du
@SimonShaoleiDu
I'm excited to announce that I'll be joining @UW as an assistant professor in the Paul Allen School of Computer Science & Engineering @uwcse in 2021! Looking forward to the exciting future collaborations.
-
Sham Kakade
@ShamKakade6
1/ Playing the long game: Is long horizon RL harder than short horizon RL? Clearly, H length episodes scale linearly with H, but counting learning complexity by # episodes rather than # samples accounts for this. So is it any harder? arxiv.org/abs/2005.00527
Sham Kakade
@ShamKakade6
2/ This was the COLT 2018 open problem from @nanjiang_cs and Alekh, who conjectured a poly(H) lower bound. New work refutes this, showing only logarithmic in H episodes are needed to learn. So, in a minimax sense, long horizons are not more difficult than short ones!
Sham Kakade
@ShamKakade6
3/ joint work with @RuosongW @SimonShaoleiDu @lyang36
-
Sham Kakade reposted
Gautam Kamath
@thegautamkamath
Attending a nice tutorial by Shipra Agrawal (@ashipra) on Thompson Sampling based Methods for RL. Unfortunately will miss Sham Kakade's (@ShamKakade6) talk later on policy gradient methods... Details here (statisticalml.stat.columbia.edu/event/tutorials-on-reinfo…), and livestream here (facebook.com/events/226029992009392/permalink/22657362528…)
-
Sham Kakade reposted
Zachary Lipton
@zacharylipton
Math/music nerds already know: @vihartvihart is among the most talented & charismatic educators on Earth. Her latest video walks through plans for re-opening the economy (builds on whitepaper w @ShamKakade6, @JohnCLangford, et al.) www.youtube.com/watch?v=HhRQxk9QA-o
-
Sham Kakade reposted
Sanjeev Arora
@prfsanjeevarora
Added talks by Josh Tenenbaum and @_beenkim in early evening today!
Sanjeev Arora
@prfsanjeevarora
The number of "tickets" for real-time watching is 500 according to our contract. If you can't get one, please watch the videos a few days later on youtube or the IAS website.
Sanjeev Arora
@prfsanjeevarora
"New directions in optimization, statistics, and machine learning." April 15,16. Online workshop at @the_IAS . Speakers include @pushmeet @StefanoErmon @_beenkim @RogerGrosse @zicokolter @chelseabfinn @percyliang . Enter email info for zoom link organizer.runtheworld.today/invitation/1374
-
Sham Kakade reposted
Sanjeev Arora
@prfsanjeevarora
"New directions in optimization, statistics, and machine learning." April 15,16. Online workshop at @the_IAS . Speakers include @pushmeet @StefanoErmon @_beenkim @RogerGrosse @zicokolter @chelseabfinn @percyliang . Enter email info for zoom link organizer.runtheworld.today/invitation/1374
-
Sham Kakade
@ShamKakade6
1/3 Can open democracies fight pandemics? making a PACT to set forth transparent privacy and anonymity standards, which permit adoption of mobile tracing efforts while upholding civil liberties. arxiv.org/abs/2004.03544
Sham Kakade
@ShamKakade6
2/3 “Privacy Preserving” is misleading, at best, due to re-identification risks. PACT frankly discusses the underlying privacy, security, and re-identification concerns, both to make a case for adoption and for scrutiny by civil liberties organizations.
Sham Kakade
@ShamKakade6
3/3 Anything similar? Yes, Pan European PP-PT www.pepp-pt.org and CEN similar in spirit. Also, PACT protocols are not meant to replace conventional tracing strategies employed by public health organizations. PACT adds functionalities to empower them. Tools coming soon...
-
Sham Kakade reposted
(((E. Glen Weyl/衛谷倫))) ⿻ 🇺🇸/🇩🇪/🇹🇼
@glenweyl
So proud of this white paper (ethics.harvard.edu/outpacing-virus) @HarvardEthics I wrote with @vihartvihart, @divyasiddarth, @BethanCantrell, @lilatretikov, @pde33, @JohnCLangford, @scottleibrand, @ShamKakade6, @danamlewis, Steve Latta, @stefanomtessaro : ethics.harvard.edu/outpacing-virus
-
Sham Kakade reposted
Julien Mairal
@julienmairal
Congratulations to @BachFrancis for being elected to the academy of science!!!
Académie des sciences
@AcadSciences
🌿L'Académie des sciences vient d'élire 4 nouveaux membres à des postes aux thématiques ciblées :
🔵 Nathalie PALANQUE-DELABROUILLE
🔵 Isabelle CHUINE
🔵 Nicolas MOËS
🔵 Francis BACH
🌿 🌿 🌿 Bienvenue à eux ! 🌿 🌿 🌿
👉Pour en savoir ➕: www.academie-sciences.fr/espace-presse -
Sham Kakade reposted
Bored Yann LeCun
@boredyannlecun
Don't forget to stock up on essential items, including Purell, toilet paper, pasta and red sauce, GPUs, my 1989 treatise on convolutional neural networks, the MNIST dataset, the GAN cookbook, > 420 Dance Dance Convolution step packs, and the AdaDelta optimizer. #feelthelearn
-
Sham Kakade reposted
Bored Yann LeCun
@boredyannlecun
To stem the spread of #covid19, the CDC recommends social distancing, or as I like to call it, ***a career in computer science*** #torched
-
Sham Kakade
@ShamKakade6
1/3 Two shots at few shot learning: We have T tasks and N1 samples per task. How effective is pooling these samples for few shot learning? New work: arxiv.org/abs/2002.09434. Case 1: there is a common low dim representation. Case 2: there is a common high dim representation.
Sham Kakade
@ShamKakade6
3/3 with @SimonShaoleiDu @Hoooway @jasondeanlee and Qi Lei!
Sham Kakade
@ShamKakade6
2/3 This work shows that, under either assumption, all T*N1 samples can be used to achieve a precise notion of "few shot" learning. Also, worth pointing out nice work in Maurer et al jmlr.org/papers/volume17/15-242/15-242.pdf. New work makes improvements under assumptions of good common rep!
-
Sham Kakade
@ShamKakade6
1/3 Should only Bayesians be Bayesian? No. Being Bayes is super robust! An oldie but goodie from Vovk: "Competitive Online Statistics" 2001. Beautiful work showing Bayes is awesome, even if you are not a Bayesian. (post motivated by nice thoughts from @RogerGrosse @roydanroy).
Sham Kakade
@ShamKakade6
2/3 This motivated this this line of work on Bayesian methods in the worst case: papers.nips.cc/paper/2637-online-bounds-for-bayesian-algo… and homes.cs.washington.edu/~sham/papers/ml/ieeeinf.pdf
Sham Kakade
@ShamKakade6
3/3 And the seminal papers that started this line of thought: Dawid, "The prequential approach" (1984), and Foster, "Prediction in the worst case" (1991). They def influenced my thinking! stats meets philosophy. good stuff.
-
Sham Kakade
@ShamKakade6
Great to see that Mike Jordan is thinking about Decision Theory, ML, and Econ! Super important area: lots of stats/algorithmic questions that have immediate impact on practice. Few other areas that one can say the same!
ACM Education & Learning Center
@acmeducation
Mar 25 #acmtechtalk "The Decision-Making Side of Machine Learning: Computational, Inferential, and Economic Perspective," w/Michael I. Jordan. event.on24.com/eventRegistration/unavailable?&eventid=221…
@JeffDean @smolix @etzioni @erichorvitz @lexfridman @DaphneKoller @pabbeel @ShamKakade6 @suchisaria @aleks_madry -
Sham Kakade
@ShamKakade6
1/2. No double dipping. My current worldview: the 'double dip' is not a practical concern due to that tuning of various hyperparams (early stopping, L2 reg, model size, etc) on a holdout set alleviates the 'dip'. This work lends evidence to this viewpoint!
Preetum Nakkiran
@PreetumNakkiran
Optimal Regularization can Mitigate Double Descent
arxiv.org/abs/2003.01897
Joint work with Prayaag Venkat, @ShamKakade6, @tengyuma.
We prove in certain ridge regression settings that *optimal* L2 regularization can eliminate double descent: more data never hurts (1/n)
Sham Kakade
@ShamKakade6
2/2 Why double dip, you ask? Is it a theorists' concoction? Perhaps, yes. It is a compelling demo of how SGD (and GD) behave differently in the overparameterized regime; a great question to study in its own right! As an 'asymptotic', it may not be how practitioners roll...
-
Sham Kakade
@ShamKakade6
John was a reason I moved to AI and neuroscience from physics . In his first class, he compared the human pattern-matching algo for chess playing to DeepBlue's brute force lookahead. I wondered if Go would be mastered in my lifetime! Wonderful to hear from John Hopfield again!
Lex Fridman
@lexfridman
Here's my conversation with John Hopfield. Hopfield networks were one of the early ideas that catalyzed the development of deep learning. His truly original work has explored the messy world of biology through the piercing eyes of a physicist. www.youtube.com/watch?v=DKyzcbNr8WE
-
Sham Kakade
@ShamKakade6
Looking forward to reading this one! There isn't a compelling explanation for the unreasonable effectiveness of adam/adagrad (aside from the original convex regret bounds). So this looks quite promising!
Elad Hazan
@HazanPrinceton
Trying out yet another deep learning optimizer? Graft its learning rates to better understand its performance:
arxiv.org/abs/2002.11803
w. @naman33k @_arohan_ Tomer Koren and Cyril Zhang -
Sham Kakade
@ShamKakade6
couldn't reconcile theory/practice with dropout for over a year. New: arxiv.org/abs/2002.12915 w/ @tengyuma & C. Wei. turns out dropout sometimes has an implicit regularization effect! pretty wild. just like small vs. large batch sgd. these plots def surprised us!
-
Sham Kakade reposted
Sanjeev Arora
@prfsanjeevarora
Simons Foundation and NSF propose to spend $20M to fund projects on Mathematical and Scientific Foundations of Deep Learning preview.tinyurl.com/qqws9sp
An interesting public-private partnership to fund basic research. -
Sham Kakade
@ShamKakade6
Bellairs. Day 5 @HazanPrinceton and myself: double feature on controls+RL. +spotlights: @maithra_raghu: meta-learning as rapid feature learning. Raman Arora: dropout, capacity control, and matrix sensing . @HanieSedghi: module criticality and generalization! And that is a wrap!🙂
-
Sham Kakade
@ShamKakade6
Bellairs. Theory of DL. Day 4, penultimate session from the unstoppable Boaz Barak. Average case complexity, computational limits, and relevance to DL. Front row: Yoshua Bengio, Jean Ponce, @ylecun
Sham Kakade
@ShamKakade6
Unlikely.
-
Sham Kakade
@ShamKakade6
Bellairs Research Institute ☀️🏖️. Theory of DL. Day 3: @nadavcohen: "Gen. and Opt. in DL via Trajectories". careful study of deep linear nets. second time hearing about it. now appreciate how this reveals quite different effects, relevant for DL! also, 🏊♂️🥥 🍨!
Sham Kakade
@ShamKakade6
edit. day 4.
-
Sham Kakade
@ShamKakade6
Bellairs Research Institute☀️🏖️.Theory of DL. Day 3: new insights from @roydanroy @KDziugaite on PAC-Bayes for DL. possibly gives a new lens into implicit reg 🤔 @david_rolnick cool results on expressivity of deep nets. And T. Lillicrap keeps us real on theory vs. practice!
-
Sham Kakade
@ShamKakade6
Bellairs Research Institute ☀️⛱️. Theory of DL workshop, Day 2 (eve): Thanks to Yann LeCun and Yoshua Bengio for thought provoking talks. @ylecun title: "Questions from the 80s and 90s". Good questions indeed!!
-
Sham Kakade
@ShamKakade6
At the Bellairs Research Institute: Theory of Deep Learning workshop. Day 1: great presentations on implicit regularization from @prfsanjeevarora @suriyagnskr . Day 2: lucid explanations of NTKs from @Hoooway @jasondeanlee. Good friends, sun ☀️, and sand 🏖️ a bonus.
-
Sham Kakade
@ShamKakade6
Nice notes!
Pieter Abbeel
@pabbeel
One of my favorites from most recent offering of CS287 Advanced Robotics?
Exam study handout summarizing all the main math in ~20pp. Incl. MaxEnt RL, CEM, LQR, Penalty Method, RRTs, Particle Filters, Policy Gradient, TRPO, PPO, Q-learning, DDPG, SAC,
people.eecs.berkeley.edu/~pabbeel/cs287-fa19/exam/cs287-f… -
Sham Kakade reposted
Aditya Kusupati
@adityakusupati
Can we learn pruning thresholds and, in-turn, layer-wise sparsity distribution?
Introducing STR, Soft Threshold Reparameterization, a simple trick on weight tensors to achieve SOTA accuracy in sparse DNNs while reducing inference FLOPs significantly!
arxiv.org/abs/2002.03231 1/n -
Sham Kakade
@ShamKakade6
Congratulations to the new Sloan Research Fellows! sloan.org/fellowships/2020-Fellows
-
Sham Kakade reposted
Sebastien Bubeck
@SebastienBubeck
Congratulations to our colleague Lin Xiao @MSFTResearch for the #neurips2019 test of time award!!!
Online convex optimization and mirror descent for the win!! (As always? :-).)
medium.com/@NeurIPSConf/neurips-2019-paper-awards-807e41d… -
Sham Kakade
@ShamKakade6
Just got back from MSR Montreal and had a great visit! Lots of cool projects going on there in RL/NLP/unsupervised learning. Thanks to @APTrizzle @momusbah @Drewch @JessMastronardi @philip_bachman for hosting me!
-
Sham Kakade reposted
Elad Hazan
@HazanPrinceton
Real pleasure to give a colloquium at lovely EPFL about our groups' work on non-stochastic control. Many excellent questions and control researchers in the audience.
Lê Nguyên Hoang bsky.app/profile/science4all.org
@le_science4all
Today's @ICepfl talk is given by @HazanPrinceton of @Princeton and @GoogleAI.
He'll discuss his way of tackling control theory with a machine learning background. -
Sham Kakade reposted
David Duvenaud
@DavidDuvenaud
We approximated the implicit function theorem to tune millions of hyperparameters. Now we can train data augmentation networks from scratch using gradients from the validation loss.
arxiv.org/pdf/1911.02590.pdf
With @JonLorraine and @PaulVicol -
Sham Kakade
@ShamKakade6
Beautiful post by @BachFrancis on Chebyshev polynomials: francisbach.com/chebyshev-polynomials/. Handy for algorithm design. Let's not forget the wise words of Rocco Servedio, as quoted by @mrtz, "There's only one bullet in the gun. It's called the Chebyshev polynomial."
-
Sham Kakade reposted
Yaroslav Bulatov
@yaroslavvb
Something not widely enough appreciated -- largest SGD learning rates scale like 1/trace(H), rather than 1/norm(H) of regular gradient descent. One way to get this is from eq 3. of @jainprateek_ @ShamKakade6 's
arxiv.org/abs/1610.03774 -
Sham Kakade reposted
Aaron Roth
@Aaroth
Announcing an exciting new conference: "Foundations of Responsible Computing" (FORC). Here is the call for papers: responsiblecomputing.org/forc-2020-call-for-paper/. The deadline is February 11, and (if I can say so myself :-) it has an absolutely amazing steering and program committee. 1/
-
Sham Kakade
@ShamKakade6
A nice point: better features not better classifiers are key. This is more generally an important point related to distribution shift: (also comes up in RL, related to our " is a good representation sufficient" paper).
Aleksander Madry
@aleks_madry
Video summaries for our papers "Adversarial Examples Aren't Bugs They're Features" (www.youtube.com/watch?v=-vhUWSHOqIM) and "Image Synthesis with a Single Robust Classifier" (www.youtube.com/watch?v=l3SoeIIjxTM) are now online. Enjoy! (@andrew_ilyas @tsiprasd @ShibaniSan @logan_engstrom Brandon Tran)
-
Sham Kakade
@ShamKakade6
nice to see this package is out there, which helps to support comparisons to more 'classical' algorithms. Also, given it is in PyTorch, one could do backprop through these methods.
Sasha Rush
@srush_nlp
PyTorch-Struct (v0.3 github.com/harvardnlp/pytorch-struct).
New features: autoregressive models / beam search, sparse-max dp, alignment/dtw, parallel semi-markov, k-max, pretty docs (harvardnlp.github.io/pytorch-struct)
Fun example: gradients of time-warping crf under different semirings. -
Sham Kakade
@ShamKakade6
cool stuff from @TheGregYang: Tensors, Neural Nets, GPs, and kernels! looks like we can derive a corresponding kernel/GP in a fairly general sense. very curious on broader empirical comparisons to neural nets, which (potentially) draw strength from the non-linear regime!
Greg Yang
@TheGregYang
1/ I can't teach you how to dougie but I can teach you how to compute the Gaussian Process corresponding to infinite-width neural network of ANY architecture, feedforward or recurrent, eg: resnet, GRU, transformers, etc ... RT plz💪arxiv.org/abs/1910.12478
-
Sham Kakade
@ShamKakade6
Nice talk from Rong Ge! learning two layer neural nets, with _finite_ width: arxiv.org/pdf/1810.06793.pdf A seriously awesome algebraic idea. Reminiscent of FOOBI (the coolest spectral algo in town!): they replace the 'rank-1-detector' in FOOBI with a 'one-neuron-detector'.
Sham Kakade
@ShamKakade6
Also, the FOOBI (Fourth-Order-Only Blind Identification) paper: publi-etis.ensea.fr/2007/DCC07/ldl-06-38.pdf. A beautiful algo. And a catchy acronym too! Still surprises me how one can efficiently impose that rank one constraint! Worth a read.
-
Sham Kakade
@ShamKakade6
Very cool! Years ago (a little post AlexNet) I put in (too?) many cycles trying to design such a kernel. Didn't match their performance (though didn't have much compute back then). Pretty slick that they use a derived kernel from a ConvNet!
Ruosong Wang
@RuosongW
We have released code for computing Convolutional Neural Tangent Kernel (CNTK) used in our paper "On Exact Computation with an Infinitely Wide Neural Net", which will appear in NeurIPS 2019.
Paper: arxiv.org/abs/1904.11955
Code: github.com/ruosongwang/CNTK -
Sham Kakade
@ShamKakade6
revised thoughts on Neural Tangent Kernels (after understanding the regime better. h/t @SimonShaoleiDu): def a super cool idea for designing a kernel! It does not look to be helpful of our understanding of how representations arise in deep learning. Much more needed here!
-
Sham Kakade
@ShamKakade6
nice talk! and an important direction to pursue; the older margin based ideas def need to be refined. so nice to see this here!
Tengyu Ma
@tengyuma
A new paper on improving the generalization of deep models (w.r.t clean or robust accuracy) by theory-inspired explicit regularizers. arxiv.org/abs/1910.04284
-
Sham Kakade
@ShamKakade6
Looking forward to reading this one! The original Bousquet and Elisseeff work was way ahead of its time! Epic in retrospect.
Vitaly 🇺🇦 Feldman
@vitalyFM
Indeed, really neat simplification of the bounds!
-
Sham Kakade
@ShamKakade6
Wrapped up at the "Workshop on Theory of Deep Learning: Where next?" at IAS. www.math.ias.edu/wtdl. The field has moved so much! e.g. Neural Tangent Kernel (NTK) results! A few years ago, understanding DL looked hopeless. Terrific set of talks, too!
-
Sham Kakade
@ShamKakade6
huh... so this is pretty wild. it is _formally_ equivalent to the the Polyak heavy ball momentum algorithm (with weight decay). not just 'similar behavior'.
Sanjeev Arora
@prfsanjeevarora
Conventional wisdom: slowly decay learning rate (lr) when training deep nets. Empirically, some exotic lr schedules also work, eg cosine. New work with Zhiyuan Li: exponentially increasing lr works too! Experiments + surprising math explanation. See arxiv.org/abs/1910.07454
Sham Kakade
@ShamKakade6
oh yeah. batch norm is assumed, which is clearly why things aren't exploding. But still, it is quite cool that they are formally equivalent; note their scheme only tracks one param, as opposed to two with momentum.
-
Sham Kakade reposted
Aleksander Madry
@aleks_madry
Update: Our Robustness library now comes with a number of CIFAR10 and ImageNet models (and their corresponding accuracies) trained against a spectrum of L2 and Linf adversaries. github.com/MadryLab/robustness
-
Sham Kakade reposted
Sanjeev Arora
@prfsanjeevarora
Workshop: "Theory of Deep Learning: Where Next?" at the Institute for Advanced Study, Tuesday--Friday this week. Amazing schedule of talks! www.math.ias.edu/wtdl
Registration is closed (sorry), but follow livestream here www.ias.edu/livestream -
Sham Kakade
@ShamKakade6
Wow! This is amazing. A few years ago RL was starting to be applied in the robotics domain, with many doubters. Fast forward a handful of years and this! 👏
OpenAI
@OpenAI
We've trained an AI system to solve the Rubik's Cube with a human-like robot hand.
This is an unprecedented level of dexterity for a robot, and is hard even for humans to do.
The system trains in an imperfect simulation and quickly adapts to reality: openai.com/blog/solving-rubiks-cube/ -
Sham Kakade reposted
Sukant Khurana
@Sukant_Khurana
This is the rejection letter for the work that just won the Nobel Prize.
Believe in yourself.
Everyone else will catch up eventually. -
Sham Kakade reposted
Ben Recht
@beenwrekt
Remember when OpenAI invented the ResNet? Neither do I, because it was developed at Microsoft Research.
-
Sham Kakade
@ShamKakade6
looking forward to this blog. starting off with a great post...
Francis Bach
@BachFrancis
As promised, my first blog post is out, on the "eta-trick".
francisbach.com/the-η-trick-or-the-effectiveness-of-rewei… -
Sham Kakade reposted
Sanjeev Arora
@prfsanjeevarora
Conventional wisdom: "Not enough data? Use classic learners (Random Forests, RBF SVM, ..), not deep nets." New paper: infinitely wide nets beat these and also beat finite nets. Infinite nets train faster than finite nets here (hint: Neural Tangent Kernel)! arxiv.org/abs/1910.01663
-
Sham Kakade
@ShamKakade6
A nice read!
Simon Shaolei Du
@SimonShaoleiDu
Check out new blog post on deep learning theory: ultra-wide neural network and Neural Tangent Kernel.
-
Sham Kakade
@ShamKakade6
What actually constitutes a good representation for reinforcement learning? Lots of sufficient conditions. But what's necessary? New paper: arxiv.org/abs/1910.03016. Surprisingly, good value (or policy) based representations just don't cut it! w/ @SimonShaoleiDu @RuosongW @lyang36
-
Sham Kakade
@ShamKakade6
nice post with some cool explanations!
Ferenc Huszár
@fhuszar
New post on iMAML: Meta Learning with Implicit Gradients
some animations, discussing potential limitations and of course a Bayesian/variational interpretation www.inference.vc/notes-on-imaml-meta-learning-without-dif… -
Sham Kakade
@ShamKakade6
Also, we recently posted this work arxiv.org/abs/1908.00261 on the theory of policy gradients for reinforcement learning! A long time in the works, this paper finally gets a handle on function approximation with policy gradient methods.
-
Sham Kakade
@ShamKakade6
Excited to share this new work:
Chelsea Finn
@chelseabfinn
It's hard to scale meta-learning to long inner optimizations.
We introduce iMAML, which meta-learns *without* differentiating through the inner optimization path using implicit differentiation.
arxiv.org/abs/1909.04630
to appear @NeurIPSConf
w/ @aravindr93 @ShamKakade6 @svlevine -
Sham Kakade reposted
Suchi Saria
@suchisaria
I decided to one up @ShamKakade6 but may not have fully succeeded. Here are some photos from the evening! Was a real treat to join the Turing award celebrations. My favorite bit was Geoff’s closing words: don’t give up on an idea you believe in even if people call it silly.
-
Sham Kakade reposted
Suchi Saria
@suchisaria
Very fun discussions but I am feeling a bit stood up.
They are all formally dressed, especially Sham in a nice red tie w/ matching red socks and a black suit while I am in my jeans & vest. I didn’t know ACM/Turing meeting automatically meant upgrade your clothes!
Association for Computing Machinery
@TheOfficialACM
Tune into the panel "Deep Learning, Reinforcement Learning, and Role of Methods in Data Science" featuring Shirley Ho, Sham Kakade (@ShamKakade6), Suchi Saria (@suchisaria) and Manuela Veloso at the ACM-IMS Data Science Summit: www.acm.org/data-science-summit/livestream
@InstMathStat -
Sham Kakade
@ShamKakade6
It’s great to be a visitor here!
Elad Hazan
@HazanPrinceton
Very excited to see this finally announced & many thanks to @JeffDean , @GoogleAI and @Princeton for the ongoing support!
+ fresh from the oven, research from the lab: arxiv.org/abs/1812.02690
Google AI
@GoogleAI
Research Scientists @HazanPrinceton and Yoram Singer discuss some early research from Google AI Princeton, a new lab opening early next year that aims to broaden research in multiple facets of #machinelearning. goo.gl/Wdcq7n
-
Sham Kakade reposted
Elad Hazan
@HazanPrinceton
Deep RL fan? Tune in to Sham Kakade's talk today @ workshop on deep reinforcement learning @ NeurIPS!
Provable exporation in RL: arxiv.org/abs/1812.02690
joint work with star student Karan Singh and PU undergrad (!) Abby Van Soest. -
Sham Kakade reposted
Allen School
@uwcse
Congrats to #uwallen theory prof Yin Tat Lee & collaborators Kevin Scaman, @SebastienBubeck, @BachFrancis, & Laurent Massoulié on their #neurips2018 Best Paper Award for presenting new algorithms for distributed optimization of non-smooth convex functions. news.cs.washington.edu/2018/12/06/allen-schools-yin-tat-l…
-
Sham Kakade
@ShamKakade6
Congrats to @SebastienBubeck @BachFrancis, Yin Tat, Laurent, and Kevin for a cool paper! Looking forward to reading it.
Sebastien Bubeck
@SebastienBubeck
At #neurips2018 where it was just announced that our paper on non-smooth distributed optimization with Kevin Scaman, @BachFrancis , Laurent Massoulie and Yin Tat Lee got a best paper award. Lots of interesting open problems left there, check out the paper arxiv.org/abs/1806.00291
-
Sham Kakade
@ShamKakade6
At #neurips18. Excited to present at my poster tonight on automatic subdiffereniation arxiv.org/abs/1809.08530 with Jason D. Lee.
-
Sham Kakade
@ShamKakade6
Hi!







































































































































































































