I’m starting a new early-stage AI fund, doing what I love: partnering with founders from company formation on bold bets, long before consensus.
The fund’s LPs will include Sequoia, Jeff Dean, Sachin Katti, Philip Wong (my Stanford professor and Chief Scientist adviser to TSMC), and others whose work has shaped our industry and how I think about AI’s future.
I’ll remain a partner at Sequoia and keep serving on my boards on Sequoia’s behalf.
Sequoia has been my home for the past 11 years. I joined as the youngest investor the firm had ever hired. I grew up here. Through years of intense debate and shared work, I found lifelong mentors and friends who took me under their wing and taught me this craft. They taught me to aim high and take big swings. I’ll carry that into what I build: outsized ambition, relentless truth-seeking and curiosity, and a refusal to become complacent.
Working alongside founders has made me want to build something of my own: a firm shaped from day one around the convictions I’ve developed over the past decade.
Life has taught me not to take time for granted. This is how I want to spend mine.
Why now
We’re seeing a new way of building companies in today’s AI era. Research and products often advance together. A breakthrough can open a market or accelerate a business far faster than expected. Some companies require extraordinary capital and compute early on. Others reach meaningful scale with remarkably small teams. The rules for how big these businesses can get, and how fast, are being rewritten.
But early on, the technology is unproven, the resources required enormous, and the vision seemingly implausible.
This is what I love most: seeing the potential right from the beginning, and helping founders realize their full scale of ambition.
More than ever, it starts with an extreme willingness to believe early, with bold, independent conviction in a founder’s technical insight before any product or market validation. Then, it means helping founders take the biggest swings, giving the rare, long-term patience to see those bets through uncertainty, ambiguity, and turbulence, and serving as a level head when the stakes are highest.
I learned this kind of partnership at Sequoia. It’s the foundation of the firm I’m building.
Believing before consensus
My fascination with AI’s potential took root at Stanford in the early-to-mid 2010s, studying computer science and working in Andrew Ng’s lab. The friendships I formed connected me to a community whose people and ideas would help shape modern AI and build some of our generation’s defining companies. What stayed with me was the gap between what those people saw was possible and what the wider world could imagine.
I’m drawn to founders pursuing ambitions that sound impossible today.
I start with a simple question: “What if everything goes right?”
Then we work backward together to make it happen.
In 2023, Deepak and Abhinav set out to build a general-purpose robot brain at @SkildAI, before physical AI was a consensus bet. Their sim-to-real breakthroughs gave me conviction where earlier robotics approaches had fallen short. But the data flywheel required deployment at scale. We made commercialization as urgent as research and brought in strategic partners like NVIDIA and SoftBank to support the ambition. Three years in, Skild is advancing the frontier of physical AI with nine figures in annualized revenue.
In 2024, Ioannis and Misha founded @reflection_ai with an insight into scaling reinforcement learning. As that grew into the ambition to build a Western frontier lab for open models, the hard question was how a startup competes with labs spending tens of billions. An introduction to NVIDIA and many compute calls and recruiting events later, the team has built Beam, now one of the leading US open models.
In 2025, Anna and Azalia, the co-creators of AlphaChip, founded @RicursiveAI to design models and hardware together. From day one, we made early customers and ecosystem partners a priority. Today, they are showing order-of-magnitude improvements on real workloads, working with some of the industry’s most consequential players, and reaching a defining inflection point.
The ambition keeps growing as these companies scale.
@linear has always had a kind of magic to it. Now it’s the system a new generation of AI companies builds on, for teams and agents alike.
@tavus saw early that we’d one day talk to AI the way we talk to each other. Today, it’s building the human computing interface for a world of AI colleagues.
@Mach_Industries began as hydrogen experiments in an MIT dorm room. It has since become one of the most exciting next-generation defense companies.
Making the ambition possible
Being a first believer is only the beginning. Compute falls through. A deep-pocketed incumbent makes your mission its top priority. A breakthrough changes the opportunity and the capital required overnight.
These are stomach-churning moments, and early belief carries a responsibility to help make the bet work.
This is the partnership I love: turning the full ambition into concrete steps. Raising capital, securing compute and strategic partners, recruiting exceptional people, and winning the first customers together. Thinking several moves ahead while solving what’s in front of us. And as breakthroughs open new possibilities, helping founders dream even bigger.
Thank you
To the founders I’ve worked with: thank you for trusting me and Sequoia to be your first partner and first believer. It’s been the joy of a lifetime, and I can’t wait to keep building together.
Thank you to Alfred and Pat for your leadership of Sequoia and your support of my next chapter. To Roelof, for bringing me into Sequoia and believing in me from the beginning. Alfred and Roelof, you have been mentors in investing and in life. So much of what I know, and the investor I’ve become, comes from your guidance and generosity over the years. To Bill, for the close partnership and wealth of knowledge, always wrapped in humility. To Shaun, for showing me how to think deeply and imaginatively. To Michael Moritz, for inspiring 22-year-old me to dream big. To Doug, for teaching me to keep things simple. To Bogie and Jess, for bringing joy to the everyday and perspective on what matters. And to all my Sequoia teammates, for the friendship, camaraderie, and lessons I’ll carry with me.
To my family, especially my husband and my brother, who have each started companies of their own: thank you for giving me the courage to take this leap.
Let’s get to work!
Quoc V. Le
Lê Viết Quốc (born 1982), or in anglicized form Quoc Viet Le, is a Vietnamese-American computer scientist and artificial intelligence researcher. He is a Google Fellow at Google DeepMind and a founding member of the Google Brain project. More on Wikipedia
-
Quoc Le reposted
Stephanie Zhan
@stephzhan
-
Quoc Le reposted
"Leigh Marie" Braswell
@LM_Braswell
Excited to partner with @DiscoLoopAI ! They are actively hiring across engineering & research - please reach out if the mission sounds interesting to you.
Discovery Loop
@DiscoLoopAI
Introducing Discovery Loop ♾. We’re building AI to run experiments at unprecedented scale and to solve the biggest bottlenecks in science and engineering.
Our founders @JeffDean, @Sanjay_Ghemawat, @quocleix, @OriolVinyalsML helped build the AI infrastructure and models that scaled the modern world. Now, we are scaling discovery itself.
The future can't wait.
Learn more at: www.discoveryloop.com -
Quoc Le reposted
Tao Tu
@taotu831
I am excited to share some of the progress we are making towards using Gemini to accelerate scientific discovery in the real-world.
We present an extension of Co-Scientist that transitions from pure in silico ideation to execution-grounded research partner, adapting to multiple levels of autonomy across different scientific domains. -
Quoc Le reposted
Jeff Dean
@JeffDean
Indeed, it has been quite a week, so I thought I'd share it with the KDD 2026 audience! For my Google colleagues who might have sent me internal chat messages after I closed my laptop for a final time on Thursday afternoon, I apologize for my lack of response!
(Zoomed in)
Dhruv Kuchhal
@kuchhal_dhruv
Legendary! Thanks for sharing the behind-the-scenes @JeffDean!
-
Quoc Le reposted
Ravi Mhatre
@ravi_lsvp
@JeffDean has built the foundations that every era of modern computing has stood on, from web-scale infrastructure to scaling deep learning itself. Exceptional people inspire great talent and the founding team around him is among the strongest I have ever seen in over 25 years of investing. @lightspeedvp is honored to join them from the start of this journey.
Jeff Dean
@JeffDean
Announcing Discovery Loop!
I am very excited to announce that, along with my longtime friends and collaborators @Sanjay_Ghemawat, @OriolVinyalsML and @quocleix, we are founding Discovery Loop (@DiscoLoopAI), a Public Benefit Corporation whose mission is to automate machine learning, science, and engineering to accelerate discoveries and progress. The four of us have worked together for 14 to 30 years, and have helped build some of the world’s most used products, infrastructure and AI models, and we’re excited to turn our attention to this ambitious endeavor.
♾
Learn more at: www.discoveryloop.com -
Quoc Le
@quocleix
RT @OriolVinyalsML: Founder Mode activated from my new office ♾
-
Quoc Le reposted
rohan anil
@_arohan_
I am going to say a turning award is due for @JeffDean and @Sanjay_Ghemawat
-
Quoc Le reposted
Yoonseok Yang
@yoonseok_yang
In 2022, as a Google intern, I sent a random calendar invite to Jeff Dean.
He accepted.
His calendar was accessible even to interns, so I figured I had nothing to lose.
The next day, I sent this photo to my team. Everyone was shocked.
It’s hard to imagine Google without Jeff. He’s been one of my biggest inspirations.
Thank you for everything, Jeff.
Jeff Dean
@JeffDean
Tomorrow will be my last day at Google after 27 years, and watching it grow from 25 people to 190,000+ has been an amazing journey. Below is a note I shared with many people internally at Google today. An excerpt is:
It has been an absolute pleasure to work with you and to help build some of the most widely used and impactful products of all time. As a kid, I dreamed of helping build software that would be used by many people, and Google now has thirteen products used by more than a billion people (amazing!). Our work has had a tremendous impact in the world, and I have been lucky enough to collaborate and form friendships with many colleagues that I deeply admire, respect, and enjoy. It still brings me joy every time I see people out in the world using our products to find information, handle email, translate documents, watch videos, learn new things, navigate and understand the physical world, browse the web, use their phone, run large-scale computations on our infrastructure, ride in an autonomous vehicle, or perform complex tasks with the help of our AI systems. I hope you all share this sense of joy, because it is a shared accomplishment! Thank you to all of my colleagues at Google over many years!
Now I'm excited to go start @DiscoLoopAI with my longtime friends and colleagues @Sanjay_Ghemawat, @OriolVinyalsML, and @quocleix.
(Updated post: slightly redacted to not have some personal info) -
Quoc Le reposted
Vinod Khosla
@vkhosla
Super excited to back this super star team at @khoslaventures on its quest for huge impact. Could be larger than the impact that @Google had when we backed them at Kleiner in 1998 and @OpenAI in 2018 at @khoslaventures
Jeff Dean
@JeffDean
Announcing Discovery Loop!
I am very excited to announce that, along with my longtime friends and collaborators @Sanjay_Ghemawat, @OriolVinyalsML and @quocleix, we are founding Discovery Loop (@DiscoLoopAI), a Public Benefit Corporation whose mission is to automate machine learning, science, and engineering to accelerate discoveries and progress. The four of us have worked together for 14 to 30 years, and have helped build some of the world’s most used products, infrastructure and AI models, and we’re excited to turn our attention to this ambitious endeavor.
♾
Learn more at: www.discoveryloop.com -
Quoc Le
@quocleix
Excited to co-found Discovery Loop with my long-time collaborators @JeffDean @Sanjay_Ghemawat @OriolVinyalsML . Our mission is to automate machine learning, engineering and science. Learn more at: www.discoveryloop.com ♾
Jeff Dean
@JeffDean
Announcing Discovery Loop!
I am very excited to announce that, along with my longtime friends and collaborators @Sanjay_Ghemawat, @OriolVinyalsML and @quocleix, we are founding Discovery Loop (@DiscoLoopAI), a Public Benefit Corporation whose mission is to automate machine learning, science, and engineering to accelerate discoveries and progress. The four of us have worked together for 14 to 30 years, and have helped build some of the world’s most used products, infrastructure and AI models, and we’re excited to turn our attention to this ambitious endeavor.
♾
Learn more at: www.discoveryloop.com -
Quoc Le reposted
koray kavukcuoglu
@koraykv
Today at Google I/O, we introduced Gemini 3.5 Flash! It has become an integral part of our daily research cycle and works with all the tools we have at Google.
We used a team of agents in Antigravity 2.0 to recreate the original AlphaZero research paper and build a playable version. They coded the reinforcement learning pipeline in JAX/Flax, trained a ResNet model from scratch via self-play on multi-TPU pods, and shipped a full-stack web app so you can play against it, from just 2 prompts. .
Here’s what else makes 3.5 Flash special 🧵 -
Quoc Le reposted
Jeff Dean
@JeffDean
1/ Today at #googleio, we’re releasing Gemini 3.5, our latest family of models combining frontier intelligence with action.
We’re starting by releasing 3.5 Flash, which is built to help you execute complex, long-horizon agentic workflows.
Gemini 3.5 Flash is our strongest model for coding and agent yet.it outscores 3.1 Pro on agentic and coding benchmarks like Terminal-Bench and MCP Atlas, while running 4x faster than other frontier models.
Used in Google Antigravity, 3.5 Flash is even further optimized to be up to 12x faster. It’s a powerful engine to deploy sub-agents that collaborate, run high-frequency iterative loops, and solve real-world problems at scale.
Some highlights we’re excited about 🔽 -
Quoc Le reposted
Thang Luong
@lmthang
Very excited to share a new milestone in AI for Math: Aletheia, powered by Gemini Deep Think, was just used to autonomously solve a Kirby problem! “Kirby’s list” is a “compendium of the most important unsolved problems in topology, the study of deformable shapes” (Quanta magazine). 🧵
-
Quoc Le reposted
Conglong Li
@conglongli
[🧵1/12] We evaluated Gemini 3.1 Pro and its Deep Think mode on regional contests of International Mathematical Olympiad, International Collegiate Programming Contest, and International Olympiad in Informatics in 8 languages. Deep Think beats/matches competitors on all contests.
-
Quoc Le reposted
Thang Luong
@lmthang
“Mathematics is undergoing the biggest change in its history.”
Glad to see our math agent #aletheia featured by @newscientist on its results at the FirstProof inaugural challenge, alongside other interesting milestones in AI for Math research! Implications worth thinking about. Link in thread. -
Quoc Le reposted
机器之心 JIQIZHIXIN
@jiqizhixin
Google just solved a theoretical physics problem using Gemini!
Google, Harvard, and CMU built a neuro-symbolic system using the Gemini Deep Think model and a tree-search framework to autonomously discover complex mathematical proofs.
The agent functions like a digital scientist, testing different analytical paths and using numerical feedback to refine its logic until it finds a perfect solution.
This approach successfully solved a major open problem regarding gravitational radiation from cosmic strings, outperforming previous AI attempts by delivering full analytical solutions where others only found partial approximations. -
Quoc Le reposted
Sundar Pichai
@sundarpichai
Gemini 3.1 Flash-Lite is the fastest and most cost-efficient Gemini 3 series model⚡️
It outperforms 2.5 Flash with a 2.5X faster Time to First Answer Token and a 45% increase in output speed, at a fraction of the cost of larger models!
koray kavukcuoglu
@koraykv
Gemini 3.1 Flash-Lite is available now! It takes an unbelievable amount of complex engineering to make AI feel instantaneous, enabling exciting new frontiers for experimentation!
This media couldn't be displayed. See it on the original post. -
Quoc Le reposted
koray kavukcuoglu
@koraykv
Gemini 3.1 Flash-Lite is available now! It takes an unbelievable amount of complex engineering to make AI feel instantaneous, enabling exciting new frontiers for experimentation!
This media couldn't be displayed. See it on the original post. -
Quoc Le reposted
koray kavukcuoglu
@koraykv
Nano Banana 2, new state of the art in image generation and editing combined with Gemini’s real-world knowledge!
You can simulate 3D CAD models purely through images. From sketch to real object! -
Quoc Le reposted
Carlos Santana
@DotCSV
🔴 ¡LA IA MATEMÁTICA de GOOGLE!
Google acaba de confirmar que su agente matemático Aletheia consiguió resolver (ahora sí, confirmado) 6 de 10 problemas matemáticos MUY difíciles del reto FirstProof! 🔥
Esto es potente, os cuento 👇🧵 -
Quoc Le
@quocleix
Exciting results in AI math research! We use Aletheia agent, powered by Gemini 3 Deep Think, to tackle the FirstProof challenge. Operating completely autonomously, Aletheia successfully solved 6 out of the 10 problems. Check out the full paper for details on the methodology and expert evaluations. arxiv.org/abs/2602.21201
Quoc Le
@quocleix
See @lmthang tweet for more info:
Thang Luong
@lmthang
Thrilled to share: #aletheia, our math research agent, just solved 6/10 notoriously hard FirstProof problems autonomously, the best result in the inaugural challenge! To me, this is even bigger than our historic IMO-gold achievement last year; these problems challenge even top mathematicians. We share our results transparently, see paper and full thoughts in the thread. 👇
-
Quoc Le reposted
-
Quoc Le reposted
Artificial Analysis
@ArtificialAnlys
Google is once again the leader in AI: Gemini 3.1 Pro Preview leads the Artificial Analysis Intelligence Index, 4 points ahead of Claude Opus 4.6 while costing less than half as much to run
@GoogleDeepMind gave us pre-release access to Gemini 3.1 Pro Preview. It leads 6 of the 10 evaluations that make up the Artificial Analysis Intelligence Index and improves significantly over Gemini 3 Pro Preview across capabilities, with the biggest gains in reasoning and knowledge, coding, and hallucination reduction.
Gemini 3.1 Pro Preview also remains relatively token efficient, using ~57M tokens to run the Artificial Analysis Intelligence Index (+1M from Gemini 3 Pro Preview), lower than other frontier models at max reasoning settings such as Opus 4.6 (max) and GPT-5.2 (xhigh). Combined with lower per-token pricing, Gemini 3.1 Pro Preview is cost-efficient among frontier peers, costing less than half as much as Opus 4.6 (max) to run the full Intelligence Index, though still nearly 2x the leading open-weights model, GLM-5.
Key Takeaways:
➤ State-of-the-art intelligence at lower costs: Gemini 3.1 Pro Preview is leading 6 of the 10 evaluations that make up the Artificial Analysis Intelligence Index at less than half the cost to run of frontier peers from @OpenAI and @AnthropicAI. It obtains the highest score in Terminal-Bench Hard (agentic coding), AA-Omniscience (knowledge & hallucination), Humanity’s Last Exam (reasoning & knowledge), GPQA-Diamond (scientific reasoning), SciCode (coding) and CritPt (research-level physics). The CritPt score is particularly notable, scoring 18% on unpublished, research-level physics reasoning problems, over 5 p.p. above the next best model
➤ Improved real-world agentic performance, but not leading: Gemini 3.1 Pro Preview shows an improvement in GDPval-AA, our agentic evaluation focusing on real-world tasks, but is still not the leading model in this area. The model increases its ELO score over 100 points to 1316 (up from Gemini 3 Pro Preview), however still sits behind Claude Sonnet 4.6, Opus 4.6, GPT-5.2 (xhigh), and GLM-5
➤ Leading coding abilities: Gemini 3.1 Pro Preview leads the Artificial Analysis Coding Index, achieving the highest score in both Terminal-Bench Hard (54%) and SciCode (59%)
➤ Reduced hallucinations: Gemini 3.1 Pro Preview shows a major improvement in tendency to guess incorrectly when it doesn’t know the answer, reducing its AA-Omniscience hallucination rate by 38 p.p. from Gemini 3 Pro Preview
➤ Maintained token and cost efficiency: Gemini 3.1 Pro Preview improves without material increases in cost or token usage. It uses only ~2% more tokens to run the Artificial Analysis Intelligence Index than Gemini 3 Pro Preview, and keeps the same pricing ($2/$12 per 1M input/output tokens for ≤200k context). Its cost to run the Artificial Analysis Intelligence Index of $892 is less than half of frontier models such as Opus 4.6 (max) and GPT-5.2 (xhigh), though still ~2x the cost of leading open weights models such as GLM 5 ($547)
➤ Google takes top 3 spots in multi-modality: Gemini 3.1 Pro Preview ranks #1 on MMMU-Pro, our multimodal understanding and reasoning benchmark, ahead of Gemini 3 Pro Preview and Gemini 3 Flash, reinforcing Google’s leadership in multimodal reasoning
➤ Other model details: Gemini 3.1 Pro Preview retains the same 1 million token context window as its predecessor, and includes support for tool calling, structured outputs, and JSON mode -
Quoc Le reposted
koray kavukcuoglu
@koraykv
Today we’re releasing a preview of Gemini 3.1 Pro and making it available to our users and developers.
Very excited to bring the upgraded core we used in Deep Think to everyone.
Learn more about Gemini 3.1 Pro: blog.google/innovation-and-ai/models-and-research/gemini-…This media couldn't be displayed. See it on the original post. -
Quoc Le reposted
Wenhao Chai
@wenhaocha1
It’s over. Gemini 3 Deep Think achieved a 3300 rating on LiveCodeBench Pro almost surpassing all humans (99.99%) and is leading GPT-5.2 by a massive margin of 1000 points. Gemini is insanely strong!
Link:livecodebenchpro.com/projects/livecodebench-pro/leaderboard -
Quoc Le reposted
Thang Luong
@lmthang
Yes, we provided 3 things for AI-assisted math:
* Human-AI interaction (HAI) card (photo), inspired by model cards
* Full transcripts github.com/google-deepmind/superhuman/tree/main/aletheia
* A label for novelty-autonomy, inspired by SAE Levels of autonomy, see #aletheia paper arxiv.org/abs/2602.10177
Daniel Litt
@littmath
Really good question (note that DeepMind shared transcripts in their recent Aletheia paper, and I think this is clearly best practice). Hopefully OAI follows suit.
-
Quoc Le reposted
Jaana Dogan ヤナ ドガン
@rakyll
Gemini 3 Deep Think is quite good in generating high fidelity 3D representations of every day objects from still images. We've been using it to redesign on some of our home items like pots. It generates STLs too!
-
Quoc Le reposted
Ivan Fioravanti
@ivanfioravanti
Eulerian Fluid simulation test! Zero-shot!
Opus 4.6 vs GPT-5.3 vs Gemini 3 Deep Think!
My personal preference:
🥇 Gemini 3 Deep Think (really strong!)
🥈 Opus 4.6
🥉 GPT 5.3 HighThis media couldn't be displayed. See it on the original post. -
Quoc Le reposted
Markus J. Buehler
@ProfBuehlerMIT
Gave Gemini 3 Deep Think an image of a 3D spider web and asked for an interactive design tool. It generated a full design suite (procedural control, simulation, optimization) with STL export capability. I used it to engineer new metamaterials and a spider-web inspired bridge design, 3D printed it, then validated the structural integrity with a @nvidia DGX Spark load test. A mind-blowing example of the future of material and architecture design - image in, fabrication-ready design out! @GeminiApp
This media couldn't be displayed. See it on the original post. -
Quoc Le reposted
Zeno
@WAnzhi0810
Gemini 3 Deep Thinking 真的太牛逼了,对3D 的理解真的是最强的,然后再配合强大的前端能力,用 Three.js 还原出来具有结构性的模型。
这下一定程度上补齐了 3D 生成不擅长的结构化模型的问题!
我直接把它生成的模型打印出来,完全可用且没问题!太牛逼了! -
Quoc Le reposted
BijanBowen
@bijanbowen
Gemini 3 Deep Think generated a real-time 3D WiFi radar that maps every network around you as glowing nodes in a Matrix-style space — in one shot. It used Pearson correlation to infer which APs are physically close, since RSSI alone isn't enough.
This media couldn't be displayed. See it on the original post. -
Quoc Le reposted
Garrett Bingham
@gjb_ai
Gemini 3 Deep Think demo: ASCII fluid dynamics
-
Quoc Le reposted
Jeff Dean
@JeffDean
We’ve updated Gemini 3 Deep Think to better tackle the complexity of real-world research, science, and engineering. ♊ 🚀
It achieves gold-medal standards on the written portions of the Physics and Chemistry Olympiads, building on gold-level performance at IMO and ICPC and has SOTA results on ARC-AGI-2, MMMU-Pro, and HLE. 🏅
From optimizing code to validating scientific data, we’re seeing it become a genuine partner in the discovery process.
Read more here: blog.google/innovation-and-ai/models-and-research/gemini-… -
Quoc Le reposted
Derya Unutmaz, MD
@DeryaTR_
Gemini 3 Deep Think benchmarks look amazing! On Codeforces, it scored 3,455 Elo. Apparently, only 7 humans in the world have a higher coding Elo score!
A friend just sent me an output about a cancer mechanism that was so great that I am now resubscribing to Ultra for DT access! -
Quoc Le reposted
Demis Hassabis
@demishassabis
Thrilled to announce a big upgrade to Gemini 3 Deep Think that hits new records on the most rigorous benchmarks in maths, science & reasoning - including 84.6% on ARC-AGI-2, 48.4% Humanity’s Last Exam without tools, and 3455 Elo rating on Codeforces!
-
Quoc Le reposted
Garrett Bingham
@gjb_ai
Gemini 3 Deep Think demo: ray tracing, fully in-browser.
-
Quoc Le reposted
Sundar Pichai
@sundarpichai
Gemini 3 Deep Think is getting a significant upgrade. We’ve refined Deep Think in close partnership with scientists and researchers to tackle tough, real-world challenges.
And it’s pushing the frontier across the most challenging benchmarks, achieving an unprecedented 84.6% on ARC-AGI-2. It also sets a new standard on Humanity’s Last Exam - 48.4% without tools. -
Quoc Le reposted
ARC Prize
@arcprize
Gemini 3 Deep Think (2/26) Semi Private Eval
- ARC-AGI-1: 96.0%, $7.17/task
- ARC-AGI-2: 84.6% $13.62/task
New ARC-AGI SOTA model from @GoogleDeepMind -
Quoc Le reposted
Google Gemini
@GeminiApp
Today, we’re releasing a significant upgrade to our specialized reasoning mode, Gemini 3 Deep Think.
Deep Think is built to drive practical applications, enabling researchers to interpret complex data and engineers to model physical systems through code.
With the updated Deep Think, you can turn a sketch into a 3D-printable reality. Deep Think analyzes the drawing, builds the complex shape, and generates a file so you can create the physical object with 3D printing.
This is rolling out now to Google AI Ultra subscribers. Select the "Deep Think" option in the tools menu to get started.
Learn more here: goo.gle/3MoiifFThis media couldn't be displayed. See it on the original post. -
Quoc Le reposted
Noam Shazeer
@NoamShazeer
An updated Gemini 3 Deep Think is out today:
📈 Achieves SOTA on ARC-AGI-2, MMMU-Pro, and HLE.
🥇Gold-medal level on Physics & Chemistry Olympiads.
It turns out the best way to solve hard problems is still to think about them. Read more: blog.google/innovation-and-ai/models-and-research/gemini-… -
Quoc Le reposted
Google DeepMind
@GoogleDeepMind
We’ve upgraded our specialized reasoning mode Gemini 3 Deep Think to help solve modern science, research, and engineering challenges – pushing the frontier of intelligence. 🧠
Watch how the Wang Lab at Duke University is using it to design new semiconductor materials. 🧵This media couldn't be displayed. See it on the original post. -
Quoc Le reposted
Ed H. Chi
@edchi
Thanks for this recognition, Mike.
/cc @denny_zhou @quocleix
Mike Knoop
@mikeknoop
The original CoT paper ("lets think step by step") from Jan 2022 is equally important as the Transformer ("attention is all you need"). arxiv.org/abs/2201.11903
-
Quoc Le
@quocleix
Read our latest blog post on how Gemini Deep Think is advancing mathematical and scientific research.
goo.gle/4aGs3Pz
Quoc Le
@quocleix
The first paper in the blog describes Aletheia, an agent based on Gemini Deep Think. Aletheia moves beyond Olympiad problem-solving to tackle PhD-level mathematics. It iteratively generates and verifies proofs, successfully collaborating on human-AI research, autonomously generating a paper on eigenweights, and solving open Erdős conjectures! 🤯 Link to paper: github.com/google-deepmind/superhuman/blob/main/aletheia/…
Quoc Le
@quocleix
The second paper show how researchers use "vibe-proving" with Gemini Deep Think to solve open problems, refute long-standing conjectures, and discover obscure mathematical links across TCS, physics, and more. Link to paper: arxiv.org/abs/2602.03837
-
Quoc Le reposted
Thang Luong
@lmthang
6 months in, after the IMO-gold achievement, I’m very excited to share another important milestone: AI can help accelerate knowledge discovery in mathematics, physics, and computer science! We’re sharing Two new papers from @GoogleDeepMind and @GoogleResearch that explore how Gemini #deepthink together with agentic workflows can empower mathematicians and scientists to tackle professional research problems. Some highlights:
The first paper built a research agent #aletheia, powered by an advanced version of Gemini Deep Think, that can autonomously produce publishable math research and crack open Erdős problems.
The second paper, built on similar agentic reasoning ideas, helped resolve bottlenecks in 18 research problems, across algorithms, ML and combinatorial optimization, information theory and economics.
See the thread for details about the two papers and the joint blog post.
Thang Luong
@lmthang
Very excited to share that an advanced version of Gemini Deep Think is the first to have achieved gold-medal level in the International Mathematical Olympiad! 🏆, solving five out of six problems perfectly, as verified by the IMO organizers! It’s been a wild run to lead this effort and I am grateful to everyone in the team for such an amazing achievement! Blog post in the thread and more to share soon!
-
Quoc Le reposted
Tao Tu
@taotu831
Excited to share our latest research published today in @NatureMedicine, demonstrating how Large Language Models (LLMs) can help bridge the critical shortage of subspecialist medical expertise. www.nature.com/articles/s41591-025-04190-9
-
Quoc Le reposted
Demis Hassabis
@demishassabis
The AI field is in need of harder benchmarks to test capabilities of the latest AI models. This update to @Kaggle Game Arena with werewolf and poker (heads-up) plus chess, gives us new objective measures of real-world skills like planning and decision making under uncertainty.
-
Quoc Le
@quocleix
Excited to share our latest work: "Semi-Autonomous Mathematics Discovery with Gemini." We used Gemini to systematically evaluate 700 "open" conjectures in the Erdős Problems database.
The result? We addressed 13 problems marked as open—finding 5 novel autonomous solutions and identifying 8 existing solutions missed by previous literature.
Read the full case study here: arxiv.org/abs/2601.22401 -
Quoc Le reposted
Thang Luong
@lmthang
Thanks to @AwaitSeedIgnite for such a cute comic about the cool collaboration btw mathematicians and #deepthink to solve Erdős 1051! cc @AcerFur
Thang Luong
@lmthang
There has been so much noise on AI for Math research. We have been working on research-level math for over a year (in parallel with our IMO Olympiad math effort) and obtained many results including solving Erdős problems (and beyond!). We haven't shared much in the past yet as we want to do things responsibly with respect to the math community. We're almost done! Yesterday, we released the first paper in our series: Solving a generalized version of Erdős-1051 problem! More to come! #deepthink
-
Quoc Le reposted
Thang Luong
@lmthang
There has been so much noise on AI for Math research. We have been working on research-level math for over a year (in parallel with our IMO Olympiad math effort) and obtained many results including solving Erdős problems (and beyond!). We haven't shared much in the past yet as we want to do things responsibly with respect to the math community. We're almost done! Yesterday, we released the first paper in our series: Solving a generalized version of Erdős-1051 problem! More to come! #deepthink
-
Quoc Le reposted
Jeff Dean
@JeffDean
We’ve pushed out the Pareto frontier of efficiency vs. intelligence again.
With Gemini 3 Flash ⚡️, we are seeing reasoning capabilities previously reserved for our largest models, now running at Flash-level latency. This opens up entirely new categories of near real-time applications that require complex thought.
It’s available in the API, and rolling out today as the default model in AI Mode in Search and Gemini app globally.
Read more on the blog at: blog.google/products-and-platforms/products/gemini/gemini…
More in thread ⬇️ -
Quoc Le reposted
Demis Hassabis
@demishassabis
I feel the need... the need for speed! ⚡️⚡️⚡️
-
Quoc Le reposted
koray kavukcuoglu
@koraykv
Tomorrow comes in a flash. ⚡⚡⚡
-
Quoc Le reposted
Jeff Dean
@JeffDean
This was a really interesting experiment, and it will only get better as model capabilities improve.
Some quotes from authors who participated:
Google Research
@GoogleResearch
Our new experiment uses an advanced version of Gemini 2.5 Deep Think to rigorously verify theoretical computer science papers. 97% of trial participants, authors for #stoc2026, found the feedback helpful for catching errors & improving clarity. More at: goo.gle/3MXslYP
-
Quoc Le
@quocleix
Gemini 3 Deep Think mode is now live in the Gemini app for Ultra users. 🚀
Building on the technology that reached a gold-medal level at the ICPC World Finals & IMO, it uses parallel thinking to excel at difficult coding and scientific tasks. blog.google/products-and-platforms/products/gemini/gemini…
Quoc Le
@quocleix
See Deep Think’s creative coding in action. 🎨
We asked it to design a fractal explosion in HTML. By using extended thinking to strategize on recursion and aesthetics before coding, it turns a simple prompt into sophisticated digital art.This media couldn't be displayed. See it on the original post.
Quoc Le
@quocleix
Benchmark scores:
-
Quoc Le
@quocleix
Excited with this new team. Consider joining us in Singapore (and swim in Marina Bay with AGI hippos):
Yi Tay
@YiTayML
Happy to share that the @GoogleDeepMind Gemini team is starting a new research team in Singapore!
This new team will be focused on advanced reasoning, LLM/RL and improving bleeding edge SOTA models such as Gemini, Gemini Deep Think and beyond. 🔥
This team will be led by yours truly and reports up to Quoc Le (@quocleix)'s broader team in Mountain View which was recently in the center of both IMO gold medal and ICPC gold medal breakthroughs with Gemini Deep Think, amongst many other significant Gemini advancements. 🚀
We’re starting out with a very small but intensely capable force because talent density is key over anything else in the LLM era. Over the past few months, we have gone around and gathered the best of the best talent (in the region and beyond) and I’m confident we’ll have a super cracked team very soon.
If you are interested in joining and have made truly exceptional contributions in any domain or area, (engineering and/or research etc) please contact me.
This is quite an exciting time, with the Gemini / GenAI team at Google Deepmind leading the charge at the frontier. This is also the best opportunity to be on the critical path to AGI from the sunny island of Singapore. 🏝️
Many thanks to leadership support from @quocleix @JeffDean @benoitschilling, @EugenieRives and @demishassabis for the support of this team.
Wonderful and fun image generated by Nano Banana 👇 -
Quoc Le reposted
Adams Wei Yu
@AdamsYu
In our latest work, we show autoregressive pixel modeling scales predictably, but is massively compute-hungry compared to text LLMs. But with 4-5x annual compute growth, raw pixel modeling will become feasible in ~5 years. So see you in 2030!
Xinchen Yan
@skywalkeryxc
For years, RAW pixel space pretraining has been sidelined: too compute-expensive.
Our new @GoogleDeepMind paper 📜 dives into the scaling trends of raw pixel models to answer the question “how far are we from scaling up next-pixel prediction?”
arxiv.org/pdf/2511.08704
Forecast: Raw next-pixel modeling will reach competitive ImageNet classification (>80% top1 accuracy) and generation metrics (90 Fr’echet Distance) in five years!
Threads 👇 -
Quoc Le reposted
Xinchen Yan
@skywalkeryxc
For years, RAW pixel space pretraining has been sidelined: too compute-expensive.
Our new @GoogleDeepMind paper 📜 dives into the scaling trends of raw pixel models to answer the question “how far are we from scaling up next-pixel prediction?”
arxiv.org/pdf/2511.08704
Forecast: Raw next-pixel modeling will reach competitive ImageNet classification (>80% top1 accuracy) and generation metrics (90 Fr’echet Distance) in five years!
Threads 👇 -
Quoc Le reposted
Ankesh Anand
@ankesh_anand
Gemini 3 Pro is the new sota on FrontierMath, importantly also improving on Tier 4 (research-level math)!
-
Quoc Le
@quocleix
Nano Banana Pro is out and its text understanding is amazing🍌
Asked it to create an infographic for the sequence to sequence paper and it did this: -
Quoc Le reposted
koray kavukcuoglu
@koraykv
Gemini 3 Pro 🤝 Nano Banana Pro
New SOTA image generation and editing.
🍌Production-ready visuals with improved precision and control
🍌Images grounded in Gemini’s real world understanding
🍌Superior text rendering
🍌Localization in multiple languages
🍌Physically accurate lighting and dynamics.
And character consistency across 14 (fluffy) inputs!
blog.google/technology/ai/nano-banana-pro -
Quoc Le reposted
Dan Hendrycks
@hendrycks
Just how significant is the jump with Gemini 3?
We just released a new leaderboard to track AI developments.
Gemini 3 is the largest leap in a long time. -
Quoc Le reposted
Kilian Lieret
@KLieret
Gemini 3 Pro sets new record on SWE-bench verified: 74%! (evaluated with minimal agent) Costs are 1.6x of GPT-5, but still cheaper than Sonnet 4.5. Gemini iterates longer than everyone; run your agent with a step limit of >100 for max performance. Details & full agent logs in 🧵
-
Quoc Le reposted
Shunyu Yao
@ShunyuYao14
Try Gemini 3!
my collaborator @vihaniaj has created a new personal webpage: alfredyao.github.io for me in minutes with one-shot prompting gemini 3. Almost no factual error, and in contrast, look at the old webpage, which is what I can do in hours :)alfredyao.github.io/posts/2025-11-18.html -
Quoc Le reposted
Logan Kilpatrick
@OfficialLoganK
And say hello to Gemini 3 Deep Think, even more SOTA compared to Gemini 3 Pro 🤯
-
Quoc Le
@quocleix
Gemini 3 Deep Think is next level. Deep Think was the the engine behind our gold medal-level wins at IMO and ICPC, and now powers an even stronger version of Gemini 3. SOTA above SOTA. More to come soon!
Quoc Le
@quocleix
ARC-AGI-2 at 45.1% is 🔥
ARC Prize
@arcprize
Gemini 3 models from @Google @GoogleDeepMind have made a significant 2X SOTA jump on ARC-AGI-2 (Semi-Private Eval)
Gemini 3 Pro:
31.11%, $0.81/task
Gemini 3 Deep Think (Preview):
45.14%, $77.16/task
Quoc Le
@quocleix
To feel the AGI, I prompted Gemini 3 Deep Think "Create an interactive analog film photography experience, all within a self-contained HTML file. I need to be able to look through the viewfinder, upload an image, have it display there, and then "take a photo" after which point it goes away from the viewfinder and into my gallery. I should be able to adjust my settings and choose a film stock, which will affect the photo style accordingly. The gallery should be of cute overlapping polaroid style prints that are slightly magnified when I hover over each one. Above all else, focus on nuanced creativity, immaculate vibes, and high fidelity. You can do this!!" and it created this amazing demo:
This media couldn't be displayed. See it on the original post. -
Quoc Le reposted
Oriol Vinyals
@OriolVinyalsML
The secret behind Gemini 3?
Simple: Improving pre-training & post-training 🤯
Pre-training: Contra the popular belief that scaling is over—which we discussed in our NeurIPS '25 talk with @ilyasut and @quocleix—the team delivered a drastic jump. The delta between 2.5 and 3.0 is as big as we've ever seen. No walls in sight!
Post-training: Still a total greenfield. There's lots of room for algorithmic progress and improvement, and 3.0 hasn't been an exception, thanks to our stellar team.
Congratulations to the whole team 💙💙💙 -
Quoc Le
@quocleix
Gemini 3 is finally out. 🚀
The numbers on the hardest benchmarks are wild. Seeing MathArena Apex go from <2% to 23.4% and ARC-AGI-2 hit 31% feels like a real turning point for reasoning. We're starting to crack problems that used to look impossible.
Huge congrats to the team.
Quoc Le
@quocleix
I've also been using Gemini 3 for fun and its ability to reason with internal knowledge is so good. For example, I gave it the prompt "Visualize Neural Architecture Search with Reinforcement Learning with p5.js" and it autonomously determined the components of the system, developed the code to generate the animation, and explained the concept in a way that's clear and accessible.
-
Quoc Le reposted
Thang Luong
@lmthang
#gemini3 is finally out! Congrats to everyone on this amazing launch! Also very excited to see how #deepthink can power Gemini3 to further the state-of-the-art performances across reasoning, deep knowledge, and multimodality: 41% HLE, 93.8% GPQA, & 45.1% on ARC-AGI-2 (big jump)!
Google DeepMind
@GoogleDeepMind
This is Gemini 3: our most intelligent model that helps you learn, build and plan anything.
It comes with state-of-the-art reasoning capabilities, world-leading multimodal understanding, and enables new agentic coding experiences. 🧵This media couldn't be displayed. See it on the original post. -
Quoc Le reposted
koray kavukcuoglu
@koraykv
Deep Think is especially powerful for complex reasoning – we used it at ICPC and IMO and now with Gemini 3!
Logan Kilpatrick
@OfficialLoganK
And say hello to Gemini 3 Deep Think, even more SOTA compared to Gemini 3 Pro 🤯
-
Quoc Le reposted
Demis Hassabis
@demishassabis
Awesome to see Veo 3.1 top the LMArena video leaderboards by a large distance with big improvements over Veo 3.0 for text-to-video (+30) and image-to-video (+70)! 🔥Huge congrats to the team! Try it for yourself in flow.google and the @GeminiApp
Arena.ai
@arena
🚨🎬 Big news from Video Arena!
@GoogleDeepMind’s latest Veo 3.1 now ranks #1 in both Text-to-Video and Image-to-Video leaderboards. 🏆
This is a +30-point leap from Veo 3.0 → 3.1, making it the first model to break 1400 in Video Arena history!
Huge congrats to the @GoogleDeepMind team for pushing the frontier of video generation forward!
More details in the thread 🧵 -
Quoc Le reposted
Arena.ai
@arena
🚨🎬 Big news from Video Arena!
@GoogleDeepMind’s latest Veo 3.1 now ranks #1 in both Text-to-Video and Image-to-Video leaderboards. 🏆
This is a +30-point leap from Veo 3.0 → 3.1, making it the first model to break 1400 in Video Arena history!
Huge congrats to the @GoogleDeepMind team for pushing the frontier of video generation forward!
More details in the thread 🧵 -
Quoc Le reposted
koray kavukcuoglu
@koraykv
Exciting update to Veo with better control, consistency and more granular control over your final scene with enhanced realism.
Available on Flow, Gemini App, AI Studio and Vertex AI!
Google DeepMind
@GoogleDeepMind
Veo is getting a major upgrade. 🚀
We’re rolling out Veo 3.1, our updated video generation model, alongside improved creative controls for filmmakers, storytellers, and developers - many of them with audio. 🧵This media couldn't be displayed. See it on the original post. -
Quoc Le
@quocleix
Deep Think achieves SoTA performance on FrontierMath:
Epoch AI
@EpochAIResearch
We evaluated Gemini 2.5 Deep Think on FrontierMath. There is no API, so we ran it manually. The results: a new record!
We also conducted a more holistic evaluation of its math capabilities. 🧵 -
Quoc Le reposted
Josh Woodward
@joshwoodward
🍌 @GeminiApp just passed 5 billion images in less than a month. What a ride, still going!
Latest trend: retro selfies of you holding a baby version of you. Can't make this stuff up! -
Quoc Le
@quocleix
(1/3) Thrilled to announce a new Gemini breakthrough! Building on our success at IMO this year, an advanced version of Gemini Deep Think achieved gold-medal level performance at the ICPC 2025 World Finals - one of the world’s leading competitive programming competitions.
Quoc Le
@quocleix
(2/3) Our model solved 10 out of 12 problems to achieve gold medal level. We were able to achieve this through breakthroughs in parallel thoughts, multi-step reasoning, and novel reinforcement learning techniques. You can find Gemini's solutions here: github.com/google-deepmind/gemini_icpc2025
Quoc Le
@quocleix
(3/3) Beyond coding capabilities, this achievement shows our models can truly reason and solve complex problems in new ways, across domains. So proud of the team!
-
Quoc Le reposted
Josh Woodward
@joshwoodward
It’s a full-on stampede to use the @GeminiApp, the team is doing heroics to keep the system up and running, had to use temporary limits to manage the peak demand earlier.
We're preparing for tomorrow’s peak and doing everything to make sure everyone can join in on the fun! 🍌🔥 -
Quoc Le reposted
Demis Hassabis
@demishassabis
Made it to no.1 in the App Store. Congrats to the @GeminiApp team for all their hard work, and this is just the start, so much more to come!
Rihard Jarc
@RihardJarc
Wow, $GOOGL Gemini has overtaken ChatGPT in top downloads in the US on iOS.
Something to keep an eye on as ChatGPT has dominated the standings for months now.
$GOOGL execution and product shipment showing results… -
Quoc Le reposted
Yi Tay
@YiTayML
Had a really wonderful time hosting @JeffDean, @quocleix, @benoitschilling and @denny_zhou in Singapore for the @GoogleDeepMind Gemini Singapore 🇸🇬 event last week! 🔥
The event went super well imo, the vibes were on-point and an overwhelming number of people told me directly they were moved and very inspired! 😁
Moreover, the consensus amongst GDM speakers was also that the level of questions from the community were solid and reassuring that something good is going on here. Basically, signal to noise ratio was generally pretty good.
As I also said in my opening remarks, I think Singapore has a lot of raw pure talent, but today it is physically and spiritually far away from the true AI frontier for a myriad of reasons. Hence, increasing "frontier awareness" and "aura diffusion" (aka ecosystem building 😂) was something particularly high ROI (for us too!)
The way I see it is that SG is like a reasonably strong base pretrained model that hasn't been properly post-trained yet. It feels like the gears have started turn though, hopefully we'll make it happen. 🤞
We also had a lot of fun roaming SG. Visiting gardens by the bay on day-0, having 2-michelin star "tarvar" as the finale dinner, pickleball with Jeff on top of a random building, spotting anteaters at the night safari and last but not least, chatting with senior minister Lee Hsien Loong and visiting the Istana.
Was an epic week! 😁
More details of the 3/9/2025 event in the thread below 👇 -
Quoc Le reposted
Demis Hassabis
@demishassabis
strange object spotted under the microscope over the weekend in the lab...
-
Quoc Le reposted
Logan Kilpatrick
@OfficialLoganK
Good news: rate limits have doubled for Deep Think in the @GeminiApp
Happy prompting : ) -
Quoc Le reposted
Josh Woodward
@joshwoodward
Great news for @GoogleAI Ultra members on @GeminiApp:
⬆️ We just doubled the 2.5 Deep Think daily limit
⬆️ Any errors on our side don't count against your limit -
Quoc Le reposted
Demis Hassabis
@demishassabis
One word: relentless. just in the past two weeks, we’ve shipped:
🌐 Genie 3 - the most advanced world simulator ever
🤔 Gemini 2.5 Pro Deep Think available to Ultra subs
🎓 Gemini Pro free for uni students & $1B for US ed
🌍 AlphaEarth - a geospatial model of the entire planet
🏛️ Aeneas - deciphering ancient text (in @Nature)
🥇 Gemini gold-medal level at the IMO
🧸 Storybook - books w/art & audio @GeminiApp
♛ New @Kaggle Game Arena benchmark for LLMs
🐙 Jules, our asynchronous coding agent, out of Beta
🇬🇧 AI Mode for Search available in the UK
📔 NotebookLM Video Overviews
🔥 Gemma passed 200m downloads
Now you know why I don't get much sleep 🛌 - too busy pushing the frontier! -
Quoc Le reposted
Demis Hassabis
@demishassabis
Thrilled to announce the @Kaggle Game Arena, a new leaderboard testing how modern LLMs perform on games (spoiler: not very well atm!). AI systems play each other, making it an objective & evergreen benchmark that will scale in difficulty as they improve.
www.kaggle.com/game-arena -
Quoc Le reposted
Jasper Dekoninck
@j_dekoninck
We launched a new competition on MathArena: Evaluation on the International Mathematics Competition for University students!
Our goal: Verify the gold medals on the IMO by testing agentic Gemini-2.5-Pro and Gemini IMO Deep Think
The results: The models aced the competition. 🧵 -
Quoc Le reposted
Taelin
@VictorTaelin
For the first time ever, an AI has managed to derive a generic "foldr" function for N-tuples in the λ-Calculus. It was Gemini Pro 2.5, with Deep Think. All other models fail to do so, including o3 and Grok 4. Nice progress @GoogleDeepMind
-
Quoc Le reposted
🚨 AI News | TestingCatalog
@testingcatalog
A classic "Pelikan riding a bicycle SVG" test 👀
GPT-5* vs Gemini Deep Think IMO
Chetaslua
@chetaslua
🚨 Breaking News 🚨
GPT-5 is already (ostensibly) available via API
Using the model `gpt-5-bench-chatcompletions-gpt41-api-ev3` via the Chat Completions API will give you what is supposedly GPT-5.
Conjecture: The "gpt41-api" portion of the name suggests that there's new functionality to this model that will require new API parameters or calls.
Source - Reddit u/segin -
Quoc Le reposted
🚨 AI News | TestingCatalog
@testingcatalog
The first time I've played an AI-generated game for more than 10 minutes. 10x speed, made by Gemini Deep Think IMO.
Why would you even need to code anything if LLMs will be able to one-shot almost any coding tasks?
The future is near 🤖This media couldn't be displayed. See it on the original post.
🚨 AI News | TestingCatalog
@testingcatalog
Documented 🗞️
www.testingcatalog.com/early-preview-of-olympiad-winning-… -
Quoc Le reposted
Ethan Mollick
@emollick
I prompted Gemini 2.5 Deep Think: "create a missile command game that incorporates relativity in realistic ways but is still playable." I then asked it to improve the design a few times.
The full design & all code & calculations came from the AI, each version ran without errors. -
Quoc Le reposted
Demis Hassabis
@demishassabis
Gemini 2.5 Deep Think is state-of-the-art performance across many challenging benchmarks!
Google DeepMind
@GoogleDeepMind
Compared to other models without tool use, it achieves state-of-the-art performance across:
🔘 LiveCodeBench V6, which evaluates competitive code performance
🔘 Humanity’s Last Exam, a challenging benchmark that measures a model’s expertise in different domains, including science and math -
Quoc Le reposted
Simon Willison
@simonw
My notes on Google Deep Think - I don't have a $250/month Ultra account but nickandbro on Hacker News got it to draw a pelican riding a bicycle and the bird actually is recognizable as a pelican! simonwillison.net/2025/Aug/1/deep-think-in-the-gemini-app/
-
Quoc Le reposted
Pushmeet Kohli
@pushmeet
Releasing a model with this kind of performance is an impressive milestone in AI. Shows the power and extensibility of the thinking paradigm and the power of reasoning/maths as a problem. Great work by a super team!
Google DeepMind
@GoogleDeepMind
For researchers, scientists, and academics tackling hard problems: Gemini 2.5 Deep Think is here. 🤯
It doesn't just answer, it brainstorms using parallel thinking and reinforcement learning techniques. We put it into the hands of mathematicians who explored what it can do ↓This media couldn't be displayed. See it on the original post. -
Quoc Le
@quocleix
Following its IMO gold-level win, @GoogleDeepMind is sharing Gemini Deep Think with mathematicians for feedback. Excited to see what they discover! 🧠
Plus, an updated Gemini 2.5 Deep Think is now rolling out for Google AI Ultra subscribers.
Learn more: blog.google/products-and-platforms/products/gemini/gemini…
Quoc Le
@quocleix
General benchmark scores are very strong. Maybe AGI is already here :-)
-
Quoc Le reposted
Ethan Mollick
@emollick
Had early access to Gemini with Deep Think. Very good model, big gains over standard Gemini 2.5 Pro for a lot of problems.
Here is the first attempt at the starship control panel prompt I try with every model. First time I have seen a model make a 3D interface in response.
Ethan Mollick
@emollick
Kinda amazing: the mystery model "summit" with the prompt "create something I can paste into p5js that will startle me with its cleverness in creating something that invokes the control panel of a starship in the distant future" & "make it better"
2,351 lines of code. First timeThis media couldn't be displayed. See it on the original post. -
Quoc Le reposted
Demis Hassabis
@demishassabis
Gemini 2.5 Deep Think now available for Ultra subscribers! Great at tackling problems that require creativity & planning, it finds the best answer by considering, revising & combining many ideas at once. A faster variation of the model that just achieved IMO gold-level. Enjoy!
Google DeepMind
@GoogleDeepMind
For researchers, scientists, and academics tackling hard problems: Gemini 2.5 Deep Think is here. 🤯
It doesn't just answer, it brainstorms using parallel thinking and reinforcement learning techniques. We put it into the hands of mathematicians who explored what it can do ↓This media couldn't be displayed. See it on the original post. -
Quoc Le reposted
Pushmeet Kohli
@pushmeet
AI will help us understand our home - Earth.
🌍Introducing AlphaEarth Foundations - an AI model to see our planet in a whole new way. By integrating public data from satellite images, radar, and more, it creates highly detailed and compact ‘embeddings’ that represent the planet. -
Quoc Le reposted
Thang Luong
@lmthang
Right before #imo2025, together with colleagues from Mountain View, NYC, Singapore, etc, we all gathered at @GoogleDeepMind headquarter in London for our final push for IMO. I believe that week was when all magic happened!
We put all individual recipes (that we figured out before) together and did a yolo run (with the compute that I had to beg various groups to loan) to train our most advanced Gemini model. We finished training 2 days before IMO :D That model achieved SOTA results, not just for math, but coding along with other reasoning tasks, unbelievable!
That week was also when we figured all details to scale our Deep Think mode, alongside with other enhanced inference strategies. We finalized our runbooks that predetermined the configs to be used during the IMO days (so there will be no human intervention at all). And the rest is history.
Thanks @CadeMetz and @nytimes for featuring that historic moment
www.nytimes.com/2025/07/21/technology/google-ai-internati…
The photo below was taken in London during that magical week. Tagging various team members in random order who worked extremely for that moment @YiTayML @theophaneweber @DawsenHwang @JiemingMao @jon_lee0 @zichengxu42 @vinayramasesh @mirrokni @blackhc @gjb_ai @g01na2 @LeiYu63 @NateKushman @jj_at_brown @quocleix @freeetext together with many others that I don't yet have X handles.
Thang Luong
@lmthang
Very excited to share that an advanced version of Gemini Deep Think is the first to have achieved gold-medal level in the International Mathematical Olympiad! 🏆, solving five out of six problems perfectly, as verified by the IMO organizers! It’s been a wild run to lead this effort and I am grateful to everyone in the team for such an amazing achievement! Blog post in the thread and more to share soon!
-
Quoc Le reposted
Auriel
@aurielws
Come work with us on helping agents reach unprecedented levels of autonomy :)
We have a ridiculously cracked team and are pushing the bounds of AGI with awesome folks like @robertarail @OriolVinyalsML @_rockt @quocleix
Come join ~~ le rocket ship ~~
Roberta Raileanu
@robertarail
I’m building a new team at @GoogleDeepMind to work on Open-Ended Discovery!
We’re looking for strong Research Scientists and Research Engineers to help us push the frontier of autonomously discovering novel artifacts such as new knowledge, capabilities, or algorithms, in an open-ended self-improving loop.
We aim to work on ambitious research projects in a fast-paced manner.
If this sounds appealing to you, apply using the link below by Friday, August 1st EOD:
job-boards.greenhouse.io/deepmind/jobs/7093349 -
Quoc Le reposted
Yi Tay
@YiTayML
Watching AI progress is like watching your child grow up. You don't notice the little day to day changes but all of a sudden after some period of time you look back and you become amazed and shocked by how much have changed. 😮
Every minor revision to the model, every incremental algorithmic upgrade, every new dataset etc may seem small and inconsequential but it's crazy how much progress stacks. 📈
Last year Gemini was barely able to perform at the IMO. But this year it managed to get gold 🥇. Just 12 months!! That's not a lot of time! I think this progress is pretty fascinating and eerie. The eerie part is that even as AI researchers we underestimate how fast or how good a job we're doing in general!
Crazy times but the true frontier of AI is very exciting and enchanting at the same time. 😎🧙 -
Quoc Le
@quocleix
Excited to share that a scaled up version of Gemini DeepThink achieves gold-medal standard at the International Mathematical Olympiad. This result is official, and certified by the IMO organizers.
Watch out this space, more to come soon!
deepmind.google/discover/blog/advanced-version-of-gemini-…Post not available here, view on X
Quoc Le
@quocleix
We got the gold-medal results last Friday but we wanted the the young participants to have fun and celebrated. That's why we delayed the announcement and didn't scramble to announce unofficial results on twitter on the weekend like another lab :-)
Quoc Le
@quocleix
A version of this model will soon be available to Gemini users. Benchmark numbers, not just math, are generally very good. Be excited :-)
-
Quoc Le reposted
Melvin Johnson
@melvinjohnsonp
So happy to see this incredible achievement.
Huge congrats to @lmthang, @quocleix, @YiTayML and the IMO team on the result.
This was a great collaboration across teams to build a general Gemini DeepThink model that can also get gold at IMO. -
Quoc Le reposted
Demis Hassabis
@demishassabis
Official results are in - Gemini achieved gold-medal level in the International Mathematical Olympiad! 🏆 An advanced version was able to solve 5 out of 6 problems. Incredible progress - huge congrats to @lmthang and the team! deepmind.google/discover/blog/advanced-version-of-gemini-…
-
Quoc Le reposted
koray kavukcuoglu
@koraykv
Advanced version of Gemini Deep Think (announced at #googleio) using parallel inference time computation achieved gold-medal performance at IMO, solving 5/6 problems with rigorous proofs as verified by official IMO judges! Congrats to all involved!
deepmind.google/discover/blog/advanced-version-of-gemini-… -
Quoc Le reposted
Thang Luong
@lmthang
Very excited to share that an advanced version of Gemini Deep Think is the first to have achieved gold-medal level in the International Mathematical Olympiad! 🏆, solving five out of six problems perfectly, as verified by the IMO organizers! It’s been a wild run to lead this effort and I am grateful to everyone in the team for such an amazing achievement! Blog post in the thread and more to share soon!
-
Quoc Le reposted
AshutoshShrivastava
@ai_for_success
We can't even be sure OpenAI's "gold medal" is legit.
OpenAI stole the spotlight and overshadowed the kids who actually won.
As per some leaks, DeepMind has won gold too but hasn’t said a word yet. Looks like they actually respected what the IMO asked.
But as usual, OpenAI will do anything for hype, I guess.
No official score. No patience. No shame.
Mikhail Samin
@Mihonarium
🚨 According to a friend, the IMO asked AI companies not to steal the spotlight from kids and to wait a week after the closing ceremony to announce results. OpenAI announced the results BEFORE the closing ceremony.
According to a Coordinator on Problem 6, the one problem OpenAI couldn't solve, "the general sense of the IMO Jury and Coordinators is that it was rude and inappropriate" for OpenAI to do this.
OpenAI wasn't one of the AI companies that cooperated with the IMO on testing their models, so unlike the likely upcoming Google DeepMind results, we can't even be sure OpenAI's "gold medal" is legit. Still, the IMO organizers directly asked OpenAI not to announce their results immediately after the olympiad.
Sadly, OpenAI desires hype and clout a lot more than it cares about letting these incredibly smart kids celebrate their achievement, and so they announced the results yesterday. -
Quoc Le reposted
Thang Luong
@lmthang
Yes, there is an official marking guideline from the IMO organizers which is not available externally. Without the evaluation based on that guideline, no medal claim can be made. With one point deducted, it is a Silver, not Gold.
Mikhail Samin
@Mihonarium
🚨 According to a friend, the IMO asked AI companies not to steal the spotlight from kids and to wait a week after the closing ceremony to announce results. OpenAI announced the results BEFORE the closing ceremony.
According to a Coordinator on Problem 6, the one problem OpenAI couldn't solve, "the general sense of the IMO Jury and Coordinators is that it was rude and inappropriate" for OpenAI to do this.
OpenAI wasn't one of the AI companies that cooperated with the IMO on testing their models, so unlike the likely upcoming Google DeepMind results, we can't even be sure OpenAI's "gold medal" is legit. Still, the IMO organizers directly asked OpenAI not to announce their results immediately after the olympiad.
Sadly, OpenAI desires hype and clout a lot more than it cares about letting these incredibly smart kids celebrate their achievement, and so they announced the results yesterday. -
Quoc Le reposted
Jasper
@zjasper
We might be heading into a plot twist in the OpenAI vs. DeepMind IMO saga.
Just saw a post from Joseph Myers (involved in the Math Olympiad since 1992): the IMO committee reportedly asked AI labs not to publish results until 7 days after the closing ceremony — out of respect for human contestants (see my post yesterday) and likely to allow time for proper verification of AI submissions and formats.
According to Joseph, OpenAI didn’t collaborate with the IMO to test their model, and none of the 91 official IMO coordinators were involved in grading its solutions. Meanwhile, it seems DeepMind is following the rules and patiently waiting their turn.
For context:
The IMO has 6 problems, each worth 7 points. This year’s gold cutoff is 35 points. Even a small deduction could knock OpenAI down to silver. And from my read of their writeups, some parts might raise questions — and possibly cost points.
Terence Tao also pointed out that while the problems stay the same, testing formats matter. A student who wouldn’t get a bronze under standard conditions might strike gold with a modified setup — which raises real questions about what “solving the IMO” means for AI.
Next week might get spicy. Stay tuned. -
Quoc Le reposted
Mikhail Samin
@Mihonarium
🚨 According to a friend, the IMO asked AI companies not to steal the spotlight from kids and to wait a week after the closing ceremony to announce results. OpenAI announced the results BEFORE the closing ceremony.
According to a Coordinator on Problem 6, the one problem OpenAI couldn't solve, "the general sense of the IMO Jury and Coordinators is that it was rude and inappropriate" for OpenAI to do this.
OpenAI wasn't one of the AI companies that cooperated with the IMO on testing their models, so unlike the likely upcoming Google DeepMind results, we can't even be sure OpenAI's "gold medal" is legit. Still, the IMO organizers directly asked OpenAI not to announce their results immediately after the olympiad.
Sadly, OpenAI desires hype and clout a lot more than it cares about letting these incredibly smart kids celebrate their achievement, and so they announced the results yesterday. -
Quoc Le reposted
Andrew M. Dai
@AndrewDai
'Semi-supervised sequence learning' is also 10 years old too. Great to see language model pretraining and supervised finetuning has since been working out for some folks. arxiv.org/abs/1511.01432 w/ @quocleix (Work started off as an accident).
Oriol Vinyals
@OriolVinyalsML
"A Neural Conversational Model" is 10 years old, w/ @quocleix . TL;DR you can train a chatbot with a large neural network (~500M params!). Samples 👇
This paper was received with mixed reviews, but I'm glad all the critics are now riding the LLM wave 🌊 arxiv.org/abs/1506.05869 -
Quoc Le
@quocleix
Ten years ago today, @OriolVinyalsML and I published this paper on arxiv. Back then, we didn't know how to evaluate chatbots so we chatted with model and showed the samples in the paper.
Glad that 10 years later, chatbots are still cool - and vibe-checking is going strong :-)
Oriol Vinyals
@OriolVinyalsML
"A Neural Conversational Model" is 10 years old, w/ @quocleix . TL;DR you can train a chatbot with a large neural network (~500M params!). Samples 👇
This paper was received with mixed reviews, but I'm glad all the critics are now riding the LLM wave 🌊 arxiv.org/abs/1506.05869 -
Quoc Le reposted
Yi Tay
@YiTayML
Gemini! 😋
The new 2.5 flash-lite has the same crazy research we landed in the 2.5 flash launched at google i/o. 😎
There's also a Gemini 2.5 tech report now. 🔥👇
Sundar Pichai
@sundarpichai
Gemini 2.5 Pro + 2.5 Flash are now stable and generally available. Plus, get a preview of Gemini 2.5 Flash-Lite, our fastest + most cost-efficient 2.5 model yet. 🔦
Exciting steps as we expand our 2.5 series of hybrid reasoning models that deliver amazing performance at the Pareto frontier of cost and speed. 🚀 -
Quoc Le reposted
Thang Luong
@lmthang
As highlighted yesterday at #googleio, Gemini 2.5 Pro with #deepthink mode achieved a new state-of-the-art performance of 49.4% on USAMO 2025, an extremely challenging Math benchmark for AI that requires detailed proof writing.
I was analyzing the model outputs with our math experts for Problem #2, a challenging algebra problem. We were really surprised that it can nail a complex solution, something we haven’t seen before in previous models! This video shows how Gemini 2.5 Pro with DeepThink solves this problem:
- First the model decided that it will prove by contradiction.
- The model then came up with different ideas such as using Rolle's theorem or Newton's inequalities.
- The model later decided to proceed with Rolle's theorem and conducted a series of advanced algebraic manipulations before arriving at a final correct proof.
The results are quite fascinating, thanks to our latest research in parallel thinking techniques!
Thang Luong
@lmthang
It's been a wild run over the last 2 weeks to land Deep Thinking just in time for I/O! Heroic efforts across many teams @GoogleDeepMind from research to production that felt almost impossible last month! Proud of our research team for the relentless effort over the past 6 months towards advanced deep reasoning with parallel thinking, with state-of-the-art performances across math (USAMO), coding (LiveCodeBench), and multimodality (MMMU)! #googleio
-
Quoc Le reposted
Jack Rae
@jack_w_rae
There was a lot of announcements at IO, easy to overlook the new 2.5 Flash.
It's pushing new boundaries in capability vs speed! -
Quoc Le
@quocleix
Amazing progress in reasoning! 🚀 Gemini 2.5 Pro Deep Think hitting 49.4% on USAMO – a feat I'd have considered impossible just a couple of years ago – & Gemini 2.5 Flash achieving 1424 Elo are huge leaps. So proud our team's research ideas contributed to this moment!
blog.google/technology/google-deepmind/google-gemini-upda… -
Quoc Le reposted
vinh q. tran
@vqctran
crazy to see a wild research idea I dreamt about for a long time land into this Gemini 2.5 Flash! it's been amazing making this happen with @YiTayML and @quocleix, and the rest of the team!!!
Arena.ai
@arena
🚨Breaking from Arena: @GoogleDeepMind's new Gemini-2.5-Flash climbs to #2 overall in chat, a major jump from its April release (#5 → #2)!
Highlights:
- Top-2 across major categories (Hard, Coding, Math)
- #3 in WebDev Arena, #2 in Vision Arena
- New model at the cost-performance Pareto frontier
AI progress continues to accelerate🔥Congrats again @GoogleDeepMind! -
Quoc Le reposted
Thang Luong
@lmthang
It's been a wild run over the last 2 weeks to land Deep Thinking just in time for I/O! Heroic efforts across many teams @GoogleDeepMind from research to production that felt almost impossible last month! Proud of our research team for the relentless effort over the past 6 months towards advanced deep reasoning with parallel thinking, with state-of-the-art performances across math (USAMO), coding (LiveCodeBench), and multimodality (MMMU)! #googleio
-
Quoc Le reposted
Yi Tay
@YiTayML
It's so over. Gemini 2.5 flash beats everything else (except 2.5 pro). 😎
I've only been back at Google for 5 months but together with @quocleix & @vqctran (and a few others) we landed some outrageous research idea (which I can't talk about 😶) into this model. 🔥
Arena.ai
@arena
🚨Breaking from Arena: @GoogleDeepMind's new Gemini-2.5-Flash climbs to #2 overall in chat, a major jump from its April release (#5 → #2)!
Highlights:
- Top-2 across major categories (Hard, Coding, Math)
- #3 in WebDev Arena, #2 in Vision Arena
- New model at the cost-performance Pareto frontier
AI progress continues to accelerate🔥Congrats again @GoogleDeepMind! -
Quoc Le
@quocleix
AlphaPokemon? 😅
Logan Kilpatrick
@OfficialLoganK
Gemini 2.5 Pro just got the final 8th badge in Pokemon Blue, incredible pace of progress by the world's most powerful model!!!
Next up: victory road and final 4 : ) -
Quoc Le reposted
Josh Woodward
@joshwoodward
@GeminiApp app update:
* Best model (Gemini 2.5 Pro) now powers Deep Research
* 20 reports per day
* 150 countries
* 45+ languages
In my testing, the analysis really shines with 2.5 Pro. Throw it your hardest questions and let us know!
Available for paying customers today. -
Quoc Le reposted
Martin Baeuml
@mbaeuml
Just shipped a few updates
1. Gemini 2.5 Pro to try for free on gemini.google.com in the model drop down. Advanced has higher limits.
2. Canvas with 2.5 Pro in Advanced. Our best coding model yet. We had so much fun building demos internally, can't wait to see what y'all come up with!
Google Gemini
@GeminiApp
Gemini 2.5 Pro is taking off 🚀🚀🚀
The team is sprinting, TPUs are running hot, and we want to get our most intelligent model into more people’s hands asap.
Which is why we decided to roll out Gemini 2.5 Pro (experimental) to all Gemini users, beginning today.
Try it at no cost at gemini.google.com -
Quoc Le reposted
Logan Kilpatrick
@OfficialLoganK
Gemini 2.5 Pro is now available to all users in the @GeminiApp for free 🚢
It’s also integrated into canvas mode which makes vibe coding easy, enjoy!
gemini.google.com -
Quoc Le
@quocleix
Gemini 2.5 Pro is now #1 across all categories on Arena leaderboard. Been playing with it and it's a really amazing model.
Arena.ai
@arena
BREAKING: Gemini 2.5 Pro is now #1 on the Arena leaderboard - the largest score jump ever (+40 pts vs Grok-3/GPT-4.5)! 🏆
Tested under codename "nebula"🌌, Gemini 2.5 Pro ranked #1🥇 across ALL categories and UNIQUELY #1 in Math, Creative Writing, Instruction Following, Longer Query, and Multi-Turn!
Massive congrats to @GoogleDeepMind for this incredible Arena milestone! 🙌
More highlights in thread👇 -
Quoc Le reposted
Demis Hassabis
@demishassabis
Feedback loop: train SOTA chip design model (AlphaChip) -> use it to design better AI chips -> use them to train better models -> to design better chips... part of the reason why our TPU stack is so good. Congrats @Azaliamirh, @annadgoldie, @JeffDean & the AlphaChip team!
Google DeepMind
@GoogleDeepMind
Our AI for chip design method AlphaChip has transformed the way we design microchips. ⚡
From helping to design state-of-the-art TPUs for building AI models to CPUs in data centers - its widespread impact can be seen across Alphabet and beyond.
Find out more → -
Quoc Le reposted
Jeff Dean
@JeffDean
Welcome, AlphaChip!
Today, we are sharing some exciting updates on our work published in @Nature in 2021 on using reinforcement learning for ASIC chip floorplanning and layout. We’re also naming this work AlphaChip.
Since we first published this work, our use of this approach internally has grown significantly. It has now been used for multiple generations of TPU chips (TPU v5e, TPU v5p, and Trillium), with AlphaChip placing an increasing number of blocks and with larger wirelength reductions vs. human experts from generation to generation:
AlphaChip has also been used with excellent results for other chips across Alphabet, including Google’s Axion chip, an Arm-based general-purpose data center CPU.
In 2022, as a companion to the Nature paper, we open-sourced the code for the AlphaChip algorithms described in the Nature paper (see link below). Since then, external researchers could use this repository to pre-train on a variety of chip blocks and then apply the pre-trained model to new blocks, as was done and described in our original paper.
Today we’re also releasing a pre-trained AlphaChip checkpoint for the open source release that makes it easier for external users to get started using AlphaChip for their own chip designs.
Original Nature paper w/ wonderful joint first authors @Azaliamirh + @annadgoldie, and @mnyazgan, @joesmemory, @ESonghori, @ShenWangURC, @xylophi, @efjohnson, @pathomkar, @Azade_na, @PakJiwoo, Andy Tong, @kavyasrinivas23, @willhang_, @emretuncer, @quocleix, @JamesLaudon, @rh00, Roger Carpenter, and myself):
www.nature.com/articles/s41586-021-03544-w (PDF: rdcu.be/cmedX)
Today’s Addendum to the paper published in Nature: www.nature.com/articles/s41586-024-08032-5 (same authors)
AlphaChip blog post: deepmind.google/discover/blog/how-alphachip-transformed-c…
Open source release: github.com/google-research/circuit_training
Pre-trained checkpoint: github.com/google-research/circuit_training/?tab=readme-o…
Three things we have observed in the external community are described in the Nature Addendum: (1) not doing any pre-training (circumventing the learning aspects of our method by removing its ability to learn from prior experience) (2) not training to convergence (standard practice in ML methods), and (3) using fewer computational resources than described in our Nature paper (using fewer resources is likely to harm performance, or require running for considerably longer to achieve the same performance).
Pre-training the model for it to learn the craft of chip layout and to be able to generalize to new designs is an important part of our method. The pre-training process requires some effort to perform, since one has to find representative blocks and then run a lengthy computational process to pre-train the model to be good at placing those blocks. To avoid external users having to perform this process and make it easier for the external community to use AlphaChip, today we are releasing an AlphaChip model checkpoint pre-trained on 20 TPU blocks. This will enable users to get good zero-shot performance and faster convergence for novel blocks right out of the box. (For best results, however, we continue to recommend that developers pre-train on their own in-distribution blocks, and we provide a tutorial on how to perform pre-training with our open-source repository: see the Addendum).
Many organizations have used AlphaChip as a building block for their own chip design efforts. For example, MediaTek, one of the top chip design companies in the world, extended AlphaChip to accelerate development of their most advanced chips (e.g. the Dimensity Flagship 5G used in Samsung mobile phones), while improving power, performance and chip area.
We’re very excited about the increasing impact of AlphaChip internally and externally, and we look forward to continued work in this space to make custom higher performance, more efficient, and more capable chips dramatically easier to design and build. -
Quoc Le reposted
Logan Kilpatrick
@OfficialLoganK
We just shipped a new set of evals measuring long context reasoning performance which are challenging for frontier models and go beyond needle-in-the-haystack measurement.
1.5 Pro is 12.5% (1.5 Flash is 6.8%) better than the next closest models. 🧮
arxiv.org/abs/2409.12640 -
Quoc Le reposted
Azalia Mirhoseini
@Azaliamirh
The data and code for our recent project “Large Language Monkeys”, which characterizes inference-time scaling laws for LLMs, are now public!
For more details, check out our
Blog post: scalingintelligence.stanford.edu/blogs/monkeys!
Dataset: huggingface.co/datasets/ScalingIntelligence/monkey_business
Code: github.com/ScalingIntelligence/large_language_monkeys
Paper: -
Quoc Le reposted
Arena.ai
@arena
Chatbot Arena update⚡!
The latest Gemini (Pro/Flash/Flash-9b) results are now live, with over 20K community votes!
Highlights:
- New Gemini-1.5-Flash (0827) makes a huge leap, climbing from #23 to #6 overall!
- New Gemini-1.5-Pro (0827) shows strong gains in coding, math over previous versions.
- The new, smaller Gemini-1.5 Flash-8b outperforms gemma-2-9b, matching llama-3-70b levels.
Big Congrats @GoogleDeepMind Gemini team on the incredible launch!
More plots in the followup posts👇
**Note: to better reflect community interests, older models nearing deprecation will soon be removed from the default leaderboard view.
Logan Kilpatrick
@OfficialLoganK
Today, we are rolling out three experimental models:
- A new smaller variant, Gemini 1.5 Flash-8B
- A stronger Gemini 1.5 Pro model (better on coding & complex prompts)
- A significantly improved Gemini 1.5 Flash model
Try them on aistudio.google.com, details in 🧵 -
Quoc Le reposted
Thang Luong
@lmthang
Introducing #haloquest, our latest effort towards improving hallucination in multimodal foundation models! We hope that HaloQuest, as both a challenging benchmark and an open-sourced dataset, will enable the field's progress in advanced reasoning! Also glad that HaloQuest played a small part in the great news yesterday that #gemini is now #1 on lmsys & accepted at #eccv2024.
Project page: haloquest.github.io
Paper: arxiv.org/abs/2407.15680
Github: github.com/google/haloquest/ -
Quoc Le
@quocleix
Excited to see Gemini #1 on lmsys. Nice ELO of 1300 :-)
Arena.ai
@arena
Exciting News from Chatbot Arena!
@GoogleDeepMind's new Gemini 1.5 Pro (Experimental 0801) has been tested in Arena for the past week, gathering over 12K community votes.
For the first time, Google Gemini has claimed the #1 spot, surpassing GPT-4o/Claude-3.5 with an impressive score of 1300 (!), and also achieving #1 on our Vision Leaderboard.
Gemini 1.5 Pro (0801) excels in multi-lingual tasks and delivers robust performance in technical areas like Math, Hard Prompts, and Coding.
Huge congrats to @GoogleDeepMind on this remarkable milestone!
Gemini (0801) Category Rankings:
- Overall: #1
- Math: #1-3
- Instruction-Following: #1-2
- Coding: #3-5
- Hard Prompts (English): #2-5
Come try the model and let us know your feedback!
More analysis below👇 -
Quoc Le
@quocleix
You should spend more compute at test time
Jordan Juravsky
@jordanjuravsky
Do you like LLMs? Do you also like for loops? Then you’ll love our new paper!
We scale inference compute through repeated sampling: we let models make hundreds or thousands of attempts when solving a problem, rather than just one. By simply sampling more, we can boost LLM performance across a range of math and coding tasks, allowing weaker models (ex. Llama-3-8B) to outperform single attempts from more capable models (ex. GPT-4o).
Notably, with DeepSeek-Coder-V2-Instruct and 250 attempts, we solve 56% of issues from SWE-bench Lite, outperforming the single-attempt SOTA of 43%.
Joint work with @brad19brown, @ryansehrlich, @ronnieclark__, @quocleix, @HazyResearch, and @Azaliamirh!
More details below: -
Quoc Le
@quocleix
Amazing achievement: Our AI system at Google DeepMind just achieved the equivalent of a silver medal at the International Mathematical Olympiad (#imo2024).
This is a major milestone for AI in mathematics. Very proud to have been a part of it!
Google DeepMind
@GoogleDeepMind
We’re presenting the first AI to solve International Mathematical Olympiad problems at a silver medalist level.🥈
It combines AlphaProof, a new breakthrough model for formal reasoning, and AlphaGeometry 2, an improved version of our previous system. 🧵 dpmd.ai/imo-silver
Quoc Le
@quocleix
More details: The system is a combination of AlphaGeometry 2 and AlphaProof. It achieves full scores for 4 out of 6 problems (P1, P2, P4, P6), total of 28 points. Gold-level starts at 29 points. It can solve P6 which is the hardest problem that very few contestants can solve.
Quoc Le
@quocleix
AlphaGeometry 2 produced a elegant proof for P4. It took only 19 seconds for AlphaGeometry 2 to solve this problem after receiving a formalization.
-
Quoc Le reposted
Swaroop Mishra
@Swarooprm7
Introducing NATURAL PLAN 🔥: a realistic planning benchmark in natural language!
Key features:
- 3 main tasks: Trip Planning, Meeting Planning, and
Calendar Scheduling.
- Supplies in the context all relevant information to the
model (e.g., Google Flights, Maps, Calendar)
- No need for a separate tool-use environment: direct
LLM calls for evaluations
- Assesses the planning capabilities of large language
models (LLMs)
Joint work with my awesome collaborators at @GoogleDeepMind : @HuaixiuZheng , @hughbzhang , (now at Scale AI), @xinyun_chen_ , @chenmm24 , @Azade_na , @Hou_Le, @HengTze , @quocleix , @edchi ,@denny_zhou .
Paper: arxiv.org/pdf/2406.04520
Dataset and evaluation code will be released
[1/5] -
Quoc Le reposted
Arena.ai
@arena
Big news – Gemini 1.5 Flash, Pro and Advanced results are out!🔥
- Gemini 1.5 Pro/Advanced at #2, closing in on GPT-4o
- Gemini 1.5 Flash at #9, outperforming Llama-3-70b and nearly reaching GPT-4-0125 (!)
Pro is significantly stronger than its April version. Flash’s cost, capabilities, and unmatched context length make it a market game-changer!
Huge congrats to @GoogleDeepMind on the incredible
Gemini launches! Can't wait to see what new applications Gemini unlocks!
More breakdown analysis below👇 -
Quoc Le reposted
Jeff Dean
@JeffDean
Consider entering the Gemini API Developer Competition! Win an environmentally-friendlier electric 1981 Delorean, or $1M in prizes. Plus it'll be fun just to see what you can do with the Gemini APIs!
Google for Developers
@googledevs
🚘✨ If you're gonna build an app, why not do it with some style?
Join our Gemini API Developer Competition! Create an app with the Gemini API & showcase the power of gen AI for a chance to win a custom electric 1981 DeLorean and more.
Details here 👀 → goo.gle/3V0it2m -
Quoc Le reposted
Thang Luong
@lmthang
Just realized that #alphageometry was highlighted at the beginning of @demishassabis's keynote talk at #googleio today. I was watching the keynote but somehow missed it, until @quocleix told me so :D. Stay tune for more updates from our team later in the year, cc @thtrieu_!
Demis Hassabis
@demishassabis
Making great progress on the Gemini Era. At #googleio we shared 2M long context breakthrough with 1.5 Pro and announced Gemini 1.5 Flash, a lighter-weight multimodal model with long context designed to be fast and cost-efficient to serve at scale. More: blog.google/technology/ai/google-gemini-update-flash-ai-a…
-
Quoc Le reposted
Jeff Dean
@JeffDean
There was a bit of confusing back and forth between myself and @ylecun recently, because in the tweet below, 'it' was referencing DistBelief, but @ylecun thought DistBelief referred to the ICML 2012 "cat detector" paper (icml.cc/2012/papers/73.pdf), not to the training system called DistBelief (which was used for the ICML 2012 paper but thousands of other things). DistBelief was initially described in the NeurIPS 2012 paper (Large Scale Distributed Deep Networks, papers.nips.cc/paper_files/paper/2012/file/6aca97005c68f1…). In some back-and-forth discussion, we've clarified that the "cat detector" paper did indeed use backprop (Yann thought it did not), and also that DistBelief supported training end-to-end convnet models starting in early 2012 (not later than the "cat detector" paper as Yann had thought, although the "cat detector" paper itself did not use convolutions because of its focus on evaluating biologically-plausible unsupervised learning: see (2) below).
Now that Yann clarified that it's the ICML 2012 paper which used DistBelief that he meant when he referred to "it" as a "dead end" (I presume Yann doesn't think the DistBelief system in the NeurIPS 2012 paper is a dead end?), let me say why I think the ICML 2012 paper was and is interesting, even if the particular unsupervised autoencoder technique described in the paper isn't widely used today.
(As an aside, I try not to refer to any research work as a "dead end", because I think much of how we progress as a field is by trying things, and that we all collectively learn a lot by seeing what specific ideas work, what general areas are interesting even if the specific ideas may need modifying, get inspired by multiple independent ideas that are combined in other work, etc.)
I think the ICML 2012 "cat detector" paper explored several things that together make it not a "dead end":
(1) It reported results on massively-scaled up neural networks (using the DistBelief software system that enabled this), training a neural net with 1B parameters (100X larger than the previously reported largest networks in the literature). This alone showed many interesting properties (see (3) below).
(2) It evaluated a biologically-plausible approach for doing unsupervised learning on unlabeled images (random frames from YouTube videos), to explore what representations would be learned at high levels in the network when we significantly scaled up an autoencoder-based approach for learning representations from unlabeled data. It did not use convolutions, because we wanted to use network architectures that were more biologically plausible (unshared weights, local receptive fields). This neuroscience-inspired hypothesis is conveyed in the first part of the paper:
Indeed, interesting feature representations did emerge:
(3) In order to show that the unsupervised representations learned were useful for downstream supervised learning, we used supervised training on a neural network initialized with the pre-trained representations that were learned from (2), and compared that with a baseline trained in a purely supervised manner trained from random weights. The results showed that (a) the unsupervised representations learned in a very large model gave significant improvements over the purely supervised approach (15.8% vs. 13.6% on ImageNet 22k), but also (b) that purely supervised learning of a very large model gave major improvements over the prior state-of-the-art (13.6% vs. 9.3% prior SoTA). To me, both (a) and (b) are interesting (and improvements of ~70% on state-of-the-art on ImageNet 22k seem interesting).
(4) The paper demonstrated how to use a combination of model parallelism (each model replica was distributed across 169 machines and also parallelized within each of these machines), as well as data parallelism (using multiple replicas of this setup) trained with asynchronous SGD to train the model robustly and reliably, even in the presence of machine failures (we used 16,000 CPU cores on ~1000 machines). I had explored and compared model parallel and data parallel training at a much smaller scale in my undergraduate thesis in 1990 (x.com/JeffDean/status/1033054187742015489), but this is the first demonstration that I'm aware of for these two parallelism approaches to be put together in training a single system. The NeurIPS 2012 paper on the DistBelief system explored a variety of different tradeoffs in this space, but the ICML 2012 paper was the first one that described this approach (almost as an aside in the paper, though). Nearly all large-scale neural network training these days combines model parallel and data parallel training, and while a lot of large-scale training, I believe asynchronous training will be more and more important as we further scale neural networks.
To me, the lessons learned in (1), (2), (3), and (4) make this not a "dead end". One major takeaway was that continued work on scaling up training of neural networks would lead to further major gains in accuracy across things like computer vision, speech recognition, and language processing. This led to future systems like TPUs, TensorFlow, Pathways, PaLM, and Gemini. Another was that using unsupervised (or self-supervised) methods can help give major boosts in accuracy and reduce the need for labeled data (known at the time, but demonstrated at large scale is still interesting).
Now, could we have done things differently? Sure!
If we'd not been as interested in the biologically-plausible unsupervised learning of local receptive fields and non-shared weights, we could have used convolutions and likely gotten much more efficient learning and probably higher accuracy (it's not very efficient to learn the same kinds of low level filters at each spatial position in your network mostly independently). However, as mentioned in (2), our goal wasn't to get really accurate ImageNet models, but rather to explore what kinds of high-level feature detectors emerged from purely biologically-plausible unsupervised learning.
So, if this is what constitutes a "dead end", I'm delighted to be associated with this paper, along with my awesome co-authors @quocleix, @MarcRanzato, @rajatmonga, Matthieu Devin, Kai Chen, @greg_corrado and @AndrewYNg .
@ylecun, sorry about the initial miscommunication, when your reference to "it" seemed to be discussing the NeurIPS 2012 paper/DistBelief system, rather than the ICML 2012 paper. Because of that, I thought you were saying the DistBelief approach of scaled-up neural network training was a "dead end" and this made me a bit defensive, because I didn't believe it (and you just asserted it without any justification, which also didn't seem reasonable). I'd be happy to hear in more detail which particular things in the ICML 2012 paper you feel are dead ends, or any response you have to the above points.
Vacation photo incoming! Cat detector fired!
Yann LeCun
@ylecun
It was a dead end.
It couldn't do supervised ConvNets efficiently. -
Quoc Le reposted
Shayne Longpre
@ShayneRedford
Excited to see our 🍮Flan-Palm🌴 work finally published in @JmlrOrg 2024!
Looking back, I see this work as pushing hard on scaling: post-training data, models, prompting, & eval.
We brought together the methods and findings of many awesome prior works, scaled them up, and evaluated the impact extensively, w/ human preference evals which were not as common or widely available then. (The key was a lot of manual test example creation and blind reviewing -- to really see the effects of different decisions.)
📜 jmlr.org/papers/v25/23-0870.html
📽️ youtu.be/oqi0QrbdgdI
It was a pleasure working w/ @_jasonwei @hwchung27 & Le Hou.
1/ -
Quoc Le
@quocleix
Our new work on evaluating and benchmarking long-form factuality. We provide a new dataset, an evaluation method, an aggregation metric that accounts for both precision and recall, and an analysis of thirteen popular LLMs (including Gemini, GPT, and Claude). We’re also open-sourcing all data and code for future work to use.
Jerry Wei
@JerryWeiAI
New @GoogleDeepMind+@Stanford paper! 📜
How can we benchmark long-form factuality in language models?
We show that LLMs can generate a large dataset and are better annotators than humans, and we use this to rank Gemini, GPT, Claude, and PaLM-2 models.
arxiv.org/abs/2403.18802 -
Quoc Le reposted
rohan anil
@_arohan_
We with @quocleix used Gemini Advanced yesterday to brainstorm for an internal research week debate. It was quite incredible experience and an effective companion in creative brainstorming and nothing else compares.
Clemens Meyer
@clemenslm
Try out Gemini Advancd, our best model yet - it's awesome!
-
Quoc Le reposted
Jack Krawczyk
@JackK
Today, Bard becomes Gemini.
Available on web (gemini.google.com) and mobile (new app in Play Store, Google app on iOS) - starting to roll out today.
AND introducing Gemini Advanced gemini.google.com/advanced - access to our most capable model (Ultra 1.0) -
Quoc Le reposted
Jeff Dean
@JeffDean
Bard (the product) is now Gemini. Gemini Advanced is powered by the Gemini Ultra 1.0 model, our most advanced multimodal model. Congrats to everyone who worked on this effort, including the Bard and Gemini teams! Lots more updates in the blog post!
blog.google/products/gemini/bard-gemini-advanced-app/ -
Quoc Le reposted
Pei Zhou
@peizNLP
There is an old Chinese saying: 千人千面 (Thousands of people and thousands of faces). Just as each person is unique, each problem is unique. How do we prepare LLMs to solve complex unseen problems through reasoning?
Our (@USCViterbi, @GoogleDeepMind) new paper: Self-Discover: LLMs Self-Compose Reasoning Structures presents evidence that models can compose task-specific reasoning strategies from a collection of general problem-solving techniques.
Self-Discover outperforms CoT on GPT-4 and PaLM 2-L on BigBench-Hard by as much as 32% while requiring 10-40x fewer inference computes than Self-Consistency.
Spoilers: the discovered structures can also transfer between Large/Small LMs and share commonality with human reasoning patterns!
🗝️arxiv.org/abs/2402.03620
@jay_mlr @xiangrenNLP @xinyun_chen_ @HengTze @quocleix @edchi @denny_zhou @Swarooprm7 @HuaixiuZheng -
Quoc Le reposted
Jack Krawczyk
@JackK
Bard with Gemini Pro is now available in over 40 languages and 230+ countries and territories, bringing its top 2 most preferred status across the world (👀 @lmsysorg).
AND you can now bring your imagination to life with image generation. It's optimized for speed and is available in English across most countries... let your imagination run wild, at no cost.
AND we’re also making it easier to corroborate Bard’s responses. We’re expanding our Double Check feature, which is already used by millions of people in English, to more than 40 languages across the globe.
Thank you, PaLM2 for all you did to get us here 🫡. Rest easy.
Ok back to work... read more here: blog.google/products/bard/google-bard-gemini-pro-image-ge… -
Quoc Le reposted
Thang Luong
@lmthang
Image generation is finally in #bard! It was an effort that I started with the first demo built 10 months ago; it took a while but I am glad that we have finally shipped it; great joint effort across many teams! Try simple queries like "generate a fantastical image" & get cool images like here g.co/bard/share/bb5011cc1521.
Google DeepMind
@GoogleDeepMind
Unleash your creativity using Imagen 2: Google DeepMind’s most advanced text-to-image technology. 🎨
Try it now on Bard, @Google's latest AI experiment ImageFX and Search Generative Experience (SGE). → dpmd.ai/google-imagen-2 -
Quoc Le
@quocleix
Bard (free) with Gemini Pro is now #2 on lmsys chatbot arena, surpassing GPT-4.
I use it a lot for email and writing purposes and its responses are much better since the first launch. Give it a try at bard.google.com
Arena.ai
@arena
🔥Breaking News from Arena
Google's Bard has just made a stunning leap, surpassing GPT-4 to the SECOND SPOT on the leaderboard! Big congrats to @Google for the remarkable achievement!
The race is heating up like never before! Super excited to see what's next for Bard + Gemini Ultra release. -
Quoc Le reposted
Yuhuai (Tony) Wu
@Yuhu_ai_
Euclidean geometry problems have been my favorite math puzzles since middle school. The most intriguing part of it is the creation of auxiliary lines, which opens a space for imagination and the freedom to explore various diagrams. Once a proof is found, these auxiliary lines almost seem magic!
In AlphaGeometry, we've devised a way to synthesize data that trains transformers to predict auxiliary lines. The main idea is straightforward:
1. Sample a random diagram.
2. Deduce all the facts about this diagram using a symbolic algorithm.
3. For each fact, we identify the minimal diagram in which the fact remains true.
4. This leads us to a trajectory from the minimal diagram to the full diagram for learning auxiliary line construction.
5. Sample a lot of random diagrams and gather a lot of data to train a big transformer.
Even though this idea seems straightforward, the underlying details are insanely hard to get right.
1. What's the action space to sample a diagram? We need an action space that allows us to sample interesting diagrams easily.
2. What's the symbolic algorithm to deduce all the facts? We built on prior works e.g. Chou et. al. from 2 decades ago, but also greatly empower their algorithm.
3. What's the algorithm to reduce the diagram to its minimal form? Super non-trivial - but Trieu solved it!!
It took Trieu (@thtrieu_) 4 years (almost his entire PhD) to work out these details. I was very fortunate to meet Trieu back in 2022, when we found we were working on exactly the same idea. During our collaboration, there are multiple times Trieu rewrote the entire project from scratch in order to tackle a new technical problem. It took an insane amount of determination and perseverance that eventually led to today's result.
People ask me how general this technique is, and what the implication is for MATHAI. I think it's clear now that solving math boils down to synthesizing high quality data. Geometry happens to have a nice action space to sample interesting problems. But for general math domains, randomly sampling likely is not the way to go. Instead, figuring out how to bootstrap from existing human data and learn the distribution of interesting problems is much more promising.
The other aspect of this work is that we utilize symbolic tools / formal algorithms to verify the correctness of the data. I think that is also fundamental to math, and how to utilize these formal tools to verify the data is one of the major future challenges. Our recent work accepted to ICLR2024, led by Jin Zhou (@jpzhou), took a small step towards this direction for general math domains: openreview.net/forum?id=V5tdi14ple, but there are still many interesting problems yet to be solved.
Lastly, the way we synthesize those 'magic' steps can be seen as some form of reconstruction, and I believe there are ways to do such reconstruction for other kinds of mathematics to learn these "magic" steps.
Thank you Trieu @thtrieu_ and the rest of the AlphaGeometry team (@lmthang, @quocleix, @hhexiy) for this amazing collaboration. We're one step closer to solving math. LFG!!!
trieu
@thtrieu_
Proud of this work. Here's my 22min video explanation of the paper: www.youtube.com/watch?v=TuZhU1CiC0k
Google DeepMind
@GoogleDeepMind
Introducing AlphaGeometry: an AI system that solves Olympiad geometry problems at a level approaching a human gold-medalist. 📐
It was trained solely on synthetic data and marks a breakthrough for AI in mathematical reasoning. 🧵 dpmd.ai/alphageometry -
Quoc Le reposted
trieu
@thtrieu_
Proud of this work. Here's my 22min video explanation of the paper: www.youtube.com/watch?v=TuZhU1CiC0k
Google DeepMind
@GoogleDeepMind
Introducing AlphaGeometry: an AI system that solves Olympiad geometry problems at a level approaching a human gold-medalist. 📐
It was trained solely on synthetic data and marks a breakthrough for AI in mathematical reasoning. 🧵 dpmd.ai/alphageometry -
Quoc Le reposted
Shane Legg
@ShaneLegg
As someone who still vividly remembers trying to solve crazy hard geometry problems at the New Zealand IMO training camp in Christchurch way back in 1990... it kind of blows my mind to see how good AI has become at this! AGI keeps getting closer. dpmd.ai/alphageometry
-
Quoc Le reposted
Eric Topol
@EricTopol
LLM #ai may not be great at math, but "AlphaGeometry is the 1st computer program to surpass the performance of the average International Mathematical Olympiad contestant in proving Euclidean geometry theorems..."
www.nature.com/articles/s41586-023-06747-5 @Nature -
Quoc Le
@quocleix
RT @demishassabis: Congrats to the #alphageometry team on their impressive breakthrough published today in @Nature using a neuro-symbolic h…
-
Quoc Le
@quocleix
New York Times article on AlphaGeometry
Geometry was my favorite subject in high school because solving them requires many step of reasoning and planning. Geometry problems however have been difficult for AI to solve. Our Nature paper shows my team’s progress in Geometry (approaching IMO Gold level). Key ideas include theorem-proof synthesis and combining search with neural nets (neuro-symbolic) to do complex multi-step theorem proving.
www.nytimes.com/2024/01/17/science/ai-computers-mathemati…Post not available here, view on X
Quoc Le
@quocleix
Google DeepMind blogpost: deepmind.google/discover/blog/alphageometry-an-olympiad-l…
Quoc Le
@quocleix
Video explaining the method and results: www.youtube.com/watch?v=TuZhU1CiC0k&ab_channel=TrieuHobbies
-
Quoc Le reposted
Chen Liang
@crazydonkey200
Just presented our work “Symbolic Discovery of Optimization Algorithms” at NeurIPS today and learned from other attendees how the Lion optimizer has been used. Please reach out if you are interested to chat at the conference.
arxiv.org/abs/2302.06675
github.com/google/automl/tree/master/lion -
Quoc Le reposted
Oriol Vinyals
@OriolVinyalsML
seq💜💙❤️seq
@ilyasut @OriolVinyalsML @quocleix
Sasha Rush
@srush_nlp
2014->2024, 100M->1T, same people, pretty similar model. It's such a crazy story.
-
Quoc Le reposted
Jeff Dean
@JeffDean
I’m very excited to share our work on Gemini today! Gemini is a family of multimodal models that demonstrate really strong capabilities across the image, audio, video, and text domains. Our most-capable model, Gemini Ultra, advances the state of the art in 30 of 32 benchmarks, including 10 of 12 popular text and reasoning benchmarks, 9 of 9 image understanding benchmarks, 6 of 6 video understanding benchmarks, and 5 of 5 speech recognition and speech translation benchmarks. Gemini Ultra is the first model to achieve human-expert performance on MMLU across 57 subjects with a score above 90%. It also achieves a new state-of-the-art score of 62.4% on the new MMMU multimodal reasoning benchmark, outperforming the previous best model by more than 5 percentage points.
Gemini was built by an awesome team of people from @GoogleDeepMind, @GoogleResearch, and elsewhere at @Google, and is one of the largest science and engineering efforts we’ve ever undertaken. As one of the two overall technical leads of the Gemini effort, along with my colleague @OriolVinyalsML, I am incredibly proud of the whole team, and we’re so excited to be sharing our work with you today!
There’s quite a lot of different material about Gemini available, starting with:
Main blog post: blog.google/technology/ai/google-gemini-ai
60-page technical report authored by th Gemini Team: deepmind.google/gemini/gemini_1_report.pdf
In this thread, I’ll walk you through some of the highlights. -
Quoc Le reposted
Demis Hassabis
@demishassabis
The Gemini era is here. Thrilled to launch Gemini 1.0, our most capable & general AI model. Built to be natively multimodal, it can understand many types of info. Efficient & flexible, it comes in 3 sizes each best-in-class & optimized for different uses blog.google/technology/ai/google-gemini-ai
-
Quoc Le reposted
Swaroop Mishra
@Swarooprm7
Overwhelmed by endless chatbot text walls? 😵💫Can't personalize them enough? 🤔
Introducing ExploreLLM🤖! 🔥🚀 It goes Beyond Chatbots with
🧠 Structured Thoughts and
💁 Personalized Model Responses.
✨ Get ready for a new level of AI interaction! 🤩
Joint work with the awesome collaborators: @infoxiao , Ariel Liu, Sophie Su, Jilin Chen, @chinmay , @HengTze , @quocleix and @edchi
For a complex task such as “I want to plan a trip to Tokyo", ExploreLLM organizes the query into sub-tasks automatically such as deciding on dates and duration, making hotel and flight arrangements, checking travel documents, etc. Such organizational structures, or schema, allow people to not only learn what aspects are important in the given task, but also act as a cue to express their own preferences.
Further, we design a dedicated user preference solicitation UI and a recommender-like user interaction within sub-tasks, aiming at improving personalization and the ability of users to easily steer model responses. -
Quoc Le reposted
Najoung Kim 🫠
@najoungkim
Belated thread on updates to the "Inverse scaling can become U-shaped" paper; so belated that I'm now presenting it at EMNLP :)
Takeaways:
(1) Inv scaling curves are hard to extrapolate
(2) Even minimal demonstrations (1-shot) are critically effective as a mitigation strategy -
Quoc Le
@quocleix
Congratulations @geoffreyhinton.
The celebration today brought back many fond memories of learning from and collaborating with Geoff. His passion for research is another level and has made a great impact on the career of researchers around him, including myself. Every time he visited Mountain View, he was generous to spend time chatting with me about research ideas and we would collaborate even on weekends.
Funny story: Around 2014, I was struggling to make a decision to stay at Google or to join CMU as a faculty. Knowing about my struggle, he cut out a piece of newspaper of Pittsburgh in a snowstorm and put it on my desk. That piece of newspaper was a key reason why I chose to stay at Google :-)
Andrew Ng
@AndrewYNg
Attending @geoffreyhinton’s retirement celebration at Google with old friends. Thank you for everything you’ve done for AI! @JeffDean @quocleix
-
Quoc Le reposted
Carlos E. Perez
@IntuitMachine
Let's discuss Step-Back Prompting
Step-Back Prompting is like taking a step back to see the bigger picture before diving into the details.
It's based on the observation that we humans often simplify complex problems by first identifying the key, high-level concepts. We extract the essence before getting lost in the weeds.
Let's take an example:
Original question: "What was the name of the first dog sent into space by the Soviet Union in 1957?"
This requires finding an obscure fact about a specific dog many years ago. Easy to get overwhelmed by the details.
Instead, we first take a step back and ask - "What were the major milestones in early space exploration history?"
Now we're dealing with summarizing the key events in space travel. Much more manageable.
The high-level concept here is "space exploration milestones". We'd retrieve key facts like:
- Yuri Gagarin was the first human in space (1961, Soviet Union)
- Sputnik was the first artificial satellite (1957, Soviet Union)
- Laika was the first animal in orbit (1957, Soviet Union)
Armed with this knowledge, we can now easily infer that Laika must have been the first dog in space sent by the Soviets in 1957.
So in a nutshell, Step-Back Prompting is like zooming out of Google Maps to first see the bigger picture and main roads before searching for a specific store. Going top-down instead of bottom-up.
We teach the LLMs this strategy of abstraction before reasoning by showing them examples of stepping back and extracting high-level concepts. This grounds their thinking and improves complex reasoning. -
Quoc Le
@quocleix
Asking LLMs to take a step back improves their ability to reason and answer highly specific questions.
Swaroop Mishra
@Swarooprm7
Introducing 🔥 ‘Step back prompting’ 🔥 fueled by the power of abstraction. Joint work with the awesome collaborators at @GoogleDeepMind : @HuaixiuZheng , @xinyun_chen_ , @HengTze , @edchi , @quocleix , @denny_zhou.
LLMs struggle to answer specific questions such as: “Estella Leopold went to which school between Aug 1954 and Nov 1954?”
We find that prompting LLMs to first step back and answer a high-level less specific question such as “What was Estella Lepold’s education history?” before answering the specific question significantly improve their performance.
Paper: arxiv.org/abs/2310.06117 [1/3] -
Quoc Le
@quocleix
A weakness of LLMs is that they don’t know recent events well. This is nice work from Tu developing a benchmark (FreshQA) to measure factuality of recent events, and a simple method to improve search integration for better performance on the benchmark.
Tu Vu
@tuvllms
🚨 New @GoogleAI paper:
🤖 LLMs are game-changers, but can they help us navigate a constantly changing world? 🤔
As of now, our work shows that LLMs, no matter their size, struggle when it comes to fast-changing knowledge & false premises.
📰: arxiv.org/abs/2310.03214
👇 -
Quoc Le reposted
Jack Krawczyk
@JackK
Today is Bard's half birthday ♥️
All we've done in the past 6 months was go:
- From waitlist to open access
- From 1 language to 40+ languages
- From US/UK to 200+ countries/territories
- Launch coding
- Give Bard vision (image upload)
- Integrate with your personal life (extensions! export!)
- Show confidence & mistakes in responses (Google It v2)
- Steady improvement of model quality
- Introduce web images in responses
Can't wait to see what the next 6 months holds - thanks for experimenting with us 😍
Jack Krawczyk
@JackK
:) excited to hear your feedback on where we get it right and where we need to get better — bard.google.com
-
Quoc Le reposted
Sang Michael Xie
@sangmichaelxie
Releasing an open-source PyTorch implementation of DoReMi! github.com/sangmichaelxie/doremi
The pretraining data mixture is a secret sauce of LLM training. Optimizing your data mixture for robust learning with DoReMi can reduce training time by 2-3x. Train smarter, not longer!
Sang Michael Xie
@sangmichaelxie
Should LMs train on more books, news, or web data?
Introducing DoReMi🎶, which optimizes the data mixture with a small 280M model.
Our data mixture makes 8B Pile models train 2.6x faster, get +6.5% few-shot acc, and get lower pplx on *all* domains!
🧵⬇️ arxiv.org/abs/2305.10429 -
Quoc Le reposted
Ethan Mollick
@emollick
In a new paper showing that AI comes up with more effective prompts for other AIs than humans do, there is this gem that shows how weird AIs are...
The single most effective prompt was to start by telling the AI "Take a deep breath and work step-by-step!" arxiv.org/pdf/2309.03409.pdf -
Quoc Le reposted
Chengrun Yang
@chengrun_yang
New preprint: Large Language Models as Optimizers (arxiv.org/abs/2309.03409)
(1/5) -
Quoc Le reposted
Jerry Wei
@JerryWeiAI
New @GoogleAI paper! 📜
Language models repeat a user’s opinion, even when that opinion is wrong. This is more prevalent in instruction-tuned and larger models.
Finetuning with simple synthetic-data (github.com/google/sycophancy-intervention) reduces this behavior.
arxiv.org/abs/2308.03958
1/ -
Quoc Le reposted
Min Choi
@minchoi
Google Bard just changed the ChatGPT game with its massive multimodal upgrade - Image Recognition
It doesn't just recognize objects in the image, it can also extract texts and understand the image.
Here are 8 mind blowing examples: -
Quoc Le reposted
Jerry Wei
@JerryWeiAI
New @GoogleAI research-blog post on our symbol tuning paper! 😁
Quick ~5 minute read summarizing the key methodology and findings.
Find it here 👇
Google AI
@GoogleAI
Today on the blog, read all about symbol tuning, a method that fine-tunes models on tasks where natural language labels are replaced with arbitrary symbols. Learn how symbol tuning can improve in-context learning for benefits across various settings →goo.gle/3rt3r8U
-
Quoc Le reposted
Jeff Dean
@JeffDean
I'm excited that bard.google.com is now available in 40 languages & 230 countries! You can use images in your prompts, listen to responses, pin conversations, share responses w/friends, export code to more places & more..
blog.google/products/bard/google-bard-new-features-update…
support.google.com/bard/answer/13575153 -
Quoc Le reposted
Xiangning Chen
@XiangningChen
I've just been told that MPT-7B is trained by our Lion optimizer (arxiv.org/abs/2302.06675).
Lion has also been successfully deployed in production systems such as Google search ads CTR model.
Glad to see that our work has real-world production impact!
www.mosaicml.com/blog/mpt-7b -
Quoc Le reposted
Sundar Pichai
@sundarpichai
We’re updating Bard with a new technique called implicit code execution. Now it runs code in the background when it detects computational prompts, improving the accuracy of word and math problems by ~30%.
blog.google/technology/ai/bard-improved-reasoning-google-… -
Quoc Le reposted
Thang Luong
@lmthang
Images have finally arrived in #bard! Proud to see what our Multi-Bard team have been hard at work, now delivered to everyone. You can ask for recommendation (travel, art, food) & search for images directly at bard.google.com! We are still improving; more examples below!
Jack Krawczyk
@JackK
📸 Bard (bard.google.com) starts its image journey today 📸
The first of many features we discussed at I/O is live right now: images in responses.
Enjoy some extra visual flair to help you imagine more possibilities.
Click an image to see the source for each one. 1/ -
Quoc Le
@quocleix
Can we speed up language model training with a better data mixture?
Our DoReMi🎶 algorithm optimizes the data mixture, speeding up 8B model training by 2.6x on The Pile.
Crucially, DoReMi🎶 just trains a small model (30x smaller) to tune the mixture.
arxiv.org/abs/2305.10429
Quoc Le
@quocleix
For more details, check out the thread from the first author @sangmichaelxie:
Sang Michael Xie
@sangmichaelxie
Should LMs train on more books, news, or web data?
Introducing DoReMi🎶, which optimizes the data mixture with a small 280M model.
Our data mixture makes 8B Pile models train 2.6x faster, get +6.5% few-shot acc, and get lower pplx on *all* domains!
🧵⬇️ arxiv.org/abs/2305.10429 -
Quoc Le
@quocleix
Cool new research from our team @Google!
Symbol tuning is a new finetuning method that allows language models to better learn input–label mappings in-context.
Jerry Wei
@JerryWeiAI
New @GoogleAI+@Stanford paper!📜
Symbol tuning is a simple method that improves in-context learning by emphasizing input–label mappings. It improves robustness to prompts without instructions/relevant labels and boosts performance on algorithmic tasks.
arxiv.org/abs/2305.08298 -
Quoc Le reposted
Paul Couvert
@itsPaulAi
ChatGPT has now a big problem.
Google just updated its free competitor, Bard.
Here are 8 things impossible on ChatGPT but that Bard can do (for free): -
Quoc Le reposted
Bindu Reddy
@bindureddy
Playing around with Bard from Google and it appears to have become significantly better than GPT-4x from Open AI
The new Palm model that replaces the previous Lamda LLM appears to be way better! -
Quoc Le reposted
Denny Zhou
@denny_zhou
This week marks ICLR 2023. Thrilled to share a thread dedicated to our 9 accepted papers. We are grateful that all our submissions were accepted and truly appreciate the constructive discussions and valuable feedback provided by the reviewers.
🧵 -
Quoc Le reposted
Jack Krawczyk
@JackK
Bard with a 🎯
minor critique: needs some tweaking to replace “one of” to “THE” -
Quoc Le reposted
Kris Kashtanova
@icreatelife
"This is how Google Bard sees and hears itself"
I used Google Bard to describe itself.
Then I used its description to generate an image, a video based on this image, and a soundtrack.
Tools I used: Google Bard, Midjourney, Kaiber AI, Mubert, RunwayML
This is the description Bard had of itself as a prompt to text-to-image, image-to-video, and text-to-music:
"I would be a large, complex machine with many different components. I would have a large, powerful processor that would allow me to access and process information quickly. I would have a large amount of memory that would allow me to store a vast amount of information. I would have a powerful network connection that would allow me to access information from the real world. I would have a user-friendly interface that would allow me to interact with humans in a natural way. I would be a powerful tool that could be used to help people in all sorts of ways. I could be used to answer questions, generate text, translate languages, write different kinds of creative content, and help people with their tasks. I would be a valuable resource for anyone who needs access to information or help with a task"This media couldn't be displayed. See it on the original post. -
Quoc Le reposted
Jack Krawczyk
@JackK
Now it's easy to follow our updates on Bard (like this one we did recently 👇 )
Check out the Experiment updates page for latest features, improvements, & bug fixes: bard.google.com/updates
New today: We’ve added additional suggested Search topics when people click “Google it”
Jack Krawczyk
@JackK
Hope you’ve had a chance to start testing out Bard! We’re constantly working on developing new features for it - today I wanted to share that we’ve improved Bard’s capabilities in math and logic by incorporating some of the advances we’ve developed in PaLM. /1
-
Quoc Le reposted
Sundar Pichai
@sundarpichai
We're expanding access to Bard in US + UK with more countries ahead, it's an early experiment that lets you collaborate with generative AI. Hope Bard sparks more creativity and curiosity, and will get better with feedback. Sign up: bard.google.com
blog.google/technology/ai/try-bard/ -
Quoc Le reposted
Jeff Dean
@JeffDean
Excited to share a number of features that build on many years of @GoogleAI research and generative AI advances:
An API to the latest generation of our PaLM language models
MakerSuite for developers,
Writing/rewriting help in Docs and Gmail (Google Workspace),
and more!
Thomas Kurian
@ThomasOrTK
Today marks the next chapter of our AI journey with new tools and experiences across Google Workspace and Google Cloud that allow users, developers and businesses to harness the power of generative AI. blog.google/technology/ai/ai-developers-google-cloud-work…
-
Quoc Le
@quocleix
We discovered an optimizer, "Lion", with symbolic program search. In our experiments, it works well for some cases. Please check out arxiv.org/abs/2302.06675 for results and limitations.
-
Quoc Le reposted
Xiangning Chen
@XiangningChen
Symbolic program search discovers an effective optimizer that only tracks momentum, Lion🦁.
It gains 88.3% 0-shot and 91.1% finetuning ImageNet accuracy, and up to 5x, 2.3x, and 2x training efficiency on ViT, diffusion models, and LM, compared to Adam.
arxiv.org/abs/2302.06675 -
Quoc Le reposted
Chen Liang
@crazydonkey200
We conducted symbolic program search and discovered a simple and effective optimizer, Lion🦁.
It gains 88.3% 0-shot and 91.1% finetuning ImageNet accuracy, and up to 5x, 2.3x, and 2x training efficiency on ViT, diffusion models, and LM, compared to Adam.
arxiv.org/abs/2302.06675 -
Quoc Le reposted
-
Quoc Le reposted
Aran Komatsuzaki
@arankomatsuzaki
Symbolic Discovery of Optimization Algorithms
Discovers a simple and effective optimization algorithm, Lion, which is more memory-efficient than Adam.
arxiv.org/abs/2302.06675 -
Quoc Le reposted
AK
@_akhaliq
Symbolic Discovery of Optimization Algorithms
method discovers a simple and effective optimization algorithm, Lion. It is more memory-efficient than Adam as it only keeps track of the momentum
abs: arxiv.org/abs/2302.06675
github: github.com/google/automl/tree/master/lion -
Quoc Le reposted
Jeff Dean
@JeffDean
We're excited to share Bard soon. It utilizes a powerful version of LaMDA and is made possible by our foundational work on Transformers, dialog models, responsible AI, model safety techniques, and more.
Sundar Pichai
@sundarpichai
1/ In 2021, we shared next-gen language + conversation capabilities powered by our Language Model for Dialogue Applications (LaMDA). Coming soon: Bard, a new experimental conversational #googleai service powered by LaMDA.
blog.google/technology/ai/bard-google-ai-search-updates/ -
Quoc Le reposted
Sundar Pichai
@sundarpichai
1/ In 2021, we shared next-gen language + conversation capabilities powered by our Language Model for Dialogue Applications (LaMDA). Coming soon: Bard, a new experimental conversational #googleai service powered by LaMDA.
blog.google/technology/ai/bard-google-ai-search-updates/ -
Quoc Le reposted
Delip Rao
@deliprao
History class: This 2015 paper is the mother of all LM based pre-training approaches, including the GPT, but few are aware of it. GPT (Radford et al 2018) was a direct application of Transformers (Vasvani et al 2017) to the result in this paper (w/ lot of work & insight ofc).
-
Quoc Le reposted
Ethan Perez
@EthanJPerez
Cool paper from @_jasonwei
@YiTayML
@quocleix
on reversing inverse scaling trends found in Round 1 of the Inverse Scaling Prize, with chain of thought prompting! arxiv.org/abs/2211.02011
H/t @_jasonwei
for the paper update- eval setup is pretty convincing now! -
Quoc Le reposted
Aman Sanger
@amanrsanger
No one is talking about the actual best open-source language model today. It isn't Bloom or OPT, not even GLM-130.
It's an 11B instruction fine-tuned model open-sourced by Google themselves: Flan-T5 11B
And the second best is Flan-T5 3B... -
Quoc Le reposted
Jason Wei
@_jasonwei
New preprint!
By evaluating 5x larger language models, inverse scaling can become “U-shaped scaling”, which means that performance increases sharply after decreasing.
arxiv.org/abs/2211.02011
These two tasks here are Third Prize winners from the Inverse Scaling Prize. -
Quoc Le reposted
Samuel Albanie 🇬🇧
@SamuelAlbanie
Just how striking are the recent language model results with Flan-PaLM?
Here's a plot.
Across 57 tasks on mathematics, US history, computer science etc., Flan-PaLM surpasses **both** the June 2023 and June 2024 SotA forecasts from this summer by competitive forecasters.
1/3 -
Quoc Le
@quocleix
New open-source language model from Google AI: Flan-T5 🍮
Flan-T5 is instruction-finetuned on 1,800+ language tasks, leading to dramatically improved prompting and multi-step reasoning abilities.
Public models: github.com/google-research/t5x/blob/main/docs/models.md
Paper: arxiv.org/abs/2210.11416Post not available here, view on X
Quoc Le
@quocleix
There's a typo while copying the model’s output into tweet (birth year of Geoff). See the main figure in the paper:
Quoc Le
@quocleix
Video about the paper:
Shayne Longpre
@ShayneRedford
Too many new papers to read this week? 📚
See the key take-aways of our new Flan-PaLM work in a 7⃣ minute video.
➕Scaling to 540B model
➕Scaling finetuning to 1.8K tasks
➕CoT tuning
➡️
🌟MMLU SOTA 75.2%
🌟Better usability
🌟Better CoT Reasoning
🌐: www.youtube.com/watch?v=oqi0QrbdgdI -
Quoc Le reposted
Yi Tay
@YiTayML
Introducing U-PaLM 540B! @GoogleAI
Training PaLM w UL2's mixture-of-denoisers with only 0.1% more compute unlocks:
- Much better scaling 📈
- Emergent abilities on BIGBench 😎
- Saving 2x compute (4.4 million TPU hours!) 🔥
- New prompting ability
link:arxiv.org/abs/2210.11399 -
Quoc Le reposted
Sebastian Gehrmann
@sebgehr
We want to identify the capabilities and limitations of LLMs.
But which existing tasks are the most challenging? And can we solve them already with newer models and prompting approaches?
Our work, led by Mirac Suzgun, investigates these questions: arxiv.org/abs/2210.09261 👇 -
Quoc Le reposted
Zoubin Ghahramani
@ZoubinGhahrama1
Getting feedback from users is an important way to improve the value and safety of AI systems. We are delighted to launch #aitestkitchen with 3 demos based on our language model #lamda. Gradual roll out for now, but expect more as we learn from this! blog.google/technology/ai/join-us-in-the-ai-test-kitchen/
-
Quoc Le reposted
Glenn Gabe
@glenngabe
Just received access to Google's AI Test Kitchen (which has 3 demos showing off LaMDA). The "List it" demo is pretty sick. Enter what you want to do and receive an endless list of things you could cover. I think content marketers might like this. :)
-
Quoc Le reposted
Sundar Pichai
@sundarpichai
As we shared at #googleio, the first 3 demos in our #aitestkitchen offer a way to experience and give feedback on our language model LaMDA. We're now gradually rolling out access to small groups of users in the US.
blog.google/technology/ai/join-us-in-the-ai-test-kitchen/ -
Quoc Le reposted
Google
@Google
Meet the #aitestkitchen, a rotating set of experimental demos designed to give you a taste of what’s becoming possible with AI in a responsible way. Learn more and reserve your spot to try it out ↓ www.blog.google/technology/ai/join-us-in-the-ai-test-kitc…
-
Quoc Le reposted
Jeff Dean
@JeffDean
Honored that our 2012 paper "Building High-level Features Using Large Scale Unsupervised Learning" received an @icmlconf Test of Time Award honorable mention!
Joint work with @quocleix, @MarcRanzato, @RajatMonga, Matthieu Devin, Kai Chen, @greg_corrado, myself, & @AndrewYNg. -
Quoc Le reposted
Yingwei Li
@yingwei_li
Tomorrow (Friday) morning, we will be presenting our
@CVPR work 𝐃𝐞𝐞𝐩𝐅𝐮𝐬𝐢𝐨𝐧: 𝐋𝐢𝐝𝐚𝐫-𝐂𝐚𝐦𝐞𝐫𝐚 𝐃𝐞𝐞𝐩 𝐅𝐮𝐬𝐢𝐨𝐧 𝐟𝐨𝐫 𝐌𝐮𝐥𝐭𝐢-𝐌𝐨𝐝𝐚𝐥 𝟑𝐃 𝐎𝐛𝐣𝐞𝐜𝐭 𝐃𝐞𝐭𝐞𝐜𝐭𝐢𝐨𝐧 at 115a.
Come visit! Code is just released here shorturl.at/lKU16
Google AI
@GoogleAI
Learn how DeepFusion, a fully end-to-end multi-modal 3D detection framework, applies a simple yet effective feature fusion strategy to unify the signals from two sensing modalities and achieve state-of-the-art performance. goo.gle/3M2UgRI
-
Quoc Le reposted
Thang Luong
@lmthang
Excited to share our work on Parti, state-of-the-art text-to-image model that generates photorealistic images with complex compositions & rich world knowledge! Parti shows that autoregressive modeling is powerful & universally applicable! (1/n)
Blog/Paper: parti.research.google -
Quoc Le
@quocleix
We can teach a language model to solve complex problems by showing it how to break down a problem into subproblems, and solve them sequentially.
This “least-to-most prompting” method solves 99.7% of SCAN benchmark while other prompting methods solve ~16%
arxiv.org/abs/2205.10625Post not available here, view on X
Quoc Le
@quocleix
The solution to each subproblem in the list is sequentially built on the answers to the previous subproblems.
E.g., in figure above, the solution to the 2nd Q “How many times can she slide before it closes?” is built on the answer to the 1st Q “How long does each trip take?”
Quoc Le
@quocleix
This work continues our work on “chain-of-thought prompting” which encourages LMs to “show their work”. Chain-of-thought prompting however doesn’t work well for very complex problems like SCAN benchmark. The decomposition step in “least-to-most prompting” addresses this weakness.
-
Quoc Le reposted
Jeff Dean
@JeffDean
New @GoogleAI work:
Input: "Two meerkats sitting next to each other on top of a mountain and looking at the beautiful landscape. There is a mountain, a river lake, and fields of yellow flowers. There are hot air balloons in the sky."
#imagen
g.co/research/imagen
Output:
Google AI
@GoogleAI
Introducing Imagen, a new text-to-image synthesis model that can generate high-fidelity, photorealistic images from a deep level of language understanding. Learn more and and check out some examples of #imagen at g.co/research/Imagen
-
Quoc Le reposted
Jeff Dean
@JeffDean
Chain of thought promoting.
Encouraging language models to "show their work" makes them both more interpretable and more accurate at complex reasoning tasks, solving math problems, etc.
Google AI
@GoogleAI
Learn about chain of thought prompting, a method that equips language models to decompose multi-step problems into intermediate steps, enabling models of sufficient scale to solve complex reasoning problems that are not solvable with standard prompting. → goo.gle/3Ft16h0
-
Quoc Le reposted
Google AI
@GoogleAI
Alpa is a framework that uses just one line of code to easily automate the complex model parallelism process for large #deeplearning models. Learn more and check out the code. goo.gle/3y7xZ1f
-
Quoc Le reposted
Jacob Steinhardt
@JacobSteinhardt
This NYT article on Azalia and Anna's excellent chip design work is gross, to the point of journalistic malpractice. It platforms a bully while drawing an absurd parallel to @timnitGebru's firing. @CadeMetz should be ashamed. (not linking so it doesn't get more clicks)
-
Quoc Le reposted
Jason Wei
@_jasonwei
FLAN is an oral at #iclr2022!
Come hang out at the poster tomorrow! I will answer any question about anything, FLAN or otherwise :)
iclr.cc/virtual/2022/oral/6255
Poster: Tues 4/26 10:30am-12:30pm Pacific Time (PT)
Oral: Thurs 4/28 1:30-1:45am PT
paper: openreview.net/forum?id=gEZrGCozdqR -
Quoc Le
@quocleix
If you like the reasoning abilities in PaLM, also check out:
"chain of thought prompting": arxiv.org/abs/2201.11903
Which adds “chain of thought” before answers in prompts for language models. This simple idea surprisingly enables LMs to do much better in many reasoning tasks.
Quoc Le
@quocleix
For PaLM, chain of thought prompting boosted the solving rate on GSM8K from 17.9% (by regular prompting) to 58.1%. Quite a big jump!
Quoc Le
@quocleix
Chain of thought prompting combined with self-consistency decoding ( arxiv.org/abs/2203.11171 ) solves 75% of the math problems in the GSM8K benchmark (prior finetuned SOTA = 55%).
-
Quoc Le reposted
Denny Zhou
@denny_zhou
Self-consistency improves chain of thought reasoning in large language models.
arxiv.org/abs/2203.11171 -
Quoc Le reposted
Adams Wei Yu
@AdamsYu
LiDAR or camera for self-driving cars? We need both! In our CVPR 2022 paper, we show that multimodality learning can advance state-of-the-art of self-driving car research – our single/ensemble DeepFusion models top the leaderboard of Waymo 3D detection challenge! 1/n
Mingxing Tan
@tanmingxing
Introducing DeepFusion: a new SoTA lidar-camera multimodal model for autonomous driving.
Use cross-attention to align and fuse lidar-camera info, and drastically improve accuracy & robustness for 3D detection.
Arxiv:arxiv.org/abs/2203.08195
Leaderboard: waymo.com/open/challenges/2020/3d-detection/↓ -
Quoc Le reposted
Mingxing Tan
@tanmingxing
Introducing DeepFusion: a new SoTA lidar-camera multimodal model for autonomous driving.
Use cross-attention to align and fuse lidar-camera info, and drastically improve accuracy & robustness for 3D detection.
Arxiv:arxiv.org/abs/2203.08195
Leaderboard: waymo.com/open/challenges/2020/3d-detection/↓ -
Quoc Le reposted
Nando de Freitas
@NandoDF
LaMDA is one of the best papers I’ve read this year. I think everyone should read it, and contrast it to other works. It has many ideas that I believe will be very influential. Congratulations @quocleix and Google teams behind this work.
Google AI
@GoogleAI
Open-domain dialog—where a model converses about any topic—is a key challenge for language models. Learn about LaMDA, a project to build dialog models that are more safe, grounded, high quality, and in line with our Responsible AI Principles ↓ goo.gle/3KJ2oXJ
-
Quoc Le reposted
Zoubin Ghahramani
@ZoubinGhahrama1
Exciting work that dramatically accelerates the mapping between amino acid sequences and protein function, adding a decade-worth of new additions to the Pfam protein family database! From @GoogleAI in collaboration with @MIT @Cambridge_Uni @emblebi
Nature Biotechnology
@NatureBiotech
Using deep learning to annotate the protein universe go.nature.com/3v4tKlv
-
Quoc Le reposted
Liam Fedus
@LiamFedus
Proud to release our last year of work on sparse expert models!
This started over a year ago when we found Switch Transformers pre-trained well, but some variants were unstable or fine-tuned poorly. The new SOTA ST-MoE-32B addresses this.
arxiv.org/abs/2202.08906 -
Quoc Le reposted
Barret Zoph
@barret_zoph
Interested in using sparse expert models, but find they are unstable, hard to design or don’t fine-tune well?
We address these key issues and train 269B param MoE model (w/ FLOPs of 32B dense model) that improves SOTA on NLP benchmarks liked SuperGLUE.
arxiv.org/abs/2202.08906 -
Quoc Le reposted
Urs Hölzle
@uhoelzle
This new paper shows that the energy footprint of ML training (including large-model training) is much smaller than currently believed. On GCP, the net carbon footprint is zero, obviously, but it's still good to focus on reducing energy! 🧵⬇️
@GoogleAI
ai.googleblog.com/2022/02/good-news-about-carbon-footprin… -
Quoc Le reposted
Google AI
@GoogleAI
In order to address the challenges of climate change, it's crucial that accurate data is used. Check out our recent study about the true operational carbon emissions of #ml model training and industry best practices that can reduce ML’s carbon footprint ↓ goo.gle/3JysVFM
-
Quoc Le reposted
Dan Zhang @ Ricursive
@DZhang50
Currently, datacenter ML training and inference uses commodity TPU and GPU devices optimized for a wide range of workloads. Given the extreme scale of large datacenter deployments, would it be practical to build custom accelerators optimized for specific workloads? (1/4)
-
Quoc Le reposted
Google AI
@GoogleAI
Introducing advanced platform-aware #neuralarchitecturesearch for ML model architectures that are optimized for the specific hardware behind powerful datacenter ML accelerators. Read more about this and the new EfficientNetX family of CV models ↓ goo.gle/3stMdou
-
Quoc Le reposted
Daily AI Papers
@papers_daily
LaMDA: Language Models for Dialog Applications by @GoogleAI
papers.labml.ai/paper/281378027a6611ec905035ac8856fc87
Fine tuned language model to get human level dialog quality in terms of quality (sensibleness/ specificity/ interestingness), safety, and factual grounding.
Here's how it works:
1/6 🧵👇 -
Quoc Le reposted
Maithra Raghu
@maithra_raghu
LaMDA: Language Models for Dialogue Applications
Paper: arxiv.org/pdf/2201.08239.pdf
Blogpost: ai.googleblog.com/2022/01/lamda-towards-safe-grounded-and…
Excited to see this paper come out! I enjoyed the weddell seal conversation with LaMDA in our 2021 research summary blogpost! -
Quoc Le reposted
Romal Thoppilan
@ThoppilanRomal
Happy to share our research on #lamda
LaMDA models are trained with up to 137B parameters and it demonstrates near human-level dialog quality and significant improvements in safety and factual grounding.
Arxiv: arxiv.org/pdf/2201.08239.pdf
Blog: shorturl.at/mCQW2 -
Quoc Le reposted
Google AI
@GoogleAI
Open-domain dialog—where a model converses about any topic—is a key challenge for language models. Learn about LaMDA, a project to build dialog models that are more safe, grounded, high quality, and in line with our Responsible AI Principles ↓ goo.gle/3KJ2oXJ
-
Quoc Le reposted
Jeff Dean
@JeffDean
"Circuit training"
Now, thanks to @sguada, @summeryue0, @tobyjboyd, @esonghori, @joesmemory, @annadgoldie & @Azaliamirh & the TensorFlow-Agents team, there's an open-source implementation of RL for chip floorplanning, based on our Nature paper (rdcu.be/cmedX).
Azalia Mirhoseini
@Azaliamirh
Our work on RL for chip floorplanning (Nature’21) is now open-source!
github.com/google-research/circuit_training
We hope "CircuitTraining" enables new advances in learning-based chip design.
Joint work w/ TF-Agents, @sguada, @summeryue0, @tobyjboyd, @esonghori, @joesmemory, @annadgoldie -
Quoc Le reposted
Sergio (🌈)🕊️#PalestineLivesMatter
@sguada
In a joint effort w/TF-Agents, @Azaliamirh, @annadgoldie, @esonghori, @joesmemory, @summeryue0, @tobyjboyd, we are open-sourcing github.com/google-research/circuit_training Deep RL for chip floorplanning (Nature’21), that would enable new collaborations and advances in learning-based chip design.
-
Quoc Le reposted
Azalia Mirhoseini
@Azaliamirh
Our work on RL for chip floorplanning (Nature’21) is now open-source!
github.com/google-research/circuit_training
We hope "CircuitTraining" enables new advances in learning-based chip design.
Joint work w/ TF-Agents, @sguada, @summeryue0, @tobyjboyd, @esonghori, @joesmemory, @annadgoldie -
Quoc Le reposted
Anna Goldie
@annadgoldie
Happy to share the "Circuit Training" code from our Nature article on RL for chip design! So grateful to the TF-Agents team, especially @sguada, and my co-authors for all of their efforts to make this available to the research community. Excited to see others build on this work!
Sergio (🌈)🕊️#PalestineLivesMatter
@sguada
In a joint effort w/TF-Agents, @Azaliamirh, @annadgoldie, @esonghori, @joesmemory, @summeryue0, @tobyjboyd, we are open-sourcing github.com/google-research/circuit_training Deep RL for chip floorplanning (Nature’21), that would enable new collaborations and advances in learning-based chip design.
-
Quoc Le reposted
Google Cloud
@googlecloud
📣: We're partnering with @Qualcomm on Vertex AI NAS (Neural Architecture Search) to accelerate neural network deployments. Learn more ⬇️
Qualcomm
@Qualcomm
.@Snapdragon will be integrating @googlecloud Vertex #ai Neural Architecture Search, enabling engineers to quickly create, test, and deploy edge experiences while using significantly fewer resources. That means faster and smoother #ai use cases. Explore: www.qualcomm.com/news/onq/2021/12/snapdragon-and-google-c…
-
Quoc Le reposted
Mingxing Tan
@tanmingxing
Nice paper with beautiful plots and catchy titles!
I was hopping it could compare EffNetV2 (ICML 2021, convs only) as well: arxiv.org/abs/2104.00298
If EffNetV2 is included, the figure should look roughly like this:
AK
@_akhaliq
A ConvNet for the 2020s
abs: arxiv.org/abs/2201.03545
github: github.com/facebookresearch/ConvNeXt
Constructed entirely from standard ConvNet modules, achieving 87.8% ImageNet top-1 accuracy and outperforming Swin Transformers on COCO detection and ADE20K segmentation -
Quoc Le reposted
Jeff Dean
@JeffDean
As in past years, I've spent part of the holiday break summarizing much of the work we've done in @GoogleResearch over the last year. On behalf of @Google's research community, I'm delighted to share this writeup (this year grouped into five themes).
ai.googleblog.com/2022/01/google-research-themes-from-202… -
Quoc Le reposted
Blaise Agüera (@blaiseaguera.bsky.social)
@blaiseaguera
Just published: Do large language models understand us? link.medium.com/0S1dajuU2lb It’s sometimes claimed that ML is "just stats" and AI can't "understand". I'm arguing that LLMs have a great deal to teach us about language, understanding, intelligence, sociality, even personhood.
-
Quoc Le
@quocleix
GLaM: Efficient Scaling of Language Models with Mixture-of-Experts
1.2T weight model that has better average zero and one-shot results than GPT-3 while using only ⅓ of compute for training.
Arxiv: arxiv.org/abs/2112.06905
Blog: goo.gle/3dBNLWQ
Quoc Le
@quocleix
oops, "... while using only 1/3 of energy for training."
-
Quoc Le reposted
Yanping Huang
@bignamehyp
Our work shows that even a basic version of MoE can work extremely well for generative tasks at scale, and hence should be used as default for future scaling!
Google AI
@GoogleAI
Today we introduce the Generalist Language Model (GLaM), a sparsely activated model that achieves better overall performance on 29 few-shot #naturallanguageprocessing benchmark tasks with just a third of the training energy cost. Learn more on the blog ↓ goo.gle/3dBNLWQ
-
Quoc Le reposted
Google AI
@GoogleAI
Today we introduce the Generalist Language Model (GLaM), a sparsely activated model that achieves better overall performance on 29 few-shot #naturallanguageprocessing benchmark tasks with just a third of the training energy cost. Learn more on the blog ↓ goo.gle/3dBNLWQ
-
Quoc Le reposted
Barret Zoph
@barret_zoph
Our new sparse model (SS-MoE) achieved SOTA on SuperGLUE (super.gluebenchmark.com/leaderboard/)!
Excited to see sparsity pushing state-of-the-art!
This new work builds heavily on our prior work on Switch Transformer: arxiv.org/abs/2101.03961
Paper and more details to come soon! -
Quoc Le reposted
Google Cloud
@googlecloud
We're collaborating with @Qualcomm Technologies to accelerate #ai development, making “it the first system-on-a-chip (SoC) customer to offer Google Cloud #vertexai Neural Architecture Search services.” Learn more from @VentureBeat ↓
#snapdragonsummit
venturebeat.com/2021/11/30/google-and-qualcomm-collaborat… -
Quoc Le reposted
Google AI
@GoogleAI
Read all about how we used neural architecture search to optimize #ml models for Google Tensor, the system-on-chip custom-built to deliver state-of-the-art innovations to Pixel users, achieving higher quality with lower latency and power → goo.gle/3BTObkR
-
Quoc Le
@quocleix
Primer: Searching for Efficient Transformers for Language Modeling
arxiv.org/abs/2109.08668
We use evolution to design a new Transformer variant, called Primer. Primer has a better scaling law, and is 3X to 4X faster for training than Transformer for language modeling.Post not available here, view on X
Quoc Le
@quocleix
In a 1.9B parameter configuration similar to GPT-3 XL, Primer only needs 24K hours to achieve the same one-shot performance that Transformer achieves at 72K hours (Primer is 3X faster):
Quoc Le
@quocleix
Primer improves the original T5 architecture on C4 auto-regressive LM, reducing the training cost by 4X at a 500M parameter size.
-
Quoc Le reposted
Google AI
@GoogleAI
As #neuralnetwork models and training data size grow, training efficiency has become more important. Today we present two families of models for image recognition that train faster and achieve state-of-the-art performance. Learn more and grab the code ↓ goo.gle/3kfIsAk
-
Quoc Le reposted
Stanford NLP Group
@stanfordnlp
As the authors (@MaartenBosma, @_jasonwei, @quocleix, and many others at @GoogleAI) note, this graph from the recent FLAN paper (arxiv.org/abs/2109.01652) “is thought-provoking”—models seem to need a certain large capacity to be able to generalize to being instructed in new tasks.
-
Quoc Le
@quocleix
Large language models can perform zero-shot learning pretty well if you finetune them on NLP tasks verbalized with instructions.
Link: arxiv.org/abs/2109.01652
Quoc Le
@quocleix
See Jason's tweet for more details:
Jason Wei
@_jasonwei
New paper: arxiv.org/abs/2109.01652
We teach a 137B parameter language model to follow instructions using “instruction tuning”, which finetunes the model on a collection of tasks described via instructions.
This improves the zero-shot abilities of the model on unseen tasks. -
Quoc Le reposted
Jason Wei
@_jasonwei
New paper: arxiv.org/abs/2109.01652
We teach a 137B parameter language model to follow instructions using “instruction tuning”, which finetunes the model on a collection of tasks described via instructions.
This improves the zero-shot abilities of the model on unseen tasks. -
Quoc Le reposted
Barret Zoph
@barret_zoph
How do we combine knowledge from multiple labeled and unlabeled datasets to train a great general model?
Multi-Task Self-Training (MuST) trains specialized teachers on labeled data, which then label unlabeled data to train a single general model.
arxiv.org/abs/2108.11353 -
Quoc Le reposted
TensorFlow
@TensorFlow
🆕 The new EfficientNet V2 models are available on #tfhub!
They’re a family of image classification models that achieves greater parameter efficiency and faster training speed.
Take a look → goo.gle/3fAfmJ3 -
Quoc Le reposted
Simone Scardapane
@s_scardapane
*Towards Domain-Agnostic Contrastive Learning*
#icml 2021 by @vikasverma1077 @lmthang Kawaguchi @hieupham789 @quocleix
Here's an interesting question: can we do self-supervised learning if we know *nothing* about the domain we are operating on?
/n -
Quoc Le reposted
Thang Luong
@lmthang
Glad to share our #icml2021 work, DACL, a domain-agnostic contrastive learning approach that works across domains (tabular, images, graphs)! DACL uses Mixup to create pos/neg examples with theoretical foundation.
Paper: proceedings.mlr.press/v139/verma21a/verma21a.pdf
ICML Poster: icml.cc/Conferences/2021/ScheduleMultitrack?event=10693 -
Quoc Le reposted
Thang Luong
@lmthang
Glad to share our recent success in applying semi-supervised learning to Google's most fundamental product, Search. Our approach, semi-supervised distillation (SSD), is a simplified version of Noisy Student and an alternative to product distillation.
Blog: ai.googleblog.com/2021/07/from-vision-to-language-semi-su… -
Quoc Le reposted
labml.ai
@labmlai
Pay Attention to MLPs (gMLP)
📝 Annotated @PyTorch code nn.labml.ai/transformers/gmlp/index.html
🖥 Github github.com/labmlai/annotated_deep_learning_paper_implemen…
@ Paper papers.labml.ai/paper/2105.08050 by @hanxiao_6 @zihangdai @davidrso1 @quocleix @Google -
Quoc Le reposted
Mingxing Tan
@tanmingxing
Happy to introduce CoAtNet: combining convolution and self-attention in a principled way to obtain better capacity and better generalization.
88.56% top-1 with ImageNet21K (13M imgs), matching ViT-huge with JFT (300M imgs).
Paper: arxiv.org/abs/2106.04803 -
Quoc Le reposted
Aran Komatsuzaki
@arankomatsuzaki
CoAtNet: Marrying Convolution and Attention for All Data Sizes
Achieves SotA perf under different resource constraints across various datasets.
Achieves 88.56% top-1 ImageNe accuracy, matching ViT-huge while using 23x less data.
arxiv.org/abs/2106.04803 -
Quoc Le reposted
Google AI
@GoogleAI
Last year we shared how #reinforcementlearning could hasten the design of accelerator chips (goo.gle/3awYoWH). Today we're publishing improved methods, which we've used in production to design the next generation of Google TPUs. Read more in Nature rdcu.be/cmedX
-
Quoc Le reposted
Jeff Dean
@JeffDean
Very excited to have collaborated with @Azaliamirh, @annadgoldie and many others on this work on using reinforcement learning to teach computers to pay the "game of ASIC chip floorplanning"!
Azalia Mirhoseini
@Azaliamirh
Thrilled to announce that our work on RL for chip floorplanning was published in Nature & used in production to design next generation Google TPUs, with potential to save thousands of hours of engineering effort for each next generation ASIC:
www.nature.com/articles/s41586-021-03544-w (1/7) -
Quoc Le reposted
Anna Goldie
@annadgoldie
Excited to share that our work has been published in Nature! Our RL agent generates chip layouts in just a few hours, whereas human experts can take months. These superhuman AI-generated layouts were used in Google's latest AI accelerator (TPU-v5)! www.nature.com/articles/s41586-021-03544-w
-
Quoc Le reposted
Azalia Mirhoseini
@Azaliamirh
Thrilled to announce that our work on RL for chip floorplanning was published in Nature & used in production to design next generation Google TPUs, with potential to save thousands of hours of engineering effort for each next generation ASIC:
www.nature.com/articles/s41586-021-03544-w (1/7) -
Quoc Le reposted
Azalia Mirhoseini
@Azaliamirh
Does it make sense to build more variety of accelerators? We explored designing highly custom inference accelerators for vision models. We optimized over a HW/SW co-design space of size 10^2300 and found vision accelerators with >6x better Perf/TDP!
-
Quoc Le
@quocleix
Good meme for recent MLP papers ;)
Pay attention to MLPs: arxiv.org/abs/2105.08050
GuilleFix — life/acc | /dd ▶️
@guillefix
Glad to see machine learning finally growing up
-
Quoc Le reposted
Hanxiao Liu
@Hanxiao_6
Pay attention to MLPs: arxiv.org/abs/2105.08050
We show that MLPs with gating work well for key applications that Transformers are good at: BERT for NLP and ViT for vision.
Is MLP all you need? -
Quoc Le reposted
Zoubin Ghahramani
@ZoubinGhahrama1
LaMDA: our breakthrough conversation technology @google blog.google/technology/ai/lamda/
-
Quoc Le reposted
Mingxing Tan
@tanmingxing
Excited to share the open source of EfficientNetV2 code and modes.
Paper: arxiv.org/abs/2104.00298(accepted to ICML'21)
Code: github.com/google/automl/tree/master/efficientnetv2
Bonus: amazing speed on both TPUs and V100/A100 GPUs with TensorRT.
Mingxing Tan
@tanmingxing
Happy to introduce EfficientNetV2: Smaller Models and Faster Training
Achieved faster training and inference speed, AND also with better parameters efficiency.
Arxiv: arxiv.org/abs/2104.00298
Thread 1/4 -
Quoc Le reposted
Google AI
@GoogleAI
Building visual and vision-language representations can rely on curated datasets that require expert knowledge and extensive labelling, hindering the scale of models. ALIGN is a new approach that leverages existing, noisy alt-text data with SotA results. goo.gle/33Bxw6U
-
Quoc Le reposted
Aleksandra Faust
@AleksandraFaust
Very excited to share our recent work, which enables automated design of new RL algorithms.
We will be presenting it as oral at ICLR on Wed, 5-May.
sites.google.com/corp/view/evolvingrl
w/@JDCoReyes @yingjieMiao @dypeng @estebitius @svlevine @quocleix @honglaklee @AleksandraFaust
Google AI
@GoogleAI
Today we present a new approach for automated discovery of generalizable #reinforcementlearning algorithms that evolves a population of loss functions — represented by computational graphs — over a set of simple training environments. Learn more at goo.gle/3dFzDML
-
Quoc Le reposted
Thang Luong
@lmthang
Interesting reproducibility study by @philipvollet to conclude that: ELECTRA small (14M) can be easily run on a single GPU with great performances! @clark_kev @quocleix @chrmanning
Reproducibility paper: arxiv.org/abs/2104.02756
ELECTRA paper: arxiv.org/abs/2003.10555
Philip Vollet
@philipvollet
Efficient transfer learning for NLP with ELECTRA • Can we use ELECTRA to achieve close to SOTA performances for NLP in low-resource settings in term of compute cost?
Paper arxiv.org/abs/2104.02756
Model github.com/google-research/electra
_
#nlp #nlproc -
Quoc Le
@quocleix
When in doubt, try larger models with more data :-) Joint training of a big model on many speech datasets leads to impressive improvements across the board on many important benchmarks.
Link: arxiv.org/abs/2104.02133 -
Quoc Le reposted
AK
@_akhaliq
SpeechStew: Simply Mix All Available Speech Recognition Data to Train One Large Neural Network
pdf: arxiv.org/pdf/2104.02133.pdf
abs: arxiv.org/abs/2104.02133
"SpeechStew achieves SoTA or near SoTA results across a variety of tasks, without the use of an external language
model." -
Quoc Le reposted
Hanxiao Liu
@Hanxiao_6
Check out our paper & models! Besides delivering SoTA hardware-specific detectors, this work also underscores the importance of search space design for mobile tasks.
Yunyang Xiong
@YoungXiong1
MobileDets: SoTA object detectors for mobile accelerators.
Our models outperform MobileNetV2 by +1.9/+3.7/+3.4/+2.7 mAP on CPU/EdgeTPU/Hexagon DSP/Jetson GPU at comparable inference latencies.
Paper (Accepted to CVPR 2021): arxiv.org/pdf/2004.14525
Code: github.com/tensorflow/models/tree/master/research/object_… -
Quoc Le reposted
Aleksa Gordić (水平问题)
@gordic_aleksa
Finally! EfficientNet v2 by @GoogleAI paper explained!
EfficientNets are more efficient than NFNets after all - just append v2 - problem solved. ^^
YT: youtu.be/CTsSrOKSPNo
@quocleix @tanmingxing
#efficientnetv2 #nas #deeplearning -
Quoc Le reposted
Mingxing Tan
@tanmingxing
Happy to introduce EfficientNetV2: Smaller Models and Faster Training
Achieved faster training and inference speed, AND also with better parameters efficiency.
Arxiv: arxiv.org/abs/2104.00298
Thread 1/4 -
Quoc Le reposted
Irwan Bello
@IrwanBello
Lambda layers: a scalable alternative framework to self-attention for capturing structured interactions between inputs and their contexts.
LambdaNetworks: Modeling Long-Range Interactions without Attention
To appear @iclr_conf as a spotlight
arxiv.org/abs/2102.08602 -
Quoc Le
@quocleix
Some nice improvement on ImageNet: 90% top-1 accuracy has been achieved :-)
This result is possible by using Meta Pseudo Labels, a semi-supervised learning method, to train EfficientNet-L2.
More details here: arxiv.org/abs/2003.10580
Quoc Le
@quocleix
Short summary of the method: Meta Pseudo Labels is based on Pseudo Labels. But it has a feedback loop from student to teacher. So both teacher and student are trained concurrently to teach each other in this method.
Code: github.com/google-research/google-research/tree/master/me… -
Quoc Le reposted
Jeff Dean
@JeffDean
On behalf of the entire Google Research & @GoogleAI communities, I'm excited to share an overview of some of our research in 2020.
Thanks to everyone who helped make this work possible!
ai.googleblog.com/2021/01/google-research-looking-back-at… -
Quoc Le reposted
Barret Zoph
@barret_zoph
Introducing Switch Transformer, a simplified sparse architecture for scaling to trillion parameter language models
Switch Transformers yield 4-7x speedups over strong Transformer T5 models w/ the same computational resources
Paper: arxiv.org/abs/2101.03961 -
Quoc Le reposted
Adams Wei Yu
@AdamsYu
We achieved new SOTA on SuperGLUE with human parity. Stay tuned for more details. Happy holidays!
-
Quoc Le reposted
Papers with Code
@paperswithcode
⏪ Papers with Code: Year in Review. We’re ending the year by taking a look back at the top trending papers, libraries and benchmarks for 2020. Read on below!
medium.com/paperswithcode/papers-with-code-2020-review-93… -
Quoc Le reposted
Kevin Clark
@clark_kev
Check out our #emnlp2020 paper now on arXiv: we develop an energy-based version of BERT called "Electric" and show this essentially re-invents the ELECTRA pre-training method. Electric is useful for re-ranking the outputs of text generation systems. arxiv.org/pdf/2012.08561.pdf.
-
Quoc Le reposted
Barret Zoph
@barret_zoph
Can simply copying and pasting objects from one image to another be used to create more data to improve state-of-the-art instance segmentation?
Yes!
With Copy&Paste, we achieve 57.3 box AP and 49.1 mask AP on COCO. This is SoTA wrt @paperswithcode
arxiv.org/abs/2012.07177 -
Quoc Le reposted
Thang Luong
@lmthang
If you like ELECTRA, check out Electric, our new #emnlp2020 work that brought over energy-based model perspective ;) Electric produces fast pseudo-likelihood to improve speech recognition & translation reranking. Github: github.com/google-research/electra & paper www.aclweb.org/anthology/2020.emnlp-main.20.pdf
-
Quoc Le reposted
Jaehoon Lee
@hoonkp
Can we leverage the power of infinite-width limit to help with Neural Architecture Search (NAS)?
In this new paper (arxiv.org/abs/2011.06006), we find that empirical NNGP can provide cheap and effective signals that can be used for NAS! -
Quoc Le
@quocleix
Pretty amazing progress on speech recognition thanks to pre-training and self-training with unlabeled data.
Key ingredients: Large conformer architecture + wave2vec2.0 pretraining + Noisy Student Training
Link: arxiv.org/abs/2010.10504 -
Quoc Le reposted
DeepLearning.AI
@DeepLearningAI
We’re thrilled to present "Heroes of NLP," our latest interview series.
@AndrewYNg interviews @chrmanning, Kathleen McKeown, @etzioni, & @quocleix to discuss their personal AI journeys, current research projects, & NLP trends.
Watch the full videos: www.deeplearning.ai/blog/heroes-of-nlp -
Quoc Le reposted
Sylvain Gugger
@GuggerSylvain
I just finished adding the Funnel Transformer model to 🤗 Transformers, a transformer model leveraging ideas from regular CNNs (pooling the hidden states after a block of n layers), ELECTRA pretraining and Transformer-XL positional attention. huggingface.co/funnel-transformer/intermediate
-
Quoc Le reposted
Hanxiao Liu
@Hanxiao_6
Does weight-sharing NAS work?
Yes!
Our study shows that weight-sharing NAS is better than random search, and the gap widens on larger and more challenging search spaces.
Paper and Video (CVPR 2020): openaccess.thecvf.com/content_CVPR_2020/html/Bender_Can_W…
Code: github.com/google-research/google-research/tree/master/tunas -
Quoc Le reposted
Lilian Weng
@lilianweng
Although popular and successful model architectures are mostly designed by human experts, it doesn't mean we have settled down with the best option. Neural Architecture Search (NAS) automates network architecture engineering in a more systematic way.
lilianweng.github.io/lil-log/2020/08/06/neural-architectu… -
Quoc Le reposted
Yannic Kilcher 🇸🇨
@ykilcher
New Video 🔥 Self-Training with Noisy Student is a differnt take on semi-supervised learning and distillation, first training a teacher model on supervised data, then a larger student on more data. Check it out! 📽️
youtu.be/q7PjrmGNx5A
@QizheXie @quocleix -
Quoc Le reposted
Mingxing Tan
@tanmingxing
Happy to share EfficientDet-D7x: new SOTA 55.1 COCO AP at 153ms latency (10x faster than other multi-scale models). TF2+keras supported!
Arxiv: arxiv.org/abs/1911.09070
github: github.com/google/automl/tree/master/efficientdet
paperwithcode: huggingface.co/papers/1911.09070 -
Quoc Le reposted
Barret Zoph
@barret_zoph
Models and checkpoints are now open sourced for my recent work: "Rethinking Pre-training and Self-training".
Paper link: arxiv.org/abs/2006.06882 Code Link: github.com/tensorflow/tpu/tree/master/models/official/det…. On COCO we achieve 54.3 AP and on Pascal Segmentation 90.5 mIOU! -
Quoc Le reposted
Jeff Dean
@JeffDean
AutoML-Zero: new research from @GoogleAI researchers Esteban Real, @crazydonkey200, David R. So, & @quocleix that that can rediscover fundamental ML techniques by searching a space of different ways of combining basic mathematical operations.
Arxiv: arxiv.org/abs/2003.03384
Google AI
@GoogleAI
Extending research into evolutionary #automl, we present an approach that evolves algorithms from scratch—using only basic mathematical operations—rediscovering fundamental ML techniques & showing the potential to discover novel algorithms. Read more ↓ goo.gle/3edHPBc
-
Quoc Le reposted
Zihang Dai
In NLP, the O(TD^2) linear projections in Transformer often cost more FLOPs than the O(T^2D) attention as commonly D > T. While many efforts focus on reducing quadratic attention to linear O(TKD), our Funnel-Transformer explores reducing T with clear gains:github.com/laiguokun/Funnel-Transformer
-
Quoc Le reposted
Google AI
@GoogleAI
Extending research into evolutionary #automl, we present an approach that evolves algorithms from scratch—using only basic mathematical operations—rediscovering fundamental ML techniques & showing the potential to discover novel algorithms. Read more ↓ goo.gle/3edHPBc
-
Quoc Le reposted
Dmitry (Dima) Lepikhin
@lepikhin
arxiv.org/abs/2006.16668
We scaled the Transformer model with Sparsely-Gated Mixture-of-Experts using GShard, and trained a 600B multilingual translation model in about 4 days (for 100 languages) achieving 13.5 BLEU gain compared to the baseline. -
Quoc Le reposted
Google AI
@GoogleAI
Today we present SpineNet, a novel alternative to standard scale-decreased backbone models for visual recognition tasks, which uses reordered network blocks with cross-scale connections to better preserve spatial information. Learn more below: goo.gle/3dNq9fa
-
Quoc Le
@quocleix
A surprising result: We found that smooth activation functions are better than ReLU for adversarial training and can lead to substantial improvements in adversarial robustness.
arxiv.org/abs/2006.14536
Quoc Le
@quocleix
Applying this simple method to EfficientNet produces much more robust models on ImageNet than previous state-of-the-art.
Quoc Le
@quocleix
Our work also reveals that bigger models are also more robust models, which is unexpected and also good news. It's good news because more compute can be used to tackle adversarial robustness.
-
Quoc Le reposted
Mingxing Tan
@tanmingxing
Glad to see EfficientNets help detect fake videos: all winners use pre-trained EfficientNets, with B7 as the most popular one, in the recent DeepFake challenge ai.facebook.com/blog/deepfake-detection-challenge-results…
-
Quoc Le
@quocleix
We researchers love pre-training. Our new paper shows that pre-training is unhelpful when we have a lot of labeled data. In contrast, self-training works well even when we have a lot of labeled data. SOTA on PASCAL segmentation & COCO detection.
Link: arxiv.org/abs/2006.06882Post not available here, view on X
Quoc Le
@quocleix
In the paper, we evaluate two “pre-training” methods: supervised and self-supervised. Both do not transfer well to the COCO dataset.
Quoc Le
@quocleix
Here's how we use self-training (w/ Noisy Student) in our work:
1) Train a model on COCO
2) Infer labels on ImageNet images
3) Train a new model on the combined set with noise added
4) Optional: Iterate the process
Quoc Le
@quocleix
A nice video that summarizes the paper by @CShorten30
www.youtube.com/watch?v=QSjMLGA7e2o -
Quoc Le
@quocleix
Some new progress on Semi-supervised learning for speech recognition. Noisy student training improves accuracy on the LibriSpeech benchmark. The gains are especially big on the noisy test set (“Test-other”).
Link: arxiv.org/abs/2005.09629 -
Quoc Le reposted
Azalia Mirhoseini
@Azaliamirh
So excited to share our work on RL for Chip Placement, which enables chip optimization in < 6 hrs, whereas baselines require manual efforts and can take weeks. Joint work with my co-TL @annadgoldie, @JeffDean (cont.)
Paper: arxiv.org/abs/2004.10746
Blog: ai.googleblog.com/2020/04/chip-design-with-deep-reinforce… -
Quoc Le reposted
Anna Goldie
@annadgoldie
Just released the paper and blog post for Chip Design with Deep Reinforcement Learning! (ai.googleblog.com/2020/04/chip-design-with-deep-reinforce…, arxiv.org/pdf/2004.10746.pdf)
Really excited to finally share this work! Thank you to all of my amazing collaborators, especially my co-lead @Azaliamirh! -
Quoc Le reposted
Mingxing Tan
@tanmingxing
Interesting paper on more efficient COVID-19 detection in X-ray images: arxiv.org/pdf/2004.05717.pdf
By tuning the network architecture, it significantly reduces the model size and computation (from 3.5billion to 4.5million FLOPs), with comparable or better accuracy. -
Quoc Le
@quocleix
@GoogleAI blogpost about EfficientDet.
Google AI
@GoogleAI
Check out EfficientDet: a new class of state-of-the-art object detectors that provide an order of magnitude better efficiency, released along with all code and pre-trained models. Learn more below! goo.gle/3adBgMI
Quoc Le
@quocleix
Our implementation of EfficientDet in TensorFlow:
github.com/google/automl/tree/master/efficientdet
Good implementation of EfficientDet in PyTorch (not by us):
github.com/zylo117/Yet-Another-EfficientDet-Pytorch -
Quoc Le
@quocleix
Cool results from our collaboration with colleagues at @DeepMind on searching for new layers as alternatives for BatchNorm-ReLU. Excited with the potential use of AutoML for discovering novel ML concepts from low level primitives.
Hanxiao Liu
@Hanxiao_6
New paper: Evolving Normalization-Activation Layers.
We use evolution to design new layers called EvoNorms, which outperform BatchNorm-ReLU on many tasks. A promising use of AutoML to discover fundamental ML building blocks.
arxiv.org/abs/2004.02967
Joint work with @DeepMind
Quoc Le
@quocleix
Nice video explaining the paper by @CShorten30:
Connor Shorten
@CShorten30
Evolving Normalization-Activation Layers!
The latest large-scale AutoML study from Google and DeepMind, producing the EvoNorm layers.
ResNet-50 with BN-ReLU achieves 76.1% ImageNet accuracy, replacing BN-ReLU w/ EvoNorm hits 77.8%!
youtu.be/RFn5eH5ZCVo
#100daysofmlcode -
Quoc Le
@quocleix
Waymo's blogpost about the use of automated data augmentation methods for self-driving cars (AutoAugment, RandAugment, Progressive Population Based Augmentation - PPBA).
PPBA is "... up to 10 times more data efficient than training nets without augmentation".
Waymo
@Waymo
Our newest research in collaboration with our @googleAI colleagues will allow us to train better machine learning models with less data and improve perception tasks for the Waymo Driver. blog.waymo.com/2020/04/using-automated-data-augmentation-…
-
Quoc Le reposted
Connor Shorten
@CShorten30
This video explains Meta Pseudo Labels!
A really interesting way to adapt the ground truth target labels while training a classification network. This achieves remarkable performance in Semi-Supervised and Limited Data settings!
youtu.be/yhItocvAaq0
#100daysofmlcode -
Quoc Le reposted
Maithra Raghu
@maithra_raghu
A Survey of Deep Learning for Scientific Discovery
To help facilitate using DL in science, we survey a broad range of deep learning methods, new research results, implementation tips & many links to code/tutorials
Paper arxiv.org/abs/2003.11755
Work with @ericschmidt
Thread⬇️ -
Quoc Le
@quocleix
New paper: Meta Pseudo Labels
Self-training has a pre-trained teacher to generate pseudo labels to train a student. Here we use the student’s performance to meta-train the teacher to generate better pseudo labels. Works well on ImageNet 10%.
Link: arxiv.org/abs/2003.10580
Quoc Le
@quocleix
This work continues our efforts on semi-supervised learning such as
UDA: arxiv.org/abs/1904.12848
MixMatch: arxiv.org/abs/1905.02249
FixMatch: arxiv.org/abs/2001.07685
Noisy Student: arxiv.org/abs/1911.04252
etc.
Joint work with @hieupham789 @QizheXie @ZihangDai
Quoc Le
@quocleix
Some people pointed out some bugs with the figure. We fixed the comparisons and the updated figure is below.
-
Quoc Le reposted
Anna Goldie
@annadgoldie
spectrum.ieee.org/tech-talk/semiconductors/design/google-…
IEEE Spectrum just published a great article on the ML for Chip Design research that I started with @Azaliamirh and @JeffDean! Will post the paper soon with the full author list, but this work would not have been possible without our amazing collaborators: -
Quoc Le reposted
Mingxing Tan
@tanmingxing
Excited to announce the open source of EfficientDet: better accuracy & efficiency on COCO detection. Bonus: it also works pretty well for semantic segmentation (Table 3)!
Paper: arxiv.org/abs/1911.09070
Code: github.com/google/automl/tree/master/efficientdet -
Quoc Le reposted
Mingxing Tan
@tanmingxing
EfficientNet-lite: enjoy 80+% imagenet top-1 accuracy with real time latency on mobile CPU/GPU/EdgeTPU.
Thanks to TF team @TensorFlow
TensorFlow
@TensorFlow
Check out EfficientNet-Lite, a new family of vision models that is optimized for mobile inference using TensorFlow Lite.
Read the blog → goo.gle/2U8topS -
Quoc Le
@quocleix
Fun AutoML-Zero experiments: Evolutionary search discovers fundamental ML algorithms from scratch, e.g., small neural nets with backprop.
Can evolution be the “Master Algorithm”? ;)
Paper: arxiv.org/abs/2003.03384
Code: git.io/JvKrZ
Quoc Le
@quocleix
Nice summary video about AutoML-Zero by @CShorten30:
www.youtube.com/watch?v=aumUBLMfIA4 -
Quoc Le reposted
Kevin Clark
@clark_kev
Excited to share the code and blog post for ELECTRA, a more efficient pre-training method for NLP Transformers. Pre-train a model to good accuracy using only 1 GPU!
Joint work with @lmthang @quocleix @chrmanning
Code: github.com/google-research/electra
Blog Post: ai.googleblog.com/2020/03/more-efficient-nlp-model-pre-tr… -
Quoc Le reposted
Stanford NLP Group
@stanfordnlp
ELECTRA’s Replaced Token Detection pre-training is not only more compute efficient but gives a new best single model result on SQuAD v2 benchmark!
6 Mar 2020 ELECTRA: 88.716 EM 91.365 F1
By @clark_kev @lmthang @quocleix @chrmanning
rajpurkar.github.io/SQuAD-explorer/
openreview.net/pdf?id=r1xMH1BtvB -
Quoc Le reposted
Kevin Clark
@clark_kev
Our pre-trained transformer ELECTRA (joint work with @lmthang @quocleix @chrmanning) is the top single model on the SQuAD 2.0 leaderboard! rajpurkar.github.io/SQuAD-explorer/
Stay tuned for the code release coming next week!
Paper: openreview.net/pdf?id=r1xMH1BtvB -
Quoc Le reposted
Thang Luong
@lmthang
What's the difference between recent works in semi-supervised learning? Key: consistency training (UDA, FixMatch, ReMixMatch, ICT, etc.) and self-training (#noisystudent), see photo below and my recent talk (drive.google.com/file/d/1ax1-XprJHDRRv2Ru3dJwPLs3ShxcpQ3r… with bonus slides on #meenabot!).
-
Quoc Le
@quocleix
Last week we released the checkpoints for SOTA ImageNet models trained by NoisyStudent. Due to popular demand, we’ve also opensourced an implementation of NoisyStudent. The code uses SVHN for demonstration purposes.
Link: github.com/google-research/noisystudent
Paper: arxiv.org/abs/1911.04252
Quoc Le
@quocleix
Give NoisyStudent a try if you want to use unlabeled data to improve your supervised learning.
See my earlier tweet for more context.
Quoc Le
@quocleix
Happy to announce that we've released a number of models trained with Noisy Student (a semi-supervised learning method). The best model achieves 88.4% top-1 accuracy on ImageNet (SOTA).
Enjoy finetuning!
Link: github.com/tensorflow/tpu/tree/master/models/official/eff…
Paper: arxiv.org/abs/1911.04252 -
Quoc Le
@quocleix
Nice work on speeding up physics simulations with convolutional nets & neural architecture search.
-
Quoc Le
@quocleix
Happy to announce that we've released a number of models trained with Noisy Student (a semi-supervised learning method). The best model achieves 88.4% top-1 accuracy on ImageNet (SOTA).
Enjoy finetuning!
Link: github.com/tensorflow/tpu/tree/master/models/official/eff…
Paper: arxiv.org/abs/1911.04252Post not available here, view on X
Quoc Le
@quocleix
A nice video about the paper is below. Note that first version of the paper said 87.4% top-1 accuracy but since then we improved the result to 88.4%.
www.youtube.com/watch?v=Y8YaU9mv_us&t=3s
Quoc Le
@quocleix
Model checkpoints in PyTorch are here:
Ross Wightman
@wightmanr
A new batch of SOTA shattering EfficientNet Tensorflow weights released by @quocleix @QizheXie and team yesterday. Thanks! I got them up n running in #pytorch today. Bare-bones at github.com/rwightman/gen-efficientnet-pytorch
-
Quoc Le reposted
Thang Luong
@lmthang
About #meenabot data (asked by @GaryMarcus & others):
1. Only one occurrence of "Hayvard" in the training data.
2. The sentence that contains "Hayvard" has meaning similar to "Guess what, I obtained my bachelor from Hayvard."
3. No occurrence of "cow*" in the same conversation.
Quoc Le
@quocleix
My favorite conversation is below. The Hayvard pun was funny but I totally missed the steer joke at the end until it was pointed out today by @Blonkhart
-
Quoc Le reposted
Daniel De Freitas
@dan_defr
Enabling people to converse with chatbots about anything has been a passion of a lifetime for me, and I'm sure of others as well. So I'm very thankful to be able to finally share our results with you all. Hopefully, this will help inform efforts in the area. (1/4)
Thang Luong
@lmthang
Introducing #meenabot, a 2.6B-param open-domain chatbot with near-human quality. Remarkably, we show strong correlation between perplexity & humanlikeness!
Paper: arxiv.org/abs/2001.09977
Sample conversations: github.com/google-research/google-research/tree/master/meena -
Quoc Le reposted
Connor Shorten
@CShorten30
This video explains @GoogleAI 's amazing new Meena chatbot! An Evolved Transformer with 2.6B parameters on 341 GB / 40B words of conversation data to achieves remarkable chatbot performance!
"Horses go to Hayvard!"
youtu.be/STrrlLG15OY
#100daysofmlcode -
Quoc Le
@quocleix
New paper: Towards a Human-like Open-Domain Chatbot. Key takeaways:
1. "Perplexity is all a chatbot needs" ;)
2. We're getting closer to a high-quality chatbot that can chat about anything
Paper: arxiv.org/abs/2001.09977
Blog: ai.googleblog.com/2020/01/towards-conversational-agent-th…Post not available here, view on X
Quoc Le
@quocleix
My favorite conversation is below. The Hayvard pun was funny but I totally missed the steer joke at the end until it was pointed out today by @Blonkhart
Quoc Le
@quocleix
I had another conversation with Meena just now. It's not as funny and I don't understand the first answer. But the replies to the next two questions are quite funny.
-
Quoc Le reposted
Connor Shorten
@CShorten30
This video explains AdvProp from @GoogleAI! This technique leverages Adversarial Examples for ImageNet classification by using separate Batch Normalization layers for clean and adversarial mini-batches.
youtu.be/KTCztkNJm50
#100daysofmlcode -
Quoc Le reposted
Jeff Dean
@JeffDean
Some nice case studies about how @GCPcloud's AutoML products can help tackle real-world problems in visual inspection across a number of different manufacturing domains, being used by companies like Global Foundries and Siemens.
cloud.google.com/blog/products/ai-machine-learning/ai-and… -
Quoc Le
@quocleix
AdvProp: One weird trick to use adversarial examples to reduce overfitting.
Key idea is to use two BatchNorms, one for normal examples and another one for adversarial examples.
Significant gains on ImageNet and other test sets.
Mingxing Tan
@tanmingxing
Can adversarial examples improve image recognition? Check out our recent work: AdvProp, achieving ImageNet top-1 accuracy 85.5% (no extra data) with adversarial examples!
Arxiv: arxiv.org/abs/1911.09665
Checkpoints: git.io/JeopWPost not available here, view on X
Quoc Le
@quocleix
AdvProp improves accuracy for a wide range of image models, from small to large. But the improvement seems bigger when the model is larger.
Quoc Le
@quocleix
Pretrained checkpoints in Pytorch: github.com/rwightman/gen-efficientnet-pytorch
h/t to @wightmanr -
Quoc Le
@quocleix
EfficientDet: a new family of efficient object detectors. It is based on EfficientNet, and many times more efficient than state of art models.
Link: arxiv.org/abs/1911.09070
Code: coming soon
Quoc Le
@quocleix
Architecture of EfficientDet
Quoc Le
@quocleix
And latency on CPU and GPU:
-
Quoc Le
@quocleix
RandAugment was one of the secret sources behind Noisy Student that I tweeted last week. Code for RandAugment is now opensourced.
Barret Zoph
@barret_zoph
*New paper* RandAugment: a new data augmentation. Better & simpler than AutoAugment. Main idea is to select transformations at random, and tune their magnitude. It achieves 85.0% top-1 on ImageNet.
Paper: arxiv.org/abs/1909.13719
Code: git.io/Jeopl -
Quoc Le
@quocleix
Want to improve accuracy and robustness of your model? Use unlabeled data!
Our new work uses self-training on unlabeled data to achieve 87.4% top-1 on ImageNet, 1% better than SOTA. Huge gains are seen on harder benchmarks (ImageNet-A, C and P).
Link: arxiv.org/abs/1911.04252Post not available here, view on X
Quoc Le
@quocleix
Method is also super simple:
1) Train a classifier on ImageNet
2) Infer labels on a much larger unlabeled dataset
3) Train a larger classifier on the combined set
4) Iterate the process, adding noise
Quoc Le
@quocleix
I also highly recommend this nice video that explains the paper very well:
www.youtube.com/watch?v=Y8YaU9mv_us -
Quoc Le reposted
Ruben Villegas
@RubenEVillegas
Check out our study on the effects of inductive bias and model capacity in video prediction models. This is work with @arkanathpathak @harinidkannan @doomie @quocleix @honglaklee
Paper: arxiv.org/abs/1911.01655
Website: sites.google.com/view/videopredictioncapacity -
Quoc Le reposted
Yifeng Lu
@yifenglou
Congratulations, Kaggle and Cloud teams. Glad to see our AutoML research could help data scientists on Kaggle. @quocleix @dhuangcn
Kaggle
@kaggle
We're thrilled to bring the power of AutoML to Kaggle! Read more about the integration and why we're excited to put it in the hands of our 3.5M strong community in this blog post by @GoogleCloud: cloud.google.com/blog/products/ai-machine-learning/bringi…
-
Quoc Le reposted
Google AI
@GoogleAI
Neural architecture evolution, a new method for automatically finding optimal neural networks for video understanding, has yielded architectures that outperform existing hand-made models and show improvements to network runtime of 10-100x. Learn more ↓ goo.gle/32siUVa
-
Quoc Le reposted
Jeff Dean
@JeffDean
Nice evaluation of @GCPcloud AutoML!
"This article shows that physicians with no coding experience can use automated deep learning to develop algorithms that can do clinical classification tasks to a level comparable with traditional deep learning models ..."
Pearse Keane
@pearsekeane
Soooo excited to see this out @LancetDigitalH - deep learning for medical image classification by doctors @Moorfields with NO coding experience using @googlecloud #deeplearning #automl #opensource authors.elsevier.com/sd/article/S2589-7500(19)30108-6
-
Quoc Le reposted
Gamaleldin Elsayed
@gamaleldinfe
Our paper "Saccader: Improving Accuracy of Hard Attention Models for Vision" with @skornblith and @quocleix is now online. We design a novel hard attention model that achieves good accuracy on ImageNet while providing more interpretable predictions.
arxiv.org/abs/1908.07644 -
Quoc Le reposted
Thang Luong
@lmthang
Wanna make your #nlp datasets larger? Check out this simple tutorial "Data Augmentation by Backtranslation (DAB)" github.com/vietai/dab, interactive demos & pretrained models in English, French, & Vietnamese! A community project led by @thtrieu_ with @tcs2711, @joocxi. #nlproc
New Turing Institute
@newturing
Check out #vietai first github repository: Data Augmentation by Backtranslation (DAB) github.com/vietai/dab, made easy with interactive demos, pretrained models, and a case-study on low-resource languages! Brought to you by @thtrieu_, @tcs2711, @joocxi. #nlpproc
-
Quoc Le reposted
Mingxing Tan
@tanmingxing
Introducing EfficientNet-EdgeTPU: customized for mobile accelerators, with higher accuracy and 10x faster inference speed.
blog post: ai.googleblog.com/2019/08/efficientnet-edgetpu-creating.html
Code and pertained models: github.com/tensorflow/tpu/tree/master/models/official/eff…
Google AI
@GoogleAI
Announcing EfficientNet-EdgeTPU, a family of image classification models created using #automl and optimized for use on #edgetpu, that provide real-time performance while achieving the accuracy of much larger, server-side models. Check it out below ↓ goo.gle/2MK8Vpe
-
Quoc Le
@quocleix
We released all checkpoints and training recipes of EfficientNets, including the best model EfficientNet-B7 that achieves accuracy of 84.5% top-1 on ImageNet.
Link: github.com/tensorflow/tpu/tree/master/models/official/eff…
Quoc Le
@quocleix
For more context about EfficientNet, check out my earlier tweet:
Quoc Le
@quocleix
EfficientNets: a family of more efficient & accurate image classification models. Found by architecture search and scaled up by one weird trick.
Link: arxiv.org/abs/1905.11946
Github: github.com/tensorflow/tpu/tree/master/models/official/eff…
Blog: research.google/blog/efficientnet-improving-accuracy-and-… -
Quoc Le reposted
Mingxing Tan
@tanmingxing
Introducing MixNet: AutoML + a new mixed depthwise conv (MDConv). SOTA results for mobile: 78.9% ImageNet top-1 accuracy under typical mobile settings (<600M FLOPS).
Paper: arxiv.org/abs/1907.09595
Code & models: github.com/tensorflow/tpu/tree/master/models/official/mna… -
Quoc Le reposted
Russ Salakhutdinov
@rsalakhu
A Fair Comparison Study of XLNet and BERT with Large Models: medium.com/@xlnet.team/a-fair-comparison-study-of-xlnet-a…
-
Quoc Le reposted
Google AI
@GoogleAI
Recent work on "Unsupervised Data Augmentation" (UDA) reveals that better data augmentation leads to better semi-supervised learning, with state-of-the-art results on various language and vision benchmarks, using one or two orders of magnitude less data. goo.gle/2G3Tq7u
-
Quoc Le reposted
Kevin Clark
@clark_kev
BAM! Our new #acl2019 paper presents "Born-Again Multi-Task Networks," a simple way to improve multi-task learning using knowledge distillation. With @lmthang @ukhndlwl @quocleix @chrmanning. Paper: arxiv.org/pdf/1907.04829.pdf Code: github.com/google-research/google-research/tree/master/bam
-
Quoc Le
@quocleix
We opensourced AutoAugment strategy for object detection. This strategy significantly improves detection models in our benchmarks. Please try it on your problems.
Code: github.com/tensorflow/tpu/tree/master/models/official/det…
Paper: arxiv.org/abs/1906.11172
More details & results 👇
Ekin Dogus Cubuk
@ekindogus
Data augmentation is even more crucial for detection. We present AutoAugment for object detection, achieving SOTA on COCO validation set (50.7 mAP). Policy transfers to different models & datasets. Paper: arxiv.org/abs/1906.11172, Code: github.com/tensorflow/tpu/tree/master/models/official/det…, details in thread.
-
Quoc Le reposted
Azalia Mirhoseini
@Azaliamirh
I’m very honored to have received the MIT TR 35 innovator award for my work on #mlforsystems. Grateful to have been able to work with some of the most wonderful mentors and collaborators thus far in my career!
MIT Technology Review
@techreview
Announcing the new class of #35innovatorsunder35. Past honorees include Mark Zuckerberg, Sergey Brin, Larry Page, and Jonny Ive. This crew should be just as successful. Here's your chance to meet them first. www.technologyreview.com/innovators-under-35/2019
-
Quoc Le
@quocleix
XLNet: a new pretraining method for NLP that significantly improves upon BERT on 20 tasks (e.g., SQuAD, GLUE, RACE)
arxiv: arxiv.org/abs/1906.08237
github (code + pretrained models): github.com/zihangdai/xlnet
with Zhilin Yang, @ZihangDai, Yiming Yang, Jaime Carbonell, @rsalakhu -
Quoc Le reposted
Thang Luong
@lmthang
Introducing Selfie, an unsupervised pretraining method that extends the concept of masked language model (e.g., BERT) towards continuous data, such as images. Overall challenging, but promising in low-data regime! Joint work with @thtrieu_ @quocleix arxiv.org/abs/1906.02940
-
Quoc Le
@quocleix
EfficientNets: a family of more efficient & accurate image classification models. Found by architecture search and scaled up by one weird trick.
Link: arxiv.org/abs/1905.11946
Github: github.com/tensorflow/tpu/tree/master/models/official/eff…
Blog: research.google/blog/efficientnet-improving-accuracy-and-… -
Quoc Le
@quocleix
Nice blog post titled "The Quiet Semi-Supervised Revolution" by Vincent Vanhoucke. It discusses two related works by the Google Brain team: Unsupervised Data Augmentation and MixMatch.
towardsdatascience.com/the-quiet-semi-supervised-revoluti…Post not available here, view on X
Quoc Le
@quocleix
Key idea in these two papers is to ensure prediction(x) = prediction(x + noise) , where x is an unlabeled example.
People have tried all kind of noise, e.g., Gaussian noise, adversarial noise etc. But it looks like data augmentation noise is the real winner.
Quoc Le
@quocleix
To add to Vincent's point above, new findings also include:
1. The method is general (works well for images & texts).
2. The method works well on top of transfer learning (e.g., BERT).
You can find these results in Unsupervised Data Augmentation paper: arxiv.org/abs/1904.12848 -
Quoc Le reposted
Jascha Sohl-Dickstein
@jaschasd
A careful empirical study of the effect of network width on generalization and fixed learning rate SGD, for MLPs, convnets, resnets, and batch norm. With superstar resident Daniel Park, and @quocleix + Sam Smith. arxiv.org/abs/1905.03776
-
Quoc Le reposted
Google AI
@GoogleAI
Interested in applying machine learning to tabular data? Check out our work on an end-to-end AutoML solution, making ML scalable and accessible for both research and industry applications, and see how it performed in the recent @Kaggle_Days challenge! goo.gle/2VRDwGj
-
Quoc Le
@quocleix
Introducing MobileNetV3: Based on MNASNet, found by architecture search, we applied additional methods to go even further (quantization friendly SqueezeExcite & Swish + NetAdapt + Compact layers). Result: 2x faster and more accurate than MobileNetV2. Link: arxiv.org/abs/1905.02244
-
Quoc Le reposted
Mihail Eric
@mihail_eric
Yum! Unsupervised data augmentation that works from @GoogleAI
@QizheXie @quocleix. New state-of-the-art on various language and vision tasks: arxiv.org/pdf/1904.12848.pdf -
Quoc Le reposted
Ran Reichman
@ranreichman
I just published "Attention Augmented Convolutional Networks", explaining a great @GoogleAI paper presenting a new way to add attention to CNNs. Great results with no significant compute/memory cost. Congrats @IrwanBello @quocleix @ashVaswani @barret_zoph www.lyrn.ai/2019/05/03/attention-augmented-convolutional-…
-
Quoc Le
@quocleix
Data augmentation is often associated with supervised learning. We find *unsupervised* data augmentation works better. It combines well with transfer learning (e.g. BERT) and improves everything when datasets have a small number of labeled examples. Link: arxiv.org/abs/1904.12848
Thang Luong
@lmthang
Introducing UDA, our new work on "Unsupervised data augmentation" for semi-supervised learning (SSL) with Qizhe Xie, Zihang Dai, Eduard Hovy, & @quocleix. SOTA results on IMDB (with just 20 labeled examples!), SSL Cifar10 & SVHN (30% error reduction)! arxiv.org/abs/1904.12848
Quoc Le
@quocleix
Key idea behind the method is consistency training: model prediction on an unlabeled example and model prediction on an *augmented* unlabeled example are consistent. This idea is also related to some recent papers and our own paper on Cross View Training: arxiv.org/abs/1809.08370
Quoc Le
@quocleix
Results with Unsupervised Data Augmentation: It achieves SOTA on IMDb with 20 labeled examples (~1000x less labeled data than other methods), and achieves SOTA on CIFAR10 4k, SVHN 1k.
-
Quoc Le
@quocleix
Exciting new work on replacing convolutions with self-attention for vision. Our paper shows that full attention is good, but loses a few percents in accuracy. And a middle ground that combines convolutions and self-attention is better. Link: arxiv.org/abs/1904.09925
-
Quoc Le
@quocleix
Wanted to apply AutoAugment to speech, but a handcrafted augmentation policy already improves SOTA. Idea: randomly drop out certain time & frequency blocks, and warp input spectrogram. Results: state-of-art on LibriSpeech 960h & Switchboard 300h. Link: arxiv.org/abs/1904.08779
Google AI
@GoogleAI
Automatic Speech Recognition (ASR) struggles in the absence of an extensive volume of training data. We present SpecAugment, a new approach to augmenting audio data that treats it as a visual problem rather than an audio one. Learn more at → blog.research.google/2019/04/specaugment-new-data-augment…
-
Quoc Le
@quocleix
We used architecture search to find a better architecture for object detection. Results: Better and faster architectures than Mask-RCNN, FPN and SSD architectures. Architecture also looks unexpected and pretty funky. Link: arxiv.org/abs/1904.07392
-
Quoc Le
@quocleix
Had fun talking to @tsimonite from @WIRED about Google AutoML during the live Kaggle competition last Thursday.
www.wired.com/story/googles-ai-experts-try-automate-thems… -
Quoc Le reposted
Ekin Dogus Cubuk
@ekindogus
Latest version of the AutoAugment paper is up: arxiv.org/abs/1805.09501. Stop by our oral presentation at CVPR to learn more! Joint work with @barret_zoph @decentralion Vijay Vasudevan and @quocleix.
-
Quoc Le
@quocleix
Update from #kaggledays , 5 hours into the competition and Google AutoML still maintains its lead. Three hours to go (five hours since I took the pictures).
Quoc Le
@quocleix
One hour to the final deadline and Google AutoML is dropped to the second place. Humans are still good :)
Quoc Le
@quocleix
Final results are in, and Google AutoML came in second in the final ranking.
-
Quoc Le
@quocleix
Exciting new AutoML lineup for @googlecloud : AutoML Tables & Mobile Vision. Collaboration between Google Brain + AI + Cloud. Some benchmark data are below. Google Brain AutoML team will also participate in a Kaggle competition tomorrow. This will be fun!
cloud.google.com/blog/products/ai-machine-learning/expand… -
Quoc Le reposted
Anthony Goldbloom
@antgoldbloom
We're hosting a #kaggledays conference in San Francisco on April 10 and 11 (kaggledays.com/sanfrancisco). Speakers include Kaggle Grandmasters @mikb0b, @DmitryLarko and @tunguz, Keras creator @fchollet, XGBoost contributor & Kaggle Grandmaster Bing Xu and Google Brain's @quocleix.
-
Quoc Le reposted
Sara Hooker
@sarahookr
Wow, great to (belatedly) find out this work w @skornblith, @quocleix, Shlens was recognized with an oral at CVPR 2019. arxiv.org/abs/1805.08974 Wealth of empirical data + I find myself regularly returning to the appendix for hyperparameters listed for image datasets
-
Quoc Le reposted
Rani Horev
@rani_horev
A new summary of The Evolved Transformer!
The recipe:
1. Take the human-designed Transformer
2. Add a new neural architecture search - PDH
3. Whisk with ~200 TPUs to get a much better model - ET
www.lyrn.ai/2019/03/12/the-evolved-transformer/
By Google Brain (@quocleix) -
Quoc Le reposted
Google AI
@GoogleAI
Check out GPipe, an open source library that uses pipeline parallelism to scale up the training of deep neural networks, helping to overcome the memory limitation on currently available accelerators. research.google/blog/introducing-gpipe-an-open-source-lib…
-
Quoc Le reposted
John Platt
@johnplattml
Benchmarking Neural Architecture Search by precomputing performance of 423k different convnets on cifar-10 (on #tpu ). arxiv.org/abs/1902.09635 Clever idea! Random search gets you pretty far. #deeplearning
-
Quoc Le
@quocleix
For me, ELMo is my favorite work of 2018. Together with ULMFit, CoVE, GPT, BERT and others, they represent a breakthrough. I just mean that these ideas have a history, and want to bring up some history from my part. Also check out some comments in the thread for related works.
Quoc Le
@quocleix
Peters et al 2017 and 2018 (ELMo) get the credit for the discoveries of 1) Using language models for transfer learning and 2) Embedding words through a language model ("contextualized word vectors"). ELMo is great, but these methods were proposed earlier. (1/3)
-
Quoc Le
@quocleix
Peters et al 2017 and 2018 (ELMo) get the credit for the discoveries of 1) Using language models for transfer learning and 2) Embedding words through a language model ("contextualized word vectors"). ELMo is great, but these methods were proposed earlier. (1/3)
Quoc Le
@quocleix
First, the idea of using pre-trained language models for transfer learning was proposed in our paper in 2015: arxiv.org/abs/1511.01432 . (2/3)
Quoc Le
@quocleix
Second, the idea of embedding words through a language model was proposed in our followup paper in 2016: arxiv.org/abs/1611.02683 . (3/3)
-
Quoc Le
@quocleix
Since our work on "Semi-supervised sequence learning", ELMo, BERT and others have shown changes in the algorithm give big accuracy gains. But now given these nice results with a vanilla language model, it's possible that a big factor for gains can come from scale. Exciting!
OpenAI
@OpenAI
We've trained an unsupervised language model that can generate coherent paragraphs and perform rudimentary reading comprehension, machine translation, question answering, and summarization — all without task-specific training: blog.openai.com/better-language-models/
-
Quoc Le reposted
trieu
@thtrieu_
As also observed by OpenAI's GPT-2, training data quality is important. We release the STORIES corpus introduced in our work arxiv.org/abs/1806.02847. The corpus is a high quality subset of CommonCrawl with a total of ~7B words (~32GB) can be found here: console.cloud.google.com/storage/browser/commonsense-reas…
-
Quoc Le
@quocleix
Evolved Transformer is now opensourced in Tensor2Tensor:
github.com/tensorflow/tensor2tensor/blob/master/tensor2te…
Quoc Le
@quocleix
We used architecture search to improve Transformer architecture. Key is to use evolution and seed initial population with Transformer itself. The found architecture, Evolved Transformer, is better and more efficient, especially for small size models. Link: arxiv.org/abs/1901.11117
-
Quoc Le
@quocleix
We used architecture search to improve Transformer architecture. Key is to use evolution and seed initial population with Transformer itself. The found architecture, Evolved Transformer, is better and more efficient, especially for small size models. Link: arxiv.org/abs/1901.11117
-
Quoc Le
@quocleix
Google AI's blog post about our work on Transformer-XL.
Google AI
@GoogleAI
Introducing Transformer-XL, a novel architecture that enables natural language understanding beyond a fixed-length context. You can learn more and find the paper, code, pretrained models and hyperparameters at ↓ research.google/blog/transformer-xl-unleashing-the-potent…
-
Quoc Le
@quocleix
Introducing Natural Questions, a new dataset for Q&A research. Many questions in the dataset are not yet answered well by Google:
where does the energy in a nuclear explosion come from?
when are hops added to the brewing process?
The dataset is still difficult for SoTA models.
Google AI
@GoogleAI
Introducing Natural Questions, a new, large-scale corpus and challenge for training and evaluating open-domain question answering systems, and the first to replicate the end-to-end process in which people find answers to questions. Learn more at ↓ research.google/blog/natural-questions-a-new-corpus-and-c…
-
Quoc Le reposted
hardmaru
@hardmaru
Transformer-XL: Combining Transformers and RNNs Into a State-of-the-art Language Model
Blog post by @HorevRani giving an overview of the model and key concepts such as the recurrence mechanism and the relative positional encoding scheme.
www.lyrn.ai/2019/01/16/transformer-xl-sota-language-model/
hardmaru
@hardmaru
Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context
Up to 1800x faster than vanilla Transformer during evaluation. New SoTA results on Wikipedia (enwik8, text8, WikiText-103), One Billion Words, and PennTree Bank. 🔥
arxiv.org/abs/1901.02860 -
Quoc Le reposted
Google AI
@GoogleAI
We are happy to collaborate with @Waymo researchers to automate the design of #machinelearning models to make self-driving cars a reality. For more details, check out their blog post at medium.com/waymo/automl-automating-the-design-of-machine-…
-
Quoc Le
@quocleix
Blog post about our collaboration to automate the design of machine learning models for @Waymo 's perception problems.
Waymo
@Waymo
Our researchers have teamed up with @GoogleAI to put cutting-edge AutoML research into practice by automatically generating neural nets for our self-driving cars. The results? Faster and more accurate nets for our vehicles. medium.com/waymo/automl-automating-the-design-of-machine-…
-
Quoc Le
@quocleix
Our work on transformer-xl which uses smart caching to improve the learning of long-term dependency in transformer. Key results: state-of-art on five language modeling benchmarks, including ppl of 21.8 on LM1B and 0.99 on enwiki8. Link: arxiv.org/abs/1901.02860
-
Quoc Le reposted
Thang Luong
@lmthang
Nice summary of top NLP ideas. Glad to see our work on “Cross-view training” with @clark_kev, @quocleix, and @chrmanning got featured arxiv.org/abs/1809.08370. I’d also vote for HotpotQA a very well-designed multi-hop reasoning dataset!
Sebastian Ruder
@seb_ruder
10 Exciting Ideas of 2018 in NLP: A collection of 10 ideas that I found exciting and impactful this year—and that we'll likely see more of in the future.
ruder.io/10-exciting-ideas-of-2018-in-nlp/ -
Quoc Le reposted
Matthew Honnibal
@honnibal
The "Cross-View Training" paper by Clark et al came out two weeks before BERT, and I think it's got lost in the noise. It's worth paying attention to: their self-training trick is very elegant, and seems easy to apply and extend. arxiv.org/abs/1809.08370
-
Quoc Le reposted
Mohammad Norouzi
@mo_norouzi
Chen did a great job presenting memory augmented policy optimization (mapo) for program synthesis. This is a simple and effective tool for variance reduction.
Come by our poster #137 in Room 517 #neurips18 @crazydonkey200 @jonathanberant @quocleix @lao_ni -
Quoc Le
@quocleix
This blog post did a great job at summarizing our recent GPipe paper.
Ran Reichman
@ranreichman
Very cool breakthrough. I wrote a short summary of the main ideas in the GPipe paper, hope it helps people understand it. www.lyrn.ai/2018/11/30/gpipe-training-giant-neural-nets-u…
-
Quoc Le
@quocleix
Introducing MAPO, an policy gradient method augmented with memory of good trajectories to make better updates. Well suited for generating programs to query databases. Good accuracies on WikiTableQuestions and WikiSQL. Video: www.youtube.com/watch?v=HyUA_VQur-4 #neurips2018
Quoc Le
@quocleix
Link to paper: arxiv.org/abs/1807.02322
Link to code: github.com/crazydonkey200/neural-symbolic-machines -
Quoc Le
@quocleix
We studied transfer learning from ImageNet to other datasets. Finding: better ImageNet architectures tend to work better on other datasets too. Surprise: pretraining on ImageNet dataset sometimes doesn't help very much. More info: arxiv.org/abs/1805.08974
Simon Kornblith
@skornblith
We updated our paper "Do Better ImageNet Models Transfer Better?" arxiv.org/abs/1805.08974
In v2, we show that regularization settings for ImageNet training matter a lot for transfer learning on fixed features. ImageNet accuracy now correlates with transfer acc in all settings. -
Quoc Le
@quocleix
Bigger models are better models so we built GPipe to enable training of large models. Results: 84.3% on ImageNet with AmoebaNet (big jump from other state-of-art models) and 99% on CIFAR-10 with transfer learning. Link: arxiv.org/abs/1811.06965
-
Quoc Le reposted
Stanford NLP Group
@stanfordnlp
Much of the #nlproc action this week is at #emnlp2018 in Brussels, but back home at @Stanford, we’re delighted to have @AdamsYu telling us all about his amazing work on using an efficient Transformer architecture for effective Neural Question Answering. arxiv.org/abs/1804.09541
-
Quoc Le
@quocleix
Our latest paper on DropBlock, a new regularization method for convolutional networks. DropBlock drops continuous region of a feature map. It improves ImageNet top1 of ResNet from 76.5% to 78.1%, and COCO mAP of RetinaNet from 36.8% to 38.4%.
Paper: arxiv.org/abs/1810.12890 -
Quoc Le
@quocleix
Auto-Augment policies are now opensourced in TF-Hub.
TensorFlow
@TensorFlow
Image augmentation lets you get the most of your dataset. We released state-of-the-art AutoAugment Modules on #tfhub allowing you to train better image models with less data! #tensorflowhub #googleai
Check it out here → tfhub.dev/s?keywords=image_augmentation&linkId=58679401 -
Quoc Le
@quocleix
Look forward to speaking at AI Frontiers Conference on "Using Machine Learning to Automate Machine Learning" Nov 9. See you there. @AI_Frontiers @GoogleAI aifrontiers.com
-
Quoc Le
@quocleix
Great progress in NLP using unsupervised pre-trained bidirectional language model.
Thang Luong
@lmthang
A new era of NLP has just begun a few days ago: large pretraining models (Transformer 24 layers, 1024 dim, 16 heads) + massive compute is all you need. BERT from @GoogleAI: SOTA results on everything arxiv.org/abs/1810.04805. Results on SQuAD are just mind-blowing. Fun time ahead!
Quoc Le
@quocleix
I also want to point out that the masked LM idea is similar in principle to the idea of sequence autoencoders with input word dropout and embedding dropout and also LM that @andrewdai & I tried in 2015: arxiv.org/abs/1511.01432
-
Quoc Le reposted
Jeff Dean
@JeffDean
If you want to do ML research, consider applying for the 2019 Google AI Residency program! You'll have the opportunity to conduct cutting-edge research working in a wide variety of areas, and this year we're expanding to host residents in even more locations.
Google AI
@GoogleAI
Applications for the 2019 Google AI Residency program are now open! Visit g.co/airesidency/apply for more information on how to apply. To learn more about the accomplishments of the recently graduated second class of residents, visit ↓ research.google/blog/highlights-from-the-google-ai-reside…
-
Quoc Le
@quocleix
Interesting analysis of keywords in ICLR papers.
Sasha Rush
@srush_nlp
First pass at up <-> down keywords. Seems like "meta-learning", "imitation learning", and "architecture" search are up 🚀. Whereas "gan", "rnn", and "nlp" are down. 😬
-
Quoc Le reposted
Google Open Source
@GoogleOSS
Researchers on the Google Brain team, @ekindogus, @barret_zoph, and @quocleix, open sourced AutoAugment, a learning algo that improves on the performance of the previous best computer vision models. Find code and white paper at github.com/tensorflow/models/tree/master/research/autoaug….
Google AI
@GoogleAI
Introducing AutoAugment, a reinforcement learning algorithm which increases both the amount and diversity of existing data by finding optimal image augmentation policies, leading to state-of-the-art performance in computer vision models. Learn more at blog.research.google/2018/06/improving-deep-learning-perf…
-
Quoc Le reposted
Google AI
@GoogleAI
Google Brain team researchers @ekindogus, @barret_zoph, and @quocleix have open sourced the AutoAugment image classification policies that achieve state-of-the-art results. Check out the the paper and code is located in tensorflow/models/research/autoaugment. #automl
Google AI
@GoogleAI
Introducing AutoAugment, a reinforcement learning algorithm which increases both the amount and diversity of existing data by finding optimal image augmentation policies, leading to state-of-the-art performance in computer vision models. Learn more at blog.research.google/2018/06/improving-deep-learning-perf…
-
Quoc Le reposted
Thang Luong
@lmthang
Excited to share our #emnlp2018 work: SOTA on many NLP tasks where performances already feel saturated! ideas: models self-train on unlabeled data by refraining access from full representations of text sequences + multitask-traing. by @clark_kev, @chrmanning, @quocleix & I.
Stanford NLP Group
@stanfordnlp
Cross-View Training—A semi-supervised learning technique by @clark_kev @lmthang Quoc Le @chrmanning at #emnlp2018. Allows training for your #nlproc task on large-scale unannotated data, not only using such data for task-agnostic word representations arxiv.org/abs/1809.08370

































































































































































































































































































