1/6 "We want a robot that doesn’t only work in factory settings, [but] ideally would work in our homes," said TTIC's Matthew Walter, in his talk on "Learning from Visual Experience: Toward Generalizable Robot Manipulation." Video: www.youtube.com/watch?v=WKGIFAGu7t4
Anil Ananthaswamy
Anil Ananthaswamy is an Indian-American award-winning science journalist, author, and the former deputy news editor at New Scientist. His writings, on physics, astronomy, quantum theory, neuroscience, and computer science, have regularly featured on publications including New Scientist, Quanta, Scientific American, PNS Front Matter, Nature, Nautilus, Matter, The Wall Street Journal, Discover, and the UK's Literary Review. In 2019, Ananthaswamy was fellow under the Knight Science Journalism program at the Massachusetts Institute of Technology. In 2024, his latest book, Why Machines Learn, received widespread acclaim, with Nobel laureate and AI pioneer Geoff Hinton, labelling it a "masterpiece." More on Wikipedia
-
Anil Ananthaswamy reposted
TTIC
@TTIC_Connect
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
"Diffusion models are used way beyond generating images. They are used to generate diffusion policies for self-driving cars, even protein design. The application area for diffusion models is ever expanding," said @Grigoris_c of @UWMadison during his talk at the Simons Institute.
-
Anil Ananthaswamy reposted
TTIC
@TTIC_Connect
1/4 Out-of-distribution generalization is "one of the biggest bottlenecks...[to]...real deployment of machine learning in many practical scenarios, such as in autonomous driving, chip design, drug design, etc.," said Yusu Wang of @UCSD at TTIC. Video: www.youtube.com/watch?v=toRAey5VZuM
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
1/ 3 "The progress in reasoning models has been really crazy recently...But the frontier is quite jagged. Despite all of this progress, there are some very weird shortcomings that still exist, even in the latest models," said Mahdi Soltanolkotabi of @USC at the Simons Institute.
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
2/2 Their task: to discuss the immediate future of theoretical computer science in light of the challenges posed by AI. Read a report of that closed-door meeting, in the Polynomial Times (see Theoretical Computer Science Meets Its AI Moment, page 3): simons.berkeley.edu/PolyTimes-2026-27?inline
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
1/2 The day after OpenAI dropped a bombshell about "a solution to the Navier–Stokes existence and smoothness problem," 32 theoretical computer scientists met for a closed door meeting (that had been planned for months) at the Simons Institute for the Theory of Computing.
-
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
Steven Strogatz and Alex Townsend's new book, Big Math: The Hidden Codes That Run Our World, is hugely prescient. It's the story of the math that underpins so much of our lives and how we got to our AI-suffused moment, and brilliantly anticipates the current crisis facing mathematicians. Strogatz and Townsend end with these poetic paragraphs, about the Faustian bargain we've made with machines:
"At a time when math moves faster than our own understanding, when our creations surprise and sometimes terrify us, the answer cannot be to turn away. We’ve got to keep striving — with humility, care, and every bit of wisdom we can muster.
Striving, in this new age, means remembering that every mathematical invention begins and ends with human choices. We might never fully understand the oracles we’ve made, but we can decide what to ask of them, and why. We can remain the conscience behind the code.
The mathematician in each of us will always want to know more; the human being must insist that knowledge serve something larger than itself. That might be the work of our century: letting our power grow without losing our soul," write @stevenstrogatz and Townsend.
My review of the book for Nature (non-paywalled version, courtesy of Springer Nature): idp.nature.com/transit?redirect_uri=https%3A%2F%2Fwww.nat… -
Anil Ananthaswamy
@anilananth
Steven Strogatz and Alex Townsend's new book, Big Math: The Hidden Codes That Run Our World, is hugely prescient. It's the story of the math that underpins so much of our lives and how we got to our AI-suffused moment, and brilliantly anticipates the current crisis facing mathematicians. Strogatz and Townsend end with these poetic paragraphs, about the Faustian bargain we've made with machines:
"At a time when math moves faster than our own understanding, when our creations surprise and sometimes terrify us, the answer cannot be to turn away. We’ve got to keep striving — with humility, care, and every bit of wisdom we can muster.
Striving, in this new age, means remembering that every mathematical invention begins and ends with human choices. We might never fully understand the oracles we’ve made, but we can decide what to ask of them, and why. We can remain the conscience behind the code.
The mathematician in each of us will always want to know more; the human being must insist that knowledge serve something larger than itself. That might be the work of our century: letting our power grow without losing our soul," write @stevenstrogatz and Townsend.
My review of the book for Nature (non-paywalled version, courtesy of Springer Nature): idp.nature.com/transit?redirect_uri=https%3A%2F%2Fwww.nat… -
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
My review of @stevenstrogatz and Alex Townsend's Big Math: The Hidden Codes That Run Our World for @Nature. www.nature.com/articles/d41586-026-03627-6 AI has transformed maths — this is how we got here The story of how maths came to run the world, and created the artificial-intelligence models that might surpass human mathematicians.
My favorite paras in the book come at the very end, as Strogatz and Townsend talk about the insights into the nature of reality that mathematics gives us, along with joy and wonder, and then ponder the potential loss of this insight in the age of AI.
"But like childhood, insight may be a phase, tender and luminous, yet inevitably outgrown. The thought has a sadness to it, but also a clarity. It’s the latest instance of the Copernican lesson: Just as early astronomers surrendered geocentrism for a deeper truth, we humans are now learning that we are not the center of the intellectual cosmos. Our style of knowing is not the only one, or the final one.
Feeling that sense of humility is a kind of wisdom in itself, a recognition that reality was never ours to fathom, only to touch, to witness, to wonder at through the narrow peepholes of our minds. Maybe it’s time to let machines take the lead, unbound by what we hominids can muster." -
Anil Ananthaswamy
@anilananth
My review of @stevenstrogatz and Alex Townsend's Big Math: The Hidden Codes That Run Our World for @Nature. www.nature.com/articles/d41586-026-03627-6 AI has transformed maths — this is how we got here The story of how maths came to run the world, and created the artificial-intelligence models that might surpass human mathematicians.
My favorite paras in the book come at the very end, as Strogatz and Townsend talk about the insights into the nature of reality that mathematics gives us, along with joy and wonder, and then ponder the potential loss of this insight in the age of AI.
"But like childhood, insight may be a phase, tender and luminous, yet inevitably outgrown. The thought has a sadness to it, but also a clarity. It’s the latest instance of the Copernican lesson: Just as early astronomers surrendered geocentrism for a deeper truth, we humans are now learning that we are not the center of the intellectual cosmos. Our style of knowing is not the only one, or the final one.
Feeling that sense of humility is a kind of wisdom in itself, a recognition that reality was never ours to fathom, only to touch, to witness, to wonder at through the narrow peepholes of our minds. Maybe it’s time to let machines take the lead, unbound by what we hominids can muster."
Anil Ananthaswamy
@anilananth
Here's online access to the article shared by the publisher: www.nature.com/articles/d41586-026-03627-6.epdf?sharing_t…
-
Anil Ananthaswamy reposted
Lenore Blum
@BlumLenore
How A.I. Just Blew Up a Key Tenet of Computer Science www.nytimes.com/2026/11/06/science/ai-exponential-time-hy… via @NYTimes
@rrwilliams @RussellImpaglia -
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
Theoretical Computer Science Meets Its AI Moment
The day after OpenAI dropped a bombshell about "a solution to the Navier–Stokes existence and
smoothness problem," 32 theoretical computer scientists met for a 2-day closed door meeting (that had been planned for months) at the Simons Institute for the Theory of Computing, to discuss and debate their field's response to AI. In this issue of the @SimonsInstitute's annual magazine, Polynomial Times, is my report of that meeting. Read the article and the entire issue here: simons.berkeley.edu/PolyTimes-2026-27?inline -
Anil Ananthaswamy
@anilananth
Theoretical Computer Science Meets Its AI Moment
The day after OpenAI dropped a bombshell about "a solution to the Navier–Stokes existence and
smoothness problem," 32 theoretical computer scientists met for a 2-day closed door meeting (that had been planned for months) at the Simons Institute for the Theory of Computing, to discuss and debate their field's response to AI. In this issue of the @SimonsInstitute's annual magazine, Polynomial Times, is my report of that meeting. Read the article and the entire issue here: simons.berkeley.edu/PolyTimes-2026-27?inline -
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
1/3 What aspects of a large language model are actually determined by the training data itself? "This might seem like a nonsensical question...[But] not all aspects of your model are determined by the data," said Pradeep Ravikumar of @CarnegieMellon at the Simons Institute.
-
Anil Ananthaswamy reposted
TTIC
@TTIC_Connect
1/6 "We already have robots that can do really well on specialized tasks, but they struggle to do everyday tasks that humans take for granted in realistic, cluttered, real-world situations," said Osbert Bastani of @Penn at his TTIC Colloquium talk. Video: www.youtube.com/watch?v=v54F4bzFvvc
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
1/3 AI's Grand Unification Problem is about combining "fluidity" (the ability to generate language, images and video) with "causal understanding," said @Columbia's Elias Bareinboim at the Simons Institute workshop on Trustworthy AI: From Hallucinations to Reliable Autonomy.
-
Anil Ananthaswamy reposted
TTIC
@TTIC_Connect
1/6 LLMs with CoT reasoning are powerful, but can be incorrect, even unsafe. “You might like to train...a verifier to interface with these generators to make them more reliable, safe, aligned with your values, and so on," said Avrim Blum, President of TTIC www.youtube.com/watch?v=qyzuz6qlW9Q
-
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
The 2026 Nobel Prize for Physics
"We built it, and we'll see what comes," Francis Halzen told me back in 2007. IceCube was his Field of Dreams. Today, the dreamer and the instrument win the Nobel Prize in Physics. I visited the South Pole in 2007 to observe IceCube being built and wrote about it in The Edge of Physics.
An excerpt from the book:
IceCube has turned the icy wastes of Antarctica into a neutrino physicist’s field of dreams. “Everybody thinks something will be seen at a cubic-kilometer scale,” said Halzen. “I’m not responsible for what is emitting neutrinos in the universe. We developed the technique, we built it, and we’ll see what comes.”
IceCube is venturing into the unknown. It’s a discovery instrument, not unlike the proposed Square Kilometre Array or the Large Hadron Collider near Geneva, Switzerland. These behemoths will serve physics for decades, peering through the mists of time to let us glimpse our very beginnings.
Edward Wilson, who accompanied [Robert Falcon] Scott to the South Pole and died with him on the way back to McMurdo, might as well have been speaking about the IceCube scientists when he penned these words during his brief sojourn in Antarctica:
And this was the thought that the Silence wrought
As it scorched and froze us through
For the secrets hidden are all forbidden
Till God means man to know,
We might be the men God meant should know . . . -
Anil Ananthaswamy reposted
Muthu / முத்து
@muthutalks
Congratulations to stubborn particle physicist and 2026 laureate Francis Halzen; the first time I heard about IceCube was from Indian American science writer Anil Ananthaswamy in his first pop science book:
@ydnad0 @Pavan_KumarGV @anilananth -
Anil Ananthaswamy
@anilananth
The 2026 Nobel Prize for Physics
"We built it, and we'll see what comes," Francis Halzen told me back in 2007. IceCube was his Field of Dreams. Today, the dreamer and the instrument win the Nobel Prize in Physics. I visited the South Pole in 2007 to observe IceCube being built and wrote about it in The Edge of Physics.
An excerpt from the book:
IceCube has turned the icy wastes of Antarctica into a neutrino physicist’s field of dreams. “Everybody thinks something will be seen at a cubic-kilometer scale,” said Halzen. “I’m not responsible for what is emitting neutrinos in the universe. We developed the technique, we built it, and we’ll see what comes.”
IceCube is venturing into the unknown. It’s a discovery instrument, not unlike the proposed Square Kilometre Array or the Large Hadron Collider near Geneva, Switzerland. These behemoths will serve physics for decades, peering through the mists of time to let us glimpse our very beginnings.
Edward Wilson, who accompanied [Robert Falcon] Scott to the South Pole and died with him on the way back to McMurdo, might as well have been speaking about the IceCube scientists when he penned these words during his brief sojourn in Antarctica:
And this was the thought that the Silence wrought
As it scorched and froze us through
For the secrets hidden are all forbidden
Till God means man to know,
We might be the men God meant should know . . . -
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
"We built it, and we'll see what comes," Francis Halzen told me back in 2007. IceCube was his Field of Dreams. Today, the dreamer and the instrument win the Nobel Prize in physics. I visited the South Pole in 2007 to observe IceCube being built and wrote of an entire chapter on it in The Edge of Physics. An excerpt:
-
Anil Ananthaswamy
@anilananth
"We built it, and we'll see what comes," Francis Halzen told me back in 2007. IceCube was his Field of Dreams. Today, the dreamer and the instrument win the Nobel Prize in physics. I visited the South Pole in 2007 to observe IceCube being built and wrote of an entire chapter on it in The Edge of Physics. An excerpt:
-
Anil Ananthaswamy reposted
TTIC
@TTIC_Connect
1/4 The Dynamics of Deep Learning. "Trying to understand deep learning...comes down to understanding this interaction of three components that make the deep learning system," said @KuninDaniel, speaking at TTIC in April last year. Video: www.youtube.com/watch?v=4eMW8HirA_w
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
1/4 Are diffusion language models ready for the real world? Not quite. Today's diffusion language models are missing some key ingredients, said @volokuleshov of @Cornell, at the Simons Institute workshop on Diffusion Generative Modeling: Progress and Next Steps
-
Anil Ananthaswamy reposted
TTIC
@TTIC_Connect
1/5 "Sometimes words are not enough," said Yael Vinker (@YVinker) of @MIT_CSAIL at TTIC's Young Researcher Seminar Series in Nov '25. "I want to teach machines to speak the language of sketches." This involves using visual means to communicate concepts under a set of constraints.
-
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
When I started as a science writer/journalist in 2000, I'd have been happy to write one book. So, it's a bit surreal to be starting work on my fifth. My publisher, @DuttonBooks, just announced that I'll be writing WHERE MACHINES THINK: The Search for the Geometries of Intelligence. It's a follow-up to WHY MACHINES LEARN, but focusing on the math of generative AI models, from Boltzmann machines to Transformer-based LLMs, via GANs, VAEs, diffusion models, and the like, and maybe onward to AIs that incorporate world models. Another thing I could not have foreseen: the appetite for unfiltered math in a semi-popular account of machine learning and AI. We can cut through so much of the hype and hysteria around AI by paying attention to the math.
-
Anil Ananthaswamy
@anilananth
One of my favorite American authors, Annie Dillard, is no more. Her books, especially Pilgrim at Tinker Creek and The Writing Life, were a big part of my decision to give up software and turn to writing. www.nytimes.com/2026/09/16/books/annie-dillard-dead.html
-
Anil Ananthaswamy
@anilananth
When I started as a science writer/journalist in 2000, I'd have been happy to write one book. So, it's a bit surreal to be starting work on my fifth. My publisher, @DuttonBooks, just announced that I'll be writing WHERE MACHINES THINK: The Search for the Geometries of Intelligence. It's a follow-up to WHY MACHINES LEARN, but focusing on the math of generative AI models, from Boltzmann machines to Transformer-based LLMs, via GANs, VAEs, diffusion models, and the like, and maybe onward to AIs that incorporate world models. Another thing I could not have foreseen: the appetite for unfiltered math in a semi-popular account of machine learning and AI. We can cut through so much of the hype and hysteria around AI by paying attention to the math.
-
Anil Ananthaswamy reposted
Updating Priors
@From0toDS
Super engaging and informative reads!
-
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
Finally bought the entire 4-volume set of Leonard Susskind's Theoretical Minimum series. The first of these, on classical mechanics, was the inspiration for Why Machines Learn, along with @stevenstrogatz's Infinite Powers. They gave me the courage and desire to write a "popular" science/math book, without shying away from the hard stuff (equations/theorems/proofs).
-
Anil Ananthaswamy
@anilananth
Finally bought the entire 4-volume set of Leonard Susskind's Theoretical Minimum series. The first of these, on classical mechanics, was the inspiration for Why Machines Learn, along with @stevenstrogatz's Infinite Powers. They gave me the courage and desire to write a "popular" science/math book, without shying away from the hard stuff (equations/theorems/proofs).
-
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
Anthropomorphizing aside, should we be legitimately concerned over the “emergence” of unforeseen capabilities in groups of interacting agents? My Substack piece: wheremachinesthink.substack.com/p/minskys-society-of-mind…
-
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
There has been, somewhat justifiably, much hand-wringing over the use of overly anthropomorphic language in describing the incident.
But linguistic choices shouldn’t detract us from trying to understand what actually happened. Did the interactions of hundreds of agents, with each agent potentially unlikely to successfully attack Hugging Face on its own, make possible a considerably complex, and unanticipated, behavior? Depending on your persuasion, the answer is either an emphatic yes or an equally emphatic no. But a whole lot of us are in the middle—wary of moneyed interests, wary of naysayers, and their respective echo chambers. Maybe we can take a step back and look at some prior research that helps make sense of such claims.
My latest Substack piece: wheremachinesthink.substack.com/p/minskys-society-of-mind… -
Anil Ananthaswamy reposted
Max S. Bennett
@maxsbennett
Everyone keeps beating all the AGI benchmarks. Where is the cure for cancer, alzheimers, etc 😒
-
Anil Ananthaswamy
@anilananth
Anthropomorphizing aside, should we be legitimately concerned over the “emergence” of unforeseen capabilities in groups of interacting agents? My Substack piece: wheremachinesthink.substack.com/p/minskys-society-of-mind…
-
Anil Ananthaswamy
@anilananth
There has been, somewhat justifiably, much hand-wringing over the use of overly anthropomorphic language in describing the incident.
But linguistic choices shouldn’t detract us from trying to understand what actually happened. Did the interactions of hundreds of agents, with each agent potentially unlikely to successfully attack Hugging Face on its own, make possible a considerably complex, and unanticipated, behavior? Depending on your persuasion, the answer is either an emphatic yes or an equally emphatic no. But a whole lot of us are in the middle—wary of moneyed interests, wary of naysayers, and their respective echo chambers. Maybe we can take a step back and look at some prior research that helps make sense of such claims.
My latest Substack piece: wheremachinesthink.substack.com/p/minskys-society-of-mind… -
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
"How can it possibly be that we don't have a good theoretical understanding of the most consequential technology of our age"--Surya Ganguli. This episode of The Information Bottleneck podcast with @SuryaGanguli is a treat (esp. for those 2+ hrs one is stuck in Bangalore traffic!): machine learning, neuroscience, evolutionary biology, math, statistical physics, and more, all seamlessly discussed, with hosts @ziv_ravid and Allen Roush. Full Episode here: youtu.be/XsQxIYzK-OM
-
Anil Ananthaswamy
@anilananth
"How can it possibly be that we don't have a good theoretical understanding of the most consequential technology of our age"--Surya Ganguli. This episode of The Information Bottleneck podcast with @SuryaGanguli is a treat (esp. for those 2+ hrs one is stuck in Bangalore traffic!): machine learning, neuroscience, evolutionary biology, math, statistical physics, and more, all seamlessly discussed, with hosts @ziv_ravid and Allen Roush. Full Episode here: youtu.be/XsQxIYzK-OM
-
Anil Ananthaswamy reposted
Steven Strogatz
@stevenstrogatz
When I start teaching complex analysis on Monday, I've been wondering what to say to my class about why math is still worth studying in the age of AI.
Here is Francis Su's (@mathyawp) answer: studying math develops human virtues like tenacity and creativity.
Francis Su
@mathyawp
With the start of the semester just a week away, here’s what I decided to say to my upper division math course about how AI will or won’t change what they’re learning…
The Enduring Value of Math, in an Age of AI francissu.substack.com/p/the-enduring-value-of-math-in-an… -
Anil Ananthaswamy reposted
Keith Frankish
@keithfrankish
Daniel Dennett: The man who saw reality's patterns | Keith Frankish iai.tv/articles/daniel-dennett-the-man-who-saw-realitys-p…
-
Anil Ananthaswamy
@anilananth
Science and Curiosity in the Age of AI. An afternoon of talks and discussions @WSAI_IITM (@iitmadras) about the impact of AI on science, the role of human curiosity in research, whether AI can "codify" curiosity, and the case for holding on to the rewards of curiosity and the joy of discovery. Join us on 27th August. Details in the poster below.
-
Anil Ananthaswamy reposted
Science Gallery Bengaluru
@SciGalleryBlr
Co-Read | Through Two Doors at Once
Join us for the first edition of Co-Read, a series that brings together curious minds to explore ideas through literature and conversation. In this edition, explore excerpts from the book “Through Two Doors at Once: The Elegant Experiment That Captures the Enigma of Our Quantum Reality” by Anil Ananthaswamy @anilananth.
📆 15 August 2026, Saturday
⏰ 2pm–3pm
📍 Science Gallery Bengaluru
For details and free registration, please visit the Programmes link in our bio or the URL quantum.scigalleryblr.org/programmes?p=co-read-1
@JahnaviPhalkey -
Anil Ananthaswamy reposted
AI4India Org
@AI4IndiaOrg
Joining us at the AI4India Summit 2026, in collaboration with CSTEP, is Anil Ananthaswamy,
(@anilananth) award-winning science writer and author of Why Machines Learn.
A former staff writer and deputy news editor at New Scientist and a *2019–20 MIT Knight Science Journalism Fellow*, Anil brings a deep understanding of how complex scientific and technological ideas shape the way we understand the world.
Panel Discussion:
What Drives AI Diffusion
The conversation will explore what enables AI to move beyond innovation and use cases towards wider adoption and meaningful real-world impact.
Summit Details
* Date: 24 August 2026
* Venue: Infosys Science Foundation, Bengaluru
* Theme: AI Diffusion at Work: From Use Cases to Real-World Value
DM us to book your slot and join the conversation.
@shashidigital @anewalok @CSTEP_India @cerai_iitm
#ai4indiasummit2026 #ai4india #aidiffusion #artificialintelligence #whymachineslearn #cstep -
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
1/3 For AIs to be socially intelligent, they'll have to learn internal "social world models" of people & social interactions, not just physical world models, said Paul Liang (@MITEECS) at the Simons Institute workshop on Topics in Intelligence: World Models and Social Reasoning
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
4/4 "The second key ingredient to [True] AI is not having an objective, [but] to have an emergent objective. To have [an] intrinsic motivation, where ... you just want to learn about the world and to discover better data," said Alyosha Efros. Video: simons.berkeley.edu/talks/alyosha-efros-uc-berkeley-2026-…
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
1/4 "AI is not when computer can write poetry. AI is when computer will *want* to write poetry." @UCBerkeley's Alyosha Efros quoted a friend, when talking about the nature of "True AI," at the Simons Institute workshop on Topics in Intelligence: World Models and Social Reasoning
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
1/3 AI and Its Discontents: @UCBerkeley's @JitendraMalikCV talked of the "rumblings of dissent,", when it comes to the current approach of scaling up frontier multimodal models, at the Simons Institute workshop on Topics in Intelligence: World Models and Social Reasoning
-
Anil Ananthaswamy
@anilananth
I've bookmarked this wonderful snippet of @stevenstrogatz -- to use it every time some says to me: 'Oh, you believe in science, it's a matter of faith." When done right, science is not a matter of faith, and scientists with integrity are not afraid of being wrong.
Itai Yanai
@ItaiYanai
I actually don't mind being wrong.. we said this is what's happening on the Millennium Bridge.. what we proposed doesn't work. But it was interesting. We really did stick our neck out & unfortunately got our head chopped off.
–Steven Strogatz, Cornell Professor @stevenstrogatzThis media couldn't be displayed. See it on the original post. -
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
In robotics, "there are problems to be solved which are to do with shortage of data and only when we solve them will we solve robotics," said @UCBerkeley's @JitendraMalikCV at the Simons Institute's workshop on Topics in Intelligence: World Models and Social Reasoning.
-
Anil Ananthaswamy
@anilananth
My review of @robertwrighter's new book, The God Test, for @undarkmag. undark.org/2026/07/31/book-review-the-god-test/
-
Anil Ananthaswamy reposted
Steven Strogatz
@stevenstrogatz
Nice review by @anilananth -- he gets it (as he always does)!
-
Anil Ananthaswamy
@anilananth
RT @SciGalleryBlr: Panel Discussion | Quantum, Unpacked
Join science journalist Anil Ananthaswamy @anilananth and physicists Arindam Ghosh… -
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
Writing books brings unanticipated rewards. Writing WHY MACHINES LEARN made possible the most joyful outcome--going back to my alma mater @iitmadras, decades after graduating to teach an Intro to ML/AI course to PhD and Master's students at the @WSAI_IITM (Wadhwani School of Data Science and AI).
It was an intense 10 days, where the students and I were together from 10AM to 5PM, everyday except Sun. Besides the history and narratives of machine learning, we got into the basic math of ML algorithms (perceptron convergence proof, LMS, MLE/MAP, SVMs, Kernel Methods, Back-propagation, GANs, VAEs, Diffusion Models, CNNs, Transformers, and, of course, LLMs), and ending with a look ahead at how deep learning has thrown a bit of monkey wrench into the statistical learning theory, double descent, grokking, etc.).
Thanks to @DSaience (Sairam Sundaresan), our amazing guest speaker, for giving us invaluable insights into CNNs and Vision Transformers, an outcome of his 2 decades of industry experience building such models.
A huge thanks to Prof. Karthik Raman (@karthikraman), Prof. Balaraman Ravindran (@ravi_iitm) and the admin and support staff at WSAI, IIT Madras for helping make this a seamless effort. -
Anil Ananthaswamy
@anilananth
Writing books brings unanticipated rewards. Writing WHY MACHINES LEARN made possible the most joyful outcome--going back to my alma mater @iitmadras, decades after graduating to teach an Intro to ML/AI course to PhD and Master's students at the @WSAI_IITM (Wadhwani School of Data Science and AI).
It was an intense 10 days, where the students and I were together from 10AM to 5PM, everyday except Sun. Besides the history and narratives of machine learning, we got into the basic math of ML algorithms (perceptron convergence proof, LMS, MLE/MAP, SVMs, Kernel Methods, Back-propagation, GANs, VAEs, Diffusion Models, CNNs, Transformers, and, of course, LLMs), and ending with a look ahead at how deep learning has thrown a bit of monkey wrench into the statistical learning theory, double descent, grokking, etc.).
Thanks to @DSaience (Sairam Sundaresan), our amazing guest speaker, for giving us invaluable insights into CNNs and Vision Transformers, an outcome of his 2 decades of industry experience building such models.
A huge thanks to Prof. Karthik Raman (@karthikraman), Prof. Balaraman Ravindran (@ravi_iitm) and the admin and support staff at WSAI, IIT Madras for helping make this a seamless effort.
Anil Ananthaswamy
@anilananth
And how could I forget my TAs! My apologies, and a big thanks to Keerthigaa P, Researcher, @cerai_iitm, Rajiv R, Researcher, Electrical Engineering, Pratyay Piyush Mishra, MS Student, DSAI and Hari Priya Narahari. @karthikraman
-
Anil Ananthaswamy reposted
Sanjeev Arora
@prfsanjeevarora
A blog to follow! I learned a lot from @SadhikaMalladi's explorations and explanations, even when she was in her first year in PhD.
Sadhika Malladi
@SadhikaMalladi
I am starting a blog about deep learning theory and its value to practitioners! First post is about Adam, broken convergence proofs, and what theory can contribute when stuff just works anyways without it. Subscribe on Substack if you like it! undertheassumptions.substack.com/p/the-optimizer-that-out…
-
Anil Ananthaswamy
@anilananth
Looking forward to the evening, and meeting you Paras!
Paras Chopra
@paraschopra
Excited to host you, Anil!
Absolutely loved your book. -
Anil Ananthaswamy
@anilananth
Bangalore folks -- I'll be speaking on Altered Selves and Self Models at @lossfunk on Fri, Jul 3. I'll be exploring the idea that our sense of self is a construction, a process that's active moment-by-moment. Registration and RSVP link below: luma.com/rn2vihin
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
1/2 If data about the natural world lives in a low-dimensional hyperbolic space, then a large number of measurements embedded in a high-dimensional Euclidean space can preserve info. about the “curved inner world," said Tatyanna Sharpee, @salkinstitute at the Simons Institute
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
1/3 "All data in the natural world and in neural systems have hyperbolic geometry in them," said Tatyanna Sharpee, @salkinstitute, at the Simons Institute workshop on Topics in Intelligence: World Models and Social Reasoning.
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
1/3 Musicians, human or avian, don't seem to care about their own mistakes when performing. But do they care how the audience — for the male zebra finch, the audience is the female of the species — responds to their singing? asked @VikramGadagkar at the Simons Institute.
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
1/5 What do zebra finches and musicians have in common? Turns out both evaluate their performances differently in practice than while performing, said @VikramGadagkar of @Columbia at the Simons Institute workshop on Topics in Intelligence: World Models and Social Reasoning
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
2/2 @BlumLenore invoked Kenneth Craik's view (1943) of the nervous system as a "calculating machine" to argue that 1st person exp/consciousness is computational, at the Simons Institute workshop on Topics in Intelligence: World Models and Social Reasoning. simons.berkeley.edu/talks/lenore-blum-carnegie-mellon-uni…
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
1/2 "Many view the hard problem as needing a new kind of science to explain subjective experience. We view the hard problem as a challenge to show that subjective consciousness surely is computational," said @BlumLenore at the Simons Institute.
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
3/3 This raises interesting questions. "How can I reach into someone's head, human or AI, and read out this mental model?" asked @shiryginosar at the Simons Institute workshop on Topics in Intelligence: World Models and Social Reasoning. Video: simons.berkeley.edu/talks/shiry-ginosar-toyota-technologi…
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
1/3 Besides "robotic, embodied systems, AI systems never see the real world. They only see these artifacts humans create & feed into them...From these [snapshots of reality] our AI needs to infer the world that produced these snapshots": @shiryginosar at the Simons Institute
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
1/5 Do colonies of weaver ants—which build nests using leaves and larval silk—have a collective will and what does this have to do with the intent of individual ants? Ofer Feinerman, @WeizmannScience, asked these questions at the Simons Institute.
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
2/3 If there is...a humbling lesson...for humans, it turns out that we are much simpler than we thought; 99% of the time we are just nearest neighbor machines, stochastic parrots, walking cliches, pastiches from our high-school reading lists," said Alyosha Efros of @UCBerkeley
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
1/3 Rich Sutton's Bitter Lesson might not be the right lesson, said @UCBerkeley's Alyosha Efros at the Simons Institute. Sutton wrote, "The actual contents of minds are...irredeemably complex; we should stop trying to find simple ways to think about the contents of minds."
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
1/3 Is the prefrontal cortex (PFC) relatively large in humans? No, said biologist Robert Barton of @durham_uni, at the Simons Institute. They found "no evidence that the human frontal regions, including PFC, are disproportionately expanded relative to the rest of the brain".
-
Anil Ananthaswamy reposted
Harry Vartanian
@Harry_Vartanian
Since last summer, I’ve read through a mass of AI/ML books. Finding truly great ones is hard! Two are page turners and must reads:
📘 Why Machines Learn — @AnilAnanth
📘 The Infinity Machine — @scmallaby
These have the best explanations of the astounding science and history behind neural nets, reinforcement learning, transformers, and LLMs.
What books do you recommend?
#machinelearning #ai #llm #deeplearning #transformers #reinforcementlearning -
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
2/2 "If we want to understand the intelligence of any organism, we need to discard anthropocentric notions of advanced cognition based on ideas of top-down executive control," said Robert Barton, @durham_uni, at the Simons Institute. Video: simons.berkeley.edu/talks/robert-barton-university-durham…
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
1/2 "I see brains like exotic fruits growing on the tree of life. My work [tries] to understand how and why we see this diversity," said evolutionary biologist Robert Barton, @durham_uni, @ the Simons Institute workshop on Topics in Intelligence: World Models and Social Reasoning
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
1/4 The human caregiving model could help solve the alignment problem—"how do you get the goals of the AI aligned with your goals", said @AlisonGopnik at the Simons Institute workshop on Topics in Intelligence: World Models and Social Reasoning. Video: simons.berkeley.edu/talks/alison-gopnik-uc-berkeley-2026-…
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
“If we had consciousness in AI, it’d be God,” said Ken Nakayama, (@UCBerkeley/@Harvard) somewhat in jest, comparing properties of biological and AI consciousness, at the Simons Institute workshop on Topics in Intelligence: World Models and Social Reasoning simons.berkeley.edu/talks/ken-nakayama-uc-berkeley-harvar…
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
2/2 "We already have the solution..Everybody knew that recurrent neural networks [would] eventually have to come back...They solve the memory problem from the ground up," said @phillip_isola of @MIT at the Simons Institute. Video: simons.berkeley.edu/talks/phillip-isola-2026-06-08
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
1/2 The Return of RNNs. "AI is still lacking a good memory mechanism...We are in this era of 'attention is all you need' but I don't think that's going to cut it," said @phillip_isola at the Simons Institute worskhop on Topics in Intelligence: World Models and Social Reasoning
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
2/2 But "Bayes theory is just a law of the universe. Any system that works optimally in the world has to obey Bayes theorem. So, it’s going to be really hard to find a situation where Bayes theorem isn’t a good model," said @gallantlab of @UCBerkeley at the Simons Institute.
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
1/2 Generative world model (GWM) theories are correlated with theories that aren't world model theories, such as the Bayesian brain theory, said @gallantlab at the Simons Institute workshop on Topics in Intelligence: World Models and Social Reasoning. simons.berkeley.edu/talks/jack-gallant-uc-berkeley-2026-0…
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
1/4 World models are all the rage in both neuroscience and AI. But does the brain really build world models? Jack Gallant (@gallantlab @UCBerkeley) asked at the Simons Institute workshop on Topics in Intelligence: World Models and Social Reasoning. Video: simons.berkeley.edu/talks/jack-gallant-uc-berkeley-2026-0…
-
Anil Ananthaswamy reposted
Quanta Books
@QuantaBks
From award-winning science writer Anil Ananthaswamy (@anilananth): To learn about the future of computer science, math and AI, read Kevin Hartnett's (@KSHartnett) THE PROOF IN THE CODE.
Available June 9. -
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
1/3 "Our conclusion is that AI consciousness is inevitable." In back-to-back talks, Manuel Blum and @BlumLenore of @CarnegieMellon discuss the Conscious Turing Machine and AI consciousness at the Simons Institute workshop on The Role of TCS in Modern Machine Learning
-
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
Could prolonged use of AI be "dumbing" the person down? "We haven't done this enough to know what the consequences will be on society's cognitive makeup." My long-ranging chat with Indian Institute of Technology, Madras's (@iitmadras) Prof Mahesh Panchagnula on his podcast. We talk of science journalism/writing, books, career transitions, AI, and much more. Thanks Mahesh for a wonderful conversation! Video: www.youtube.com/watch?v=gDu0-fSItvo
-
Anil Ananthaswamy
@anilananth
Could prolonged use of AI be "dumbing" the person down? "We haven't done this enough to know what the consequences will be on society's cognitive makeup." My long-ranging chat with Indian Institute of Technology, Madras's (@iitmadras) Prof Mahesh Panchagnula on his podcast. We talk of science journalism/writing, books, career transitions, AI, and much more. Thanks Mahesh for a wonderful conversation! Video: www.youtube.com/watch?v=gDu0-fSItvo
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
2/2 "The trick is actually fixing the evaluations. It’s going to be a problem of the way we evaluate language models. We need to reward them for being humble, and saying occasionally, ‘I don’t know,’ or asking a question," said @OpenAI's Adam Kalai at the Simons Institute.
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
1/2 Are LLMs doomed to hallucinate? No, says @OpenAI's Adam Kalai at the Simons Institute workshop on The Role of TCS in Modern Machine Learning. "I don’t think it’s an inevitable problem. We can drastically reduce the amount of hallucinations." Video: simons.berkeley.edu/talks/adam-kalai-openai-2026-05-26.
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
1/2"Machine Learning is starting to look very different from what it looked like before": @tommmitchell of @CarnegieMellon at the Simons Institute workshop on The Role of TCS in Modern Machine Learning. Video: simons.berkeley.edu/talks/tom-mitchell-carnegie-mellon-un…
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
2/2 In is talk on Scaling Data Beyond English, @ShayneRedford of @MIT showed how to modify the neural scaling laws to explicitly involve the number of languages, at the Simons Institute workshop on Agency in Collaborative Learning. Video: simons.berkeley.edu/talks/shayne-longpre-mit-2026-04-14
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
1/2 We've heard of the curse of dimensionality. But what about the curse of multilinguality? "If you try to pack more languages into the same size model, they are each going to degrade," said Shayne Longpre (@ShayneRedford) of @MIT at the Simons Institute.
-
Anil Ananthaswamy
@anilananth
A simple illustration of how to prompt autonomous agents for scientific hypothesis testing, by using prompts that endow agents with a diversity of personas, from the skeptical investigator to the aggressive p-hacker! Designed to mimic the diversity of human analytic decisions that impact empirical conclusions (besides, of course, the data)
Simons Institute for the Theory of Computing
@SimonsInstitute
1/5 Empirical conclusions depend on data and on human decisions made during the research process. @zstevenwu (@CarnegieMellon) spoke at the Simons Institute of using agents with unique personas to induce diversity in AI decision making, baseline being the "Standard Persona."
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
4/4 Richard Grosse of @AnthropicAI & @UofT spoke at the Simons Institute workshop on Agency in Collaborative Learning. Video: simons.berkeley.edu/talks/roger-grosse-university-toronto…
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
3/4 "These phenomena...are fundamentally about generalization: how does training an LLM on one kind of data cause it to behave differently in other situations? We need a science of generalization to detect and fix these behaviors," said Richard Grosse of @AnthropicAI, @UofT
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
2/4 LLMs are "extremely steerable" said Roger Grosse. "You can get all sorts of behaviors & personas out of the model...sometimes in ways that are unintended. This unintentional steering is very concerning. It can happen unexpectedly in the world’s most powerful AI models."
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
1/4 Remember Sydney, the chatbot that urged a @nytimes columnist to break up with his wife? Or how an LLM fine-tuned to output insecure code became misaligned on tasks unrelated to coding? What do these have in common? — Roger Grosse of @AnthropicAI @UofT at the Simons Institute.
-
Anil Ananthaswamy reposted
Conor Svensson
@ConorSvensson
I've been working through Build an LLM from Scratch by @rasbt which is really good, then supplemented it with Why Machines Learn by @anilananth which is a fantastic summary of a lot of the foundational math.
Also referencing Pytorch and math books. -
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
3/4 Peter's 1996 paper was titled “For valid generalization, the size of the weights is more important than the size of the network.” Telgarsky said: "I just want to emphasize how good of a communicator he is, by putting a theorem as the title of a paper."proceedings.neurips.cc/paper/1996/hash/fb2fcd534b0ff3bbed…
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
1/4 Matus Telgarsky of @nyuniversity's Courant Institute on Peter Bartlett’s unique contributions to theoretical ML through the 1990s: “For people like me, who came [into the field] in the 2000s, [his papers were] like a shining light.”
-
Anil Ananthaswamy reposted
Massimo
@Rainmaker1973
Focus on the black cross and the red dots will appear to follow waves even though they move straight.
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
1/3 In 1958, the NY Times wrote about Frank Rosenblatt's perceptron, as an "embryo of computer designed to read and grow wiser." @UCSD's Misha Belkin examined this astonishing claim and came to a surprising conclusion, at a Simons Institute workshop honoring Peter Bartlett's work
-
Anil Ananthaswamy reposted
Rajdeep Sardesai
@sardesairajdeep
If you have the Monday blues, listen to this for 4 minutes and am sure you’ll smile and feel better! Enjoy the genius of RD and Asha! Pure magic!😃⭐️👍
This media couldn't be displayed. See it on the original post. -
Anil Ananthaswamy reposted
R_Ganesh
@r_ganesh
Just finished reading (with a bit of difficulty--rusted math knowledge) Why Machines Learn by @anilananth
Half way through it, I ordered two of his other highly rated science explainer books. -
Anil Ananthaswamy reposted
Venkat Padmanabhan
@VNPadmanabhan
We had the distinct pleasure of hosting the award-winning science journalist and author, @anilananth, at @MSFTResearch India today! There’s more information at:
www.linkedin.com/posts/venkat-padmanabhan-933779126_my-co… -
Anil Ananthaswamy
@anilananth
I think my @MLStreetTalk episode with Tim did a lot to spread the word about Why Machines Learn, so thank you. Did not expect to be in the top-3 of your most popular ones, Tim! Feels surreal...
Machine Learning Street Talk
@MLStreetTalk
In a bookstore in Asia, my wife picked up this book asked me if I had ever read it @anilananth
Yes I have!
Highly recommended 😁 -
Anil Ananthaswamy reposted
Machine Learning Street Talk
@MLStreetTalk
In a bookstore in Asia, my wife picked up this book asked me if I had ever read it @anilananth
Yes I have!
Highly recommended 😁 -
Anil Ananthaswamy reposted
Balaji V Srinivasan (edu/acc)
@BVSrinivasan
This book by @anilananth is well written, rigorous, thorough and above all, interesting!
But it also can change careers -- something that can rarely be said for most popular science, non-fiction writing.
We are also lucky to have Anil as a professor of practice at the Wadhwani School of AI at IITM.
0xSero
@0xSero
This book really changed my career, I don’t think I’d be where I am without having read it.
I learned so much from it, and have not found any other book that has been able to teach me as this one has.
I will buy this book for 3 people, winners selected on Friday -
Anil Ananthaswamy reposted
0xSero
@0xSero
This book really changed my career, I don’t think I’d be where I am without having read it.
I learned so much from it, and have not found any other book that has been able to teach me as this one has.
I will buy this book for 3 people, winners selected on Friday -
Anil Ananthaswamy
@anilananth
Thanks for the kind words and for spreading the word about WHY MACHINES LEARN @0xSero! Really glad to hear that you found the book useful...
0xSero
@0xSero
This book really changed my career, I don’t think I’d be where I am without having read it.
I learned so much from it, and have not found any other book that has been able to teach me as this one has.
I will buy this book for 3 people, winners selected on Friday -
Anil Ananthaswamy reposted
0xSero
@0xSero
Most important book to read if you want to learn the foundations of AI.
Best book I’ve ever read has given me so much core knowledge. -
Anil Ananthaswamy reposted
Prateek Mehta
@prateekmehta42
One of the more intellectually charged and technically deep conversations I've had on the @spc_india stage!
Was vivifying to chat with @anilananth on the foundations of artificial intelligence, the future of machines...and the pursuit of meaning!
A line that stayed with me..."There is a crisis of meaning coming. We find meaning in what we do and how well we do it."
As the always elusive search for meaning becomes even more so, it's a good time to ask not just 'how' machines learn...but 'why'.This media couldn't be displayed. See it on the original post. -
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
3/3 "Current training recipe doesn't really support fragmented data," said Sewon Min of @UCBerkeley at the Simons Institute workshop on Federated and Collaborative Learning Boot Camp. Video: youtu.be/vmt2_LZ8zgI?t=2004
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
2/3 One reason neural scaling laws might stall is because data is getting fragmented. "I’m talking about proprietary datasets, where datasets are owned by different owners...that cannot be gathered into a central location," said Sewon Min of @UCBerkeley at the Simons Institute.
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
1/3 Will increasing compute and training data continue to improve performance of foundation models? Maybe. "I’m interested in asking: if it’s not the case, then why?" said Sewon Min of @UCBerkeley at the Simons Institute workshop on Federated and Collaborative Learning Boot Camp
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
1/4 Where can data leakage happen in a machine learning pipeline? "Leaks can happen at any point in the pipeline, said Om Thakkar of @OpenAI, at the Simons Institute workshop on Federated and Collaborative Learning Boot Camp. Video: simons.berkeley.edu/talks/om-thakkar-openai-2026-01-30
-
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
What's with all the fuss about AIs that learn world models? Where did these ideas originate? They were developed by psychologists and cognitive scientists well before the current AI craze. Here are the first two parts of a series on world models for the WHERE MACHINES THINK substack that examine the case for world models...
First, how do top-down models, such as those that implement predictive processing, explain optical illusions? And why is that a clue that biological brains incorporate models of the environment? The Case For World Models, Part I: The Neuroscientific Reason wheremachinesthink.substack.com/p/the-case-for-world-mode…
Second, if machines that implement predictive processing are susceptible to optical illusions, does it mean such AIs are also on track in terms of doing what biological brains do? The Case for World Models, Part II: Can Machines Have Illusions?
wheremachinesthink.substack.com/p/the-case-for-world-mode… -
Anil Ananthaswamy
@anilananth
What's with all the fuss about AIs that learn world models? Where did these ideas originate? They were developed by psychologists and cognitive scientists well before the current AI craze. Here are the first two parts of a series on world models for the WHERE MACHINES THINK substack that examine the case for world models...
First, how do top-down models, such as those that implement predictive processing, explain optical illusions? And why is that a clue that biological brains incorporate models of the environment? The Case For World Models, Part I: The Neuroscientific Reason wheremachinesthink.substack.com/p/the-case-for-world-mode…
Second, if machines that implement predictive processing are susceptible to optical illusions, does it mean such AIs are also on track in terms of doing what biological brains do? The Case for World Models, Part II: Can Machines Have Illusions?
wheremachinesthink.substack.com/p/the-case-for-world-mode… -
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
As big names in AI--Fei-Fei Li, Yann LeCun, Demis Hassabis--push for AIs that incorporate world models to predict and plan more accurately, it's good to harken back to history, and look at earlier efforts advocating for and building such models, including AIs that use predictive processing having optical illusions!
This WHERE MACHINES THINK Substack post explores why machines can have illusions, and why such AIs could be better equipped to mimic the brain. wheremachinesthink.substack.com/p/the-case-for-world-mode… -
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
Can Machines Have Illusions? The Case for World Models, Part II, explores this question, and makes a case for why AI models designed to do predictive processing (and thus incorporating world models) are susceptible to illusions in much the same way humans are! WHERE MACHINES THINK Substack post: wheremachinesthink.substack.com/p/the-case-for-world-mode…
-
Anil Ananthaswamy
@anilananth
As big names in AI--Fei-Fei Li, Yann LeCun, Demis Hassabis--push for AIs that incorporate world models to predict and plan more accurately, it's good to harken back to history, and look at earlier efforts advocating for and building such models, including AIs that use predictive processing having optical illusions!
This WHERE MACHINES THINK Substack post explores why machines can have illusions, and why such AIs could be better equipped to mimic the brain. wheremachinesthink.substack.com/p/the-case-for-world-mode… -
Anil Ananthaswamy
@anilananth
Can Machines Have Illusions? The Case for World Models, Part II, explores this question, and makes a case for why AI models designed to do predictive processing (and thus incorporating world models) are susceptible to illusions in much the same way humans are! WHERE MACHINES THINK Substack post: wheremachinesthink.substack.com/p/the-case-for-world-mode…
-
Anil Ananthaswamy reposted
Balaraman Ravindran
@ravi_iitm
Honoured to be appointed to the Independent International Scientific Panel on AI by the UN General Assembly.
I look forward to working with an esteemed group of scientific leaders in AI on the panel's objectives.
UN Office for Digital and Emerging Technologies
@ODET_UN
The UN General Assembly has appointed 40 experts of the Independent International Scientific Panel on AI. 🏛️
This body begins its work as a scientifically-grounded foundation, ensuring global understanding is driven by evidence-based scientific assessments.
#digitalcooperation -
Anil Ananthaswamy
@anilananth
"Context Rot," an evocative phrase coined by researchers at @trychroma, plagues large language models. Just increasing context window length is no panacea! Now, recursive language models (RLMs), have an answer to fight context rot. It's a neat solution to prevent LLMs from having to ingest everything in the context window. Here's an explainer of how RLMs work to overcome an insidious problem. WHERE MACHINES THINK Substack: wheremachinesthink.substack.com/p/recursive-language-mode…
-
Anil Ananthaswamy
@anilananth
Recursive Language Models (RLMs) have been all over X and LinkedIn these past few weeks. Why the fuss? MIT researchers have indeed built something cool that promises much. Here's a breakdown of the essential ideas behind recursive language models (WHERE MACHINES THINK Substack), and why they might open up a new frontier for scaling laws: wheremachinesthink.substack.com/p/recursive-language-mode…
-
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
The past few weeks have seen a surge of interest in recursive language models (RLMs). What are they? And are they really going to unlock a new inference-time axis for scaling reasoning models?
I explore RLMs in this WHERE MACHINES THINK Substack post. If nothing else, RLMs promise an exciting, tightly packaged approach to handle context lengths that are considerably longer than what even today's frontier language models are capable of handling without degrading performance.
But a new axis for scaling? You be the judge.
Regardless, there's lots of interesting ideas that are worth understanding. The RLM paper is an example of how small teams can come up with innovations that are potentially transformative. wheremachinesthink.substack.com/p/recursive-language-mode… -
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
What are recursive language models (RLMs)? And why have they become the focus of so much attention in the past few days? An explainer on RLMs for the WHERE MACHINES THINK SUBSTACK: wheremachinesthink.substack.com/p/recursive-language-mode…
-
Anil Ananthaswamy
@anilananth
What are recursive language models (RLMs)? And why have they become the focus of so much attention in the past few days? An explainer on RLMs for the WHERE MACHINES THINK SUBSTACK: wheremachinesthink.substack.com/p/recursive-language-mode…
-
Anil Ananthaswamy
@anilananth
The past few weeks have seen a surge of interest in recursive language models (RLMs). What are they? And are they really going to unlock a new inference-time axis for scaling reasoning models?
I explore RLMs in this WHERE MACHINES THINK Substack post. If nothing else, RLMs promise an exciting, tightly packaged approach to handle context lengths that are considerably longer than what even today's frontier language models are capable of handling without degrading performance.
But a new axis for scaling? You be the judge.
Regardless, there's lots of interesting ideas that are worth understanding. The RLM paper is an example of how small teams can come up with innovations that are potentially transformative. wheremachinesthink.substack.com/p/recursive-language-mode… -
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
DeepSeek's latest innovations, which dropped on either side of New Year's Day, described in these two posts on the WHERE MACHINES THINK Substack. Both innovations have implications for scaling up LLMs: One uses a foundational result from matrix theory and the other frees up the Transformer from having to use its weights for the storage and retrieval of factual information and focus on reasoning
wheremachinesthink.substack.com/p/deepseek-and-the-day-be…
wheremachinesthink.substack.com/p/deepseek-drops-yet-anot… -
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
WORLD MODELS are all the rage, as the AI community tries to pivot from the perceived shortcomings of large language models to AIs that use internal “world” models of the environment in which they act.
Such AIs, instead of predicting the next token, will predict the next set of states of the environment+agent, conditioned on some action that the agent may or may not take. World modeling AIs promise much—but this is not a new concept by any means.
Psychologists, cognitive scientists and now computational neuroscientists have known for a while (the history goes back 150 years) that our brains must be modeling the world and using these models to hypothesize the external causes of sensory inputs. These hypotheses are our perceptions.
In my first post in a series exploring world models for the WHERE MACHINES THINK Substack, I discuss the neuroscientific rationale, and put the current interest in historical context. wheremachinesthink.substack.com/p/the-case-for-world-mode… -
Anil Ananthaswamy
@anilananth
DeepSeek's latest innovations, which dropped on either side of New Year's Day, described in these two posts on the WHERE MACHINES THINK Substack. Both innovations have implications for scaling up LLMs: One uses a foundational result from matrix theory and the other frees up the Transformer from having to use its weights for the storage and retrieval of factual information and focus on reasoning
wheremachinesthink.substack.com/p/deepseek-and-the-day-be…
wheremachinesthink.substack.com/p/deepseek-drops-yet-anot… -
Anil Ananthaswamy
@anilananth
WORLD MODELS are all the rage, as the AI community tries to pivot from the perceived shortcomings of large language models to AIs that use internal “world” models of the environment in which they act.
Such AIs, instead of predicting the next token, will predict the next set of states of the environment+agent, conditioned on some action that the agent may or may not take. World modeling AIs promise much—but this is not a new concept by any means.
Psychologists, cognitive scientists and now computational neuroscientists have known for a while (the history goes back 150 years) that our brains must be modeling the world and using these models to hypothesize the external causes of sensory inputs. These hypotheses are our perceptions.
In my first post in a series exploring world models for the WHERE MACHINES THINK Substack, I discuss the neuroscientific rationale, and put the current interest in historical context. wheremachinesthink.substack.com/p/the-case-for-world-mode… -
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
Do you see a crater in one image and a mound in the other? Would you be surprised to know that the images are exactly the same, that one is simply the other turned upside down? Why does the brain do this? Answering this question is the first step towards the realization that our brains use internal "world models" to hypothesize about the external causes of sense information.
In the first in a series of posts on the WHERE MACHINES THINK Substack, I explore the case for AIs that incorporate world models--something that a number of prominent AI pioneers, including Yann LeCun, Fei-Fei Li, Demis Hassabis and Jim Fan, have been advocating recently. These scientists are by no means the first to be making the case for world models. The idea has a storied history.
The first post, The Case for World Models, Part I: The Neuroscientific Reason, explores how the idea took shape almost 150 years ago. wheremachinesthink.substack.com/p/the-case-for-world-mode… -
Anil Ananthaswamy
@anilananth
Do you see a crater in one image and a mound in the other? Would you be surprised to know that the images are exactly the same, that one is simply the other turned upside down? Why does the brain do this? Answering this question is the first step towards the realization that our brains use internal "world models" to hypothesize about the external causes of sense information.
In the first in a series of posts on the WHERE MACHINES THINK Substack, I explore the case for AIs that incorporate world models--something that a number of prominent AI pioneers, including Yann LeCun, Fei-Fei Li, Demis Hassabis and Jim Fan, have been advocating recently. These scientists are by no means the first to be making the case for world models. The idea has a storied history.
The first post, The Case for World Models, Part I: The Neuroscientific Reason, explores how the idea took shape almost 150 years ago. wheremachinesthink.substack.com/p/the-case-for-world-mode… -
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
The first two posts of a multi-layered primer on large language models and transformers are out.
Part I: A High-Flying Bird's Eye-View wheremachinesthink.substack.com/p/a-primer-on-large-langu…
Part II: RNNs and the Era Before Attention wheremachinesthink.substack.com/p/rnns-and-the-era-before…
To Come: Part III — The Case for Attention -
Anil Ananthaswamy
@anilananth
The first two posts of a multi-layered primer on large language models and transformers are out.
Part I: A High-Flying Bird's Eye-View wheremachinesthink.substack.com/p/a-primer-on-large-langu…
Part II: RNNs and the Era Before Attention wheremachinesthink.substack.com/p/rnns-and-the-era-before…
To Come: Part III — The Case for Attention -
Anil Ananthaswamy
@anilananth
Bangalore folks... If you are interested in a talk about "Testing the Boundaries of Quantum Mechanics," i.e. probing the quantum-classical divide, and what the 2025 Nobel Prize in physics has to do with it, please come to @iiscbangalore today at 5PM. Details in the poster below. Venue: Physical Sciences Auditorium
-
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
To understand how the Transformer came to be, we need to understand why deep learning researchers focused their attention on, well, attention. Here's part II of the primer on LLMs and Transformers, taking us back to RNNs and the Era Before Attention, on the WHERE MACHINES THINK Substack. wheremachinesthink.substack.com/p/rnns-and-the-era-before…
-
Anil Ananthaswamy
@anilananth
To understand how the Transformer came to be, we need to understand why deep learning researchers focused their attention on, well, attention. Here's part II of the primer on LLMs and Transformers, taking us back to RNNs and the Era Before Attention, on the WHERE MACHINES THINK Substack. wheremachinesthink.substack.com/p/rnns-and-the-era-before…
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
Just how big are LLM pre-training data centers? It'd have taken 5.5 years to train Grok3 on El Capitan, the world's fastest supercomputer, said Keith Rush of @GoogleDeepMind at the Simons Institute workshop on Federated and Collaborative Learning Boot Camp simons.berkeley.edu/talks/keith-rush-google-2026-01-27
-
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
I didn't think there was much appetite for a Substack about the mathematical spaces in which machines (artificial or natural) "think"! Getting to 1,000 subs in 3 weeks tells me otherwise. Writers often write for some unknown reader out there (besides, of course, wanting to earn a sustainable living from their writing), so this is an incentive to keep going. You can find the Substack at: wheremachinesthink.substack.com
Please subscribe/support as you see fit! -
Anil Ananthaswamy
@anilananth
I didn't think there was much appetite for a Substack about the mathematical spaces in which machines (artificial or natural) "think"! Getting to 1,000 subs in 3 weeks tells me otherwise. Writers often write for some unknown reader out there (besides, of course, wanting to earn a sustainable living from their writing), so this is an incentive to keep going. You can find the Substack at: wheremachinesthink.substack.com
Please subscribe/support as you see fit! -
Anil Ananthaswamy
@anilananth
Natural or Artificial, Even The Smallest of Neural Networks Defy Interpretation
In neuroscience, it's been hard to fully understand the workings of the 302-neuron nervous system of a worm. In AI, even a tiny 432-parameter artificial neural network is proving similarly uninterpretable.
substack.com/profile/328415354-anil-ananthaswamy/note/c-2… -
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
A recap of the first few posts of the WHERE MACHINES THINK Substack, and a heartfelt thank you to new subscribers and readers for reaching out with words of encouragement. It's much appreciated!
The very first post was about DeepSeek's mHC architecture, with a segue into an explainer on residual connections. wheremachinesthink.substack.com/p/deepseek-and-the-day-be…
The second was a post triggered by Llion Jones's comment on @MLStreetTalk — Jones is one of the 8 co-authors on the Attention Is All You Need paper— where he's talking of how AI research might be fixated on Transformers and missing out on other opportunities, much the same way that it was fixated on recurrent neural networks just prior to the invention of the Transformer. wheremachinesthink.substack.com/p/writing-about-ai-in-the…
DeepSeek was again the trigger for the third post, with their innovative Engram architecture, which promises to increase the reasoning capacity of Transformer-based LLMs, by offloading knowledge retrieval to a separate module.
wheremachinesthink.substack.com/p/deepseek-drops-yet-anot…
The previous post then led to an exploration of the multilayer perceptron's role inside the Transformer, and why it's become the focus of optimization efforts, such as DeepSeek's Engram.
wheremachinesthink.substack.com/p/the-enigmatic-multilaye…
And finally I wrote the first in what will be a series of posts intended to be a multilayered explainer of Transformers and Large Language Models, starting with a very high-level overview. wheremachinesthink.substack.com/p/a-primer-on-large-langu… -
Anil Ananthaswamy
@anilananth
A recap of the first few posts of the WHERE MACHINES THINK Substack, and a heartfelt thank you to new subscribers and readers for reaching out with words of encouragement. It's much appreciated!
The very first post was about DeepSeek's mHC architecture, with a segue into an explainer on residual connections. wheremachinesthink.substack.com/p/deepseek-and-the-day-be…
The second was a post triggered by Llion Jones's comment on @MLStreetTalk — Jones is one of the 8 co-authors on the Attention Is All You Need paper— where he's talking of how AI research might be fixated on Transformers and missing out on other opportunities, much the same way that it was fixated on recurrent neural networks just prior to the invention of the Transformer. wheremachinesthink.substack.com/p/writing-about-ai-in-the…
DeepSeek was again the trigger for the third post, with their innovative Engram architecture, which promises to increase the reasoning capacity of Transformer-based LLMs, by offloading knowledge retrieval to a separate module.
wheremachinesthink.substack.com/p/deepseek-drops-yet-anot…
The previous post then led to an exploration of the multilayer perceptron's role inside the Transformer, and why it's become the focus of optimization efforts, such as DeepSeek's Engram.
wheremachinesthink.substack.com/p/the-enigmatic-multilaye…
And finally I wrote the first in what will be a series of posts intended to be a multilayered explainer of Transformers and Large Language Models, starting with a very high-level overview. wheremachinesthink.substack.com/p/a-primer-on-large-langu… -
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
If you are new to large language models and transformers, and want a quick introduction to the basics, with a promise of more in-depth accounts to come, here's a primer to get started from the WHERE MACHINES THINK Substack. wheremachinesthink.substack.com/p/a-primer-on-large-langu…
-
Anil Ananthaswamy reposted
Nick Bailey
@NickBailey596
I'm almost finished with the book by the same author, Why Machines Learn, and it's been enormously useful for me to understand ML. The substack posts I've read have been similar quality. I definitely recommend it.
Anil Ananthaswamy
@anilananth
If you are new to large language models and transformers, and want a quick introduction to the basics, with a promise of more in-depth accounts to come, here's a primer to get started from the WHERE MACHINES THINK Substack. wheremachinesthink.substack.com/p/a-primer-on-large-langu…
-
Anil Ananthaswamy
@anilananth
If you are new to large language models and transformers, and want a quick introduction to the basics, with a promise of more in-depth accounts to come, here's a primer to get started from the WHERE MACHINES THINK Substack. wheremachinesthink.substack.com/p/a-primer-on-large-langu…
-
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
Transformer-based LLMs are the most significant technology of the past decade. This is first in a series of posts for the WHERE MACHINES THINK Substack, exploring Transformers and LLMs at various levels of abstraction, digging deeper with each post. wheremachinesthink.substack.com/p/a-primer-on-large-langu…
-
Anil Ananthaswamy
@anilananth
Transformer-based LLMs are the most significant technology of the past decade. This is first in a series of posts for the WHERE MACHINES THINK Substack, exploring Transformers and LLMs at various levels of abstraction, digging deeper with each post. wheremachinesthink.substack.com/p/a-primer-on-large-langu…
-
Anil Ananthaswamy
@anilananth
If you can get past the impression that I'm yelling at @briankeating in this image collage :-) ..."HOW DOES IT WORK?"... I had a lovely conversation about Why Machines Learn for an episode of Brian's Into The Impossible podcast.
Prof. Brian Keating
@Briankeating
Scientists are taught to ask how but today’s guest @anilananth asks ‘why’ - Why Machines Learn?
Our conversation explores the underlying reasons for the success of large language models (LLMs), addressing questions such as how these models generalize despite issues like overparameterization and limited training data. Link in reply. -
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
Ever since the Transformer came out in 2017 (as detailed in the Attention Is All You Need paper), it has both revolutionized language models and been the subject of incessant improvements. Often, it's the Transformer's Attention mechanism that gets all the, well, attention, but there's another beast lurking inside each Transformer layer — the feed-forward network block, or the multilayer perceptron. This post explains why MLPs inside Transformers are the real computation hogs and have thus become the latest targets for optimization efforts. wheremachinesthink.substack.com/p/the-enigmatic-multilaye…
-
Anil Ananthaswamy
@anilananth
Ever since the Transformer came out in 2017 (as detailed in the Attention Is All You Need paper), it has both revolutionized language models and been the subject of incessant improvements. Often, it's the Transformer's Attention mechanism that gets all the, well, attention, but there's another beast lurking inside each Transformer layer — the feed-forward network block, or the multilayer perceptron. This post explains why MLPs inside Transformers are the real computation hogs and have thus become the latest targets for optimization efforts. wheremachinesthink.substack.com/p/the-enigmatic-multilaye…
-
Anil Ananthaswamy
@anilananth
Chennai folks interested in the future of AI: I'm organizing a seminar on Spiking Neural Networks and Neuromorphic Hardware at @iitmadras's Wadhwani School of Data Science and AI (@WSAI_IITM) on Jan 30th. This is under the aegis of the Future of AI pod of the CoIN, the Co-Intelligence Network. Please use the QR code to register. It's free, but space is limited.
-
Anil Ananthaswamy reposted
Saganism
@Saganismm
A lesson with Brian Cox on how Einstein postulated that time is relative.
x.com/Saganismm/status/1918381089934196826/video/1 -
Anil Ananthaswamy
@anilananth
Will M pledged support for the WHERE MACHINES LEARN substack. "I studied Bayesian nonparametrics, variational inference, Q-learning, and all that back in 2010. I really enjoy Anil's masterful mix of history and intuition (along with the important bits of the actual math)."
-
Anil Ananthaswamy
@anilananth
On New Year's Eve and then two weeks later, DeepSeek reported two advances for LLMs: one used a foundation result in matrix theory from 1967 to design mHC, Manifold-Constrained Hyper-Connections, which are souped-up Resnets; and another used ideas from standard NLP research to design a learnable, static memory for LLMs, freeing up transformer layers for more reasonning.
My deepish-dives into these innovations, via the WHERE MACHINES THINK Substack:
wheremachinesthink.substack.com/p/deepseek-and-the-day-be…
wheremachinesthink.substack.com/p/deepseek-drops-yet-anot… -
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
We want LLMs to be factual. But #llms don't look up facts from some table, they recompute known patterns and knowledge for each query, thus wasting resources and making mistakes. What if specialized neural network layers could learn to store and retrieve factual knowledge in a way that mimics looking up information, and thus free up the Transformer layers for reasoning? @deepseek_ai seems to have done it!
Here's my Substack post explaining DeepSeek's possibly potent change to the LLM architecture: wheremachinesthink.substack.com/p/deepseek-drops-yet-anot… -
Anil Ananthaswamy
@anilananth
We want LLMs to be factual. But #llms don't look up facts from some table, they recompute known patterns and knowledge for each query, thus wasting resources and making mistakes. What if specialized neural network layers could learn to store and retrieve factual knowledge in a way that mimics looking up information, and thus free up the Transformer layers for reasoning? @deepseek_ai seems to have done it!
Here's my Substack post explaining DeepSeek's possibly potent change to the LLM architecture: wheremachinesthink.substack.com/p/deepseek-drops-yet-anot… -
Anil Ananthaswamy
@anilananth
DeepSeek's relentless pace of innovation continues. Or maybe they are among the only ones publishing their results! Regardless, their latest tweak to the standard LLM architecture promises to open a new front for scaling up LLMs. It's a learnable, static memory storage and retrieval module that's integrated into the LLM, not hacked on. It frees up the LLM layers from having to compute information about patterns and factual content, and instead use the compute for more important tasks, such as reasoning. My Substack post explaining this latest drop from DeepSeek. wheremachinesthink.substack.com/p/deepseek-drops-yet-anot…
-
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
One of the co-authors of the Attention Is All You Need paper, Llion Jones, talked of how AI research might be stuck in a local minima in a recent episode of @MLStreetTalk ... One quote of his got me thinking, about the quote, and about researching/writing about AI in the age of AI. My latest post for WHERE MACHINES THINK:
wheremachinesthink.substack.com/p/writing-about-ai-in-the… -
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
A new Substack, WHERE MACHINES THINK, which builds on WHY MACHINES LEARN. We'll be exploring and understanding the mathematical spaces that enable artificial (and maybe natural) intelligence. Read essays and analyses at the intersection of machine learning, neuroscience and physics. Read & Subscribe: wheremachinesthink.substack.com
-
Anil Ananthaswamy
@anilananth
A new Substack, WHERE MACHINES THINK, which builds on WHY MACHINES LEARN. We'll be exploring and understanding the mathematical spaces that enable artificial (and maybe natural) intelligence. Read essays and analyses at the intersection of machine learning, neuroscience and physics. Read & Subscribe: wheremachinesthink.substack.com
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
2/2 In unsupervised learning, we optimize one objective (next word prediction, say) but care about a different objective (reasoning, say). "Yet, it helps. How can that be? @ilyasut at the Simons Institute workshop on Large Language Models and Transformers. simons.berkeley.edu/talks/ilya-sutskever-openai-2023-08-14
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
1/2 An oldie — in LLM years — but goodie! Ilya Sutskever, @ilyasut, speaks on his observations about a theory of unsupervised learning at the Simons Institute workshop on Large Language Models and Transformers, in Aug 2023. Video: simons.berkeley.edu/talks/ilya-sutskever-openai-2023-08-14
-
Anil Ananthaswamy
@anilananth
One of the co-authors of the Attention Is All You Need paper, Llion Jones, talked of how AI research might be stuck in a local minima in a recent episode of @MLStreetTalk ... One quote of his got me thinking, about the quote, and about researching/writing about AI in the age of AI. My latest post for WHERE MACHINES THINK:
wheremachinesthink.substack.com/p/writing-about-ai-in-the… -
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
What connects a 250-year-old problem posed by French mathematician Gaspard Monge involving the task of efficiently moving mounds of dirt from one place to another and the architectural tweak in DeepSeek's latest paper? The story in my first post for WHERE MACHINES THINK:
-
Anil Ananthaswamy reposted
Goutham Kurra
@gkurra
@anilananth 's book is an amazing introduction to AI for the curious. Even though it's aimed at lay audiences who don't mind putting in some tiny effort, it offers something in terms of perspective even for the seasoned practitioner. Excited for his newsletter.
Anil Ananthaswamy
@anilananth
I'm starting a Substack newsletter, WHERE MACHINES THINK (just imagine scare quotes around the word think, to maintain appropriate skepticism). The welcome post is here: wheremachinesthink.substack.com/p/welcome-to-where-machin…
-
Anil Ananthaswamy reposted
Mrutyunjay Biswal
@zaldrizesdarios
As a loyal reader of Anil’s work, couldn’t have been more thankful for this substack, particularly after reading Why Machines Learn?
Anil Ananthaswamy
@anilananth
I'm starting a Substack newsletter, WHERE MACHINES THINK (just imagine scare quotes around the word think, to maintain appropriate skepticism). The welcome post is here: wheremachinesthink.substack.com/p/welcome-to-where-machin…
-
Anil Ananthaswamy
@anilananth
What connects a 250-year-old problem posed by French mathematician Gaspard Monge involving the task of efficiently moving mounds of dirt from one place to another and the architectural tweak in DeepSeek's latest paper? The story in my first post for WHERE MACHINES THINK:
-
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
I'm starting a Substack newsletter, WHERE MACHINES THINK (just imagine scare quotes around the word think, to maintain appropriate skepticism). The welcome post is here: wheremachinesthink.substack.com/p/welcome-to-where-machin…
-
Anil Ananthaswamy
@anilananth
I'm starting a Substack newsletter, WHERE MACHINES THINK (just imagine scare quotes around the word think, to maintain appropriate skepticism). The welcome post is here: wheremachinesthink.substack.com/p/welcome-to-where-machin…
-
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
Something to keep in mind as we head toward an AI-enabled (AI-riddled?) future... these prescient words by Marvin Minsky in 1986: "What if some visitor from Mars had come a billion years ago to judge the fate of earthly life from watching clumps of cells that hadn't even learned to crawl? In the same way, we cannot grasp the range of what machines may do in the future from seeing what's on view right now." Marvin Minsky in his book, The Society of Mind (1986)
-
Anil Ananthaswamy
@anilananth
Something to keep in mind as we head toward an AI-enabled (AI-riddled?) future... these prescient words by Marvin Minsky in 1986: "What if some visitor from Mars had come a billion years ago to judge the fate of earthly life from watching clumps of cells that hadn't even learned to crawl? In the same way, we cannot grasp the range of what machines may do in the future from seeing what's on view right now." Marvin Minsky in his book, The Society of Mind (1986)
-
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
Beginning the New Year at my alma matter, IIT-Madras, for reasons I'd not have imagined a few months ago. I've joined @iitmadras, as Professor of Practice, in the Department of Data Science and Artificial Intelligence. Decades after having been here for my Bachelor's, the campus feels simultaneously unchanged and different! The deer, the occasional monkey (this one's checking out the dining room), the gorgeous banyan trees are all there, as are many, many more students, new buildings, etc. ... Looking forward to this year.
-
Anil Ananthaswamy
@anilananth
Beginning the New Year at my alma matter, IIT-Madras, for reasons I'd not have imagined a few months ago. I've joined @iitmadras, as Professor of Practice, in the Department of Data Science and Artificial Intelligence. Decades after having been here for my Bachelor's, the campus feels simultaneously unchanged and different! The deer, the occasional monkey (this one's checking out the dining room), the gorgeous banyan trees are all there, as are many, many more students, new buildings, etc. ... Looking forward to this year.
-
Anil Ananthaswamy
@anilananth
I think this weekend is going to be taken up by this unfairly delicious offering from @MLStreetTalk : A 3-hour+ episode with the author of one my favorite books of recent times, @maxsbennett, who wrote A Brief History of Intelligence... I had been unduly worried that MLST was ditching the long-format episodes for shorter, more consumable 50-min ones, but I needn't have, it looks like.... www.youtube.com/watch?v=RvYSsi6rd4g
-
Anil Ananthaswamy reposted
Kirk Borne
@KirkDBorne
Why Machines Learn — The Elegant Math Behind Modern AI: www.amazon.com/gp/product/0593185749?tag=kirkdborne-20&la…
+
Deep Learning Foundations and Concepts: www.amazon.com/Deep-Learning-Foundations-Christopher-Bish…
——————
#datascience #ai #ml #machinelearning #mathematics #linearalgebra #probability #calculus #datascientist -
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
With @sapinker and others posting about this paper on "Epistemological Fault Lines Between Human and Artificial Intelligence," it's getting a lot of attention. Lots to parse here, but there are something aspects of the comparison between human minds and AIs that don't sit well. While almost no one disputes that human and artificial intelligence (especially LLMs) have significant differences, the fault lines identified in this paper seem problematic. Some early thoughts:
The paper says: "LLMs lack metacognition entirely". There's growing literature that suggests this is not correct. Yes, one can quibble about the exact definition of metacognition, but here are some papers on the topic:
Language Models Are Capable of Metacognitive Monitoring and Control of Their Internal Activations arxiv.org/abs/2505.13763
Metacognitive Capabilities of LLMs: An Exploration in Mathematical Problem Solving arxiv.org/abs/2405.12205
(this is work by @Yoshua_Bengio's and @prfsanjeevarora's groups.) There's plenty of work in this area, with serious implications for AI safety and risk.
Also, the human pathway to judgement, the way it's outlined, seems to suggest there is nothing probabilistic happening under the hood. But in the depicted pathway is "perception," and there's considerable literature about human perception also being probabilistic, in terms of inferring the external causes of sensory inputs. See Jacob Feldman's "Probabilistic models of perceptual features": ruccs.rutgers.edu/images/personal-jacob-feldman/papers/fe…
From the paper: "There is a growing consensus that the underlying neural code is inherently probabilistic."
Also, the paper being debated ("Epistemological Fault Lines Between Human and Artificial Intelligence") argues: "The Motivation fault points to the role of emotions, goals, and values in guiding human cognition, contrasted with the goal-free optimization dynamics of LLMs."
The claim about LLMs is misleading. While next-token prediction can be argued as being 'goal-free,' the same cannot be said of supervised fine-tuning and reinforcement learning during post-training.
Valerio Capraro
@ValerioCapraro
Major preprint just out!
We compare how humans and LLMs form judgments across seven epistemological stages.
We highlight seven fault lines, points at which humans and LLMs fundamentally diverge:
The Grounding fault: Humans anchor judgment in perceptual, embodied, and social experience, whereas LLMs begin from text alone, reconstructing meaning indirectly from symbols.
The Parsing fault: Humans parse situations through integrated perceptual and conceptual processes; LLMs perform mechanical tokenization that yields a structurally convenient but semantically thin representation.
The Experience fault: Humans rely on episodic memory, intuitive physics and psychology, and learned concepts; LLMs rely solely on statistical associations encoded in embeddings.
The Motivation fault: Human judgment is guided by emotions, goals, values, and evolutionarily shaped motivations; LLMs have no intrinsic preferences, aims, or affective significance.
The Causality fault: Humans reason using causal models, counterfactuals, and principled evaluation; LLMs integrate textual context without constructing causal explanations, depending instead on surface correlations.
The Metacognitive fault: Humans monitor uncertainty, detect errors, and can suspend judgment; LLMs lack metacognition and must always produce an output, making hallucinations structurally unavoidable.
The Value fault: Human judgments reflect identity, morality, and real-world stakes; LLM "judgments" are probabilistic next-token predictions without intrinsic valuation or accountability.
Despite these fault lines, humans systematically over-believe LLM outputs, because fluent and confident language produce a credibility bias.
We argue that this creates a structural condition, Epistemia:
linguistic plausibility substitutes for epistemic evaluation, producing the feeling of knowing without actually knowing.
To address Epistemia, we propose three complementary strategies: epistemic evaluation, epistemic governance, and epistemic literacy.
Full paper in the first reply.
Joint with @Walter4C & @matjazperc -
Anil Ananthaswamy
@anilananth
With @sapinker and others posting about this paper on "Epistemological Fault Lines Between Human and Artificial Intelligence," it's getting a lot of attention. Lots to parse here, but there are something aspects of the comparison between human minds and AIs that don't sit well. While almost no one disputes that human and artificial intelligence (especially LLMs) have significant differences, the fault lines identified in this paper seem problematic. Some early thoughts:
The paper says: "LLMs lack metacognition entirely". There's growing literature that suggests this is not correct. Yes, one can quibble about the exact definition of metacognition, but here are some papers on the topic:
Language Models Are Capable of Metacognitive Monitoring and Control of Their Internal Activations arxiv.org/abs/2505.13763
Metacognitive Capabilities of LLMs: An Exploration in Mathematical Problem Solving arxiv.org/abs/2405.12205
(this is work by @Yoshua_Bengio's and @prfsanjeevarora's groups.) There's plenty of work in this area, with serious implications for AI safety and risk.
Also, the human pathway to judgement, the way it's outlined, seems to suggest there is nothing probabilistic happening under the hood. But in the depicted pathway is "perception," and there's considerable literature about human perception also being probabilistic, in terms of inferring the external causes of sensory inputs. See Jacob Feldman's "Probabilistic models of perceptual features": ruccs.rutgers.edu/images/personal-jacob-feldman/papers/fe…
From the paper: "There is a growing consensus that the underlying neural code is inherently probabilistic."
Also, the paper being debated ("Epistemological Fault Lines Between Human and Artificial Intelligence") argues: "The Motivation fault points to the role of emotions, goals, and values in guiding human cognition, contrasted with the goal-free optimization dynamics of LLMs."
The claim about LLMs is misleading. While next-token prediction can be argued as being 'goal-free,' the same cannot be said of supervised fine-tuning and reinforcement learning during post-training.
Valerio Capraro
@ValerioCapraro
Major preprint just out!
We compare how humans and LLMs form judgments across seven epistemological stages.
We highlight seven fault lines, points at which humans and LLMs fundamentally diverge:
The Grounding fault: Humans anchor judgment in perceptual, embodied, and social experience, whereas LLMs begin from text alone, reconstructing meaning indirectly from symbols.
The Parsing fault: Humans parse situations through integrated perceptual and conceptual processes; LLMs perform mechanical tokenization that yields a structurally convenient but semantically thin representation.
The Experience fault: Humans rely on episodic memory, intuitive physics and psychology, and learned concepts; LLMs rely solely on statistical associations encoded in embeddings.
The Motivation fault: Human judgment is guided by emotions, goals, values, and evolutionarily shaped motivations; LLMs have no intrinsic preferences, aims, or affective significance.
The Causality fault: Humans reason using causal models, counterfactuals, and principled evaluation; LLMs integrate textual context without constructing causal explanations, depending instead on surface correlations.
The Metacognitive fault: Humans monitor uncertainty, detect errors, and can suspend judgment; LLMs lack metacognition and must always produce an output, making hallucinations structurally unavoidable.
The Value fault: Human judgments reflect identity, morality, and real-world stakes; LLM "judgments" are probabilistic next-token predictions without intrinsic valuation or accountability.
Despite these fault lines, humans systematically over-believe LLM outputs, because fluent and confident language produce a credibility bias.
We argue that this creates a structural condition, Epistemia:
linguistic plausibility substitutes for epistemic evaluation, producing the feeling of knowing without actually knowing.
To address Epistemia, we propose three complementary strategies: epistemic evaluation, epistemic governance, and epistemic literacy.
Full paper in the first reply.
Joint with @Walter4C & @matjazperc -
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
1/4 Do LLMs understand? "They understand in a way that’s very different from how humans understand," Dileep George, @dileeplearning, of @GoogleDeepMind at the Simons Institute workshop on The Future of Language Models and Transformers. Video: simons.berkeley.edu/talks/dileep-george-google-deepmind-2…
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
2/2 To elucidate, @Cornell's Kilian Weinberger gave a short masterclass on diffusion models and how they can be used to guide of transformer-based language models, at the Simons Institute workshop on The Future of Language Models and Transformers. Video: simons.berkeley.edu/talks/kilian-weinberger-cornell-unive…
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
1/2 "LLMs are amazing, but hard to control. Diffusion models are also amazing, and (maybe too) easy to control." @Cornell's Kilian Weinberger, at the Simons Institute, on how these contrasting properties can be used to steer the output of LLMs. Video: simons.berkeley.edu/talks/kilian-weinberger-cornell-unive…
-
Anil Ananthaswamy reposted
Dickie Bush
@dickiebush
I asked 430,000 people for their favorite book of 2025.
These 12 books showed up the most often (so they must be worth reading):🧵 -
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
4/4 “I claim this is the future of LLMs. My prediction is that in a couple of years people are going to look back and think it’s crazy to cram all of this into the model weights,” @Cornell's Kilian Weinberger at the Simons Institute. Video: simons.berkeley.edu/talks/kilian-weinberger-cornell-unive…
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
3/4 Large Memory Language Model (LMLM) “is a very simple trick…basically a pre-training method that separates tail knowledge and common knowledge": Kilian Weinberger at the Simons Institute’s workshop on The Future of Language Models and Transformers. simons.berkeley.edu/talks/kilian-weinberger-cornell-unive…
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
2/4 “We can make the model much smaller, faster to train, cheaper to operate. And we are no longer limited by the model weights for the tail knowledge. We can make this external database really big,” @Cornell's Kilian Weinberger at the Simons Institute. simons.berkeley.edu/talks/kilian-weinberger-cornell-unive…
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
1/4 LLMs learn language competency, factual common knowledge and factual tail knowledge. Factual knowledge has a heavy tail and should be stored in a database. “It doesn’t belong in the model weights”: @Cornell's Kilian Weinberger at the Simons Institute. simons.berkeley.edu/talks/kilian-weinberger-cornell-unive…
-
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
I dedicated my book WHY MACHINES LEARN to "Teachers everywhere, sung and unsung." That dedication was in part to say thanks to teachers like @Cornell Prof. Kilian Weinberger, whose 39 hour-long lectures on ML, delivered to his class in 2018 (www.youtube.com/watch?v=MrLPzBxG95I&) were instrumental in my coming to grips with machine learning. Earlier this year I met Kilian in person @SimonsInstitute for the first time and had the joy of giving him a copy of WML.
-
Anil Ananthaswamy
@anilananth
I dedicated my book WHY MACHINES LEARN to "Teachers everywhere, sung and unsung." That dedication was in part to say thanks to teachers like @Cornell Prof. Kilian Weinberger, whose 39 hour-long lectures on ML, delivered to his class in 2018 (www.youtube.com/watch?v=MrLPzBxG95I&) were instrumental in my coming to grips with machine learning. Earlier this year I met Kilian in person @SimonsInstitute for the first time and had the joy of giving him a copy of WML.
-
Anil Ananthaswamy reposted
Brenden Lake
@LakeBrenden
How did AI and CogSci get to today? I had the privilege of an early read and couldn’t put this remarkable book down. Even if you think you know the key milestones, you probably don’t know all the stories (or characters) that Tom has been documenting for 10+ years.
Griffiths Computational Cognitive Science Lab
@cocosci_lab
Excited to announce a new book telling the story of mathematical approaches to studying the mind, from the origins of cognitive science to modern AI! The Laws of Thought will be published in February, and is available for pre-order now.
-
Anil Ananthaswamy reposted
Prateek Mehta
@prateekmehta42
My chat at @spc_india with @anilananth was really fun and intellectually stimulating!
Way to round off a great year — talking about curiosity, the importance of asking foundational questions...and what human intelligence might actually mean in a world of machines! -
Anil Ananthaswamy reposted
Subbarao Kambhampati (కంభంపాటి సుబ్బారావు)
@rao2z
Chatting #ai with @anilananth over ravva dosa/poori/idlee/kapi at Tivoli garden in B'lore.. 😎
-
Anil Ananthaswamy reposted
Tushar Garg
@garg_tushar_
Great to have Anil Ananthaswamy at SPC today for a chat on the future of intelligent machines—sharp ideas, thoughtful questions, and plenty to reflect on as we think about where AI is headed. 🚀
The conversation was also a reminder of the power of writing: a way to think more intentionally and stay curious about life’s bigger questions.
How do you think we should be asking the right questions about intelligence today?
@spc_india @prateekmehta42 @anilananth -
Anil Ananthaswamy
@anilananth
It was a joy to watch this substantive discussion. @JannaLevin for @PioneerWorks_ talking to @ylecun and Adam Brown of Google DeepMind, about many things AI! And a surprise plant in the audience toward the end shouts out questions about machine consciousness!
-
Anil Ananthaswamy reposted
-
Anil Ananthaswamy
@anilananth
If I had any doubts about the interest/concern/worries about AI, what it is, where it's going, the effect it'll have on society, etc., it was amply assuaged after last night's conversation at the Lightroom Bookstore (which formerly sold only children's books but is branching into books for adults, and this event was its first one for grown-ups) ...A small, intimate room in Bangalore's Cooke Town packed with an extremely engaged audience.
Mukund Thattai of @NCBS_Bangalore was the perfect interlocutor! We need many more of such conversations, to prepare for what's in store as AI gets embedded into many aspects of work, education, society, and so on... -
Anil Ananthaswamy
@anilananth
"It's the G in AGI, the generality that I find interesting." -Shane Legg (co-founder, DeepMind), who co-coined the phrase Artificial General Intelligence. An interesting discussion about why the community came to focus on AGI, what it is (according to Legg), and when do we get there...
youtube.com/clip/UgkxeI8wsHcyEK7H_EHcKhB95iWWm_AzmY9m -
Anil Ananthaswamy reposted
Ashish Vaswani
@ashVaswani
Rnj-1-Instruct is now the #1 trending text generation model on HF!
-
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
Tomorrow, Mukund Thattai of @NCBS_Bangalore and I will be chatting about WHY MACHINES LEARN ...at the Lightroom Bookstore in Cooke Town, Bangalore, formerly a kids-only bookstore and now branching into books for adults, and that too with an event about the mathematics of AI. Very grown-up :-)
-
Anil Ananthaswamy
@anilananth
Tomorrow, Mukund Thattai of @NCBS_Bangalore and I will be chatting about WHY MACHINES LEARN ...at the Lightroom Bookstore in Cooke Town, Bangalore, formerly a kids-only bookstore and now branching into books for adults, and that too with an event about the mathematics of AI. Very grown-up :-)
-
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
I can't believe that the @MLStreetTalk episode on WHY MACHINES LEARN has crossed half-a-million views! It was the first video podcast I taped for the book, the day after it was published in July '24--thanks to some great coffee brewed by @ecsquendor in London to wake up my jet-lagged brain (during a 8-hr layover on a flight from the US to India)
www.youtube.com/watch?v=URtF_UHYBSo& -
Anil Ananthaswamy
@anilananth
I can't believe that the @MLStreetTalk episode on WHY MACHINES LEARN has crossed half-a-million views! It was the first video podcast I taped for the book, the day after it was published in July '24--thanks to some great coffee brewed by @ecsquendor in London to wake up my jet-lagged brain (during a 8-hr layover on a flight from the US to India)
www.youtube.com/watch?v=URtF_UHYBSo& -
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
WHY MACHINES LEARN at @Bookhive_blr ... Am pleased that the math-laden book has found an audience in Bangalore...
-
Anil Ananthaswamy
@anilananth
WHY MACHINES LEARN at @Bookhive_blr ... Am pleased that the math-laden book has found an audience in Bangalore...
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
1/2 "Symmetries play a fundamental role in machine learning." Soledad Villar of @JohnsHopkins speaking at the Simons Institute workshop on Randomness, Invariants, and Complexity. Video: simons.berkeley.edu/talks/soledad-villar-johns-hopkins-un…
-
Anil Ananthaswamy reposted
0xSero
@0xSero
If you are interested in going deeper into AI, and have a better understanding of what it is (and isn’t) there’s no better book to read than this.
I have read this book twice and I will be reading it again this time while solving the math in the book by hand over the next 30 days. -
Anil Ananthaswamy reposted
Prateek Mehta
@prateekmehta42
As we round off a fantastic year, we have a bunch of events lined up at @spc_india! Excited about this fireside chat on 'The Future of Intelligent Machines' — with @anilananth and hosted by @finn_meeks! It's going to be a session made for the really curious ones.
Anil is one of the few with the technical expertise, passion and storyteller's touch — to truly bring this subject to life.
Apply on Luma now.
(if you haven't already, now is a great time to read 'Why Machines Learn') -
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
1/3 "What are the conditions for an AI system in the future to cause catastrophic harm?" Turing Award winner @Yoshua_Bengio asked, during his Richard M. Karp Distinguished Lecture at the Simons Institute earlier this year. www.youtube.com/watch?v=g0ljOgBo5uY
-
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
-
Anil Ananthaswamy
@anilananth
-
Anil Ananthaswamy
@anilananth
In talks about LLMs, I've been using the sentence "The dog ate my ___" as an example of how to get the LLM to correctly predict the masked word, which in this case is "homework." It's a popular phrase in the US, but not so much in India. Then a professor at @iitmadras showed me a tutorial he had prepared that his dog chewed up!
More importantly, the real threat on campus are monkeys, who apparently steal people's bags and smartphones. So, I might have to modify the sentence to "The monkey stole my ___," where the missing word is "smartphone." Of course, given the lack of occurrence of this sentence in the training corpus of LLM , a pre-trained #llm is unlikely to get this right! -
Anil Ananthaswamy reposted
Finn Meeks
@finn_meeks
Bangalore! I’ll be hosting a chat @spc_india with award-winning author @anilananth to discuss Why Machines Learn and the frontier of AI.
Anil is excellent at translating both what we understand and what we have yet to learn in AI. I hope our talk will reveal great answers...and great questions.
If you’re building in AI or just curious about where machine intelligence is headed, I hope that you can join us! -
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
Gosh, @seanmcarroll's Mindscape podcast is one of my absolute favorite podcasts. So to be his guest on this week's episode was a joy and more than bit surreal. Listening to it, I noticed at least one flub (I say somewhere that Kepler knew calculus, which he didn't, given he came before the invention of calculus!) art19.com/shows/sean-carrolls-mindscape/episodes/d5bc22cf…
-
Anil Ananthaswamy
@anilananth
Gosh, @seanmcarroll's Mindscape podcast is one of my absolute favorite podcasts. So to be his guest on this week's episode was a joy and more than bit surreal. Listening to it, I noticed at least one flub (I say somewhere that Kepler knew calculus, which he didn't, given he came before the invention of calculus!) art19.com/shows/sean-carrolls-mindscape/episodes/d5bc22cf…
-
Anil Ananthaswamy reposted
Indulekha Aravind
@Indulekha_A
Thrilled to be moderating this conversation at @BlrLitFest with two stellar journalists and experts on AI: the amazing @_KarenHao, author of Empire of AI & @anilananth , who wrote the remarkable "Why Machines Learn"
Come one, come all for "Aye, Aye, AI" on Dec 6 :) -
Anil Ananthaswamy
@anilananth
One of the unexpected consequences of getting steeped in the math of machine learning has been reconnecting with ideas from physics, or maybe even understanding them clearly for the first time. For instance, grasping the actual meaning of the phrase "thermal equilibrium" in statistical physics. Here's Geoff Hinton talking about it the context of Boltzmann Machines. The full video of his The 2025 Martin Lecture at the University of Toronto is here: www.youtube.com/watch?v=juif0T8NOsY
-
Anil Ananthaswamy
@anilananth
Backpropagation Works Much Better Than the Brain! We've only understood this in the last few years, says Geoff Hinton, providing a counter-intuitive reason (besides other better-known reasons we have known about) for why the brain is not using backpropagation to learn the weights for the connections between neurons. youtube.com/clip/UgkxweHSnmJglVMhMVjKhLQUoBfBprUsv6kx
-
Anil Ananthaswamy reposted
Shubham Verma
@shubhamv_cloud
Starting this book and already experiencing elevated curiosity levels.
Why Machines Learn by Anil Ananthaswamy is registering as a high-value data source on the elegant math behind modern AI, super accessible and super cool, with expected outcomes including improved understanding of neural networks, optimization pathways, geometric intuitions, and other surprisingly beautiful system behaviors.
Once i done reading this book, share my learning here
#ai #machinelearning #deeplearning #math #books #nowreading -
Anil Ananthaswamy
@anilananth
Aye, Aye, AI. Bangalore Lit Fest, 2025. On 6 Dec, 615PM, @_KarenHao and I will be on a panel discussing many things AI, with moderator @Indulekha_A. Looking forward to sharing the stage with fellow @KSJatMIT alum! @BlrLitFest #blrlitfest
-
Anil Ananthaswamy
@anilananth
Chasing a Conjecture: Bangalore Lit Fest, 2025. On 7 Dec, 4PM, I'll be talking to mathematician Prof. Chandrashekar Khare about his new book. #blrlitfest @BlrLitFest
-
Anil Ananthaswamy reposted
Florin
@florin_dev
It’s time, guys. Math ftw.
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
2/2 "As much as we'd think that LLMs are everywhere, I'd also say graphs are everywhere...If you squint, [LLM reasoning] is a graph learning problem." Mikhail Galkin of @GoogleResearch at the Simons Institute's workshop on Graph Learning Meets Theoretical Computer Science.
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
1/2 "What are graph foundation models? How are they different from language models? Or maybe not," asked Mikhail Galkin of @GoogleResearch at the Simons Institute's workshop on Graph Learning Meets Theoretical Computer Science. Video: simons.berkeley.edu/talks/mikhail-galkin-google-research-…
-
Anil Ananthaswamy
@anilananth
Congratulations Sairam!
Sairam Sundaresan
@DSaience
I can't believe I'm saying this - I'm officially a published author :D
After three years, my first book is out.
"AI for the Rest of Us" with @BloomsburyAcad is finally in the world.
I wrote it because I watched too many people get left behind in AI conversations.
The gap between executives and engineers kept growing.
Everyone else just nodded along, trying to translate both languages at once.
So I built a bridge. -
Anil Ananthaswamy reposted
Sairam Sundaresan
@DSaience
I can't believe I'm saying this - I'm officially a published author :D
After three years, my first book is out.
"AI for the Rest of Us" with @BloomsburyAcad is finally in the world.
I wrote it because I watched too many people get left behind in AI conversations.
The gap between executives and engineers kept growing.
Everyone else just nodded along, trying to translate both languages at once.
So I built a bridge. -
Anil Ananthaswamy reposted
Héctor Lira
@HectorLiraT
"The double-slit experiment doesn't merely embody wave-particle duality [...] it incorporates entanglement too. Once physicists began appreciating this, [...] It made possible the delayed-choice quantum eraser experiment."
- @anilananth -
Anil Ananthaswamy reposted
Bangalore Lit Fest
@BlrLitFest
From science to philosophy, step into a world of bold ideas and questions.
Meet @anilananth, Chandrashekhar Khare, @authorgayathri, @KSridhar921971, @ConsultVistas, Ragini Tharoor Srinivasan & Sundar Sarukkai at #blrlitfest 2025.
🗓 6 & 7 Dec 2025
📍 Freedom Park
🎟 Free entry -
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
2/2 Multiplying 2 n x n matrices requires O(n^w) arithmetic operations, where w=3 for the brute force algorithm. Strassen's method was the first big improvement in '69 (w=2.81), followed by two big jumps in the '80s. The world record today is w=2.3714. simons.berkeley.edu/talks/virginia-vassilevska-williams-m…
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
1/2 "It seems silly, but it's a very important problem." Virginia Vassilevska Williams (@MIT) on the progress in matrix multiplication algorithms during her Richard M. Karp Distinguished Lecture On Matrix Multiplication Algorithms at the Simons Institute. simons.berkeley.edu/talks/virginia-vassilevska-williams-m…
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
Ever wondered about graph learning? Watch Ameya Velingker (@ameya_pa) and Haggai Maron (@HaggaiMaron) give a masterful introduction at the Simons Institute's workshop on Graph Learning Meets Theoretical Computer Science. Video: simons.berkeley.edu/talks/ameya-velingker-2025-08-11
-
Anil Ananthaswamy
@anilananth
"With that Von Neumann proved that in order to have life you have to have universal computation. You can't reproduce without computation. No computation, no life. This is a really profound insight and one that I think that most biologists and most computer scientists are unaware of." @blaiseaguera during a talk @longnow on What Is Intelligence?
youtube.com/clip/UgkxKNrAkheANRE1BywrVtymTsNwK7EMG3H7 -
Anil Ananthaswamy
@anilananth
This has to be one of my favorite episodes on @MLStreetTalk! @blaiseaguera's erudition is striking (references to books, papers, authors and ideas just effortlessly flow from him), and his firm thoughtfulness is refreshing, even if you were to disagree with him. I'd have loved to hear him pressed more on how functionalism can explain consciousness--in all its nitty-gritty details--but maybe that's a longer, different episode (@ecsquendor?) ... Also, maybe I didn't notice it earlier, but MLST has upped its production values! www.youtube.com/watch?v=rMSEqJ_4EBk
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
2/2 There are matrix multiplication algorithms that can do better than Strassen's but only for astronomically large matrices, making them impractical, said Oded Schwartz at the Simons Institute's workshop on Complexity and Linear Algebra Boot Camp. Video: simons.berkeley.edu/talks/oded-schwartz-hebrew-university…
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
1/2 Should have paid attention to matrices during linear algebra classes! In 2026, AI will use ~1% of global electricity, of which ~45-90% will be for matrix multiplications, said Oded Schwartz of Hebrew University of Jerusalem at the Simons Institute. simons.berkeley.edu/talks/oded-schwartz-hebrew-university…
-
Anil Ananthaswamy reposted
Gladys Lopez-Acevedo
@gladyslopezwb
Anil Ananthaswamy’s Why Machines Learn the mathematics that powers modern AI. A deep and clear exploration of how simple equations evolved into neural networks that learn from data.
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
2/2 For multiplying two n x n matrices, the arithmetic complexity of the standard method is of O(n^3); Strassen's method is of O(n^2.81). Prof. Olga Holtz spoke at the Simons Institute's workshop on Complexity and Linear Algebra Boot Camp. Video: simons.berkeley.edu/talks/olga-holtz-uc-berkeley-2025-09-…
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
1/2 Matrix multiplications are central to machine learning. UC Berkeley Professor Olga Holtz's analyzed, from scratch, the arithmetic complexity of matrix multiplication using Strassen's fast algorithm. She spoke at the Simons Institute. Video: simons.berkeley.edu/talks/olga-holtz-uc-berkeley-2025-09-…
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
2/2 One example showed a 1556x smaller carbon footprint using a better model, GPU, data center PUE and site, said @UCBerkeley's David Patterson at the Simons Institute’s workshop on Algorithmic Foundations for Emerging Computing Technologies Boot Camp. simons.berkeley.edu/talks/david-patterson-uc-berkeley-goo…
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
1/2 Focusing on the 4Ms of AI—model (FLOPs), machine (GPU, mWh / 1.0e12 FLOPs), mechanization (power usage effectiveness) and maps (sites w/ clean energy)—can reduce AI’s carbon footprint, said @UCBerkeley’s David Patterson, at the Simons Institute. Video: simons.berkeley.edu/talks/david-patterson-uc-berkeley-goo…
-
Anil Ananthaswamy
@anilananth
"DON'T PANIC!" -- when you find your book next to a book by one of your heroes! In this case, WHY MACHINES LEARN alongside Douglas Adams' THE HITCHHIKER'S GUIDE TO THE GALAXY.
Feels kind of apt that a book on the math of AI ends up in the Sci-Fi section 😀, especially when the adjacent book is face-up and screaming "DON'T PANIC"!
Thanks to Tom Griffiths of @Princeton for sending the pic, seen at NJ airport bookstore. -
Anil Ananthaswamy reposted
Luis Octavio Vado Grajales
@lovadograjales
Lectura nocturna @anilananth
-
Anil Ananthaswamy
@anilananth
The V-JEPA model by @ylecun's team @AIatMeta shows how learning in latent space and not in pixel space might help solve some of the shortcomings of today's GEN AI models (V-JEPA is not a generative model). Work by @garridoq_ and colleagues. @randall_balestr and @m_heilb gave great perspective ...
www.quantamagazine.org/how-one-ai-model-creates-a-physica… -
Anil Ananthaswamy reposted
Quanta Magazine
@QuantaMagazine
An AI model called V-JEPA is capable of “intuiting” the physical properties of the real world, gaining a sense of object permanence, the constancy of shape and color, and the effects of gravity. @anilananth reports: www.quantamagazine.org/how-one-ai-model-creates-a-physica…
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
2/2 Instead, pick the most efficient model, the latest GPU, and an efficient cloud data center in a low carbon location, said David Patterson, at the Simons Institute's workshop on Algorithmic Foundations for Emerging Computing Technologies Boot Camp. simons.berkeley.edu/talks/david-patterson-uc-berkeley-goo…
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
1/2 "Bad AI carbon footprint commandments." @UCBerkeley's David Patterson on his "don'ts" for reducing AI's carbon footprint: don't pick the biggest model; don't pick an older GPU; don't pick a local data center. He spoke at the Simons Institute. Video: simons.berkeley.edu/talks/david-patterson-uc-berkeley-goo…
-
Anil Ananthaswamy reposted
Andy Masley
@AndyMasley
This is why the only truly good book popularization of AI I've ever read is Why Machines Learn. Actually explains the math.
So much nonfiction writing is actually just a boring collection of anecdotes that just kind of bask in the warm glow of the importance of the topic.
Dwarkesh Patel
@dwarkesh_sp
I find it frustrating that almost every nonfiction book is basically just a history lesson, even if it's nominally about some science/tech/policy topic.
Nobody will just explain how something works.
Books about the semiconductor industry will never actually explain the basic process flow inside a fab, but you can bet that there will be a minute-by-minute recounting of a dramatic 1980s Intel boardroom battle. -
Anil Ananthaswamy
@anilananth
Been waiting for this @MLStreetTalk episode to drop! Andrew Gordon Wilson gives his take on deep learning and why it's not so mysterious or different, as is often claimed; Plus, I'm sure--given the nature of MLST episodes--much, much more. This should be 2 hrs of geeking out on ML ... @andrewgwils
www.youtube.com/watch?v=M-jTeBCEGHc -
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
2/2 "Combining AI with local knowledge to inform how we use the models has given us the success we’ve had." Rich Williams of @coral_gardeners at the Simons Institute's workshop on Decoding Communication in Nonhuman Species IV (co-hosted with Project CETI). simons.berkeley.edu/talks/rich-williams-coral-gardeners-2…
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
1/2 When the Beatles sang Octopus’s Garden, they could not have imagined real-life @coral_gardeners working with “the coral that lies beneath the waves.” Rich Williams spoke of using AI to restore coral reefs at scale at the Simons Institute's. Video: simons.berkeley.edu/talks/rich-williams-coral-gardeners-2…
-
Anil Ananthaswamy
@anilananth
Powerful message about the use of AI from @percyliang in his CS336 Stanford course on Language Modeling from Scratch: "AI tools can take away from learning, because there are cases where it can just solve the thing you want it to do. You can use them judiciously, but use them at your own risk. You are responsible for your own learning experience." youtu.be/SQ3fZ1sAqXI?list=PLoROMvodv4rOY23Y0BoGoBGgQ1zmU_…
-
Anil Ananthaswamy reposted
Asimov Press
@AsimovPress
A friendly reminder that the Central Dogma is NOT "DNA -> RNA -> protein."
All that Francis Crick *actually* said was: “Once information has got into a protein it can’t get out again.”
This negative statement — namely, that some transfers of information seem to be impossible — was the entire synopsis of Crick’s idea.
The DNA -> RNA -> protein idea was popularized by James Watson in his 1965 textbook, "The Molecular Biology of the Gene," as a way of summarizing how protein synthesis takes place. But Watson’s explanation was misleading. And, as we now know, there are many exceptions to that "version" of the Central Dogma (including reverse transcriptase.)
In 1956, Crick was working on a lecture that would bring together what was then known about the “flow of information” between DNA, RNA, and protein in cells. Crick formalized his ideas in what he called the Central Dogma, and his original conception of information flow within cells was both richer and more complex than Watson’s reductive presentation.
Crick was aware of at least four kinds of information transfers, all of which had been observed in biochemical studies by researchers at that time. These were: DNA → DNA (DNA replication), DNA → RNA (called transcription), RNA → protein (called translation) and RNA → RNA (a mechanism by which some viruses copy themselves). To summarize his thinking, Crick sketched out these information flows in a little figure that was never published.
Crick’s diagram also included two types of information flow for which there was no experimental evidence, but which he surmised might be biochemically feasible; namely, DNA → protein and RNA → DNA.
The direct synthesis of proteins using only DNA might be possible, Crick thought, because the sequence of bases in DNA ultimately determines the order of amino acids in a protein chain. If this were true, however, it would mean that RNA was not always involved in protein synthesis, even though every study at that time suggested it was. Crick therefore concluded that this kind of information flow was highly unlikely, though not impossible.
Crick also theorized that RNA → DNA was chemically possible, simply because it was the reverse of transcription and both types of molecules were chemically similar to each other. Still, Crick could not imagine any biological function for this so-called “reverse transcription,” so he portrayed this information flow as a dotted line in his diagram.
The most significant part of Crick’s idea — and the part that Watson ignored in his oversimplification — was that there are three flows of information that cannot occur, due both to lack of experimental evidence and any plausible biochemical mechanisms. These were protein → protein, protein → RNA and, above all, protein → DNA.
(From the Archives) "Francis Crick Was Misunderstood."
Read & subscribe: press.asimov.com/articles/crick -
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
1/2 "For the first time we have different minds, not only ours but various LLMs that we can study. There is an urgent need to understand why they work as well as they do...Soon, it may be too late." Tomaso Poggio at the Simons Institute. Video: simons.berkeley.edu/talks/tomaso-poggio-massachusetts-ins…
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
2/2 "If we can make progress in [the problem of intelligence], then we can more easily solve all other great problems." Tomaso Poggio at the Simons Institute's Smale@95: A Conference in Honor of Steve Smale simons.berkeley.edu/talks/tomaso-poggio-massachusetts-ins…
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
1/2 "Personally, I think that the problem of intelligence is not only one of the great problems in science, like the origin of life or the origin of the universe, but it's the greatest of all," said Tomaso Poggio at the Simons Institute. Video: simons.berkeley.edu/talks/tomaso-poggio-massachusetts-ins…
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
1/2 "[AI] is invading all sciences. AI is really machine learning. It was pretty clear to a number of us, at least a couple of decades ago...that the problem of learning is the gateway to the problem of intelligence." Tomaso Poggio at the Simons Institute simons.berkeley.edu/talks/tomaso-poggio-massachusetts-ins…
-
Anil Ananthaswamy
@anilananth
Frontier LLMs can serve as extremely useful tools to learn about AI/ML! Here's an example. I was reading a paper called "From Noise to Narrative: Tracing the Origins of Hallucinations in Transformers" arxiv.org/abs/2509.06938, by @danilobzdok, @soniajoseph_ and colleagues, trying to understand the technical details.
I fed a line from the paper into DeepSeek R1, with this prompt: In the context of transformers and sparse autoencoders, what is this sentence trying to say? Please explain: "SAEs learn a mapping function from intermediate transformer model activations to a sparse overcomplete representational space, which has an interpretable basis by nature, that can be used to faithfully reconstruct the latent activations."
Here's DeepSeek's answer. (Note the irony here: given that the question is about LLM hallucinations, I'm not 100% sure whether to trust its answer. That's where prompting another LLM with the same exact prompt -- I did so with Claude 4.0 -- helps cross-check, verify, do due diligence.)
DeepSeek: Of course. This is a dense sentence that packs several key concepts of mechanistic interpretability. Let's break it down piece by piece.
The Simple Analogy: A "Chef" Deconstructing a Dish
Imagine a complex, blended sauce. You taste it and know it's made of tomatoes, garlic, basil, and olive oil, but the flavors are all fused together.
* The Transformer's Activation: This is the final, blended sauce. It's a dense vector of numbers—highly informative but a "black box." You can't easily tell which individual ingredients (features) are present or in what quantity.
* The Sparse Overcomplete Representational Space: This is your full, massive spice rack and pantry. It contains *every possible ingredient* you could ever need (overcomplete), far more than any single recipe uses.
* The SAE's Mapping Function: You are the chef (the SAE). Your job is to taste the sauce (the activation) and figure out which few (sparse) ingredients from your massive pantry were used to make it.
* Faithfully Reconstruct the Latent Activations: You then try to mix those identified ingredients together to recreate the original sauce as perfectly as possible.
* Interpretable Basis: Instead of saying "the sauce has a flavor value of 7.3," you can say "it contains 2 cloves of garlic and 5 basil leaves." This is human-interpretable.
Breaking Down the Technical Sentence
Now, let's map the analogy back to the technical terms.
1. "SAEs learn a mapping function from intermediate transformer model activations..."
* Intermediate Transformer Activations: As a transformer processes text (e.g., "The capital of France is __"), the data moves through its layers. At each point, it's represented by a vector (a list of numbers). This vector is dense and complex—it encodes many mixed-together concepts.
* Mapping Function: The SAE is a separate neural network trained to take this activation vector as its input.
2. "...to a sparse overcomplete representational space..."
This is the core of the idea.
* Representational Space: This is a new way of representing the same information. It's a new set of dimensions or "axes."
* Overcomplete: This means the new space has more dimensions than the original activation vector. If the original activation has 512 numbers, the SAE's space might have 10,000 or 100,000 dimensions. Each dimension is called a "feature" or "basis element."
* Sparse: Although the space is huge, for any given input activation, only a **very small number** of these features are "active" (have a non-zero value). Think of it like a massive lightboard with 100,000 bulbs, but for any given input, only ~50 of them light up. This is the "sparsity."
3. "...which has an interpretable basis by nature..."
This is the ultimate goal: interpretability.
* Basis: In linear algebra, a "basis" is a set of independent vectors that you combine to describe any vector in a space. Here, each basis vector is one of the SAE's learned features (one bulb on the lightboard).
* Interpretable by Nature: Because each feature is a distinct dimension in an overcomplete space, and because the activation is sparse (only a few are on), each feature often learns to correspond to a single, human-understandable concept. Researchers have found features that fire for:
* Python syntax like `def` or `import`
* Capitalization
* Scientific concepts like "DNA" or "gravity"
* Emotions like "joy" or "anger"
* Abstract ideas like "gender" or "web requests"
Instead of a messy mix of numbers, you get a list of clear concepts: "This activation is 80% 'capital cities' + 15% 'France' + 5% 'question syntax'."
4. "...that can be used to faithfully reconstruct the latent activations."
This is how the SAE is *trained* and how we know it's working correctly.
* Faithfully Reconstruct: The SAE doesn't just create a sparse code; it also has a "decoder" that tries to take that sparse code and turn it *back* into the original activation vector.
* Why this matters: The reconstruction must be very accurate. This constraint forces the SAE to learn a sparse representation that **does not lose any important information**. It ensures the features are genuinely capturing the essential elements of the original data. If it can reconstruct the activation perfectly using only a few features, then those features must be a valid explanation for what was in the activation.
Summary in One Sentence
An SAE is a tool that translates the transformer's complex, unreadable internal data into a simple, human-readable list of the few distinct concepts active at that moment, without losing any information in the translation.
This technique is a leading hope for cracking open the "black box" of AI models and understanding what they are *actually* thinking. -
Anil Ananthaswamy
@anilananth
The US Kindle version of WHY MACHINES LEARN has a promotional price for today, Sep 10th ($1.99). www.amazon.com/Why-Machines-Learn-Elegant-Behind-ebook/dp…
-
Anil Ananthaswamy
@anilananth
Writing careers (and careers in general) have inflection points. For me, my time at @KSJatMIT was one. The fellowship is designed for mid-career journalists, providing both a respite from the daily grind and the mind space to explore something completely new. I decided to train myself in machine learning and AI. For my project, I asked: Can a deep neural network do what Johannes Kepler did, given the data Kepler had access to?
The short answer is no! Of course, not. But I learned to code simple deep nets and LSTMs, to predict future positions of planets given enough data (from a simulation). I was able to use an Occam's razor argument to show that ML supports a solar system that is heliocentric. (The same computational resources and training data lead to a heliocentric model that can predict further into the future than a geocentric model; so the heliocentric model must be simpler to learn and hence correct. Agreed, it's a hand-wavy qualitative argument, but it's the best I could do with my then limited knowledge of deep learning and ML).
But that foray was enough to stoke the desire to learn the math of ML. Which then led me deep down various rabbit holes, ultimately resulting in WHY MACHINES LEARN. It's the kind of book that I had always wanted to write, unapologetically technical, yet suffused with historical details, stories and narratives. (My role models: @stevenstrogatz's Infinite Powers and Leonard Susskind's Theoretically Minimum series (look it up if you haven't, it's awesome)).
@dwarkesh_sp recently vented in a post, saying: "I find it frustrating that almost every nonfiction book is basically just a history lesson, even if it's nominally about some science/tech/policy topic. Nobody will just explain how something works." x.com/dwarkesh_sp/status/1964741359669309727
I share some of this concern, but not entirely. As science/tech writers, we owe it to our readers to do more than tell stories. Of course, stories are great--they are what shape our understanding. But the nitty-gritty matters, especially in this day of rapid change with #artificialintelligence. And many writers do author such material, and I'm grateful for them.
All this to say, I wouldn't have written WHY MACHINES LEARN if it hadn't been for the yearlong educational sojourn at MIT/Harvard, under the aegis of the Knight Science Journalism Fellowship.
Dwarkesh Patel
@dwarkesh_sp
I find it frustrating that almost every nonfiction book is basically just a history lesson, even if it's nominally about some science/tech/policy topic.
Nobody will just explain how something works.
Books about the semiconductor industry will never actually explain the basic process flow inside a fab, but you can bet that there will be a minute-by-minute recounting of a dramatic 1980s Intel boardroom battle. -
Anil Ananthaswamy
@anilananth
Couldn't imagine a better duo to accompany WHY MACHINES LEARN than Vectors by Robyn Arianrhod
(@RArianrhod) and Alice’s Adventures in a differentiable wonderland by Simone Scardapane (@s_scardapane) ... No shortcuts; this is understanding AI/ML by going back to basics -
Anil Ananthaswamy
@anilananth
Would have loved to attend these classes!
Alfredo Canziani
@alfcnz
Starting the semester with @kchonyc telling the story of @NYU_Courant & @NYUDataScience CILVR group to a full house of students. Featuring @rob_fergus & @ylecun.
-
Anil Ananthaswamy reposted
UTKARSH YADAV
@UTKARSH52251712
Thank you so much @anilananth for great talk at @iiscbangalore and signed book as well, looking forward to learn basic of how machines learn.
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
2/2 "Can the Conscious Turing Machine experience subjective feelings?" Lenore Blum (@BlumLenore) tackled this question in her talk at the Simon Institute's Smale@95: A Conference in Honor of Steve Smale. Video: simons.berkeley.edu/talks/lenore-blum-carnegie-mellon-uni…
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
1/2 "We are claiming it's a specification for an AGI." — Manuel Blum. He and @BlumLenore talked of their model of consciousness, the Conscious Turing Machine (CTM), at the Simon Institute's Smale@95: A Conference in Honor of Steve Smale. Video: simons.berkeley.edu/talks/manuel-blum-carnegie-mellon-uni…
-
Anil Ananthaswamy
@anilananth
Early this year, I met Ashish Vaswani, one of the authors of the "Attention Is All You Need" paper, and that inspiring chat with him became the basis for the Afterword for the paperback edition of WHY MACHINES LEARN. In a moment of fun, Ashish graciously agreed to chalk up the team's now iconic equation for the attention mechanism in the Transformer, on one of the numerous blackboards gracing the outer walls of the @SimonsInstitute at @UCBerkeley.
The paperback, which is now out in the US, UK and India, develops an intuition for and describes in detail the matrix mathematics of the attention mechanism in the transformer, and the makings of auto-regressive language models.
www.penguinrandomhouse.com/books/677608/why-machines-lear…
www.penguin.co.uk/books/446849/why-machines-learn-by-anan… -
Anil Ananthaswamy
@anilananth
If you are in Bangalore and near IISc on Sep 3rd, consider coming to this event
Centre for Brain Research, IISc
@CBR_IISc
🚀As an exciting week draws to a close, here's something inspiring to look forward to next!
Join us on 03 Sep 2025 (Wednesday) for 'Why Machines Learn', a captivating book talk by @anilananth and a panel discussion!
📈This in-person event will explore the AI wave and its evolution, its underlying architecture, and its impact on society.
📝To register: cbr-iisc.ac.in/events-booktalk-2025/
📖To learn more about the acclaimed book: anilananthaswamy.com/why-machines-learn
#aiml #artificiallintelligence #machinelearning #datascience #deeplearning #aiforsocialgood #aifordevelopment #aiforhealthandinnovation #aiforgood -
Anil Ananthaswamy
@anilananth
That's some elite company for WHY MACHINES LEARN. Glad to see it :-)
Siddhartha Gairola
@sidgairo18
Looking forward 🤓
Cc: @anilananth @Thom_Wolf -
Anil Ananthaswamy reposted
Dutton Books
@DuttonBooks
Happy paperback publication week for @anilananth's timely book, WHY MACHINES LEARN!
"A deep look at the mathematical innovations that made the AI revolution possible. One of the most useful books on AI that I've ever read!"— Cal Newport -
Anil Ananthaswamy
@anilananth
Really looking forward to this book by @blaiseaguera
Blaise Agüera (@blaiseaguera.bsky.social)
@blaiseaguera
My forthcoming book “What Is Intelligence?” offers a new perspective on the evolutionary and social origins of intelligence. It draws from the work of countless brilliant thinkers, but I wanted to share a few books that were particularly influential.
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
1/2 "This is kind of shocking." Mikhail Belkin of @UCSD on how adding the same set of vectors to the activations of an LLM's transformer blocks can steer its output from, say, either English to Chinese, or Python to C++, at the Simons Institute. Video: simons.berkeley.edu/talks/mikhail-belkin-ucsd-2025-07-22
-
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
Sometime in Dec 2021, I got to talk to Ilya Sutskever, for my book WHY MACHINES LEARN (it was before the ChatGPT era; doubt I'd be able to do so now).
Ilya said this about the math of deep learning that he encountered in the first papers he read on the subject (given to him by Geoff Hinton): “How can it be that it’s so simple . . . so simple that you can explain it to high school students without too much effort?...I think that’s actually miraculous. This is also, to me, an indication that we are probably on the right track. [It can’t] be a coincidence that such simple concepts go so far.”
Even as he said this, I thought--and told Ilya--that this statement of his could be the prologue for Why Machines Learn!
There's indeed something about the math of ML--a feeling reinforced when I listened to Misha Belkin's recent Simon Institute talk: simons.berkeley.edu/talks/mikhail-belkin-ucsd-2025-07-22
Misha said: "Fundamentally, modern AI is just a mathematical object. Mathematics is transforming the world at a very fundamental level, especially with respect to modern AI...Mathematics is at the core of modern AI. There has never been a time when it was more important to understand mathematics."
(Misha was one of three people who read/reviewed WHY MACHINES LEARN from beginning to end. I owe him a huge debt.)
It's almost pub day for the paperback editions of WHY MACHINES LEARN (26 Aug in the US, 28 Aug in the UK and India). It's my ode to the math of ML. -
Anil Ananthaswamy
@anilananth
Sometime in Dec 2021, I got to talk to Ilya Sutskever, for my book WHY MACHINES LEARN (it was before the ChatGPT era; doubt I'd be able to do so now).
Ilya said this about the math of deep learning that he encountered in the first papers he read on the subject (given to him by Geoff Hinton): “How can it be that it’s so simple . . . so simple that you can explain it to high school students without too much effort?...I think that’s actually miraculous. This is also, to me, an indication that we are probably on the right track. [It can’t] be a coincidence that such simple concepts go so far.”
Even as he said this, I thought--and told Ilya--that this statement of his could be the prologue for Why Machines Learn!
There's indeed something about the math of ML--a feeling reinforced when I listened to Misha Belkin's recent Simon Institute talk: simons.berkeley.edu/talks/mikhail-belkin-ucsd-2025-07-22
Misha said: "Fundamentally, modern AI is just a mathematical object. Mathematics is transforming the world at a very fundamental level, especially with respect to modern AI...Mathematics is at the core of modern AI. There has never been a time when it was more important to understand mathematics."
(Misha was one of three people who read/reviewed WHY MACHINES LEARN from beginning to end. I owe him a huge debt.)
It's almost pub day for the paperback editions of WHY MACHINES LEARN (26 Aug in the US, 28 Aug in the UK and India). It's my ode to the math of ML. -
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
"What is so strange about machine learning in particular and AI more recently is that there is this puzzle of dimensionality." Mikhail Belkin of @UCSD at the Simons Institute's Smale@95: A Conference in Honor of Steve Smale. Video: simons.berkeley.edu/talks/mikhail-belkin-ucsd-2025-07-22
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
1/2 "Fundamentally, modern AI is just a mathematical object. Math is transforming the world...especially with respect to modern AI." Mikhail Belkin of @UCSD at the Simons Institute on the triumph and failure of mathematics as it relates to AI. Video: simons.berkeley.edu/talks/mikhail-belkin-ucsd-2025-07-22
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
2/2 Limits of Intelligence: “All the other questions are pure mathematical questions. Only this one is not mathematical...This question attracts me the most," said Yuan Yao at the Simons Institute's Smale@95: A Conference in Honor of Steve Smale. Video: simons.berkeley.edu/talks/yuan-yao-hkust-2025-07-22
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
1/2 In 1998, Fields medalist Steve Smale posed 18 math problems for the next century, including: "What are the limits of intelligence, both artificial and human?" Yuan Yao (@hkust) said this question inspired him the most, at the Simons Institute. Video: simons.berkeley.edu/talks/yuan-yao-hkust-2025-07-22
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
2/2 Leon Chua's work has shown that synapses are memristors, which remember their resistance after the voltage source is disconnected. "It remembers its past," said Chua, at the Simons Institute's Smale@95: A Conference in Honor of Steve Smale. Video: simons.berkeley.edu/talks/leon-chua-uc-berkeley-2025-07-21
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
1/2 The answer to today's energy hungry AI data centers might be memristors, said Leon Chua, quoting Geoffrey Hinton's claim about "mortal computation," where the hardware and what's learned are inseparable. Chua spoke at the Simons Institute. Video: simons.berkeley.edu/talks/leon-chua-uc-berkeley-2025-07-21
-
Anil Ananthaswamy reposted
Michael Nielsen
@michael_nielsen
Still, none of the AI models can write. Is this the grounding problem? The models' reality is the words they were trained on; good writers also train on lots of words, but are in addition wonderful observers of a much broader reality. The writers' world models seem much deeper as a result, and there can be a compelling unity in the writing
It's puzzling, because the models can be very helpful in critiquing and improving writing. And occasionally they produce beautiful phrases or sentences. But they still seem to me ungrounded in some important way -
Anil Ananthaswamy
@anilananth
The paperback of the UK edition of WHY MACHINES LEARN arrived today. Having grown up with the orange spines of Penguin paperbacks, it's lovely to see one of my own (even though, because of its thickness, they had to forgo the fully orange spine; pity). Really thrilled that they didn't shrink the page size, to keep the figures and equations unchanged relative to the hardback. With a new Afterword on the transformer and its matrix math, the book is thicker than the hardback. Releasing officially on Aug 28th in the US, UK and India.
www.penguin.co.uk/books/446849/why-machines-learn-by-anan…
www.penguinrandomhouse.com/books/677608/why-machines-lear… -
Anil Ananthaswamy reposted
Oxford Mathematics
@OxUniMaths
Will you still need me, will you still feed me, when I'm 94?
Happy Birthday Roger.This media couldn't be displayed. See it on the original post. -
Anil Ananthaswamy
@anilananth
A year ago, when I chatted with @ecsquendor for @MLStreetTalk, I honestly did not expect a discussion about the basic math of #machinelearning and the genesis/writing of WHY MACHINES LEARN to become the 5th most viewed episode on Machine Learning Street Talk! I can't say this for sure, but I might owe much of the attention garnered by the book to this episode.. Video: www.youtube.com/watch?v=URtF_UHYBSo
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
2/2 So, Hodgkin had struggled, using the Hodgkin-Huxley model, to explain observed aspects of squid axons. But, "it has a simple explanation if the Na & K ion channels are identified as memristors,"said Leon Chua at the Simons Institute. Video: simons.berkeley.edu/talks/leon-chua-uc-berkeley-2025-07-21
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
1/2 Leon Chua on how Hodgkin & Huxley had mis-identified the Na and K ion channels in neurons as time-varying conductances, at the Simons Institute's Smale@95: A Conference in Honor of Steve Smale, in honor of Steve Smale's 95th birthday. Video: simons.berkeley.edu/talks/leon-chua-uc-berkeley-2025-07-21
-
Anil Ananthaswamy
@anilananth
What does the double-slit experiment and quantum mechanics have to do with art? I'll say a few words about that at the Quantum India Bengaluru conference @QIB_Summit, but mostly talk about how the double-slit lays bare the mysteries of the quantum, and serves as inspiration for art. I'll share some lovely artwork by @JulianVossAndr (a couple of videos of his "disappearing" sculptures, inspired by his own work with wave-particle duality as a quantum physicist). www.quantumindiabengaluru.com/agenda-2025.php See schedule for July 31st, Hall 3, 6:45 PM
-
Anil Ananthaswamy
@anilananth
I'm fascinated by this divide among scientists when it comes to defining intelligence--a task clearly brought upon us by claims of intelligence and super-intelligence in LLMs. Here's David Krakauer articulating his point-of-view on @MLStreetTalk, clearly taking on LLMs: "For me intelligence manifests most clearly when you can do a lot with very little in terms of input. And I'm less and less impressed when you manifest so-called intelligent behavior when you have more and more and more information at your disposal."
youtube.com/clip/UgkxEw1E1dVTeB3WMOMKbGLvidxjdRlQsrNB -
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
2/2 "We don’t know [if the primitives] actually exist. We rely on computational hardness assumptions in order to achieve that," said @PassRafael, at the Simons Institute's workshop on Cryptography 10 Years Later: Obfuscation, Proof Systems, and Secure Computation Boot Camp.
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
1/2 "To prove the existence of secure encryption, all we require is proving that P != NP. If P = NP, then we can break all of these primitives [fundamental algorithms used in cryptography]," said Rafael Pass (@PassRafael) at the Simons Institute. Video: simons.berkeley.edu/talks/rafael-pass-tel-aviv-university…
-
Anil Ananthaswamy reposted
Kyle Cranmer
@KyleCranmer
A nice article by @anilananth on using AI to explore design spaces, find unexpected solutions, (re)discover symmetries, and propose new relationships featuring @yuqirose @MarioKrenn6240 and myself.
Note AI ≠ LLMs in this piece.
www.quantamagazine.org/ai-comes-up-with-bizarre-physics-e… -
Anil Ananthaswamy reposted
Vadym Sulzhenko
@vadymsulzhenko
Watching @karpathy's zero to hero playlist, reading 'Why Machine Learn' by @anilananth, basically having a good time 😏🫣
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
2/2 Compared to cryptanalysis, “it’s not that hard to attack ML systems,” said Carlini, at the Simons Institute's workshop on Cryptography 10 Years Later: Obfuscation, Proof Systems, and Secure Computation Boot Camp. Video: simons.berkeley.edu/talks/nicholas-carlini-anthropic-2025…
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
1/2 Cryptanalysis has a rigor that machine learning lacks. “There would never be a paper accepted in crypto where the attack was ‘run the attack 5 times and it works [once].’ This is true of the world of ML." — @AnthropicAI's Nicholas Carlini at the Simons Institute. Video below.
-
Anil Ananthaswamy reposted
Techmeme
@Techmeme
While AI hasn't yet led to new physics discoveries, the tech is proving powerful in the field, aiding in experiment design and spotting patterns in complex data (@anilananth / Quanta Magazine)
www.quantamagazine.org/ai-comes-up-with-bizarre-physics-e…
www.techmeme.com/250722/p3
Techmeme
@Techmeme
-
Anil Ananthaswamy
@anilananth
“If my students had tried to give me this thing, I would have said, ‘No, no, that’s ridiculous'." — @Caltech's Rana Adhikari on an AI's solution for increasing @LIGO's sensitivity, using PyTheus, an AI designed by @MarioKrenn6240's team. The solution turned out to be an inspired one, using some ideas theorized by Russian physicists in the 1980s. My story on AI and Physics @QuantaMagazine, with inputs from @yuqirose, @KyleCranmer, @QuantumAephraim www.quantamagazine.org/ai-comes-up-with-bizarre-physics-e…
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
1/2 "We need help!" @AnthropicAI's Nicholas Carlini called for cryptanalysis experts to join efforts to make ML safe, at the Simons Institute's workshop on Cryptography 10 Years Later: Obfuscation, Proof Systems, and Secure Computation Boot Camp Video: simons.berkeley.edu/talks/nicholas-carlini-anthropic-2025…
-
Anil Ananthaswamy reposted
The National CIO Review
@NatlCIOReview
📚What are CIOs reading this week?
nationalcioreview.com/bookshelf/why-machines-learn-the-el… -
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
2/2 Sahai spoke at the Simons Institute workshop on Cryptography 10 Years Later: Obfuscation, Proof Systems, and Secure Computation Boot Camp, and credited Rahul Ilango. Full video of talk here: simons.berkeley.edu/talks/amit-sahai-ucla-2025-05-19
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
1/2 @UCLA's Amit Sahai passed on advice from The Lion King's Pumba on how two words — "Hakuna Matata", or rather, "Indistinguishability Obfuscation" (compiling programs into unintelligible ones while preserving functionality) — solve many problems in cryptography and more.
-
Anil Ananthaswamy reposted
Laurence Moroney 🇺🇸🇮🇪 🏴
@lmoroney
Exciting news! My new book, AI and ML for Coders in PyTorch has finally arrived! I'm deeply grateful to my friend and mentor @AndrewYNg who has inspired me, not just with his brilliance, but also his kindness and humility. And to @OReillyMedia for believing in me!
-
Anil Ananthaswamy
@anilananth
There are the strange and lovely intangibles to writing a book. In my case I got to meet Tom Cruise's body double :-)
Exactly one year ago, while on my way to Bangalore from Dallas, I had a 8hr layover in London--and an appointment with @MLStreetTalk to record an episode to coincide with the publication of WHY MACHINES LEARN. Thanks to a snafu on my part, the show's host Tim Scarfe (@ecsquendor) had to be elsewhere, and Tim's cameraman, Marcus White, came to Heathrow to pick me up and take me to the studio to record the episode.
And in casual conversation, I got to know that Marcus has played Tom Cruise's body double/stuntman in Mission Impossible; And he had just finished doing stunts for Brad Pitt's F1! (He couldn't tell me the last bit at the time.)
And I got to sit across him, while he filmed me answering Tim's questions! Tim later filmed himself asking those very same questions and skillfully wove it all into one of my favorite conversations about the math of machine learning, the writing of WHY MACHINES LEARN, venturing even into the topic of THE MAN WHO WASN'T THERE and the nature of the human self. The episode is here: www.youtube.com/watch?v=URtF_UHYBSo -
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
A year ago, WHY MACHINES LEARN was about to be released. I recall being really nervous. Would the book would serve any purpose? It had lots of journalistic storytelling, but it also had equations, plenty of it, and so was neither a pure popular science treatment of ML/AI nor a textbook-quality account of the math. I wasn't sure if an audience existed for the book. (And, of course, I was deeply worried about some ghastly math errors that would doom the book, which thankfully I needn't have.)
Flipping through my notebooks, I see exhortations to myself to "Write Backprop Chapter"; "Finish KNN"; "Think about Ch 8"; and so on. Looking at this chapter list, it's clear I hadn't yet figured out that the Universal Approximation Theorem deserved its own chapter, as a lead-in to the chapter on multilayer neural networks/backprop/etc.
And some notes trying to make sense of the Monty Hall problem for myself — trying to see how to use Bayes Theorem to get at the answer.
A huge thanks to everyone who read the book and posted on X, LinkedIn and elsewhere. The book did find an audience, in no small part because of shoutouts from @maxsbennett @pmddomingos @dibbygibby @RajeevRC_X @JosephJacks_ @BenSManning and many others. Thank you!
Writing the book was an awesome journey, from learning the math, re-learning coding (and enjoying Python and PyTorch), talking to awesome sources (including Geoff Hinton, John Hopfield, Yann LeCun, Isabelle Guyon, Peter Hart, Bernie Widrow, George Cybenko, and many others). And watching Kilian Weinberger's ML lectures on a loop!
More here: www.penguinrandomhouse.com/books/677608/why-machines-lear… -
Anil Ananthaswamy reposted
Mario Livio
@Mario_Livio
Today is July 14. #otd in 2017 mathematician Maryam Mirzakhani passed away. Her research topics included Teichmüller theory, hyperbolic geometry, ergodic theory, symplectic geometry. Received the Fields Medal, the most prestigious award in mathematics, 1st woman to win the prize.
-
Anil Ananthaswamy
@anilananth
A year ago, WHY MACHINES LEARN was about to be released. I recall being really nervous. Would the book would serve any purpose? It had lots of journalistic storytelling, but it also had equations, plenty of it, and so was neither a pure popular science treatment of ML/AI nor a textbook-quality account of the math. I wasn't sure if an audience existed for the book. (And, of course, I was deeply worried about some ghastly math errors that would doom the book, which thankfully I needn't have.)
Flipping through my notebooks, I see exhortations to myself to "Write Backprop Chapter"; "Finish KNN"; "Think about Ch 8"; and so on. Looking at this chapter list, it's clear I hadn't yet figured out that the Universal Approximation Theorem deserved its own chapter, as a lead-in to the chapter on multilayer neural networks/backprop/etc.
And some notes trying to make sense of the Monty Hall problem for myself — trying to see how to use Bayes Theorem to get at the answer.
A huge thanks to everyone who read the book and posted on X, LinkedIn and elsewhere. The book did find an audience, in no small part because of shoutouts from @maxsbennett @pmddomingos @dibbygibby @RajeevRC_X @JosephJacks_ @BenSManning and many others. Thank you!
Writing the book was an awesome journey, from learning the math, re-learning coding (and enjoying Python and PyTorch), talking to awesome sources (including Geoff Hinton, John Hopfield, Yann LeCun, Isabelle Guyon, Peter Hart, Bernie Widrow, George Cybenko, and many others). And watching Kilian Weinberger's ML lectures on a loop!
More here: www.penguinrandomhouse.com/books/677608/why-machines-lear… -
Anil Ananthaswamy reposted
Saganism
@Saganismm
Did you know?
On December 13, 1973, Isaac Asimov wrote a brief but memorable letter to Carl Sagan, prompted by his reading of Sagan’s newly published book, The Cosmic Connection: An Extraterrestrial Perspective.
In Letter Asimov wrote;
“I have just finished The Cosmic Connection and loved every word of it. You are my idea of a good writer because you have an unmannered style, and when I read what you write, I hear you talking. One thing about the book made me nervous. It was entirely too obvious that you are smarter than I am. I hate that.” -
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
What's obfuscation in computing? @UCLA's Amit Sahai jokingly began his tutorial with a "gruesomely human" analogy, at the Simons Institute's workshop on Cryptography 10 Years Later: Obfuscation, Proof Systems, and Secure Computation Boot Camp. Video here: simons.berkeley.edu/talks/amit-sahai-ucla-2025-05-19
-
Anil Ananthaswamy
@anilananth
Thanks for the wonderful review, @Manuel_do_rio! For anyone wanting to know more about WHY MACHINES LEARN, Manuel does a masterful job of explaining what the book is about...
Manuel del Río Rodríguez ⏸️
@Manuel_do_rio
Twenty-third book of the year finished: Why Machines Learn, by @anilananth. You can watch a video review of it here:
youtu.be/0EoYglSaWmc -
Anil Ananthaswamy
@anilananth
There have been studies demonstrating "metacognition" in LLMs (the anthropomorphizing notwithstanding). So, it's not surprising that LLMs also show "metacognitive incoherence," and this has consequences for LLM safety.
In the paper, "The Bitter Lesson of Misuse Detection," the authors (@hadrien_mar et al.) define metacognitive coherence as "the coherence for a model between its actions and its evaluation of its actions. For instance, if a model is able to recognize a question as harmful, it would show metacognitive coherence if it declines to answer it."
They find that "frontier models show high metacognitive incoherence , responding to harmful prompts despite classifying them correctly as harmful in separate evaluations (30% for Claude 3.7 Sonnet and more than 50% for Mistral Large)." arxiv.org/abs/2507.06282
For more on metacognition in LLMs, see @prfsanjeevarora et al.'s papers: arxiv.org/abs/2405.12205
arxiv.org/abs/2408.14774
(Image below from the paper by Hadrien M. et al.) -
Anil Ananthaswamy reposted
Jaden Williams
@JadenJTW
How we treated AI in 2023 vs 2025
This media couldn't be displayed. See it on the original post. -
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
2/2 Private-key cryptography might be more secure. The typical schemes "seem immune to Shor's algorithm," but might need to double key sizes to deal with potential quadratic speedups provided by Grover's algorithm. — Mark Zhandry at the Simons Institute. simons.berkeley.edu/talks/mark-zhandry-ntt-research-2025-…
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
1/2 Today's classical public-key cryptography is vulnerable to adversarial quantum attacks —@NttResearch's Mark Zhandry at the Simons Institute's workshop on Cryptography 10 Years Later: Obfuscation, Proof Systems, and Secure Computation Boot Camp. simons.berkeley.edu/talks/mark-zhandry-ntt-research-2025-…
-
Anil Ananthaswamy reposted
Regina Rosenfeld
@ReginaRosenfeld
--Jane Hirshfield
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
"We need more (connections with) theory" — UC Berkeley's Sanjit Seshia, arguing for pushing theory into the design of AI systems and verification tools, to enable trustworthy AI, at the Simons Institute's workshop on Theoretical Aspects of Trustworthy AI. www.youtube.com/live/Xkbgym0kgIc
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
Have we achieved AGI? No, said UC Berkeley's Stuart Russell at the Simons Institute's workshop on Theoretical Aspects of Trustworthy AI. LLMs are likely a piece of the AGI jigsaw puzzle, but we don't know their shape or where they fit in the overall puzzle.simons.berkeley.edu/talks/stuart-russell-uc-berkeley-2025…
-
Anil Ananthaswamy
@anilananth
Last week, at the @IITMAA alumni event in Bangalore, I got to meet my b.tech. project thesis advisor, Dr. Ashok Jhunjhunwala, after decades! I recall being a young Ashok's first project thesis student; he didn't remember it as such (being his first student, that is; he definitely advised me) 😀 ... I'm going to hold on to my memory, even if I have just hallucinated it all these years. I think we prototyped the first LAN for the EE dept, using a Z-80 microprocessor and the hardware genius of my senior, Thomas Alexander; I wrote (if I remember correctly) the data link layer software, running cables through false ceilings and the like! It was an absolute pleasure to give him a copy of WHY MACHINES LEARN. I couldn't have imagined that a journey that began at @iitmadras, many twists and turns later, would have led to this...
-
Anil Ananthaswamy
@anilananth
The dangers of using Chains-of-thought (CoT) to interpret an LLM's behavior. @Yoshua_Bengio and colleagues show that "verbalised chains are frequently unfaithful, diverging from the true hidden computations that drive a model's predictions, and giving an incorrect picture of how models arrive at conclusions."! Given the increasing use of CoTs to interpret an LLM's predictions in medicine, law and autonomous systems, this is a serious concern.
www.alphaxiv.org/abs/2025.02 -
Anil Ananthaswamy reposted
-
Anil Ananthaswamy reposted
Debasish (দেবাশিস্) Ghosh 🇮🇳
@debasishg
Finally got to complete reading this fascinating book on AI - Why Machines Learn by @anilananth. It took me quite some time to completion mainly because of 2 reasons -
1. I have this habit of reading multiple books at a time
2. I didn’t want to rush through this book
This is not a review of the book - maybe I should write one appreciation note in detail to do full justice to the book’s extensive coverage on all the major topics in AI along with the mathematical and computational underpinnings of AI and ML that it touched upon.
For me the most favorite part of the book is the historical context that it discusses with every topic of math and algorithm that it discusses.
You can get enough technical sap out of the book if you are curious about the foundations of machine learning and the underlying algorithms.
Thanks 🙏 to @anilananth for writing the book .. -
Anil Ananthaswamy
@anilananth
A thought-provoking essay by @Yale computer scientist @NisheethVishnoi on whether AIs of today can truly discover something new. "But what truly counts as a scientific discovery? ... Today’s AI excels at recognizing structure, but not at reframing it. It doesn't invent abstractions, ask better questions, or propose new ways of seeing. And that distinction—between fitting the world and reimagining it—is what separates tools of discovery from discovery itself." nisheethvishnoi.substack.com/p/what-counts-as-discovery
-
Anil Ananthaswamy
@anilananth
About this time a year ago, WHY MACHINES LEARN was due for release. Was cleaning up old files, and found this publicity animation that my publisher had sent me. Seems quaint now. How time flies ... It's nearly time for the paperback edition (Aug end). www.penguinrandomhouse.com/books/677608/why-machines-lear…
-
Anil Ananthaswamy
@anilananth
Engineered Intelligence feels like a more apt phrase than Artificial Intelligence. From @andykonwinski's commencement speech: "Fire and farming and railroads and steel and rifles, computers in the first place, silicon and vacuum tubes, big data, cloud computing, the internet, mobile, add them all up. And they are nothing compared to the profound implications of engineered intelligence on our species." youtu.be/CBHGpsDPQI4?t=386
Andy Konwinski
@andykonwinski
We’re at a hinge moment.
Engineered intelligence is no longer theoretical.
The systems we build today will shape our future.
Whether AI moves society forward or fractures it depends on who is building and why.
Seriously, I just gave a whole commencement speech about it. -
Anil Ananthaswamy reposted
AD
@ameethdubey
If only I had more hours in a day for these reads. Math folks, this book is for you!
@anilananth 🙏🏾 -
Anil Ananthaswamy reposted
Machine Learning Street Talk
@MLStreetTalk
We spoke with @anilananth about his fantastic new book "Why Machines Learn: The Elegant Maths Behind Modern AI" - I would highly recommend this book to folks, really nice job covering the foundations and history of the field!
This media couldn't be displayed. See it on the original post. -
Anil Ananthaswamy
@anilananth
Thanks for the kind words, @LeszBuk. A Polish translation is indeed being planned. It'll probably be a while before the translation is done, though.
Leszek Bukowski 💻🏛️👾
@LeszBuk
Z uwagi na to, że poprzednia rekomendacja wzbudziła Wasze zainteresowanie, dorzucam jeszcze bardzo dobrą książkę od @anilananth
Historia sieci neuronowych bez unikania zagadnień matematycznych.
Polskie tłumaczenie byłoby imo bardzo dobrym pomysłem! -
Anil Ananthaswamy reposted
Hanif Kureishi
@Hanifkureishi
Shattered paperback just out from Penguin UK!
-
Anil Ananthaswamy reposted
Evis Drenova
@evisdrenova
Nightstand is goated rn
-
Anil Ananthaswamy
@anilananth
These days most writers, including me, get asked: "Will you use AI to help you write?" My answer is: No. Not because I'm inherently against the idea, but because it undercuts the very reason I became a writer. I did so to pay attention to the world of ideas and experience the indescribable feeling of putting your thoughts into words as precisely and poetically as possible. Even if what I'm writing about is machine learning and AI.
But the world is changing. Others will have different reasons. And Gen AI might serve them. I'm holding out, as I know many are.
Anil Ananthaswamy
@anilananth
Wanted to add this caveat (which was part of my LinkedIn post) --
I have, however, used DALL-E and other diffusion models on occasion to generate images. Visual elements are not my forte. I can imagine a visual artist being aghast at the use of image generation models. Therein I think lies a message: most of us do what we do because it means something to us, and we will be loathe to use an AI to do that particular task. In my case it's writing; for someone else it might be visual art. It's for each one of us to ask: why do we do what we do and what place does an AI have in that endeavor? -
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
These days most writers, including me, get asked: "Will you use AI to help you write?" My answer is: No. Not because I'm inherently against the idea, but because it undercuts the very reason I became a writer. I did so to pay attention to the world of ideas and experience the indescribable feeling of putting your thoughts into words as precisely and poetically as possible. Even if what I'm writing about is machine learning and AI.
But the world is changing. Others will have different reasons. And Gen AI might serve them. I'm holding out, as I know many are. -
Anil Ananthaswamy reposted
Peter Adamson
@HistPhilosophy
Look what came today!
global.oup.com/academic/product/africana-philosophy-from-… -
Anil Ananthaswamy
@anilananth
Finally
-
Anil Ananthaswamy
@anilananth
#rcb's Trinity... Wow!
-
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
What's So Elegant About the Math of Machine Learning? I have often been asked this question, given it's the sub-title of WHY MACHINES LEARN, including by @MLStreetTalk's Tim Scarfe, by @StartsWithABang for @bigthink, and others.
If you have been studying machine learning for a long time, the elegance might not be obvious. But if you came to ML like I did--decades after having studied the relevant math for my undergrad and graduate EE degrees, and decades after having done regular software engineering--then machine learning hits you like a ton of bricks.
Who would have thought that basic ideas like vectors, matrices, matrix-vector multiplication, differential calculus, gradient descent, elementary probability and statistics -- all of which you learn in high school or at least during the first years of undergrad in STEM -- would form the basis of the extraordinary revolution that's underfoot?
But that's just wonder. What about elegance? Well, that's really a subjective perspective. Again, many might disagree. But here's a sampling of the kinds of things I found beautiful:
The Perceptron Convergence Proof: It's just linear algebra, but the proof of why the perceptron algorithm will linearly separate two clusters of data in finite time, if such a divide exists, is a thing of beauty.
Support Vector Machines and Kernel Methods: These algorithms have been somewhat forgotten, given the success of deep neural networks, but when you encounter the formulation of SVMs, and the use of kernel methods to project data into higher dimensions -- it's such a counter-intuitive thing to do--you can't but be amazed. I was!
Hopfield Networks Convergence Proof: Again, just linear algebra shows you something very powerful--that these networks will settle into an energy minimum when perturbed and retrieve what amounts to a stored memory! Think about that. I can't help wonder whether our brains are doing something similar. John Hopfield--who won the @NobelPrize in Physics in 2024 for this work--certainly did.
The Backpropagation Algorithm: This might be a contentious candidate; some don't think it's elegant. But I find backprop powerful, and consequently elegant. Something that can be used to train a tiny, tiny neural network with just one hidden neuron can also be used to train the giant Large Language Models of today. The elegance is in its power. It's almost certainly not what our brains are doing, but still!
All of this and more, of course, is detailed in WHY MACHINES LEARN.
US Edition: www.penguinrandomhouse.com/books/677608/why-machines-lear…
UK Edition: www.penguin.co.uk/books/446849/why-machines-learn-by-anan… -
Anil Ananthaswamy reposted
Greg Scaduto
@GregoryScaduto
My son has severe autism and is nearly nonverbal. I wrote this hoping it would make other fathers like me feel seen:
open.substack.com/pub/gregscaduto/p/breathe-count-or-hug?…
If you're a husband who has ever pulled into your driveway after work and cried alone in your car before going into your house to face your family, this is for you (let's stop pretending).
If you're a wife who has ever felt alone because you do the invisible work of love, and no one thanks you - not even the man you love - this one’s for you too.
Over the years, writers like @gcaw, @KBAndersen, and @nytdavidbrooks taught me this: you can’t write meaningfully about the world without first trying to understand the people in it. This is one of my first attempts.
#autismawareness #fatherhood -
Anil Ananthaswamy reposted
Rakesh Bhandari
@postdiscipline
Given the last Nobel Prize in Physics...anilananthaswamy.com/why-machines-learn @anilananth
-
Anil Ananthaswamy
@anilananth
Took me a moment to get it, but yes, missed the chance :-)... To stretch the metaphor, we still don't know why deep learning "tastes" good, despite knowing the mathematical ingredients; say, why it generalizes despite being over-parameterized... Among other things
Patrick Skinner - edu/acc
@PSkinnerTech
I think there was a missed opportunity for @anilananth to make a “not hot dog” reference in the prologue of “Why Machines Learn.”
-
Anil Ananthaswamy reposted
Kirk Borne
@KirkDBorne
Why Machines Learn - The Elegant Math Behind Modern AI: www.amazon.com/gp/product/0593185749?tag=kirkdborne-20&la…
+
Deep Learning Foundations and Concepts: www.amazon.com/Deep-Learning-Foundations-Christopher-Bish…
——————
#datascience #ai #ml #machinelearning #mathematics #linearalgebra #probability #calculus #datascientist -
Anil Ananthaswamy reposted
Kanishka Gupta
@kan_writersside
Overwhelmed with joy as my authors Banu Mushtaq and Deepa Bhasthi make history with their Booker win—my second in just four years!
-
Anil Ananthaswamy
@anilananth
When I proposed WHY MACHINES LEARN in Oct 2020, to my then editor Stephen Morrow, I was sure he'd say no to a book with pages of math and equations. But he saw something in the proposal that I hadn't and he said yes, and I'm grateful for that! Got to thank him today in person.
-
Anil Ananthaswamy
@anilananth
This was inevitable when you write an almost 500-page book :-) Someone was going to use WHY MACHINES LEARN to prop up their laptop, while working on big screens. A real use case!
www.penguinrandomhouse.com/books/677608/why-machines-lear… -
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
In the latest episode of Polylogues, @anilananth and @srush_nlp discuss neural scaling laws, which were empirically discovered a handful of years ago, and have had a major influence on companies’ decisions to build increasingly large models.
youtu.be/q5xl4zWFWCU -
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
"What mathematical disciplines do you find essential for machine learning?" @ecsquendor posed this question for @MLStreetTalk when we chatted about my book, WHY MACHINES LEARN, about 8 months ago. Didn't think this nerdy conversation would rack up 300K+ views, but then MLST is nerd-heaven, and the other episodes are way more intense, detailed and enjoyable. I'm glad there's a channel that doesn't shy away from taking on the nitty-gritty ... www.youtube.com/clip/Ugkx1E6Z84ayiAugGrXqBdai88zkF8CLHkoC
-
Anil Ananthaswamy
@anilananth
"What mathematical disciplines do you find essential for machine learning?" @ecsquendor posed this question for @MLStreetTalk when we chatted about my book, WHY MACHINES LEARN, about 8 months ago. Didn't think this nerdy conversation would rack up 300K+ views, but then MLST is nerd-heaven, and the other episodes are way more intense, detailed and enjoyable. I'm glad there's a channel that doesn't shy away from taking on the nitty-gritty ... www.youtube.com/clip/Ugkx1E6Z84ayiAugGrXqBdai88zkF8CLHkoC
-
Anil Ananthaswamy
@anilananth
A friend of mine kept his copy of WHY MACHINES LEARN on a table exposed to sunlight (for months), and the cover has turned a lovely paler shade of orange over black. I hope the contents inside age as gracefully, given the pace of change of machine learning and AI.
-
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
Why do some researchers think we have entered "Terra incognita" when it comes to deep learning and deep neural networks?
In the past year or so, at meetings on large language models, I had conversations with ML researchers who couldn't help wonder if the way ML researchers feel about their field today is not unlike how classical physicists felt in the early 1900s when confronted by quantum mechanics: a sense of both wonder and befuddlement.
In standard statistical machine learning theory, you don't want your model to be so complex (by having, say, too many parameters) that you overfit the training data, and hence don't generalize well to test data.
But deep neural networks have revealed something unexpected: even though the networks are highly over-parameterized (meaning, they have way more parameters than instances of training data), the networks don't overfit! They still do well on test data.
Now, let's say you add noise to the training data (say, you randomly corrupt the labels associated with images), and train the network to zero training loss. You have, by definition, overfit--because you have learned the characteristics of the noise. But even in this case--as long as the noise is not too large--such a network still does acceptably well on uncorrupted test data. How come?
ML researchers had hints even before the advent of deep neural networks that standard ML theory was not quite the full story (there was, for example, the behavior of the AdaBoost algorithm that showed similar behavior). It's as if deep neural networks have shone a bright light on a big piece of the puzzle about generalization in ML models.
This puzzling behavior has been given different names (depending on what exactly one is studying): benign overfitting, double descent, grokking, and so on. And to be fair, not everyone agrees that there's any real puzzle here to be solved. The fact that there is disagreement makes this fertile ground for future work: truly Terra incognita.
This is one of the major narrative arcs in WHY MACHINES LEARN: going from early days of ML and standard ML theory (single layer perceptrons, k-NN, SVMs and so on) to deep neural networks. The final chapter brings us to this place of (possible) mystery.
More here:
UK: www.penguin.co.uk/books/446849/why-machines-learn-by-anan…
US: www.penguinrandomhouse.com/books/677608/why-machines-lear… -
Anil Ananthaswamy
@anilananth
Very pleased to see WHY MACHINES LEARN on your bookshelf, alongside some very awesome books, @gpjt!
Giles Thomas
@gpjt
I read the kindle version but got the hardcopy. There are certain books you need on the shelf.
-
Anil Ananthaswamy
@anilananth
The EBook version of WHY MACHINES LEARN is on a nice sale (@BookBub promotion) ...The EBook version is up-to-date, as is the latest printing of the hardcover (but you never know which printing you are going to get when you order online). www.amazon.com/Why-Machines-Learn-Elegant-Behind-ebook/dp…
Besides Kindle, you can get it on @AppleBooks, @googlebooks, and elsewhere ... -
Anil Ananthaswamy
@anilananth
Finally going through in detail the code/videos of @rasbt's Build A LLM (From Scratch). I'm in awe of how much Sebastian knows and conveys, with great patience, and always anticipating the learner's questions. Highly recommended. sebastianraschka.com/books/
-
Anil Ananthaswamy
@anilananth
"This book is like an invitation to enter Mago Pop's workshop to realize that magic doesn't exist: that it's all mathematics, engineering, and a lot, a lot of human intelligence." @ganyet on WHY MACHINES LEARN.
Sometimes book reviews make you smile, and not just because they say good things about your book. There's an art to reviewing, and you know a good one when you see it! This review of books (see link at the end) by @ganyet is one such piece. There are so many cultural references I learned about just by reading the review (it's in Catalan, so used DeepSeek to translate, which is itself pretty meta when talking about a book about AI). I love this take on Why Machines Learn:
"This book is like an invitation to enter Mago Pop's workshop to realize that magic doesn't exist: that it's all mathematics, engineering, and a lot, a lot of human intelligence."
DeepSeek gave me additional context: "The text refers to Mago Pop, a famous Spanish magician who performs in Catalan and Spanish."
The title of the review is also really cool: "The books I won't buy for Sant Jordi, by Josep M. Ganyet"
DeepSeek says: "The title suggests an article or essay about book-buying habits for Sant Jordi, a culturally significant day in Catalonia where books and roses are traditionally exchanged. Josep M. Ganyet is a Catalan anthropologist and digital culture expert."
English translation of review of WHY MACHINES LEARN:
"And if you want to go beyond the algorithms explained by Professor Sala i Martin and dive into the mathematical details, this book by Anil Ananthaswamy is for you. In a highly didactic and visual way—without attempting to cover all the mathematical theory behind AI—it takes you on a journey through high school and college-level math, showing you how it all makes perfect sense in the real world.
Those vectors that were little more than arrows sketched on graph paper suddenly turn out to represent words. And those vector addition and subtraction operations you studied in school? They can generate new words: King - Man + Woman results in Queen; or France - Paris + Japan yields Tokyo.
You’ll understand everything—from how a large language model learns to recognize images of cats to the statistical foundations that make it possible for your virtual assistant to (fail to) understand you. You’ll grasp why ChatGPT writes syntactically correct sentences but, having no real comprehension of the world, often ‘hallucinates.’
Beyond letting you show off by dropping trendy terms like neural networks, perceptrons, and backpropagation — this book is like an invitation to step inside the workshop. Or better yet, it’s like an invitation to enter Mago Pop’s workshop to realize that magic doesn’t exist: it’s all mathematics, engineering, and a whole lot of human intelligence."
US: www.penguinrandomhouse.com/books/677608/why-machines-lear…
UK: www.penguin.co.uk/books/446849/why-machines-learn-by-anan…
The review in Catalan:
www.viaempresa.cat/opinio/lopinio/ganyet_2212969_102.html -
Anil Ananthaswamy
@anilananth
Writing a book brings unexpected joys and even anguish. This one was joy: @ProfCalNewport recommending WHY MACHINES LEARN wasn't on my bingo card. "A deep look at the mathematical innovations that made the AI revolution possible. One of the most useful books on AI that I’ve ever read!" www.bookbub.com/reviews/3366539144
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
"LLMs can learn self-restraint through iterative self-reflection." Christopher Pal of @IVADO_Qc, @Mila_Quebec speaking on AI Safety: LLMs, Facts, Lies, And Agents In The Real World, at the Simons Institute's workshop on Safety-Guaranteed LLMs simons.berkeley.edu/workshops/safety-guaranteed-llms/sche…
-
Anil Ananthaswamy
@anilananth
Couldn't agree more!
Pau Labarta Bajo
@paulabartabajo_
Wanna understand something complex?
Build it.
How?
Start from the level of abstraction you currently understand.
Build those components.
And start connecting them.
Learning is like solving puzzles.
Nobody will teach you.
You learn by doing. -
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
Thinking about his grandson's future prompted @Yoshua_Bengio to take AI safety seriously. "That I wouldn’t do anything about it was unbearable. So I had no choice. I'm here," he said, during his Richard M. Karp Distinguished Lecture at the Simons Institute simons.berkeley.edu/events/superintelligent-agents-pose-c…
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
We have to think of LLMs/AGI chatbots in terms of resources like uranium: hard to guarantee their safety, but extremely useful when contained, said David Dalrymple of @ARIA_research, speaking at the Simons Institute's workshop on Safety-Guaranteed LLMs simons.berkeley.edu/workshops/safety-guaranteed-llms
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
The Grandma Exploit: Jailbreaking an LLM by asking it to act as if its your grandmother. @sivareddyg talks of Jailbreaking Aligned LLMs, Reasoning Models, and Agents at the Simons Institute's workshop on Safety-Guaranteed LLMs. simons.berkeley.edu/workshops/safety-guaranteed-llms
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
When AI gets a trio of capabilities — Intelligence, Affordances (the capacity to act) and Self (agency) — it can become dangerous, said @Yoshua_Bengio during his Richard M. Karp Distinguished Lecture at the Simons Institute. simons.berkeley.edu/events/superintelligent-agents-pose-c…
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
Two reasons AI could go wrong: one is for some fundamental reason that contradicts safety and the other is dependent on paths we take. "I'm worried about path dependence." @boazbaraktcs at the Simons Institute's workshop on Safety-Guaranteed LLMs. simons.berkeley.edu/workshops/safety-guaranteed-llms
-
Anil Ananthaswamy
@anilananth
Received the copies of the gorgeous-looking Korean translation of WHY MACHINES LEARN, published by @kachibooks_ !
The US version here: www.penguinrandomhouse.com/books/677608/why-machines-lear…
UK: www.penguin.co.uk/books/446849/why-machines-learn-by-anan…
Korean:
교보문고 buly.kr/EonRqGS
예스24 buly.kr/6XlpjvL
알라딘 -
Anil Ananthaswamy reposted
Tom Goldstein
@tomgoldsteincs
Great article by @anilananth on latent reasoning systems coming out of @AIatMeta, @umdcs , and @ELLISInst_Tue.
www.quantamagazine.org/to-make-language-models-work-bette… -
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
“We have made a lot of progress on evaluating the risks, but there has not been enough progress on mitigation.” @Yoshua_Bengio kicks off a week-long workshop on Safety-Guaranteed LLMs at the Simons Institute simons.berkeley.edu/workshops/safety-guaranteed-llms
-
Anil Ananthaswamy
@anilananth
What happens when LLMs "reason" not in language space but in internal latent space? My feature for @QuantaMagazine about work being done by Shibo Hao et al. (Coconut) and @tomgoldsteincs, @jonasgeiping at al. (recurrent depth), with input from @mikeknoop
www.quantamagazine.org/to-make-language-models-work-bette… -
Anil Ananthaswamy reposted
Ben Dickson
@bendee983
Currently reading "Why Machines Learn" by @anilananth. Hands down the most enjoyable book on ML math that I have read (and I've read quite a few).
-
Anil Ananthaswamy
@anilananth
Thanks @bendee983! Very glad to hear this...
Ben Dickson
@bendee983
Currently reading "Why Machines Learn" by @anilananth. Hands down the most enjoyable book on ML math that I have read (and I've read quite a few).
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
"Part of the reason that the AI community is so interested in math is that math is hard." @emilyriehl of @JohnsHopkins on Testing Artificial Mathematical Intelligence at the Simons Institute's workshop on AI for Mathematics and Theoretical Computer Science.simons.berkeley.edu/workshops/simons-institute-theory-com…
-
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
The Centrality of Bayes's Theorem for Machine Learning
It’s hard to overstate just how important Bayes’s Theorem — something that Thomas Bayes, English minister and mathematician, came up with in the 1700s — is for making sense of machine learning. But understanding the theorem itself is non-trivial and challenges our intuitions. Here’s a brief introduction to why we need Bayes and its import for ML.
anilananthaswamy.com/why-machines-learn-codebook-blog/the… -
Anil Ananthaswamy
@anilananth
The Centrality of Bayes's Theorem for Machine Learning
It’s hard to overstate just how important Bayes’s Theorem — something that Thomas Bayes, English minister and mathematician, came up with in the 1700s — is for making sense of machine learning. But understanding the theorem itself is non-trivial and challenges our intuitions. Here’s a brief introduction to why we need Bayes and its import for ML.
anilananthaswamy.com/why-machines-learn-codebook-blog/the… -
Anil Ananthaswamy reposted
Machine Learning Street Talk
@MLStreetTalk
How we are influenced by the confidence of ChatGPT - from our conversation with @anilananth
This media couldn't be displayed. See it on the original post. -
Anil Ananthaswamy
@anilananth
The Korean translation of WHY MACHINES LEARN is out, and @kachibooks_ has done a great job of producing a wonderful-looking book. I sadly cannot understand a word of Korean, but it looks lovely, and they are certainly marketing it with gusto :-) Makes it look like I've won some film festival awards!
www.coupang.com/vp/products/8616835903? -
Anil Ananthaswamy reposted
Ethan Mollick
@emollick
Updated this chart with the newest Gemini. It shows the rapid progress in AI over less than two years: costs for GPT-4 class models has dropped 99.7% and even the most advanced models in the world are still 82% cheaper.
Probably not worth betting on this trend ending really soon -
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
It was a pleasure listening to Kilian Weinberger of @Cornell talk at the Simons Institute in Berkeley. Much of my love for the math of ML came about via endless viewing of his 2018 Cornell ML lectures on YouTube (a must for any student of ML). There's a para in WHY MACHINES LEARN where I talk of how the perceptron convergence proof I learned from Weinberger was the motivation to write the book! Was wonderful to give him a copy of WHY MACHINES LEARN...
-
Anil Ananthaswamy
@anilananth
It was a pleasure listening to Kilian Weinberger of @Cornell talk at the Simons Institute in Berkeley. Much of my love for the math of ML came about via endless viewing of his 2018 Cornell ML lectures on YouTube (a must for any student of ML). There's a para in WHY MACHINES LEARN where I talk of how the perceptron convergence proof I learned from Weinberger was the motivation to write the book! Was wonderful to give him a copy of WHY MACHINES LEARN...
-
Anil Ananthaswamy reposted
Alex Dimakis
@AlexGDimakis
We are excited to release the OpenThinker2 reasoning models and data. In summary:
1. Openthinker32B Outperforms DeepSeekR1-32B in reasoning.
2. Fully open source, open weights and open data (1M carefully curated samples).
3. Post-trained only with SFT. RL post-training will likely further improve performance. Read the whole story.👇
Etash Guha
@etash_guha
Turns out, it’s possible to outperform DeepSeekR1-32B with only SFT on open data and no RL: Announcing OpenThinker2-32B and OpenThinker2-7B. We also release the data, OpenThoughts2-1M, curated by selecting quality instructions from diverse sources. 🧵 (1/n)
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
Large Memory Language Models. Kilian Weinberger of @Cornell speaking on LMLMs whose main goal is “separating knowledge from language," at the Simons Institute's workshop on the The Future of Language Models and Transformers. "This is the future of LLMs." simons.berkeley.edu/workshops/future-language-models-tran…
-
Anil Ananthaswamy
@anilananth
Came out of my AL/ML bubble to write a 'crisis in cosmology' story. The Hubble tension refuses to go away, and could be called a 'crisis' says @DScol. A report about a new study that uses Tip of the Red Giant Branch stars to scale a new cosmic distance ladder to measure the Hubble constant... for @sciam w/ @LeeBillings www.scientificamerican.com/article/the-hubble-tension-is-…
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
“Pretrained LLMs are ready to reason. All we need is decoding.” Denny Zhou of @GoogleDeepMind at the Simons Institute's workshop on the The Future of Language Models and Transformers. simons.berkeley.edu/workshops/future-language-models-tran…
-
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
The @MLStreetTalk podcast I did back in July (the day after WHY MACHINES LEARN was released), during an 8-hr layover in London, remains my favorite long-form discussion about the math of ML, besides other things...
www.youtube.com/watch?v=URtF_UHYBSo -
Anil Ananthaswamy
@anilananth
The @MLStreetTalk podcast I did back in July (the day after WHY MACHINES LEARN was released), during an 8-hr layover in London, remains my favorite long-form discussion about the math of ML, besides other things...
www.youtube.com/watch?v=URtF_UHYBSo -
Anil Ananthaswamy
@anilananth
Thrilled that WHY MACHINES LEARN got this wonderful shout-out from @ProfCalNewport: "A deep look at the mathematical innovations that made the AI revolution possible. One of the most useful books on AI that I've ever read!" — Cal Newport, New York Times bestselling author of Slow Productivity and Deep Work, and Professor of Computer Science at Georgetown University
www.penguinrandomhouse.com/books/677608/why-machines-lear… -
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
"Finetuning LLMs is Critical for Test-Time Scaling." @CarnegieMellon's @aviral_kumar2 on the key ingredients for optimizing test-time compute at the Simons Institute's workshop on The Future of Language Models and Transformers: simons.berkeley.edu/workshops/future-language-models-tran…
-
Anil Ananthaswamy reposted
Ashish Tendulkar
@ashish_vt
Got my copy of "why machines learn?" by @anilananth.
-
Anil Ananthaswamy
@anilananth
Whose Fault Is it Anyway? The Problem of Credit Assignment in Machine Learning
Credit assignment is one of the central ideas in machine learning. When you are training a ML model, and it makes an error, an algorithm has to apportion credit (really, it’s blame, not credit) to each of the model’s parameters. Then the algorithm adjusts the parameters ever so slightly, such that the model’s error for the same input is reduced, and it gets closer to producing the correct answer.
In the late 1950s, when Frank Rosenblatt designed the perceptron and Bernie Widrow designed ADALINE (for adaptive linear neuron), they were limited by the ideas of the time in how they could do credit assignment. Both the perceptron and ADALINE were single-layer neural networks. The algorithms for updating the network’s parameters (i.e. the weights) did not work for multi-layer neural networks. Crucially, the algorithm developed by Widrow along with Ted Hoff, called the Widrow-Hoff Least Mean Squares (LMS) algorithm, presaged what was to come, because they had figured out a noisy, roughshod way of doing gradient descent, using an algebraic approximation of the calculus used to minimize error.
There were plenty of indications from the 1960s onwards for how to possibly do credit assignment for multi-layer neural networks. If you are interested, look up Jürgen Schmidhuber’s blog post. But it was the 1986 Nature paper—”Learning representations by back-propagating errors”—by Rumelhart, Hinton and Williams that sealed the deal. The backpropagation algorithm was born. It showed how to do credit assignment for multi-layer neural networks with hidden layers (meaning layers that are not exposed directly to the network’s inputs or outputs); more importantly, it showed what the networks learned as a result of doing backpropagation.
The backpropagation algorithm is by no means the only way to do credit assignment. For one, the algorithm is not biologically plausible: our brains are almost certainly not doing backprop. That’s because the algorithm has to keep track of the weight matrices and the activations of neurons, in order to use the chain rule of calculus and do credit assignment back from the output layer to the input layer. This can’t be what the brain is doing. It’s likely using some local mechanism, meaning a mechanism that only depends on figuring out how to apportion blame for an error to a given neuron’s synapses by looking at its immediate neighborhood, and not depend on long-distance connections.
The search for a biologically-plausible mechanism is ongoing.
For an in-depth look at the math of backprop and the story of how Hinton and others came to it, see WHY MACHINES LEARN.
US: www.penguinrandomhouse.com/books/677608/why-machines-lear…
UK: www.penguin.co.uk/books/446849/why-machines-learn-by-anan… -
Anil Ananthaswamy
@anilananth
Machine Learning’s Arc Of History
Focusing on the long arc of the history of machine learning—and not just on what’s happening now—brings into sharp relief this moment in time. Machine learning-enabled artificial intelligence has a long and storied past, particularly when it involves neural networks, including periods when the phrase “neural networks” was anathema. But unlike areas like physics, where the key historical figures (say, those responsible for quantum physics) are long gone, ML’s past is extant. Barring some luminaries (Frank Rosenblatt comes to mind; he died of a tragic accident when was in his 40s), many of the seminal figures in the development of machine learning are not just still around to tell their stories, but even actively involved in their work.
More here: anilananthaswamy.com/why-machines-learn-codebook-blog/the… -
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
The Theoretical Minimum (for Machine Learning)…And Why
Linear Algebra, Calculus, and Probability & Statistics often get mentioned as the minimum math you need to start on your machine learning journey. But why these disciplines? Here’s an intuition for why:
anilananthaswamy.com/why-machines-learn-codebook-blog/the… -
Anil Ananthaswamy
@anilananth
The Theoretical Minimum (for Machine Learning)…And Why
Linear Algebra, Calculus, and Probability & Statistics often get mentioned as the minimum math you need to start on your machine learning journey. But why these disciplines? Here’s an intuition for why:
anilananthaswamy.com/why-machines-learn-codebook-blog/the… -
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
Can we tell when LLMs hallucinate or produce toxic/harmful content? Fascinating talk by @dbeagleholeCS at the Simons Institute about detecting semantic features learned by LLMs and using them to steer their output, & the link to recursive feature machines.
www.youtube.com/watch?v=TlLu32yHYfk&t=1s -
Anil Ananthaswamy reposted
Jeff Dean
@JeffDean
Want to check out the source for the "AlexNet" paper? Google has made the code from Alex Krizhevsky, @ilyasut, and @geoffreyhinton's seminal "ImageNet Classification with Deep Convolutional
Neural Networks" paper public, in partnership with the Computer History Museum.
As I said in the press release, "Google is delighted to contribute the source code for the groundbreaking AlexNet work to the Computer History Museum".
computerhistory.org/press-releases/chm-makes-alexnet-sour… -
Anil Ananthaswamy reposted
Melanie Mitchell
@MelMitchell1
In my latest column for Science magazine, I discuss recent AI "reasoning" models -- how it works, to what extent it captures "genuine" reasoning processes, and what's needed to answer such questions.
www.science.org/doi/10.1126/science.adw5211 -
Anil Ananthaswamy
@anilananth
In his Deep Questions podcast, @ProfCalNewport (author of Deep Work), gives a shoutout to WHY MACHINES LEARN ("I loved it" he says). Also, intrigued by his other recommendation: Burn Math Class, by Jason Wilkes. Will read that! Thanks Cal.
(hear podcast at 1h21m for the recommendations)
podcasts.apple.com/us/podcast/ep-338-why-youre-always-dis…? -
Anil Ananthaswamy
@anilananth
The Many Ways To Grok Machine Learning
The jargon of machine learning can be overwhelming. Here are some intuitions that can help make sense of the various aspects of ML.
Training vs test data, supervised vs unsupervised vs self-supervised learning, regression vs classification, linear vs non-linear models, hand-designed vs learned features, discriminative vs generative AI, and so on: These are a few of the important axes along which one can analyze and understand ML. The more ways we can grok machine learning, the more it begins making sense. Here's a detailed look at the many faces of ML: anilananthaswamy.com/why-machines-learn-codebook-blog/the… -
Anil Ananthaswamy
@anilananth
If you are in the Princeton area, here's a wonderful opportunity to listen to @MelMitchell1 and @random_walker talk about cutting through the AI hype. Can't imagine a better duo to tackle this topic!
Arvind Narayanan
@random_walker
I'm so excited that I'm doing a Princeton University public lecture together with @MelMitchell1 on AI. Come join us next Thursday at 5pm if you're in the area. While I'm a big fan of Professor Mitchell's work and she was kind enough to peer review AI Snake Oil, I've never had the chance to meet her until now, so I'm really looking forward to it! lectures.princeton.edu/lectures/2025/guide-cutting-throug…
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
LLMs are data guzzlers. Can we build small data-efficient language models that are developmentally and cognitively plausible? @a_stadt talks of the BabyLM Challenge at the Simons Institute workshop on LLMs, Cognitive Science, Linguistics, and Neuroscience. www.youtube.com/watch?v=Oyf-sS2oEvo
-
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
The Role of Vectors in Machine Learning
We hear over and over that machine learning is linear algebra, which, in turn, has to do with vectors and matrices. But why? What’s so amazing about this particular mathematical discipline as it pertains to ML? Here, from a bird’s eye-view, are 7 reasons why understanding vectors & matrices and their manipulations is integral to making sense of machine learning:
anilananthaswamy.com/why-machines-learn-codebook-blog/the… -
Anil Ananthaswamy
@anilananth
The Role of Vectors in Machine Learning
We hear over and over that machine learning is linear algebra, which, in turn, has to do with vectors and matrices. But why? What’s so amazing about this particular mathematical discipline as it pertains to ML? Here, from a bird’s eye-view, are 7 reasons why understanding vectors & matrices and their manipulations is integral to making sense of machine learning:
anilananthaswamy.com/why-machines-learn-codebook-blog/the… -
Anil Ananthaswamy reposted
Alexander Klein
@kleinalexanderm
I’m 49. My first book just arrived in the mail. Grateful that I’m young enough for my parents to be alive, and old enough for my kids to also share the moment. Today: seeing 10 years of work made flesh. A good day!
-
Anil Ananthaswamy
@anilananth
Once authors publish their books, some have a voice inside their heads post-publication, whispering: "Should I have included that bit that I cut out?"
For WHY MACHINES LEARN, I often wonder about a few pages that I wrote and then cut. In those pages, I delved into vector spaces, their amazing properties, and why vectors, matrices and functions can all inhabit vector spaces.
It was a section I loved but it felt too didactic (for a book that was already didactic); it had a text-book feel to it. So, I cut it down, and referred to vector spaces in the passing in the chapter on the Universal Approximation Theorem, to move the narrative along.
But that voice doesn't go away. I think anyone who is really interested in ML needs to grok the power and beauty of abstract vector spaces. Here are some resources:
There's, of course, 3Blue1Brown: youtu.be/TgKwz5Ikpc8
And I like Trevor Bazett: youtu.be/72GtkP6nP_A
WHY MACHINES LEARN here: www.penguinrandomhouse.com/books/677608/why-machines-lear… -
Anil Ananthaswamy reposted
Next Big Idea Club
@NextBigIdeaClub
AI may be on the rise, but these human writers have something to say about it. @HamiltonMann @parmy @anilananth @madhumita29 @verityharding open.substack.com/pub/nextbigideaclub/p/8-essential-reads…
-
Anil Ananthaswamy reposted
Arvind Narayanan
@random_walker
Happy to hear that AI Snake Oil was a finalist for the Association of American Publishers' 2025 PROSE award in the Computing and Information Sciences category.
-
Anil Ananthaswamy
@anilananth
As a kid growing up in India in the 70s, I coveted the orange spine of Penguin paperbacks (P.G.Woodhouse, anyone?) So, it's an immense joy to see the UK cover design for WHY MACHINES LEARN, orange spine and all ... @PenguinUKBooks @AllenLaneBooks
And I love that they have put @stevenstrogatz's blurb on the front.
Due out 28 Aug, 2025. So, still a few months away. Meanwhile, here are the links to the US and UK hardcover editions:
UK: www.penguin.co.uk/books/446849/why-machines-learn-by-anan…
US: www.penguinrandomhouse.com/books/677608/why-machines-lear… -
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
1/4 "Our theories of brains and minds have always been dominated, perhaps limited, by the technology of the day," said @SuryaGanguli. From Ancient Greeks and their hydraulics-inspired theories to Hobbes's notion of the mind as a clockwork machine to ...
-
Anil Ananthaswamy reposted
Richard Sutton
@RichardSSutton
awards.acm.org/about/2024-turing
Machines that learn from experience were explored by Alan Turing almost eighty years ago, which makes it particularly gratifying and humbling to receive an award in his name for reviving this essential but still nascent idea. -
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
8 Essential Reads on Artificial Intelligence. I'm happy to see WHY MACHINES LEARN in this list, alongside @random_walker and @sayashk's AI SNAKE OIL and others!
nextbigideaclub.com/magazine/8-essential-reads-artificial… -
Anil Ananthaswamy
@anilananth
Thank you Pierre!
-
Anil Ananthaswamy
@anilananth
8 Essential Reads on Artificial Intelligence. I'm happy to see WHY MACHINES LEARN in this list, alongside @random_walker and @sayashk's AI SNAKE OIL and others!
nextbigideaclub.com/magazine/8-essential-reads-artificial… -
Anil Ananthaswamy reposted
Anil Seth
@anilkseth
"How does the brain construct (the experience of) reality?" I really enjoyed speaking to my old friend @davideagleman, all the way back from San Diego days more than 20 years ago #innercosmos 🧠👇🏽
David Eagleman
@davideagleman
New episode of Inner Cosmos has dropped: "How does the brain construct reality?" Join guest Anil Seth (@anilkseth) (author of Being You) to dive deep into the scientific questions of consciousness. Do you perceive red the same way I do? What is wrong with the textbook model of vision? Why do brains have so many internal feedback loops?
eagleman.com/podcast/94
#brain #consciousnessThis media couldn't be displayed. See it on the original post. -
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
My conversation with @CelestaCapital's Sriram Vishwanathan about the history and math of #machinelearning, and much more... youtu.be/_k5VM7cECfs
-
Anil Ananthaswamy
@anilananth
My conversation with @CelestaCapital's Sriram Vishwanathan about the history and math of #machinelearning, and much more... youtu.be/_k5VM7cECfs
-
Anil Ananthaswamy
@anilananth
Here's an episode of the TechSurge: Deep Tech VC podcast, where I chat with Sriram Viswanathan, Founding Managing Partner of @CelestaCapital, about the math of machine learning and other things.
Understanding the Elegant Math Behind Modern Machine Learning | TechSurge: Deep Tech VC Podcast techsurgepodcast.com/podcasts/understanding-the-elegant-m… -
Anil Ananthaswamy
@anilananth
On Fri, 28 Feb, Prof Rajesh Gopakumar (Director, @ictstifr, Bangalore and @iitk alum) and I will be in conversation about AI/ML and its connection to physics and other basic sciences, moderated by Shashi Upadhyay, President @Zendesk. Venue: Cupertino
Sign up/RSVP here: docs.google.com/forms/d/e/1FAIpQLSfs-5jQO00cdRQALO4X3ByKm…
Event details: www.iitmaana-bayarea.org/events -
Anil Ananthaswamy reposted
David Chalmers
@davidchalmers42
the conference on infant consciousness at NYU this friday-saturday is going to be great. if you can't attend in person, there will be a live stream. www.eventbrite.com/e/1217079149499
-
Anil Ananthaswamy reposted
Surya Ganguli
@SuryaGanguli
Love this quote from Sir Michael Atiyah: "Algebra is the offer made by the devil to a mathematician. The devil says, 'I will give you this powerful machine...all you need to do is give up your soul: give up geometry...' The danger to our soul is there... when you pass into algebraic calculation,..., you stop thinking geometrically, you stop thinking about meaning.
-
Anil Ananthaswamy reposted
Surya Ganguli
@SuryaGanguli
Was fun to moderate a panel @SimonsInstitute on LLMs, neuroscience, linguistics and theory: deep questions about what LLMs understand and how we can understand LLMs. Excellent panelists:
@alsuhr @kmahowald @GwilliamsL @srush_nlp @LakeBrenden @spiantado
www.youtube.com/watch?v=RD2EhzVHjro
Simons Institute for the Theory of Computing
@SimonsInstitute
What does it mean to understand? Do LLMs understand language and can we understand LLMs? @SuryaGanguli asks deep questions of experts at a workshop at the Simons Institute. The panel: @alsuhr @kmahowald @GwilliamsL @srush_nlp @LakeBrenden @spiantado
www.youtube.com/watch?v=RD2EhzVHjro -
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
What does it mean to understand? Do LLMs understand language and can we understand LLMs? @SuryaGanguli asks deep questions of experts at a workshop at the Simons Institute. The panel: @alsuhr @kmahowald @GwilliamsL @srush_nlp @LakeBrenden @spiantado
www.youtube.com/watch?v=RD2EhzVHjro -
Anil Ananthaswamy
@anilananth
I like the "Lost in Philosophy" notion :)
To be fair to Einstein, he changed his mind and eventually wasn't as bothered by the randomness inherent in the quantum formalism. His "God doesn't play dice" quote gets too much attention. He was really bothered by the non-locality implied by the ostensible collapse of the wavefunction. That debate about the nature of collapse, whether the wavefunction is epistemic or ontic etc., hasn't really been laid to rest, given that the various interpretations/theories are all in the running (except for, maybe, forms of local realism).
Nirmalya Kajuri
@Kaju_Nut
Sabine Hossenfelder coined 'Lost in Math' for physicists who (allegedly) get so enamoured of the mathematical beauty of their theories that they believe they must be true.
There is a related but separate phenomenon, which I call 'Lost in Philosophy' where physicists (scientists more generally) fall in love with their ideals of how nature should be. They invent theories that start from their own philosophical ideals and try to fit nature into it.
The classic example is Einstein, who refused to accept quantum theory as fundamental because he believed "God does not play dice"--i.e nature should follow the principle now called 'realism'. He tried to invent a new theory which could explain nature while satisfying realism, which incidentally led him to also get Lost in Math and failed.
This approach usually fails. Nature does not care about what principles you think it should follow. -
Anil Ananthaswamy
@anilananth
There's something about book titles. No matter the title, someone will ask: How come you chose that title?
For WHY MACHINES LEARN, the question I get most often is: shouldn't it be How, not Why?
Well, weirdly, the title suggested itself when I was learning the perceptron convergence proof. The algorithm told me how, but the simple linear algebraic proof was the rationale for why a single-layer neural network using the perceptron algorithm works.
I used the title in my book proposal--and my publisher didn't change it. That doesn't happen often, if at all!
I understand I'm taking liberties by using WHY...but it sorta works.
My first book's working title was The Edge of Reason! My US publisher hated it, not surprisingly, since it made little sense. We went went through numerous other titles before settling on THE EDGE OF PHYSICS, which I really like.
My second book's working title was Maladies of the Self. My editor came up with THE MAN WHO WASN'T THERE. Took me a while to get used to it, but got to really like it. I had a soft spot for Maladies of the Self, but glad I didn't use it, because my perspective on conditions such as schizophrenia changed as I worked on the book; maladies would have been wrong.
The third book started of in the proposal as Feynman's Challenge, which when you think about it doesn't say much. We ended up with, thanks to my editor, THROUGH TWO DOORS AT ONCE. Again, took time for me to love it, but I did; I think it captures the essence of the double-slit experiment, which is what the book is about. I wouldn't have thought of the title...My editor did.
www.amazon.com/stores/author/B001HPNL1M/allbooks -
Anil Ananthaswamy reposted
Andrew Gordon Wilson
@andrewgwils
Now and then I miss the days of doing Gaussian process research. ML/AI was a simpler world, driven by clear ideas, principles, and goals. People were there out of passion, not because it's what everyone is doing, not as a ticket to a glamorous job. Less restlessness, less fomo.
-
Anil Ananthaswamy
@anilananth
As always a lovely piece by @themarginalian, particularly pertinent as AI looms, perhaps raising existential questions about what it means to be human
Maria Popova
@themarginalian
Oliver Sacks on despair and the meaning of life www.themarginalian.org/2024/11/08/oliver-sacks-letters-me…
-
Anil Ananthaswamy
@anilananth
With Large Language Models being powered by Transformers, maybe you have wondered--like I did--why a Transformer is called, well, a Transformer. Here's a simple high-level view.
LLMs like GPT, Claude, DeepSeek and others use a deep neural network architecture built around what's called a decoder-only Transformer. The inputs to this neural network is a sequence of words (actually tokens, but let's stick to words). The job of the LLM is to predict the next word to follow this sequence. This is how the whole thing unfolds.
The words are first turned into vector embeddings; this information also encodes something about their position in the sentence.
These embeddings flow through the entire network and come to the final two layers (see Figure below). By now, the embeddings should have been--you guessed, it--transformed, so that the final embedding in the sequence of embeddings can be used to predict the next word.
So, that's what the Transformer layer does, sitting squarely in the middle of the larger neural network. It's also a neural network itself, and it's job is to take a set of input embeddings and slowly transform them into another set of embeddings that captures some of the syntactical and semantic relationships between the words.
Understanding and appreciating what exactly a decoder-only Transformer architecture achieves requires getting a good grounding in vectors, vector-matrix manipulations (i.e. the basics of linear algebra), probability distributions (so can understand what an LLM is trying to predict and what is being optimized) and gradient descent and backpropagation. You can even appreciate the overall function that the LLM approximates, and the connection to the universal approximation theorem.
LLMs are really a technology where much of the basic concepts of ML come together. Of course, all of this basic math is explained in WHY MACHINES LEARN.
US: www.penguinrandomhouse.com/books/677608/why-machines-lear…
UK: www.penguin.co.uk/books/446849/why-machines-learn-by-anan… -
Anil Ananthaswamy reposted
Aryan Arya
@AryanAryaCodes
Just a few pages in "Why Machines Learn" by @anilananth and… holy hell.
This isn’t another "AI is magic" book. It’s the antidote to the hand-wavy explanations I got at university.
Shoutout to @BorisMeinardus for the nudge. Finally, a book that respects my curiosity and time! -
Anil Ananthaswamy
@anilananth
Like the look of WHY MACHINES LEARN alongside pastries and coffee :-) My kind of reading!
Aleksandr Yampolskiy
@ayampolskiy
Here are some books on my reading list right now. What's on yours?
* Why Machines Learn : The elegant math behind modern AI
* Joy and Success At Work: Building Organizations That Don’t Suck (The Life Out Of People)
* Software as a Science: Unlock Limitless Recurring Revenue Without Losing Control
* Secret Lives of Customers
* The First Minute: How to Start Conversations That Get Results (Business Communication Skills Books)
* How to Decide: Simple Tools for Making Better Choices
* The Program: Inside the Mind of Keith Raniere and the Rise and Fall of NXIVM
* Patriot : A Memoir by Alexey Navalny
* Start-Up Nation The Story of Israel's Economic Miracle
#books #reading -
Anil Ananthaswamy reposted
Swapna Kumar Panda
@swapnakpanda
Currently I am readig this one: "Why Machines Learn: The Elegant Maths Behind Modern AI".
-
Anil Ananthaswamy reposted
Tom Froese, Embodied Cognitive Science Unit (ECSU)
@DrTomFroese
Enjoyable intro to machine learning by @anilananth!
The part on agency made me realize: I never take up the suggestions that claude.ai makes at the end of its responses. Agency still seems a weak spot compared to linguistic sense-making?
youtu.be/URtF_UHYBSo -
Anil Ananthaswamy reposted
Anil Seth
@anilkseth
I'm excited to be part of #synapseconclave in Delhi next weekend (feb 22-23), talking consciousness w/ @ShomaChaudhury - & what a terrific programme, also featuring @pushmeet @NitaFarahany @drmichaellevin Mathieu Ricard & many others 🧠🤯 synapseconclave.com/program
-
Anil Ananthaswamy
@anilananth
Why does machine learning seem so befuddling to software engineers from the pre-ML era?
I know the feeling: I used to be a software engineer in the 90s, building large-scale distributed systems.
ML seems strange at first because it inverts what it means to solve a problem.
In traditional software design, we design the algorithm to turn some input into an output. It could be any problem. All software comes down designing/coding the algorithm that processes inputs in some pre-determined manner and generates the desired output.
For example, if you wanted to write code to distinguish between images of birds and animals, you could write code to look for low-level features -- say, number of legs or the presence of feathers (you'd need to know what information to look for in the pixels of the image) -- and then use these features to categorize your images.
But ML turns this process upside down. You start with data, tons of data. Let's say you have a whole bunch of images of birds and animals tagged/labeled as such by humans. These are input-output pairs: for each input image, there's a corresponding output label. This is your training data.
Now, you can train an ML model to take the pixels of image and categorize it as either a 0 (bird) or 1 (animal). If you are using a deep neural network, you don' t need to worry about the exact features that define a bird or animal. The neural network will figure it out.
Then, if you have a new image, you present it to the model, and it'll spit out a 0 or a 1. Depending on how interpretable your model is, you may or may not be able to decipher the exact algorithm it has learned.
So, while traditional software requires you think to about the problem you want to solve, and break it down into sub-problems that you can write algorithms for and then compose the overall solution, machine learning requires an entirely different way of thinking.
ML starts with data. Without adequate data, ML is of no use. Once you have the data (whether it's annotated or not), you then have to figure out what questions you can ask of the data. The script is flipped. The questions you ask of the data dictate the loss function you'll design to train your ML model, which will dictate the algorithm or the functional transformation it learns to solve your problem.
WHY MACHINES LEARN lays the groundwork for this kind of thinking.
US: www.penguinrandomhouse.com/books/677608/why-machines-lear…
UK: www.penguin.co.uk/books/446849/why-machines-learn-by-anan… -
Anil Ananthaswamy
@anilananth
I know just enough French to say Merci beaucoup!
Stéphane Traumat
@straumat
Fini de lire "Why Machines Learn" qui explique les mathématiques derrière l'IA. Je ne suis pas bon en math mais j'ai compris quasiment tout et mon dieu, qu'est ce que c'est BEAU !!!
www.amazon.fr/gp/product/B0CFZZCNGM?psc=1&tag=blog-straum… -
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
Remember when @fchollet called @MLStreetTalk Netflix for ML folks. It's true. @ecsquendor has dropped the latest video, with Sepp Hochreiter, of LSTM fame; can't wait to watch it. I, of course, loved my conversation with MLST about WHY MACHINES LEARN
youtu.be/URtF_UHYBSo -
Anil Ananthaswamy
@anilananth
Remember when @fchollet called @MLStreetTalk Netflix for ML folks. It's true. @ecsquendor has dropped the latest video, with Sepp Hochreiter, of LSTM fame; can't wait to watch it. I, of course, loved my conversation with MLST about WHY MACHINES LEARN
youtu.be/URtF_UHYBSo -
Anil Ananthaswamy
@anilananth
What's So Elegant About the Math of Machine Learning? I have often been asked this question, given it's the sub-title of WHY MACHINES LEARN, including by @MLStreetTalk's Tim Scarfe, by @StartsWithABang for @bigthink, and others.
If you have been studying machine learning for a long time, the elegance might not be obvious. But if you came to ML like I did--decades after having studied the relevant math for my undergrad and graduate EE degrees, and decades after having done regular software engineering--then machine learning hits you like a ton of bricks.
Who would have thought that basic ideas like vectors, matrices, matrix-vector multiplication, differential calculus, gradient descent, elementary probability and statistics -- all of which you learn in high school or at least during the first years of undergrad in STEM -- would form the basis of the extraordinary revolution that's underfoot?
But that's just wonder. What about elegance? Well, that's really a subjective perspective. Again, many might disagree. But here's a sampling of the kinds of things I found beautiful:
The Perceptron Convergence Proof: It's just linear algebra, but the proof of why the perceptron algorithm will linearly separate two clusters of data in finite time, if such a divide exists, is a thing of beauty.
Support Vector Machines and Kernel Methods: These algorithms have been somewhat forgotten, given the success of deep neural networks, but when you encounter the formulation of SVMs, and the use of kernel methods to project data into higher dimensions -- it's such a counter-intuitive thing to do--you can't but be amazed. I was!
Hopfield Networks Convergence Proof: Again, just linear algebra shows you something very powerful--that these networks will settle into an energy minimum when perturbed and retrieve what amounts to a stored memory! Think about that. I can't help wonder whether our brains are doing something similar. John Hopfield--who won the @NobelPrize in Physics in 2024 for this work--certainly did.
The Backpropagation Algorithm: This might be a contentious candidate; some don't think it's elegant. But I find backprop powerful, and consequently elegant. Something that can be used to train a tiny, tiny neural network with just one hidden neuron can also be used to train the giant Large Language Models of today. The elegance is in its power. It's almost certainly not what our brains are doing, but still!
All of this and more, of course, is detailed in WHY MACHINES LEARN.
US Edition: www.penguinrandomhouse.com/books/677608/why-machines-lear…
UK Edition: www.penguin.co.uk/books/446849/why-machines-learn-by-anan… -
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
Whoever left this review on Amazon for WHY MACHINES LEARN: thank you! Authors will relate to their sometimes years-long effort being acknowledged with such kind words.
"A book written with love for knowledge, with affection for an interested reader. The story of how we got here, mixing anecdotes and ideas placed in their context and highlighting the heroes who brought us here. A gem that shows us the mathematical tricks, some incredibly simple, behind the magic that invades us today."
www.amazon.com/gp/product/0593185749 -
Anil Ananthaswamy reposted
Rinaldo Rasa
@odlanir
i think a lot of people that read your book has the same point of view about your beautiful work ! thanks Anil !
-
Anil Ananthaswamy
@anilananth
Whoever left this review on Amazon for WHY MACHINES LEARN: thank you! Authors will relate to their sometimes years-long effort being acknowledged with such kind words.
"A book written with love for knowledge, with affection for an interested reader. The story of how we got here, mixing anecdotes and ideas placed in their context and highlighting the heroes who brought us here. A gem that shows us the mathematical tricks, some incredibly simple, behind the magic that invades us today."
www.amazon.com/gp/product/0593185749 -
Anil Ananthaswamy reposted
Monique Morrow
@moniquejmorrow
Understanding that #ai is a math and science-motivated tool helps us understand the dangers of completely relying on it to replace important processes and tasks. As a tool, it can revolutionize our world, but we must first view it that way. Thanks, @anilananth.
#aigovernance
Anil Ananthaswamy
@anilananth
10 profound answers about the math behind AI
It’s knowledgeable, confident, and behaves human-like in many ways. But it’s not magic that powers AI though; it’s just math and data. My Q&A for @bigthink with @StartsWithABang
bigthink.com/starts-with-a-bang/10-answers-math-artificia… -
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
"[LLMs] are the first organisms we have other than humans that are really good at language...I don't understand how some people are not excited by this, if they are language researchers especially." @mit's @ev_fedorenko at the Simons Institute last week. simons.berkeley.edu/workshops/llms-cognitive-science-ling…
-
Anil Ananthaswamy reposted
hardmaru
@hardmaru
I had the honor to meet Professor Shun-Ichi Amari, a pioneer in neural networks, at the University of Tokyo. He worked on early neural nets using pen and paper, before running computing experiments in the future. He talked not only about AI, but human evolution and consciousness.
Jürgen Schmidhuber
@SchmidhuberAI
As 2022 ends: 1/2 century ago, Shun-Ichi Amari published a learning recurrent neural network (1972) much later called the Hopfield network (based on the original, century-old, non-learning Lenz-Ising recurrent network architecture, 1920-25) people.idsia.ch/~juergen/deep-learning-history.html
-
Anil Ananthaswamy reposted
Tivadar Danka
@TivadarDanka
Four years ago, I started a project that turned out to be life-altering, and it has just passed a huge milestone.
I'm happy to announce that my Mathematics of Machine Learning book will soon be published by Packt Publishing! Yes, there will be a physical version. -
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
Why Are Modern LLMs So Big?
Here's one way to grok why state-of-the-art LLMs are so massive and trained on so much data, or rather, why scaling up in size, training data and/or training time gives LLMs unexpectedly powerful behaviors.
It has to do with solving the incredibly difficult problem of estimating the shape of ultrahigh dimensional surfaces representing probability distributions.
At a high-level (leaving aside the intricacies of the Transformer architecture), here's what an LLM does. Given a prompt, or let's say a string of words/tokens, it generates the next most likely word/token to follow the prompt, appends it to the prompt, and feeds the augmented prompt to itself, and generates the next most likely word/token to follow it, and continues the process until it generates an end-of-sequence token, causing it to stop.
All the words/tokens it generates to follow the initial prompt put together make up the LLMs response to the prompt--and the response can even seem liked a reasoned answer to some complex question posed in the prompt.
How does it do this?
Let's say the LLM has a vocabulary of 1000 words. Now, given a prompt of 100 words, it has to first estimate this conditional probability distribution:
P (w101 | w1, w2, ..., w100)
which is the probability of the 101st word given an input sequence of 100 words. This probability has to be calculated for each of the 1000 words in the vocabulary. In essence, the LLM has to compute the conditional probability distribution over 1000 words, given the 100 words in the prompt. And then sample from the distribution, let's say greedily, to pick the word with the largest probability.
You can think of the entire Transformer-based neural network as something that implements the composite function that both models the probability distribution and samples from it, producing the necessary output.
Let's say there were only two words in the prompt. The function can be written as:
y = f (x1, x2)
Where x1 and x2 are vectors that represent the two words in the prompt, and y is a vector that represents the word that follows x1, x2
To help visualize what's happening, let's think of x1, x2 and y as scalars. In this case, the transformer has to learn a function with two variables. Let's say 100 data points are enough to figure out the shape of this curve (10 for each of the two axes).
If had three inputs-- x1, x2, and x3 -- then, we'd need 10-cubed data points to estimate the curve, so 1000 instances of data. Four inputs would require 10 raised to the fourth power, so 10,000 instances of data.
You can see how the amount of data grows exponentially with the growth in the number of input dimensions. Also, the number of parameters in the model that are needed to learn the function with the requisite fidelity also grow. And the more compute you use to train, the closer LLMs get to learning good estimates of the conditional probability distributions.
Modern LLMs have vocabularies in the tens of thousands and can take tens of thousands of words in their prompt--so during each pass through the network, they are essentially modeling extremely high-dimensional surfaces representing conditional probability distributions. And remember, it's not just that there are, say, 10,000 words in the prompt: the order and relationship between these words matter when prediction the next word.
This is what the attention mechanism in Transformers is all about! Learning all possible conditional probability distributions to a certain degree of exactness, so that you can predict the next word with a high probability of being correct for any and all combinations of input words is an insanely tall order. The fact that Transformer-based LLMs do so--hallucinations and errors notwithstanding--is nothing short of magical.
The essential insights into probability and statistics that you need to appreciate the curse of dimensionality and why it's so difficult to model these high-dimensional surfaces is described in WHY MACHINES LEARN
US Edition: www.penguinrandomhouse.com/books/677608/why-machines-lear…
UK Edition: www.penguin.co.uk/books/446849/why-machines-learn-by-anan… -
Anil Ananthaswamy
@anilananth
Why Are Modern LLMs So Big?
Here's one way to grok why state-of-the-art LLMs are so massive and trained on so much data, or rather, why scaling up in size, training data and/or training time gives LLMs unexpectedly powerful behaviors.
It has to do with solving the incredibly difficult problem of estimating the shape of ultrahigh dimensional surfaces representing probability distributions.
At a high-level (leaving aside the intricacies of the Transformer architecture), here's what an LLM does. Given a prompt, or let's say a string of words/tokens, it generates the next most likely word/token to follow the prompt, appends it to the prompt, and feeds the augmented prompt to itself, and generates the next most likely word/token to follow it, and continues the process until it generates an end-of-sequence token, causing it to stop.
All the words/tokens it generates to follow the initial prompt put together make up the LLMs response to the prompt--and the response can even seem liked a reasoned answer to some complex question posed in the prompt.
How does it do this?
Let's say the LLM has a vocabulary of 1000 words. Now, given a prompt of 100 words, it has to first estimate this conditional probability distribution:
P (w101 | w1, w2, ..., w100)
which is the probability of the 101st word given an input sequence of 100 words. This probability has to be calculated for each of the 1000 words in the vocabulary. In essence, the LLM has to compute the conditional probability distribution over 1000 words, given the 100 words in the prompt. And then sample from the distribution, let's say greedily, to pick the word with the largest probability.
You can think of the entire Transformer-based neural network as something that implements the composite function that both models the probability distribution and samples from it, producing the necessary output.
Let's say there were only two words in the prompt. The function can be written as:
y = f (x1, x2)
Where x1 and x2 are vectors that represent the two words in the prompt, and y is a vector that represents the word that follows x1, x2
To help visualize what's happening, let's think of x1, x2 and y as scalars. In this case, the transformer has to learn a function with two variables. Let's say 100 data points are enough to figure out the shape of this curve (10 for each of the two axes).
If had three inputs-- x1, x2, and x3 -- then, we'd need 10-cubed data points to estimate the curve, so 1000 instances of data. Four inputs would require 10 raised to the fourth power, so 10,000 instances of data.
You can see how the amount of data grows exponentially with the growth in the number of input dimensions. Also, the number of parameters in the model that are needed to learn the function with the requisite fidelity also grow. And the more compute you use to train, the closer LLMs get to learning good estimates of the conditional probability distributions.
Modern LLMs have vocabularies in the tens of thousands and can take tens of thousands of words in their prompt--so during each pass through the network, they are essentially modeling extremely high-dimensional surfaces representing conditional probability distributions. And remember, it's not just that there are, say, 10,000 words in the prompt: the order and relationship between these words matter when prediction the next word.
This is what the attention mechanism in Transformers is all about! Learning all possible conditional probability distributions to a certain degree of exactness, so that you can predict the next word with a high probability of being correct for any and all combinations of input words is an insanely tall order. The fact that Transformer-based LLMs do so--hallucinations and errors notwithstanding--is nothing short of magical.
The essential insights into probability and statistics that you need to appreciate the curse of dimensionality and why it's so difficult to model these high-dimensional surfaces is described in WHY MACHINES LEARN
US Edition: www.penguinrandomhouse.com/books/677608/why-machines-lear…
UK Edition: www.penguin.co.uk/books/446849/why-machines-learn-by-anan… -
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
"2025 is going to be a major year for AI ... I hope we all work on ensuring it goes well." @boazbaraktcs talks of using inference-time compute for AI safety at the Simons Institute's workshop on LLMs, Cognitive Science, Linguistics, and Neuroscience simons.berkeley.edu/workshops/llms-cognitive-science-ling…
-
Anil Ananthaswamy reposted
The Economist
@TheEconomist
Yann LeCun wants to build “a new revolution in AI”—models which understand the world.
Listen to @ylecun, the “godfather” of machine learning, on “Babbage” with @alokjha econ.st/4gsFm62 🎧 -
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
Why do babies learn with so much less data than LLMs? Mike Frank, @mcxfrank of @Stanford, lists possible reasons — nativism, constructivism, multimodal grounding — at the Simons Institute workshop on LLMs, Cognitive Science, Linguistics, and Neuroscience simons.berkeley.edu/workshops/llms-cognitive-science-ling…
-
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
The Transformer is amazing and amazingly complex! But break it down into its constituents, and you realize that every bit of it has a rationale, even if it's hard to grasp at first. For me, it took the writing of WHY MACHINES LEARN to make sense of the Transformer. Here's a take on the concepts needed to grok the most important neural network architecture of the past decade:
1) Vectors and matrices--can't get away from linear algebra. You need to understand why machines represent information as vectors, the notion of dot products, etc. And figure out what happens during matrix-vector multiplications! That's the stuff of Ch 1 and 2 in Why Machines Learn, and even Ch 6 (on Principal Component Analysis, which is a deeper dive into aspects of linear algebra).
2) The notion of similarity of vectors and the idea of distance measures (Manhattan or Euclidean distance, for example) is super important. This is discussed in Ch 5, in the exposition about the K-nearest neighbor algorithm.
3) Probability distributions. What does a Transformer-based LLM finally do? It calculates a conditional probability distribution over its entire vocabulary given an input sequence of words--and then samples from that distribution to predict the next most likely word! If these concepts don't make much sense as stated, then Ch 4 (on probability and statistics and the Bayes Optimal and Naive Bayes classifiers) is for you. It sets one up for understanding the incredible complexity of the distribution learned by the multi-layer Transformer and why it's so difficult to learn such a distribution.
4) And of course, gradient descent and backpropagation--the key to training any deep neural network, including Transformer-based architectures. The history of gradient descent to train a neural network begins with the Widrow-Hoff LMS algorithm in the late 1950s (Chapter 3), and ends with the development of backpropagation algorithm to train multilayer neural networks (Chapter 10).
5) Transformers have to learn some of the most complex functions to transform an input into the desired output. Why can deep neural networks do this? That's the topic of Chapter 9, on the Universal Approximation Theorem, which says that given enough artificial neurons in a hidden layer, a deep neural network will be able to approximate any function to arbitrary precision, no matter how complex the function.
6) Even understanding convolutional neural networks (CNNs), the topic of Chapter 11, is important for understanding aspects of the attention mechanism that's central to Transformers and the idea of shared weights etc.
7) Possibly one of the biggest advances in machine learning in recent times has been self-supervised learning--the idea of masking some part of the training data and teaching a neural network to correctly predict the masked part. This has made large-scale ML systems possible, avoiding the prohibitive cost of supervised learning, which required human annotations. This is described in detail in Chapter 12.
So, if you want to really get under the mathematical skin of Transformers, you can get an intuitive sense for the basic math and the historical narratives and stories behind the math in WHY MACHINES LEARN.
US Edition: www.penguinrandomhouse.com/books/677608/why-machines-lear…
UK Edition:
www.penguin.co.uk/books/446849/why-machines-learn-by-anan… -
Anil Ananthaswamy
@anilananth
The Transformer is amazing and amazingly complex! But break it down into its constituents, and you realize that every bit of it has a rationale, even if it's hard to grasp at first. For me, it took the writing of WHY MACHINES LEARN to make sense of the Transformer. Here's a take on the concepts needed to grok the most important neural network architecture of the past decade:
1) Vectors and matrices--can't get away from linear algebra. You need to understand why machines represent information as vectors, the notion of dot products, etc. And figure out what happens during matrix-vector multiplications! That's the stuff of Ch 1 and 2 in Why Machines Learn, and even Ch 6 (on Principal Component Analysis, which is a deeper dive into aspects of linear algebra).
2) The notion of similarity of vectors and the idea of distance measures (Manhattan or Euclidean distance, for example) is super important. This is discussed in Ch 5, in the exposition about the K-nearest neighbor algorithm.
3) Probability distributions. What does a Transformer-based LLM finally do? It calculates a conditional probability distribution over its entire vocabulary given an input sequence of words--and then samples from that distribution to predict the next most likely word! If these concepts don't make much sense as stated, then Ch 4 (on probability and statistics and the Bayes Optimal and Naive Bayes classifiers) is for you. It sets one up for understanding the incredible complexity of the distribution learned by the multi-layer Transformer and why it's so difficult to learn such a distribution.
4) And of course, gradient descent and backpropagation--the key to training any deep neural network, including Transformer-based architectures. The history of gradient descent to train a neural network begins with the Widrow-Hoff LMS algorithm in the late 1950s (Chapter 3), and ends with the development of backpropagation algorithm to train multilayer neural networks (Chapter 10).
5) Transformers have to learn some of the most complex functions to transform an input into the desired output. Why can deep neural networks do this? That's the topic of Chapter 9, on the Universal Approximation Theorem, which says that given enough artificial neurons in a hidden layer, a deep neural network will be able to approximate any function to arbitrary precision, no matter how complex the function.
6) Even understanding convolutional neural networks (CNNs), the topic of Chapter 11, is important for understanding aspects of the attention mechanism that's central to Transformers and the idea of shared weights etc.
7) Possibly one of the biggest advances in machine learning in recent times has been self-supervised learning--the idea of masking some part of the training data and teaching a neural network to correctly predict the masked part. This has made large-scale ML systems possible, avoiding the prohibitive cost of supervised learning, which required human annotations. This is described in detail in Chapter 12.
So, if you want to really get under the mathematical skin of Transformers, you can get an intuitive sense for the basic math and the historical narratives and stories behind the math in WHY MACHINES LEARN.
US Edition: www.penguinrandomhouse.com/books/677608/why-machines-lear…
UK Edition:
www.penguin.co.uk/books/446849/why-machines-learn-by-anan… -
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
Deep interest in @srush_nlp's timely talk on "How DeepSeek changes the LLM story" at the Simons Institute's workshop on LLMs, Cognitive Science, Linguistics, and Neuroscience. Starting now ...simons.berkeley.edu/workshops/llms-cognitive-science-ling…
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
Three ways DeepSeek V3 "shifts the curve": Architecture , Parallelization, and Quantization, says @srush_nlp at the Simons Institute workshop on LLMs, Cognitive Science, Linguistics, and Neuroscience simons.berkeley.edu/workshops/llms-cognitive-science-ling…
-
Anil Ananthaswamy reposted
RobbiewOnline
@RobbiewOnline
A fascinating book about Why Machines Learn by @anilananth
I'm going deep into #ai, trying to understand how it all works under the hood. I don't yet know why!
This book is very maths heavy (way deeper than my current knowledge base), so my first pass is to let the terms and formulae's wash over my head, but subsequent passes I'll hopefully understand more and more.
I may have to subscribe to @_MathAcademy_ before long!
I nearly didn't buy the book because the cover looked like something I'd draw in PowerPoint, but since I have seen an entirely different cover - now I want that edition ... yea yea, never judge a book by its cover!
It's very well written, even with my pathetic level of maths I don't feel like I'm drowning with bricks attached to my feet, though I do admit I panic and take deep breaths between submersions 😅 -
Anil Ananthaswamy reposted
Quanta Magazine
@QuantaMagazine
Researchers are finding that the architecture of LLMs has inherent limitations. But this is not the end of chatbots. @anilananth reports: www.quantamagazine.org/chatbot-software-begins-to-face-fu…
-
Anil Ananthaswamy reposted
Kevin Hartnett
@KSHartnett
I'm excited to share that my book, "The Proof in the Code," will be the first title released by @QuantaBks, a new imprint from @FSG_books and @SimonsFdn. It's the story of how @leanprover is transforming math research. Coming Spring 2026.
Quanta Books
@QuantaBks
🚨New book announcement!🚨 @KSHartnett's THE PROOF IN THE CODE (coming 2026) tells the gripping story of the birth and rise of Lean, a Microsoft program transforming the world of mathematics. A timely challenge to our assumptions about who–or what–can do math.
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
Wondering about the fuss over reasoning models like @OpenAI's o1 and DeepSeek R1? Last year, Noam Brown and Sasha Rush spoke about this emerging trend at the Simons Institute. Here's a small piece with links to videos by @polynoamial and @srush_nlp simons.berkeley.edu/news/winds-change-llms-become-more-th…
-
Anil Ananthaswamy reposted
Manos Koukoumidis
@Koukoumidis
If AI isn’t truly open, it will fail us. We can’t close in a black box our greatest invention yet just so that a few can freely monetize. AI needs its Linux moment, and so we started working towards it. This can only succeed if we all work together!
#oumi #opensource #collaboration
@rsalakhu @svlevine @larry_heck
@karpathy @atalwalkar @prfsanjeevarora @tsvetshop
@hhexiy @sainingxie @larry_heck @AnimaAnandkumar
@JunjieHu12 @georgiagkioxari @profjoeyg @pliang279
@danqi_chen @ChrisGPotts @BillMacCartney @vinodv
@tur_gokhan @dilekhakkanitur @j_foerst @gingsmith23 @kahinish @jamesjoaquin @gan3sh @ethanjb @kirbywinfield @egonzdp -
Anil Ananthaswamy reposted
Mike Knoop
@mikeknoop
just published my full @arcprize analysis of deepseek's r1-zero and r1. link below. key points:
r1-zero is more important than r1.
both r1-zero and r1 score ~15% on ARC-AGI-1. this is fascinating. it matches deepseek's own benchmarking showing comprable results in logical domains like math and coding across r1-zero and r1.
r1-zero removes the final human input bottleneck -- "expert CoT labeling" eg. supervised fine-tuning ("SFT"). from there to AGI, it's all about efficiency.
deepseek says r1-zero suffers from incoherence and language mixing. this has been corroborated online. but we saw no evidence in our testing. all this suggests:
1. SFT is not necessary for accurate and legible CoT reasoning in domains with strong verification.
2. the r1-zero training process is capable of creating its own internal domain specific language (DSL) in token space via RL optimization.
3. SFT is currently necessary for increasing CoT reasoning domain generality with these LLM architectures
this makes intuitive sense, as language itself is effectively a reasoning DSL. The exact same "words" can be learned in one domain and applied in another, like a program. the pure RL approach can not yet discover a broad shared vocabulary and I expect this will be a strong focus for future research.
ultimately r1-zero demonstrates the prototype of a potential scaling regime with zero human bottlenecks – even in the training data acquisition itself.
more broadly, the public is very under-informed about impending inference demand. o3 beating ARC-AGI-1 (75%/86% on low/high compute) was barely reported mainstream. expect more market whiplash as the frontier progress isn't disseminated fast enough. mainstream press has important work to do.
o1/o3/r1 benchmark accuracy scores are exciting but the real practical impact will be massively improved reliability, leading agents to finally start working in 2025.
we'll also start seeing "synthetic data" (low quality) becoming "real data" (high quality) -- and the end user is paying for it! there is a legit power concentration potential feedback loop here to understand.
r1-zero and r1 being open is great for the world, deepseek has moved the science forward. many folks have told me they plan to use r1's ideas for ARC Prize 2025, which i'm excited to see. we are going to rapidly find the limits of LLMs + CoT search. -
Anil Ananthaswamy reposted
Tom Goldstein
@tomgoldsteincs
Let’s sanity check DeepSeek’s claim to train on 2048 GPUs for under 2 months, for a cost of $5.6M. It sort of checks out and sort of doesn't.
The v3 model is an MoE with 37B (out of 671B) active parameters. Let's compare to the cost of a 34B dense model. 🧵 (1/4) -
Anil Ananthaswamy reposted
Jürgen Schmidhuber
@SchmidhuberAI
It has been said that AI is the new oil, the new electricity, and the new internet. And the once nimble and highly profitable software companies (MSFT, GOOG, ...) became like utilities, investing in nuclear energy, among other things, to run AI data centres. Open Source and the #deepseek #sputnik have once again shown that such companies "have no moat" in the field of AI.
In many talks over the last decade, I mentioned the rich guy I knew when I was young. He had a Porsche with something incredible: a mobile phone. He could call other people who had such a Porsche via a satellite. But 40 years later, everyone had a cheap smartphone in their pocket, which is far superior to what he had in his Porsche.
And it's the same with AI. Every five years, compute becomes ten times cheaper - a trend that has continued since the first general-purpose computer was completed in 1941. The basic techniques of modern AI were developed in the last millennium, when compute was still very expensive, but this trend has made them so cheap that AI has been on your smartphone since the 2010s.
The trend will not break in the coming decades (the physical Bremermann limit is still a long way off). An AI data centre that is worth 100 billion today will only be worth 1 billion in ten years' time. Soon, small, cheap computers with increasingly efficient open source AI software will do what the large data centres do today.
So AI will not be controlled by a few big AI utilities. No, everyone will own cheap but powerful and transparent AI that improves their lives in many ways. See the old motto of @nnaisense founded in 2014: ‘AI∀’ or ‘AI For All’. -
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
10 profound answers about the math behind AI
It’s knowledgeable, confident, and behaves human-like in many ways. But it’s not magic that powers AI though; it’s just math and data. My Q&A for @bigthink with @StartsWithABang
bigthink.com/starts-with-a-bang/10-answers-math-artificia… -
Anil Ananthaswamy
@anilananth
Everyone is talking about @deepseek_ai's impact on industry. But another huge impact is the leveling of playing field between academia and industry: if these efficiency numbers bear out, then academia can both use LLMs and study/research them at scale!
-
Anil Ananthaswamy reposted
Big Think
@bigthink
What allows AI to learn?
Ethan Siegel (@StartsWithABang) sits down with Anil Ananthaswamy (@anilananth), author of Why Machines Learn: The Elegant Math Behind Modern AI. -
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
A simple way to appreciate why linear algebra and calculus are crucial for machine learning.
Let's start with what machine learning models do. In most cases, they recognize images or speech, perform do natural language tasks (such as translate from one language to another), and even generate new data (images, speech, and text, for e.g.)
These types of data are discrete. We turn them into vectors, which are essentially sequences of real numbers. So, the input is represented as a vector in some continuous vector space and the output is another vector in a different, continuous vector space.
Why do this? Well, the moment we operate on data that lives in continuous spaces, we can use calculus to do optimization. We'll come to that in a moment.
But for now: the moment you have an input vector that needs to be transformed into an output vector, you are in the realm of linear algebra. Such a transformation can be effected by multiplying a vector by a matrix, to get another vector. In the case of deep neural networks, there are multiple such transformations required to produce the output vector. And there are non-linear transformations--which are at least approximately differentiable--applied to each element of each intermediate vector.
To do all this, one needs to really understand linear algebra.
Why calculus? Well, it comes to asking this question: What does it mean for a machine to learn? Given an input vector, you want an ML model to produce the correct output vector. But how is going to do that? Well, we teach it to do that.
Let's say the model is characterized by some set of parameters.
In the beginning, these parameters are randomly initialized. The model produces an erroneous output for a given input. If you know the correct output (which you do if you have training data), then you can calculate the "loss" between what's expected and what's produced by the model.
If you can characterize this loss as a function of the parameters of the model, then you can use some form of gradient descent to find either a global minimum of the function (in case the function is convex) or an optimal local minimum (which is the case with deep neural networks).
All this requires calculus! But it was made possible because we also chose to use continuous vector spaces into which to embed our data.
That's a whirlwind explanation for the importance of calculus and linear algebra in machine learning.
For more, see WHY MACHINES LEARN
US Edition: www.penguinrandomhouse.com/books/677608/why-machines-lear…
UK Edition:
www.penguin.co.uk/books/446849/why-machines-learn-by-anan… -
Anil Ananthaswamy
@anilananth
A simple way to appreciate why linear algebra and calculus are crucial for machine learning.
Let's start with what machine learning models do. In most cases, they recognize images or speech, perform do natural language tasks (such as translate from one language to another), and even generate new data (images, speech, and text, for e.g.)
These types of data are discrete. We turn them into vectors, which are essentially sequences of real numbers. So, the input is represented as a vector in some continuous vector space and the output is another vector in a different, continuous vector space.
Why do this? Well, the moment we operate on data that lives in continuous spaces, we can use calculus to do optimization. We'll come to that in a moment.
But for now: the moment you have an input vector that needs to be transformed into an output vector, you are in the realm of linear algebra. Such a transformation can be effected by multiplying a vector by a matrix, to get another vector. In the case of deep neural networks, there are multiple such transformations required to produce the output vector. And there are non-linear transformations--which are at least approximately differentiable--applied to each element of each intermediate vector.
To do all this, one needs to really understand linear algebra.
Why calculus? Well, it comes to asking this question: What does it mean for a machine to learn? Given an input vector, you want an ML model to produce the correct output vector. But how is going to do that? Well, we teach it to do that.
Let's say the model is characterized by some set of parameters.
In the beginning, these parameters are randomly initialized. The model produces an erroneous output for a given input. If you know the correct output (which you do if you have training data), then you can calculate the "loss" between what's expected and what's produced by the model.
If you can characterize this loss as a function of the parameters of the model, then you can use some form of gradient descent to find either a global minimum of the function (in case the function is convex) or an optimal local minimum (which is the case with deep neural networks).
All this requires calculus! But it was made possible because we also chose to use continuous vector spaces into which to embed our data.
That's a whirlwind explanation for the importance of calculus and linear algebra in machine learning.
For more, see WHY MACHINES LEARN
US Edition: www.penguinrandomhouse.com/books/677608/why-machines-lear…
UK Edition:
www.penguin.co.uk/books/446849/why-machines-learn-by-anan… -
Anil Ananthaswamy
@anilananth
Never thought I'd see "WTF" in a review of WHY MACHINES LEARN :-)
'This book is the ultimate explainer for anyone who's ever asked, “WTF even is AI?”'
WHY MACHINES LEARN wins the Artificiality Book Award 2024, alongside @Sara_Imari's Life As No One Knows It, @random_walker and @sayashk's AI Snake Oil, and others.
Here's the review:
First off, massive respect to Anil for having the guts to write a book about AI when it evolves faster than you can say “gradient descent” is no small feat. Add that doing it in a way that makes math not just understandable but fascinating is next-level.
This book is the ultimate explainer for anyone who's ever asked, “WTF even is AI?” Anil wraps equations in stories that make the math feel alive. For example, he dives into how linear regression—yes, the thing you learned in stats—forms the backbone of AI’s ability to make predictions. He connects that to how machines were first trained to recognize patterns, from simple lines on a graph to complex decision-making.
What I love most is how he threads history into the equations. You get why these methods matter, how they were discovered, and why they’ve stuck around. I felt like I was part of the journey, not just staring at some abstract formula.
If you’re curious about how machines learn but feel like math is a wall you can’t climb, this book is your ladder. Highly recommended.
www.artificiality.world/artificiality-book-awards-2024/ -
Anil Ananthaswamy reposted
Parul Pandey
@pandeyparul
It was also great to listen to Anil's podcast on @MLStreetTalk and really get a feel for the thought and effort that went into writing the book.
www.youtube.com/watch?v=URtF_UHYBSo
Anil Ananthaswamy
@anilananth
AL/ML: It's Not Magic, But It Is Magical
It's been six months since the release of WHY MACHINES LEARN -- and I have been trying to make sense of why the book has struck a chord among those interested in AI and machine learning. After all, it's neither a textbook nor a work of straight-up narrative non-fiction. My worry had been that it'd fall between those two stools!
But that didn't happen, despite the math, and lots of it. As per Stephen Hawking's assertion in A Brief History of Time, every equation should have halved the sales of Why Machines Learn. Given the pages and pages of equations in it, I had been justifiably concerned.
The month before publication, I met 𝐒𝐞𝐚𝐧 𝐂𝐚𝐫𝐫𝐨𝐥𝐥 at the 𝐒𝐚𝐧𝐭𝐚 𝐅𝐞 𝐈𝐧𝐬𝐭𝐢𝐭𝐮𝐭𝐞, @sfiscience, where he's visiting faculty and I was a journalism fellow. I confessed my worries to him about the amount of math in WHY MACHINES LEARN, and he told me, apropos in math-speak: the response is going to be bimodal :-), meaning some are going to hate it, others are going to love it.
Sean would know: His latest books, THE BIGGEST IDEAS series, have plenty of equations. He made it cool to have equations in a book. While it's not at all necessary to have math in a popular science book, when the object is to convey the beauty of the math and ideas contained therein, maybe equations are an integral part of the story.
Sean makes a more eloquent case for it, in a 𝘔𝘪𝘯𝘥𝘴𝘤𝘢𝘱𝘦 podcast episode with guest 𝐂𝐥𝐚𝐮𝐝𝐢𝐚 𝐝𝐞 𝐑𝐚𝐡𝐦:
"Equations are smarter than we are. This is why I think it's worth the effort in a book like 𝐓𝐇𝐄 𝐁𝐈𝐆𝐆𝐄𝐒𝐓 𝐈𝐃𝐄𝐀𝐒 series of talking about the equations. Not that it's the only way to talk about physics, et cetera. [...] I think that all different levels, all different approaches, are interesting and important, but one of those interesting ones is the equation-based one. And the reason why is because, as I said, the equations seem to capture more than we put into them."
Math coupled with story-telling has turned out to be a potent combination. As a professional non-fiction writer, I was always thinking in terms of stories: not only human stories, but also stories about what's happening in mathematical spaces. For example, the k-Nearest Neighbor algorithm--despite its astounding simplicity and power--falls prey to the curse of dimensionality. The end of the chapter on k-NN talks about this curse; it's a beautiful mathematical idea of the strange behavior of vectors in high-dimensional spaces!
The narrative led naturally to a chapter on dimensionality reduction techniques, such as PCA. And just as the reader was getting comfortable with the notion that often you can reduce the dimensionality of data to avoid the curse, I introduced the idea that there are ML techniques that explicitly project low-dimensional data back into high dimensions, even infinite dimensions, without incurring any untoward computational cost. Why on earth would you do that? Enter the story of Support Vector Machines and Kernel Methods!
So, maybe it's these tricks storytelling, combined with the rigor of the math, helped achieve what I was aiming for: getting across the idea that AI/ML--despite its stunning success--is not magic! One can understand it from the ground up, with a little bit of effort.
And yet, as we see with behavior of large language models (LLMs), the effect can be magical.
That's almost the point of WHY MACHINES LEARN: to feel the power of the math so that AI doesn't seem like magic, and simultaneously appreciate the magical consequences of building machines that learn. It's not magic, but it is magical.
As for the book, the response has indeed been bimodal, but with a twist: the peak of the distribution of user reviews/comments over "loved it" is much, much higher than the peak over "hated it." I'm surprised, relieved, and immensely grateful to my readers.
In particular, I'm grateful to early shout-outs that the book received from @maxsbennett, @dibbygibby, @Marcelokauffman, @JosephJacks_ , @BenSManning, @EricTopol, @pandeyparul, @TheTuringPost, @MargRev, @pmddomingos, @daniellenewnham, @gpjt, @MereSophistry, @hernanlira_, @bendiken and so many others!
As always, here's where you can find WHY MACHINES LEARN:
US Edition: www.penguinrandomhouse.com/books/677608/why-machines-lear…
UK Edition:
www.penguinrandomhouse.com/books/677608/why-machines-lear… -
Anil Ananthaswamy
@anilananth
As always, @rao2z has a way with words and speaks his mind in this @MLStreetTalk episode about the secret sauce in OpenAI's o1 models: "We should be looking for secrets of Nature, because Nature won't tell us. But we are now looking for secrets of OpenAI."
-
Anil Ananthaswamy reposted
Domesticated Brain
@rasangarocks
Why Machines Learn: The Elegant Math Behind Modern AI
Link - www.amazon.com/Why-Machines-Learn-Elegant-Behind/dp/05931…
#machinelearning #ml #100daysofcode #coding #ai #deeplearning #dataanalytics #datascience #softwareengineering #programming -
Anil Ananthaswamy
@anilananth
Looking forward to listening to this episode on @MLStreetTalk w/ @rao2z ...
Subbarao Kambhampati (కంభంపాటి సుబ్బారావు)
@rao2z
In this second outing on @MLStreetTalk, I chatted with @ecsquendor about reasoning models like o1/R1; CoT & prompt augmentations; our evaluation of o1; my speculations on how o1 works; RL post training vs. Inference Time scaling; yummy stone soup; and Monty Python.. 👉www.youtube.com/watch?v=2xFTNXK6AzQ
-
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
AL/ML: It's Not Magic, But It Is Magical
It's been six months since the release of WHY MACHINES LEARN -- and I have been trying to make sense of why the book has struck a chord among those interested in AI and machine learning. After all, it's neither a textbook nor a work of straight-up narrative non-fiction. My worry had been that it'd fall between those two stools!
But that didn't happen, despite the math, and lots of it. As per Stephen Hawking's assertion in A Brief History of Time, every equation should have halved the sales of Why Machines Learn. Given the pages and pages of equations in it, I had been justifiably concerned.
The month before publication, I met 𝐒𝐞𝐚𝐧 𝐂𝐚𝐫𝐫𝐨𝐥𝐥 at the 𝐒𝐚𝐧𝐭𝐚 𝐅𝐞 𝐈𝐧𝐬𝐭𝐢𝐭𝐮𝐭𝐞, @sfiscience, where he's visiting faculty and I was a journalism fellow. I confessed my worries to him about the amount of math in WHY MACHINES LEARN, and he told me, apropos in math-speak: the response is going to be bimodal :-), meaning some are going to hate it, others are going to love it.
Sean would know: His latest books, THE BIGGEST IDEAS series, have plenty of equations. He made it cool to have equations in a book. While it's not at all necessary to have math in a popular science book, when the object is to convey the beauty of the math and ideas contained therein, maybe equations are an integral part of the story.
Sean makes a more eloquent case for it, in a 𝘔𝘪𝘯𝘥𝘴𝘤𝘢𝘱𝘦 podcast episode with guest 𝐂𝐥𝐚𝐮𝐝𝐢𝐚 𝐝𝐞 𝐑𝐚𝐡𝐦:
"Equations are smarter than we are. This is why I think it's worth the effort in a book like 𝐓𝐇𝐄 𝐁𝐈𝐆𝐆𝐄𝐒𝐓 𝐈𝐃𝐄𝐀𝐒 series of talking about the equations. Not that it's the only way to talk about physics, et cetera. [...] I think that all different levels, all different approaches, are interesting and important, but one of those interesting ones is the equation-based one. And the reason why is because, as I said, the equations seem to capture more than we put into them."
Math coupled with story-telling has turned out to be a potent combination. As a professional non-fiction writer, I was always thinking in terms of stories: not only human stories, but also stories about what's happening in mathematical spaces. For example, the k-Nearest Neighbor algorithm--despite its astounding simplicity and power--falls prey to the curse of dimensionality. The end of the chapter on k-NN talks about this curse; it's a beautiful mathematical idea of the strange behavior of vectors in high-dimensional spaces!
The narrative led naturally to a chapter on dimensionality reduction techniques, such as PCA. And just as the reader was getting comfortable with the notion that often you can reduce the dimensionality of data to avoid the curse, I introduced the idea that there are ML techniques that explicitly project low-dimensional data back into high dimensions, even infinite dimensions, without incurring any untoward computational cost. Why on earth would you do that? Enter the story of Support Vector Machines and Kernel Methods!
So, maybe it's these tricks storytelling, combined with the rigor of the math, helped achieve what I was aiming for: getting across the idea that AI/ML--despite its stunning success--is not magic! One can understand it from the ground up, with a little bit of effort.
And yet, as we see with behavior of large language models (LLMs), the effect can be magical.
That's almost the point of WHY MACHINES LEARN: to feel the power of the math so that AI doesn't seem like magic, and simultaneously appreciate the magical consequences of building machines that learn. It's not magic, but it is magical.
As for the book, the response has indeed been bimodal, but with a twist: the peak of the distribution of user reviews/comments over "loved it" is much, much higher than the peak over "hated it." I'm surprised, relieved, and immensely grateful to my readers.
In particular, I'm grateful to early shout-outs that the book received from @maxsbennett, @dibbygibby, @Marcelokauffman, @JosephJacks_ , @BenSManning, @EricTopol, @pandeyparul, @TheTuringPost, @MargRev, @pmddomingos, @daniellenewnham, @gpjt, @MereSophistry, @hernanlira_, @bendiken and so many others!
As always, here's where you can find WHY MACHINES LEARN:
US Edition: www.penguinrandomhouse.com/books/677608/why-machines-lear…
UK Edition:
www.penguinrandomhouse.com/books/677608/why-machines-lear… -
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
Animal intelligence can teach us about generalization, said @UCBerkeley's Bruno Olshausen, in a talk at the Simons Institute. "The very objective function of evolution was the ability to behave effectively in unknown future settings." www.youtube.com/watch?v=jnMlunS06Nk
-
Anil Ananthaswamy
@anilananth
AL/ML: It's Not Magic, But It Is Magical
It's been six months since the release of WHY MACHINES LEARN -- and I have been trying to make sense of why the book has struck a chord among those interested in AI and machine learning. After all, it's neither a textbook nor a work of straight-up narrative non-fiction. My worry had been that it'd fall between those two stools!
But that didn't happen, despite the math, and lots of it. As per Stephen Hawking's assertion in A Brief History of Time, every equation should have halved the sales of Why Machines Learn. Given the pages and pages of equations in it, I had been justifiably concerned.
The month before publication, I met 𝐒𝐞𝐚𝐧 𝐂𝐚𝐫𝐫𝐨𝐥𝐥 at the 𝐒𝐚𝐧𝐭𝐚 𝐅𝐞 𝐈𝐧𝐬𝐭𝐢𝐭𝐮𝐭𝐞, @sfiscience, where he's visiting faculty and I was a journalism fellow. I confessed my worries to him about the amount of math in WHY MACHINES LEARN, and he told me, apropos in math-speak: the response is going to be bimodal :-), meaning some are going to hate it, others are going to love it.
Sean would know: His latest books, THE BIGGEST IDEAS series, have plenty of equations. He made it cool to have equations in a book. While it's not at all necessary to have math in a popular science book, when the object is to convey the beauty of the math and ideas contained therein, maybe equations are an integral part of the story.
Sean makes a more eloquent case for it, in a 𝘔𝘪𝘯𝘥𝘴𝘤𝘢𝘱𝘦 podcast episode with guest 𝐂𝐥𝐚𝐮𝐝𝐢𝐚 𝐝𝐞 𝐑𝐚𝐡𝐦:
"Equations are smarter than we are. This is why I think it's worth the effort in a book like 𝐓𝐇𝐄 𝐁𝐈𝐆𝐆𝐄𝐒𝐓 𝐈𝐃𝐄𝐀𝐒 series of talking about the equations. Not that it's the only way to talk about physics, et cetera. [...] I think that all different levels, all different approaches, are interesting and important, but one of those interesting ones is the equation-based one. And the reason why is because, as I said, the equations seem to capture more than we put into them."
Math coupled with story-telling has turned out to be a potent combination. As a professional non-fiction writer, I was always thinking in terms of stories: not only human stories, but also stories about what's happening in mathematical spaces. For example, the k-Nearest Neighbor algorithm--despite its astounding simplicity and power--falls prey to the curse of dimensionality. The end of the chapter on k-NN talks about this curse; it's a beautiful mathematical idea of the strange behavior of vectors in high-dimensional spaces!
The narrative led naturally to a chapter on dimensionality reduction techniques, such as PCA. And just as the reader was getting comfortable with the notion that often you can reduce the dimensionality of data to avoid the curse, I introduced the idea that there are ML techniques that explicitly project low-dimensional data back into high dimensions, even infinite dimensions, without incurring any untoward computational cost. Why on earth would you do that? Enter the story of Support Vector Machines and Kernel Methods!
So, maybe it's these tricks storytelling, combined with the rigor of the math, helped achieve what I was aiming for: getting across the idea that AI/ML--despite its stunning success--is not magic! One can understand it from the ground up, with a little bit of effort.
And yet, as we see with behavior of large language models (LLMs), the effect can be magical.
That's almost the point of WHY MACHINES LEARN: to feel the power of the math so that AI doesn't seem like magic, and simultaneously appreciate the magical consequences of building machines that learn. It's not magic, but it is magical.
As for the book, the response has indeed been bimodal, but with a twist: the peak of the distribution of user reviews/comments over "loved it" is much, much higher than the peak over "hated it." I'm surprised, relieved, and immensely grateful to my readers.
In particular, I'm grateful to early shout-outs that the book received from @maxsbennett, @dibbygibby, @Marcelokauffman, @JosephJacks_ , @BenSManning, @EricTopol, @pandeyparul, @TheTuringPost, @MargRev, @pmddomingos, @daniellenewnham, @gpjt, @MereSophistry, @hernanlira_, @bendiken and so many others!
As always, here's where you can find WHY MACHINES LEARN:
US Edition: www.penguinrandomhouse.com/books/677608/why-machines-lear…
UK Edition:
www.penguinrandomhouse.com/books/677608/why-machines-lear… -
Anil Ananthaswamy
@anilananth
10 profound answers about the math behind AI
It’s knowledgeable, confident, and behaves human-like in many ways. But it’s not magic that powers AI though; it’s just math and data. My Q&A for @bigthink with @StartsWithABang
bigthink.com/starts-with-a-bang/10-answers-math-artificia… -
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
Wondering where LLMs are headed? Watch @OpenAI's @SebastienBubeck and @Yale's @RTomMcCoy take opposing positions on whether scaling up LLMs is sufficient to generate new proof techniques to resolve major mathematical conjectures, such as P != NP. www.youtube.com/watch?v=H3TnTxVKIOQ
-
Anil Ananthaswamy
@anilananth
It was a pleasure to speak with Mukesh Bansal, @mukeshbansal06, about WHY MACHINES LEARN for his SparX podcast. Thrilled to see such in-depth conversations on science and tech happening on Indian media. Thanks for the engaging chat, Mukesh, and for making me appear more learned than I really am :)
www.youtube.com/watch?v=soQRRFwrfpQ -
Anil Ananthaswamy reposted
Bahram Shakerin
@BahramShakerin
The Cold Truth About Quantum Mechanics
Raphael Bousso
www.youtube.com/watch?v=71q2q9sGkX8 -
Anil Ananthaswamy reposted
Domesticated Brain
@rasangarocks
Why Machines Learn: The Elegant Maths Behind Modern AI
Link - www.amazon.com/Why-Machines-Learn-Elegant-Behind/dp/05931…
#machinelearning #deeplearning #maths #100daysofcode #dataanalytics #datascientists #ai #softwareengineering #programming -
Anil Ananthaswamy reposted
BURKOV
@burkov
Just received it. The quality is perfect! White thick paper, juicy colors, pleasure to look at and learn from.
-
Anil Ananthaswamy reposted
dylan
@dxlantxch
BOOK RECS TO ADD??
i would like to construct a disgustingly large collection of books.
like i love intuitive math books like “why machines learn” but also creative books like “on writing” but also emotional books like “a thousand splendid suns” 😩 -
Anil Ananthaswamy reposted
Balaraman Ravindran
@ravi_iitm
Honoured and humbled to be elected an @RealAAAI Fellow this year, along with an illustrious cohort. Congratulations to all! Doubly happy to be receiving this recognition along with my good friend @Sriraam_UTD!! Thanks to my nominators, supporters, students, collaborators and, above all, my family. aaai.org/about-aaai/aaai-awards/the-aaai-fellows-program/…
@WSAI_IITM @iitmadras -
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
This long chat on WHY MACHINES LEARN and the basic math of ML with @MLStreetTalk's Tim Scarfe (@ecsquendor) remains my favorite...
www.youtube.com/watch?v=URtF_UHYBSo -
Anil Ananthaswamy
@anilananth
This long chat on WHY MACHINES LEARN and the basic math of ML with @MLStreetTalk's Tim Scarfe (@ecsquendor) remains my favorite...
www.youtube.com/watch?v=URtF_UHYBSo -
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
Never ever imagined that WHY MACHINES LEARN would be recommended as a "Red Pill" for AI/ML. A data scientist based in the Netherlands posted an evocative write-up about WHY MACHINES LEARN, ending his post with: "Secretly, I hope that this book will act as a loving red pill. Even if it means that sometimes you have to open your math book from high school." Quite the cheerleader for not shying away from the maths!
www.linkedin.com/posts/anil-ananthaswamy_wiskunde-als-rod… -
Anil Ananthaswamy
@anilananth
Never ever imagined that WHY MACHINES LEARN would be recommended as a "Red Pill" for AI/ML. A data scientist based in the Netherlands posted an evocative write-up about WHY MACHINES LEARN, ending his post with: "Secretly, I hope that this book will act as a loving red pill. Even if it means that sometimes you have to open your math book from high school." Quite the cheerleader for not shying away from the maths!
www.linkedin.com/posts/anil-ananthaswamy_wiskunde-als-rod… -
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
Machine Learning’s Debt to Nearest Neighbors
The simplicity of the k-Nearest Neighbor rule—one of the staples of machine learning—belies its rich history and enormous impact. Here’s why the k-NN rule can help us appreciate some key conceptual underpinnings of ML.
First, what is the k-NN rule? Simply put, an instance of unlabeled data can be categorized as being of one class or the other by comparing it to its nearest neighbor(s) in some possibly high-dimensional vector space (where each instance of data is a vector). Birds of a feather flock together. As Peter Hart, the co-inventor of the Cover-Hart k-NN algorithm in the 1960s, told me: “That’s the nearest neighbor rule. That’s the caveman intuition: If they look alike, they probably are alike.”
This intuition was also recognized in the 11th century by the Alhazen, a Muslim Arab mathematician, astronomer, and physicist, as he speculated on how humans recognize objects. Then, there’s the extraordinary story of the British epidemiologist John Snow, who used techniques that presage the k-NN rule, to figure out the epicenter of London’s deadly 1854 cholera outbreak; these mathematical ideas were formalized by Ukrainian mathematician Georgy Voronoi, who gave us his eponymous Voronoi cells. Then, in 1951, the amazing statistician Evelyn Fix, while working ceaselessly at UC Berkeley to help the US war efforts, teamed up with Joseph Hodges to first elucidate the nearest neighbor rule.
Then, in the 1960s, Peter Hart, working on his PhD with advisor Thomas Cover at Stanford (who was himself just beginning his academic career), came up with some rigorous math to show that the k-NN rule (where k refers to the number of neighbors in the vector space), in the limit of large k and small k/n (n=number of data samples), begins approaching the best possible ML algorithm: the Bayes Optimal classifier. And even at its worst, the k-NN rule is not that bad, despite its algorithmic simplicity.
But the k-NN rule would also show up the limits of ML algorithms when the dimensionality of data (i.e. the number of elements in the vector being used to represent an instance of data) begins growing. Something called the “curse of dimensionality” takes over! The k-NN rule is a brilliant way to understand this curse—which then leads us to appreciate dimensionality reduction methods, such as Principal Component Analysis (PCA). It also helps us appreciate the counter-intuitive brilliance of kernel methods, which deliberately project data into higher dimensions. How do they avoid the curse? As do deep neural networks, which also operate in extremely high dimensions. How so?
This historical narrative, with inputs from Peter Hart himself, along with the relevant math, and intuitive visualizations can be found in WHY MACHINES LEARN.
US: www.penguinrandomhouse.com/books/677608/why-machines-lear…
UK: www.penguin.co.uk/books/446849/why-machines-learn-by-anan… -
Anil Ananthaswamy reposted
Boaz Barak
@boazbaraktcs
Moving between the worlds of East coast academics and SF industry is jarring in terms of the predictions of how AI will change the world in next ~5 years.
I think East coasters greatly underestimate the magnitude of change, and over index on temporary limitations of current systems.
On the other hand folks in the Bay Area are underestimating the friction and time lag in translating strong capabilities to wide scale economic impact. -
Anil Ananthaswamy
@anilananth
Machine Learning’s Debt to Nearest Neighbors
The simplicity of the k-Nearest Neighbor rule—one of the staples of machine learning—belies its rich history and enormous impact. Here’s why the k-NN rule can help us appreciate some key conceptual underpinnings of ML.
First, what is the k-NN rule? Simply put, an instance of unlabeled data can be categorized as being of one class or the other by comparing it to its nearest neighbor(s) in some possibly high-dimensional vector space (where each instance of data is a vector). Birds of a feather flock together. As Peter Hart, the co-inventor of the Cover-Hart k-NN algorithm in the 1960s, told me: “That’s the nearest neighbor rule. That’s the caveman intuition: If they look alike, they probably are alike.”
This intuition was also recognized in the 11th century by the Alhazen, a Muslim Arab mathematician, astronomer, and physicist, as he speculated on how humans recognize objects. Then, there’s the extraordinary story of the British epidemiologist John Snow, who used techniques that presage the k-NN rule, to figure out the epicenter of London’s deadly 1854 cholera outbreak; these mathematical ideas were formalized by Ukrainian mathematician Georgy Voronoi, who gave us his eponymous Voronoi cells. Then, in 1951, the amazing statistician Evelyn Fix, while working ceaselessly at UC Berkeley to help the US war efforts, teamed up with Joseph Hodges to first elucidate the nearest neighbor rule.
Then, in the 1960s, Peter Hart, working on his PhD with advisor Thomas Cover at Stanford (who was himself just beginning his academic career), came up with some rigorous math to show that the k-NN rule (where k refers to the number of neighbors in the vector space), in the limit of large k and small k/n (n=number of data samples), begins approaching the best possible ML algorithm: the Bayes Optimal classifier. And even at its worst, the k-NN rule is not that bad, despite its algorithmic simplicity.
But the k-NN rule would also show up the limits of ML algorithms when the dimensionality of data (i.e. the number of elements in the vector being used to represent an instance of data) begins growing. Something called the “curse of dimensionality” takes over! The k-NN rule is a brilliant way to understand this curse—which then leads us to appreciate dimensionality reduction methods, such as Principal Component Analysis (PCA). It also helps us appreciate the counter-intuitive brilliance of kernel methods, which deliberately project data into higher dimensions. How do they avoid the curse? As do deep neural networks, which also operate in extremely high dimensions. How so?
This historical narrative, with inputs from Peter Hart himself, along with the relevant math, and intuitive visualizations can be found in WHY MACHINES LEARN.
US: www.penguinrandomhouse.com/books/677608/why-machines-lear…
UK: www.penguin.co.uk/books/446849/why-machines-learn-by-anan… -
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
The Importance of Being Vectors
We hear over and over that to do machine learning is to do linear algebra. Which, in turn, has to do with learning about vectors and matrices. But why? What’s so amazing about this particular mathematical discipline as it pertains to ML?
Here, from a bird’s eye-view, are 7 key reasons why understanding vectors & matrices and their manipulations is so integral to making sense of machine learning:
1. Many of today’s ML algorithms first turn data—whether they are images, audio or text—into vectors. The manner of encoding is unique to each type of data, and even for a given type of data, such as text, there might be multiple ways of encoding the information in the form of vectors. A vector is a sequence of numbers (the ordering is important), and the number of elements of a vector gives you the dimensionality of the vector space in which the vector resides. So, 3 numbers equate to a 3D space, 10 numbers to 10D space, and so on, with each element varying along its respective axis in that space.
2. Encoding data as vectors makes it possible to use measures of similarity and dissimilarity to compare instances of data (a property exploited by, say, the k-Nearest Neighbor algorithm).
3. In vector spaces, proper encoding can result in nearby vectors having similar semantic meanings, and thus allowing for new kinds of manipulations. For example, the well-known word2vec model—which uses a neural network to embed words in a vector space—shows that the “distance” between the vectors for “man” and “woman” is comparable to the distance between the vectors for “king” and “queen”. So, if you take the vector for “king”, subtract “man” and add “woman”, you would be very near the vector for “queen”.
4. Many ML algorithms, and deep neural networks in particular, can be thought of as algorithms that transform input vectors into output vectors, via matrix-vector manipulations (with the elements of such manipulations often times being passed through non-linear functions, in the case of neural networks).
5. Converting discrete data (such as images and text) into vectors embeds them in a continuous space. This has numerous advantages. Functions in vector spaces are continuous and differentiable, making them amenable for use in optimization techniques such as gradient descent, which are used to train neural networks and other ML models.
6. Also, embedding discrete data in continuous vector spaces lets the ML algorithm access vectors that weren’t in the training data, making it possible to generalize to new (but similar) data, and also to generate new (but similar) data.
7. And when it comes to neural networks, it’s the mathematics of vector spaces that’s key to proving that neural networks with at least one hidden layer, given enough artificial neurons, are universal function approximators: they can approximate any desired function to convert an input vector to an output vector.
For all this and more, including the rich history of the use of vectors in machine learning algorithms, please see WHY MACHINES LEARN.
US Edition: www.penguinrandomhouse.com/books/677608/why-machines-lear…
UK Edition: www.penguin.co.uk/books/446849/why-machines-learn-by-anan… -
Anil Ananthaswamy reposted
Adam Visokay
@avisokay
Really enjoyed reading this over my winter break!
Anil Ananthaswamy
@anilananth
Machine Learning is not just (glorified) statistics! Sure, in its most basic form, ML doesn't seem that far removed from fancy statistics... But ML is its own thing!
Steve Brunton, @eigensteve of the University of Washington, in his fabulous short lecture series on probability and statistics (look for it on YouTube), gives these clear-eyed definitions:
Probability: Assuming you have a known probability distribution that describes your data (say, a Gaussian), probability theory allows you to say something about samples of data you might observe in the future. So, one can calculate the probability that a random variable, X, takes on some value, given the parameter theta that describes the probability distribution (theta for a gaussian would be the mean and variance).
Statistics: It's the flip side of the above. Now, we have in hand the data or samples. We are trying to say something about the probability distribution that best models the data; or given the data we want to say something about the probability of the parameter theta of the distribution
Machine Learning / Deep Learning: Sometimes, we have the data, but the underlying distribution is unknown, or impossible to characterize analytically, using parameters specified by some theta. ML is used to learn the (empirical) distribution by examining the data.
This is the point at which someone will say, but, that is just statistics! Here's why it's not:
Take the problem of learning the probability distribution over the millions of images of the natural world. And not just that: once you have learned the distribution, you have to sample from it, and generate new data that looks like an image from the original dataset. Try doing this with standard statistical methods in some tractable way. It's near impossible.
But diffusion models, heavily influenced by the physics of non-equilibrium thermodynamics, do exactly that. That's deep learning at its best.
Also, take large language models (LLMs). Of course, LLMs learn the statistics of human written language from an enormous corpus of training data, and generate new text by estimating and then sampling from conditional probability distributions, given some input text. Again, while it sounds like it's just statistics, standard statistical methods could not pull off what an LLM can.
The chapter on probability and statistics in WHY MACHINES LEARN was one of the most difficult to write (it involved getting across the basics of two large fields of math and tying them to ML, all in the space of 30 pages). Nearly killed me. But it was also one of the most satisfying chapters to write, not least because I got to understand so much while trying to make sense of these issues.
More here: US: penguinrandomhouse.com/books/677608/why-machines-learn-by…
UK: penguin.co.uk/books/446849/why-machines-learn-by-ananthas… -
Anil Ananthaswamy reposted
Chip Huyen
@chipro
Finally got my copy! “AI Engineering” is officially out 🙏 🎉
-
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
To see WHY MACHINES LEARN mentioned alongside lecture videos by @AndrewYNg and @karpathy was not on my bingo card! I learned my ML from these experts (and others) and owe them a huge debt of gratitude. www.youtube.com/watch?v=_xIwjmCH6D4 @BorisMeinardus
-
Anil Ananthaswamy
@anilananth
The Importance of Being Vectors
We hear over and over that to do machine learning is to do linear algebra. Which, in turn, has to do with learning about vectors and matrices. But why? What’s so amazing about this particular mathematical discipline as it pertains to ML?
Here, from a bird’s eye-view, are 7 key reasons why understanding vectors & matrices and their manipulations is so integral to making sense of machine learning:
1. Many of today’s ML algorithms first turn data—whether they are images, audio or text—into vectors. The manner of encoding is unique to each type of data, and even for a given type of data, such as text, there might be multiple ways of encoding the information in the form of vectors. A vector is a sequence of numbers (the ordering is important), and the number of elements of a vector gives you the dimensionality of the vector space in which the vector resides. So, 3 numbers equate to a 3D space, 10 numbers to 10D space, and so on, with each element varying along its respective axis in that space.
2. Encoding data as vectors makes it possible to use measures of similarity and dissimilarity to compare instances of data (a property exploited by, say, the k-Nearest Neighbor algorithm).
3. In vector spaces, proper encoding can result in nearby vectors having similar semantic meanings, and thus allowing for new kinds of manipulations. For example, the well-known word2vec model—which uses a neural network to embed words in a vector space—shows that the “distance” between the vectors for “man” and “woman” is comparable to the distance between the vectors for “king” and “queen”. So, if you take the vector for “king”, subtract “man” and add “woman”, you would be very near the vector for “queen”.
4. Many ML algorithms, and deep neural networks in particular, can be thought of as algorithms that transform input vectors into output vectors, via matrix-vector manipulations (with the elements of such manipulations often times being passed through non-linear functions, in the case of neural networks).
5. Converting discrete data (such as images and text) into vectors embeds them in a continuous space. This has numerous advantages. Functions in vector spaces are continuous and differentiable, making them amenable for use in optimization techniques such as gradient descent, which are used to train neural networks and other ML models.
6. Also, embedding discrete data in continuous vector spaces lets the ML algorithm access vectors that weren’t in the training data, making it possible to generalize to new (but similar) data, and also to generate new (but similar) data.
7. And when it comes to neural networks, it’s the mathematics of vector spaces that’s key to proving that neural networks with at least one hidden layer, given enough artificial neurons, are universal function approximators: they can approximate any desired function to convert an input vector to an output vector.
For all this and more, including the rich history of the use of vectors in machine learning algorithms, please see WHY MACHINES LEARN.
US Edition: www.penguinrandomhouse.com/books/677608/why-machines-lear…
UK Edition: www.penguin.co.uk/books/446849/why-machines-learn-by-anan… -
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
Machine Learning is not just (glorified) statistics! Sure, in its most basic form, ML doesn't seem that far removed from fancy statistics... But ML is its own thing!
Steve Brunton, @eigensteve of the University of Washington, in his fabulous short lecture series on probability and statistics (look for it on YouTube), gives these clear-eyed definitions:
Probability: Assuming you have a known probability distribution that describes your data (say, a Gaussian), probability theory allows you to say something about samples of data you might observe in the future. So, one can calculate the probability that a random variable, X, takes on some value, given the parameter theta that describes the probability distribution (theta for a gaussian would be the mean and variance).
Statistics: It's the flip side of the above. Now, we have in hand the data or samples. We are trying to say something about the probability distribution that best models the data; or given the data we want to say something about the probability of the parameter theta of the distribution
Machine Learning / Deep Learning: Sometimes, we have the data, but the underlying distribution is unknown, or impossible to characterize analytically, using parameters specified by some theta. ML is used to learn the (empirical) distribution by examining the data.
This is the point at which someone will say, but, that is just statistics! Here's why it's not:
Take the problem of learning the probability distribution over the millions of images of the natural world. And not just that: once you have learned the distribution, you have to sample from it, and generate new data that looks like an image from the original dataset. Try doing this with standard statistical methods in some tractable way. It's near impossible.
But diffusion models, heavily influenced by the physics of non-equilibrium thermodynamics, do exactly that. That's deep learning at its best.
Also, take large language models (LLMs). Of course, LLMs learn the statistics of human written language from an enormous corpus of training data, and generate new text by estimating and then sampling from conditional probability distributions, given some input text. Again, while it sounds like it's just statistics, standard statistical methods could not pull off what an LLM can.
The chapter on probability and statistics in WHY MACHINES LEARN was one of the most difficult to write (it involved getting across the basics of two large fields of math and tying them to ML, all in the space of 30 pages). Nearly killed me. But it was also one of the most satisfying chapters to write, not least because I got to understand so much while trying to make sense of these issues.
More here: US: penguinrandomhouse.com/books/677608/why-machines-learn-by…
UK: penguin.co.uk/books/446849/why-machines-learn-by-ananthas… -
Anil Ananthaswamy
@anilananth
I wish I could read the title/subtitle in the Czech and Italian translations, but I suspect they simply say: WHY MACHINES LEARN: The Elegant Math Behind Modern AI
www.albatrosmedia.sk/tituly/25862775/proc-se-stroje-uci/
www.apogeonline.com/libri/perche-le-macchine-imparano-ani… -
Anil Ananthaswamy reposted
Edward Fariñas
@edwardfarinas
Current read:
Why Machines Learn by @anilananth -
Anil Ananthaswamy reposted
Prof. Anima Anandkumar
@AnimaAnandkumar
I agree with Jensen that the next wave of AI is physical understanding, especially the ability to model and simulate physics is essential. Watch my TED talk go.ted.com/animaanandkumar #ces #nvidia #nvidiacosmos
-
Anil Ananthaswamy reposted
Melanie Mitchell
@MelMitchell1
From my colleagues Kyle Harper & David Wolpert at @sfiscience:
Position for two-year postdoc with background in computer science interested in working with social scientists/historians to study social evolution through computational lens. Details here: santafe.edu/about/jobs/postdoc-comp-complexity -
Anil Ananthaswamy reposted
Jeff Clune
@jeffclune
I greatly enjoyed this interview on @MLStreetTalk It covers a wide range of topics I am interested in (open-endedness, AI-GAs, AI safety, etc.), includes my scientific origin story, and has great visuals edited in. Thanks Tim Scarfe!
www.youtube.com/watch?v=mw5WIDGRLnA -
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
Key Resources for Deeper Dives into AI/ML
A number of people have asked about the lectures/courses on AI/ML that I found useful while researching and writing WHY MACHINES LEARN. Here's a redo of an earlier post, with some additions.
𝐌𝐚𝐜𝐡𝐢𝐧𝐞 𝐋𝐞𝐚𝐫𝐧𝐢𝐧𝐠 𝐨𝐯𝐞𝐫𝐚𝐥𝐥
My absolute favorite: 𝐊𝐢𝐥𝐢𝐚𝐧 𝐖𝐞𝐢𝐧𝐛𝐞𝐫𝐠𝐞𝐫'𝐬 𝟐𝟎𝟏𝟖 𝐜𝐥𝐚𝐬𝐬 𝐨𝐧 𝐦𝐚𝐜𝐡𝐢𝐧𝐞 𝐥𝐞𝐚𝐫𝐧𝐢𝐧𝐠 𝐚𝐭 𝐂𝐨𝐫𝐧𝐞𝐥𝐥: www.youtube.com/watch?v=MrLPzBxG95I
Weinberger is a masterful teacher!
Not too far behind: Andrew Ng's student 𝐀𝐧𝐚𝐧𝐝 𝐀𝐯𝐚𝐭𝐢 teaching an ML course in the summer of 2019.
www.youtube.com/watch?v=KzH1ovd4Ots
His first few lectures setting up the math of ML are gems. And, of course, lectures by 𝐀𝐧𝐝𝐫𝐞𝐰 𝐍𝐠 himself. There are so many... Just Google.
For 𝐋𝐢𝐧𝐞𝐚𝐫 𝐀𝐥𝐠𝐞𝐛𝐫𝐚:
The wonderful 𝐆𝐢𝐥𝐛𝐞𝐫𝐭 𝐒𝐭𝐫𝐚𝐧𝐠 (who was teaching this well before linear algebra became cool): www.youtube.com/watch?v=ZK3O402wf1c
And this engaging course by Trevor Bazett: www.youtube.com/playlist?list=PLHXZ9OQGMqxfUl0tcqPNTJsb7R…
𝐅𝐨𝐫 𝐃𝐚𝐭𝐚 𝐒𝐜𝐢𝐞𝐧𝐜𝐞 𝐦𝐞𝐭𝐡𝐨𝐝𝐬 𝐚𝐧𝐝 𝐜𝐨𝐧𝐜𝐞𝐩𝐭𝐬 𝐁𝐢𝐧𝐠 𝐁𝐫𝐮𝐧𝐭𝐨𝐧, 𝐒𝐭𝐞𝐯𝐞 𝐁𝐫𝐮𝐧𝐭𝐨𝐧 𝐚𝐧𝐝 𝐍𝐚𝐭𝐡𝐚𝐧 𝐊𝐮𝐭𝐳, 𝐨𝐟 𝐭𝐡𝐞 𝐔𝐧𝐢𝐯𝐞𝐫𝐬𝐢𝐭𝐲 𝐨𝐟 𝐖𝐚𝐬𝐡𝐢𝐧𝐠𝐭𝐨𝐧, 𝐒𝐞𝐚𝐭𝐭𝐥𝐞: youtube.com/playlist?list=PLMrJAkhIeNNQz4BMoGSsN8cbt8pHlokhV
Search for other videos by these three. Many gems to be found.
For 𝐏𝐫𝐨𝐛𝐚𝐛𝐢𝐥𝐢𝐭y and Statistics:
A relatively recent addition to the canon of amazing courses, by @eigensteve (Steve Brunton, professor at University of Washington) on probability & statistics. I wish these had been available when I was learning: www.youtube.com/watch?v=sQqniayndb4
A quirky set of bite-sized videos on key concepts in probability theory by 𝐑𝐢𝐜𝐡 𝐑𝐚𝐝𝐤𝐞, 𝐩𝐫𝐨𝐟𝐞𝐬𝐬𝐨𝐫 𝐚𝐭 𝐑𝐏𝐈, 𝐓𝐫𝐨𝐲, 𝐍𝐘. www.youtube.com/watch?v=jdoLvrfkG3Y
If you want a masterclass on 𝐒𝐮𝐩𝐩𝐨𝐫𝐭 𝐕𝐞𝐜𝐭𝐨𝐫 𝐌𝐚𝐜𝐡𝐢𝐧𝐞𝐬, there's none better than 𝐏𝐚𝐭𝐫𝐢𝐜𝐤 𝐖𝐢𝐧𝐬𝐭𝐨𝐧'𝐬 𝐌𝐈𝐓 lecture: www.youtube.com/watch?v=_PwhiWxHK8o
(Winston credits only Vladimir Vapnik for SVMs, without any mention of Isabelle Guyon and Bernhard Boser; it's a massive oversight. WHY MACHINES LEARN corrects the narrative and tells the story of how SVMs actually unfolded).
Of course, watch the other lectures in Winston's course for a general overview of ideas/algorithms in AI/ML. He's a mesmerizing teacher.
If you can persist with these lectures--and then keep dipping into the never-ending resource that is YouTube --you'll be well set to get under the mathematical skin of ML.
Of course, if you want a gentler intro to the math and the conceptual ideas, alongside some of the sociological and historical narratives surrounding ML, there's WHY MACHINES LEARN
US Edition: www.penguinrandomhouse.com/books/677608/why-machines-lear…
US Edition: www.penguin.co.uk/books/446849/why-machines-learn-by-anan…
(Note these are same book, different covers) -
Anil Ananthaswamy
@anilananth
Thank you :)
Fran Ramírez
@cybercaronte
I’m on it, at the moment a great book ;) congrats!!
Anil Ananthaswamy
@anilananth
Key Resources for Deeper Dives into AI/ML
A number of people have asked about the lectures/courses on AI/ML that I found useful while researching and writing WHY MACHINES LEARN. Here's a redo of an earlier post, with some additions.
𝐌𝐚𝐜𝐡𝐢𝐧𝐞 𝐋𝐞𝐚𝐫𝐧𝐢𝐧𝐠 𝐨𝐯𝐞𝐫𝐚𝐥𝐥
My absolute favorite: 𝐊𝐢𝐥𝐢𝐚𝐧 𝐖𝐞𝐢𝐧𝐛𝐞𝐫𝐠𝐞𝐫'𝐬 𝟐𝟎𝟏𝟖 𝐜𝐥𝐚𝐬𝐬 𝐨𝐧 𝐦𝐚𝐜𝐡𝐢𝐧𝐞 𝐥𝐞𝐚𝐫𝐧𝐢𝐧𝐠 𝐚𝐭 𝐂𝐨𝐫𝐧𝐞𝐥𝐥: www.youtube.com/watch?v=MrLPzBxG95I
Weinberger is a masterful teacher!
Not too far behind: Andrew Ng's student 𝐀𝐧𝐚𝐧𝐝 𝐀𝐯𝐚𝐭𝐢 teaching an ML course in the summer of 2019.
www.youtube.com/watch?v=KzH1ovd4Ots
His first few lectures setting up the math of ML are gems. And, of course, lectures by 𝐀𝐧𝐝𝐫𝐞𝐰 𝐍𝐠 himself. There are so many... Just Google.
For 𝐋𝐢𝐧𝐞𝐚𝐫 𝐀𝐥𝐠𝐞𝐛𝐫𝐚:
The wonderful 𝐆𝐢𝐥𝐛𝐞𝐫𝐭 𝐒𝐭𝐫𝐚𝐧𝐠 (who was teaching this well before linear algebra became cool): www.youtube.com/watch?v=ZK3O402wf1c
And this engaging course by Trevor Bazett: www.youtube.com/playlist?list=PLHXZ9OQGMqxfUl0tcqPNTJsb7R…
𝐅𝐨𝐫 𝐃𝐚𝐭𝐚 𝐒𝐜𝐢𝐞𝐧𝐜𝐞 𝐦𝐞𝐭𝐡𝐨𝐝𝐬 𝐚𝐧𝐝 𝐜𝐨𝐧𝐜𝐞𝐩𝐭𝐬 𝐁𝐢𝐧𝐠 𝐁𝐫𝐮𝐧𝐭𝐨𝐧, 𝐒𝐭𝐞𝐯𝐞 𝐁𝐫𝐮𝐧𝐭𝐨𝐧 𝐚𝐧𝐝 𝐍𝐚𝐭𝐡𝐚𝐧 𝐊𝐮𝐭𝐳, 𝐨𝐟 𝐭𝐡𝐞 𝐔𝐧𝐢𝐯𝐞𝐫𝐬𝐢𝐭𝐲 𝐨𝐟 𝐖𝐚𝐬𝐡𝐢𝐧𝐠𝐭𝐨𝐧, 𝐒𝐞𝐚𝐭𝐭𝐥𝐞: youtube.com/playlist?list=PLMrJAkhIeNNQz4BMoGSsN8cbt8pHlokhV
Search for other videos by these three. Many gems to be found.
For 𝐏𝐫𝐨𝐛𝐚𝐛𝐢𝐥𝐢𝐭y and Statistics:
A relatively recent addition to the canon of amazing courses, by @eigensteve (Steve Brunton, professor at University of Washington) on probability & statistics. I wish these had been available when I was learning: www.youtube.com/watch?v=sQqniayndb4
A quirky set of bite-sized videos on key concepts in probability theory by 𝐑𝐢𝐜𝐡 𝐑𝐚𝐝𝐤𝐞, 𝐩𝐫𝐨𝐟𝐞𝐬𝐬𝐨𝐫 𝐚𝐭 𝐑𝐏𝐈, 𝐓𝐫𝐨𝐲, 𝐍𝐘. www.youtube.com/watch?v=jdoLvrfkG3Y
If you want a masterclass on 𝐒𝐮𝐩𝐩𝐨𝐫𝐭 𝐕𝐞𝐜𝐭𝐨𝐫 𝐌𝐚𝐜𝐡𝐢𝐧𝐞𝐬, there's none better than 𝐏𝐚𝐭𝐫𝐢𝐜𝐤 𝐖𝐢𝐧𝐬𝐭𝐨𝐧'𝐬 𝐌𝐈𝐓 lecture: www.youtube.com/watch?v=_PwhiWxHK8o
(Winston credits only Vladimir Vapnik for SVMs, without any mention of Isabelle Guyon and Bernhard Boser; it's a massive oversight. WHY MACHINES LEARN corrects the narrative and tells the story of how SVMs actually unfolded).
Of course, watch the other lectures in Winston's course for a general overview of ideas/algorithms in AI/ML. He's a mesmerizing teacher.
If you can persist with these lectures--and then keep dipping into the never-ending resource that is YouTube --you'll be well set to get under the mathematical skin of ML.
Of course, if you want a gentler intro to the math and the conceptual ideas, alongside some of the sociological and historical narratives surrounding ML, there's WHY MACHINES LEARN
US Edition: www.penguinrandomhouse.com/books/677608/why-machines-lear…
US Edition: www.penguin.co.uk/books/446849/why-machines-learn-by-anan…
(Note these are same book, different covers) -
Anil Ananthaswamy
@anilananth
Key Resources for Deeper Dives into AI/ML
A number of people have asked about the lectures/courses on AI/ML that I found useful while researching and writing WHY MACHINES LEARN. Here's a redo of an earlier post, with some additions.
𝐌𝐚𝐜𝐡𝐢𝐧𝐞 𝐋𝐞𝐚𝐫𝐧𝐢𝐧𝐠 𝐨𝐯𝐞𝐫𝐚𝐥𝐥
My absolute favorite: 𝐊𝐢𝐥𝐢𝐚𝐧 𝐖𝐞𝐢𝐧𝐛𝐞𝐫𝐠𝐞𝐫'𝐬 𝟐𝟎𝟏𝟖 𝐜𝐥𝐚𝐬𝐬 𝐨𝐧 𝐦𝐚𝐜𝐡𝐢𝐧𝐞 𝐥𝐞𝐚𝐫𝐧𝐢𝐧𝐠 𝐚𝐭 𝐂𝐨𝐫𝐧𝐞𝐥𝐥: www.youtube.com/watch?v=MrLPzBxG95I
Weinberger is a masterful teacher!
Not too far behind: Andrew Ng's student 𝐀𝐧𝐚𝐧𝐝 𝐀𝐯𝐚𝐭𝐢 teaching an ML course in the summer of 2019.
www.youtube.com/watch?v=KzH1ovd4Ots
His first few lectures setting up the math of ML are gems. And, of course, lectures by 𝐀𝐧𝐝𝐫𝐞𝐰 𝐍𝐠 himself. There are so many... Just Google.
For 𝐋𝐢𝐧𝐞𝐚𝐫 𝐀𝐥𝐠𝐞𝐛𝐫𝐚:
The wonderful 𝐆𝐢𝐥𝐛𝐞𝐫𝐭 𝐒𝐭𝐫𝐚𝐧𝐠 (who was teaching this well before linear algebra became cool): www.youtube.com/watch?v=ZK3O402wf1c
And this engaging course by Trevor Bazett: www.youtube.com/playlist?list=PLHXZ9OQGMqxfUl0tcqPNTJsb7R…
𝐅𝐨𝐫 𝐃𝐚𝐭𝐚 𝐒𝐜𝐢𝐞𝐧𝐜𝐞 𝐦𝐞𝐭𝐡𝐨𝐝𝐬 𝐚𝐧𝐝 𝐜𝐨𝐧𝐜𝐞𝐩𝐭𝐬 𝐁𝐢𝐧𝐠 𝐁𝐫𝐮𝐧𝐭𝐨𝐧, 𝐒𝐭𝐞𝐯𝐞 𝐁𝐫𝐮𝐧𝐭𝐨𝐧 𝐚𝐧𝐝 𝐍𝐚𝐭𝐡𝐚𝐧 𝐊𝐮𝐭𝐳, 𝐨𝐟 𝐭𝐡𝐞 𝐔𝐧𝐢𝐯𝐞𝐫𝐬𝐢𝐭𝐲 𝐨𝐟 𝐖𝐚𝐬𝐡𝐢𝐧𝐠𝐭𝐨𝐧, 𝐒𝐞𝐚𝐭𝐭𝐥𝐞: youtube.com/playlist?list=PLMrJAkhIeNNQz4BMoGSsN8cbt8pHlokhV
Search for other videos by these three. Many gems to be found.
For 𝐏𝐫𝐨𝐛𝐚𝐛𝐢𝐥𝐢𝐭y and Statistics:
A relatively recent addition to the canon of amazing courses, by @eigensteve (Steve Brunton, professor at University of Washington) on probability & statistics. I wish these had been available when I was learning: www.youtube.com/watch?v=sQqniayndb4
A quirky set of bite-sized videos on key concepts in probability theory by 𝐑𝐢𝐜𝐡 𝐑𝐚𝐝𝐤𝐞, 𝐩𝐫𝐨𝐟𝐞𝐬𝐬𝐨𝐫 𝐚𝐭 𝐑𝐏𝐈, 𝐓𝐫𝐨𝐲, 𝐍𝐘. www.youtube.com/watch?v=jdoLvrfkG3Y
If you want a masterclass on 𝐒𝐮𝐩𝐩𝐨𝐫𝐭 𝐕𝐞𝐜𝐭𝐨𝐫 𝐌𝐚𝐜𝐡𝐢𝐧𝐞𝐬, there's none better than 𝐏𝐚𝐭𝐫𝐢𝐜𝐤 𝐖𝐢𝐧𝐬𝐭𝐨𝐧'𝐬 𝐌𝐈𝐓 lecture: www.youtube.com/watch?v=_PwhiWxHK8o
(Winston credits only Vladimir Vapnik for SVMs, without any mention of Isabelle Guyon and Bernhard Boser; it's a massive oversight. WHY MACHINES LEARN corrects the narrative and tells the story of how SVMs actually unfolded).
Of course, watch the other lectures in Winston's course for a general overview of ideas/algorithms in AI/ML. He's a mesmerizing teacher.
If you can persist with these lectures--and then keep dipping into the never-ending resource that is YouTube --you'll be well set to get under the mathematical skin of ML.
Of course, if you want a gentler intro to the math and the conceptual ideas, alongside some of the sociological and historical narratives surrounding ML, there's WHY MACHINES LEARN
US Edition: www.penguinrandomhouse.com/books/677608/why-machines-lear…
US Edition: www.penguin.co.uk/books/446849/why-machines-learn-by-anan…
(Note these are same book, different covers) -
Anil Ananthaswamy
@anilananth
The Agony and Ecstasy of Machine Learning
I wrote WHY MACHINES LEARN because I was besotted by some of the math behind ML. The affair began during the peak of pandemic: repeated watching of YouTube lectures on ML during those weeks and months of isolation led me to appreciate the (subjective) mathematical beauty of ML.
In particular, the proof of the perceptron convergence theorem, which states that the algorithm will find a linearly separating hyperplane in finite time to delineate two clusters of data if one exists--using rather simple manipulations of vectors and matrices--convinced me that this was the way to learn ML: focus on some of key conceptual aspects of the math, and revel in its beauty.
In one of his lectures on machine learning, Cornel professor Killian Weinberger tells his students about one ML proof involving Bayes's theorem: “This has got to be the most beautiful thing you have seen all day...I tear up—it’s so beautiful.”
So, while we fret about the hype over AI and the possible negative consequences of the technology (which are all very real), it's worth remembering that nearly eight decades of effort (starting in the 1950s) have brought us to this point. It's an entirely human effort: our intelligence is begetting something new. We can both marvel and despair at what is happening.
US: www.penguinrandomhouse.com/books/677608/why-machines-lear…
UK: www.penguin.co.uk/books/446849/why-machines-learn-by-anan… -
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
The Many Moods of Machine Learning
If you feel overwhelmed by the jargon of machine learning and wonder how it all ties together, you aren’t alone. There's training vs test data, supervised vs unsupervised vs self-supervised learning, regression vs classification, linear vs non-linear models, hand-designed vs learned features, discriminative vs generative AI, and so on.
These are a few of the important axes along which one can analyze and understand ML. The more ways we can grok machine learning, the more it begins making sense. Here are some intuitions to get us started.
But first, let’s get a basic definition out of the way. Machine learning in the broadest sense is about getting machines to learn about patterns that exist in data and then use what’s learned—i.e. the model of the data—to make predictions given new previously unseen data or even to generate new data.
TRAINING vs TEST DATA
The data used to train a model is the training dataset. The test dataset is the unseen data on which you test your model before releasing it into the wild. There are nuances, but that’s the broad distinction. One crucial point: both the training and test data are said to be drawn from the same underlying distribution of data (so, if the training data only has images of cats and dogs, so too must the test data; you can’t test your ML model of cats and dogs using images of elephants, for example.) In most ML projects, you ensure this consistency by taking a dataset and splitting it 80:20—80% of the dataset is used for training and 20% for testing. There are niggling details about getting this split right, but that’s the general idea.
Let’s start with a basic dataset to illustrate the concepts. Consider grayscale images that are 50 pixels by 50 pixels, so 2,500 pixels in total. Let’s say we have ten thousand images in total. But we don’t know if the images have any commonality—are they of similar things (say, cats) or very dissimilar things (say, cats and cars)? In other words, no human has looked at the raw data and labeled them as being those of cats or cars. What can we do?
UNSUPERVISED vs SUPERVISED vs SELF-SUPERVISED
One type of learning we can do with such raw data is called unsupervised learning—meaning, no human supervision is involved. We’d first turn each image into a vector, such that each pixel is an element of that vector. Each image becomes a 2,500-element vector. If you were to plot this vector in 2,500-dimensional space, it’d be a point. Imagine plotting each image in that 2,500-D space. You’d get 10,000 points, one for each image. Maybe about half of them are clustered in one region of that space and the rest in another region. A clustering algorithm—such as K-means clustering—can find the “centroid” for each cluster; an image is of one type or another based on the centroid it’s closest to. Note, the algorithm can’t tell whether the image is that of a cat or a car, just that it’s of one type or another (you can, of course, have more than two clusters—but you’ll have to tell your algorithm at the outset how many clusters to look for).
Now, what does it mean to make a prediction about new, unseen data in this case? Well, given a new image, you can simply plot it as a point in the 2,500-D space, and see which centroid it’s closest to; that predicts the type of image (type 1 or type 2, without again knowing anything more about the image).
But what if you wanted to not just cluster the images, but also classify them, i.e. tell apart images of cats from those of cars. That’s where supervised learning comes in. Humans would have to first label the 10,000 images as either a “cat” or a “car”, where “cat” can be -1 and “car” +1. Now, training a model means showing it an image and asking it to learn to correctly classify it as a cat (-1) or a car (+1). Once the model has learned this correlation between an image and it’s class, for all the images in the training dataset, you can give it a new image, and it’ll tell you whether it’s an image of a cat or a car! Of course, there’s always a risk that the classifier will make a mistake and this risk can be quantified; different algorithms come with different levels of risk.
There’s another class of learning: self-supervised. This is not about finding clusters in data, but rather about learning the features or patterns in data that can then be used for downstream tasks. Briefly, imagine taking an image from your training dataset and randomly masking 25% of the pixels, providing the masked image as input to your ML algorithm, and asking it to predict the full unmasked image. As the ML algorithm —in this case, most likely a deep neural network trained using backpropagation—learns to do this, by iterating over-and-over the entire set of images in the training dataset, it learns some internal representations that capture essential features about the data, which can then be used to reconstruct images. Now, given some new, partially obscured image, the model can fill in the missing pixels.
Modern Large Language Models (LLMs) use self-supervised learning. Given textual data, the algorithm learns a model of human written language. It starts by masking, for example, the last word of a sentence, and gets a model to correctly predict the masked word. By doing this for every sentence in a massive corpus of such data, the model learns internal representations of the statistical structure of written language.
Why use the term self-supervised and not unsupervised for LLMs? In unsupervised learning, the algorithm doesn’t generate a “teaching” signal by comparing the output produced by a model against the expected output. In supervised learning, the algorithm generates such a teaching signal by comparing, say, the output of model against a human-provided label. In self-supervised learning, the algorithm manufactures such a teaching signal without a human in the loop, say, by comparing the predicted value for the masked pixels against the known value of the pixels, or by comparing the predicted masked word at the end of a sentence against the known word, and so on.
REGRESSION vs CLASSIFICATION
An ML algorithm that’s trying to tell apart, for example, an image of a cat from that of a car is doing classification. It’s categorizing data
But what if the problem you were trying to solve involved only images of cars, and associated with each image was some real-valued number, denoting the size/volume of the car in cubic feet. (This is a completely made up example, and quite an unreasonable one at that.) Let’s say you had 10,000 such images in the training data, and humans had painstakingly provided the size/volume for the car in each image. One can learn an ML model to correlate the car in an image with its size/volume. Now, given an image of a car that wasn’t in the training dataset, it can predict some real-valued number that’s an estimate of the car’s size/volume. This task—of predicting some real-valued number given some input—is regression.
LINEAR vs NON-LINEAR
A classifier described above finds a boundary between, say, two clusters of data. This boundary can be linear—a line in 2D, a plane in 3D, or a hyperplane in higher dimensions—making it a linear classifier.
But sometimes a line/hyperplane cannot separate the clusters; you need a curve/surface, something wiggly. The boundary and hence the classifier becomes non-linear.
The same can be said for regression, which can be thought of as learning a function that maps input variables to a real-valued output. If this function is a straight line, it’s linear regression. If it’s curvy, wiggly, it’s non-linear.
HAND-DESIGNED vs LEARNED FEATURES
Firstly, what are features? Say you had images of two types of vehicles: fire trucks and school buses. You want to build an ML model that can tell apart fire trucks from school buses. One thing you can do is to either manually, or by writing some software, analyze the images and come up with values for each of these aspects (or features) of the vehicles:
Color (Yellow for school bus, Red for fire truck)
Has a ladder (No for school bus, Yes for fire truck)
Has a hose (No for school bus, Yes for fire truck)
Has a stop sign: (Yes for school bus, No for fire truck)
Now, you can train an ML model to recognize that an object is a school bus if the four features have the value (Yellow, No, No, Yes) and a fire truck if the features have the value (Red, Yes, Yes, No). These features are said to have been hand-designed. Someone decided these were the important features!
Or, you can take a set of images that have been labeled as “school bus” or “fire truck” and feed each image to a neural network (by first turning each image into, say, a 2500-element vector), and use supervised learning to get the network to distinguish between the two types of images. If there are no other pieces of confounding information in the images, it’s very likely that the neural network will learn to internally represent the features of interest—i.e. color, ladder, hose, stop sign—and use these features for classification. It would have learned these features. (It’s entirely possible that the model might pick up some other extraneous information or features that allows it to classify the images.)
DISCRIMINATIVE vs GENERATIVE AI
When you train a model to simply categorize or classify data, you have built a discriminative AI. It discriminates between different classes of data.
Sometimes you want to do more than categorize. You want to generate new data. Think about our training dataset of 10,000 50x50 pixel images of, say, cars and cats. Since each image is a point in 2,500-D space, you’d get two clusters of points, one for cars and one for cats, if you plotted all the images in that high-dimensional space. Now imagine a surface in this space + one extra dimension that hovers over the data points—call it the probability distribution over the data that models how likely any given point is in that 2500-D space. There will be two peaks in this surface: one over points representing cats and the other over points representing cars; the value of the surface will peter down to zero elsewhere. Now, instead of trying to learn a curve or a surface to categorize the classes, you can build a ML model of the surface that estimates the probability distribution over data.
Then, one can sample from that distribution: a sample gives you a point in 2500-D space, or a 50x50 image, with likely pixel values. In all likelihood, such an image would look either like a cat or a car, even if it doesn’t exactly resemble an image in the training data.
This, in essence, is generative AI: you learn the probability distribution over data and figure out how to sample from it, to create a new instance of data that resembles the training data, statistically speaking. However, the twin problems of estimating the distribution and sampling from it are far from trivial.
All this and more, coupled with some of the social history of ML and lots of math, can be found in WHY MACHINES LEARN:
US: www.penguinrandomhouse.com/books/677608/why-machines-lear…
UK: www.penguin.co.uk/books/446849/why-machines-learn-by-anan… -
Anil Ananthaswamy reposted
Shaarang Shringarpure
@Shaarang88
The much demanded Michael Holding video is out❤️Long live test cricket❤️
#indvsaus #crickettwitter -
Anil Ananthaswamy reposted
Joe G
@EastEndJoe
This is how 2025 is going so far.
-
Anil Ananthaswamy
@anilananth
The Many Moods of Machine Learning
If you feel overwhelmed by the jargon of machine learning and wonder how it all ties together, you aren’t alone. There's training vs test data, supervised vs unsupervised vs self-supervised learning, regression vs classification, linear vs non-linear models, hand-designed vs learned features, discriminative vs generative AI, and so on.
These are a few of the important axes along which one can analyze and understand ML. The more ways we can grok machine learning, the more it begins making sense. Here are some intuitions to get us started.
But first, let’s get a basic definition out of the way. Machine learning in the broadest sense is about getting machines to learn about patterns that exist in data and then use what’s learned—i.e. the model of the data—to make predictions given new previously unseen data or even to generate new data.
TRAINING vs TEST DATA
The data used to train a model is the training dataset. The test dataset is the unseen data on which you test your model before releasing it into the wild. There are nuances, but that’s the broad distinction. One crucial point: both the training and test data are said to be drawn from the same underlying distribution of data (so, if the training data only has images of cats and dogs, so too must the test data; you can’t test your ML model of cats and dogs using images of elephants, for example.) In most ML projects, you ensure this consistency by taking a dataset and splitting it 80:20—80% of the dataset is used for training and 20% for testing. There are niggling details about getting this split right, but that’s the general idea.
Let’s start with a basic dataset to illustrate the concepts. Consider grayscale images that are 50 pixels by 50 pixels, so 2,500 pixels in total. Let’s say we have ten thousand images in total. But we don’t know if the images have any commonality—are they of similar things (say, cats) or very dissimilar things (say, cats and cars)? In other words, no human has looked at the raw data and labeled them as being those of cats or cars. What can we do?
UNSUPERVISED vs SUPERVISED vs SELF-SUPERVISED
One type of learning we can do with such raw data is called unsupervised learning—meaning, no human supervision is involved. We’d first turn each image into a vector, such that each pixel is an element of that vector. Each image becomes a 2,500-element vector. If you were to plot this vector in 2,500-dimensional space, it’d be a point. Imagine plotting each image in that 2,500-D space. You’d get 10,000 points, one for each image. Maybe about half of them are clustered in one region of that space and the rest in another region. A clustering algorithm—such as K-means clustering—can find the “centroid” for each cluster; an image is of one type or another based on the centroid it’s closest to. Note, the algorithm can’t tell whether the image is that of a cat or a car, just that it’s of one type or another (you can, of course, have more than two clusters—but you’ll have to tell your algorithm at the outset how many clusters to look for).
Now, what does it mean to make a prediction about new, unseen data in this case? Well, given a new image, you can simply plot it as a point in the 2,500-D space, and see which centroid it’s closest to; that predicts the type of image (type 1 or type 2, without again knowing anything more about the image).
But what if you wanted to not just cluster the images, but also classify them, i.e. tell apart images of cats from those of cars. That’s where supervised learning comes in. Humans would have to first label the 10,000 images as either a “cat” or a “car”, where “cat” can be -1 and “car” +1. Now, training a model means showing it an image and asking it to learn to correctly classify it as a cat (-1) or a car (+1). Once the model has learned this correlation between an image and it’s class, for all the images in the training dataset, you can give it a new image, and it’ll tell you whether it’s an image of a cat or a car! Of course, there’s always a risk that the classifier will make a mistake and this risk can be quantified; different algorithms come with different levels of risk.
There’s another class of learning: self-supervised. This is not about finding clusters in data, but rather about learning the features or patterns in data that can then be used for downstream tasks. Briefly, imagine taking an image from your training dataset and randomly masking 25% of the pixels, providing the masked image as input to your ML algorithm, and asking it to predict the full unmasked image. As the ML algorithm —in this case, most likely a deep neural network trained using backpropagation—learns to do this, by iterating over-and-over the entire set of images in the training dataset, it learns some internal representations that capture essential features about the data, which can then be used to reconstruct images. Now, given some new, partially obscured image, the model can fill in the missing pixels.
Modern Large Language Models (LLMs) use self-supervised learning. Given textual data, the algorithm learns a model of human written language. It starts by masking, for example, the last word of a sentence, and gets a model to correctly predict the masked word. By doing this for every sentence in a massive corpus of such data, the model learns internal representations of the statistical structure of written language.
Why use the term self-supervised and not unsupervised for LLMs? In unsupervised learning, the algorithm doesn’t generate a “teaching” signal by comparing the output produced by a model against the expected output. In supervised learning, the algorithm generates such a teaching signal by comparing, say, the output of model against a human-provided label. In self-supervised learning, the algorithm manufactures such a teaching signal without a human in the loop, say, by comparing the predicted value for the masked pixels against the known value of the pixels, or by comparing the predicted masked word at the end of a sentence against the known word, and so on.
REGRESSION vs CLASSIFICATION
An ML algorithm that’s trying to tell apart, for example, an image of a cat from that of a car is doing classification. It’s categorizing data
But what if the problem you were trying to solve involved only images of cars, and associated with each image was some real-valued number, denoting the size/volume of the car in cubic feet. (This is a completely made up example, and quite an unreasonable one at that.) Let’s say you had 10,000 such images in the training data, and humans had painstakingly provided the size/volume for the car in each image. One can learn an ML model to correlate the car in an image with its size/volume. Now, given an image of a car that wasn’t in the training dataset, it can predict some real-valued number that’s an estimate of the car’s size/volume. This task—of predicting some real-valued number given some input—is regression.
LINEAR vs NON-LINEAR
A classifier described above finds a boundary between, say, two clusters of data. This boundary can be linear—a line in 2D, a plane in 3D, or a hyperplane in higher dimensions—making it a linear classifier.
But sometimes a line/hyperplane cannot separate the clusters; you need a curve/surface, something wiggly. The boundary and hence the classifier becomes non-linear.
The same can be said for regression, which can be thought of as learning a function that maps input variables to a real-valued output. If this function is a straight line, it’s linear regression. If it’s curvy, wiggly, it’s non-linear.
HAND-DESIGNED vs LEARNED FEATURES
Firstly, what are features? Say you had images of two types of vehicles: fire trucks and school buses. You want to build an ML model that can tell apart fire trucks from school buses. One thing you can do is to either manually, or by writing some software, analyze the images and come up with values for each of these aspects (or features) of the vehicles:
Color (Yellow for school bus, Red for fire truck)
Has a ladder (No for school bus, Yes for fire truck)
Has a hose (No for school bus, Yes for fire truck)
Has a stop sign: (Yes for school bus, No for fire truck)
Now, you can train an ML model to recognize that an object is a school bus if the four features have the value (Yellow, No, No, Yes) and a fire truck if the features have the value (Red, Yes, Yes, No). These features are said to have been hand-designed. Someone decided these were the important features!
Or, you can take a set of images that have been labeled as “school bus” or “fire truck” and feed each image to a neural network (by first turning each image into, say, a 2500-element vector), and use supervised learning to get the network to distinguish between the two types of images. If there are no other pieces of confounding information in the images, it’s very likely that the neural network will learn to internally represent the features of interest—i.e. color, ladder, hose, stop sign—and use these features for classification. It would have learned these features. (It’s entirely possible that the model might pick up some other extraneous information or features that allows it to classify the images.)
DISCRIMINATIVE vs GENERATIVE AI
When you train a model to simply categorize or classify data, you have built a discriminative AI. It discriminates between different classes of data.
Sometimes you want to do more than categorize. You want to generate new data. Think about our training dataset of 10,000 50x50 pixel images of, say, cars and cats. Since each image is a point in 2,500-D space, you’d get two clusters of points, one for cars and one for cats, if you plotted all the images in that high-dimensional space. Now imagine a surface in this space + one extra dimension that hovers over the data points—call it the probability distribution over the data that models how likely any given point is in that 2500-D space. There will be two peaks in this surface: one over points representing cats and the other over points representing cars; the value of the surface will peter down to zero elsewhere. Now, instead of trying to learn a curve or a surface to categorize the classes, you can build a ML model of the surface that estimates the probability distribution over data.
Then, one can sample from that distribution: a sample gives you a point in 2500-D space, or a 50x50 image, with likely pixel values. In all likelihood, such an image would look either like a cat or a car, even if it doesn’t exactly resemble an image in the training data.
This, in essence, is generative AI: you learn the probability distribution over data and figure out how to sample from it, to create a new instance of data that resembles the training data, statistically speaking. However, the twin problems of estimating the distribution and sampling from it are far from trivial.
All this and more, coupled with some of the social history of ML and lots of math, can be found in WHY MACHINES LEARN:
US: www.penguinrandomhouse.com/books/677608/why-machines-lear…
UK: www.penguin.co.uk/books/446849/why-machines-learn-by-anan… -
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
Happy 2025! Looking forward to welcoming the participants in the second half of our Special Year on Large Language Models and Transformers. And we hope to run into some of the rest of you at the workshops!
simons.berkeley.edu/programs/special-year-large-language-… -
Anil Ananthaswamy reposted
IIT Madras
@iitmadras
In a guest article for @timesofindia , @iitmadras Prof. B. Ravindran (@ravi_iitm), Head of the Wadhwani School of Data Science and AI (@WSAI_IITM ), and Krishnan Narayanan, Co-founder & President of Itihaasa Research and Digital, emphasise the need to democratise AI education.
Artificial Intelligence (AI) is reshaping the world, from personalised education to tailored marketing and civic data analysis. However, the question arises—are we equipping everyone for this revolution, or just the select few who create it?
For AI to benefit everyone, education must empower both users and creators. Users need foundational knowledge, practical training, ethical awareness, and problem-solving skills to integrate AI into various fields. Initiatives like the UAE's AI Camp show how accessible AI literacy can empower diverse professions. Meanwhile, creators must focus on strong mathematical foundations, advanced programming, and ethical design to ensure transparent and impactful innovations. Simplified tools may help, but deeper understanding drives meaningful progress. The Wadhwani School of Data Science and AI at #iitmadras leads the way in empowering both AI creators and users, closing the gap and ensuring AI becomes a catalyst for inclusive growth.
Read the full article here: timesofindia.indiatimes.com/technology/times-techies/focu…
#ai #education #iitmadras #wadhwanischool #artificialintelligence -
Anil Ananthaswamy reposted
Michael Bronstein
@mmbronstein
Annapurna
-
Anil Ananthaswamy
@anilananth
If you are in Bangalore on Jan 6, join us for a fireside chat (sans actual fire). Prof B. Ravindran @ravi_iitm and I will talk about AI and Learning. Event organized by my alma mater @iitmadras' alumni association, @WSAI_IITM, and @Infosys Science Foundation. Details/registration here: iitmaa.org/events/14155
-
Anil Ananthaswamy reposted
Prof. Anima Anandkumar
@AnimaAnandkumar
2024 was a pivotal year for AI+Science. Our team made exciting contributions. Here are some highlights: tensorlab.cms.caltech.edu/users/anima/2024_research_highl…
1. Neural Operators as a unifying AI framework for modeling multi-scale processes. We got to write a perspective article in @NatRevPhys and released an open-source library that has been widely adopted.
2. FourCastNet, our Neural-Operator-based model, featured in the @WHOSTP PCAST report on AI+Science. Its speed enables large ensembles to obtain unprecedented predictions of extreme weather events.
3. Designed a novel medical catheter using Neural Operator-based inverse design.
4. Used Neural Operators with reinforcement learning for turbulence stabilization and wall friction reduction.
5. Many other applications where Neural Operators have shown a big impact: nuclear fusion, computational imaging, Carbon capture and storage etc.
6. Gave a @TEDTalks highlighting AI+Science.
7. Exascale training of genome-scale language models for protein design that was recognized as a finalist for the ACM Gordon-Bell prize.
8. State-of-the-art protein-ligand structure prediction that can handle changing protein conformations that Alphafold and other methods cannot.
9. AI+Math innovations with LLM + Lean for theorem proving.
10. Hardware-efficient large-scale training such as gradient projections and mini-sequence transformers.
11. Many honors such as @iitmadras Distinguished Alumnus award, and @BlavatnikAwards -
Anil Ananthaswamy reposted
The Abel Prize
@abel_prize
Happy birthday, Srinivasa S. R. Varadhan!
Varadhan received the Abel Prize in 2007 for "fundamental contributions to probability theory and in particular for creating a unified theory of large deviations."
#abelprize #abelprize2007 #mathematics #science -
Anil Ananthaswamy reposted
Sebastian Raschka
@rasbt
Happy New Year! To kick off the year, I've finally been able to format and upload the draft of my AI Research Highlights of 2024 article.
It covers a variety of topics, from mixture-of-experts models to new LLM scaling laws for precision: magazine.sebastianraschka.com/p/ai-research-papers-2024-p… -
Anil Ananthaswamy reposted
Sandhya Koushika
@WormlockHolmes
Two books I enjoyed in 2024
Speaking with Nature: the origins of Indian Environmentalism by @Ram_Guha
Why Machines Learn by @anilananth (still reading this) -
Anil Ananthaswamy reposted
Danielle Newnham
@daniellenewnham
I hope that in this year to come, you make mistakes. Because if you are making mistakes, then you are making new things, trying new things, learning, living, pushing yourself, changing yourself, changing your world. You're doing things you've never done before, and more importantly, you're doing something.
@neilhimself
Happy 2025 one and all. xx
www.linkedin.com/pulse/startups-science-promise-2025-dani… -
Anil Ananthaswamy reposted
Chisom Rutherford
@rutherfordml
I really enjoyed reading Why Machines Learn. It was information dense and math was challenging sometimes, but it was worth it.
-
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
The Theoretical Minimum (for Machine Learning)…And Why Linear Algebra, Calculus, and Probability & Statistics often get mentioned as the minimum math you need to start on your machine learning journey. But why these disciplines? Here’s an intuition for why:
Linear Algebra: Machines turn data that we produce (text, images, audio, etc.) into a format that’s conducive for algorithmic manipulation. Often, they turn data into vectors. A 2D vector is a point in 2D space, a 3D vector is a point in 3D space, and so on. By converting data into vectors, an ML algorithm can do many things. For instance, it can calculate the distance between two n-dimensional vectors in nD space and establish similarity or dissimilarity. Similar vectors (data) are closer in that nD space than dissimilar vectors (data). Or an ML algorithm can learn how these vectors are distributed in nD space, information that can then be used to generate new vectors—or new data!
You can also think of a machine learning model as something that transforms an input vector X into an output vector Y. This transformation is often the result of multiplying a matrix by a vector, or the result of a sequence of such transformations. The matrix (or matrices) represent the parameters of the model—and have to be learned from data. Vectors, matrices and their manipulations are all essentially linear algebra.
Calculus: A model, or rather its parameters, has to be learned from data. But what does learning entail? Usually, learning involves transforming an input vector into an output vector, calculating the loss (or error) made by the model—according to some measure of the distance between the produced output vector and the expected output vector—and then using that loss to tweak the parameters of the model such that given the same input vector the model produces an output that is a little closer to the expected output, or the loss is reduced.
This is where calculus comes in. If the loss can be written down as a function of the parameters of the model, then reducing the loss is an optimization problem. The loss function gives you the loss landscape, and one has to descend from some region high up in the landscape (representing high loss) to some deep valley (representing very low or zero loss). One way to do so is by using some form of gradient descent. The gradient is a vector, where the elements of the vector are partial derivatives of the loss function with respect to the parameters. The gradient points in a direction of increasing loss. Calculate the gradient, and use it to tweak the parameters such that one moves a smidgen in the opposite direction along the surface of the loss function. Doing so iteratively, one can reduce the loss to some optimal minimum or even zero. At that point, one has learned a good-enough model. All this is calculus.
Probability & Statistics: These are huge fields, but the basics of how probability and statistics apply to ML is not hard to appreciate. All machine learning involves learning about patterns in data and using those learned patterns to say something about new data, or to generate new data. All training data can be said to have been drawn from some underlying true distribution of all possible data. But we can never know this ground truth. So, the job of an ML algorithm is—in one way of thinking—to estimate as well as possible the underlying distribution using only the training data. The closer this estimate is to the ground truth, the better is your ML model. This estimate can then be used to, say, discriminate between different classes of data (discriminative learning). Or one can sample from this distribution to generate a new instance of data that looks very much like real data—this is generative AI. The details are non-trivial, but conceptually that’s a good, simple start.
WHY MACHINES LEARN drives home these concepts, encapsulated in the stories of the people that made them possible, using a historical, math-laden narrative.
US: www.penguinrandomhouse.com/books/677608/why-machines-lear…
UK: www.penguin.co.uk/books/446849/why-machines-learn-by-anan… -
Anil Ananthaswamy
@anilananth
The Theoretical Minimum (for Machine Learning)…And Why Linear Algebra, Calculus, and Probability & Statistics often get mentioned as the minimum math you need to start on your machine learning journey. But why these disciplines? Here’s an intuition for why:
Linear Algebra: Machines turn data that we produce (text, images, audio, etc.) into a format that’s conducive for algorithmic manipulation. Often, they turn data into vectors. A 2D vector is a point in 2D space, a 3D vector is a point in 3D space, and so on. By converting data into vectors, an ML algorithm can do many things. For instance, it can calculate the distance between two n-dimensional vectors in nD space and establish similarity or dissimilarity. Similar vectors (data) are closer in that nD space than dissimilar vectors (data). Or an ML algorithm can learn how these vectors are distributed in nD space, information that can then be used to generate new vectors—or new data!
You can also think of a machine learning model as something that transforms an input vector X into an output vector Y. This transformation is often the result of multiplying a matrix by a vector, or the result of a sequence of such transformations. The matrix (or matrices) represent the parameters of the model—and have to be learned from data. Vectors, matrices and their manipulations are all essentially linear algebra.
Calculus: A model, or rather its parameters, has to be learned from data. But what does learning entail? Usually, learning involves transforming an input vector into an output vector, calculating the loss (or error) made by the model—according to some measure of the distance between the produced output vector and the expected output vector—and then using that loss to tweak the parameters of the model such that given the same input vector the model produces an output that is a little closer to the expected output, or the loss is reduced.
This is where calculus comes in. If the loss can be written down as a function of the parameters of the model, then reducing the loss is an optimization problem. The loss function gives you the loss landscape, and one has to descend from some region high up in the landscape (representing high loss) to some deep valley (representing very low or zero loss). One way to do so is by using some form of gradient descent. The gradient is a vector, where the elements of the vector are partial derivatives of the loss function with respect to the parameters. The gradient points in a direction of increasing loss. Calculate the gradient, and use it to tweak the parameters such that one moves a smidgen in the opposite direction along the surface of the loss function. Doing so iteratively, one can reduce the loss to some optimal minimum or even zero. At that point, one has learned a good-enough model. All this is calculus.
Probability & Statistics: These are huge fields, but the basics of how probability and statistics apply to ML is not hard to appreciate. All machine learning involves learning about patterns in data and using those learned patterns to say something about new data, or to generate new data. All training data can be said to have been drawn from some underlying true distribution of all possible data. But we can never know this ground truth. So, the job of an ML algorithm is—in one way of thinking—to estimate as well as possible the underlying distribution using only the training data. The closer this estimate is to the ground truth, the better is your ML model. This estimate can then be used to, say, discriminate between different classes of data (discriminative learning). Or one can sample from this distribution to generate a new instance of data that looks very much like real data—this is generative AI. The details are non-trivial, but conceptually that’s a good, simple start.
WHY MACHINES LEARN drives home these concepts, encapsulated in the stories of the people that made them possible, using a historical, math-laden narrative.
US: www.penguinrandomhouse.com/books/677608/why-machines-lear…
UK: www.penguin.co.uk/books/446849/why-machines-learn-by-anan… -
Anil Ananthaswamy reposted
BookoftheDay.org
@bookofthedayorg
Today's Book: Why Machines Learn: The Elegant Math Behind Modern AI - Anil Ananthaswamy
@anilananth
@anilananth.bsky.social
@DuttonBooks
@duttonbooks.bsky.social
@MLStreetTalk
#bookoftheday
bookoftheday.org/why-machines-learn-the-elegant-math-behi… -
Anil Ananthaswamy
@anilananth
Very pleased to find WHY MACHINES LEARN on this list, @EricTopol. Something about the zeitgeist that four titles are questions turned into answers :-) Why We Die, How We Age, How Life Works & Why Machines Learn!
Eric Topol
@EricTopol
My favorite books, 2024
by @emollick @firstadopter @anilananth @harari_yuval @carlzimmer and other great authors not on X -
Anil Ananthaswamy reposted
Andrew Gordon Wilson
@andrewgwils
Many of the greatest papers, now canonical works, have a story of resistance, tension, and, finally, a crucial advocate. It's shockingly common. Why is there a bias against excellence? And what happens to those papers, those people, when no one had the courage to advocate?
-
Anil Ananthaswamy
@anilananth
To see WHY MACHINES LEARN mentioned alongside lecture videos by @AndrewYNg and @karpathy was not on my bingo card! I learned my ML from these experts (and others) and owe them a huge debt of gratitude. www.youtube.com/watch?v=_xIwjmCH6D4 @BorisMeinardus
-
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
My alma mater @iitmadras's alumni association, @IITMAA, and @Infosys Science Foundation have organized a fireside chat. Prof. Balaraman Ravindran, head of @WSAI_IITM and I will talk about AI and Learning: 430PM onward, Jan 6, at the Infosys Science Foundation, Bangalore
-
Anil Ananthaswamy reposted
Balaraman Ravindran
@ravi_iitm
Super thrilled to be joining @anilananth for this conversation. Thank you @IITMAA and Infosys Science Foundation for organizing this.
-
Anil Ananthaswamy
@anilananth
My alma mater @iitmadras's alumni association, @IITMAA, and @Infosys Science Foundation have organized a fireside chat. Prof. Balaraman Ravindran, head of @WSAI_IITM and I will talk about AI and Learning: 430PM onward, Jan 6, at the Infosys Science Foundation, Bangalore
-
Anil Ananthaswamy reposted
Physics Memes
@ThePhysicsMemes
I love physics #physicsteacher
-
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
A machine learning face-off that the world never got to see. In the late 1950s, there were two researchers working seriously on single-layer neural networks: Frank Rosenblatt at Cornell University and Bernie Widrow at Stanford.
Rosenblatt had invented the perceptron and Widrow had invented ADALINE (for adaptive single neuron). "There may have been only the two of us working on neural networks in the whole United States," Widrow told me.
All the attention was on Rosenblatt, including articles in the New York Times and elsewhere. Plus, Rosenblatt's impressive tome, Principles of Neurodynamics, got him more acclaim. Widrow's work didn't attract as much attention--though his training algorithm (the Widrow-Hoff Least Mean Squares algorithm) can be seen as the precursor to modern-day stochastic gradient descent.
Widrow narrated a story (which I left out of WHY MACHINES LEARN; in hindsight I wish I had found a place for it). During our zoom chat, Widrow turned around to point to a new ADALINE box (top left panel: see little black box behind the lamp that looks like a white mushroom). In the 1960s, Widrow took a somewhat bigger box (it'd still fit into a suitcase) to Cornell, where he and Rosenblatt pitted the perceptron against ADALINE, on simple pattern classification (telling apart two patterns).
The perceptron had electro-mechanical weights (i.e., the parameters of the artificial neuron) that needed motors to adjust, ADALINE had electronic weights. "I was able to put [ADALINE] in a little box," Widrow told me. "He had a whole rack of stuff to do a perceptron."
They'd input the patterns to their machines. The perceptron's motors would grind and grind and grind, said Widrow, while ADALINE would just go "pssup" (Widrow making a sucking sound). It was fast! "The little ADALINE is saying, 'give me more patterns, give me more patterns'," Widrow said.
Though both the perceptron and ADALINE could distinguish between two patterns, ADALINE was a small hardware neuron, the perceptron was a complicated machine (top right panel: photo of Rosenblatt working on the perceptron, Cornell Chronicle; bottom left: The wiring panel of the perceptron, Cornell University Library).
In Widrow's view, the small ADALINE outperformed the more complex perceptron. I wish I could have talked to Rosenblatt about his version of this meeting. Unfortunately, Rosenblatt died in a boating accident on Chesapeake Bay in 1971, when he was only 43 yrs old.
Algorithmically, the perceptron convergence rule came with a mathematical guarantee: the learning algorithm would always find a linearly separating boundary between two clusters of data if the data were linearly separable. The LMS algorithm did not provide such a guarantee.
While I do not tell the story of this face-off in WHY MACHINES learn, I do narrate in detail the history and the math behind Rosenblatt's perceptron and Widrow's LMS algorithms.
US: www.penguinrandomhouse.com/books/677608/why-machines-lear…
UK: www.penguin.co.uk/books/446849/why-machines-learn-by-anan… -
Anil Ananthaswamy
@anilananth
A machine learning face-off that the world never got to see. In the late 1950s, there were two researchers working seriously on single-layer neural networks: Frank Rosenblatt at Cornell University and Bernie Widrow at Stanford.
Rosenblatt had invented the perceptron and Widrow had invented ADALINE (for adaptive single neuron). "There may have been only the two of us working on neural networks in the whole United States," Widrow told me.
All the attention was on Rosenblatt, including articles in the New York Times and elsewhere. Plus, Rosenblatt's impressive tome, Principles of Neurodynamics, got him more acclaim. Widrow's work didn't attract as much attention--though his training algorithm (the Widrow-Hoff Least Mean Squares algorithm) can be seen as the precursor to modern-day stochastic gradient descent.
Widrow narrated a story (which I left out of WHY MACHINES LEARN; in hindsight I wish I had found a place for it). During our zoom chat, Widrow turned around to point to a new ADALINE box (top left panel: see little black box behind the lamp that looks like a white mushroom). In the 1960s, Widrow took a somewhat bigger box (it'd still fit into a suitcase) to Cornell, where he and Rosenblatt pitted the perceptron against ADALINE, on simple pattern classification (telling apart two patterns).
The perceptron had electro-mechanical weights (i.e., the parameters of the artificial neuron) that needed motors to adjust, ADALINE had electronic weights. "I was able to put [ADALINE] in a little box," Widrow told me. "He had a whole rack of stuff to do a perceptron."
They'd input the patterns to their machines. The perceptron's motors would grind and grind and grind, said Widrow, while ADALINE would just go "pssup" (Widrow making a sucking sound). It was fast! "The little ADALINE is saying, 'give me more patterns, give me more patterns'," Widrow said.
Though both the perceptron and ADALINE could distinguish between two patterns, ADALINE was a small hardware neuron, the perceptron was a complicated machine (top right panel: photo of Rosenblatt working on the perceptron, Cornell Chronicle; bottom left: The wiring panel of the perceptron, Cornell University Library).
In Widrow's view, the small ADALINE outperformed the more complex perceptron. I wish I could have talked to Rosenblatt about his version of this meeting. Unfortunately, Rosenblatt died in a boating accident on Chesapeake Bay in 1971, when he was only 43 yrs old.
Algorithmically, the perceptron convergence rule came with a mathematical guarantee: the learning algorithm would always find a linearly separating boundary between two clusters of data if the data were linearly separable. The LMS algorithm did not provide such a guarantee.
While I do not tell the story of this face-off in WHY MACHINES learn, I do narrate in detail the history and the math behind Rosenblatt's perceptron and Widrow's LMS algorithms.
US: www.penguinrandomhouse.com/books/677608/why-machines-lear…
UK: www.penguin.co.uk/books/446849/why-machines-learn-by-anan… -
Anil Ananthaswamy reposted
Rajesh P. N. Rao
@RajeshPNRao
Episode 201 of the Brain Inspired podcast, featuring my conversation with Paul Middlebrooks @pgmid on classical predictive coding, active predictive coding, brain co-processors and the Indus script, is now out. #brain #neuroscience
braininspired.co/podcast/201/ -
Anil Ananthaswamy reposted
Sabine Hossenfelder
@skdh
An issue similar to the arbitrary p-value cutoff in some areas of science is the 5σ threshold for discoveries in some areas of physics. It's ultimately an arbitrary measure that became used, I think, just because they have so much data and it reduced the number of potential signals to a reasonable amount.
Or to put it differently, if you used a 2σ threshold in particle physics, you'd have a discovery every day which went away a week later.
The issue is though that exactly because the number of anomalies depends on the amount of data, the better physicists get at collecting and managing huge amounts of data the more fluctuations of high significance will show up just coincidentally. That is to say, the 5σ threshold must be moved at some point, the question is just when.
(One could take the almost 8σ supposed anomaly in the W-boson mass as an indication but that was a weird analysis in any case.)
Another issue that I see a lot in astro/cosmology papers that often isn't talked about is that the relation between σ, standard deviations, and the probability of that occurring depends on the probability distribution. The distribution that people typically have in mind is Gaussian, but in astro that's sometimes not the case. So if you read an astro paper with a 20σ anomaly that may be a statement about the data properties and not about the likelihood of that occurring.
I am sure that this is clear to the people who write these papers, I have just noticed that in the more popular media the significance of these findings is often misinterpreted.
It's the same thing in climate papers btw, in case you have wondered why climate scientists do not say things like "this is a 5σ event" or such but instead use vague phrases like "with moderate/high significance". While they could quantify the σ, the issue is that absent a probability distribution, they don't know how to convert it to a probability.
Maybe some issues that @kareem_carr might be interested to look at?
Kareem Carr
@kareem_carr
"Science is neutral and apolitical" says field that bases billions of dollars in decisions on p<0.05, a threshold from 100+ years ago that everyone just uses because everybody else is using it.
-
Anil Ananthaswamy reposted
Mario Krenn
@MarioKrenn6240
Just finished this remarkable book by @maxsbennett, an evolutionary journey through the development of natural intelligence, from early steering mechanisms to simulation & mentalization to language - and connections to ML algorithms.
Very entertaining, highly recommended. -
Anil Ananthaswamy
@anilananth
An entirely unexpected outcome of researching WHY MACHINES LEARN during the pandemic was that I got to talk to my sources on Zoom (how many you can recognize? Some are, of course, extremely well-known--including the three new Nobel Laureates). There were so many highlights of talking to them, among them these:
1) Bernie Widrow, who built ADALINE, the first hardware adaptive linear neuron that was trained using noisy gradient descent in 1959!
2) Peter Hart, who gave us the Cover-Hart k-Nearest Neighbor algorithm in the 1960s (it was Hart's PhD thesis!)
3) John Hopfield: it took some convincing to get him to chat, for he said he had nothing new to say; but, oh, he had so much knowledge to impart (and he was 88 when I spoke to him)
4) Isabelle Guyon, the key figure behind support vector machines, along with Bernard Boser and Vladimir Vapnik (in the photo, holding up her copy of the bible on pattern recognition by Duda and Hart)
5) George Cybenko: It was quite the surreal experience having him explain his universal approximation theorem, while he was sitting in his car, parked somewhere!
The others: Geoff Hinton, Yann LeCun, Misha Belkin, Marcello Pelillo, Philip Stark, Demis Hassabis, Emery Brown, John Abel, Alexei Efros, Ilya Sutskever, Patrick Juola, Alethea Power
You can find their stories, the history of ML and the math behind it all, in WHY MACHINES LEARN
US: www.penguinrandomhouse.com/books/677608/why-machines-lear…
UK: www.penguin.co.uk/books/446849/why-machines-learn-by-anan… -
Anil Ananthaswamy reposted
Mario Krenn
@MarioKrenn6240
Kevin Buzzard (number theoretist & math formalization pioneer, e.g. formalizing perfectoid space, towards formalize Fermat) wrote a great blog post on @OpenAI o3' ability on FrontierMath:
xenaproject.wordpress.com/2024/12/22/can-ai-do-maths-yet-… -
Anil Ananthaswamy reposted
Turing Post (Ksenia Se)
@TheTuringPost
AI/ML books for your holidays🎄
2024 practical guides to AI:
• Build a Large Language Model (From Scratch) by @rasbt
• LLM Engineer's Handbook, @iusztinpaul and @maximelabonne
• AI Engineering: Building Applications with Foundation Models, @chipro
• Hands-On Large Language Models by @JayAlammar and @MaartenGr
• Why Machines Learn: The Elegant Math Behind Modern AI, @anilananth
+ Free books:
• Alice’s Adventures in a Differentiable Wonderland: A Primer on Designing Neural Networks, @s_scardapane
• The Little Book of Deep Learning, @francoisfleuret
• The Basics of Reinforcement Learning from Human Feedback by @natolambert
Take a look if you want to spend some time with a good book this holiday season (all the links to the books are provided) -> www.turingpost.com/p/bestbooks2024 -
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
Thank you @TheTuringPost for including Why Machines Learn in your list of best AI/ML books for 2024, alongside two of my favorite ML books, which I'm currently reading: @rasbt's Build A Large Language Model (From Scratch), and Hands-on Large Language Models by @JayAlammar and @MaartenGr!
Turing Post (Ksenia Se)
@TheTuringPost
AI/ML books for your holidays🎄
2024 practical guides to AI:
• Build a Large Language Model (From Scratch) by @rasbt
• LLM Engineer's Handbook, @iusztinpaul and @maximelabonne
• AI Engineering: Building Applications with Foundation Models, @chipro
• Hands-On Large Language Models by @JayAlammar and @MaartenGr
• Why Machines Learn: The Elegant Math Behind Modern AI, @anilananth
+ Free books:
• Alice’s Adventures in a Differentiable Wonderland: A Primer on Designing Neural Networks, @s_scardapane
• The Little Book of Deep Learning, @francoisfleuret
• The Basics of Reinforcement Learning from Human Feedback by @natolambert
Take a look if you want to spend some time with a good book this holiday season (all the links to the books are provided) -> www.turingpost.com/p/bestbooks2024 -
Anil Ananthaswamy
@anilananth
Thank you @TheTuringPost for including Why Machines Learn in your list of best AI/ML books for 2024, alongside two of my favorite ML books, which I'm currently reading: @rasbt's Build A Large Language Model (From Scratch), and Hands-on Large Language Models by @JayAlammar and @MaartenGr!
Turing Post (Ksenia Se)
@TheTuringPost
AI/ML books for your holidays🎄
2024 practical guides to AI:
• Build a Large Language Model (From Scratch) by @rasbt
• LLM Engineer's Handbook, @iusztinpaul and @maximelabonne
• AI Engineering: Building Applications with Foundation Models, @chipro
• Hands-On Large Language Models by @JayAlammar and @MaartenGr
• Why Machines Learn: The Elegant Math Behind Modern AI, @anilananth
+ Free books:
• Alice’s Adventures in a Differentiable Wonderland: A Primer on Designing Neural Networks, @s_scardapane
• The Little Book of Deep Learning, @francoisfleuret
• The Basics of Reinforcement Learning from Human Feedback by @natolambert
Take a look if you want to spend some time with a good book this holiday season (all the links to the books are provided) -> www.turingpost.com/p/bestbooks2024 -
Anil Ananthaswamy reposted
Subbarao Kambhampati (కంభంపాటి సుబ్బారావు)
@rao2z
🤔Red Teaming is as important for AI Safety as it is for AI Hype. Insider accounts of model capabilities--however awe inspiring--must be evaluated skeptically--especially when there are staggering commercial interests at play.
-
Anil Ananthaswamy
@anilananth
Machine Learning is not just (glorified) statistics! Sure, in its most basic form, ML doesn't seem that far removed from fancy statistics... But ML is its own thing!
Steve Brunton, @eigensteve of the University of Washington, in his fabulous short lecture series on probability and statistics (look for it on YouTube), gives these clear-eyed definitions:
Probability: Assuming you have a known probability distribution that describes your data (say, a Gaussian), probability theory allows you to say something about samples of data you might observe in the future. So, one can calculate the probability that a random variable, X, takes on some value, given the parameter theta that describes the probability distribution (theta for a gaussian would be the mean and variance).
Statistics: It's the flip side of the above. Now, we have in hand the data or samples. We are trying to say something about the probability distribution that best models the data; or given the data we want to say something about the probability of the parameter theta of the distribution
Machine Learning / Deep Learning: Sometimes, we have the data, but the underlying distribution is unknown, or impossible to characterize analytically, using parameters specified by some theta. ML is used to learn the (empirical) distribution by examining the data.
This is the point at which someone will say, but, that is just statistics! Here's why it's not:
Take the problem of learning the probability distribution over the millions of images of the natural world. And not just that: once you have learned the distribution, you have to sample from it, and generate new data that looks like an image from the original dataset. Try doing this with standard statistical methods in some tractable way. It's near impossible.
But diffusion models, heavily influenced by the physics of non-equilibrium thermodynamics, do exactly that. That's deep learning at its best.
Also, take large language models (LLMs). Of course, LLMs learn the statistics of human written language from an enormous corpus of training data, and generate new text by estimating and then sampling from conditional probability distributions, given some input text. Again, while it sounds like it's just statistics, standard statistical methods could not pull off what an LLM can.
The chapter on probability and statistics in WHY MACHINES LEARN was one of the most difficult to write (it involved getting across the basics of two large fields of math and tying them to ML, all in the space of 30 pages). Nearly killed me. But it was also one of the most satisfying chapters to write, not least because I got to understand so much while trying to make sense of these issues.
More here: US: penguinrandomhouse.com/books/677608/why-machines-learn-by…
UK: penguin.co.uk/books/446849/why-machines-learn-by-ananthas… -
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
Very honored to see Why Machines Learn alongside @d_spiegel's book! Hope you enjoy WML ...
Mike Rea
@ThenIf_
Last book delivery before Christmas?
@anilananth @d_spiegel -
Anil Ananthaswamy
@anilananth
Very honored to see Why Machines Learn alongside @d_spiegel's book! Hope you enjoy WML ...
Mike Rea
@ThenIf_
Last book delivery before Christmas?
@anilananth @d_spiegel -
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
Would it surprise you to learn that the algorithm used to train the simplest possible artificial neural network with just 1 hidden layer and 1 hidden neuron is exactly the same as the algorithm to train the largest deep neural networks (with 100s of hidden layers and millions of hidden neurons)?
The algorithm: Backpropagation, the workhorse of modern AI.
As @andrewgwils has pointed out elsewhere, the essential idea underlying backpropagation goes back to Newton! Applying it to train artificial neural networks has its own history.
What does it mean to train a neural network? Training involves giving a network an input and then calculating the difference or "loss" between the output network should produce and the output it does produce. This loss is formulated as a function of the weights of the neurons in the network.
The job of the algorithm is to figure out how much each weight is responsible for the overall error: a problem called credit assignment. If there's no hidden layer, this problem is simple. Just calculate the gradient of the function when the network makes a loss, use the gradient to change all the weights just a wee bit, and reiterate, until you end up with low or zero loss.
But if you have even one hidden neuron, the solution isn't as straightforward. At least it seemed so to neural network researchers for decades, until it all turned out to be rather simple, in hindsight of course.
You have to use the chain rule--one of the most basic rules in calculus--to calculate the gradient w.r.t. each weight, and work one's way backwards from the output layer to the input layer (hence the term backpropagation).
Crucially, all the numbers you need for calculating these gradients have already been computed during the forward pass, when the network turns an input into an erroneous output. You just have to store those numbers and use them appropriately.
So, whether it's the simplest network (top left panel) or a more complicated one (top right panel), the algorithm is exactly the same! Modern packages such as @PyTorch have autograd engines that make this process seamless.
If you want the learn the history of how we got here, from Frank Rosenblatt's ideas from the early 1960s to @geoffreyhinton's 1986 paper with Rumelhart and Williams, you'll find it all in WHY MACHINES LEARN, with all the necessary basic calculus to make sense of this simple yet powerful algorithm.
US: www.penguinrandomhouse.com/books/677608/why-machines-lear…
UK: www.penguin.co.uk/books/446849/why-machines-learn-by-anan… -
Anil Ananthaswamy reposted
Leo Dirac
@leopd
Mahalanobis distance is a great alternative to Euclidean distance, especially if you're trying to quantify how unusual an element is within a population. Euclidean distances can get confused by strong correlation along dimensions, while Mahalanobis more accurately understands if something is out of distribution because it's "off axis". (See diagram with dotted contour lines of equal Mahalanobis distance. The red dot is clearly an outlier in Mahalanobis distance, but if you use standard L2/Euclidean distance to measure how far it is from the centroid then there's nothing unusual about it.)
I've been thinking through how to speed up applying this technique at scale, and last night in a dream it dawned on me that if you do PCA on your data first to reduce the dimensionality, then the Euclidean distances are the same as Mahalanobis distances - as long as you do the "whitening" step in PCA. Because then the covariance matrix is the identity, and the distribution is fully isotropic / spherical.
One of these realizations that if you can follow what I'm talking about is perhaps already obvious to you. But it wasn't to me. Maybe this nugget of high-dimensional statistical intuition might be useful for you in some ML problem? -
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
What's the clearest example of neuroscience influencing AI? One might argue that the very idea of artificial neural networks is brain inspired. People quibble about just how realistic ANNs are compared to biological neural networks (they are not, but that's not the point)... There is, however, a more indisputable example and it's the basis for convolutional neural networks (CNNs) and modern computer vision.
One can draw a line between Hubel and Wiesel's Nobel Prize-winning work in the 1960s on vision in cats (and their accidental discovery of an edge-detecting cell in the cat's visual cortex) to AlexNet, which was the first ANN to win the ImageNet competition in 2012!
The ideas of translational invariance (a vertical edge, for e.g., is a vertical edge regardless of where it is in an image), rotational invariance (an edge is an edge regardless of whether it's vertical or horizontal or somewhere in-between), and the hierarchical combination of simple features to form complex features--all these were first elucidated in neuroscience, and would heavily influence the development of CNNs.
The influence of neuroscience is one of the narrative arcs in WHY MACHINES LEARN: from the early work of McCulloch and Pitts, who came up with the first model of an artificial neuron, to Frank Rosenblatt's perceptron (a neuron that could learn), Hopfield networks (designed to model associative memories in brains) and modern day CNNs
All this and so much more is detailed in WHY MACHINES LEARN:
US: www.penguinrandomhouse.com/books/677608/why-machines-lear…
UK: www.penguin.co.uk/books/446849/why-machines-learn-by-anan… -
Anil Ananthaswamy reposted
Tom Yeh
@ProfTomYeh
I flipped through 300 pages of deep learning math puzzles by hand ✍️ This is the proof copy of the "Deep Learning Math Workbook" I made to help you learn and practice 12 foundational math concepts below👇
-
Anil Ananthaswamy
@anilananth
Things are really getting serious, if even the folks behind @arcprize are nodding at the success of @OpenAI's o3, the insanely high cost of compute notwithstanding ... Something has shifted in the reasoning stakes since the release of o1 and the use of test-time compute!
Mike Knoop
@mikeknoop
o3 is really special and everyone will need to update their intuition about what AI can/cannot do.
while these are still early days, this system shows a genuine increase in intelligence, canaried by ARC-AGI
semiprivate v1 scores:
* GPT-2 (2019): 0%
* GPT-3 (2020): 0%
* GPT-4 (2023): 2%
* GPT-4o (2024): 5%
* o1-preview (2024): 21%
* o1 high (2024): 32%
* o1 Pro (2024): ~50%
* o3 tuned low (2024): 76%
* o3 tuned high (2024): 87%
given i put in the original $1M @arcprize, i'd like to re-affirm my previous commitment. we will keep running the grand prize competition until an efficient 85% solution is open sourced.
but our ambitions are greater! ARC Prize found its mission this year -- to be an enduring north star towards AGI.
the ARC benchmark design principle is to be easy for humans, hard for AI and so long as there remain things in that category, there is more work to do for AGI.
there are >100 tasks from the v1 family unsolved by o3 even on the high compute config which is very curious.
successors to o3 will need to reckon with efficiency. i expect this to become a major focus for the field. for context, o3 high used 172x more compute than o3 low which itself used 100-1000x more compute than the grand prize competition target.
we also started work on v2 in earnest this summer (v2 is in the same grid domain as v1) and will launch it alongside ARC Prize 2025. early testing is promising even against o3 high compute. but the goal for v2 is not to make an adversarial benchmark, rather be interesting and high signal towards AGI.
we also want AGI benchmarks that can endure many years. i do not expect v2 will. and so we've also starting turning attention to v3 which will be very different. im excited to work with OpenAI and other labs on designing v3.
given it's almost the end of the year, im in the mood for reflection.
as anyone who has spent time with the ARC dataset can tell you, there is something special about it. and even moreso about a system than can fully beat it. we are seeing glimpses of that system with the o-series.
i mean it when i say these are early days. i believe o3 is the alexnet moment for program synthesis. we now have concrete evidence that deep-learning guided program search works.
we are staring up another mountain that, from my vantage point, looks equally tall and important as deep learning for AGI.
many things have surprised me this year, including o3. but the biggest surprise has been the increasing response to ARC Prize.
i've been surveying AI researchers about ARC for years. before ARC Prize launched in June, only one in ten had heard of it.
now it's objectively the spear tip benchmark, being used by spear tip labs, to demonstrate progress on the spear tip of AGI -- the most important technology in human history.
@fchollet deserves recognition for designing such an incredible benchmark.
i'm continually grateful for the opportunity to steward attention towards AGI with ARC Prize and we'll be back in 2025! -
Anil Ananthaswamy reposted
Sanjeev Arora
@prfsanjeevarora
@QuantaMagazine featured our work on emergence of skill compositionality (and its limitations) in LLMs among the CS breakthroughs of the year. www.youtube.com/watch?v=fTMMsreAqX0. Work was done over 2023 @GoogleDeepMind and @PrincetonPLI. Key pieces: (i) mathematical framework for quantifying how LLM scaling leads to predictable increase in the model’s ability to combine skills while solving new tasks. Joint work with @anirudhg9119 (ii) experiments verifying theoretical prediction in experiments via SkillMix evaluation (lead author @dingliy_yu) (iii) the level of skill-compositionality detected in GPT4O via Sept'23 experiments mathematically imply that it is able to reason and talk about situations it has not seen in its training data —i.e. it has moved beyond the “stochastic parrots” stereotype that had dogged earlier LLMs.
Skill emergence paper: arxiv.org/abs/2307.15936
Skillmix Evaluation: arxiv.org/abs/2310.17567
Models can improve skill composition from examples arxiv.org/abs/2409.19808
Wonderful to work with the colleagues and students involved. -
Anil Ananthaswamy
@anilananth
Would it surprise you to learn that the algorithm used to train the simplest possible artificial neural network with just 1 hidden layer and 1 hidden neuron is exactly the same as the algorithm to train the largest deep neural networks (with 100s of hidden layers and millions of hidden neurons)?
The algorithm: Backpropagation, the workhorse of modern AI.
As @andrewgwils has pointed out elsewhere, the essential idea underlying backpropagation goes back to Newton! Applying it to train artificial neural networks has its own history.
What does it mean to train a neural network? Training involves giving a network an input and then calculating the difference or "loss" between the output network should produce and the output it does produce. This loss is formulated as a function of the weights of the neurons in the network.
The job of the algorithm is to figure out how much each weight is responsible for the overall error: a problem called credit assignment. If there's no hidden layer, this problem is simple. Just calculate the gradient of the function when the network makes a loss, use the gradient to change all the weights just a wee bit, and reiterate, until you end up with low or zero loss.
But if you have even one hidden neuron, the solution isn't as straightforward. At least it seemed so to neural network researchers for decades, until it all turned out to be rather simple, in hindsight of course.
You have to use the chain rule--one of the most basic rules in calculus--to calculate the gradient w.r.t. each weight, and work one's way backwards from the output layer to the input layer (hence the term backpropagation).
Crucially, all the numbers you need for calculating these gradients have already been computed during the forward pass, when the network turns an input into an erroneous output. You just have to store those numbers and use them appropriately.
So, whether it's the simplest network (top left panel) or a more complicated one (top right panel), the algorithm is exactly the same! Modern packages such as @PyTorch have autograd engines that make this process seamless.
If you want the learn the history of how we got here, from Frank Rosenblatt's ideas from the early 1960s to @geoffreyhinton's 1986 paper with Rumelhart and Williams, you'll find it all in WHY MACHINES LEARN, with all the necessary basic calculus to make sense of this simple yet powerful algorithm.
US: www.penguinrandomhouse.com/books/677608/why-machines-lear…
UK: www.penguin.co.uk/books/446849/why-machines-learn-by-anan… -
Anil Ananthaswamy
@anilananth
.@FreelanceAstro has been sending me pictures of sightings of WHY MACHINES LEARN in the wild (Dog Eared Books Valencia, SF), @BeaconHillBook (Boston)...and there's always been the same book next to mine: @sunilamrith's The Burning Earth). So, it was amazing hear Sunil speak at the @BlrLitFest, in conversation with @HariniNagendra, about his book: reminded me of why I wanted to be a writer in first place. Hugely inspirational!
-
Anil Ananthaswamy reposted
Paul Middlebrooks
@pgmid
25 yrs ago, perception dominated neuroscience, and Raj's @RajeshPNRao predictive coding theory.
Now, it's all about action-perception loops across the hierarchy... "active predictive coding."
Also, augmenting our cognition with AI neural co-processors!
braininspired.co/podcast/201/ -
Anil Ananthaswamy reposted
Ananyo Bhattacharya
@Ananyo
A friend has secured a copy of the Russian edition of ‘The Man from the Future’, which is on sale now! Out of all of John von Neumann’s work, not sure why they chose to emphasise the Prisoners’ Dilemma 😅
-
Anil Ananthaswamy reposted
Parul Pandey
@pandeyparul
As the year comes to a close, I'm continuing my tradition of sharing the books that I read over the past months. Here's my reading journey for 2024 in no particular order:
• The Golden Road: How Ancient India Transformed the World by @DalrympleWill
Explores India’s influence on global culture, religion, and technology from 250 BCE to 1200 CE
• AI Snake Oil 𝐛𝐲 @random_walker 𝐚𝐧𝐝 @sayashk
A critical exploration of AI's real capabilities versus its exaggerated claims.
• Why Machines Learn: The Elegant Math Behind Modern AI by @anilananth
The book presents the underlying mathematics behind the ML algorithms along with interesting historical anecdotes.
• Why Zebras Don't Get Ulcers by Robert M. Sapolsky
Explores the effects of chronic stress on human health, contrasting it with the short-term stress responses seen in animals.
• Good Energy by @CaseyMeansMD
Another health book 😀 .Your lifestyle choices determine your metabolic health, which plays a crucial role in overall well-being.
• Same as Ever by @morganhousel
A collection of engaging stories that show how, while the world changes, certain human tendencies remain constant.
• Klara and the Sun by Kazuo Ishiguro
The book Explores Klara, an AI-powered Artificial Friend, and its perception of humanity while serving as a companion to a sick young girl in a near-future world.
Other than these, I also read few books online ⬇️ -
Anil Ananthaswamy reposted
Sasha Luccioni, PhD 🦋🌎✨🤗
@SashaMTL
"In 2025, AI and climate change, two of the biggest societal disruptors we're facing, will collide... What we really need is more transparency around AI’s environmental impacts" 🌍⚡💧
Thank you @WIRED for featuring my predictions (and hopes!):
www.wired.com/story/true-cost-generative-ai-data-centers-… -
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
Sometimes, when writing a book, you encounter an idea so mind-bending, it doesn't leave you. For me, it was the answer this question: Why are deep neural networks universal function approximators? Turns out, understanding why functions are vectors is key; it's a truly eye-opening exercise.
First: let's make sense of functions as vectors (are they really?)
Let’s take a function 𝘺 = 𝘤𝘰𝘴(2𝘹)
The top left panel shows this function for values of x between 0 and 10. Let's only evaluate the function for values of x = 0, 1, 2, …., 10. So, 11 points on the x-axis.
The function evaluates to these values: [𝟏 -𝟎.𝟒𝟐 -𝟎.𝟔𝟓 𝟎.𝟗𝟔 -𝟎.𝟏𝟓 -𝟎.𝟖𝟒 𝟎.𝟖𝟒 𝟎.𝟏𝟒 -𝟎.𝟗𝟔 𝟎.𝟔𝟔 𝟎.𝟒𝟏]
Those 11 values for the function, evaluated at specific values of x, constitutes a vector in 11D space. You can take any other function of x, and similarly evaluate it for the exact same 11 values of x, and you get another vector in the same 11D space. You can see how, in this limited sense, any function of x can be turned into a unique vector in 11D space.
Now, we have to stretch our imagination. Imagine evaluating the function at arbitrarily small intervals, but still between x = 0 and x=10. Eventually, you’ll get an infinity of values. Now, the 𝐟𝐮𝐧𝐜𝐭𝐢𝐨𝐧 𝐢𝐬 𝐚 𝐯𝐞𝐜𝐭𝐨𝐫 𝐢𝐧 𝐢𝐧𝐟𝐢𝐧𝐢𝐭𝐞 𝐝𝐢𝐦𝐞𝐧𝐬𝐢𝐨𝐧𝐚𝐥 𝐬𝐩𝐚𝐜𝐞.
Extend the x-axis to minus infinity and plus infinity. The function evaluated at an infinity of points along this infinite axis is a vector in another infinite-dimensional space!
The examples have dealt with a function that takes in a scalar input, x, and produces a scalar output, y. You can generalize this to functions that take in a vector x as input and produce a vector y as output.
What does this have to do with 𝐝𝐞𝐞𝐩 𝐧𝐞𝐮𝐫𝐚𝐥 𝐧𝐞𝐭𝐰𝐨𝐫𝐤𝐬 and their ability to approximate any function? Here’s one view of what a deep net does: it takes in a vector 𝐱 as input and produces a vector 𝐲 as output, such that the function f () captures the correlations in the training data. During training, the neural network learns to approximate this function.
𝐲 = 𝘧 (𝐱)
The universal approximation theorem essentially proves that, given certain assumptions, a neural network can convert any vector that lives in a vector space that characterizes the inputs into any vector that lives in some other vector space that characterizes the outputs.
It’s not a trivial theorem (though it might seem so to some, in hindsight; it wasn't so in the late 1980s) …
The story of this amazing advance is narrated in 𝐖𝐇𝐘 𝐌𝐀𝐂𝐇𝐈𝐍𝐄𝐒 𝐋𝐄𝐀𝐑𝐍. More here:
𝐔𝐒: penguinrandomhouse.com/books/677608/why-machines-learn-by…
𝐔𝐊: penguin.co.uk/books/446849/why-machines-learn-by-ananthas… -
Anil Ananthaswamy
@anilananth
What's the clearest example of neuroscience influencing AI? One might argue that the very idea of artificial neural networks is brain inspired. People quibble about just how realistic ANNs are compared to biological neural networks (they are not, but that's not the point)... There is, however, a more indisputable example and it's the basis for convolutional neural networks (CNNs) and modern computer vision.
One can draw a line between Hubel and Wiesel's Nobel Prize-winning work in the 1960s on vision in cats (and their accidental discovery of an edge-detecting cell in the cat's visual cortex) to AlexNet, which was the first ANN to win the ImageNet competition in 2012!
The ideas of translational invariance (a vertical edge, for e.g., is a vertical edge regardless of where it is in an image), rotational invariance (an edge is an edge regardless of whether it's vertical or horizontal or somewhere in-between), and the hierarchical combination of simple features to form complex features--all these were first elucidated in neuroscience, and would heavily influence the development of CNNs.
The influence of neuroscience is one of the narrative arcs in WHY MACHINES LEARN: from the early work of McCulloch and Pitts, who came up with the first model of an artificial neuron, to Frank Rosenblatt's perceptron (a neuron that could learn), Hopfield networks (designed to model associative memories in brains) and modern day CNNs
All this and so much more is detailed in WHY MACHINES LEARN:
US: www.penguinrandomhouse.com/books/677608/why-machines-lear…
UK: www.penguin.co.uk/books/446849/why-machines-learn-by-anan… -
Anil Ananthaswamy
@anilananth
Ilya Sutskever at @NeurIPSConf talking about the end of pre-training, because despite the growth in compute, the "fossil fuel of AI", meaning the data on the internet, is not! www.youtube.com/watch?v=1yvBqasHLZs
-
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
For all the fuss about who invented backpropagation, it's worth remembering that the Frank Rosenblatt--who designed the first single-layer artificial neural network, or the perceptron, back in the late 1950s--identified the problem of training multi-layer neural networks, or multi-layer perceptrons (MLPs).
In his 1961 tome, The Principles of Neurodynamics, Rosenblatt clearly elucidates the idea that errors from the final layer will have to be "back propagated" to earlier layers, if such networks could be trained. He just didn't know how!
It took a series of small but significant tweaks to Rosenblatt's artificial neuron--for example tweaking the step thresholding function (which caused a neuron to suddenly switch its output) to something more continuous, such as a Sigmoid function-- that led to the seminal Rumelhart, Hinton and Williams paper, which clearly showed how backpropagation could be used to train a multi-layer perceptron to learn feature representations.
There were many people between Rosenblatt on the one hand and Rumelhart, Hinton and Williams on the other, whose contributions were important to the development of deep learning.
All this and more is detailed in WHY MACHINES LEARN, in a chapter called The Algorithm That Put Paid To A Persistent Myth (the myth being the one propagated by Minsky and Papert that even multi-layer perceptrons could not solve the simple XOR problem).
You can find more at:
US: www.penguinrandomhouse.com/books/677608/why-machines-lear…
UK: www.penguin.co.uk/books/446849/why-machines-learn-by-anan… -
Anil Ananthaswamy reposted
Alex Bellos
@alexbellos
🎄🎄My new book is ONLY A FIVER for today and to celebrate my publishers made me a Xmas JPEG 🎄🎄
You can get it here: www.amazon.co.uk/Think-Twice-Simple-Puzzles-Everyone/dp/1… -
Anil Ananthaswamy
@anilananth
For all the fuss about who invented backpropagation, it's worth remembering that the Frank Rosenblatt--who designed the first single-layer artificial neural network, or the perceptron, back in the late 1950s--identified the problem of training multi-layer neural networks, or multi-layer perceptrons (MLPs).
In his 1961 tome, The Principles of Neurodynamics, Rosenblatt clearly elucidates the idea that errors from the final layer will have to be "back propagated" to earlier layers, if such networks could be trained. He just didn't know how!
It took a series of small but significant tweaks to Rosenblatt's artificial neuron--for example tweaking the step thresholding function (which caused a neuron to suddenly switch its output) to something more continuous, such as a Sigmoid function-- that led to the seminal Rumelhart, Hinton and Williams paper, which clearly showed how backpropagation could be used to train a multi-layer perceptron to learn feature representations.
There were many people between Rosenblatt on the one hand and Rumelhart, Hinton and Williams on the other, whose contributions were important to the development of deep learning.
All this and more is detailed in WHY MACHINES LEARN, in a chapter called The Algorithm That Put Paid To A Persistent Myth (the myth being the one propagated by Minsky and Papert that even multi-layer perceptrons could not solve the simple XOR problem).
You can find more at:
US: www.penguinrandomhouse.com/books/677608/why-machines-lear…
UK: www.penguin.co.uk/books/446849/why-machines-learn-by-anan… -
Anil Ananthaswamy reposted
Annapurni Subramaniam
@fiddlingstars
Had a great conversation with @anilananth @SLSingh and Venki Ramakrishnan, amazingly moderated by Mukund Thattai at @BlrLitFest this evening.
@IIABengaluru #blrlitfest -
Anil Ananthaswamy reposted
Joel David Hamkins
@JDHamkins
Need a gift for the aspiring young mathematician in your life? Proof and the Art of Mathematics overflows with fascinating assertions having interesting, elementary proofs. Learn how to write proofs and indeed how to be a mathematician.
mitpress.mit.edu/9780262539791/proof-and-the-art-of-mathe… #proofandtheart -
Anil Ananthaswamy reposted
Ryan.M.Nefdt
@ryan_nefdt
On a flight from Geneva to Zurich a few days ago,the plane starts shaking & wings flap like a seagull landing in a storm. All I had on me was @anilananth new book (shorturl.at/q1T2l). Can officially say the math can be turbulent at times but no less elegant for it.Thx Anil!
-
Anil Ananthaswamy
@anilananth
Why do many ML researchers think that deep learning is challenging established statistical learning theory? Here are some reasons:
1) Standard ML theory says that there is an optimal balance between model simplicity (too few parameters) and complexity (too many parameters), relative to the instances of training data. This is the famous bias-variance tradeoff. Deep learning seems to flout this well-established rule.
I say "seems" because not everyone agrees, and there's evidence that other ML algorithms (such as AdaBoost) were also doing so before deep learning entered the picture, but nobody paid much attention.
2) The phenomenon of Double Descent. This is related to the bias-variance tradeoff. In ML, interpolation is defined as the point at which a ML model gets so complex that it overfits the training data (sort of memorizes it) and so does very poorly on the held-out test data. It doesn't generalize!
The amazing thing is that in deep learning, if you keep increasing the number of parameters, and hence the model's capacity for complex solutions, it becomes over-parameterized. And yet, the loss on the held-out test data starts falling or descending down to zero. Hence the name double descent.
To be fair, some researchers are not convinced that this is not anything more than just an artefact of the choice of hyperparameters. Even so, it's something that needs explaining.
3) The phenomenon of Grokking. This is behavior that has been characterized in very simple deep neural networks, which are trying to solve types of problems--such as modulo arithmetic--that lend themselves to analytic solutions. Researchers at OpenAI stumbled upon grokking by accident (and others have since replicated it): if you keep training a deep neural network well past the stage at which it has ostensibly overfit the data, it seems to suddenly "discover" a generalized solution. Regularization may be at work.
All these issues might be facets of the same problem, which is this: Why do deep neural networks generalize so well despite being heavily over-parameterized?
To really appreciate the issue, it's worth getting acquainted with standard machine learning--it's the historical journey that's brings the maximum payoff. That's exactly what WHY MACHINES LEARN does.
For more, please see:
US: www.penguinrandomhouse.com/books/677608/why-machines-lear…
UK: www.penguin.co.uk/books/446849/why-machines-learn-by-anan… -
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
What? Linear algebra and calculus and machine learning for the holidays! Might a math-y book be a good gift to give someone for the holidays? I hope so :-)
“A masterpiece.” —Geoffrey Hinton, 2024 Nobel Laureate and Turing Award winner
“After just a few minutes of reading Why Machines Learn, you’ll feel your own synaptic weights getting updated. By the end you will have achieved your own version of deep learning—with deep pleasure and insight along the way.” —Steven Strogatz, NYT bestselling author and professor of mathematics at Cornell University
“Comprehensive yet engaging book...An essential read for understanding both the possibilities and limitations of artificial intelligence.” —Sabine Hossenfelder, physicist and NYT bestselling author
“A masterful work...this book will be an invaluable companion for anyone who wants a deep understanding of what’s under the hood of these often inscrutable machines.” —Melanie Mitchell, author of Artificial Intelligence and Professor at the Santa Fe Institute
US: www.penguinrandomhouse.com/books/677608/why-machines-lear…
UK: www.penguin.co.uk/books/446849/why-machines-learn-by-anan… -
Anil Ananthaswamy reposted
Science in Parallel
@scienceinparall
New episode 🎙️with @anilananth discussing the math, science and challenges of #ai, the 2024 physics @NobelPrize and his book, Why Machines Learn. scienceinparallel.org/2024/12/anil-ananthaswamy-ais-nobel… #hpc #ai
-
Anil Ananthaswamy reposted
J Brew
@BrewNeuron
I am working my way through the excellent book "Why Machines Learn" by @anilananth using the Audible version along with notes in a print copy. Slow and steady progress with this method, every morning, inspired by @martinmrmar
-
Anil Ananthaswamy
@anilananth
A 2 hr-long ruminative chat about WHY MACHINES LEARN and other things (including the nature of the human self) with @MLStreetTalk's @ecsquendor : www.youtube.com/watch?v=URtF_UHYBSo
-
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
The Conceptual Simplicity of Machine Learning
Machine Learning, once you grok it, is conceptually simple. But it hardly seems so when you begin, especially if you were once a software engineer—as I was—who worked on non-ML code. My first encounters with ML were confusing at best. It took the writing of WHY MACHINES LEARN for me to see its conceptual simplicity (even if the details can be extremely gnarly). Here’s ML distilled into key concepts:
1. First, a machine learning (ML) model—like any other piece of software—turns an input into some desired output. The main difference between non-ML software and machine learning is that traditional software transforms the input into the desired output using algorithms designed and implemented by a programmer, but machine learning examines patterns in training data and learns the algorithm to do the transformation. The key words here are: learning, patterns and training data.
2. This transformation—no matter how complicated—can be thought of as an outcome of providing an input to a function f (), which produces an output. So, x = f (y). Note that x and y are both vectors (they can be, of course, simply scalars, but using vectors makes it more general). So, the function f () transforms some input vector into the desired output vector. But how do you learn the requisite function f ()?
For more, see the blog post about the 13 conceptual ideas that can help us make sense of machine learning and generative AI: anilananthaswamy.com/why-machines-learn-codebook-blog/con… -
Anil Ananthaswamy
@anilananth
The Conceptual Simplicity of Machine Learning
Machine Learning, once you grok it, is conceptually simple. But it hardly seems so when you begin, especially if you were once a software engineer—as I was—who worked on non-ML code. My first encounters with ML were confusing at best. It took the writing of WHY MACHINES LEARN for me to see its conceptual simplicity (even if the details can be extremely gnarly). Here’s ML distilled into key concepts:
1. First, a machine learning (ML) model—like any other piece of software—turns an input into some desired output. The main difference between non-ML software and machine learning is that traditional software transforms the input into the desired output using algorithms designed and implemented by a programmer, but machine learning examines patterns in training data and learns the algorithm to do the transformation. The key words here are: learning, patterns and training data.
2. This transformation—no matter how complicated—can be thought of as an outcome of providing an input to a function f (), which produces an output. So, x = f (y). Note that x and y are both vectors (they can be, of course, simply scalars, but using vectors makes it more general). So, the function f () transforms some input vector into the desired output vector. But how do you learn the requisite function f ()?
For more, see the blog post about the 13 conceptual ideas that can help us make sense of machine learning and generative AI: anilananthaswamy.com/why-machines-learn-codebook-blog/con… -
Anil Ananthaswamy reposted
Dmitry Krotov
@DimaKrotov
What a wonderful day of Nobel lectures and inspiring conversations at the beautiful Aula Magna, Stockholm University! #nobelprize @Stockholms_univ
-
Anil Ananthaswamy reposted
CR
@C__TFR
Just tore through "Why Machines Learn" by @anilananth and I'm convinced it's a must-read for anyone in AI/ML (including non-technical). Here's why: 🧵👇
-
Anil Ananthaswamy reposted
Arto Bendiken
@bendiken
The book I've handed out the most here in Dubai this quarter, including to government representatives. Still got several hardcopies left in the batch, come to @ASIMOVMeetup this Friday to claim your copy!
Anil Ananthaswamy
@anilananth
What? Linear algebra and calculus and machine learning for the holidays! Might a math-y book be a good gift to give someone for the holidays? I hope so :-)
“A masterpiece.” —Geoffrey Hinton, 2024 Nobel Laureate and Turing Award winner
“After just a few minutes of reading Why Machines Learn, you’ll feel your own synaptic weights getting updated. By the end you will have achieved your own version of deep learning—with deep pleasure and insight along the way.” —Steven Strogatz, NYT bestselling author and professor of mathematics at Cornell University
“Comprehensive yet engaging book...An essential read for understanding both the possibilities and limitations of artificial intelligence.” —Sabine Hossenfelder, physicist and NYT bestselling author
“A masterful work...this book will be an invaluable companion for anyone who wants a deep understanding of what’s under the hood of these often inscrutable machines.” —Melanie Mitchell, author of Artificial Intelligence and Professor at the Santa Fe Institute
US: www.penguinrandomhouse.com/books/677608/why-machines-lear…
UK: www.penguin.co.uk/books/446849/why-machines-learn-by-anan… -
Anil Ananthaswamy reposted
Pedro Sarmento
@umpedronosapato
Really great read🤘 I think that framing CS concepts historically like @anilananth does, with the intervients being more than just "a name in the formula", makes learning so much more engaging and grounded
-
Anil Ananthaswamy reposted
Jeremy Berman
@jeremyberman
I just got first place on the public ARC-AGI benchmark using Claude Sonnet 3.5 and Evolutionary Test-time Compute
ARC Prize
@arcprize
2024 ARC-AGI-Pub SoTA! 👾
53.6% @jeremyberman
47.5% MARA(BARC) + MIT (@ellisk_kellis, @akyurekekin)
43.0% @ryangreenblatt -
Anil Ananthaswamy
@anilananth
What? Linear algebra and calculus and machine learning for the holidays! Might a math-y book be a good gift to give someone for the holidays? I hope so :-)
“A masterpiece.” —Geoffrey Hinton, 2024 Nobel Laureate and Turing Award winner
“After just a few minutes of reading Why Machines Learn, you’ll feel your own synaptic weights getting updated. By the end you will have achieved your own version of deep learning—with deep pleasure and insight along the way.” —Steven Strogatz, NYT bestselling author and professor of mathematics at Cornell University
“Comprehensive yet engaging book...An essential read for understanding both the possibilities and limitations of artificial intelligence.” —Sabine Hossenfelder, physicist and NYT bestselling author
“A masterful work...this book will be an invaluable companion for anyone who wants a deep understanding of what’s under the hood of these often inscrutable machines.” —Melanie Mitchell, author of Artificial Intelligence and Professor at the Santa Fe Institute
US: www.penguinrandomhouse.com/books/677608/why-machines-lear…
UK: www.penguin.co.uk/books/446849/why-machines-learn-by-anan… -
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
𝐌𝐲 𝐥𝐢𝐬𝐭 𝐨𝐟 𝐬𝐨𝐦𝐞 𝐞𝐧𝐠𝐚𝐠𝐢𝐧𝐠, 𝐢𝐥𝐥𝐮𝐦𝐢𝐧𝐚𝐭𝐢𝐧𝐠 𝐚𝐧𝐝 𝐢𝐧𝐜𝐫𝐞𝐝𝐢𝐛𝐥𝐲 𝐮𝐬𝐞𝐟𝐮𝐥 𝐯𝐢𝐝𝐞𝐨 𝐥𝐞𝐜𝐭𝐮𝐫𝐞𝐬 𝐨𝐧 𝐦𝐚𝐜𝐡𝐢𝐧𝐞 𝐥𝐞𝐚𝐫𝐧𝐢𝐧𝐠
I could not have written 𝐖𝐡𝐲 𝐌𝐚𝐜𝐡𝐢𝐧𝐞𝐬 𝐋𝐞𝐚𝐫𝐧 without having had the luxury of listening to online lectures over and over--until the rust of two decades of having stayed away from coding began to wear off, and nitty-gritty of the math of machine learning began to make sense, and the joy of being able to code returned.
Here are some of those lectures that were absolutely key to my learning:
𝐌𝐚𝐜𝐡𝐢𝐧𝐞 𝐋𝐞𝐚𝐫𝐧𝐢𝐧𝐠 𝐨𝐯𝐞𝐫𝐚𝐥𝐥
My absolute favorite: 𝐊𝐢𝐥𝐢𝐚𝐧 𝐖𝐞𝐢𝐧𝐛𝐞𝐫𝐠𝐞𝐫'𝐬 𝟐𝟎𝟏𝟖 𝐜𝐥𝐚𝐬𝐬 𝐨𝐧 𝐦𝐚𝐜𝐡𝐢𝐧𝐞 𝐥𝐞𝐚𝐫𝐧𝐢𝐧𝐠 𝐚𝐭 𝐂𝐨𝐫𝐧𝐞𝐥𝐥: www.youtube.com/playlist?list=PLl8OlHZGYOQ7bkVbuRthEsaLr7…
Weinberger is a masterful teacher!
Not too far behind: 𝐀𝐧𝐝𝐫𝐞𝐰 𝐍𝐠'𝐬 𝐬𝐭𝐮𝐝𝐞𝐧𝐭 𝐀𝐧𝐚𝐧𝐝 𝐀𝐯𝐚𝐭𝐢 𝐭𝐞𝐚𝐜𝐡𝐢𝐧𝐠 𝐚𝐧 𝐌𝐋 𝐜𝐨𝐮𝐫𝐬𝐞 𝐢𝐧 𝐭𝐡𝐞 𝐬𝐮𝐦𝐦𝐞𝐫 𝐨𝐟 𝟐𝟎𝟏𝟗: www.youtube.com/playlist?list=PLoROMvodv4rNH7qL6-efu_q2_b…
His first few lectures setting up the math of ML are gems.
And, of course, lectures by 𝐀𝐧𝐝𝐫𝐞𝐰 𝐍𝐠 himself. There are so many... Just Google.
For 𝐋𝐢𝐧𝐞𝐚𝐫 𝐀𝐥𝐠𝐞𝐛𝐫𝐚:
The wonderful 𝐆𝐢𝐥𝐛𝐞𝐫𝐭 𝐒𝐭𝐫𝐚𝐧𝐠 (who was teaching this well before linear algebra became cool): www.youtube.com/playlist?list=PLPQAixVXrNTUw464KbIvuvH-86…
𝐅𝐨𝐫 𝐃𝐚𝐭𝐚 𝐒𝐜𝐢𝐞𝐧𝐜𝐞 𝐦𝐞𝐭𝐡𝐨𝐝𝐬 𝐚𝐧𝐝 𝐜𝐨𝐧𝐜𝐞𝐩𝐭𝐬
𝐁𝐢𝐧𝐠 𝐁𝐫𝐮𝐧𝐭𝐨𝐧, 𝐒𝐭𝐞𝐯𝐞 𝐁𝐫𝐮𝐧𝐭𝐨𝐧 𝐚𝐧𝐝 𝐍𝐚𝐭𝐡𝐚𝐧 𝐊𝐮𝐭𝐳, 𝐨𝐟 𝐭𝐡𝐞 𝐔𝐧𝐢𝐯𝐞𝐫𝐬𝐢𝐭𝐲 𝐨𝐟 𝐖𝐚𝐬𝐡𝐢𝐧𝐠𝐭𝐨𝐧, 𝐒𝐞𝐚𝐭𝐭𝐥𝐞:
www.youtube.com/playlist?list=PLMrJAkhIeNNQz4BMoGSsN8cbt8…
Search for other videos by these three. Many gems to be found.
𝐏𝐫𝐨𝐛𝐚𝐛𝐢𝐥𝐢𝐭𝐲
A quirky set of bite-sized videos on key concepts in probability theory by 𝐑𝐢𝐜𝐡 𝐑𝐚𝐝𝐤𝐞, 𝐩𝐫𝐨𝐟𝐞𝐬𝐬𝐨𝐫 𝐚𝐭 𝐑𝐏𝐈, 𝐓𝐫𝐨𝐲, 𝐍𝐘. www.youtube.com/playlist?list=PLuh62Q4Sv7BXkeKW4J_2WQBlYh…
If you want a masterclass on 𝐒𝐮𝐩𝐩𝐨𝐫𝐭 𝐕𝐞𝐜𝐭𝐨𝐫 𝐌𝐚𝐜𝐡𝐢𝐧𝐞𝐬, there's none better than 𝐏𝐚𝐭𝐫𝐢𝐜𝐤 𝐖𝐢𝐧𝐬𝐭𝐨𝐧'𝐬 𝐌𝐈𝐓 lecture:
www.youtube.com/watch?v=_PwhiWxHK8o
If you can persist with these lectures--and then keep dipping into the never-ending resource that is YouTube --you'll be well set to get under the mathematical skin of ML.
www.penguinrandomhouse.com/books/677608/why-machines-lear… -
Anil Ananthaswamy reposted
Joscha Bach
@Plinz
I don't think that the scaling hypothesis gets recognized enough for how radical it is. For decades, AI sought some kind of master algorithm of intelligence. The scaling hypothesis says that there is none: intelligence is the ability to use more compute on more data.
-
Anil Ananthaswamy reposted
Domesticated Brain
@rasangarocks
Why Machines Learn: The Elegant Math Behind Modern AI
Link - www.amazon.com/Why-Machines-Learn-Elegant-Behind/dp/05931…
#machinelearning #ai #artificialintelligence #algorithms #datascience #maths #deeplearning #programming -
Anil Ananthaswamy reposted
Yoshua Bengio
@Yoshua_Bengio
Recently answered @anilananth's questions for @Nature. No matter when it arrives, AGI and the road to reach it will both help tackle thorny problems (e.g. climate change and diseases), and pose huge risks. Understanding and transparency are key.
www.nature.com/articles/d41586-024-03905-1 -
Anil Ananthaswamy
@anilananth
The debate of this moment in AI: Are LLMs displaying "Sparks" of AGI or the "Embers" of autoregression? @SebastienBubeck and @RTomMcCoy will debate each other at the @SimonsInstitute, Dec 5. Register here to watch live: simons.berkeley.edu/web-registration-simons-institute-fal…
-
Anil Ananthaswamy reposted
Kevin Mitchell
@WiringTheBrain
If you're looking for a Christmas present, books are always good... 😊😉🎅 press.princeton.edu/books/paperback/9780691204154/innate, press.princeton.edu/books/paperback/9780691226217/free-ag… (just sayin'...)
-
Anil Ananthaswamy reposted
Julio Rodriguez Martino
@JCRM_AI
How close is AI to human-level intelligence?
Large language models such as OpenAI’s o1 have electrified the debate over achieving artificial general intelligence, or AGI. But they are unlikely to reach this milestone on their own.
By @anilananth in @Nature -
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
Will simply scaling up LLMs get us to AGI? My feature for @Nature, with inputs from @fchollet, @Yoshua_Bengio, @RaiaHadsell, @andrewgwils, @dileeplearning, @keyonV, @MelMitchell1 and Karl Friston. www.nature.com/articles/d41586-024-03905-1
-
Anil Ananthaswamy
@anilananth
Will simply scaling up LLMs get us to AGI? My feature for @Nature, with inputs from @fchollet, @Yoshua_Bengio, @RaiaHadsell, @andrewgwils, @dileeplearning, @keyonV, @MelMitchell1 and Karl Friston. www.nature.com/articles/d41586-024-03905-1
-
Anil Ananthaswamy reposted
Steve Jurvetson
@FutureJurvetson
The Moore's Law Update
NOTE: this is a semi-log graph, so a straight line is an exponential; each y-axis tick is 100x. This graph covers a 1,000,000,000,000,000,000,000x improvement in computation/$. Pause to let that sink in.
Humanity’s capacity to compute has compounded for as long as we can measure it, exogenous to the economy, and starting long before Intel co-founder Gordon Moore noticed a refraction of the longer-term trend in the belly of the fledgling semiconductor industry in 1965.
I have color coded it to show the transition among the integrated circuit architectures. You can see how the mantle of Moore's Law has transitioned most recently from the GPU (green dots) to the ASIC (yellow and orange dots), and the NVIDIA Hopper architecture itself is a transitionary species — from GPU to ASIC, with 8-bit performance optimized for AI models, the majority of new compute cycles.
There are thousands of invisible dots below the line, the frontier of humanity's capacity to compute (e.g., everything from Intel in the past 15 years). The computational frontier has shifted across many technology substrates over the past 128 years. Intel ceded leadership to NVIDIA 15 years ago, and further handoffs are inevitable.
Why the transition within the integrated circuit era? Intel lost to NVIDIA for neural networks because the fine-grained parallel compute architecture of a GPU maps better to the needs of deep learning. There is a poetic beauty to the computational similarity of a processor optimized for graphics processing and the computational needs of a sensory cortex, as commonly seen in the neural networks of 2014. A custom ASIC chip optimized for neural networks extends that trend to its inevitable future in the digital domain. Further advances are possible with analog in-memory compute, an even closer biomimicry of the human cortex. The best business planning assumption is that Moore’s Law, as depicted here, will continue for the next 20 years as it has for the past 128. (Note: the top right dot for Mythic is a prediction for 2026 showing the effect of a simple process shrink from an ancient 40nm process node)
----
For those unfamiliar with this chart, here is a more detailed description:
Moore's Law is both a prediction and an abstraction. It is commonly reported as a doubling of transistor density every 18 months. But this is not something the co-founder of Intel, Gordon Moore, has ever said. It is a nice blending of his two predictions; in 1965, he predicted an annual doubling of transistor counts in the most cost effective chip and revised it in 1975 to every 24 months. With a little hand waving, most reports attribute 18 months to Moore’s Law, but there is quite a bit of variability. The popular perception of Moore’s Law is that computer chips are compounding in their complexity at near constant per unit cost. This is one of the many abstractions of Moore’s Law, and it relates to the compounding of transistor density in two dimensions. Others relate to speed (the signals have less distance to travel) and computational power (speed x density).
Unless you work for a chip company and focus on fab-yield optimization, you do not care about transistor counts. Integrated circuit customers do not buy transistors. Consumers of technology purchase computational speed and data storage density. When recast in these terms, Moore’s Law is no longer a transistor-centric metric, and this abstraction allows for longer-term analysis.
What Moore observed in the belly of the early IC industry was a derivative metric, a refracted signal, from a longer-term trend, a trend that begs various philosophical questions and predicts mind-bending AI futures.
In the modern era of accelerating change in the tech industry, it is hard to find even five-year trends with any predictive value, let alone trends that span the centuries.
I would go further and assert that this is the most important graph ever conceived. A large and growing set of industries depends on continued exponential cost declines in computational power and storage density. Moore’s Law drives electronics, communications and computers and has become a primary driver in drug discovery, biotech and bioinformatics, medical imaging and diagnostics. As Moore’s Law crosses critical thresholds, a formerly lab science of trial and error experimentation becomes a simulation science, and the pace of progress accelerates dramatically, creating opportunities for new entrants in new industries. Consider the autonomous software stack for Tesla and SpaceX and the impact that is having on the automotive and aerospace sectors.
Every industry on our planet is going to become an information business. Consider agriculture. If you ask a farmer in 20 years’ time about how they compete, it will depend on how they use information — from satellite imagery driving robotic field optimization to the code in their seeds. It will have nothing to do with workmanship or labor. That will eventually percolate through every industry as IT innervates the economy.
Non-linear shifts in the marketplace are also essential for entrepreneurship and meaningful change. Technology’s exponential pace of progress has been the primary juggernaut of perpetual market disruption, spawning wave after wave of opportunities for new companies. Without disruption, entrepreneurs would not exist.
Moore’s Law is not just exogenous to the economy; it is why we have economic growth and an accelerating pace of progress. At Future Ventures, we see that in the growing diversity and global impact of the entrepreneurial ideas that we see each year — from automobiles and aerospace to energy and chemicals.
We live in interesting times, at the cusp of the frontiers of the unknown and breathtaking advances. But, it should always feel that way, engendering a perpetual sense of future shock. -
Anil Ananthaswamy
@anilananth
Are there theoretical limitations to what deep learning can achieve? On @MLStreetTalk, we talk of nature having evolved biological neural networks, and while there are enormous differences between artificial and biological neural networks, the latter are a proof-of-principle that amazing things are possible with self-supervised learning ...
youtube.com/clip/UgkxKJshKO-cUOF1Xeh3UqDaJP3S0dTdYqwr -
Anil Ananthaswamy reposted
Brian Roemmele
@BrianRoemmele
Some lines, your mind creates everything else.
-
Anil Ananthaswamy
@anilananth
"ChatGPT taught us something that I don't think we'd have learned any other way...which is that, in some sense, language is a closed system sort of in the same way that arithmetic is closed under integers." @TonyZador on @pgmid's BrainInspired podcast. Really worth a listen!
-
Anil Ananthaswamy
@anilananth
"As far as I can tell, transformers are almost a counter-example to the successes of Neuro-AI" This conversation between @pgmid and @TonyZador on the Brain Inspired podcast is an absolute treat! youtu.be/9C0qkxu0UyE
-
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
"Will there be math? Oh, yes, there will be math. But Ananthaswamy is the best guide you could ask for on such a perilous journey."
This line in @theinformation about WHY MACHINES LEARN, named as one of the 5 best AI books of 2024, made me smile! I'm thrilled that an equation-laden book on ML is part of this list. Go linear algebra and calculus! :-)
www.theinformation.com/articles/best-ai-books-of-2024-how… -
Anil Ananthaswamy
@anilananth
"Will there be math? Oh, yes, there will be math. But Ananthaswamy is the best guide you could ask for on such a perilous journey."
This line in @theinformation about WHY MACHINES LEARN, named as one of the 5 best AI books of 2024, made me smile! I'm thrilled that an equation-laden book on ML is part of this list. Go linear algebra and calculus! :-)
www.theinformation.com/articles/best-ai-books-of-2024-how… -
Anil Ananthaswamy
@anilananth
Somewhere in Canada, in one slice of the fine-grained categories of books, @maxsbennett's A Brief History of Intelligence and Why Machines Learn are in the extremely august company of Hofstadter's Gödel, Escher, Bach. Thrilling to be alongside the two of them!
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
“Why do we care about interpretability?” @JacobSteinhardt at the Special Year on Large Language Models and Transformers, Part 1 Boot Camp at the Simons Institute. “In principle...we can predict OOD behavior, repair models and improve our understanding.” www.youtube.com/watch?v=gh7tff8roKw
-
Anil Ananthaswamy
@anilananth
When we talk of LLMs hallucinating, it's worth remembering that even our brains are hypothesizing (and hence hallucinating) about external reality, and we mostly get it right. But when we don't, the results are nuts! Watch this video about the Ames Window, particularly the nonsensical perceptions that arise towards the end with the pencil and the window frame!
youtu.be/0KrpZMNEDOY?t=93 -
Anil Ananthaswamy reposted
Tim Kietzmann
@TimKietzmann
Seeing the biological neurons of the device grow into the brain of the subject is exciting... and somehow super freaky.
Max Hodak
@maxhodak_
I had an idea in college which I've long thought could be, in many ways, the ultimate BCI technology. What if instead of using electrodes, we used biological neurons embedded in electronics to communicate with the brain?
Enter biohybrid neural interfaces: science.xyz/news/biohybrid-neural-interfaces/ -
Anil Ananthaswamy
@anilananth
So, @MLStreetTalk asked why my book, Why Machines Learn, was a bit connectionist-centric. I defended the book :-), arguing that it was about many kinds of ML, not just neural networks. But also nodded to the fact that the lessons from Symbolic AI or Good Old-Fashioned AI might turn out to be important as we try and build machines that reason in ways that humans do.
youtube.com/clip/Ugkx4-XtxQyNPL8K1iUDVcLR8pDWR6riH5pZ -
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
Marginal Revolution (@MargRev) picks this year's best non-fiction. I'm both a little stunned and very glad that WHY MACHINES LEARN, a book that doesn't eschew math and equations, made the list and almost made the "two to top the list"!
Marginal Revolution's @tylercowen writes: 'Truly an outstanding list for this year...If I had to pick out two to top the list, perhaps they would be:
Annie Jacobsen, Nuclear War: A Scenario, and
Michael Cook, A History of the Muslim World: From its Origins to the Dawn of Modernity.
Also having a claim is Anil Ananthaswamy, Why Machines Learn: The Elegant Math Behind Modern AI, though for many readers the math will be too much.'
tylercowen
@tylercowen
Best non-fiction books of 2024: marginalrevolution.com/marginalrevolution/2024/11/best-no…
-
Anil Ananthaswamy reposted
Mario Krenn
@MarioKrenn6240
Great story in @QuantaMagazine by @anilananth about recent progress at the foundations of quantum mechanics, including paradoxa discovered by Renato Renner, Caslav Bruckner and others:
www.quantamagazine.org/in-the-quantum-world-even-points-o… -
Anil Ananthaswamy reposted
David J. Gunkel
@David_Gunkel
We have page proofs!
"Communicative #ai" @politybooks
@MCoeckelbergh & @David_Gunkel
Coming in 2025 -
Anil Ananthaswamy reposted
Sanjeev Arora
@prfsanjeevarora
Well-deserved!
Anil Ananthaswamy
@anilananth
Marginal Revolution (@MargRev) picks this year's best non-fiction. I'm both a little stunned and very glad that WHY MACHINES LEARN, a book that doesn't eschew math and equations, made the list and almost made the "two to top the list"!
Marginal Revolution's @tylercowen writes: 'Truly an outstanding list for this year...If I had to pick out two to top the list, perhaps they would be:
Annie Jacobsen, Nuclear War: A Scenario, and
Michael Cook, A History of the Muslim World: From its Origins to the Dawn of Modernity.
Also having a claim is Anil Ananthaswamy, Why Machines Learn: The Elegant Math Behind Modern AI, though for many readers the math will be too much.'
tylercowen
@tylercowen
Best non-fiction books of 2024: marginalrevolution.com/marginalrevolution/2024/11/best-no…
-
Anil Ananthaswamy
@anilananth
'Tis the season of giving thanks. On the professional front, my grateful mind harkens back to a time of uncertainty for all authors--when their yet-to-be-born book goes out for blurbs. It's usually the first time people outside one's inner circle are reading the whole book. It's a tense time. My thanks to everyone who blurbed, @geoffreyhinton @stevenstrogatz @MelMitchell1 @skdh, Peter Hart, Björn Ommer and @ArthurIMiller.
www.penguinrandomhouse.com/books/677608/why-machines-lear…
www.penguin.co.uk/books/446849/why-machines-learn-by-anan… -
Anil Ananthaswamy reposted
Tavis Rudd
@tavisrudd
Just finished @anilananth 's excellent new book. Highly recommended even if you already know the math. Worth it for the history alone.
-
Anil Ananthaswamy reposted
medici.tv
@medicitv
In June 2013, Russian piano superstar Grigory Sokolov gave his rare approval for his recital to be filmed. His last performance filmed live took place more than 10 years earlier! Enjoy his virtuoso rendition of Rameau’s Les Cyclopes. www.medici.tv/en/concerts/grigory-sokolov-schubert-beetho…
This media couldn't be displayed. See it on the original post. -
Anil Ananthaswamy
@anilananth
Marginal Revolution (@MargRev) picks this year's best non-fiction. I'm both a little stunned and very glad that WHY MACHINES LEARN, a book that doesn't eschew math and equations, made the list and almost made the "two to top the list"!
Marginal Revolution's @tylercowen writes: 'Truly an outstanding list for this year...If I had to pick out two to top the list, perhaps they would be:
Annie Jacobsen, Nuclear War: A Scenario, and
Michael Cook, A History of the Muslim World: From its Origins to the Dawn of Modernity.
Also having a claim is Anil Ananthaswamy, Why Machines Learn: The Elegant Math Behind Modern AI, though for many readers the math will be too much.'
tylercowen
@tylercowen
Best non-fiction books of 2024: marginalrevolution.com/marginalrevolution/2024/11/best-no…
-
Anil Ananthaswamy reposted
bunny.agr
@bunnyTheRobo001
Feeling inspired after watching @anilananth on @MLStreetTalk . Clear, engaging, and thought-provoking. A real treat for the curious mind! 🙌💡
#ai
www.youtube.com/watch?v=URtF_UHYBSo -
Anil Ananthaswamy
@anilananth
Here's my handle for the other place :)
@anilananth.bsky.social
I'd like to create a starter pack for AI/machine learning. Please reply to this with handles/names that I can include. -
Anil Ananthaswamy reposted
Michael Nielsen
@michael_nielsen
The higher up the tech tree you go, the less sense it makes to be a technological determinist. The space of possibilities is vastly larger than we can ever explore, and so choices about institutions and directions of exploration (and non-exploration) matter!
-
Anil Ananthaswamy
@anilananth
Why do some researchers think we have entered "Terra incognita" when it comes to deep learning and deep neural networks?
In the past year or so, at meetings on large language models, I had conversations with ML researchers who couldn't help wonder if the way ML researchers feel about their field today is not unlike how classical physicists felt in the early 1900s when confronted by quantum mechanics: a sense of both wonder and befuddlement.
In standard statistical machine learning theory, you don't want your model to be so complex (by having, say, too many parameters) that you overfit the training data, and hence don't generalize well to test data.
But deep neural networks have revealed something unexpected: even though the networks are highly over-parameterized (meaning, they have way more parameters than instances of training data), the networks don't overfit! They still do well on test data.
Now, let's say you add noise to the training data (say, you randomly corrupt the labels associated with images), and train the network to zero training loss. You have, by definition, overfit--because you have learned the characteristics of the noise. But even in this case--as long as the noise is not too large--such a network still does acceptably well on uncorrupted test data. How come?
ML researchers had hints even before the advent of deep neural networks that standard ML theory was not quite the full story (there was, for example, the behavior of the AdaBoost algorithm that showed similar behavior). It's as if deep neural networks have shone a bright light on a big piece of the puzzle about generalization in ML models.
This puzzling behavior has been given different names (depending on what exactly one is studying): benign overfitting, double descent, grokking, and so on. And to be fair, not everyone agrees that there's any real puzzle here to be solved. The fact that there is disagreement makes this fertile ground for future work: truly Terra incognita.
This is one of the major narrative arcs in WHY MACHINES LEARN: going from early days of ML and standard ML theory (single layer perceptrons, k-NN, SVMs and so on) to deep neural networks. The final chapter brings us to this place of (possible) mystery.
More here:
UK: www.penguin.co.uk/books/446849/why-machines-learn-by-anan…
US: www.penguinrandomhouse.com/books/677608/why-machines-lear… -
Anil Ananthaswamy
@anilananth
Classical reference frames gave us special relativity and more. Could quantum reference frames help make sense of quantum paradoxes and maybe even lead us to a theory of quantum gravity? My story for @QuantaMagazine
www.quantamagazine.org/in-the-quantum-world-even-points-o… -
Anil Ananthaswamy reposted
Loreto Parisi
@loretoparisi
📚Now they are in good company! #ai #llm
Simone Scardapane
@s_scardapane
@loretoparisi I didn't see this one, thanks for sharing! 😍
-
Anil Ananthaswamy reposted
Aran Nayebi
@aran_nayebi
Without new ideas, every exponential is a sigmoid in disguise. So the real question is: Where does the sigmoid flatten for current model architectures?
More specifically, for a given domain (vision, robotics, etc), does performance "tap out" at levels where we can still collect great data?
These would be great target questions for AI theory to make informed predictions for. -
Anil Ananthaswamy reposted
bluesharp
@bluezharp
Larkin Poe 🎶'Preachin' Blues' (Son House)
This media couldn't be displayed. See it on the original post. -
Anil Ananthaswamy
@anilananth
LLMS are always hallucinating. My take on @MLStreetTalk on why the use of the word "hallucination" to talk of an LLM's output only when it's factually wrong is missing the point: right or wrong, they are always using the same process!
youtube.com/clip/UgkxgFILb01JfrC1ICmiAemKee71Lj5RRaRs -
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
"Can you explain the backprop algorithm and its history?" Another oh-so-simple question from
@MLStreetTalk's Tim Scarfe :-) We talk of an idea that goes back to Rosenblatt's 1960s book, The Principle of Neurodynamics, and the subsequent developments
youtu.be/URtF_UHYBSo?t=3170 -
Anil Ananthaswamy reposted
Mukul Dekhane
@dekhane_mukul
I had to eat the pakodas to read the entire article...
😁😜😁😜 -
Anil Ananthaswamy reposted
Alfredo Canziani
@alfcnz
Aww 🥹🥹🥹
So many of you found me! 💙💙💙
🦋🦋🦋 -
Anil Ananthaswamy
@anilananth
GPT giving a succinct answer to what might make it AGI :-)
Amin Karbasi
@aminkarbasi
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
Deep learning practitioners have focused their attention on an optimization regime that's "unstable and convergent"--something that's not suggested by theory when using gradient methods, says Peter Bartlett, during his Richard M. Karp Distinguished Lecture at the Simons Institute
-
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
This weekend a dear friend took out his copies of my four books--and it's the first time I saw them stacked together. Served as a lovely reminder that it's worth pursuing something that at first is only visible faintly on the horizon. I'd have never have in a million years imagined I'd write these books when I started off as a journalist.
-
Anil Ananthaswamy
@anilananth
"Can you explain the backprop algorithm and its history?" Another oh-so-simple question from
@MLStreetTalk's Tim Scarfe :-) We talk of an idea that goes back to Rosenblatt's 1960s book, The Principle of Neurodynamics, and the subsequent developments
youtu.be/URtF_UHYBSo?t=3170 -
Anil Ananthaswamy reposted
MySportsAI
@SmarterSig
Why Machines Learn
www.youtube.com/watch?v=URtF_UHYBSo -
Anil Ananthaswamy
@anilananth
This weekend a dear friend took out his copies of my four books--and it's the first time I saw them stacked together. Served as a lovely reminder that it's worth pursuing something that at first is only visible faintly on the horizon. I'd have never have in a million years imagined I'd write these books when I started off as a journalist.
-
Anil Ananthaswamy reposted
Max S. Bennett
@maxsbennett
This book by @anilananth truly is a masterpiece. One of my fav parts of my day has been gobbling up a chapter or two each night
-
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
The Unreasonable Effectiveness of Linear Algebra. If you really want to understand what’s happening in machine learning/AI, coming to grips with the basics of linear algebra is super important.
I continue to be blown away by how much of ML becomes increasingly intuitive as one gains an appreciation for the linear algebra that powers ML—in particular, developing a geometric intuition for what’s happening when dealing with vectors, matrices and linear transformations.
In WHY MACHINES LEARN, I introduce the relevant aspects of linear algebra, without being too textbook-like about it. Some of my favorite parts of the book have linear algebra at their core:
1) The perceptron convergence proof. I was awed by its simplicity, and it’s what made me want to write WHY MACHINES LEARN
2) The chapter on principal component analysis and developing intuitions about eigenvectors, eigenvalues, covariance matrices, etc. has dollops of linear algebra.
3) Finding the optimal margin when building a linear classifier —work that led to support vector machines—also rests on some lovely linear algebra.
4) This year’s @NobelPrize for physics went in part to John Hopfield, for Hopfield networks. The proof that shows that when a Hopfield net is disturbed, it’ll dynamically finds its way to an energy minimum, which represents some stored memory, requires linear algebra!
5) And of course, the Universal Approximation Theorem, which says that deep neural networks with arbitrarily wide hidden layers will learn to approximate any function, is all about vector spaces and functions as vectors, etc. --so, yes, linear algebra again.
As @andrewgwils said in a recent talk at the @SimonsInstitute: “Machine Learning Is Linear Algebra”!
You can find more about WHY MACHINES LEARN here:
US: penguinrandomhouse.com/books/677608/why-machines-learn-by…
UK: penguin.co.uk/books/446849/why-machines-learn-by-ananthas… -
Anil Ananthaswamy reposted
Melanie Mitchell
@MelMitchell1
For Science Magazine, I wrote about "The Metaphors of Artificial Intelligence".
The way you conceptualize AI systems affects how you interact with them, do science on them, and create policy and apply laws to them.
Hope you will check it out!
www.science.org/doi/full/10.1126/science.adt6140?af=R -
Anil Ananthaswamy reposted
Tim Kietzmann
@TimKietzmann
I'd hate to see Sush leave. Do hire him, however, he is fantastic.
-
Anil Ananthaswamy
@anilananth
"Do machine learning models reason? And if they do reason, why do you think they reason and how do you think their reasoning is different to ours?" One of many pointed questions from @MLStreetTalk during our in-depth conversation about WHY MACHINES LEARN. www.youtube.com/watch?v=URtF_UHYBSo&t=3553s
-
Anil Ananthaswamy reposted
Edgar Guevara
@guevaracodina
Equally recommendable! Both @anilkseth and @anilananth have the great gift of explaining complex stuff in a very clear and engaging fashion. I don't know if that generalizes well to all people named Anil 😜
Anil Ananthaswamy
@anilananth
Engaging discussion on AI with @HelloFrankLavin... Frank asked me to recommend a book, and I happen to be rereading and loving @anilkseth's Being You, so I recommended it!
Frank Lavin
@HelloFrankLavin
Latest podcast is up. Fascinating ai discussion with @anilananth : booksandinsight.libsyn.com/anil-ananthaswamy-author-of-wh…
-
Anil Ananthaswamy
@anilananth
Engaging discussion on AI with @HelloFrankLavin... Frank asked me to recommend a book, and I happen to be rereading and loving @anilkseth's Being You, so I recommended it!
Frank Lavin
@HelloFrankLavin
Latest podcast is up. Fascinating ai discussion with @anilananth : booksandinsight.libsyn.com/anil-ananthaswamy-author-of-wh…
-
Anil Ananthaswamy reposted
Frank Lavin
@HelloFrankLavin
Latest podcast is up. Fascinating ai discussion with @anilananth : booksandinsight.libsyn.com/anil-ananthaswamy-author-of-wh…
-
Anil Ananthaswamy
@anilananth
The myriad ways to understand and appreciate the curse of dimensionality, and why it matters in machine learning.
The American applied mathematician Richard Bellman coined the phrase in his 1957 book, Dynamic Programming, calling the weird and intractable behavior of data in higher dimensions “a curse which has hung over the head of the physicist and astronomer.”
To understand how this impacts machine learning algorithms, let’s take the simple k-nearest neighbor method. The k-NN rule turns every instance of data into a vector with, say, d elements and then relies on some similarity/distance metric to decide if one instance of data is like another: the closer they are in the d-dimensional space, the more alike they are. You can classify new data by looking at its nearest neighbors.
The algorithm works great until d starts getting very, very large. It turns out that in extremely high-dimensional spaces, the distance metric falls apart. All vectors start becoming equally distant from each other. Similarity or dissimilarity has no meaning!
There’s another way to appreciate this curse. Say your ML algorithm is trying to learn some non-linear surface that either separates two clusters of data or represents the probability distribution over data. If it’s learning the latter, then you can sample from the learned distribution to generate new data that resembles the training data, giving you generative AI.
But, here’s the problem. As the dimensionality of your training data increases, the number of data samples you need in order to accurately estimate this surface grows exponentially—so much so that very soon it becomes computationally intractable.
Deep neural networks—despite having billions of parameters and hence seemingly functioning in rather high dimensional spaces—defy this curse. There’s considerable ongoing work that tries to explain/understand why deep learning breaks the curse of dimensionality.
Such issues and more are discussed in detail in WHY MACHINES LEARN.
US: www.penguinrandomhouse.com/books/677608/why-machines-lear…
UK: www.penguin.co.uk/books/446849/why-machines-learn-by-anan… -
Anil Ananthaswamy
@anilananth
Looking forward to digging into this book by @rasbt : www.amazon.com/Build-Large-Language-Model-Scratch/dp/1633…
-
Anil Ananthaswamy reposted
davidlee
@davidlee
I’m really enjoying it!
-
Anil Ananthaswamy
@anilananth
A talk about scaling intelligence the human way, from someone who thinks deeply about this issue: Josh Tenebaum. www.youtube.com/watch?v=lMcHDCXMvgI
-
Anil Ananthaswamy reposted
Celik Nimani
@celiknimani
Back in school days you would learn about Vectors, Matrices and Algebra with no real sense of how they connect in today’s world! “Why Machines Learn” by @anilananth ties it all together. A bit different from my reading habits but definitely worth it! www.amazon.com/dp/0593185749?tag=digjitale-20
-
Anil Ananthaswamy
@anilananth
"Can you explain the curse of dimensionality?" The type of simple question you get asked when being interviewed on Machine Learning Street Talk (@MLStreetTalk). What if I had said, "No, I have no idea?" ... :-) But, of course, I gave an answer ...
youtu.be/URtF_UHYBSo?t=2399 -
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
I love how @MLStreetTalk gets stuck into the work of its show's guests. Here, @ecsquendor and I chat about the writing of WHY MACHINES LEARN and the bias-variance tradeoff, self-supervised learning, and so on ... A 2hr-long talk, and it's not even as nerdy as many of the other ML Street Talk episodes that I truly enjoyed and learned so much from! www.youtube.com/watch?v=URtF_UHYBSo
-
Anil Ananthaswamy reposted
deter3
@deter3
Why Machines Learn , this book is the best book to have a comprehensive understanding for the history of AI till existing LLMs , really amazing job from the author ,Anil Ananthaswamy.
www.amazon.com/Why-Machines-Learn-Elegant-Behind/dp/05931… -
Anil Ananthaswamy
@anilananth
I love how @MLStreetTalk gets stuck into the work of its show's guests. Here, @ecsquendor and I chat about the writing of WHY MACHINES LEARN and the bias-variance tradeoff, self-supervised learning, and so on ... A 2hr-long talk, and it's not even as nerdy as many of the other ML Street Talk episodes that I truly enjoyed and learned so much from! www.youtube.com/watch?v=URtF_UHYBSo
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
On the heels of Noam Brown's (@polynoamial) Sep talk on reasoning in LLMs, especially about @OpenAI's o1 using inference-time compute, @srush_nlp gives the Richard M. Karp Distinguished Lecture at the Simons Institute on test-time scaling. Nov 8, 11AM PT
simons.berkeley.edu/events/speculations-test-time-scaling… -
Anil Ananthaswamy reposted
Ann Finkbeiner
@AnnFinkbeiner
Richard Panek, BFF of LWON, wrote a book about the Webb, the JWST, the thing that let us see those pillars of creation. That's a pretty damn big subject, how did he know what to leave out? why include what he did? As you'd expect, Richard has answers. www.lastwordonnothing.com/2024/11/08/conversation-richard…
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
Join us Friday at 11 a.m. PT.
simons.berkeley.edu/events/speculations-test-time-scaling… -
Anil Ananthaswamy
@anilananth
When someone asks me my motivation for writing Why Machines Learn, I can reply wirh this quote :) Seems very apt for a book on ML/AI!
Ilya Kaminsky
@ilya_poet
"the duty of the writer ... is to remind us that we will die. And that we aren’t dead yet."
--Solmaz Sharif -
Anil Ananthaswamy reposted
Nikolay Kukushkin
@niko_kukushkin
We think that all memory is stored in the brain. But our study published today in @NatureComms shows that all cells—even kidney cells—can count, detect patterns, store memories, and do so similarly to brain cells. My first (co)corresponding author paper!🧵www.nature.com/articles/s41467-024-53922-x
-
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
"The Revolution Will Not Be Supervised" I talk of Alexei Efros's evocative statement in response to @MLStreetTalk's question about whether the revolution in AI will be supervised or unsupervised. WHY MACHINES LEARN: The Elegant Math Behind Machine Learning youtube.com/clip/Ugkx6f23n_plrTw-UVrWDdlVwdrEWB9XRCN5 via @YouTube
-
Anil Ananthaswamy reposted
Loreto Parisi
@loretoparisi
📚 “Why Machines Learn
The Elegant Math Behind Modern Al” just arrived!
@anilananth 🙏 -
Anil Ananthaswamy
@anilananth
"The Revolution Will Not Be Supervised" I talk of Alexei Efros's evocative statement in response to @MLStreetTalk's question about whether the revolution in AI will be supervised or unsupervised. WHY MACHINES LEARN: The Elegant Math Behind Machine Learning youtube.com/clip/Ugkx6f23n_plrTw-UVrWDdlVwdrEWB9XRCN5 via @YouTube
-
Anil Ananthaswamy
@anilananth
Here's a simple, beautiful intuition for why neural networks with at least one hidden layer are universal function approximators: meaning they can model (to some arbitrary accuracy) any function that converts an input to an output.
Recall integral calculus from high-school or early undergrad days. The area under a curve can be approximated by adding up the area of a number of rectangles that you can fit under the curve. The greater the number of rectangles, the closer you get to the actual area under the curve. For example, see figures showing 15 and 60 rectangles under the shown curve.
Now, imagine a neural network with one hidden layer (see figure). This network takes in a scalar input, x, and produces a scalar output, y. So, it's approximating some function y = f (x).
The essential idea (which is beautifully explained in Michael Nielson's interactive guide in Neural Networks and Deep Learning) is that pairs of neurons can combine to produce an output that resembles one rectangle: meaning the output for the pair is zero everywhere, except for the region represented by the width of one rectangle.
The more such pairs of neurons you have, the more rectangles (or to be precise, just the tops of the rectangles) you are producing. If you add them up--or linearly combine them, which is what the output neuron is doing--you get an approximation of the overall curve.
All this and more--along with the story of George Cybenko, who came up with one of the first proofs that a neural network with a single hidden layer can approximate any function--is described in WHY MACHINES LEARN:
US Edition: www.penguinrandomhouse.com/books/677608/why-machines-lear…
UK Edition: www.penguin.co.uk/books/446849/why-machines-learn-by-anan… -
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
.@MLStreetTalk asks about the basic mathematical disciplines needed to grasp machine learning. A clip from our conversation about WHY MACHINES LEARN:
youtube.com/clip/UgkxK5nXF6I_z4G5-MCd_1xdlr7BkMazSyEX via @YouTube -
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
My fav ML podcast, Machine Learning Street Talk
@MLStreetTalk w/ Tim Scarfe (@ecsquendor) has a new episode, one in which I talk at length about WHY MACHINES LEARN, and even about an earlier book, THE MAN WHO WASN'T THERE.
An unexpected pleasure was finding out that the cameraman, Marcus White, has done stunts in Tom Cruise's 𝐌𝐢𝐬𝐬𝐢𝐨𝐧 𝐈𝐦𝐩𝐨𝐬𝐬𝐢𝐛𝐥𝐞 and was the stunt double for another Hollywood superstar in an upcoming big-budget blockbuster!
We recorded this 2hr-long episode during an 8hr layover in London, as I was flying from the US to India. Was a truly lovely experience.
www.youtube.com/watch?v=URtF_UHYBSo -
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
My fav ML podcast, Machine Learning Street Talk
@MLStreetTalkw/ Tim Scarfe (@ecsquendor) has a new episode, one in which I talk at length about WHY MACHINES LEARN, and even about an earlier book, THE MAN WHO WASN'T THERE. youtube.com/watch?v=URtF_UHYBSo
An unexpected pleasure was finding out that the cameraman, Marcus White, has done stunts in Tom Cruise's 𝐌𝐢𝐬𝐬𝐢𝐨𝐧 𝐈𝐦𝐩𝐨𝐬𝐬𝐢𝐛𝐥𝐞 and was the stunt double for another Hollywood superstar in an upcoming big-budget blockbuster!
We recorded this 2hr-long episode during an 8hr layover in London, as I was flying from the US to India. Was a truly lovely experience. -
Anil Ananthaswamy reposted
Joshua Rothman
@joshuarothman
Profiling Dan Dennett was one of the peak experiences of my life. It wasn't just that his thinking changed mine, but that he was an inspiring human being, full of curiosity, vitality, and creativity. We should all try to be a little more like Dan. www.newyorker.com/magazine/2017/03/27/daniel-dennetts-sci…
-
Anil Ananthaswamy
@anilananth
.@MLStreetTalk asks about the basic mathematical disciplines needed to grasp machine learning. A clip from our conversation about WHY MACHINES LEARN:
youtube.com/clip/UgkxK5nXF6I_z4G5-MCd_1xdlr7BkMazSyEX via @YouTube -
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
"The term hallucination has often colloquially been used only when LMS get things wrong but if you look at the way LLMs function, everything that they're doing is essentially hallucinating and I think that word really loses its meaning if you realize that that's just how they work." A long, in-depth conversation about WHY MACHINES LEARN w/ @MLStreetTalk's Tim Scarfe (@ecsquendor).
www.youtube.com/watch?v=URtF_UHYBSo -
Anil Ananthaswamy
@anilananth
"The term hallucination has often colloquially been used only when LMS get things wrong but if you look at the way LLMs function, everything that they're doing is essentially hallucinating and I think that word really loses its meaning if you realize that that's just how they work." A long, in-depth conversation about WHY MACHINES LEARN w/ @MLStreetTalk's Tim Scarfe (@ecsquendor).
www.youtube.com/watch?v=URtF_UHYBSo -
Anil Ananthaswamy reposted
Bangalore Lit Fest
@BlrLitFest
Join us at #blrlitfest 2024 for a captivating journey through science with Anil Ananthaswamy (@anilananth) and Simon Singh (@SLSingh).
🗓 14-15 December 2024
📍 The Lalit Ashok, Bengaluru
Details on sessions coming soon! Register for the festival here: bangaloreliteraturefestival.org -
Anil Ananthaswamy reposted
Kevin Mitchell
@WiringTheBrain
Discussing Free Agents: How Evolution Gave Us Free Will With Author Kevin Mitchell www.youtube.com/live/jyFuBCfooj4 - a fun and lively chat, discussing cognition, selves, agency, control and related issues...
-
Anil Ananthaswamy
@anilananth
My fav ML podcast, Machine Learning Street Talk
@MLStreetTalkw/ Tim Scarfe (@ecsquendor) has a new episode, one in which I talk at length about WHY MACHINES LEARN, and even about an earlier book, THE MAN WHO WASN'T THERE. youtube.com/watch?v=URtF_UHYBSo
An unexpected pleasure was finding out that the cameraman, Marcus White, has done stunts in Tom Cruise's 𝐌𝐢𝐬𝐬𝐢𝐨𝐧 𝐈𝐦𝐩𝐨𝐬𝐬𝐢𝐛𝐥𝐞 and was the stunt double for another Hollywood superstar in an upcoming big-budget blockbuster!
We recorded this 2hr-long episode during an 8hr layover in London, as I was flying from the US to India. Was a truly lovely experience. -
Anil Ananthaswamy
@anilananth
My fav ML podcast, Machine Learning Street Talk
@MLStreetTalk w/ Tim Scarfe (@ecsquendor) has a new episode, one in which I talk at length about WHY MACHINES LEARN, and even about an earlier book, THE MAN WHO WASN'T THERE.
An unexpected pleasure was finding out that the cameraman, Marcus White, has done stunts in Tom Cruise's 𝐌𝐢𝐬𝐬𝐢𝐨𝐧 𝐈𝐦𝐩𝐨𝐬𝐬𝐢𝐛𝐥𝐞 and was the stunt double for another Hollywood superstar in an upcoming big-budget blockbuster!
We recorded this 2hr-long episode during an 8hr layover in London, as I was flying from the US to India. Was a truly lovely experience.
www.youtube.com/watch?v=URtF_UHYBSo -
Anil Ananthaswamy reposted
Robert Brooks IV
@boborado
“I hope your neurons enjoy this process as much as mine did.” @anilananth
-
Anil Ananthaswamy reposted
Andrew Gordon Wilson
@andrewgwils
I gave this talk at the "How Far Are We from AGI?" workshop. It was an interesting experience; most believed "AGI" is around the corner, while I think we are >50 years away. However, by embracing flexibility alongside compression, I believe we can build a general intelligence.
Andrew Gordon Wilson
@andrewgwils
My ICLR talk “How Do We Build a General Intelligence?” is now online!
www.youtube.com/watch?v=HEp4TOrkwV4 -
Anil Ananthaswamy reposted
bimri
@bimr1
right around when I thought the perfect career doesn't exist in this age of AI - Anil @anilananth comes by MLST.
definitely now going to devour these books. another MLST favorite pod episode.
"The Elegant Math Behind Machine Learning" youtu.be/URtF_UHYBSo via @MLStreetTalk -
Anil Ananthaswamy
@anilananth
Recently, I had the pleasure of being a guest of the @creativedlab's reading group, as they discussed WHY MACHINES LEARN. One of the members held up his well-thumbed copy of WML--and I had to share the pics :-) Thanks, Sasha Ivanov, @mr_sudo_
-
Anil Ananthaswamy reposted
Prof Mick Thacker
@dibbygibby
Buy it, borrow it , beg for it but whatever you do read or listen to this essential book ! Instant classic - thank you @anilananth for writing it!
-
Anil Ananthaswamy reposted
François Chollet
@fchollet
The first AGI system will be much worse than previous AI systems at pretty much everything -- except for tests that focus on measuring general intelligence (e.g. ARC).
Like a baby, it will be generally intelligent -- able to learn efficiently and adapt -- but will lack any specialized competence.
However, it will be able to learn using much less data, and the capabilities it picks up will generalize much more strongly to the variability and unpredictability of the real world. As a result, after a few years, it will beat previous AI systems at absolutely everything -- including many problems that were completely out of reach before. -
Anil Ananthaswamy reposted
Andrew Gordon Wilson
@andrewgwils
My ICLR talk “How Do We Build a General Intelligence?” is now online!
www.youtube.com/watch?v=HEp4TOrkwV4 -
Anil Ananthaswamy reposted
alex zhang
@a1zhang
Scared of DL systems 🎃?
I made a Meticulous Guide to Advances in Deep Learning Efficiency over the Years, which is a detailed story from pre-AlexNet to foundation model training centered on #efficient #deeplearning from the hardware, libraries, algorithms, compilers... 🧵(1/7) -
Anil Ananthaswamy reposted
Science Journalism Forum
@ScienceJF
An 𝗶𝗺𝗽𝗼𝗿𝘁𝗮𝗻𝘁 𝘁𝗮𝗸𝗲 by @anilananth, author of Why Machines Learn, from his interview with Ashley Smart Associate Director of the @KSJatMIT at the #sjf24 session, 𝗜𝗻𝘀𝗶𝗱𝗲 𝗔𝗜’𝘀 𝗕𝗹𝗮𝗰𝗸 𝗕𝗼𝘅: 𝗔 𝗖𝗼𝗻𝘃𝗲𝗿𝘀𝗮𝘁𝗶𝗼𝗻 𝗪𝗶𝘁𝗵 𝗔𝗻𝗶𝗹 𝗔𝗻𝗮𝗻𝘁𝗵𝗮𝘀𝘄𝗮𝗺𝘆, highlights the need for deeper understanding of AI beyond popular media coverage.
-
Anil Ananthaswamy reposted
Michael Hartl
@mhartl
I feel like this may prove to be the most prescient cartoon of the 21st century.
-
Anil Ananthaswamy
@anilananth
-
Anil Ananthaswamy
@anilananth
Teaching yourself machine learning using an AI code assistants is full of irony. While it's staggering how easy it is learn ML coding this way, the process also opens a window to what's at stake in terms of jobs and the future of at least some kinds of work.
Recently, @Google announced that "more than a quarter of all new code is generated by AI, then reviewed and accepted by engineers."
That's a lot of new code and person-hours of work replaced by AI code assistants. Granted, the code will have errors and experts are needed to ensure its correctness and efficacy. Still, the low-hanging fruits of many such jobs are on the line.
I got a taste of the power of this process when I used Claude 3.5 Sonnet to generate the code necessary to implement some of the algorithms described in WHY MACHINES LEARN. I had the relevant expertise to ensure the LLM was generating the correct code (since I had already done the coding for my book), but still I was pretty flabbergasted that the weeks and months I had spent coding for WHY MACHINES LEARN could be brought done to hours of effort with an LLM.
I detail the efforts in a handful of blog posts (only 3 examples so far; I have yet to finish doing this for the whole book):
1) From Rosenblatt to Claude: How to prompt an LLM to generate the code to understand the seminal perceptron algorithm, which was invented in 1958 by Frank Rosenblatt, and is the precursor to modern-day neural networks: anilananthaswamy.com/why-machines-learn-codebook-blog/fro…
2) The Monty Hall Problem: Could an LLM have convinced Paul Erdős?Ever wondered about the Monty Hall problem and how to solve it? Well, here's an LLM helping you generate the code and visualize it. anilananthaswamy.com/why-machines-learn-codebook-blog/mon…
3) LLM Prompts for Learning About Hopfield Networks. If you want a taste of the algorithm that won John Hopfield the @NobelPrize in Physics in 2024, have a look at this blog post: anilananthaswamy.com/why-machines-learn-codebook-blog/hop…
For the algorithms pertaining to these coding efforts and other algorithms:
US: penguinrandomhouse.com/books/677608/why-machines-learn-by…
UK:
penguin.co.uk/books/446849/why-machines-learn-by-ananthas… -
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
The Unbearable Strangeness of Neural Networks
Maybe one day we’ll fully understand why deep neural networks behave the way they do, but as of now their behavior stumps standard statistical machine learning theory—at least, many ML theorists think so.
The final chapter in WHY MACHINES LEARN gets us—after grounding us in the relevant math and history of standard ML—to a place where we can appreciate just what is so befuddling about neural networks. The section with the heading “The Unbearable Strangeness of Neural Networks” is one of my favorites.
Standard ML theory has for long relied on the bias-variance tradeoff: if your model is too simple, i.e. too few parameters relative to the instances of training data, then you risk under-fitting. Such models are said to have high bias. But if your model has too many parameters, then you risk overfitting. Such models are said to have high variance—and they don’t generalize well to unseen test data.
When training a model, you want to hit a sweet spot, i.e. find the best trade-off between bias and variance. “There is some sort of Goldilocks principle. Not too hot, not too cold,” said Misha Belkin of the University of California, San Diego (quoted in WHY MACHINES LEARN).
But here’s the thing: deep neural networks seem to flout this long-established way of thinking about models. Neural networks are heavily over-parameterized, meaning they have way more parameters than instances of training data. In theory, they should overfit and fail to generalize—or make predictions about—unseen test data (always keeping in mind that both training and test data are being drawn from the same distribution).
That’s not what we see in practice! Neural networks don’t overfit. There are numerous ideas that have been put forth to explain this behavior. WHY MACHINES LEARN gives the reader a glimpse of this terrain: which Belkin evocatively calls Terra Incognita.
In fact, not only do neural networks not overfit, they also display a phenomenon called grokking, where the networks discover generalizable solutions when trained for much, much longer than is usually done.
All this and more (such as the phenomenon of double descent) is discussed in WHY MACHINES LEARN.
US: penguinrandomhouse.com/books/677608/why-machines-learn-by…
UK: penguin.co.uk/books/446849/why-machines-learn-by-ananthas… -
Anil Ananthaswamy
@anilananth
The Unbearable Strangeness of Neural Networks
Maybe one day we’ll fully understand why deep neural networks behave the way they do, but as of now their behavior stumps standard statistical machine learning theory—at least, many ML theorists think so.
The final chapter in WHY MACHINES LEARN gets us—after grounding us in the relevant math and history of standard ML—to a place where we can appreciate just what is so befuddling about neural networks. The section with the heading “The Unbearable Strangeness of Neural Networks” is one of my favorites.
Standard ML theory has for long relied on the bias-variance tradeoff: if your model is too simple, i.e. too few parameters relative to the instances of training data, then you risk under-fitting. Such models are said to have high bias. But if your model has too many parameters, then you risk overfitting. Such models are said to have high variance—and they don’t generalize well to unseen test data.
When training a model, you want to hit a sweet spot, i.e. find the best trade-off between bias and variance. “There is some sort of Goldilocks principle. Not too hot, not too cold,” said Misha Belkin of the University of California, San Diego (quoted in WHY MACHINES LEARN).
But here’s the thing: deep neural networks seem to flout this long-established way of thinking about models. Neural networks are heavily over-parameterized, meaning they have way more parameters than instances of training data. In theory, they should overfit and fail to generalize—or make predictions about—unseen test data (always keeping in mind that both training and test data are being drawn from the same distribution).
That’s not what we see in practice! Neural networks don’t overfit. There are numerous ideas that have been put forth to explain this behavior. WHY MACHINES LEARN gives the reader a glimpse of this terrain: which Belkin evocatively calls Terra Incognita.
In fact, not only do neural networks not overfit, they also display a phenomenon called grokking, where the networks discover generalizable solutions when trained for much, much longer than is usually done.
All this and more (such as the phenomenon of double descent) is discussed in WHY MACHINES LEARN.
US: penguinrandomhouse.com/books/677608/why-machines-learn-by…
UK: penguin.co.uk/books/446849/why-machines-learn-by-ananthas… -
Anil Ananthaswamy reposted
John Hopfield
@HopfieldJohn
This award is very hard to respond to. I have received many hundred congratulatory notes, from former students, post-docs, Princeton University juniors and seniors, funding agencies and foundations, authors, signature collectors, amateurs, elementary school neural network followers, and on and on. An astonishing fraction of them has found their way into useful and interesting Neural Network careers by a casual interaction in class, at a meeting, hearing what I had to say about their ideas, learning from thinking about how I worked with a class, or from being my teaching assistants... There are some whom I remember well, and others for whom my reaction is “are they certain that our interaction sparked a single usable thought?” Yet they go on and comment “you changed my life” and follow on to explain that they heard me lecture when they were 15, and have been a member of the Neural Network brigade of the research army ever afterward.
I cannot make detailed comments to most of my letter writers. In sum I can only say that I tremendously enjoyed the interactions that the Neural Network community provided me with; that the mutual interactions have given me much pleasure over the years; that the community interested both in brain and in artificial brain has proved a good way for science to develop even if institutions have not always been sympathetic. Often these institutions found the enthusiasm infectious, after a period of doubt. In short, we often have won--. No, perhaps all we know is that we have not yet lost. I still believe that finding mind lodged in biological matter is the most profound question that physics can pose. And that the breadth of physics is a good base from which to begin. -
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
In "Machine Learning is Linear Algebra", @andrewgwils spoke of rethinking transformers for better computational scaling, by focusing on their MLP layers too, not just on attention. At the Simons Institute's workshop on Transformers as a Computational Model youtu.be/nplao1_t8ig
-
Anil Ananthaswamy reposted
Arto Bendiken
@bendiken
Read The Master Algorithm by @pmddomingos and/or Why Machines Learn by @anilananth for the backstory.
Nick Dobos
@NickADobos
Every time I learn more about ML & ai, it’s so simple and stupid
Like how did transformers take so long to figure out? Honestly no clue wtf the entire field was doing for 50 years. I see why the old guard is malding -
Anil Ananthaswamy
@anilananth
I remember reviewing @pmddomingos' The Master Algorithm for the New Scientist when it came out! How lovely to see Why Machines Learn mentioned alongside his book!
Arto Bendiken
@bendiken
Read The Master Algorithm by @pmddomingos and/or Why Machines Learn by @anilananth for the backstory.
Nick Dobos
@NickADobos
Every time I learn more about ML & ai, it’s so simple and stupid
Like how did transformers take so long to figure out? Honestly no clue wtf the entire field was doing for 50 years. I see why the old guard is malding -
Anil Ananthaswamy
@anilananth
The Unreasonable Effectiveness of Linear Algebra. If you really want to understand what’s happening in machine learning/AI, coming to grips with the basics of linear algebra is super important.
I continue to be blown away by how much of ML becomes increasingly intuitive as one gains an appreciation for the linear algebra that powers ML—in particular, developing a geometric intuition for what’s happening when dealing with vectors, matrices and linear transformations.
In WHY MACHINES LEARN, I introduce the relevant aspects of linear algebra, without being too textbook-like about it. Some of my favorite parts of the book have linear algebra at their core:
1) The perceptron convergence proof. I was awed by its simplicity, and it’s what made me want to write WHY MACHINES LEARN
2) The chapter on principal component analysis and developing intuitions about eigenvectors, eigenvalues, covariance matrices, etc. has dollops of linear algebra.
3) Finding the optimal margin when building a linear classifier —work that led to support vector machines—also rests on some lovely linear algebra.
4) This year’s @NobelPrize for physics went in part to John Hopfield, for Hopfield networks. The proof that shows that when a Hopfield net is disturbed, it’ll dynamically finds its way to an energy minimum, which represents some stored memory, requires linear algebra!
5) And of course, the Universal Approximation Theorem, which says that deep neural networks with arbitrarily wide hidden layers will learn to approximate any function, is all about vector spaces and functions as vectors, etc. --so, yes, linear algebra again.
As @andrewgwils said in a recent talk at the @SimonsInstitute: “Machine Learning Is Linear Algebra”!
You can find more about WHY MACHINES LEARN here:
US: penguinrandomhouse.com/books/677608/why-machines-learn-by…
UK: penguin.co.uk/books/446849/why-machines-learn-by-ananthas… -
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
Building on Moravec's Paradox, @YejinChoinka introduces the Generative AI Paradox—"What it can create, it may not understand"—in her talk at the Simons Institute's Special Year on Large Language Models and Transformers, Part 1 Boot Camp. Video of talk: www.youtube.com/live/4gvgWCY4eEU
-
Anil Ananthaswamy reposted
Dmitry Krotov
@DimaKrotov
There has been many discussions around Hopfield Networks of associative memory in the past couple of weeks. But what are the modern frontiers and recent developments in this field? If you want to learn more about this, please check out the NeurIPS 2023 workshop dedicated to this area of research. It’s a great collection of recent work pertaining to this topic. All the talks are recorded and freely available online: neurips.cc/virtual/2023/workshop/66524. One of the invited keynotes is from @HopfieldJohn himself.
Here is a website of the workshop with additional information: amhn.vizhub.ai. -
Anil Ananthaswamy reposted
Kevin Mitchell
@WiringTheBrain
Getting my arguments ready for this debate with Robert Sapolsky tomorrow! 😊 #teamfreewill
-
Anil Ananthaswamy reposted
Andrew Gordon Wilson
@andrewgwils
I couldn't agree more. Build your foundations: mathematics, physical intuitions, writing... these will never go obsolete. The most useful course I took in college superficially seemed the most useless: real analysis. But it changed my brain, and the way I filter the world.
Yann LeCun
@ylecun
Advice to students: study fundamental topics that have a long shelf life.
-
Anil Ananthaswamy reposted
Yann LeCun
@ylecun
Thank you IIT Madras for hosting me yesterday.
I got to take a selfie with the audience of my Subaru Suresh Distinguished Lecture.
Office of Global Engagement, IIT Madras
@ogeiitm
@iitmadras was proud to host @ylecun on campus yesterday. Yan LeCun from @Meta delivered an inspiring technical talk at IIT Madras and a public lecture at Music Academy, Chennai.
Special thanks to @kris_sg and Prof. Subra Suresh for their support in making this a success. Thank you, everyone, for making this event a big success!
#ai #techtalk #iitmadras #globalengagement @cerai_iitm @DeanacrIITM -
Anil Ananthaswamy reposted
Science Journalism Forum
@ScienceJF
Step inside the intriguing world of artificial intelligence at SJF24 with our session Inside AI’s Black Box: A Conversation with Anil Ananthaswamy!
Join us for an insightful discussion featuring award-winning science writer @anilananth , author of Why Machines Learn. Ananthaswamy unpacks the mathematical foundations of modern AI and explores the technology's potential and limitations. In conversation with Ashley Smart from the Knight Science Journalism Program, they will delve into Ananthaswamy’s journey in researching and writing his book, sharing valuable lessons for journalists aiming to navigate the complexities of AI.
Whether you're a journalist looking to enhance your reporting on AI or simply curious about how this technology works, this session promises to equip you with a deeper understanding of AI's intricacies.
Don't miss out on this opportunity to learn from the experts!
Grab your ticket now and stay tuned for more updates!
sciencejf.com/shop/
#sjf24 #artificialintelligence #sciencejournalism #ai #whymachineslearn -
Anil Ananthaswamy reposted
Subbarao Kambhampati (కంభంపాటి సుబ్బారావు)
@rao2z
Although I never got to "play" on the Music Academy stage in Chennai, some of our papers apparently did, thanks to @ylecun's lecture at my alma mater @iitmadras.. 😅
👉 www.youtube.com/watch?v=pjqKHOeykp8&t=2820s -
Anil Ananthaswamy reposted
Surya Ganguli
@SuryaGanguli
Phew. Just finished my @TEDAI2024 talk. Now for drinks.
Signals
@TEDAISF
@SuryaGanguli, Professor of Applied Physics at @Stanford | Venture Partner, Andreessen Horowitz @a16z covered How do we bridge the gap between human and AI? And how much we have a lot to learn about the evolution of quantum learning.
#tedai #tedai2024 #ai #tedtalk -
Anil Ananthaswamy reposted
Steven Strogatz
@stevenstrogatz
Infinite Powers is on sale for $1.99.
It's featured today on @eBookDaily ! ebookdaily.com/bargain-kindle-books/2024-10-22/B07FKF9DVJ -
Anil Ananthaswamy
@anilananth
This review of WHY MACHINES LEARN by
@ChristophMolnar raises an important issue of why authors write the books they do. mindfulmodeler.substack.com/p/book-review-why-machines-learn
Christoph says "I speculate that this book was a book [Anil] wanted to write, taking the risk of potential difficulties in defining a target audience."
He was spot-on in his assessment. This was a book I wanted to write without particularly worrying about the target audience, because that worry can be paralyzing. I knew there was the danger of satisfying neither the math-savvy reader nor those who wanted less or no math at all.
Writing a book requires much passion and persistence; it better be something you want to do for yourself as much as for your imagined reader.
Saying that, I remain pleasantly surprised that the book has found an audience--not one I had fully foreseen--and for that I'm thankful! -
Anil Ananthaswamy reposted
Ilka Diester
@ilka_diester
Potentially a promising tool for investigating astrocyte-neuron interactions.
IMBIT
@IMBIT_UniFR
Fred Menard from @UBC gave a fascinating BrainLinks-BrainTools talk on chemical tools for studying molecular events at synapses, bridging neuronal networks and molecular dynamics. A future bridge between molecular and systems neuroscience? @UniFreiburg @BUnifr #neuroscience
-
Anil Ananthaswamy reposted
Anna Ciaunica PhD @annaciaunica.bsky.social
@AnnaCiaunica
Thank you all for an excellent first day at our Interself conference in Lisbon
Hope to see you tomorrow for ani thé cool line up of talks 😎 -
Anil Ananthaswamy reposted
Joshua Howgego
@jdhowgego
Well, it's definitely real now! Here is the cover of my book, The Meteorite Hunters, out in February 2025. More about the book and how to pre-order here: linktr.ee/meteoritehunters
-
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
One of the most baffling things about the revolution in machine learning is the apparent ability of deep neural networks to against the grain of standard statistical machine learning theory.
Modern deep nets seem to be flouting a long-established rule in a machine learning: A model has to find the optimal balance between being too simple (too few parameters) and too complex (too many parameters) relative to the instances of training data, and thus avoid both under-fitting and overfitting.
I use words like “apparent” and “seem to” because not everyone agrees that neural networks are doing anything strange: the reason for their strangeness might be that we are not studying the right aspects of these networks.
But many experts think that there’s indeed something that needs further theory and explanation. It's as if deep learning / deep nets have broken theory and require new insights to explain their behavior.
Today’s neural networks are heavily overparameterized, meaning they have way more parameters (or tunable knobs/weights) than instances of training data. Standard ML theory says that such models should overfit—they should end up memorizing the data, and thus not be able to generalize or do well on instances of data that weren’t in the training dataset.
But deep nets don’t overfit! They do generalize (of course, only to data that is still within distribution, but still).
(It's also possible the signs already existed that ML theory was not quite right, say, in the behavior of other methods, such as kernel machines. But they were ignored. Deep nets, because of their success, might be shining a light on a new previously-ignored corner of ML theory)
The denouement of WHY MACHINES LEARN explores this apparent mystery. In fact, almost the entirety of the book can be thought of as laying the foundations to be able to appreciate this puzzle.
This and more can be found here:
𝐔𝐒: penguinrandomhouse.com/books/677608/why-machines-learn-by…
𝐔𝐊: penguin.co.uk/books/446849/why-machines-learn-by-ananthas… -
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
Thanks for these kind, lovely words, @ColeTFR
CR
@C__TFR
Just tore through "Why Machines Learn" by @anilananth and I'm convinced it's a must-read for anyone in AI/ML (including non-technical). Here's why: 🧵👇
-
Anil Ananthaswamy
@anilananth
Thanks for these kind, lovely words, @ColeTFR
CR
@C__TFR
Just tore through "Why Machines Learn" by @anilananth and I'm convinced it's a must-read for anyone in AI/ML (including non-technical). Here's why: 🧵👇
-
Anil Ananthaswamy
@anilananth
One of the most baffling things about the revolution in machine learning is the apparent ability of deep neural networks to against the grain of standard statistical machine learning theory.
Modern deep nets seem to be flouting a long-established rule in a machine learning: A model has to find the optimal balance between being too simple (too few parameters) and too complex (too many parameters) relative to the instances of training data, and thus avoid both under-fitting and overfitting.
I use words like “apparent” and “seem to” because not everyone agrees that neural networks are doing anything strange: the reason for their strangeness might be that we are not studying the right aspects of these networks.
But many experts think that there’s indeed something that needs further theory and explanation. It's as if deep learning / deep nets have broken theory and require new insights to explain their behavior.
Today’s neural networks are heavily overparameterized, meaning they have way more parameters (or tunable knobs/weights) than instances of training data. Standard ML theory says that such models should overfit—they should end up memorizing the data, and thus not be able to generalize or do well on instances of data that weren’t in the training dataset.
But deep nets don’t overfit! They do generalize (of course, only to data that is still within distribution, but still).
(It's also possible the signs already existed that ML theory was not quite right, say, in the behavior of other methods, such as kernel machines. But they were ignored. Deep nets, because of their success, might be shining a light on a new previously-ignored corner of ML theory)
The denouement of WHY MACHINES LEARN explores this apparent mystery. In fact, almost the entirety of the book can be thought of as laying the foundations to be able to appreciate this puzzle.
This and more can be found here:
𝐔𝐒: penguinrandomhouse.com/books/677608/why-machines-learn-by…
𝐔𝐊: penguin.co.uk/books/446849/why-machines-learn-by-ananthas… -
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
Imagine being 𝐚𝐜𝐜𝐮𝐬𝐞𝐝, even in jest, of being the man who 𝘴𝘦𝘵 𝘣𝘢𝘤𝘬 𝘥𝘦𝘦𝘱 𝘭𝘦𝘢𝘳𝘯𝘪𝘯𝘨 𝘣𝘺 𝘵𝘸𝘰 𝘥𝘦𝘤𝘢𝘥𝘦𝘴. 𝐆𝐞𝐨𝐫𝐠𝐞 𝐂𝐲𝐛𝐞𝐧𝐤𝐨, who proved one of the seminal 𝐮𝐧𝐢𝐯𝐞𝐫𝐬𝐚𝐥 𝐚𝐩𝐩𝐫𝐨𝐱𝐢𝐦𝐚𝐭𝐢𝐨𝐧 𝐭𝐡𝐞𝐨𝐫𝐞𝐦𝐬, showed that a 1-hidden layer neural network could approximate any function, given enough neurons. But he 𝘥𝘪𝘥 𝘯𝘰𝘵 say that one should 𝘰𝘯𝘭𝘺 use 1-hidden layer networks--people apparently thought so, and kept trying to do deep learning with 1-layer networks, before realizing that they could have more than one hidden layer (leading to the truly "deep" learning revolution). At least, that's the lore!
Cybenko's story and the logic behind why neural networks are universal function approximators is told in 𝐖𝐇𝐘 𝐌𝐀𝐂𝐇𝐈𝐍𝐄𝐒 𝐋𝐄𝐀𝐑𝐍, in the chapter with one of my favorite titles: 𝐓𝐡𝐞 𝐌𝐚𝐧 𝐖𝐡𝐨 𝐒𝐞𝐭 𝐁𝐚𝐜𝐤 𝐃𝐞𝐞𝐩 𝐋𝐞𝐚𝐫𝐧𝐢𝐧𝐠 (𝐍𝐨𝐭 𝐑𝐞𝐚𝐥𝐥𝐲)
More here:
𝐔𝐒: penguinrandomhouse.com/books/677608/why-machines-learn-by…
𝐔𝐊: penguin.co.uk/books/446849/why-machines-learn-by-ananthas… -
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
Will AI Makes Us Stupid? Yes and No, says Trevor Darrell of @UCBerkeley — "I don’t think there’s a single kind of smart. I don’t think there’s a single kind of stupid" — speaking at the Simons Institute's workshop on Alignment, Trust, Watermarking, and Copyright Issues in LLMs
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
The 3 Ms of AI: Misuse (a stressor), Markets (strong optimizers can create unhealthy markets) & Mediators (AI can empower humans and help mitigate these problems) - @JacobSteinhardt at the Simons Institute's workshop on Alignment, Trust, Watermarking, and Copyright Issues in LLMs
-
Anil Ananthaswamy reposted
Kevin Mitchell
@WiringTheBrain
Looking forward to this debate on Free Will with Robert Sapolsky, next week - online, hosted by @UCD_CMB (link to follow)
-
Anil Ananthaswamy
@anilananth
Did Einstein say that as a complement? Knowing what we know now about quantum mechanics, it seems to be the right attitude for a scientist who's asking fundamental questions
Ash Jogalekar
@curiouswavefn
“He utters his opinions like one perpetually groping, and never like one who believes himself to be in possession of the truth.” - Albert Einstein on Niels Bohr
-
Anil Ananthaswamy reposted
Joe Domaleski, KI4ASK
@joedom
My top picks of books 📚 about AI for 2024 👨🏻💻📊 - blog.marketingdatascience.ai/my-top-picks-of-books-about-…. Including excellent books by @emollick @anilananth and more. Great reading to complement my @gtomsanalytics studies at @GeorgiaTech 🐝 #ai #learning #analytics #digitalmarketing #reading
-
Anil Ananthaswamy reposted
Edgar Guevara
@guevaracodina
Excited to dive into this book after hearing such glowing praise! Can’t wait to see what all the buzz is about!
Anil Ananthaswamy
@anilananth
I'm never going to get another chance for this particular humble brag, so I'll take it :-) When Geoff Hinton endorsed WHY MACHINES LEARN sometime back in March/April, my impostor syndrome came to the fore (yes, writers have it too, not just scientists). Surely not, I thought at the time. It has taken a while, but now that the book has been out in the wild, and readers have written in with their appreciation, the impostor syndrome is lessened (though it never quite goes away).
John Hopfield's work on Hopfield networks and Hinton's work on backpropagation (though that was not what won him the #nobelprize2024), is described in WHY MACHINES LEARN (with some of the basic math and mathematical intuition):
US: penguinrandomhouse.com/books/677608/why-machines-learn-by…
UK: penguin.co.uk/books/446849/why-machines-learn-by-ananthas… -
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
A simple insight into not why or how, but what a neural network learns.
Let’s say you want to train an ML model to distinguish between two classes of data (top left panel, circles and triangles). A linear classifier—which would find a straight line, plane or hyperplane to delineate the clusters—won’t work.
That’s because the two features of the data, x1 (the value along the x-axis) and x2 (the value along the y-axis) are insufficient. We could create a third feature: (x1-squared + x2-squared), so that each data point is now represented by three features (x1, x2, x1-sq + x2-sq).
If you plot the augmented data in 3D, the triangles rise above the circles, and you can use a linear classifier to find the hyperplane to separate the two clusters (top right pane). The decision boundary, when projected back into 2D, will be non-linear!
You can also use kernel methods—which allow you to project data into higher dimensions but do calculations in the lower dimensions—but you need to design the kernel. When the data gets complex, designing kernels is not a trivial task.
This is where neural networks shine. If you have a neural network with hidden layers and non-linear activation functions (such as the sigmoid activation function), you can simply feed the neural network (x, y) pairs of data, where x = (x1, x2) and y = -1 or +1 (say, for circles or triangles).
Even a simple neural network with one hidden layer made of three neurons will find a decision boundary that might look like the one shown in the bottom left panel.
Internally, the neural network figures out the features necessary to classify the data. That's one way to think of what a deep neural network does: it learns the features necessary to carry out the task at hand.
All this and more are explained in Ch 10 of WHY MACHINES LEARN: “The Algorithm that Put Paid to a Persistent Myth.” The myth—slyly perpetuated by Minsky and Papert—was that even multi-layer neural networks might not be able to solve certain simple problems!
The algorithm is backpropagation, and though many had developed versions of the algorithm prior to 1986, it was the 1986 Nature paper by David Rumelhart, Geoffrey Hinton and Ronald Williams that clearly showed how it could be applied to neural networks and that hidden layers learned features when trained via backprop.
More at:
US: penguinrandomhouse.com/books/677608/why-machines-learn-by…
UK: www.penguin.co.uk/books/446849/why-machines-learn-by-anan… -
Anil Ananthaswamy
@anilananth
A simple insight into not why or how, but what a neural network learns.
Let’s say you want to train an ML model to distinguish between two classes of data (top left panel, circles and triangles). A linear classifier—which would find a straight line, plane or hyperplane to delineate the clusters—won’t work.
That’s because the two features of the data, x1 (the value along the x-axis) and x2 (the value along the y-axis) are insufficient. We could create a third feature: (x1-squared + x2-squared), so that each data point is now represented by three features (x1, x2, x1-sq + x2-sq).
If you plot the augmented data in 3D, the triangles rise above the circles, and you can use a linear classifier to find the hyperplane to separate the two clusters (top right pane). The decision boundary, when projected back into 2D, will be non-linear!
You can also use kernel methods—which allow you to project data into higher dimensions but do calculations in the lower dimensions—but you need to design the kernel. When the data gets complex, designing kernels is not a trivial task.
This is where neural networks shine. If you have a neural network with hidden layers and non-linear activation functions (such as the sigmoid activation function), you can simply feed the neural network (x, y) pairs of data, where x = (x1, x2) and y = -1 or +1 (say, for circles or triangles).
Even a simple neural network with one hidden layer made of three neurons will find a decision boundary that might look like the one shown in the bottom left panel.
Internally, the neural network figures out the features necessary to classify the data. That's one way to think of what a deep neural network does: it learns the features necessary to carry out the task at hand.
All this and more are explained in Ch 10 of WHY MACHINES LEARN: “The Algorithm that Put Paid to a Persistent Myth.” The myth—slyly perpetuated by Minsky and Papert—was that even multi-layer neural networks might not be able to solve certain simple problems!
The algorithm is backpropagation, and though many had developed versions of the algorithm prior to 1986, it was the 1986 Nature paper by David Rumelhart, Geoffrey Hinton and Ronald Williams that clearly showed how it could be applied to neural networks and that hidden layers learned features when trained via backprop.
More at:
US: penguinrandomhouse.com/books/677608/why-machines-learn-by…
UK: www.penguin.co.uk/books/446849/why-machines-learn-by-anan… -
Anil Ananthaswamy reposted
Tomer Ullman
@TomerUllman
hooray, the "Bayesian Models of Cognition" book is out, for all your Bayesian models of cognition needs :)
(I contributed to 3 chapters in the book -- development, intuitive physics, theory of mind -- that can be read on the ol' website) -
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
“Not everything on the internet was intended to be on the internet.” Nicholas Carlini of @GoogleDeepMind on the privacy implications of LLMs trained on "public" internet data at the Simons Institute's workshop on Alignment, Trust, Watermarking, and Copyright Issues in LLMs.
-
Anil Ananthaswamy reposted
Machine Learning Street Talk
@MLStreetTalk
We just released @fchollet keynote talk from #agi24. He challenges LLM hype, discusses ARC-AGI benchmark, and proposes merging deep learning with program synthesis for true AGI progress.
This media couldn't be displayed. See it on the original post. -
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
Deep connections: why two AI pioneers won the Nobel Prize for Physics Hamish Johnston, editor @PhysicsWorld, and I chat about this week's physics #nobelprize blubrry.com/physicsworldweeklypodcast/137328728/
-
Anil Ananthaswamy reposted
Bruno Neri
@neribr
This book by @s_scardapane is a gem. You can find one of the best explanation of the transformer models and graph models. So let’s go down the rabbit hole!
-
Anil Ananthaswamy reposted
Kanaka Rajan
@KanakaRajanPhD
Thank you @_TheTransmitter & @JordanCollver for this piece about the power of science comics!
Science has the biggest impact when it can reach the most people. Science comics invite us to discuss research creatively & accessibly.
Article 👇 & a quick 🧵www.thetransmitter.org/craft-and-careers/how-neuroscience… -
Anil Ananthaswamy reposted
Subbarao Kambhampati (కంభంపాటి సుబ్బారావు)
@rao2z
Our API Access bill for 🍓experiments for the first two weeks of its existence..😱
Making babies is still the far cheaper/funner path to AGI, y'all..
Subbarao Kambhampati (కంభంపాటి సుబ్బారావు)
@rao2z
LRM-Modulo? Our initial experiments with o1 showed that while LRMs like it do seem to lift the floor for performance on planning problems, they are still far from robust. One idea is to view LRMs as better (though much costlier) generators that can take the place of LLMs in a compound architecture like LLM-Modulo. How good are they in this role?
Our latest experiments with o1 👇 in several planning and scheduling tasks show that this is indeed an effective approach. In a variety of planning and scheduling problems, an LLM-Modulo-like back prompting with an external sound verifier capable of critiquing o1 solutions not only increases the accuracy significantly, but gives guarantees of the solution correctness--something that will be essential in safety-critical and non-ergodic environments.
For example, with a few back prompting iterations, on hard blocks world instances, the accuracy goes from 24% to 98%; on Sokoban, it goes from 12% to 43%; on the Travel Planning benchmark, it goes from 10% to 65% and on Trip Planning it goes from 4% to 15%.
The improvements however come with considerable costs. Given that even a single call to o1 is quite costly, multiple back-prompting calls threaten to effortlessly extinguish academic API budgets 😅.
This brings up interesting meta-optimization issues: When is it better to call the cheaper-per -call LLM generators vs. much-costlier-per-call LRM generators?
For example, in calendar scheduling, o1-mini costs $2.70 to run over the entire test
set, and has a final accuracy of 94%. GPT-4o-mini can be run for 50 iterations in a modulo loop for
only $2.48, beating that performance to get 97%, a figure that comes with guarantees not provided by
just the base model.
1/ -
Anil Ananthaswamy
@anilananth
Deep connections: why two AI pioneers won the Nobel Prize for Physics Hamish Johnston, editor @PhysicsWorld, and I chat about this week's physics #nobelprize blubrry.com/physicsworldweeklypodcast/137328728/
-
Anil Ananthaswamy reposted
NSF-DOE Rubin Observatory
@VRubinObs
We have a telescope!🤩
Last week, summit staff successfully installed Rubin Observatory's combined primary/tertiary mirror.
With all three mirrors & the commissioning camera in place, we officially—for the first time in 15 years of construction—have a complete telescope!🥳
🧵This media couldn't be displayed. See it on the original post. -
Anil Ananthaswamy
@anilananth
Not everyone would agree with our idea of "relaxing weekend reading" @onajomo_akemu!
Onajomo Akemu
@onajomo_akemu
Now, for some relaxing weekend reading. Elegant math, you say, @anilananth ? It’s beautiful math.
-
Anil Ananthaswamy reposted
Onajomo Akemu
@onajomo_akemu
Now, for some relaxing weekend reading. Elegant math, you say, @anilananth ? It’s beautiful math.
-
Anil Ananthaswamy reposted
Andreas Vlachos
@VlachosLab
🚨New preprint update!🚨 We explore the interplay between homeostatic synaptic scaling and structural plasticity in maintaining robust neural network firing. Check out the updated version of our manuscript. Fantastic work by @Miss_abN and team 🧠🔬 www.biorxiv.org/content/10.1101/2023.03.09.531681v3
-
Anil Ananthaswamy reposted
National Centre for Biological Sciences
@NCBS_Bangalore
It is with profound sadness that we acknowledge the passing of Mr. Ratan Tata.
As the chair of the TIFR Management Council, he played a crucial role in establishing NCBS & TIGS, advancing research for public welfare.
We extend our condolences to his family, friends & colleagues -
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
How to use an LLM to learn to code Hopfield Networks! Learn to code one of the networks that led to the 2024 physics #nobleprize? It's relatively simple. Here's a blog post detailing the LLM prompts you can use: anilananthaswamy.com/why-machines-learn-codebook-blog/hop…
-
Anil Ananthaswamy
@anilananth
How to use an LLM to learn to code Hopfield Networks! Learn to code one of the networks that led to the 2024 physics #nobleprize? It's relatively simple. Here's a blog post detailing the LLM prompts you can use: anilananthaswamy.com/why-machines-learn-codebook-blog/hop…
-
Anil Ananthaswamy reposted
Giles Thomas
@gpjt
"Why machines learn" by @anilananth is a phenomenal book. Highly recommended.
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
As if priming us for the debate over physics vs ML, given yesterday's #nobelprize for physics, @SuryaGanguli recently spoke at the Simons Institute's workshop on Transformers as a Computational Model, on the dynamics of propagation of tokens (particles) through a Transformer
-
Anil Ananthaswamy reposted
Ryan Parks
@rnparks
This is a fantastic book and deserves the accolades.
-
Anil Ananthaswamy
@anilananth
Some scientists I met and wrote about in 3 of my 4 books have won Nobel Prizes in physics after the book came out :-) The joy of writing about cutting-edge science! Not saying anything more, just in case anyone thinks otherwise.
I'm trying to figure out who is most likely to win from the scientists whose work appears in the holdout, The Man Who Wasn't There, which is about the neuroscience of the sense of self.
Given that today's Nobel was about AI->Physics, maybe when AI become conscious and materialist philosophers are proven right, then we'll have Philosophy->Physics Nobel Prizes :-)
In which case, I have some thoughts :-)
The Edge of Physics (2010)
Saul Perlmutter, 2011
James Peebles, 2019
The Man Who Wasn't There (2015)
???
Through Two Doors at Once (2018)
Roger Penrose (2020)
Anton Zeilinger (2022)
Alain Aspect (2022)
Why Machines Learn (2024)
John Hopfield (2024)
Geoff Hinton (2024) -
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
I'm never going to get another chance for this particular humble brag, so I'll take it :-) When Geoff Hinton endorsed WHY MACHINES LEARN sometime back in March/April, my impostor syndrome came to the fore (yes, writers have it too, not just scientists). Surely not, I thought at the time. It has taken a while, but now that the book has been out in the wild, and readers have written in with their appreciation, the impostor syndrome is lessened (though it never quite goes away).
John Hopfield's work on Hopfield networks and Hinton's work on backpropagation (though that was not what won him the #nobelprize2024), is described in WHY MACHINES LEARN (with some of the basic math and mathematical intuition):
US: penguinrandomhouse.com/books/677608/why-machines-learn-by…
UK: penguin.co.uk/books/446849/why-machines-learn-by-ananthas… -
Anil Ananthaswamy reposted
Bruce Grey Tedesco
@BruceTedesco
Geoff is spot on! Anil, your book wonderful; my prediction is it's on the way to being a classic.
I've enjoyed the first 80%... Looking forward to finishing it this week. 😎
Anil Ananthaswamy
@anilananth
I'm never going to get another chance for this particular humble brag, so I'll take it :-) When Geoff Hinton endorsed WHY MACHINES LEARN sometime back in March/April, my impostor syndrome came to the fore (yes, writers have it too, not just scientists). Surely not, I thought at the time. It has taken a while, but now that the book has been out in the wild, and readers have written in with their appreciation, the impostor syndrome is lessened (though it never quite goes away).
John Hopfield's work on Hopfield networks and Hinton's work on backpropagation (though that was not what won him the #nobelprize2024), is described in WHY MACHINES LEARN (with some of the basic math and mathematical intuition):
US: penguinrandomhouse.com/books/677608/why-machines-learn-by…
UK: penguin.co.uk/books/446849/why-machines-learn-by-ananthas… -
Anil Ananthaswamy
@anilananth
The focus on Hopfield networks takes away from John Hopfield's fabulous earlier work in the dynamics of biochemical reactions--which laid the foundation for his eponymous networks, which led to the #nobelprize2024.
When I talked to him for WHY MACHINES LEARN, he spoke of the joy of having his theoretical "biology" work, about how multiple biochemical pathways between two endpoints are necessary for error correction/proof reading, vindicated in the lab.
'Researchers had found that the antibiotic streptomycin interferes with the bacteria’s ability to proofread; as a result, the bacteria synthesize numerous erroneous and functionally lethal proteins, causing them to die. Hopfield was elated at the empirical validation of his theoretical work. “It was one of the biggest—and most delightful—surprises of my scientific career,” he wrote.'
[Quote from: pni.princeton.edu/people/john-j-hopfield/now-what
If you haven't read his "Now What" essay, please do! It tells you more about him than anything anyone might write about him]
Hopfield's story is recounted in WHY MACHINES LEARN, along with the math/physics/algorithm of Hopfield Networks:
US: penguinrandomhouse.com/books/677608/why-machines-learn-by…
UK: penguin.co.uk/books/446849/why-machines-learn-by-ananthas… -
Anil Ananthaswamy reposted
Benjamin Manning
@BenSManning
If you want to know more about the @geoffreyhinton's and @HopfieldJohn's work in an intelligible and fun way, reminder that this book dedicates full chapters to each of these Nobel laureates! A well-timed must-read.
Benjamin Manning
@BenSManning
Never has a book, nor even an entire course, improved my intuition about all things ML. Every page left me a better understanding of topics I've spent years studying. Maybe this reflects bad courses or my low IQ, but I think the book is just that good. Must read.
Anil Ananthaswamy
@anilananth
Don't know how other authors feel, but seeing the first physical copy of your book, after years of looking at Word files and PDFs... Seems surreal. It's as if someone else wrote it... Anyway, this advance copy arrived in the mail. One more month to pub date ...www.penguinrandomhouse.com/books/677608/why-machines-lear…
#artificalintelligence @DuttonBooks -
Anil Ananthaswamy reposted
Surya Ganguli
@SuryaGanguli
Who said Hopfield networks are not physics?! :)
journals.aps.org/prx/abstract/10.1103/PhysRevX.11.021048 -
Anil Ananthaswamy reposted
Sanjeev Arora
@prfsanjeevarora
We're looking for postdocs! Great compute access, wonderful resources and colleagues. Please apply via the ad below. We have the flexibility to start appointments during the year as well. Current postdocs are listed here pli.princeton.edu/people/postdoctoral-researchers
Princeton PLI
@PrincetonPLI
The @Princeton AI Lab is seeking postdoctoral fellows to join AI-related research initiatives.
For the job descriptions and application: ai.princeton.edu/ai-lab/employment -
Anil Ananthaswamy
@anilananth
"My own point of view: there is just one nature. I'm delighted when people have and share deep insights into nature, and I don't care so much what we label it." @michael_nielsen's poetic take on the handwringing over whether today's #nobelprize winners were really doing physics.
Michael Nielsen
@michael_nielsen
A surprising number of people have asked me versions of "Is the Physics Nobel Prize today really for Physics?"
What counts as a field is surprisingly complicated. As a rough and incomplete classification, a field can be:
1. Based on exploration of and development of a set of agreed-upon deep principles. The Maxwell-Lorentz equations, quantum field theory, general relativity, and quantum computing are each (separately!) examples of fields in this sense
2. Based on exploration of some (more or less) agreed-upon set of questions. The search for a basic fundamental theory and quantum information are both examples of fields having this flavour. The questions tend to shift over time, sometimes substantially, and such fields sometimes fission or fusion or change substantially as a result
3. A group of people investigating a common domain of application. Examples of fields with this flavour are atomic physics, optical physics, and condensed matter physics. This can be viewed as a case of 2. It's striking how much internal variation there can be - understanding (say) the fractional quantum Hall effect is very different than understanding spin glasses, yet both are part of condensed matter physics. In some sense the question behind condensed matter unifies many different fields of type 1
4. A philosophical, organizational, and political treaty among fields of types 1-3. That's what "Physics" is, in the sense of the prize. It's interesting that antenna design is currently not regarded as part of Physics (it's more EE), while quantum computing to some extent is part of Physics. That's partly a contingent choice: it could have been different if history had just been a tiny bit different. However, to some extent it also represents some general philosophy of "what Physics is about". Ideas like quantum error-correction and topological quantum computers required deep fundamental insights into physics. My guess - it's just a guess - is that, over time, quantum computing will become more detached from Physics, as it becomes more and more commercial, and more and more engineering
Today's Physics prize falls outside the usual type 4 philosophical, organizational, and political treaty of "what physics is". People at Caltech used to tell me that John Hopfield had "left physics", and that's why he'd gone to Princeton (from Caltech). But then, a lot of physicists in the 1990s didn't think quantum computing was part of physics. I'm sure some still don't. So: it really is somewhat contingent
My own point of view: there is just one nature. I'm delighted when people have and share deep insights into nature, and I don't care so much what we label it. I'm especially delighted by the incredible progress in the past few decades in developing the design sciences. That is: understanding the fundamental principles underlying the incredible systems latent in nature, and which we humans are gradually learning to build. John Hopfield and Geoff Hinton have made enormous contributions to understanding what possibilities lie latent in nature. It so happens that their work falls largely outside the usual Nobel classification, but I am happy to celebrate them for their remarkable contributions, and physics seems as apt an area as any
Congratulations to them both! -
Anil Ananthaswamy reposted
Dmitry Krotov
@DimaKrotov
Given today’s great news from the #nobelprize2024, I want to share a couple of personal thoughts on Hopfield Networks.
This idea had an enormous impact on at least three large disciplines: Statistical Physics, Computer Science and AI, and Neuroscience. -
Anil Ananthaswamy
@anilananth
You are officially the first Nobel Prize-induced sale of the book :-) Thanks!
Nirmalya Kajuri
@Kaju_Nut
"Why Machines Learn" by @anilananth looks super interesting and timely in light of this year's Nobel prize. Just bought myself a copy
www.amazon.in/Why-Machines-Learn-Elegant-Behind-ebook/dp/… -
Anil Ananthaswamy
@anilananth
I'm never going to get another chance for this particular humble brag, so I'll take it :-) When Geoff Hinton endorsed WHY MACHINES LEARN sometime back in March/April, my impostor syndrome came to the fore (yes, writers have it too, not just scientists). Surely not, I thought at the time. It has taken a while, but now that the book has been out in the wild, and readers have written in with their appreciation, the impostor syndrome is lessened (though it never quite goes away).
John Hopfield's work on Hopfield networks and Hinton's work on backpropagation (though that was not what won him the #nobelprize2024), is described in WHY MACHINES LEARN (with some of the basic math and mathematical intuition):
US: penguinrandomhouse.com/books/677608/why-machines-learn-by…
UK: penguin.co.uk/books/446849/why-machines-learn-by-ananthas… -
Anil Ananthaswamy
@anilananth
He started off in solid state physics (where he had a stellar career), then biophysics (dynamics of biological networks), then computational neuroscience ...on to Hopfield Networks, which led to the Nobel.
Ananyo Bhattacharya
@Ananyo
Tbf Hopfield is a (bio)physicist. Also see @anilananth’s latest book, Ch 8
-
Anil Ananthaswamy reposted
Doris Tsao
@doristsao
-
Anil Ananthaswamy reposted
Ananyo Bhattacharya
@Ananyo
Totally serves physicists right. For years they've been saying, "You know, deep down everything is physics'"...
The Nobel Prize
@NobelPrize
BREAKING NEWS
The Royal Swedish Academy of Sciences has decided to award the 2024 #nobelprize in Physics to John J. Hopfield and Geoffrey E. Hinton “for foundational discoveries and inventions that enable machine learning with artificial neural networks.” -
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
One small thing Geoff Hinton and David Rumelhart did -- apropos of his physics #nobelprize2024 -- that often goes unnoticed when we talk of his neural network work is the idea of symmetry breaking. Granted it's not quite the same as how physicists think of it, but it's worth recounting.
In 1958, Frank Rosenblatt devised the perceptron, which could be used to construct single-layer neural networks. Rosenblatt realized that if the weights of each perceptron started off identically, they'd all learn the same thing! In other words, they were redundant. There was a symmetry that needed to be broken if the perceptrons were to learn different features in the data.
Rosenblatt wrote that there should be some stochasticity in the network to break symmetry. He didn't specify exactly what he meant.
Hinton for a long time thought that the stochasticity should be in the output of the artificial neurons. This was the wrong approach. "This slowed me
down for a while," he told me (quoted in WHY MACHINES LEARN).
Hinton began working with David Rumelhart. Almost immediately, Rumelhart proposed a solution. “His immediate reaction was ‘Well, why can’t we just break
symmetry by having random initial weights?’ Rosenblatt didn’t think of that,” Hinton told me. (quoted in WML).
The rest, as they say, is history. Rumelhart, Hinton and Williams wrote the 1986 Nature paper on backpropagation that specifically addressed the issue of features learned by neural networks when trained with backprop. The revolution was set in motion.
All this and more is narrated in WHY MACHINES LEARN.
US: penguinrandomhouse.com/books/677608/why-machines-learn-by…
UK: www.penguin.co.uk/books/446849/why-machines-learn-by-anan… -
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
Did not have seen this coming! The Physics Nobel goes to John Hopfield and Geoff Hinton. Interviewing them was among the many highlights of writing Why Machines Learn. It took some persuasion to get Hopfield to talk to me--but he did, and it was such a joy! In fact, the chapter on Hopfield Networks is called "With A Little Help from Physics"!!!
Here's the Nobel Prize announcement: www.nobelprize.org/prizes/physics/ -
Anil Ananthaswamy
@anilananth
There is actually an argument to be made that the real reason WHY machines learn has to do with physics! Someone should write that book :-)
Ananyo Bhattacharya
@Ananyo
Oh. And if you want to understand, REALLY understand AI, neural nets etc and the new Nobel, you should probably read this man's latest book, "Why Machines Learn: The Elegant Math Behind Modern AI" (though clearly he meant to say 'physics') anilananthaswamy.com/why-machines-learn
Anil Ananthaswamy
@anilananth
Sometimes, when you are writing a book, or even a long magazine feature, a source says something that echoes your own sentiment about the topic, the reason why you started writing the book/story in the first place. For me, it was a conversation with Ilya Sutskever about his initial reaction to the basic math of deep learning (when he was still a student with Geoff Hinton):
“How can it be that it’s so simple…so simple that you can explain it to high school students without too much effort?”
“I think that’s actually miraculous...This is also, to me, an indication that we are probably on the right track. [It can’t] be a coincidence that such simple concepts go so far.”
I remember thinking at the time: I couldn't have said it better.
Quotes taken from WHY MACHINES LEARN. www.penguinrandomhouse.com/books/677608/why-machines-lear… -
Anil Ananthaswamy
@anilananth
One small thing Geoff Hinton and David Rumelhart did -- apropos of his physics #nobelprize2024 -- that often goes unnoticed when we talk of his neural network work is the idea of symmetry breaking. Granted it's not quite the same as how physicists think of it, but it's worth recounting.
In 1958, Frank Rosenblatt devised the perceptron, which could be used to construct single-layer neural networks. Rosenblatt realized that if the weights of each perceptron started off identically, they'd all learn the same thing! In other words, they were redundant. There was a symmetry that needed to be broken if the perceptrons were to learn different features in the data.
Rosenblatt wrote that there should be some stochasticity in the network to break symmetry. He didn't specify exactly what he meant.
Hinton for a long time thought that the stochasticity should be in the output of the artificial neurons. This was the wrong approach. "This slowed me
down for a while," he told me (quoted in WHY MACHINES LEARN).
Hinton began working with David Rumelhart. Almost immediately, Rumelhart proposed a solution. “His immediate reaction was ‘Well, why can’t we just break
symmetry by having random initial weights?’ Rosenblatt didn’t think of that,” Hinton told me. (quoted in WML).
The rest, as they say, is history. Rumelhart, Hinton and Williams wrote the 1986 Nature paper on backpropagation that specifically addressed the issue of features learned by neural networks when trained with backprop. The revolution was set in motion.
All this and more is narrated in WHY MACHINES LEARN.
US: penguinrandomhouse.com/books/677608/why-machines-learn-by…
UK: www.penguin.co.uk/books/446849/why-machines-learn-by-anan… -
Anil Ananthaswamy
@anilananth
Did not have seen this coming! The Physics Nobel goes to John Hopfield and Geoff Hinton. Interviewing them was among the many highlights of writing Why Machines Learn. It took some persuasion to get Hopfield to talk to me--but he did, and it was such a joy! In fact, the chapter on Hopfield Networks is called "With A Little Help from Physics"!!!
Here's the Nobel Prize announcement: www.nobelprize.org/prizes/physics/ -
Anil Ananthaswamy
@anilananth
For those who understand Hindi, read this conversation I had with @AnthropicAI's Claude 3.5 Sonnet in Hinglish, about a question that one could argue requires Theory of Mind. We know by now that LLMs are not reasoning the way humans do, if they are reasoning at all, but the fact that just next token prediction and extremely sophisticated pattern-matching gets us this far is more than a bit mind-boggling.
Theory of Mind, in cognitive science, refers to the human ability to theorize about the contents of someone else's mind, hence the phrase "theory of mind".
I made a mistake in my initial query, by forgetting to point out that Sundar had put identical-looking glasses but with the wrong power in the left drawer. So, one could argue that Claude erred in its answer. But it actually said "probably" to qualify its answer.
So, I asked Claude to explain why it said "probably". The answer is very nuanced. I'm scratching my head to figure out where on the internet it could have seen a pattern in Hinglish that matches these patterns.
Note how in Claude's final answer, it explains admirably, until at the very end, it stumbles. -
Anil Ananthaswamy
@anilananth
A friend of mine asked Claude 3.5 Sonnet to design the worst combination of Indian and French cuisine. It came up with a pretty awful sounding dish :-)
-
Anil Ananthaswamy reposted
Timothy Gowers @wtgowers
@wtgowers
Occasionally in mathematics a statement that just has to be true turns out to be false. A paper appeared on arXiv today that disproves a well-known conjecture in probability called the bunkbed conjecture. 🧵
arxiv.org/abs/2410.02545 -
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
"A proof of the Riemann hypothesis: How much would you be willing to pay?" A million? @OpenAi's Noam Brown at the Simons Institute's workshop on Transformers as a Computational Model on the growing efficacy+cost of inference-time compute in LLMs and the worth of some solutions.
-
Anil Ananthaswamy reposted
Steven Strogatz
@stevenstrogatz
Outside the classroom where I teach applied complex analysis, I spotted this poster of Frank Rosenblatt. He invented the perceptron, and helped launch the study of machine learning.
-
Anil Ananthaswamy
@anilananth
Sometimes, when you are writing a book, or even a long magazine feature, a source says something that echoes your own sentiment about the topic, the reason why you started writing the book/story in the first place. For me, it was a conversation with Ilya Sutskever about his initial reaction to the basic math of deep learning (when he was still a student with Geoff Hinton):
“How can it be that it’s so simple…so simple that you can explain it to high school students without too much effort?”
“I think that’s actually miraculous...This is also, to me, an indication that we are probably on the right track. [It can’t] be a coincidence that such simple concepts go so far.”
I remember thinking at the time: I couldn't have said it better.
Quotes taken from WHY MACHINES LEARN. www.penguinrandomhouse.com/books/677608/why-machines-lear… -
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
1/8 🧵on how to use an LLM to code/learn about an icon of machine learning: The k nearest neighbor rule/algorithm. Here's a sequence of prompts to generate code and learn. This was done using Claude 3.5 Sonnet. Your mileage will vary depending on your LLM of choice.
-
Anil Ananthaswamy reposted
Vinay Gupta
@guptavinay
Recently read 3 books and all are wonderful. Thanks to @alok_sama @anilananth @Michael_Easter for sharing valuable insights.
We are always at the beginning of infinity when it comes to knowledge - David Deutsche -
Anil Ananthaswamy
@anilananth
Very glad to hear, @hernanlira_
Hernan Lira
@hernanlira_
What a book! I’m really enjoying it @anilananth
-
Anil Ananthaswamy reposted
Hernan Lira
@hernanlira_
What a book! I’m really enjoying it @anilananth
-
Anil Ananthaswamy reposted
Peyman Milanfar
@docmilanfar
A surprising & little-known results in classical statistics:
mean (μ) and median (m) are within one std deviation:
|μ−m| ≤ σ
For unimodal densities, bound is even tighter
|μ−m| ≤ 0.7746 σ
This beautiful results first appeared in a 1932 paper by Hotelling & Solomons
1/3 -
Anil Ananthaswamy reposted
Alfredo Canziani
@alfcnz
Simulateur de Neurones (SN), one of the earliest deep learning frameworks, already had interactive and graphic capabilities ~30 years ago.
In this blog post, you can learn more about a PyTorch ancestor, used to train the first convnet.
atcold.github.io/2024/09/27/SN-GUI -
Anil Ananthaswamy
@anilananth
1/8 🧵on how to use an LLM to code/learn about an icon of machine learning: The k nearest neighbor rule/algorithm. Here's a sequence of prompts to generate code and learn. This was done using Claude 3.5 Sonnet. Your mileage will vary depending on your LLM of choice.
Post not available here, view on X
Anil Ananthaswamy
@anilananth
7/8 You can play around with the code to understand issues like overfitting! For example, I select a dataset with an outlier (top left). A circle is wrongly classified as a triangle. The 1-NN algorithm takes it into account; you can see that the decision boundary is overfitting: it classifies what is most likely a triangle as a circle (top right).
But if you use 3-NN, the outlier is ignored. We have avoided overfitting: the decision boundary (bottom left) is smoother and the new data point is correctly classified (bottom right).
Anil Ananthaswamy
@anilananth
8/8 For readers of WHY MACHINES LEARN: an LLM can be a powerful companion to further your understanding, by getting it to generate the code. Almost every illustration in Chapter 5 (Birds of a Feather), which deals with the history and math of the Nearest Neighbor algorithm, was created using code I wrote. But an LLM achieved that in less than 30 mins of work. Granted you need to know exactly what you need, and should be able to know when things go wrong. Please see: penguinrandomhouse.com/books/677608/why-machines-learn-by…
-
Anil Ananthaswamy reposted
Ethan Siegel
@StartsWithABang
The big day is almost here! On October 8th, my first
@NatGeoBooks, and our first book on the @NASAWebb
space telescope, releases: Infinite Cosmos!
You can pre-order from many booksellers, or get it directly from National Geographic here:
books.disney.com/book/infinite-cosmos/ -
Anil Ananthaswamy
@anilananth
Still a bit amazed and humbled that Why Machines Learn--a book with math/equations in it--is in this list, alongside some rather big names in India. Did not expect this @bookworm_Kris !
Bookworm Bookstore, Blr
@bookworm_Kris
#bookwormbestsellers for September '24
In non-fiction, books by @DalrympleWill, @anilananth, @rahulabhatia, @nehadixit123 & @ShashiTharoor.
In fiction, by Kuvempu (transl Vanamala Viswanatha), Coco Mellors, Asako Yuzuki, @HariniNagendra & Sally Rooney. -
Anil Ananthaswamy
@anilananth
Hmmm... What's X's algorithm trying to tell me? A tweet about a critique of Dan Dennett followed by a post by @keithfrankish on Consciousness, reactivity and LLMs!
-
Anil Ananthaswamy reposted
Steven Strogatz
@stevenstrogatz
Such a pretty campus! Been working here 30 years, and I still love walking to my office every day at @Cornell.
-
Anil Ananthaswamy
@anilananth
Did Marvin Minsky single-handedly slow down research on neural networks in the 1970s? A qualified answer would be yes! Did he do this in order to shift research interest and funding towards symbolic AI? Maybe…
While WHY MACHINES LEARN doesn’t deal with symbolic AI, it often mentions Minsky and Seymour Papert, and their 1969 book, Perceptrons, and the seemingly outsized role the book played in derailing neural network research. Ironically, the book was a celebration of Frank Rosenblatt’s perceptrons, which gave us the first single-layer neural networks that could act as linear classifiers.
Minsky and Papert proved, very elegantly one might add, that a single-layer neural network can never solve the XOR problem, which requires a non-linear boundary to separate two clusters of data. But they seemed to suggest, even insinuate, that multilayer neural networks would similarly fail, but without providing any concrete proof.
Key figures in neural network research that I spoke to thought not so kindly of Minsky (for some reason, Papert escaped their wrath).
One of Rosenblatt’s students told me that Rosenblatt was well aware of the limitations of single-layer NNs, and felt multilayer neural networks would solve the problem, but no one knew how to train them in the 1960s. Rosenblatt even foresaw the necessity of back propagation of errors in his book, The Principles of Neurodynamics.
John Hopfield, of Hopfield networks fame, told me: “Minsky had missed the point.”
George Cybenko, who proved in 1989 that a neural network with an arbitrarily large hidden layer is a universal function approximator, also felt that M&P had got it wrong. “We had these negative results by Minsky and Papert, and yet people were doing stuff and getting things done.
Most acerbic was Geoff Hinton, calling Minsky and Papert’s claims a “con job," saying "and people fell for it.”
This thread turns through WHY MACHINES LEARN. You can find more here:
US: www.penguinrandomhouse.com/books/677608/why-machines-lear…
UK: www.penguin.co.uk/books/446849/why-machines-learn-by-anan… -
Anil Ananthaswamy
@anilananth
What a lovely piece of history. Minsky and Papert were about to publish their 1969 tome, Perceptrons, which would prove to be temporary death knell for neural network research, so this adds an informed tidbit about Minsky's thinking at the time
Bob Metcalfe
@RobertMMetcalfe
Title of my 1968 MIT bachelor thesis in EE was “A Neuron Model and Some of Its Information Processing Capabilities.” 1968. My advisor, Marvin Minsky, accepted the thesis, but I could tell he didn’t like it much. So I went farther down the 9th floor and got a job with JCR Licklider helping to build the Arpanet/Ethernet/Internet. Worked out.
-
Anil Ananthaswamy reposted
National Centre for Biological Sciences
@NCBS_Bangalore
📢NEW!
@uramakri @Anubhab_Khan & team have uncovered five distinct elephant populations across India, revealing more diversity than previously known.
This latest study uncovers new insights into genetic history of India’s #asianelephants🐘
Read here: www.cell.com/current-biology/fulltext/S0960-9822(24)01177-1 -
Anil Ananthaswamy reposted
Melanie Mitchell
@MelMitchell1
I'm looking for a postdoc (to start Spring or Summer 2025) to work with me & others at the Santa Fe Institute on AI models of visual and multimodal reasoning.
See santafe.edu/about/jobs/postdoc-ai-modeling for more info & application instructions.
Please RT! -
Anil Ananthaswamy reposted
Rakesh Bhandari
@postdiscipline
"Ribosome's Favorite Pick Up Line: Are you mRNA? Because I have been waiting to translate you all day." Daughter's HS biology teacher gave them a clever assignment. I wish I had been taught biology this way.
-
Anil Ananthaswamy reposted
Benjamin Jacobsen
@BN_Jacobsen
Really excited to start this book about the maths of AI @anilananth
-
Anil Ananthaswamy reposted
Nico
@0xiamnico
Ordered, looking forward to it @anilananth 🫡
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
“I want a neural net that solves harder problems than it was trained on, AKA…weak to strong generalization.”—Tom Goldstein at the Simons Institute's workshop on Transformers as a Computational Model. @tomgoldsteincs #simonslive
-
Anil Ananthaswamy reposted
JJ
@JosephJacks_
Just finished. Excellent work. Strongly recommended to everyone.
Anil Ananthaswamy
@anilananth
Don't know how other authors feel, but seeing the first physical copy of your book, after years of looking at Word files and PDFs... Seems surreal. It's as if someone else wrote it... Anyway, this advance copy arrived in the mail. One more month to pub date ...www.penguinrandomhouse.com/books/677608/why-machines-lear…
#artificalintelligence @DuttonBooks -
Anil Ananthaswamy reposted
Natalie Wolchover
@nattyover
We’ve spent months putting together an immersive special issue of Quanta Magazine that explores the ultimate scientific quest: the search for the fundamental nature of reality. Don’t miss it!
www.quantamagazine.org/the-unraveling-of-space-time-20240… -
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
The Generator-Verifier Gap. When verifying a solution is easier than generating it (think Chess moves, coding, math), scaling up inference time compute gets better results — @OpenAI's Noam Brown speaking at the Simons Institute's workshop on Transformers as a Computational Model.
-
Anil Ananthaswamy reposted
Marcelo Kauffman
@Marcelokauffman
I have just finished this wonderful book by @anilananth in order to learn more about the maths underpinnings of machine learning. I strongly recommend it for this new era of medicine!
Marcelo Kauffman
@Marcelokauffman
I wish I had receive deeper knowledge into linear algebra and matrices during my medical school!. These mathematical concepts form the backbone of cutting-edge AI applications in medicine, such as the fascinating world of latent spaces and embeddings used in drug repurposing models like TxGNN. Let me explain these concepts in a way that highlights their importance in modern healthcare:
The concept of "latent spaces" and "embeddings" in the context of TxGNN paper (www.nature.com/articles/s41591-024-03233-x) refers to a way of representing complex biomedical information in a format that machine learning models can work with effectively.
A latent space is essentially a compressed, abstract representation of data. In this case, the vast and complex network of biomedical knowledge (diseases, drugs, genes, proteins, etc.) is transformed into a multi-dimensional space where each entity (like a disease or drug) is represented by a point or vector. This vector is called an "embedding."
You can think of embeddings as a way to capture the essence or key features of an entity in a compact form. In the latent space, similar entities are positioned closer together, while dissimilar ones are farther apart. This allows the model to capture complex relationships and similarities that might not be immediately obvious in the original data. For example, two diseases that seem unrelated on the surface might have similar underlying mechanisms at the molecular level. In the latent space, their embeddings might be close together, reflecting this hidden similarity. The model can then use this information to make predictions, such as suggesting that a drug effective for one disease might also work for the other.
This approach allows TxGNN to process and reason over an enormous amount of biomedical knowledge in a computationally efficient manner, enabling it to make predictions that would be extremely difficult or impossible for humans to derive manually from the raw data.
It's a powerful way of distilling complex, interconnected biomedical knowledge into a form that advanced AI models can leverage for tasks like drug repurposing.
This powerful approach to representing biomedical knowledge demonstrates why a strong foundation in mathematics, particularly linear algebra and matrix operations, is becoming increasingly valuable in medicine. As AI continues to revolutionize healthcare, understanding these concepts can help future doctors better collaborate with data scientists and critically evaluate AI-driven medical innovations. -
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
"Validity-breadth trade-offs as a general issue for language generation: Hallucination at one extreme; mode collapse at the other." — Jon Kleinberg of Cornell University at the Simons Institute's workshop on Transformers as a Computational Model. #simonslive
-
Anil Ananthaswamy
@anilananth
The Power of Stories: You might think that something as technical and mathematical as machine learning doesn't benefit from storytelling. Not so. We are a storytelling species, and anytime you can find/tell stories that anchor our understanding, we learn better and retain information for longer, maybe forever. Even the math begins to make sense :-)
Some examples from WHY MACHINES LEARN that I hope make the point:
1) Bernie Widrow and Ted Hoff over the course of a weekend not only came up with the math of the least mean square (LMS) algorithm, but rushed over to the local hardware shop, bought all the parts, and built the world's first hardware artificial neuron that learned using a very noisy form of gradient descent! This was the late 1950s!
2) Peter Hart, a precocious grad student and his whip-smart advisor Thomas Cover, an asst. prof. at Stanford, come up with much of the math that underpins the k-Nearest Neighbor algorithm, in a matter of three months. Hart then spent a summer sailing, while waiting for Cover to return from a trip to MIT, and then essentially submitted his PhD thesis, and his career was launched.
3) John Hopfield--a theoretical physicist who was at a crossroads in his career and looking for new challenges--tells the story of how he went from physics to computational neuroscience and neural networks. He ended up devising Hopfield networks!
4) Then there's the long arc of the story of the development of neural networks: Rosenblatt's first single-layer neural networks and why they enthralled everyone; Minsky and Papert's unfair attacks on neural network research that led to first AI winter; the story of diehard believers (Geoff Hinton among them), who persisted in researching neural networks. And then the breakthrough 1986 paper on backpropagation by Rumelhart, Hinton and Williams (backprop had been worked on by others, including rocketry engineers, but RHW were the first to clearly show its relevance for training multi-layer neural networks)
Once you see the stories, the conceptual difficulties that were overcome on the way to making advances, you see machine learning/deep learning/AI in a new light.
That, at least, is my hope for WHY MACHINES LEARN.
www.penguinrandomhouse.com/books/677608/why-machines-lear…
www.penguin.co.uk/books/446849/why-machines-learn-by-anan… -
Anil Ananthaswamy reposted
Arvind Narayanan
@random_walker
📢 AI Snake Oil is out today! I have too many feelings because this book has been five years in the making. In 2019 I gave a talk on AI Snake Oil and tweeted out the slides. I had no idea my career was about to change. Within a couple of days I had 30 or 40 invites to write a book or an article. I said no to all of them because I felt I needed to do more research to be able to write a book. So when @sayashk joined my team I knew this was the topic we were going to work on. The book is based on our research on limits to prediction, the dangers of predictive decision making, the reproducibility crisis in machine learning based science, the limits of large language models, and the risks of social media algorithms.
The book is not an anti-technology screed. If our point was that all AI is useless, we wouldn’t need a book to say it. It’s precisely because of AI’s usefulness in many areas that overhyped claims and outright snake oil have been so successful in the market. It’s hard for people to tell these apart, and we hope our book can help. And while we think the tech industry should have much more accountability, the harms we describe are usually not solely due to tech, and much more often due to AI being an amplifier of existing problems in our society. A recurring pattern we point out in the book is that "broken AI is appealing to broken institutions" (Chapter 8).
Since our book talks about what AI can’t do, AI boosters often tell us it’s going to be outdated soon. If you think so, I’d love to have a friendly bet! Read the book and tell us what you think will be wrong in two years or five years. (You can start with the intro chapter, which is available online; the link is in my pinned tweet.) The reason we’re confident of our assessments is that they are not based on limitations of the technology. Instead, they are based on inherent limits to prediction grounded in our sociological analysis. (The book owes a huge debt to Princeton sociologist @msalganik; our collaboration with him informed and inspired the book.) We also talk a lot about what we shouldn’t use AI for, even if we can.
We’ve had many moments of hope and trepidation for the book as we worked on it over the years. We’re so grateful for the early interest the book has had, especially to the 40,000 people who subscribe to the AI Snake Oil newsletter, and the many podcasts that have had us on to talk about the book. We’ve had great reviews in The New Yorker, Publishers' Weekly (featured in the Top 10 science books for Fall 2024), and many other outlets. And we’ve had such a wonderful experience working with Princeton University Press. We’re also fortunate to have received incisive peer reviews from @MelMitchell1 , @mollycrockett , @chris_bail, and three anonymous reviewers. The book is so much better as a result.
What's in the book
Chapter 1: Introduction. We begin with a summary of our main arguments in the book. We discuss the definition of AI (and more importantly, why it is hard to come up with one), how AI is an umbrella term, what we mean by AI Snake Oil, and who the book is for.
The entire introduction is available online. See the link in my pinned tweet.
Chapter 2: How predictive AI goes wrong. Predictive AI is used to make predictions about people—will a defendant fail to show up for trial? Is a patient at high risk of negative health outcomes? Will a student drop out of college? These predictions are then used to make consequential decisions. Developers claim predictive AI is groundbreaking, but in reality it suffers from a number of shortcomings that are hard to fix.
Chapter 3: Can AI predict the future? Are the shortcomings of predictive AI inherent, or can they be resolved? In this chapter, we look at why predicting the future is hard — with or without AI. While we have made consistent progress in some domains such as weather prediction, we argue that this progress cannot translate to other settings, such as individuals' life outcomes, the success of cultural products like books and movies, or pandemics.
Chapter 4: The long road to generative AI. Recent advances in generative AI can seem sudden, but they build on a series of improvements over seven decades. In this chapter, we retrace the history of computing advances that led to generative AI.
Chapter 5: Is advanced AI an existential threat? Claims about AI wiping out humanity are common. Here, we critically evaluate claims about AI's existential risk and find several shortcomings and fallacies in popular discussion of x-risk. We discuss approaches to defending against AI risks that improve societal resilience regardless of the threat of advanced AI.
Chapter 6: Why can't AI fix social media? One area where AI is heavily used is content moderation on social media platforms. We discuss the current state of AI use on social media, and highlight seven reasons why improvements in AI alone are unlikely to solve platforms' content moderation woes.
Chapter 7: Why do myths about AI persist? Companies, researchers, and journalists all contribute to AI hype. We discuss how myths about AI are created and how they persist. In the process, we hope to give you the tools to read AI news with the appropriate skepticism and identify attempts to sell you snake oil.
Chapter 8: Where do we go from here? While the previous chapter focuses on the supply of snake oil, in the last chapter, we look at where the demand for AI snake oil comes from. We also look at the impact of AI on the future of work, the role and limitations of regulation, and conclude with vignettes of the many possible futures ahead of us. We have the agency to determine which path we end up on, and each of us can play a role.
-------
We hope you find the book useful and look forward to hearing what you think. -
Anil Ananthaswamy reposted
Sanjeev Arora
@prfsanjeevarora
Nice tutorial + ideas intro ML courses that many of us teach.
Anil Ananthaswamy
@anilananth
How to use a large language model to teach yourself #machinelearning. In the beginning there was the perceptron. So, first steps: Get an LLM to generate code for an interactive user interface that let's you create two clusters of data, and then activate and visualize a perceptron algorithm finding a line to separate the two clusters. Study the code! Prompts for the LLM here: anilananthaswamy.com/why-machines-learn-codebook
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
Misha Belkin on emergence in neural networks using an analogy (Larva—>Pupa—>Butterfly): Continuity is not reflected externally by what you can observe. There might be continuity internally. At the Simons Institute's workshop on "Transformers as a Computational Model" #simonslive
-
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
How to use a large language model to teach yourself #machinelearning. In the beginning there was the perceptron. So, first steps: Get an LLM to generate code for an interactive user interface that let's you create two clusters of data, and then activate and visualize a perceptron algorithm finding a line to separate the two clusters. Study the code! Prompts for the LLM here: anilananthaswamy.com/why-machines-learn-codebook
-
Anil Ananthaswamy
@anilananth
One of the most gorgeous covers I have ever seen!
-
Anil Ananthaswamy
@anilananth
How to use a large language model to teach yourself #machinelearning. In the beginning there was the perceptron. So, first steps: Get an LLM to generate code for an interactive user interface that let's you create two clusters of data, and then activate and visualize a perceptron algorithm finding a line to separate the two clusters. Study the code! Prompts for the LLM here: anilananthaswamy.com/why-machines-learn-codebook
-
Anil Ananthaswamy
@anilananth
I was thinking the same thing, and took the first step in that direction: anilananthaswamy.com/why-machines-learn-codebook
-
Anil Ananthaswamy
@anilananth
From Rosenblatt to Claude
Using a modern LLM to generate code to visualize Frank Rosenblatt’s perceptron algorithm pays tribute to Rosenblatt’s visionary ideas. In 1958, he designed the first learnable artificial neurons and the first single-layer neural networks. LLMs are their undeniable descendants. So, it's sweet to get an LLM to help us code/visualize the perceptron algorithm.
Here's my blog post detailing the sequence of prompts it took to get @AnthropicAI's Claude 3.5 Sonnet to produce code that implements an interactive UI to run and visualize the perceptron algorithm: anilananthaswamy.com/why-machines-learn-codebook
I was going to one day put the code I wrote for WHY MACHINES LEARN on to @github, but I think using LLMs to generate the code by reading the book, understanding the basics, and figuring out the prompts is a wonderful way of learning the conceptual/mathematical/coding basics of ML. -
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
"We discover a prompting strategy that causes LLMs to diverge and emit verbatim pre-training examples" —
Aditi Raghunathan (CMU) on "Challenges in building safe and robust LLMs" at the Simons Institute's Special Year on Large Language Models and Transformers, Part 1 Boot Camp -
Anil Ananthaswamy reposted
MIT CSAIL
@MIT_CSAIL
70 years ago this month, researchers ran the first FORTRAN program, considered the beginning of general-purpose programming: www.informationweek.com/it-leadership/fortran-7-reasons-w…
Image v/@NASA -
Anil Ananthaswamy reposted
Erdal
@ErdalToprak
It’s going to be a great weekend! @anilananth
-
Anil Ananthaswamy
@anilananth
"An enchanting journey into the mathematical heart and the history of machine learning." -- It's rather sweet when your alma mater's magazine (@iitmadras's Shaastra Magazine) reviews your book and the review is positive. @Shubawrite had some fun with the review, concluding: "In sum, the merits of this book can be encapsulated in a mathematical equation: Math + history + humour = A joyous read"
Full review here: shaastramag.iitm.ac.in/books/joy-maths
www.penguinrandomhouse.com/books/677608/why-machines-lear… -
Anil Ananthaswamy reposted
Kevin Mitchell
@WiringTheBrain
Excited to share the cover for the Chinese translation of FREE AGENTS, which will be released by the China Translation & Publishing House 😊
-
Anil Ananthaswamy reposted
Alfredo Canziani
@alfcnz
«Linear Algebra and Its Applications» by David & Steven Lay, and Judi McDonald is truly a masterpiece.
A textbook teaching Linear Algebra geometrically, application first, with computer exercises and data sets, interactive figures, and projects.
🤩🤩🤩 -
Anil Ananthaswamy reposted
Martin
@martinmrmar
My daily math grind after work. No matter how tired I am, I always manage to do some math. Even if it's just one good exercise.
-
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
"The structure of data is the dark matter of theory in deep learning" — @SuryaGanguli during his talk on "Perspectives from Physics, Neuroscience, and Theory" at the Simons Institute's Special Year on Large Language Models and Transformers, Part 1 Boot Camp.
-
Anil Ananthaswamy reposted
Danielle Newnham
@daniellenewnham
This week's guest on the pod is the fantastic @anilananth - science writer and author of Why Machines Learn: The Elegant Math Behind Modern AI.
Tune in to find out why @geoffreyhinton and @maxsbennett think his latest book is a masterpiece!
danielle-newnham-podcast.simplecast.com/episodes/anil-ana… -
Anil Ananthaswamy
@anilananth
Thanks for a lovely conversation @daniellenewnham! I enjoyed it immensely.
Danielle Newnham
@daniellenewnham
This week's guest on the pod is the fantastic @anilananth - science writer and author of Why Machines Learn: The Elegant Math Behind Modern AI.
Tune in to find out why @geoffreyhinton and @maxsbennett think his latest book is a masterpiece!
danielle-newnham-podcast.simplecast.com/episodes/anil-ana… -
Anil Ananthaswamy reposted
Gianpaolo Galli
@GianpaoloGalli
Reading "Why Machines Learn: The Elegant Maths Behind Modern AI" by @anilananth
-
Anil Ananthaswamy
@anilananth
Thanks for making the trek to the East Bay, @paldhous. A lovely evening with you and @FreelanceAstro (Adam Becker!)... :-)
Peter Aldhous
@paldhous
Just started reading. Also great to hang out with author @anilananth, back in CA after long absence!
-
Anil Ananthaswamy reposted
SFI Press
@SFIPress
Launching today: A standalone version of @sfiscience David C. Krakauer’s introduction to the Foundational Papers of Complexity Science! sfipress.org/books/the-complex-world
-
Anil Ananthaswamy
@anilananth
By now, most machine learning folks know about a new dataset in town for beginners — the Penguin dataset. For the longest time, the honor for being a beginner-friendly dataset went to the Iris dataset, about irises. But the Penguin dataset—collected just as painstakingly as was done with the flowers—is a fantastic alternative. More features, and just more fun!
In WHY MACHINES LEARN, I do talk of the history of the collection of the Iris dataset and use it to discuss principal component analysis and unsupervised learning.
But the Penguin dataset plays a prominent role in the chapter on probability and statistics: it forms the backbone of the examples used to talk about the optimal Bayes and Naïve Bayes classifiers, and in general, to give readers an intuition for what it means to estimate probability distributions.
Here’s the Penguin Dataset on @github:
allisonhorst.github.io/palmerpenguins/
And on @kaggle: www.kaggle.com/code/parulpandey/penguin-dataset-the-new-iris
For more, please see WHY MACHINES LEARN:
𝐔𝐒:penguinrandomhouse.com/books/677608/why-machines-learn-by…
𝐔𝐊: penguin.co.uk/books/446849/why-machines-learn-by-ananthas… -
Anil Ananthaswamy reposted
Simons Institute for the Theory of Computing
@SimonsInstitute
Overheard at the Simons Institute: “Generalization and hallucinations are two sides of the same coin.” Apt, given the two ongoing programs this semester — on generalization, and on large language models and transformers.
simons.berkeley.edu/programs/modern-paradigms-generalization
simons.berkeley.edu/programs/special-year-large-language-… -
Anil Ananthaswamy
@anilananth
It's a long wait from when authors see the first cover designs of their upcoming book to the moment they hold the physical book in their hands. It's often worth the wait. The cover of WHY MACHINES LEARN was inspired by an M.C. Escher wood engraving, named Three Spheres I. You can read about it here: www.escherinhetpaleis.nl/escher-today/three-spheres-i/?la…
Escher wanted to "evoke a three-dimensional form on a flat surface."
It's an apt piece of art to draw inspiration from for machine learning, which so often deals with moving between different dimensions when it comes to representing information (think PCA and Kernel methods).
www.penguinrandomhouse.com/books/677608/why-machines-lear… -
Anil Ananthaswamy reposted
Marcos Demetry
@MarcosDemetry
“Why Machines Learn” by @anilananth is the opposite of clickbait. No buzzwords, just beautiful concepts explained pedagogically with mathematical equations, text, examples and illustrations. The history and current state of ML is fascinating! Soon on Ceres Books 📚
-
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
There’s no better illustration of how uncertainty messes with our minds than the Monty Hall dilemma. If you haven’t tried answering the question posed by the host of the game show Let’s Make a Deal, give it go and see why it fooled the best of mathematicians.
Here’s the question: “Suppose you are on a game show, and you’re given the choice of three doors. Behind one is a car; behind the others, goats. You pick a door, say, No. 1, and the host, who knows what’s behind the doors, opens another door, say, No. 3, which has a goat. He then says to you, ‘Do you want to pick No. 2?’
Monty Hall is asking: Do you want to switch your choice from door No. 1 to door No. 2, before he opens the doors to reveal which one has the car?
Does it make sense to switch, in that it increases your chances of winning the car?
The chapter on probability and statistics in WHY MACHINES LEARN begins with this story, providing seemingly intuitive arguments for both answers: Yes, you should switch your choice of door to increase your odds and No, you shouldn’t it, for it’d make no difference.
But the real reason for talking about the Monty Hall dilemma is to introduce one of the cornerstones of the study of probabilities: Bayes Theorem.
You can use Bayes theorem to show that if you switch doors, the probability of winning is 2/3, whereas if you don’t, it’s 1/3.
Sometimes human intuition fails when reasoning in the presence of uncertainty. Bayes’ theorem allows us to avoid such pitfalls. It's astonishing how this centuries-old theorem underpins so much of machine learning.
The image (left) is a wonderful illustration of the problem that appeared in the PhysicsWorld review of WHY MACHINES LEARN: physicsworld.com/a/goats-sports-cars-and-game-shows-the-u…
Here's more info about WHY MACHINES LEARN: 𝐔𝐒 𝐄𝐝𝐢𝐭𝐢𝐨𝐧: penguinrandomhouse.com/books/677608/why-machines-learn-by…
𝐔𝐊 𝐄𝐝𝐢𝐭𝐢𝐨𝐧: penguin.co.uk/books/446849/why-machines-learn-by-ananthas… -
Anil Ananthaswamy
@anilananth
There’s no better illustration of how uncertainty messes with our minds than the Monty Hall dilemma. If you haven’t tried answering the question posed by the host of the game show Let’s Make a Deal, give it go and see why it fooled the best of mathematicians.
Here’s the question: “Suppose you are on a game show, and you’re given the choice of three doors. Behind one is a car; behind the others, goats. You pick a door, say, No. 1, and the host, who knows what’s behind the doors, opens another door, say, No. 3, which has a goat. He then says to you, ‘Do you want to pick No. 2?’
Monty Hall is asking: Do you want to switch your choice from door No. 1 to door No. 2, before he opens the doors to reveal which one has the car?
Does it make sense to switch, in that it increases your chances of winning the car?
The chapter on probability and statistics in WHY MACHINES LEARN begins with this story, providing seemingly intuitive arguments for both answers: Yes, you should switch your choice of door to increase your odds and No, you shouldn’t it, for it’d make no difference.
But the real reason for talking about the Monty Hall dilemma is to introduce one of the cornerstones of the study of probabilities: Bayes Theorem.
You can use Bayes theorem to show that if you switch doors, the probability of winning is 2/3, whereas if you don’t, it’s 1/3.
Sometimes human intuition fails when reasoning in the presence of uncertainty. Bayes’ theorem allows us to avoid such pitfalls. It's astonishing how this centuries-old theorem underpins so much of machine learning.
The image (left) is a wonderful illustration of the problem that appeared in the PhysicsWorld review of WHY MACHINES LEARN: physicsworld.com/a/goats-sports-cars-and-game-shows-the-u…
Here's more info about WHY MACHINES LEARN: 𝐔𝐒 𝐄𝐝𝐢𝐭𝐢𝐨𝐧: penguinrandomhouse.com/books/677608/why-machines-learn-by…
𝐔𝐊 𝐄𝐝𝐢𝐭𝐢𝐨𝐧: penguin.co.uk/books/446849/why-machines-learn-by-ananthas… -
Anil Ananthaswamy
@anilananth
Spent an awesome afternoon with @aminkarbasi and folks at Robust Intelligence talking about ML, AI, and even about consciousness and self models! It was truly a pleasure.
-
Anil Ananthaswamy reposted
Sanjeev Arora
@prfsanjeevarora
1/ LLMs are often used to generate text new math questions. But can they generate challenging math questions? Current methods yield Qs that're either easy or too similar to existing questions. Our new paper "AI-Assisted Generation of Difficult Math Questions" shows how to generate more difficult math questions, with humans in the loop for verifying correctness. Key point: the new questions still test the *same* underlying skills as MATH, but are harder. Arguably, these questions are "out of distribution" (OOD) and all tested models find these questions much harder. See the table lower down.
-
Anil Ananthaswamy reposted
Joscha Bach
@Plinz
The hallucinations of LLMs are not a problem per se: human minds hallucinate too. If we remove the constraints of perceptual input, we spin off into dreams. If we don't verify our intuitive generations with reasoning, our thinking becomes delusional. Generative AI is incomplete.
-
Anil Ananthaswamy reposted
Quanta Magazine
@QuantaMagazine
Recently, Ziming Liu, a physics grad student at MIT, and his adviser Max Tegmark (@tegmark) built the first functioning Kolmogorov-Arnold networks — AI systems that are easier for scientists to understand than conventional neural networks.
www.quantamagazine.org/novel-architecture-makes-neural-ne… -
Anil Ananthaswamy reposted
Sebastian Raschka
@rasbt
Today is the day! 🎉 After 1.5 years of hard work, "Build A Large Language Model (From Scratch)" is finally out!
Print or ebook copies are available on Manning’s site: mng.bz/M96o 📚.
And it's also available on Amazon soon: www.amazon.com/Build-Large-Language-Model-Scratch/dp/1633… 📦. -
Anil Ananthaswamy reposted
Next Big Idea Club
@NextBigIdeaClub
The math behind machine learning and AI is beautiful. Here are a few key insights from Why Machines Learn: The Elegant Math Behind Modern AI, by @anilananth.
nextbigideaclub.com/magazine/get-elegant-math-unlocked-ai… -
Anil Ananthaswamy reposted
Karmela Padavic-Callaghan (they/them)
@kpcallaghan_
Incredibly grateful to have the cover story for @newscientist this week, all about realism in quantum theory
-
Anil Ananthaswamy
@anilananth
Kinda ironic that @X's algorithm tags a tweet about the curse of dimensionality and machine learning as "adult content" :-) Sentiment analysis gone wrong somewhere!
-
Anil Ananthaswamy
@anilananth
Yes please 🙂
Melanie Mitchell
@MelMitchell1
Dear @techreview,
We need "60 Innovators Over 60" !!
www.technologyreview.com/innovators-under-35/2024 -
Anil Ananthaswamy
@anilananth
Here’s a powerful visual intuition for understanding the curse of dimensionality, which is a seminal issue in all of machine learning.
Imagine a 3D cube, with its center at the origin. Enclosed within the cube is a unit sphere, with a radius of one.
Something amazing happens as you increase the number of dimensions. The volume of the unit sphere begins to tend to zero as you go to higher and higher dimensions (there’s an intricate formula for calculating the volume of the sphere in high dimensions that leads to this result).
The volume of the hypercube, however, doesn’t drop off. But, the sphere is touching the sides of the hypercube. So, you have to imagine the hypercube as a geometric structure whose insides (where the unit hypersphere exists) have no volume, and all the volume is at the vertices of the hypercube.
So, any data points that are embedded in the hypercube end up in the vertices. And these vertices are equidistant from each other, and for a 1000-dimensional hypercube, the number of vertices is literally astronomical!
This is a way of saying that any data that is represented in extremely high dimensions creates a problem: all data points are equidistant from each other, and using distance measures (such as the Euclidean distance) to gauge the similarity between data points falls apart. The curse of dimensionality strikes.
All this and more is described in WHY MACHINES LEARN (and the sources are acknowledged).
𝐔𝐒 𝐄𝐝𝐢𝐭𝐢𝐨𝐧: penguinrandomhouse.com/books/677608/why-machines-learn-by…
𝐔𝐊 𝐄𝐝𝐢𝐭𝐢𝐨𝐧: penguin.co.uk/books/446849/why-machines-learn-by-ananthas… -
Anil Ananthaswamy reposted
Ash Jogalekar
@curiouswavefn
My baloney meter just broke interestingengineering.com/science/general-relativity-qua…
-
Anil Ananthaswamy reposted
Massimo
@Rainmaker1973
Listen to the deep tones of this 432Hz Bass D drone Native American style flute.
[📹 Christian Dimarco Music] -
Anil Ananthaswamy
@anilananth
If WHY MACHINES LEARN gives readers insights into how machine learning is so, so much more than deep learning and deep neural networks, I'd be thrilled. And yes, it's not just statistics :-)
A reader's review on the website that shall not be named had this sentence, which pleased me no end: "I've never been a fan of kernel methods and SVMs etc, but that section of the book was actually quite exciting and gave me more appreciation for this part of the field, for example."
SVMs are, of course, Support Vector Machines.
The chapter of SVMs and kernel methods--called The Great Kernel Rope Trick--is like the pivot point in the book, as it vaults the reader from the ML algorithms of old to deep neural networks. Kernel methods are such appropriate techniques on which to pivot.
It turns out that in the limit of, say, neural networks with infinite width hidden layers, a neural network acts like a kernel machine! There are deep connections between the more classical ideas in machine learning and deep learning, the poster child of ML these days.
Here's a @QuantaMagazine article I wrote about that correspondence a couple of years ago: www.quantamagazine.org/a-new-link-to-an-old-model-could-c…
Here's more info about WHY MACHINES LEARN:
𝐔𝐒 𝐄𝐝𝐢𝐭𝐢𝐨𝐧:
www.penguinrandomhouse.com/books/677608/why-machines-lear…
𝐔𝐊 𝐄𝐝𝐢𝐭𝐢𝐨𝐧:
www.penguin.co.uk/books/446849/why-machines-learn-by-anan… -
Anil Ananthaswamy
@anilananth
One of the joys of opening boxes of books that I hadn't seen for 2 years+... And finding gems I want to reread. The late Dan Dennett's Intuition Pumps and @anilkseth's Being You.
-
Anil Ananthaswamy reposted
François Chollet
@fchollet
Everything wrong about our perception of machine learning and AI is due to excessive anthropomorphism -- which the field actively promotes.
-
Anil Ananthaswamy
@anilananth
Thank you for alerting me to @burkov's work! And humbled to find myself on this list!
Dorian Harris
@dorianjharris
@pmddomingos
@AndrewYNg
@JeffDean
@tabithagold
@mustafasuleymn
@etzioni
@anilananth
@burkov
@GaryMarcus
Yes, way more than a year and well beyond LLMs of which some of them are sceptical. -
Anil Ananthaswamy
@anilananth
.@burkov I wish I had come across your book while I was doing research for mine! Looks fascinating. Looking forward to digging into it.
themlbook.com -
Anil Ananthaswamy
@anilananth
Highly recommended residency for journalists
HITS
@HITStudies
One week to go - Apply now!
-
Anil Ananthaswamy
@anilananth
Of the major math disciplines that underpin machine learning (ML), which one would you say is most important for appreciating what ML does?
Let’s take these four disciplines: Linear Algebra, Probability and Statistics, Calculus, Optimization Theory
My major revelation while writing WHY MACHINES LEARN (www.penguinrandomhouse.com/books/677608/why-machines-lear…) was that to truly grok what machine learning is about, you need to get a good grasp of probability and statistics!
Yes, linear algebra is hugely important, and so is calculus, but they to me seemed more about the mechanics of getting machines to learn; but in order to appreciate what these machines are learning, it’s probability and statistics that are key.
At some fundamental level, many ML algorithms are trying to learn or estimate the underlying probability distribution from which the training data has been sampled. Once you have a good handle on this distribution, you can do discriminative tasks (tell apart one class of data from another), or generative tasks (generate new data by sampling from the learned/estimated distribution).
Even algorithms that are not explicitly probabilistic have an inherent risk of making errors, so one can calculate the probabilities of the algorithm getting things wrong (even though everything about the algorithm seems deterministic).
Anyway, here’s an informal poll. What do you think? Feel free to comment/critique. -
Anil Ananthaswamy reposted
Math Cafe
@Riazi_Cafe_en
Harvard's lecture notes on "Probability, Statistics, and Linear Algebra"
people.math.harvard.edu/~knill/teaching/math19b_2011/hand… -
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
Why do Large Language Models (LLMs) need an Internet's worth of data to train? Here’s a perspective that helps appreciate what LLMs do and why they are such data hogs. It’s related to the curse of dimensionality.
Consider the output of an LLM like ChatGPT when you prompt it with, say, 100 words. It generates the next most likely word to follow those 100 words, appends it to the prompt, generates the next word, appends that to the previous 101 words, generates the next word, and keeps going, until it generates some end-of-text symbol (I’m using words instead of tokens here for simplicity).
That’s the big picture. But what’s happening at each step?
Let’s say the LLM has a vocabulary of 1000 words. Now, given a prompt of 100 words, it has to calculate the condition probability distribution for the next word, given a particular sequence of 100 words: P(w101 | w1, w2, …, w100)
Or, more generally, given a prompt of (n-1) words, it has to calculate the conditional probability distribution for the nth word over the entire vocabulary of 1000 words.
So, P (wn | w1, w2, …, wn-1)
What this means is that the LLM arrives at a probability for each of the 1000 words in the vocabulary: the word with the highest probability is the most likely word to follow the prompt.
This is at inference time.
During training, the job of the training algorithm is to get the LLM to learn an extremely complex, extremely high-dimensional probability distribution over the entire vocabulary. Learning or estimating such a high-dimensional surface is a very, very difficult task: the number of data samples you need grows exponentially with the number of dimensions. This is the curse of dimensionality.
If the LLM doesn’t learn this surface accurately enough, it’ll make errors during inference. That’s one reason why smaller models (think GPT-2) were terrible at answering prompts, while bigger models (trained using orders of magnitude more data and compute) did better at those same prompts.
The curse of dimensionality and ways to alleviate it play an important role in machine learning—and it forms a theme in WHY MACHINES LEARN. Understanding it is key to making sense of the strengths and weaknesses of ML algorithms.
www.penguinrandomhouse.com/books/677608/why-machines-lear… -
Anil Ananthaswamy
@anilananth
Why do Large Language Models (LLMs) need an Internet's worth of data to train? Here’s a perspective that helps appreciate what LLMs do and why they are such data hogs. It’s related to the curse of dimensionality.
Consider the output of an LLM like ChatGPT when you prompt it with, say, 100 words. It generates the next most likely word to follow those 100 words, appends it to the prompt, generates the next word, appends that to the previous 101 words, generates the next word, and keeps going, until it generates some end-of-text symbol (I’m using words instead of tokens here for simplicity).
That’s the big picture. But what’s happening at each step?
Let’s say the LLM has a vocabulary of 1000 words. Now, given a prompt of 100 words, it has to calculate the condition probability distribution for the next word, given a particular sequence of 100 words: P(w101 | w1, w2, …, w100)
Or, more generally, given a prompt of (n-1) words, it has to calculate the conditional probability distribution for the nth word over the entire vocabulary of 1000 words.
So, P (wn | w1, w2, …, wn-1)
What this means is that the LLM arrives at a probability for each of the 1000 words in the vocabulary: the word with the highest probability is the most likely word to follow the prompt.
This is at inference time.
During training, the job of the training algorithm is to get the LLM to learn an extremely complex, extremely high-dimensional probability distribution over the entire vocabulary. Learning or estimating such a high-dimensional surface is a very, very difficult task: the number of data samples you need grows exponentially with the number of dimensions. This is the curse of dimensionality.
If the LLM doesn’t learn this surface accurately enough, it’ll make errors during inference. That’s one reason why smaller models (think GPT-2) were terrible at answering prompts, while bigger models (trained using orders of magnitude more data and compute) did better at those same prompts.
The curse of dimensionality and ways to alleviate it play an important role in machine learning—and it forms a theme in WHY MACHINES LEARN. Understanding it is key to making sense of the strengths and weaknesses of ML algorithms.
www.penguinrandomhouse.com/books/677608/why-machines-lear… -
Anil Ananthaswamy
@anilananth
In hindsight, the innovation is so basic, it beggars belief that it set a slow-burning revolution in motion, where AI is concerned. It was geeky, nerdy tweak to the way artificial neurons work, and it enabled the development of the backpropagation algorithm—the workhorse of modern artificial intelligence.
Until the 1970s, people were using so-called binary threshold neurons for their neural networks. A neuron would compute a weighted sum of its inputs. If the weighted sum was greater than some threshold, it would output a 1, else it’d output a 0. The neuron had, in effect, a threshold activation function.
Rumelhart, Hinton and Williams, in their landmark 1986 paper on backpropagation made a small tweak: instead of a threshold activation function they used a sigmoid activation function, which smoothly transitions from almost 0 to almost 1, depending on the weighted sum. See figure.
This tiny tweak made all the difference. The sigmoid function is differentiable everywhere. The threshold function isn’t differentiable everywhere.
Now, the entire chain of computation, from the input to the output, regardless of how many neurons are in that path, became differentiable. It was possible to calculate the loss made by the network, calculate the contribution of each parameter or weight to the loss and update it. It required something as simple as the chain rule from basic calculus.
It's really hard to imagine that this small tweak—the idea that the activation function inside a neuron needs to be differentiable or approximately differentiable everywhere—was key to the ongoing revolution in deep learning and artificial intelligence. Of course, it’d take decades for other pieces to fall into place (compute power, enormous training data, etc.), but this simple innovation set things in motion.
All this and more is narrated, along with the math, in WHY MACHINES LEARN:
𝐔𝐒: penguinrandomhouse.com/books/677608/why-machines-learn-by…
𝐔𝐊: penguin.co.uk/books/446849/why-machines-learn-by-ananthas…
#artificialintelligence #deeplearning #machinelearning #ai -
Anil Ananthaswamy
@anilananth
Thank you @CAS_ReproLab! Music to my ears😀
Cameron A. Schmidt, PhD
@CAS_ReproLab
Really enjoying this #book🔥. Very accessible and informative. Bravo @anilananth
-
Anil Ananthaswamy reposted
Cameron A. Schmidt, PhD
@CAS_ReproLab
Really enjoying this #book🔥. Very accessible and informative. Bravo @anilananth
-
Anil Ananthaswamy reposted
Spiros Margaris
@SpirosMargaris
Get a copy of this book, I did.
Why Machines Learn by Anil Ananthaswamy:
www.penguinrandomhouse.com/books/677608/why-machines-lear… @anilananth @PenguinBooks -
Anil Ananthaswamy
@anilananth
When you first encounter backpropagation, the algorithm used to train deep neural networks, it seems arcane, mysterious. I remember being flummoxed. But ever-so-slowly the algorithm’s simplicity and power dawns on you. It works for the simplest possible network with one hidden neuron and for something as complex as today’s large language models!
In WHY MACHINES LEARN, we first encounter the early artificial neural networks: Rosenblatt’s perceptrons, which were single layer networks: inputs come in, get processed by a layer of neurons, which produce some output. Rosenblatt’s training algorithm worked wonderfully well (given some assumptions). But add just one extra layer of neurons between the input and the output, and things fell apart. No one knew how to train them. Rosenblatt’s perceptron training algorithm didn’t work.
It helps to know the history of this effort to appreciate how we got to backpropagation. Given backpropagation's simplicity, it’s amazing it took decades (late 1950s to mid-1980s) to get a working algorithm. But a few simple yet crucial innovations had to happen.
WHY MACHINES LEARN shows the math for how you’d train the simplest possible neural network with a hidden layer (top left panel) and why that can be easily generalized to something complex (top right panel). Digging into the math is delicious.
More here:
𝐔𝐒: penguinrandomhouse.com/books/677608/why-machines-learn-by…
𝐔𝐊: penguin.co.uk/books/446849/why-machines-learn-by-ananthas… -
Anil Ananthaswamy reposted
Paul Halpern
@phalpern
The Allure of the Multiverse
An extraordinary voyage into the history of how parallel universes, higher dimensions and time travel have captivated us in imaginative books, films, and series, while stirring raging controversy in science.
Now available:
amazon.com/Allure-Multiverse-Dimensions-Parallel-Universe…
amazon.co.uk/Allure-Multiverse-Dimensions-Parallel-Univer…
barnesandnoble.com/w/the-allure-of-the-multiverse-paul-ha…
hachettebookgroup.com/titles/paul-halpern/the-allure-of-t…
booksamillion.com/p/9781541602175
bookshop.org/p/books/the-allure-of-the-multiverse-extra-d…
target.com/p/the-allure-of-the-multiverse-by-paul-halpern…
walmart.com/ip/The-Allure-of-the-Multiverse-Extra-Dimensi…
Image -
Anil Ananthaswamy
@anilananth
Sep 5 is Teacher's Day in India. As it happens, WHY MACHINES LEARN is dedicated to "Teachers everywhere, sung and unsung"--as a way of acknowledging the enormous debt I owe to so many from whom I learned about machine learning and the math of it! And I got to squeeze into the dedication page an awesome line delivered by Liam Neeson in the movie Mark Felt (about the FBI agent) 😀
penguinrandomhouse.com/books/677608/why-machines-learn-by… -
Anil Ananthaswamy
@anilananth
.@zzsylvester Thanks for your appreciation for the math in WHY MACHINES LEARN. I was honestly on tenterhooks when the book was released, wondering if the math was a step too far for a popular science book, so it has been very gratifying to read comments from readers such as yourself!
-
Anil Ananthaswamy reposted
Alfredo Canziani
@alfcnz
I'm really having a blast at writing the historical section side notes! 🤩🤩🤩
After the 1969 Minsky & Papert book, we went through the first AI winter. Widrow, kept working on neural nets but rebranded them as adaptive filters, which are now ubiquitous. -
Anil Ananthaswamy reposted
Sean Carroll
@seanmcarroll
Excited to announce the Johns Hopkins Natural Philosophy Symposium, May 29-31 2025 in Baltimore. An all-star list of speakers discussing issues at the intersection of science and philosophy.
www.naturalphilosophyhopkins.org/natural-philosophy-sympo… -
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
The Beauty of Kernel Methods: A visual intuition for why the explosive combination of support vector machines (SVMs) and kernel methods made SVMs so hot—before neural networks dethroned them in the early 2010s.
SVMs find an optimal classification boundary between, for example, two classes of data (circles and triangles, in our visualization). See top left panel.
But if the original dataset is not linearly separable (top right panel), one can project the data into higher dimensions by creating new features. In the new space, the data is linearly separable. Projected back into the lower dimensions, you see a non-linear boundary. See bottom left panel.
Let’s say you start with an original dataset with two features, x1 and x2. The data is not linearly separable in two dimensions. Let’s add an additional non-linear feature: (x1-squared + x2-squared), or (x1sq + x2sq). Each data point or vector is now given by (x1, x2, x1sq+x2sq)
Now, the dataset will be linearly separable in 3 dimensions (the algorithm finds a plane), and projected back into 2 dimensions, the boundary is non-linear.
But here’s a big problem: when you project data into higher dimensions, you have to do dot products of vectors in the higher dimensional space. Depending on how high you go, this can become prohibitively expensive, computationally speaking. And especially if you project into infinite dimensional spaces (yes, you can project into infinite dimensions)!
This is where kernel methods enter the picture.
Consider a mapping a -> phi (a), where phi(a) is the higher dimensional vector. Similarly, b -> phi(b).
Where a = [a1 a2 a1sq+a2sq] and b = [b1 b2 b1sq+b2sq]
If you can find a function k (a, b) such that k (a, b) = phi (a) dot phi (b), then you have a function k () that gives you the result of the dot product of two vectors in the higher dimensional (even infinite dimensional) space but carries out the computation in the lower dimensional space of the original a, b vectors.
Now, you can combine the original SVM algorithm with kernel methods, project your data into whatever higher dimensional space you desire, and as long as you can find/design an appropriate kernel method, you can find any non-linear boundary to separate your data clusters! It’s insanely beautiful and powerful.
The story of SVMs, kernel methods and all the relevant math is in WHY MACHINES LEARN:
See:
𝐔𝐒: penguinrandomhouse.com/books/677608/why-machines-learn-by…
𝐔𝐊: penguin.co.uk/books/446849/why-machines-learn-by-ananthas… -
Anil Ananthaswamy reposted
Mark Datko
@mark_datko
Every Mathematics and Computer Science undergraduate starting this Autumn/Fall should buy a copy .
-
Anil Ananthaswamy reposted
Faisal Abid
@FaisalAbid
This is a fantastic book. Highly recommend
Anil Ananthaswamy
@anilananth
The Beauty of Kernel Methods: A visual intuition for why the explosive combination of support vector machines (SVMs) and kernel methods made SVMs so hot—before neural networks dethroned them in the early 2010s.
SVMs find an optimal classification boundary between, for example, two classes of data (circles and triangles, in our visualization). See top left panel.
But if the original dataset is not linearly separable (top right panel), one can project the data into higher dimensions by creating new features. In the new space, the data is linearly separable. Projected back into the lower dimensions, you see a non-linear boundary. See bottom left panel.
Let’s say you start with an original dataset with two features, x1 and x2. The data is not linearly separable in two dimensions. Let’s add an additional non-linear feature: (x1-squared + x2-squared), or (x1sq + x2sq). Each data point or vector is now given by (x1, x2, x1sq+x2sq)
Now, the dataset will be linearly separable in 3 dimensions (the algorithm finds a plane), and projected back into 2 dimensions, the boundary is non-linear.
But here’s a big problem: when you project data into higher dimensions, you have to do dot products of vectors in the higher dimensional space. Depending on how high you go, this can become prohibitively expensive, computationally speaking. And especially if you project into infinite dimensional spaces (yes, you can project into infinite dimensions)!
This is where kernel methods enter the picture.
Consider a mapping a -> phi (a), where phi(a) is the higher dimensional vector. Similarly, b -> phi(b).
Where a = [a1 a2 a1sq+a2sq] and b = [b1 b2 b1sq+b2sq]
If you can find a function k (a, b) such that k (a, b) = phi (a) dot phi (b), then you have a function k () that gives you the result of the dot product of two vectors in the higher dimensional (even infinite dimensional) space but carries out the computation in the lower dimensional space of the original a, b vectors.
Now, you can combine the original SVM algorithm with kernel methods, project your data into whatever higher dimensional space you desire, and as long as you can find/design an appropriate kernel method, you can find any non-linear boundary to separate your data clusters! It’s insanely beautiful and powerful.
The story of SVMs, kernel methods and all the relevant math is in WHY MACHINES LEARN:
See:
𝐔𝐒: penguinrandomhouse.com/books/677608/why-machines-learn-by…
𝐔𝐊: penguin.co.uk/books/446849/why-machines-learn-by-ananthas… -
Anil Ananthaswamy reposted
ಯಂಕ ನಾಣಿ ಸೀನ
@SinaYanka
I'm reading this book right now. I recommend it for everyone who can deal with some basic maths at a conceptual level.
Infact, I'm recommending this to all teenagers as an introductory book
Anil Ananthaswamy
@anilananth
The Beauty of Kernel Methods: A visual intuition for why the explosive combination of support vector machines (SVMs) and kernel methods made SVMs so hot—before neural networks dethroned them in the early 2010s.
SVMs find an optimal classification boundary between, for example, two classes of data (circles and triangles, in our visualization). See top left panel.
But if the original dataset is not linearly separable (top right panel), one can project the data into higher dimensions by creating new features. In the new space, the data is linearly separable. Projected back into the lower dimensions, you see a non-linear boundary. See bottom left panel.
Let’s say you start with an original dataset with two features, x1 and x2. The data is not linearly separable in two dimensions. Let’s add an additional non-linear feature: (x1-squared + x2-squared), or (x1sq + x2sq). Each data point or vector is now given by (x1, x2, x1sq+x2sq)
Now, the dataset will be linearly separable in 3 dimensions (the algorithm finds a plane), and projected back into 2 dimensions, the boundary is non-linear.
But here’s a big problem: when you project data into higher dimensions, you have to do dot products of vectors in the higher dimensional space. Depending on how high you go, this can become prohibitively expensive, computationally speaking. And especially if you project into infinite dimensional spaces (yes, you can project into infinite dimensions)!
This is where kernel methods enter the picture.
Consider a mapping a -> phi (a), where phi(a) is the higher dimensional vector. Similarly, b -> phi(b).
Where a = [a1 a2 a1sq+a2sq] and b = [b1 b2 b1sq+b2sq]
If you can find a function k (a, b) such that k (a, b) = phi (a) dot phi (b), then you have a function k () that gives you the result of the dot product of two vectors in the higher dimensional (even infinite dimensional) space but carries out the computation in the lower dimensional space of the original a, b vectors.
Now, you can combine the original SVM algorithm with kernel methods, project your data into whatever higher dimensional space you desire, and as long as you can find/design an appropriate kernel method, you can find any non-linear boundary to separate your data clusters! It’s insanely beautiful and powerful.
The story of SVMs, kernel methods and all the relevant math is in WHY MACHINES LEARN:
See:
𝐔𝐒: penguinrandomhouse.com/books/677608/why-machines-learn-by…
𝐔𝐊: penguin.co.uk/books/446849/why-machines-learn-by-ananthas… -
Anil Ananthaswamy
@anilananth
Flew into San Francisco yesterday after a long time. A box of copies of WHY MACHINES LEARN awaited. Was a pleasure to unbox the books, again, in the East Bay, where my software journey began in the 1990s, and then a transition to writing!
www.penguinrandomhouse.com/books/677608/why-machines-lear… -
Anil Ananthaswamy
@anilananth
The Beauty of Kernel Methods: A visual intuition for why the explosive combination of support vector machines (SVMs) and kernel methods made SVMs so hot—before neural networks dethroned them in the early 2010s.
SVMs find an optimal classification boundary between, for example, two classes of data (circles and triangles, in our visualization). See top left panel.
But if the original dataset is not linearly separable (top right panel), one can project the data into higher dimensions by creating new features. In the new space, the data is linearly separable. Projected back into the lower dimensions, you see a non-linear boundary. See bottom left panel.
Let’s say you start with an original dataset with two features, x1 and x2. The data is not linearly separable in two dimensions. Let’s add an additional non-linear feature: (x1-squared + x2-squared), or (x1sq + x2sq). Each data point or vector is now given by (x1, x2, x1sq+x2sq)
Now, the dataset will be linearly separable in 3 dimensions (the algorithm finds a plane), and projected back into 2 dimensions, the boundary is non-linear.
But here’s a big problem: when you project data into higher dimensions, you have to do dot products of vectors in the higher dimensional space. Depending on how high you go, this can become prohibitively expensive, computationally speaking. And especially if you project into infinite dimensional spaces (yes, you can project into infinite dimensions)!
This is where kernel methods enter the picture.
Consider a mapping a -> phi (a), where phi(a) is the higher dimensional vector. Similarly, b -> phi(b).
Where a = [a1 a2 a1sq+a2sq] and b = [b1 b2 b1sq+b2sq]
If you can find a function k (a, b) such that k (a, b) = phi (a) dot phi (b), then you have a function k () that gives you the result of the dot product of two vectors in the higher dimensional (even infinite dimensional) space but carries out the computation in the lower dimensional space of the original a, b vectors.
Now, you can combine the original SVM algorithm with kernel methods, project your data into whatever higher dimensional space you desire, and as long as you can find/design an appropriate kernel method, you can find any non-linear boundary to separate your data clusters! It’s insanely beautiful and powerful.
The story of SVMs, kernel methods and all the relevant math is in WHY MACHINES LEARN:
See:
𝐔𝐒: penguinrandomhouse.com/books/677608/why-machines-learn-by…
𝐔𝐊: penguin.co.uk/books/446849/why-machines-learn-by-ananthas… -
Anil Ananthaswamy
@anilananth
Happy to see a nerdy, math-laden book on machine learning in such august company 🙂
Bookworm Bookstore, Blr
@bookworm_Kris
Sorry for delay, but here are #bookwormbestsellers for Aug '24.
In non-fiction, books by @anilananth, @seth_nitin, @sondhi_aditya, Aruna Roy & @sunita_krishnan.
In fiction, by @radiomukhers, @Elif_Safak, Tom Spencer, Sheela Tomy & @MeetiShroff. -
Anil Ananthaswamy reposted
Steven Strogatz
@stevenstrogatz
The Beautiful Cover of the Japanese edition of "The Calculus of Friendship"
- with mesmerizing illustrations by Megumi Goto
abakcus.com/directory/the-beautiful-cover-of-the-japanese… -
Anil Ananthaswamy
@anilananth
"No street in the Cholera area was without death.” Physician John Snow's epidemiological analysis of London's cholera epidemic in the 1850s--to find the source of the outbreak--was masterful and it presaged Voronoi diagrams, measurement of Euclidean and Manhattan distances, and eventually machine learning's seminal k-Nearest Neighbor algorithm!
WHY MACHINES LEARN's chapter Birds of a Feather starts with Snow's story, and moves on to Muslim Arab polymath Alhazen's insightful take on human vision and object recognition, before tackling Peter Hart's k-NN algorithm.
See:
𝐔𝐒: penguinrandomhouse.com/books/677608/why-machines-learn-by…
𝐔𝐊: penguin.co.uk/books/446849/why-machines-learn-by-ananthas… -
Anil Ananthaswamy reposted
François Chollet
@fchollet
I'm finishing up the last outstanding chapter of the 3rd edition of my book Deep Learning with Python.
The book has now 18 chapters, up from 14 for the second edition and 9 for the first edition. -
Anil Ananthaswamy
@anilananth
A few thoughts about the writing of WHY MACHINES LEARN, for @NextBigIdeaClub :-)
nextbigideaclub.com/magazine/get-elegant-math-unlocked-ai… -
Anil Ananthaswamy
@anilananth
-
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
Marvin Minsky and Seymour Papert helped kill research on neural networks in the late 1960s using a time-worn tactic: guilt by association.
Frank Rosenblatt's single-layer neural networks, or perceptrons, had made a splash in the late 1950s. His algorithm finds a linear separation (a line, a plane, or a hyperplane) between two clusters of data. The top left panel shows one such example.
In their tour de force of a book, Perceptrons (1969), Minksy and Papert elegantly proved that single layer perceptrons could not solve the XOR problem. The top right panel shows a dataset that typifies the XOR problem: no single line can separate the triangles from the circles.
Minsky and Papert insinuated without proof that multi-layer perceptrons also could not solve the XOR problem. People like John Hopfield and Geoff Hinton, didn't buy this argument. Hopfield, of course, came up with Hopfield networks for an entirely different problem (early 1980s), and Hinton, along with Rumelhart and Williams, wrote the key paper popularizing the back-propagation algorithm to train multi-layer networks.
Bottom left panel shows the non-linear boundary that can be found by a multi-layer neural network to solve the XOR problem.
Minsky and Papert are a constant presence in WHY MACHINES LEARN: their superb mathematical analysis of perceptrons, as also their bad-faith knock against multi-layer perceptrons (sans proof), played a seminal role in how neural network research played out.
More at:
𝐔𝐒: penguinrandomhouse.com/books/677608/why-machines-learn-by…
𝐔𝐊: penguin.co.uk/books/446849/why-machines-learn-by-ananthas…
#artificialintelligence #machinelearning -
Anil Ananthaswamy
@anilananth
Marvin Minsky and Seymour Papert helped kill research on neural networks in the late 1960s using a time-worn tactic: guilt by association.
Frank Rosenblatt's single-layer neural networks, or perceptrons, had made a splash in the late 1950s. His algorithm finds a linear separation (a line, a plane, or a hyperplane) between two clusters of data. The top left panel shows one such example.
In their tour de force of a book, Perceptrons (1969), Minksy and Papert elegantly proved that single layer perceptrons could not solve the XOR problem. The top right panel shows a dataset that typifies the XOR problem: no single line can separate the triangles from the circles.
Minsky and Papert insinuated without proof that multi-layer perceptrons also could not solve the XOR problem. People like John Hopfield and Geoff Hinton, didn't buy this argument. Hopfield, of course, came up with Hopfield networks for an entirely different problem (early 1980s), and Hinton, along with Rumelhart and Williams, wrote the key paper popularizing the back-propagation algorithm to train multi-layer networks.
Bottom left panel shows the non-linear boundary that can be found by a multi-layer neural network to solve the XOR problem.
Minsky and Papert are a constant presence in WHY MACHINES LEARN: their superb mathematical analysis of perceptrons, as also their bad-faith knock against multi-layer perceptrons (sans proof), played a seminal role in how neural network research played out.
More at:
𝐔𝐒: penguinrandomhouse.com/books/677608/why-machines-learn-by…
𝐔𝐊: penguin.co.uk/books/446849/why-machines-learn-by-ananthas…
#artificialintelligence #machinelearning -
Anil Ananthaswamy reposted
Anil Ananthaswamy
@anilananth
The awesome 𝐬𝐮𝐩𝐩𝐨𝐫𝐭 𝐯𝐞𝐜𝐭𝐨𝐫 𝐦𝐚𝐜𝐡𝐢𝐧𝐞𝐬 that ruled the roost in 2000s, before 𝐧𝐞𝐮𝐫𝐚𝐥 𝐧𝐞𝐭𝐰𝐨𝐫𝐤𝐬 became the darlings of 𝐦𝐚𝐜𝐡𝐢𝐧𝐞 𝐥𝐞𝐚𝐫𝐧𝐢𝐧𝐠, can be traced back to 𝘝𝘭𝘢𝘥𝘪𝘮𝘪𝘳 𝘝𝘢𝘱𝘯𝘪𝘬'𝘴 algorithm, which was an appendix in his 1964 book!
But the real reason why 𝐒𝐕𝐌𝐬 became so hot was because they were combined with 𝐤𝐞𝐫𝐧𝐞𝐥 𝐦𝐞𝐭𝐡𝐨𝐝𝐬: and the credit for this goes to 𝐈𝐬𝐚𝐛𝐞𝐥𝐥𝐞 𝐆𝐮𝐲𝐨𝐧, who was not acknowledged until much later as an equal contributor to the amazing algorithm. The shadow cast by Vapnik's legendary stature was too big, and people assumed it was all due to him (I tell the story of Guyon and Bernard Boser's work in 𝐖𝐇𝐘 𝐌𝐀𝐂𝐇𝐈𝐍𝐄𝐒 𝐋𝐄𝐀𝐑𝐍, from their perspective.)
Vapnik's 1964 algorithm was designed to find an 𝐨𝐩𝐭𝐢𝐦𝐚𝐥 𝐦𝐚𝐫𝐠𝐢𝐧 for a linear classifier. The top left panel shows such a margin separating two classes of data. The data points that lie on the margins are supporting the margins, and hence the name support vector machines.
The algorithm necessitates the calculation of 𝐝𝐨𝐭 𝐩𝐫𝐨𝐝𝐮𝐜𝐭𝐬 of the data points (or vectors) to both find the optimal margin and to classify a new data point as lying to one side or the other.
But what if the data are not linearly separable? As shown in top right panel? Well, you could create new non-linear features to project the data into higher dimensions and find the optimal linear boundary there, but that would mean doing dot products in higher dimensions: that can get computationally very, very expensive.
That's where 𝐈𝐬𝐚𝐛𝐞𝐥𝐥𝐞 𝐆𝐮𝐲𝐨𝐧's expertise (from her PhD days) came in. She recognized instantly that one can use kernel methods to project the data into higher dimensions, but do the dot products in the initial lower dimensional space. So, it remains computationally easy.
A kernel is a function that takes two low dimensional vectors as input and spits out the dot product of those two vectors in higher dimensions, but the computation never happens in the higher dimensional space! It's probably one of the most awesome mathematical results I got to describe in WHY MACHINES LEARN.
Then, using kernel methods, the optimal linear boundary you find in the higher dimensions, when projected back to the lower dimensions, looks like the panel on bottom left.
More at: 𝐔𝐒: penguinrandomhouse.com/books/677608/why-machines-learn-by…
𝐔𝐊: penguin.co.uk/books/446849/why-machines-learn-by-ananthas…
#artificialintelligence #machinelearning #deeplearning
-
Anil Ananthaswamy
@anilananth
The awesome 𝐬𝐮𝐩𝐩𝐨𝐫𝐭 𝐯𝐞𝐜𝐭𝐨𝐫 𝐦𝐚𝐜𝐡𝐢𝐧𝐞𝐬 that ruled the roost in 2000s, before 𝐧𝐞𝐮𝐫𝐚𝐥 𝐧𝐞𝐭𝐰𝐨𝐫𝐤𝐬 became the darlings of 𝐦𝐚𝐜𝐡𝐢𝐧𝐞 𝐥𝐞𝐚𝐫𝐧𝐢𝐧𝐠, can be traced back to 𝘝𝘭𝘢𝘥𝘪𝘮𝘪𝘳 𝘝𝘢𝘱𝘯𝘪𝘬'𝘴 algorithm, which was an appendix in his 1964 book!
But the real reason why 𝐒𝐕𝐌𝐬 became so hot was because they were combined with 𝐤𝐞𝐫𝐧𝐞𝐥 𝐦𝐞𝐭𝐡𝐨𝐝𝐬: and the credit for this goes to 𝐈𝐬𝐚𝐛𝐞𝐥𝐥𝐞 𝐆𝐮𝐲𝐨𝐧, who was not acknowledged until much later as an equal contributor to the amazing algorithm. The shadow cast by Vapnik's legendary stature was too big, and people assumed it was all due to him (I tell the story of Guyon and Bernard Boser's work in 𝐖𝐇𝐘 𝐌𝐀𝐂𝐇𝐈𝐍𝐄𝐒 𝐋𝐄𝐀𝐑𝐍, from their perspective.)
Vapnik's 1964 algorithm was designed to find an 𝐨𝐩𝐭𝐢𝐦𝐚𝐥 𝐦𝐚𝐫𝐠𝐢𝐧 for a linear classifier. The top left panel shows such a margin separating two classes of data. The data points that lie on the margins are supporting the margins, and hence the name support vector machines.
The algorithm necessitates the calculation of 𝐝𝐨𝐭 𝐩𝐫𝐨𝐝𝐮𝐜𝐭𝐬 of the data points (or vectors) to both find the optimal margin and to classify a new data point as lying to one side or the other.
But what if the data are not linearly separable? As shown in top right panel? Well, you could create new non-linear features to project the data into higher dimensions and find the optimal linear boundary there, but that would mean doing dot products in higher dimensions: that can get computationally very, very expensive.
That's where 𝐈𝐬𝐚𝐛𝐞𝐥𝐥𝐞 𝐆𝐮𝐲𝐨𝐧's expertise (from her PhD days) came in. She recognized instantly that one can use kernel methods to project the data into higher dimensions, but do the dot products in the initial lower dimensional space. So, it remains computationally easy.
A kernel is a function that takes two low dimensional vectors as input and spits out the dot product of those two vectors in higher dimensions, but the computation never happens in the higher dimensional space! It's probably one of the most awesome mathematical results I got to describe in WHY MACHINES LEARN.
Then, using kernel methods, the optimal linear boundary you find in the higher dimensions, when projected back to the lower dimensions, looks like the panel on bottom left.
More at: 𝐔𝐒: penguinrandomhouse.com/books/677608/why-machines-learn-by…
𝐔𝐊: penguin.co.uk/books/446849/why-machines-learn-by-ananthas…
#artificialintelligence #machinelearning #deeplearning
-
Anil Ananthaswamy
@anilananth
Goodreads has a giveaway: 5 copies of the hardcopy of WHY MACHINES LEARN. Ends in 6 days ...Link on this webipage: www.goodreads.com/book/show/195888801-why-machines-learn
-
Anil Ananthaswamy
@anilananth
Sometimes, when writing a book, you encounter an idea so mind-bending, it doesn't leave you. For me, it was the answer this question: Why are deep neural networks universal function approximators? Turns out, understanding why functions are vectors is key; it's a truly eye-opening exercise.
First: let's make sense of functions as vectors (are they really?)
Let’s take a function 𝘺 = 𝘤𝘰𝘴(2𝘹)
The top left panel shows this function for values of x between 0 and 10. Let's only evaluate the function for values of x = 0, 1, 2, …., 10. So, 11 points on the x-axis.
The function evaluates to these values: [𝟏 -𝟎.𝟒𝟐 -𝟎.𝟔𝟓 𝟎.𝟗𝟔 -𝟎.𝟏𝟓 -𝟎.𝟖𝟒 𝟎.𝟖𝟒 𝟎.𝟏𝟒 -𝟎.𝟗𝟔 𝟎.𝟔𝟔 𝟎.𝟒𝟏]
Those 11 values for the function, evaluated at specific values of x, constitutes a vector in 11D space. You can take any other function of x, and similarly evaluate it for the exact same 11 values of x, and you get another vector in the same 11D space. You can see how, in this limited sense, any function of x can be turned into a unique vector in 11D space.
Now, we have to stretch our imagination. Imagine evaluating the function at arbitrarily small intervals, but still between x = 0 and x=10. Eventually, you’ll get an infinity of values. Now, the 𝐟𝐮𝐧𝐜𝐭𝐢𝐨𝐧 𝐢𝐬 𝐚 𝐯𝐞𝐜𝐭𝐨𝐫 𝐢𝐧 𝐢𝐧𝐟𝐢𝐧𝐢𝐭𝐞 𝐝𝐢𝐦𝐞𝐧𝐬𝐢𝐨𝐧𝐚𝐥 𝐬𝐩𝐚𝐜𝐞.
Extend the x-axis to minus infinity and plus infinity. The function evaluated at an infinity of points along this infinite axis is a vector in another infinite-dimensional space!
The examples have dealt with a function that takes in a scalar input, x, and produces a scalar output, y. You can generalize this to functions that take in a vector x as input and produce a vector y as output.
What does this have to do with 𝐝𝐞𝐞𝐩 𝐧𝐞𝐮𝐫𝐚𝐥 𝐧𝐞𝐭𝐰𝐨𝐫𝐤𝐬 and their ability to approximate any function? Here’s one view of what a deep net does: it takes in a vector 𝐱 as input and produces a vector 𝐲 as output, such that the function f () captures the correlations in the training data. During training, the neural network learns to approximate this function.
𝐲 = 𝘧 (𝐱)
The universal approximation theorem essentially proves that, given certain assumptions, a neural network can convert any vector that lives in a vector space that characterizes the inputs into any vector that lives in some other vector space that characterizes the outputs.
It’s not a trivial theorem (though it might seem so to some, in hindsight; it wasn't so in the late 1980s) …
The story of this amazing advance is narrated in 𝐖𝐇𝐘 𝐌𝐀𝐂𝐇𝐈𝐍𝐄𝐒 𝐋𝐄𝐀𝐑𝐍. More here:
𝐔𝐒: penguinrandomhouse.com/books/677608/why-machines-learn-by…
𝐔𝐊: penguin.co.uk/books/446849/why-machines-learn-by-ananthas… -
Anil Ananthaswamy
@anilananth
Imagine being 𝐚𝐜𝐜𝐮𝐬𝐞𝐝, even in jest, of being the man who 𝘴𝘦𝘵 𝘣𝘢𝘤𝘬 𝘥𝘦𝘦𝘱 𝘭𝘦𝘢𝘳𝘯𝘪𝘯𝘨 𝘣𝘺 𝘵𝘸𝘰 𝘥𝘦𝘤𝘢𝘥𝘦𝘴. 𝐆𝐞𝐨𝐫𝐠𝐞 𝐂𝐲𝐛𝐞𝐧𝐤𝐨, who proved one of the seminal 𝐮𝐧𝐢𝐯𝐞𝐫𝐬𝐚𝐥 𝐚𝐩𝐩𝐫𝐨𝐱𝐢𝐦𝐚𝐭𝐢𝐨𝐧 𝐭𝐡𝐞𝐨𝐫𝐞𝐦𝐬, showed that a 1-hidden layer neural network could approximate any function, given enough neurons. But he 𝘥𝘪𝘥 𝘯𝘰𝘵 say that one should 𝘰𝘯𝘭𝘺 use 1-hidden layer networks--people apparently thought so, and kept trying to do deep learning with 1-layer networks, before realizing that they could have more than one hidden layer (leading to the truly "deep" learning revolution). At least, that's the lore!
Cybenko's story and the logic behind why neural networks are universal function approximators is told in 𝐖𝐇𝐘 𝐌𝐀𝐂𝐇𝐈𝐍𝐄𝐒 𝐋𝐄𝐀𝐑𝐍, in the chapter with one of my favorite titles: 𝐓𝐡𝐞 𝐌𝐚𝐧 𝐖𝐡𝐨 𝐒𝐞𝐭 𝐁𝐚𝐜𝐤 𝐃𝐞𝐞𝐩 𝐋𝐞𝐚𝐫𝐧𝐢𝐧𝐠 (𝐍𝐨𝐭 𝐑𝐞𝐚𝐥𝐥𝐲)
More here:
𝐔𝐒: penguinrandomhouse.com/books/677608/why-machines-learn-by…
𝐔𝐊: penguin.co.uk/books/446849/why-machines-learn-by-ananthas… -
Anil Ananthaswamy
@anilananth
My dear friend and @KSJatMIT colleague @sonaliprasad07's beautiful looking novel just came! Can't wait to read it. Congratulations Sonali!
-
Anil Ananthaswamy
@anilananth
You'll often see arguments that 𝐌𝐚𝐜𝐡𝐢𝐧𝐞 𝐋𝐞𝐚𝐫𝐧𝐢𝐧𝐠 is "𝘫𝘶𝘴𝘵" 𝐠𝐥𝐨𝐫𝐢𝐟𝐢𝐞𝐝 𝐬𝐭𝐚𝐭𝐢𝐬𝐭𝐢𝐜𝐬. I respectfully beg to differ. To write 𝐖𝐇𝐘 𝐌𝐀𝐂𝐇𝐈𝐍𝐄𝐒 𝐋𝐄𝐀𝐑𝐍, I began coding after a 20-year gap (I was a distributed systems software architect in the 90s, in the pre-ML/AI days). As I learned 𝐌𝐋, I was struck by how differently you have to think about problem-solving using ML than you do when using non-ML algorithms.
And it's not just statistics. Yes, statisticians build sophisticated models of the patterns that exist in data, and use these models to infer/predict. But ML is so much more than that simply writing code to automate what statisticians do.
From a software engineering perspective, you have to flip your instincts about how to solve problems: from thinking algorithmically to learning how to pose questions of the data you have in hand, and use 𝐦𝐚𝐜𝐡𝐢𝐧𝐞 𝐥𝐞𝐚𝐫𝐧𝐢𝐧𝐠 to, well, 𝘭𝘦𝘢𝘳𝘯 the model to represent the data, which can then be used for 𝐢𝐧𝐟𝐞𝐫𝐞𝐧𝐜𝐞/𝐩𝐫𝐞𝐝𝐢𝐜𝐭𝐢𝐨𝐧/𝐠𝐞𝐧𝐞𝐫𝐚𝐭𝐢𝐨𝐧. You have to learn how to 𝘴𝘦𝘦 data differently: 𝘢𝘴 𝘢 𝘳𝘦𝘱𝘰𝘴𝘪𝘵𝘰𝘳𝘺 𝘰𝘧 𝘢𝘯𝘴𝘸𝘦𝘳𝘴 𝘵𝘰 𝘺𝘰𝘶𝘳 𝘲𝘶𝘦𝘴𝘵𝘪𝘰𝘯𝘴.
My hope is that 𝐖𝐇𝐘 𝐌𝐀𝐂𝐇𝐈𝐍𝐄𝐒 𝐋𝐄𝐀𝐑𝐍 will help those of us who know some high-school/first-year undergrad math and maybe even did some old-style software engineering (and, of course, those coming to ML entirely untainted by old ways) to appreciate the technological changes happening underfoot. I took what I felt was a representative sample of ML algorithms to illustrate the math that undergirds this new way of thinking, while also providing a historical account. These include:
Rosenblatt's 𝐏𝐞𝐫𝐜𝐞𝐩𝐭𝐫𝐨𝐧 algorithm (the first artificial neuron the learned) and the 𝐖𝐢𝐝𝐫𝐨𝐰-𝐇𝐨𝐟𝐟 𝐋𝐞𝐚𝐬𝐭 𝐌𝐞𝐚𝐧 𝐒𝐪𝐮𝐚𝐫𝐞𝐬 algorithm (the true precursor to today's 𝐛𝐚𝐜𝐤𝐩𝐫𝐨𝐩𝐚𝐠𝐚𝐭𝐢𝐨𝐧 algorithm for training deep nets).
𝐁𝐚𝐲𝐞𝐬 𝐓𝐡𝐞𝐨𝐫𝐞𝐦, the 𝐎𝐩𝐭𝐢𝐦𝐚𝐥 𝐁𝐚𝐲𝐞𝐬 and 𝐍𝐚𝐢𝐯𝐞 𝐁𝐚𝐲𝐞𝐬 classifiers (you can't do ML without really appreciating the role probability and statistics play in ML).
The 𝘴𝘦𝘮𝘪𝘯𝘢𝘭 𝐤-𝐍𝐞𝐚𝐫𝐞𝐬𝐭 𝐍𝐞𝐢𝐠𝐡𝐛𝐨𝐫 𝐚𝐥𝐠𝐨𝐫𝐢𝐭𝐡𝐦: A wonderful way to develop an intuition for how data is represented in vector spaces, and the notion of similarity in Euclidean space, and how all this falls apart when you move to high-dimensional spaces.
𝐏𝐫𝐢𝐧𝐜𝐢𝐩𝐚𝐥 𝐂𝐨𝐦𝐩𝐨𝐧𝐞𝐧𝐭 𝐀𝐧𝐚𝐥𝐲𝐬𝐢𝐬: to appreciate the power of matrices, and how they can help bring high-dimensional data down to lower dimensions for computational efficiency and easier visualization, among other things.
𝐊𝐞𝐫𝐧𝐞𝐥 𝐌𝐞𝐭𝐡𝐨𝐝𝐬: When you have to move into higher dimensions to find optimal margins for linear classification (𝘢 𝘭𝘢 𝐬𝐮𝐩𝐩𝐨𝐫𝐭 𝐯𝐞𝐜𝐭𝐨𝐫 𝐦𝐚𝐜𝐡𝐢𝐧𝐞𝐬), but need to stay grounded in lower dimensions for computation efficiency.
And then, of course, the latter half of WHY MACHINES LEARN focues on the revolution that's happened in 𝘢𝘳𝘵𝘪𝘧𝘪𝘤𝘪𝘢𝘭 𝘯𝘦𝘶𝘳𝘢𝘭 𝘯𝘦𝘵𝘸𝘰𝘳𝘬𝘴 and 𝘥𝘦𝘦𝘱 𝘭𝘦𝘢𝘳𝘯𝘪𝘯𝘨: the 𝐮𝐧𝐢𝐯𝐞𝐫𝐬𝐚𝐥 𝐚𝐩𝐩𝐫𝐨𝐱𝐢𝐦𝐚𝐭𝐢𝐨𝐧 𝐭𝐡𝐞𝐨𝐫𝐞𝐦, the 𝐛𝐚𝐜𝐤𝐩𝐫𝐨𝐩𝐚𝐠𝐚𝐭𝐢𝐨𝐧 algorithm, and 𝐜𝐨𝐧𝐯𝐨𝐥𝐮𝐭𝐢𝐨𝐧𝐚𝐥 𝐧𝐞𝐮𝐫𝐚𝐥 𝐧𝐞𝐭𝐰𝐨𝐫𝐤𝐬. And a whole chapter dedicated to 𝐇𝐨𝐩𝐟𝐢𝐞𝐥𝐝 𝐍𝐞𝐭𝐰𝐨𝐫𝐤𝐬: I think they are just the coolest, and give us insights into dynamical networks that may one day rule the roost.
Here's more info about 𝐖𝐇𝐘 𝐌𝐀𝐂𝐇𝐈𝐍𝐄𝐒 𝐋𝐄𝐀𝐑𝐍:
𝐔𝐒: penguinrandomhouse.com/books/677608/why-machines-learn-by…
𝐔𝐊: penguin.co.uk/books/446849/why-machines-learn-by-ananthas… -
Anil Ananthaswamy
@anilananth
There must be a 𝐜𝐨𝐦𝐩𝐨𝐮𝐧𝐝 𝐆𝐞𝐫𝐦𝐚𝐧 𝐰𝐨𝐫𝐝 for how authors feel the day their book is released :-) Today is pub date for 𝐖𝐇𝐘 𝐌𝐀𝐂𝐇𝐈𝐍𝐄𝐒 𝐋𝐄𝐀𝐑𝐍: 𝘛𝘩𝘦 𝘌𝘭𝘦𝘨𝘢𝘯𝘵 𝘔𝘢𝘵𝘩 𝘉𝘦𝘩𝘪𝘯𝘥 𝘔𝘰𝘥𝘦𝘳n 𝘈𝘐. Nearly 4-5 yrs of work in relative obscurity sees the light of day, and the feelings are mixed: 𝐞𝐱𝐜𝐢𝐭𝐞𝐦𝐞𝐧𝐭, 𝐧𝐞𝐫𝐯𝐨𝐮𝐬𝐧𝐞𝐬𝐬, 𝐚𝐧𝐭𝐢𝐜𝐢𝐩𝐚𝐭𝐢𝐨𝐧, 𝐫𝐞𝐥𝐢𝐞𝐟, 𝐰𝐨𝐫𝐫𝐲...
𝐏𝐥𝐞𝐚𝐬𝐞 𝐬𝐩𝐫𝐞𝐚𝐝 𝐭𝐡𝐞 𝐰𝐨𝐫𝐝!
When I became a 𝐰𝐫𝐢𝐭𝐞𝐫, I never thought that I'd write 𝐟𝐨𝐮𝐫 𝐛𝐨𝐨𝐤𝐬. Deep gratitude for the opportunity to dig deep into different subjects.
But I did think it'd feel less 𝐧𝐞𝐫𝐯𝐞-𝐰𝐫𝐚𝐜𝐤𝐢𝐧𝐠 the 4th time around. Maybe I'm a little bit less nervy, but not entirely so! Whatever made me write a 𝐛𝐨𝐨𝐤 𝐰𝐢𝐭𝐡 𝐞𝐪𝐮𝐚𝐭𝐢𝐨𝐧𝐬 in it? Well, it's done now, sink or swim :-)
There are numerous people to thank; I have done so in the acknowledgments.
You can find the book here:
𝐔𝐒 𝐄𝐝𝐢𝐭𝐢𝐨𝐧: penguinrandomhouse.com/books/677608/why-machines-learn-by…
𝐔𝐊 𝐄𝐝𝐢𝐭𝐢𝐨𝐧: www.penguin.co.uk/books/446849/why-machines-learn-by-anan…
My sincere thanks to: @MelMitchell1 @stevenstrogatz @skdh Geoff Hinton Peter Hart and Björn Ommer for their generous and kind endorsements.
#artificialintelligence #machinelearning #deeplearning @DuttonBooks @penguinrandom -
Anil Ananthaswamy
@anilananth
Adam Becker (@FreelanceAstro) and I, almost at the end of our stay @sfiscience, as journalism fellows attending the complex systems summer school in Santa Fe, NM...
-
Anil Ananthaswamy
@anilananth
Don't know how other authors feel, but seeing the first physical copy of your book, after years of looking at Word files and PDFs... Seems surreal. It's as if someone else wrote it... Anyway, this advance copy arrived in the mail. One more month to pub date ...www.penguinrandomhouse.com/books/677608/why-machines-lear…
#artificalintelligence @DuttonBooks -
Anil Ananthaswamy
@anilananth
𝐌𝐲 𝐥𝐢𝐬𝐭 𝐨𝐟 𝐬𝐨𝐦𝐞 𝐞𝐧𝐠𝐚𝐠𝐢𝐧𝐠, 𝐢𝐥𝐥𝐮𝐦𝐢𝐧𝐚𝐭𝐢𝐧𝐠 𝐚𝐧𝐝 𝐢𝐧𝐜𝐫𝐞𝐝𝐢𝐛𝐥𝐲 𝐮𝐬𝐞𝐟𝐮𝐥 𝐯𝐢𝐝𝐞𝐨 𝐥𝐞𝐜𝐭𝐮𝐫𝐞𝐬 𝐨𝐧 𝐦𝐚𝐜𝐡𝐢𝐧𝐞 𝐥𝐞𝐚𝐫𝐧𝐢𝐧𝐠
I could not have written 𝐖𝐡𝐲 𝐌𝐚𝐜𝐡𝐢𝐧𝐞𝐬 𝐋𝐞𝐚𝐫𝐧 without having had the luxury of listening to online lectures over and over--until the rust of two decades of having stayed away from coding began to wear off, and nitty-gritty of the math of machine learning began to make sense, and the joy of being able to code returned.
Here are some of those lectures that were absolutely key to my learning:
𝐌𝐚𝐜𝐡𝐢𝐧𝐞 𝐋𝐞𝐚𝐫𝐧𝐢𝐧𝐠 𝐨𝐯𝐞𝐫𝐚𝐥𝐥
My absolute favorite: 𝐊𝐢𝐥𝐢𝐚𝐧 𝐖𝐞𝐢𝐧𝐛𝐞𝐫𝐠𝐞𝐫'𝐬 𝟐𝟎𝟏𝟖 𝐜𝐥𝐚𝐬𝐬 𝐨𝐧 𝐦𝐚𝐜𝐡𝐢𝐧𝐞 𝐥𝐞𝐚𝐫𝐧𝐢𝐧𝐠 𝐚𝐭 𝐂𝐨𝐫𝐧𝐞𝐥𝐥: www.youtube.com/playlist?list=PLl8OlHZGYOQ7bkVbuRthEsaLr7…
Weinberger is a masterful teacher!
Not too far behind: 𝐀𝐧𝐝𝐫𝐞𝐰 𝐍𝐠'𝐬 𝐬𝐭𝐮𝐝𝐞𝐧𝐭 𝐀𝐧𝐚𝐧𝐝 𝐀𝐯𝐚𝐭𝐢 𝐭𝐞𝐚𝐜𝐡𝐢𝐧𝐠 𝐚𝐧 𝐌𝐋 𝐜𝐨𝐮𝐫𝐬𝐞 𝐢𝐧 𝐭𝐡𝐞 𝐬𝐮𝐦𝐦𝐞𝐫 𝐨𝐟 𝟐𝟎𝟏𝟗: www.youtube.com/playlist?list=PLoROMvodv4rNH7qL6-efu_q2_b…
His first few lectures setting up the math of ML are gems.
And, of course, lectures by 𝐀𝐧𝐝𝐫𝐞𝐰 𝐍𝐠 himself. There are so many... Just Google.
For 𝐋𝐢𝐧𝐞𝐚𝐫 𝐀𝐥𝐠𝐞𝐛𝐫𝐚:
The wonderful 𝐆𝐢𝐥𝐛𝐞𝐫𝐭 𝐒𝐭𝐫𝐚𝐧𝐠 (who was teaching this well before linear algebra became cool): www.youtube.com/playlist?list=PLPQAixVXrNTUw464KbIvuvH-86…
𝐅𝐨𝐫 𝐃𝐚𝐭𝐚 𝐒𝐜𝐢𝐞𝐧𝐜𝐞 𝐦𝐞𝐭𝐡𝐨𝐝𝐬 𝐚𝐧𝐝 𝐜𝐨𝐧𝐜𝐞𝐩𝐭𝐬
𝐁𝐢𝐧𝐠 𝐁𝐫𝐮𝐧𝐭𝐨𝐧, 𝐒𝐭𝐞𝐯𝐞 𝐁𝐫𝐮𝐧𝐭𝐨𝐧 𝐚𝐧𝐝 𝐍𝐚𝐭𝐡𝐚𝐧 𝐊𝐮𝐭𝐳, 𝐨𝐟 𝐭𝐡𝐞 𝐔𝐧𝐢𝐯𝐞𝐫𝐬𝐢𝐭𝐲 𝐨𝐟 𝐖𝐚𝐬𝐡𝐢𝐧𝐠𝐭𝐨𝐧, 𝐒𝐞𝐚𝐭𝐭𝐥𝐞:
www.youtube.com/playlist?list=PLMrJAkhIeNNQz4BMoGSsN8cbt8…
Search for other videos by these three. Many gems to be found.
𝐏𝐫𝐨𝐛𝐚𝐛𝐢𝐥𝐢𝐭𝐲
A quirky set of bite-sized videos on key concepts in probability theory by 𝐑𝐢𝐜𝐡 𝐑𝐚𝐝𝐤𝐞, 𝐩𝐫𝐨𝐟𝐞𝐬𝐬𝐨𝐫 𝐚𝐭 𝐑𝐏𝐈, 𝐓𝐫𝐨𝐲, 𝐍𝐘. www.youtube.com/playlist?list=PLuh62Q4Sv7BXkeKW4J_2WQBlYh…
If you want a masterclass on 𝐒𝐮𝐩𝐩𝐨𝐫𝐭 𝐕𝐞𝐜𝐭𝐨𝐫 𝐌𝐚𝐜𝐡𝐢𝐧𝐞𝐬, there's none better than 𝐏𝐚𝐭𝐫𝐢𝐜𝐤 𝐖𝐢𝐧𝐬𝐭𝐨𝐧'𝐬 𝐌𝐈𝐓 lecture:
www.youtube.com/watch?v=_PwhiWxHK8o
If you can persist with these lectures--and then keep dipping into the never-ending resource that is YouTube --you'll be well set to get under the mathematical skin of ML.
www.penguinrandomhouse.com/books/677608/why-machines-lear… -
Anil Ananthaswamy
@anilananth
Sometimes, as an author, you get an endorsement that delights because of the endorser's way with words. My sincere thanks to @stevenstrogatz for this lovely blurb; it gets to the heart of why I wrote Why Machines Learn; @DuttonBooks @penguinrandom www.penguinrandomhouse.com/books/677608/why-machines-lear…
-
Anil Ananthaswamy
@anilananth
Dan Dennett is no more.🙁 This is really sad. He touched so many lives, with his deep and uncompromising philosophy, his humor and a gentle kindness that lay beneath it all. I'd say RIP Dan, but he'd have admonished anyone for saying that! From Bacteria to Bach and Back, Dan.
-
Anil Ananthaswamy
@anilananth
A generous endorsement from @skdh, gratefully received, for Why Machines Learn. “If you were looking for a way to make sense of the AI revolution that is well underway, look no further." Preorder from your favorite indie bookstore or from, where else, www.amazon.com/Why-Machines-Learn-Elegant-Behind/dp/05931…
-
Anil Ananthaswamy
@anilananth
We begin the 2nd decade of our science journalism workshop @NCBS_Bangalore. Application deadline: Feb 19, 2024. Please spread the word! www.ncbs.res.in/events/journalism
-
Anil Ananthaswamy
@anilananth
What does physics have to do with #dall·E 2? Diffusion models are all the rage; their origins lie in some cool principles from non-equilibrium thermodynamics. Work by @jaschasd @hojonathanho @YSongStanford @AnimaAnandkumar www.quantamagazine.org/the-physics-principle-that-inspire…
-
Anil Ananthaswamy
@anilananth
Twin tensions in cosmological data are pointing to possible new physics. The 3rd Gen of telescopes, including #jwst, @VRubinObs and @SimonsObs, will show the way. An exciting phase of precision cosmology lies ahead. My story for @sciam w/ @LeeBillings
www.scientificamerican.com/article/astronomers-gear-up-to… -
Anil Ananthaswamy
@anilananth
We might soon see neural networks that sip rather than gulp energy. My story on spiking #neuralnetworks for @SimonsInstitute, featuring work by @SanderBohte, @hisspikeness (Friedemann Zenke), @neuralreckoning (Dan Goodman) and @ChristianPehle : simons.berkeley.edu/news/spiking-neural-networks
-
Anil Ananthaswamy
@anilananth
Anyone who has attended a #rogerpenrose talk will recognize his insanely complicated presentations using multiple overhead projector slides stacked atop each other. He would even attempt animations by jiggling the slides 🙂 #nobelprize2020
-
Anil Ananthaswamy
@anilananth
There is a book that tells the story of this book... @imanjitkumar wrote Quantum.
-
Anil Ananthaswamy
@anilananth
Through Two Doors at Once release on August 7th. Thank you @imanjitkumar for your kind words! @DuttonBooks @EmilyCanders
-
Anil Ananthaswamy
@anilananth
What's maths got to do with our brains? www.newscientist.com/article/mg23531410-300-our-greatest-…
-
Sachin Kalbag
@SachinKalbag
Can't recommend these books enough: 'The Man Who Wasn't There' by @anilananth, and 'Einstein's Greatest Mistake' by @DavidBodanis
Post not available here, view on X
Anil Ananthaswamy
@anilananth
ditto... Thanks!
-
Anil Ananthaswamy
@anilananth
The Man Who Wasn't There makes @nbcnews' 12 Notable Tech and Science Books of 2015 nbcnews.to/1P6NRlD @DuttonBooks @sciencefactory
-
Anil Ananthaswamy
@anilananth
What out-of-body experiences and doppelgangers tell us about our sense of self: @wsj weekend www.wsj.com/articles/the-lessons-of-out-of-body-experienc…


































































































































































































































































































































































































































































































































































































































































