the title of LeCun's slide says it all.
Haim Sompolinsky
Haim Sompolinsky (Hebrew: חיים סומפולינסקי; born 1949) is the William N. Skirball Professor of Neuroscience at the Edmond and Lily Safra Center for Brain Sciences (formerly the Interdisciplinary Center for Neural Computation), and a professor of physics at the Racah Institute of Physics at the Hebrew University of Jerusalem. He is also a visiting professor in the Center of Brain Science at Harvard University and the director of Harvard's Swartz Program in Theoretical Neuroscience. He is widely regarded as one of the leaders of theoretical neuroscience. More on Wikipedia
-
Haim Sompolinsky
@HSompolinsky
-
Haim Sompolinsky
@HSompolinsky
Thanks, Yann for the great inspiring talk at Harvard.
-
Haim Sompolinsky reposted
יצחק הרצוג Isaac Herzog
@Isaac_Herzog
בגאווה ישראלית גדולה בירכתי היום את פרופסור חיים סומפולינסקי מהאוניברסיטה העברית בירושלים על זכייתו בפרס Lundbeck היוקרתי - הפרס החשוב ביותר בעולם בתחום חקר המוח. מדובר בהישג ישראלי פורץ דרך, שמשקף את מצוינות האקדמיה הישראלית במחקר ובכלל.
אני סמוך ובטוח כי בעקבותיו יבואו עוד אותות ופרסים שיוקירו את תרומתם של ישראליות וישראלים לקהילה המדעית העולמית, ומקווה ופועל לכך שנדע להכיר תודה - כחברה וכמדינה - לענקי האקדמיה והמחקר שלנו. -
Haim Sompolinsky
@HSompolinsky
My Harvard/Neuro 231, 2024 Edition begins soon. It explores contemporary brain theory spanning local neuronal circuits as well as deep neural networks. It examines the relationship between network structure, dynamics, and computation. 1/3
Haim Sompolinsky
@HSompolinsky
The course introduces analytical and numerical tools from information theory, dynamical systems, statistics, statistical physics, AI, and machine learning for the study of neural computation. 2/3
Haim Sompolinsky
@HSompolinsky
Topics include principles of early sensory processing; unsupervised and supervised learning; attractors, memory, and spatial functions in cortical circuits; noise, chaos, and neural coding; learning, representations, and cognition in deep neural networks in brains and AI. 3/3
-
Haim Sompolinsky
@HSompolinsky
I am excited to announce a recent work by Yehonatan Avidan and Qianyi Li arxiv.org/abs/2309.04522
presenting an analytical theory for learning dynamics in infinitely wide neural network.
Haim Sompolinsky
@HSompolinsky
The theory unites NTK and NNGP as two limits of the same underlying process. We introduce the Neural Dynamical Kernel (NDK), derive equations for the dynamics of the mean predictor of the network, and discuss implications for the problem of representational drift in neuroscience.
-
Haim Sompolinsky
@HSompolinsky
Jet Blue just entered the Book of Guinness for World Record on Greediness. It cancelled my flight due to weather conditions but refused to fully refund me. They charged me cancellation fee! Never underestimate the creative way companies such as Jet Blue chase your money! advice?
-
Haim Sompolinsky
@HSompolinsky
I am excited to announce the publication of the wonderful paper of Julia Steinberg on Associative memory of structured knowledge in scientific reports
www.nature.com/articles/s41598-022-25708-y.pdf. Most neural models of associative memory store structureless knowledge as simple random patterns in RNNsPost not available here, view on X
Haim Sompolinsky
@HSompolinsky
and then store multiple compressed vectors as fixed points in RNN. Retrieved fixed points is followed by decoding individual linked components. Thus, model consists of two memory systems: 'dictionary' for items and 'episodic' for their linked structures.
Haim Sompolinsky
@HSompolinsky
Our model proposes a novel scheme for associative memory of temporal sequence. In contrast to using sequence attractors (Sompolinsky and Kanter, 1988),
here entire sequences are stored holistically as fixed points-a scheme that is robust to overlap between sequences. -
Haim Sompolinsky reposted
Nature Computational Science
@NatComputSci
We highlight a study by Ben Sorscher, @SuryaGanguli and @HSompolinsky in which they explore a quantitative theory of neural geometry and few shot learning in deep neural networks and in macaque inferotemporal cortex (rdcu.be/cZCJ3). www.nature.com/articles/s43588-022-00367-1
-
Haim Sompolinsky reposted
Surya Ganguli
@SuryaGanguli
Our new paper @NeuroCellPress "A unified theory for the computational and mechanistic origins of grid cells" lead by Ben Sorscher & @meldefon w/@SamOcko & @lisa_giocomo explains when & why grid cells appear (or don't) in trained neural path-integrators and
authors.elsevier.com/c/1f~Ze3BtfH1ZNm -
Haim Sompolinsky reposted
Surya Ganguli
@SuryaGanguli
Our new paper in @PNASNews: "Neural representation representation geometry underlies few shot concept learning'' lead by Ben Sorscher and @HSompolinsky: a quantitative theory of neural geometry & few shot learning, tested in both deep networks & monkey IT www.pnas.org/doi/abs/10.1073/pnas.2200800119
-
Haim Sompolinsky
@HSompolinsky
Our work on manifold geometric theory underlying fast learning of novel concepts led by Ben Sorcher of the Ganguli Lab is out. www.pnas.org/doi/pdf/10.1073/pnas.2200800119 We apply our theory to object manifold representations in deep neural network (DNN) and in macaque IT cortex.
Haim Sompolinsky
@HSompolinsky
And show that that they support highly accurate few-shot learning of novel visual concepts
and that and that variability in performance across concepts follow closely the predictions of our manifold geometric theory.
Haim Sompolinsky
@HSompolinsky
Remarkably, we find that object manifolds in DNN visual feature layers also support zero shot learning of concepts from linguistic descriptors revealing geometric alignment of semantic features in Word2Vec and high level visual features of the same concepts.
-
Haim Sompolinsky
@HSompolinsky
Postdoc position in Sompolinsky Group: If you are interested in doing exciting postdoc research at Harvard at the forefront of computational neuroscience and the interface between natural and artificial intelligence, send application and 3 letters to hsompolinsky@mcb.harvard.edu
-
Haim Sompolinsky
@HSompolinsky
Harvard Center for Brain Science has an opening for Swartz postdoctoral fellowship in theoretical neuroscience, starting fall 2023.
cbs.fas.harvard.edu/2022/09/28/the-swartz-program-seeks-a… -
Haim Sompolinsky
@HSompolinsky
#cosyne22
Open position for a phd or postdoc in theory at the interface of deep learning and neuroscience. If you have suitable background please see me at the meeting. -
Haim Sompolinsky reposted
Surya Ganguli
@SuryaGanguli
The importance of studying naturalistic behaviors. Scroll to the end of the video for extremely impressive Puffer Fish art!
Erik Solheim
@ErikSolheim
The Japanese 🇯🇵 Puffer Fish is probably nature's greatest artist 🐟
To grab a female’s attention he creates something that defies belief 😲 -
Haim Sompolinsky
@HSompolinsky
Qianyi Li and I are excited to announce the publication of our paper on the Kernel Renormalization Theory of Deep Learning in Phys Rev X link.aps.org/doi/10.1103/PhysRevX.11.031059 . We derive an exact stat mech solution of deep learning by successive integration of weights
Haim Sompolinsky
@HSompolinsky
2) Theory shows that in the finite width limit the GP kernels are renormalized by data and size dependent order parameter, and predicts the conditions in which generalization performance improves upon increasing the network width and depth.
Haim Sompolinsky
@HSompolinsky
3) Theory is exact for Deep Linear Networks and is a surprisingly good approximation for moderately deep ReLU networks. Stay tuned for extensions of the theory for e.g., convolutional and nonlinear networks.
-
Haim Sompolinsky
@HSompolinsky
I am excited to announce a new work by an outstanding PhD student Ben Sorscher from Ganguli Lab that links the geometry of neural representations to one of most remarkable cognitive human abilities, learning new concepts from only few examples. www.biorxiv.org/content/10.1101/2021.03.21.436284v1?rss=1
Haim Sompolinsky
@HSompolinsky
Surprisingly, an increase in dimension of concept neural manifolds is predicted to enhance few shot learning. Our theory is supported by observed layer-wise changes in manifold geometry of ImageNet- trained DCNNs and neural data from V4 and IT cortex. www.biorxiv.org/content/10.1101/2021.03.21.436284v1?rss=1
-
Haim Sompolinsky reposted
Haim Sompolinsky
@HSompolinsky
I am excited to announce a new work by an outstanding PhD student Ben Sorscher from Ganguli Lab that links the geometry of neural representations to one of most remarkable cognitive human abilities, learning new concepts from only few examples. www.biorxiv.org/content/10.1101/2021.03.21.436284v1?rss=1
-
Haim Sompolinsky
@HSompolinsky
I am excited to announce a recent work with Harvard student Qianyi Li deriving exact Stat Mech theory of Deep Linear Neural Networks following supervised learning, revealing exquisite richness of generalization properties and layerwise representations
arxiv.org/abs/2012.04030
Haim Sompolinsky
@HSompolinsky
We have developed the Back Propagating Renormalization Group which integrates network weights successively backward layer by layer, at each stage generating an effective Hamiltonian for the remainder weights with renormalized Kernel matrix.
Haim Sompolinsky
@HSompolinsky
Our results hold for the entire range of number of examples relative to width. Although developed rigorously for DNNs with linear neurons, extension to ReLU neurons agrees surprisingly well with simulations of ReLU DNNS with modest depth over a wide range of parameters.
-
Haim Sompolinsky reposted
eman
@froudara
Ιn our recent biorxiv preprint we study object representations across the mouse visual hierarchy. biorxiv.org/cgi/content/short/2020.08.20.258798v1
-
Haim Sompolinsky
@HSompolinsky
Does anybody know if coronavirus scare affects or should affect COSYNE 2020?
-
Haim Sompolinsky
@HSompolinsky
The Swartz Program at Harvard University seeks applications for postdoctoral fellows in theoretical and computational neuroscience. Please send application material to haim@fiz.huji.ac.il and come to see me at Cosyne 2020.
-
Haim Sompolinsky reposted
Uri Cohen, PhD
@UriCohen42
When neurons collectively represent objects, what would make a good representation? Is object representation improved along the layers of deep networks? Our Nature Communications paper got some hints! www.nature.com/articles/s41467-020-14578-5 With @s_y_chung, Dan Lee & @hsompolinsky. 1/5
-
Haim Sompolinsky
@HSompolinsky
1/ A new paper provides the first, theory-based measurement of the untangling of object manifolds by several ImageNet trained DCNNs, using ImageNet point-cloud manifolds and smoothly warped images.
www.biorxiv.org/content/10.1101/644658v1
Haim Sompolinsky
@HSompolinsky
2/ Based on recent theory (link.aps.org/doi/10.1103/PhysRevX.8.031003), we measure the capacity for linear separability of manifolds and show that it increases monotonically along the layers, accompanied by a decrease in manifold radii and dimensions, and in their correlations.
Haim Sompolinsky
@HSompolinsky
3/We identify functional network motifs (eg conv+relu+pool) that incrementally reduce manifold radius, dimension and correlations. Remarkably, the behavior of different DCNNs are very similar, suggesting that these results may generalize to neural networks in the mamallian brain.
-
Haim Sompolinsky
@HSompolinsky
Our recent journal club discussed an interesting paper from Olshausen Lab arxiv.org/abs/1806.08887 suggesting how to stack a hierarchy of expansive sparse coding stages staggered with compressive manifold embedding stages. Look forward to seeing more details about applications.
-
Haim Sompolinsky
@HSompolinsky
Most significant advance in stat mech of learning since Gardner. For the first time theory incorporates the rich structure of realistic data.
SueYeon Chung
@s_y_chung
A new theory to analyze neural manifolds in high-dimensional data. We link the geometry of neural manifolds to their classification capacity. Exciting implications on understanding deep networks and sensory systems! With Haim Sompolinsky and Dan Lee.
go.aps.org/2tVHcYm
















