Verifying ML in Lean and the future :)
Since we released TorchLean in March, quite a lot has changed!! I've finally written up some of the new features we've been working on, and one of the biggest developments is that TorchLean is becoming a more general tensor programming and execution framework inside Lean. We've expanded the tensor API with shape-safe operations, einsum for general tensor contractions, and rearrange, expand, and reduce for manipulating tensor dimensions. You can now express matrix multiplication, multi-head attention, and more complicated tensor computations directly in Lean, with shapes checked during elaboration. Even more exciting, our einsum implementation has a verified lowering path, generating fused index-based computations with Lean-checked proofs connecting them to their mathematical specifications, including the order of arithmetic operations and also matching python einsum speed!!!
We've also been developing a shared intermediate representation (IR) for execution, differentiation, verification, and compiler transformations, including support for importing computation graphs from external ML frameworks. Alongside this, we've expanded CPU/GPU execution, including compiling supported Lean functions to CUDA, and integrated LibTorch for larger-scale training now matching pytorch speed. Other developments include configurable numerical precision through FloatLib (whatever format you desire), higher-order automatic differentiation connected to mathlib, and additional tools for neural-network certification and numerical reasoning. The goal is to let researchers write computations naturally, execute them efficiently, and establish mathematical guarantees about what they actually compute.
It's been an incredibly exciting time for mathematics and lean and I believe the future will be even greater: )! With @leanprover and mathlib, we're seeing what happens when mathematical knowledge becomes something computers can independently check, build on, and extend. I believe something similar can happen in ML and scientific computing, and much of my PhD at @Caltech has been about building the foundations for that. Modern ML systems involve much more than mathematical functions: numerical approximations, automatic differentiation, compiler transformations, optimized GPU kernels, and complex training pipelines. I want us to be able to work with the computations researchers actually use, run experiments, and prove meaningful properties about those same definitions!!
There's still an enormous amount of work ahead, particularly around GPU kernel verification and connecting the existing ML ecosystem to Lean, but I believe now is the right time to start building something analogous to mathlib for ML and scientific computing! Everything is open source, and I'd love for more people to experiment with it and contribute :)!
Over the next few days, I'll also be sharing concrete examples of what we've built, verified, and discovered along the way, from training GPT 2 in Lean to checking transformer computations, finding bugs in major ML libraries to investigating numerical issues in LLM inference, and formalizing results from ML papers. I hope these examples show both what's already possible and how much more there is to explore!
Happy to chat about these topics or just in general about research, my dms are open!
Longer Blog: robertboy18.github.io/TorchLean-Verified-Examples/torchle…