Bharath Ramsundar
@rbhar90
Followers
12K
Following
95K
Media
299
Statuses
15K
Founder and CEO @deepforestsci. Creator of @deep_chem. Author @OReillyMedia. @stanford CS PhD. https://t.co/7LDcegrCsc
Joined June 2015
Respectfully, AI today is not like atomic science before the Manhattan project. The problem is there's a terrible Wizard of Oz effect. Even smart people who understand LLMs intellectually get fooled by the seeming intelligence of ChatGPT etc. Smooth text interpolation + a giant.
85
182
1K
This picture has been sticking in my brain. This is a good exemplar of why LLMs as computing platforms don't make sense to me. They learn the input distribution biases and all, and not necessarily any underlying structure or meaning. It's the hallucination problem in another form.
79
96
944
Respectfully, I strongly disagree with this view on multiple fronts. OpenAI is not the right entity to shoulder a burden on my behalf or much less on all of humanity's behalf. Open source and sunlight will do much better for responsible AI than secretive opaque entities. Hubris.
Building smarter-than-human machines is an inherently dangerous endeavor. OpenAI is shouldering an enormous responsibility on behalf of all of humanity.
33
103
871
My mental model of an LLM is an interpolative vector database. Larger LLMs correspond to bigger databases. The vector embeddings allow for interpolative behavior to generalize to some degree due to rich embeddings. Given this model, I am trying to understand the scientific.
28
76
698
The point of a PhD isn't really novel thinking or research even. It's more about learning how to interface with the centuries old chain of academic literature. The PhD trains you to construct memories ("papers") that are appended onto this chain for preservation.
12
96
579
I have a personal goal to spend more time being bored. If you're always busy, always excited, there's no room for mental rest and contemplation. But if you're bored, the mind wanders. I've come up with some of my best ideas when bored out of my head. You might too.
27
72
559
I'm excited to announce that my book with @Reza_Zadeh, "TensorFlow for Deep Learning" will be out on March 1st! Check it out for an introduction to the fundamentals of deep learning that focuses on conceptual understanding
13
81
418
Repeatedly hyping AGI fears is an excellent PR strategy. Almost no downside to the company if AGI doesn't happen in the next 5 years. It does tremendous damage to the field though and misguides policy makers. It's increasingly important to push back against the hype.
10
40
287
Two parallel trends I see in AI. First, a growing realization that deep learning isn't going to get to AGI. Second, the best AI work continues to build and extend on core deep learning insights. It sounds contradictory, but deep learning is a building block not the solution.
12
44
300
I've written up a brief paper review of "Molecular Attention Transformers," an intriguing new model for molecular machine learning that blends graph convolutional methods with transformer architectures
3
60
282
This is one of the most innovative deep learning papers I've seen in a while. Uses @PyTorch to construct differentiable dynamic programs. Allows for the use of backprop for PGM inference in a structured fashion
Our work w/ @mblondel_ml 'Differentiable Dynamic Programming for Structured Prediction and Attention' was accepted at @icmlconf ! Sparsity and backprop in CRF-like inference layers using max-smoothing, application in text + time series (NER, NMT, DTW)
1
79
282
What can't deep learning do? Worth putting together a list of known failures to guide algorithmic development.
14
69
277
I like this emerging view of LLMs as approximate retrieval engines, something like a continuous analog of a database. Practical, powerful, and thankfully distant from AGI mythology.
Don't confuse the approximate retrieval abilities of LLMs for actual reasoning abilities.
13
24
255
The "12 steps to Navier Stokes" tutorial series is amazing. I've had trouble building a mental model of Navier Stokes but the 12 notebooks really break down how the equation works into bite size chunks
2
36
240
1/ The new paper on Transformer limits ( has me thinking a lot. I've tried a lot to use ChatGPT and found it a lacking tool for my applications. It does an excellent surface level job but often fails to do deeper analyses.
9
41
248
Nowadays I learn new mathematics primarily through coding. I build a model system in Python, write unit tests, construct examples and iterate. There are limits to this style, but it's surprisingly powerful. I suspect the math classes of the future will adopt this system.
10
27
233
The Feynman lectures are available publicly online on Caltech's website. This is a great resource for learning more about physics
6
66
224
Recently, I've started reading some pure math again. Unlike when I was younger, I have no particular aims or goals and am mainly looking for beautiful things. Maybe unsurprisingly, I find myself starting to get ideas I couldn't digest before.
8
10
215
This is a good example showing LLMs are basically powerful autocomplete tools backed by large databases. I think 23 triggered a numerical flow and it kept rolling. The world model is limited at best. This isn't AGI.
Even the smartest LLM is "dumb" and overfit. Change a few numbers and watch them collapse
20
30
201
I'm sure this blog post is excellent, but @Medium's paywall is preventing me from reading it. Proliferating paywalls on blog articles is an unpleasant future. It feels like inviting academic paywalls back into the tech world. Let's start avoiding paywalled services like @Medium.
Nice blog post titled "The Quiet Semi-Supervised Revolution" by Vincent Vanhoucke. It discusses two related works by the Google Brain team: Unsupervised Data Augmentation and MixMatch.
15
24
204
I'm starting to worry silly fears around AI doom could cut off societally useful scientific AI. An LLM trained on trillion molecules is by no means superintelligent or even intelligent outside chemistry but could fall afoul of bad regulation.
I'm opposed to any AI regulation based on absolute capability thresholds, as opposed to indexing to some fraction of state-of-the-art capabilities. The Center for AI Policy is proposing thresholds which already include open source Llama 2 (7B). This is ridiculous.
17
30
202
Deep learning is becoming a subdiscipline of software engineering and systems software. Academic machine learners are drifting into causality, quantum ML, probabilistic programming and neuroscience. This is actually healthy. I think the lemon has been juiced thoroughly already.
5
35
201
AI agents make for very cool demos but I'm not sure when you'd actually want one. LLMs are still pretty mediocre at instruction following. An agent with chained LLM calls can go off the rails pretty easily. I think most of us don't want uncontrolled systems; rather we want tools.
26
17
193
I think programming language work doesn't get enough credit for deep learning. Automatic differentiation is what's enabled deep learning frameworks. If backprop had to be implemented by hand, deep learning would never have taken off.
5
21
189
I'd really love to see an "American Shenzhen" develop. The ability to cheaply build hardware prototypes locally would be an amazing resource for researchers and entrepreneurs.
10
13
193
I strongly disagree with this report. Those of us who disagree and don't think AGI is coming immediately need to get word out there more and push back against bad policy recommendations by Doomers. There are real world repercussions to bad geopolitical policy.
🚨Exclusive: a report commissioned by the U.S. government says advanced AI could pose an "extinction-level threat to the human species" and calls for urgent, sweeping new regulations.
12
35
189
I loved @fchollet's recent paper. One of the few papers about AGI that actually makes you think. Definitely recommend a careful read.
François Chollet’s core point: We can't measure an AI system's adaptability and flexibility by measuring a specific skill. With unlimited data, models memorize decisions. To advance AGI we need to quantify and measure ***skill-acquisition efficiency***. Let’s dig in👇
0
30
181
Deep learning for gravitational wave detection! Deep learning is really taking off in science. I suspect it's because a new wave of physics grad students who've done the ML coursework now understand both physics and ML and can find applications.
2
60
171
I enjoy working with code written by non-engineers. Often times in not following "best practices," they come up with creative solutions to problems. Novelty becomes unwieldy at scale but is beautiful in the small.
7
11
157
Scientists at @OpenAI, if you really see evidence of apocalyptic capabilities, you have a moral duty to publish asap. Prove it so us skeptics don't keep fighting you foolishly. If you don't have evidence, why are you raising public fears? Show proof or stop hyping please.
If this is true, please publish openly. I don't see evidence provided for these claims from any published material. I've tried open versions of ChatGPT etc. It's nowhere near this capable.
8
21
162
Random forests: a truly magical and underappreciated algorithm. My first query to newcomers to deep learning is "Have you tried a random forest to baseline?".
2
27
160
One of life's trickiest skills is learning to learn new things. In a new blog post, I share some tips for learning to learn in everyday life, such as greedy learning, pushing yourself, and reframing new concepts in terms of the everyday.
3
44
154
OpenAI does a lot of damage with AGI mythologizing. Failure to be honest about what's in the training data and how that explains seemingly intelligent behavior is core. This is why we need open LLMs. Lying about AGI will damage the field and prevent useful applications.
Goes to @ylecun 's point that LLMs do well when they are fed the answer. I'm being a little unfair but the hype cycle needs to deflate a bit.
2
18
152
I'm excited to announce that @deep_chem 2.0 has just been released! We've significantly refactored and extended our TensorGraph framework (built on @TensorFlow) to let us support more deep learning chem/bio/science applications. Please check it out!
3
45
154
I've put together a @deep_chem model wishlist with a collection of models we'd like to see added to DeepChem. Molecular ML, materials ML, physics ML and more needed. Please take a look and contribute!
3
30
151
Doing 0.1 seconds of molecular dynamics is crazy. For context, integration steps happen at the femtosecond scale. That's a 100 trillion integration steps!.
@nvidia We have analyzed our dataset (capturing 0.1 seconds of simulation, the largest in history!) to identify over 50 novel pockets across many #COVID19 proteins! These “cryptic pockets” are candidates for targeting with antivirals.
5
25
145
I think one of the biggest mistakes I made when starting research was trying to become a "great researcher". It's often more useful to just explore for fun or curiosity. The paper I worked hardest on so far still has 0 citations and thats ok.
My student sent me this list saying they have to improve themselves in many areas. Such a list can do more harm than good. While I appreciate author's intention to motivate one for greatness, I don't think it can be planned. But you can plan to be a "good researcher.".
6
14
143
Noether's theorem should be taught earlier in physics (maybe even at the high school level). The idea that energy is just the conserved quantity of a time invariant system for example is a simple definition of the otherwise mysterious term "energy".
10
16
139
TensorFlow 1.7 release candidate is out. Biggest change is that eager mode is moved into the main library. This is going to dramatically change how users write TensorFlow code. Looking forward to more dynamic graphs!
1
54
142
Join the new @deep_chem Discord server if you are interested in open source scientific machine learning! We're building up a good community to discuss scientific ML, LLMs and more
3
25
143
I've written a new blog post on "Machine Learning with Small Data." Small data ML is the future :)
8
40
140
Something worth noting here is that Baidu is actively covering up the infrastructure of genocide. If you're accepting a computer vision paper from Baidu researchers, consider what that technology could be used for.
How were we able to do this analysis? .@alisonkilling, Christo Buschek and I stumbled upon a strange phenomenon on China's Baidu Maps — light gray tiles appearing over known Xinjiang camp locations. By finding more gray tiles, @alisonkilling thought we could find more camps.
1
42
137
Another excellent take. Most of us sharing skepticism love AI and have been working on it for over a decade. Being honest is sustainable. Hype and pump and dump craters careers and misleads the public.
Of course factualism doesn't sell -- if I wanted to be more of an AI influencer I would have to be constantly tweeting about how AI is going to replace all programmers and doctors and so on in less than a year. That sounds exciting and positive, and it gets great engagement!.
5
18
92
Data is the new oil, but perhaps simulators are the new data.
5
24
146
I've personally struggled to get much out of generative AI tools. They're fun distractions but the hallucinations mean I'd rather just google+wiki most of the time. But smart people I know seem to love them. Perhaps it's a matter of working style?.
45
4
138
Some things take a really long time to learn. I think it took me 5 years before I could read and write Tamil well. There are some concepts in algebra I learned in college and I'm just starting to get now. Being willing to patiently learn something over years pays dividends.
4
11
143
My two cents, rumors like this have been circulating for a while. It's excellent marketing but I'm not buying it. I think OpenAI has far less than they imply.
I’ve heard the same (~“most papers are bad; very behind actual SOTA”) from someone at a top lab @dwarkesh_sp . @tszzl has said similar things. GPT-4 finished training *1.5 years ago*. What do we expect world-class researchers at OAI have done since then, twiddled their thumbs?
10
6
140
First two chapters of my and @Reza_Zadeh's book "Tensorflow for deep learning" available for free on @matroid blog
0
52
135
Leaving academia can be scary if you're a serious scientist. But it's deeply freeing as well. Being able to evaluate papers without worrying about alienating influential professors who could nix your tenure frees the mind. Academic hierarchy does a lot of unnoticed harm.
3
19
132
Differentiable programming seems like a really interesting extension to differential geometry. It's particularly interesting since manifolds are usually very smooth objects and programs combinatorial objects but now we're seeing a blending of the two subjects.
4
22
125
A lot of OpenAI's work is taking existing community models, running them at large scale, and then shipping them out online. Good systems software. But it's tiring how it's captured the AGI discourse especially when they don't publish or give proper credit.
3
15
136
Maybe it's my time in quarantine, but I'll share a couple unfiltered opinions about startups. Starting off, I think culture setting exercises in general are silly. Writing down a few generic statements ("We live for excellence") doesn't say much.
5
23
125
This is an important debate. I am nowhere near as qualified but I also respectfully disagree with the doomer position. It's important for scientists who disagree to speak up so there isnt the impression of scientific consensus of AI doom.
Yann LeCun thinks the risk of AI taking over is miniscule. This means he puts a big weight on his own opinion and a miniscule weight on the opinions of many other equally qualified experts.
7
8
129
This implementation of Genomic-ULMFiT provides a neat application of modern NLP models to genomic classification. Pretraining on a large corpora of genetic sequences is used to boost downstream genome sequence classification problems
0
26
130
This is a really interesting paper on biological deep learning. The researchers construct an end-to-end model of a moth's olfactory network. Trained with Hebbian rule. System capable of low data learning picking up new smells with 10 examples
3
22
123
The difficulty with judging AlphaFold3 is that we don't have a good way to test it out of distribution. I've found before that ML models can fail badly OOD where simpler physical models do just fine. It sounds like AF3 may have loosened equivariance constraints as well which may.
11
13
120
Blockchain is revolutionary for reasons I've seen few people express. It's not Bitcoin's valuation, but fundamental algorithmic advances that have the potential to remake Tech. My new blog post, "Why Blockchain Could (One Day) Topple Google," explains
4
23
100
How do you solve a truly hard problem? My undergrad math advisor said it's bit like trying to climb a mountain. You circle and look for a path to the peak that no one else has noticed. Often the searcher with most time and patience wins. Genius isn't what it appears from afar.
4
26
119
Learning how to read code is a skill worth developing. Not straightforward, since it requires you to form a mental model of code execution, but broad reading will teach you idioms and design patterns you didn't realize could exist. Pick a GitHub repo you like and start reading!.
3
24
119
Medium is really pushing their subscription model recently. So, users write free essays which we pay Medium to read? Sounds a lot like old school scientific publisher paywalls. No thanks!.
6
21
112
Have you ever wondered what a PhD is like? Check out my new essay, "A PhD in Snapshots." I've put together 10 progress reports I wrote charting the ups and downs and sideways of the PhD. It's a long and windy path!
2
20
116
Thriving in deep learning requires unlearning mathematics. Proofs are not to be trusted and only empirical results can be believed. Blockchain is the opposite. In a world where good empirical results are extraordinarily hard, solid proofs provide critical guarantees.
7
26
114
I lost interest in pure mathematics after starting CS grad school, but I've recently been getting back to it after following some interesting mathematicians on Twitter. The human angle makes such a difference. Seeing what people are excited about cuts through the noise.
5
6
114
Excited to announce that @deep_chem has crossed 1,000 stars on GitHub! Lots of interest in deep learning for the life sciences. Excited to see how our users leverage these tools to discover cures for diseases that have no treatments today!
1
29
117
Google Earth now offers up close pictures of backyards. This is pretty intrusive folks. I'm able to see my parents' backyard and their neighbors' yards in high rendered detail. When was consent asked for this?.
10
16
98
I'll be defending my thesis, "Molecular Machine Learning with DeepChem" on December 12th from 3:15-5:15 at Stanford. The first portion is open to the public, so please come by if you're interested in learning about molecular ML!
9
14
111
I really recommend this paper. Reading it highlights just how many details are left out in most "open" LLMs. The ML community should return to its strong open source roots and make more fully open LLMs to advance the fundamental science of the space.
LLM360: Towards Fully Transparent Open-Source LLMs. paper page: @AurickQ, @willieneis, @HongyiWang10, @BowenTan8, @ZhitingHu, @preslav_nakov, @mbzuai , @PetuumInc , @Meta @MistralAI @huggingface .
2
16
103
Something that surprised me when I learned it was that federal grants in the US provide ~$75K/year/grad student. But the grad student only gets about a ~$25K stipend with the rest going to the university. It feels like the ratio should be inverted with $50K stipend.
9
8
109
There's a lot of talented folks working on not terribly useful products. The on going crises are a good opportunity to reevaluate whether what you're working on is meaningful and worth your energy.
9
19
110
@schwabpa The conversational behavior really creates a poweful illusion of presence. It's fascinating how it enables us to mirror ourselves onto the chatbot. Also why these things are so dangerous as therapy agents etc. They tell us completions we want to hear pseudo-authoritatively.
5
10
112
This is kind of sorcerous. Forget the AI models. I'm here for hard light displays .
7
18
107
This has triggered a lot of controversy. To share a more positive take, I got my start in research in high school with Intel STS. My high school didn't have supporting programs so I emailed a lot of faculty and researchers. A couple were kind enough to gjve me pointers which.
NeurIPS 2024 will have a track for papers from high schoolers.
14
12
108
This article suggests that biological neurons are roughly analogous to 5-8 layer deep networks (w/ 256 nodes/layer). This seems intuitively reasonable. Deep learning practitioners know that the intelligence of a deep network is usually quite limited
1
20
104
Check out our new paper on "Spatial Graph Convolutions for Drug Discovery." Converts a 3D macro molecular structure into a graph structure that it feeds into a graph convolutional deep network. Matches state-of-art with end-to-end learning.
3
44
103
Physicists love Boltzmann distributions partly because of their analytic tractability. Computer scientists less so because of their computational intractability. Efficient generative neural models of physical systems will likely gain in prominence over coming years.
1
23
103
Something like 8 years ago, I did my first internship at Google. I remember working through a simple mapreduce job and launching it on 1000 cores. It was an exhilarating experience and one that pushed me towards CS grad school.
2
4
103
If you're working on AI, ask yourself why progress in AI is a genuine human good? Most applications of AI so far seem to involve public face recognition and audio surveillance. We need more recognition that research is being used for regressive purposes.
12
20
99
A brief personal announcement: I’ve departed my role as CTO at @computable_io. It’s been a pleasure working with a talented team to develop and ship the world's first protocol for decentralized data cooperatives!.
1
2
98
Deep learning developers should be concerned about Google's extensive patents in the deep learning space. If the patents were truly for defensive purposes, Google could open source them. Note that they haven't.
Serious question: if Google changed its license on future versions of TensorFlow, and a company using an older Apache 2.0 licensed version reimplemented newer functionality. that would violate copyright? Precedent set by Oracle vs Google over the Java API?.
3
37
92
I'm reading "Surveillance Capitalism". It's making me take a hard look at big tech and Google. I've been in and around these companies for the last decade. Many friends and mentors I deeply respect there. But I can't say I deeply grasped the consequences of the business model.
2
8
95
Part of the reason LLMs cause so much hype and confusion is that they are genuinely new bits of software. We don't yet have common understanding of what approximate retrieval from large compressed text/image/video databases can do. It's too easy to fall into the trap of imagining.
10
12
95
I'll just go ahead and make a prediction that AlphaFold will win a Nobel. Accessible, believable proteomes for humans and other species too, wow. There are a lot of proteins in there we barely understand that we now have a structure for
3
12
93
I've written a new blog post, "The Advent of Huang's Law," which makes the case that the continuing improvements in GPUs alongside the steady spread of deep learning applications indicates the start of a new epoch in modern computer science.
1
31
93
Digesting very complex ideas can take years. Often times, the concept won't click until you face an external challenge that requires you to master the concept to proceed. It's really important to be patient. The fact you don't get something now doesn't mean 5 years out you can't.
4
13
94
When you're trying to digest a deep technical idea, taking it head-on is almost always a bad strategy. Instead take a strafing run against it; skim it trying to pick out one weak point, simple enough that you can get it. That gives you a toehold. Repeat patiently till it cracks.
4
16
93
I'm really excited to be able to announce that @deep_chem 2.4.0 is out! Over a year's worth of hacking from an entire team of superb developers. DeepChem has been basically rewritten at this point and supports many new use cases.
DeepChem 2.4.0 is out!! This release features over a year's worth of development work. Support for TensorFlow 2 / PyTorch. Significantly improved production readiness. Faster datasets. More MoleculeNet datasets, materials science support and more!
5
17
89
I think real breakthroughs in AI will come from students who are currently studying things like quantum computing, low-energy circuit design, neuroscience, or cryptography. The field of machine learning, like graphics before it, is tilting towards applications right now.
3
13
93
Check out first two chapters of "Tensorflow for Deep Learning", my new book with @Reza_Zadeh and @OReillyMedia!
5
27
92
Some sad news. Bram Moolenaar, the creator of vim, has just passed
5
19
93
Blog post about the end-to-end deep protein structure prediction paper by the author. Worth reading!
3
28
90
One of the tougher things about learning new ideas is letting go of the idea that you're an "expert". Really new ideas make you question many of your deeply held instincts and will often make you feel stupid. Embracing the bewilderment will make you more creative.
2
18
89
Spending a little time in a library drives home the limitations of recommendation algorithms. It's not possible to walk around Amazon and discover hidden gems that you'd have never thought to open if it weren't in front of your eyes.
7
18
87
If you want to do good science, read widely. Papers in your field, in adjacent fields, and even in disciplines you know nothing about. Curiosity is a really good guide here. If you think something looks cool, just go ahead and read it!.
1
21
86
The reason Ethereum deeply excites me is that it's the first attempt to build a permisionless nation state. A rudimentary financial system has been built. Storage, communication, identity and more are in the works. Governance is painfully forming. This is what big tech should be.
1
15
83
I'm excited to announce @deep_chem 2.2 has just been released! This new version contains improvements to protein structure handling, better support for image datasets, and brings @deep_chem closer to being a general library for the deep life sciences
1
22
84
This is an excellent summary of logical holes in AGI fears. Why do we expect recursive self improvement? Usually iterating systems hit a fixed point fairly rapidly. Exponentials are usually S curves. AGI fear mongering seems self serving for organizations that benefit from it.
@Simeon_Cps what's particularly strange about this discourse is that so much of it is counter to what we know about systems:.- complex systems hit diminishing marginal returns.- even a billion years of evolution has not evolved the ability to auto-improve intelligence .- centralized.
15
12
81