Harald Schäfer
@___Harald___
Followers
2,793
Following
112
Media
42
Statuses
282
Explore trending content on Musk Viewer
Israel
• 4518146 Tweets
Iran
• 3105209 Tweets
ايران
• 1173898 Tweets
Presidenta
• 469251 Tweets
Middle East
• 397775 Tweets
Halloween
• 333796 Tweets
イスラエル
• 137820 Tweets
Wizkid
• 109166 Tweets
John Amos
• 105584 Tweets
Adams
• 95035 Tweets
Davido
• 93316 Tweets
60 Minutes
• 80951 Tweets
Margaret
• 59132 Tweets
Tampon Tim
• 52939 Tweets
#WWENXT
• 50512 Tweets
Moderators
• 48966 Tweets
Raiders
• 46246 Tweets
#VPDebate2024
• 42998 Tweets
からくりサーカス
• 40039 Tweets
ケンタッキー
• 33339 Tweets
Orioles
• 27662 Tweets
Frogido
• 27281 Tweets
Brewers
• 25500 Tweets
Nora
• 17621 Tweets
Giulia
• 14930 Tweets
Roxanne
• 14920 Tweets
FURIA Y LUCHI EN VIVO
• 14703 Tweets
Aces
• 14479 Tweets
#GandhiJayanti
• 14020 Tweets
महात्मा गांधी
• 12875 Tweets
Go JD
• 12686 Tweets
#マックのコーヒー記念カキコ
• 11242 Tweets
Gago
• 10097 Tweets
第三次世界大戦
• 10066 Tweets
![Aimee [Ah-Mee] Song](https://pbs.twimg.com/profile_images/1199822691248640000/hd45cB3g.jpg)
Pinned Tweet
The world is not zero sum. People can create something out of nothing, and many do. Should be encouraged more.
3
13
145
99% of all life by biomass learns just by mutating DNA, no neurons. Artificial neural nets taught us a lot, but to truly understand intelligence we also need to recreate DNA.
I made a repo on artificial DNA. It solves tic-tac-toe reliably in 30min.
50
93
1K
Interesting to see
@ylecun
arguing pure ViTs are not practical. We recently moved our compressors over to pure ViTs, no convs.
They take longer to train, but work quite well. Left image is compressed to only 224 bytes, right is original.
31
53
798
I've worked at comma for almost 7 years. Our goal has always been to make an autonomous driving agent, by letting an ML model learn to drive by watching humans. We've done that, and shipped it to users with open-source code.
It was 3 years ago when we first shipped end-to-end
comma's jobs page is now on GitHub. Link from the main website.
We are set up for serious growth in the next 2 years, and we need to hire ~5 more people. We considered using a recruiter, but the language they used to speak about candidates was so dehumanizing we couldn't do it.
97
141
2K
11
26
402
Are you a windmill farmer tired of driving around to check on your windmills? Let openpilot drive for you!
15
42
345
Out of ~11k devices
@comma_ai
sold, ~4.4k are active daily! Users drive more than 1hr a day, with 51% of miles driven by openpilot. Incredible numbers!
Users like our product because we spend our resources on making it better. No bizdev, PR, or ads. Just engineering.
11
15
296
After 8 years in the US, I got a green card. This place isn't perfect, but it was founded on a set of beautiful ideas that are worth fighting for.
I want to build, learn, and explore. And there's no place I'd rather do it than here. Ad Astra!
16
3
220
comma will be the first company to ship a robotics product trained in genAI simulation to users. Generic methods like this will soon dominate all of robotics.
We just deleted the training code for our current driving models. The next driving model merged into openpilot master will be trained in our new ML simulator.
Tomorrow, we'll live stream a drive with the latest model trained in the new simulator.
12
42
575
9
7
206
Even in the wilderness, openpilot can make driving chill!
4
9
179
The video looks a bit crappy, but the models learn to drive pretty well from it. Keeping the quality lower allows as to iterate faster while we work out other bugs and kinks.
Very optimistic about how good these models are becoming.
We are currently working on driving models trained fully on-policy in ML simulation. All driving behavior is learned e2e in the imagination of a transformer.
We plan to ship to release by November. If we do, we will print the model weights in an encyclopedia-style book series.
21
51
627
6
3
103
We will ship models in the next few months to users, trained in video based learned simulator. We expect performance will be incredible. We had first successful experiments last week, we drove on a model
@yassineyousfi_
trained only in ML video simulation.
Great
@comma_ai
by
@yassineyousfi_
talk on how they now train on "imagined data" to boost performance. They train vision model in real world & policy model on learned simulation.
Critically he said "It can scale to any robotics problem". Exciting road
1
6
19
3
6
92
Despite the graveyard of sham companies, self-driving is still the coolest robotics problem.
Reliable agent-like robotics needs training with a world-model, RL, and on-device learning. This is all needed for
good self-driving too, and immediately useful when it works.
3
1
82
@ylecun
Interesting comments, thanks for clarifying! To be clear, we use convnets or hybrids for most of our models, and I suspect Tesla does too.
For that particular use case the pure ViTs seem to work well. Likely because the resolution is so low (256x128), and global attention may be
3
3
76
Finance is the ultimate nerd-snipe. So much talent and energy is wasted on growing portfolios that could instead be used to build real things.
We need less blackrock capitals, and more wikipedias.
8
6
77
@realGeorgeHotz
@elonmusk
I think the underlying problem is that advertising and shitty digital services with network effects are so wildly profitable.
In the current legal/cultural landscape building a good internet is not financially rewarded, and making a bad internet is.
3
3
74
Sample efficiency is the most important metric for the future of ML.
The ability to improve behavior based on minimal feedback is fundamental to intelligence.
5
5
70
Turns out we have been putting green garbage into the models in openpilot at runtime for years! Silently regressing driving performance. Even mature code is filled with bugs, how many like this are left?
4
1
68
I can't wait for open-source robots to fold my laundry. It's my least favorite household task.
The autonomy team is building autonomous agents, and things are getting better fast.
2
21
156
4
2
69
This simple lane-center convergence test has been our most valuable test at
@comma_ai
Since these models were only trained to mimic human driving from real data, it proves that humans on average drive in the center of the lane.
3
2
68
I like hiding stuff, hope you like finding stuff. $100 in the blue canister for whoever finds it first.
32.7349314, -117.1639719
Let me know if it’s found so I can update.
10
2
61
Talk of AGI is dumb when we can't solve PID.
Despite all the progress in ML, "AI" can't even solve simple simulated controls problems.
All the good submissions are hand-tuned feedback controllers with a few lines of code.
Think I'm wrong? Beat the challenge and win $500!
The controls challenge is heating up! We got some great submissions this week.
5 days left to submit your winning solutions!
0
1
27
3
2
63
LLMs answer questions like they’re bullshitting their way through an open-book test.
They find some relevant info, but often miss the point.
I love that LLMs exist. Now when a human gives you a wordy bullshit answer, we know it’s worth as much as an API call, around $0.002.
1
3
61
@Scobleizer
@elonmusk
At
@comma_ai
we’ve been shipping end-to-end driving models that learned from human driving for years. Not only are there public demos but it’s free and open source.
6
0
53
@Mad_dev
Do you? Creating dogs only took a couple thousand years, and viral DNA can evolve so fast you get new strains of Flu every year that are immune to the previous round of antibodies.
5
1
52
Nvidia's valuation is roughly ~$1million per H100 built.
2
3
52
The Fed is out there tuning the world’s highest stakes PID controller.
3
7
51
@Freesurge
@ylecun
This is way more than our current openpilot models see, which is way more than any non-tesla ADAS systems see.
1
0
49
End-to-end imitation learning from human driving has been discussed for many years before 2022. We started shipping e2e models at comma in 2021!
I’m curious to find out more about how V12 is trained. Is it trained in simulation? We never got it to work without that.
Wonder how V12 end2end got started?
During a meeting in December 2022 at Tesla Headquarters, an Autopilot Engineer named Dhaval Shroff pitched a new idea to Elon Musk:
"It's like Chat-GPT, but for cars!!!" "Instead of determining the proper path of the car based on rules, we
78
91
999
4
1
46
Had a great time with silly and billy today.
1
1
43
Last week I set out to photograph an unknown cliff. This is the 17th tallest cliff in the contiguous US, rising 1400m above the creek with a 350m almost-vertical section.
As far as I know this cliff has not been documented. It's several miles from the closest trail.
3
1
43
The future is so exciting
Check out our model learning to drive in our latest "DriveGPT". DriveGPT is a simulation model that can generate realistic road simulations for policy models to train on. All video below is imagined by this model (top is narrow FOV camera, bottom is fisheye).
13
49
363
1
0
41
The US government is 35T in debt. Last year it took in 4.4T in revenue and spent 6.1T.
If the government DOUBLED income tax collections, it would still take 50+ years to get out of debt.
That's not good.
3
0
41
Here’s me in a barrel roll!
Scientists of Twitter! The world needs more reminders that we're living, breathing human beings.
Quote tweet this with a picture of you doing not-science
9
9
261
4
0
41
Anyone at openAI wanna take this opportunity to do something fun and torrent GPT4 weights?
2
1
36
ML is very good at ignoring anomalies in the dataset. That's great when they're bugs, but terrible when they're not. Real anomalies are very interesting.
If we let LLMs control the world's information flow, we may be letting them hide our most interesting data.
2
1
39
Cool that
@Tesla
FSD is going end-to-end! Excited to hear more details, from the livestream it sounds like off-policy imitation learning on curated human driving? No simulator? We never got that to work at all.
This is still mostly how our system works:
3
4
37
It makes me so happy that openAI has lost its lead in LLM/GPT products, and there is now a pretty competitive market. A monopoly is bad for progress and bad for consumers.
This bodes well for the future of ML-based products!
3
1
35
Moore’s law but for corn? Pretty incredible achievement for humanity that we now get over 6 times more food out of the same patch of soil than we did 100 years ago.
4
2
35
@Scobleizer
@elonmusk
@comma_ai
We are a tiny engineering team, no marketing or PR.
But me and the other engineers are more than happy to talk about our work. We have no secrets :)
I think we’ve done some genuinely cool stuff with regards to end-to-end ml for driving. openpilot has driven 100M+ miles.
1
0
36
It’s amazing that so many of the best tools are open-source. PyTorch, linux, git, vscode, numpy, …
Open-source would be a worth praising just for the sake of sharing knowledge, but it turns out to also be the best way to achieve quality.
4
1
34
Really proud of what we achieved with the Taco Bell drive. I’m excited to see what we’ll accomplish over the coming years.
Last year openpilot drove to taco bell using end-to-end machine learning. The model learned to drive by mimicking human driving.
Check out Harald's full talk about it:
5
14
113
3
1
35
Lightfish, an autonomous boat, is on a 2500mile ocean voyage to Hawaii. It was stuck in a fishnet for 22 days, but is back on track!
Really cool to see companies at the intersection of robotics and ocean adventure.
Live metrics and daily video here:
2
3
35
This is what intelligence looks like. So smooth, so elegant. This is what the robots are competing with.
3
1
36
It’s a pretty crazy time in ML where neural nets can write poems, translate, and make stunning images. But ML can’t efficiently solve even the most basic controls problem, like tuning a PID loop.
4
1
34
Over comma's lifetime the hiring challenges have gotten easier as the field progressed to the point where they are "solved", and a new challenge is needed.
Very excited to see how and when the controls challenge will be solved!
If you want to do robotics with end-to-end ML, your ML algorithm needs to directly control the actuators. Check out our experiments where we try to solve steering control in openpilot with RL:
2
14
134
0
1
33
@realGeorgeHotz
@elonmusk
Yes, it sucks for the end user.
I hope we can make it legally or culturally untenable to run these types of businesses. Personally I would like a law that requires a paid ad-free version of every ad-funded service. At least then I know how much they're selling my eyeballs for.
1
1
26
AI doomerism appears to stem from the idea that an "AGI" can be trained once, and then copy-pasted to take over the world.
This is unrealistic. If humans were all identical we'd have gone extinct many times over. Diversity is essential to be robust in an adversarial environment.
3
0
28
It's crazy cool that GPTs can fully simulate the physical dynamics of a car's steering system.
Try out our latest coding challenge! This repo includes a GPT model that simulates a realistic steering response of an openpilot-compatible car. Can you make a good controller to make it follow the desired trajectory?
Prizes:
- $250 for solution with lowest total cost
- $250 for
7
27
191
1
0
25
This map I made with
@nuwandavek
shows all the cliffs in the world over 400m. The cliffs were found by scanning satellite topography data.
Many of these are remote, and have never been documented. Maybe some good projects for
@AlexHonnold
🤔?
0
0
27
Ever wonder where the world's tallest cliffs are? I wrote cliff-finder to fix that. Cliff-finder scans the whole world's terrain to find the tallest cliffs. Most you've likely never heard of!
@nuwandavek
Also wrote a great 3D visualizer!
4
2
26
@jvboid
I mean solving tic-tac-toe is obviously not hard. This is more about exploring learning strategies without loss functions, gradients, and optimizers.
2
1
26
@Scobleizer
@elonmusk
@comma_ai
Sounds great, always happy to talk more about comma, openpilot, and our ML stack.
0
0
25
Talking with new LLMs always seems impressive at first, but then quickly become frustrating and annoying.
Humans may be hard to beat as conversation partners anytime soon, we've been optimizing for being good conversationalists for a while after all.
0
0
25
In a sea of noise around AI,
@fchollet
and
@RichardSSutton
continue to have the most reasonable, and most inspirational views
AI researchers seek to understand intelligence well enough to create beings of greater intelligence than current humans.
Reaching this profound intellectual milestone will enrich our economies and challenge our societal institutions. It will be unprecedented and
23
65
452
1
0
25
Today we teach the robots, one day they’ll teach us.
First day at Billy university. What should I teach the new cohort? 🧑🏫🤖
8
2
98
2
0
23
Sometimes I worry about America's fiscal situation.
But then I remember Japan is 250% of its GPD in debt, has had no economic growth in 30 years, a birth rate of 1.3, and an average age of 49. If they can survive that, maybe we're fine.
5
0
23
It's cool all the AI doom stuff is dying out. Now we can all focus again on making machines as smart as possible as fast as possible!
2
0
23
What is intelligence? Is it compute? Problem solving? Life’s ability to survive? Something to do with entropy? When will a new science demystify this?
Thermodynamics gave us heat pumps, but took away the dream of perpetual motion. What will this new science give and take away?
4
0
21
@tsarnick
Darwin’s theory on the origin of species. The theory of evolution was so disruptive for all culture and philosophy at the time that even Darwin struggled to accept its conclusions.
0
0
21
One crazy thing about the Chernobyl disaster is that within months after the accident the 3 other reactors were restarted. They remained staffed and operational for years, with reactor 3 only shutting down in 2000.
3
0
21
2
0
21
There's a bunch of B1 lancers just withering away in the desert. Can some billionaire please buy them and turn them into sick private jets?
Like what's the point of being a billionaire if your plane doesn't even fly supersonic.
3
0
20
One of the first things Americans did after walking on the moon was figure how to make the moon a drive-through experience. Apollo was awesome.
0
0
19
@elonmusk
I bet in 2029 I still can't buy a robot for less than the price of a Corolla that can do my dishes and my laundry.
3
1
19
@tsarnick
"AI experts" who have no experience with machine learning, but still feel qualified to advise congress on it. Or I hope so anyway.
1
0
18
Exponentials are only scary when you forget to turn your brain to log-scale.
0
2
16
There's a rich history of attempts at humanoid robots. Mechanical turk, automata, WABOT, ASIMO, Baxter, ATLAS...
They all have one thing in common, they're cool party tricks, and otherwise useless. Will the latest generation be any different? Maybe, hopefully, but probably not.
3
1
17
I hope the future of AI can be even a tenth as impressive as the doomers predict. But technology improves with slow exponentials, not crazy step changes.
How much has your car, phone, or robot vacuum improved in the last decade? That’s what we should expect from the next.
4
0
17
ChatGPT cannot create another ChatGPT. The amazing thing about humans is that we all contain the seeds from which an entire society can grow.
At the core of all life is the ability to reproduce and self-improve. We must create machines that are alive.
1
1
17
@ArtDeza
@yar_vol
@ylecun
@Artificio_Org
We're not training perception models. We are training agents, which need to train on-policy to work well. That means train in simulation (real world is expensive). Would obviously be better to train higher res, but simulator becomes slower then, so not practical yet.
2
0
15
@Artoftheproblem
@yassineyousfi_
@Tesla_AI
@wayve_ai
These GPT-like models can produce realistic futures conditioned on action. That makes them simulators. They can effectively simulate anything you have sensor+action data for. That scales to all of robotics. In fact we just had success doing this to model vehicle dynamics of a
0
0
13
Treasure hunt part 2
$100 prize. This time no coordinates, just two pictures taken somewhere in San Diego County.
Cache contains a book to write your name. First person to email me the code written in the book will be sent $100.
Please leave book/cache in place!
0
1
14
💛
To all the people doing the drive from San Fransisco to Seattle, consider buying a comma C3X to make your drive more chill!
17
12
259
1
0
14
There are few things more infuriating than your car not letting you pair bluetooth while moving. Who on earth came up with this?? And why did they not consider a passenger wanting to pair???
0
0
14
@Artoftheproblem
@yassineyousfi_
@Tesla_AI
As far as I know nobody has made a good driving agent trained in this type of simulation. But many are trying, including us,
@wayve_ai
, Tesla, etc... We plan to be the first to get it in hands of users.
1
0
14
Treasure hunt round 3, sunken treasure
32.716162, -117.256385
$100 prize. This one is harder so you get coordinates and pictures.
A scuba notebook is attached to the anchor with a code and email address. First person to email the code to the address wins.
1
0
12
Also cool to see how well it generalizes, since this metadrive simulation test looks nothing like the training data.
0
0
13
@ylecun
We’ll have RL policy model training working in our world model/simulator soon!
Preview of our ML simulator. Bottom video is real , top video is a simulation initialized with the real video. We make a lane change in the simulation so you can see the videos diverge. The real video is compressed with an ML compressor so looks a bit dreamy.
10
17
248
2
0
13