Tim Zaman
@tim_zaman
Followers
25K
Following
3K
Media
107
Statuses
854
AI at Google DeepMind. Previously Tesla, X/Twitter (head of AI Infra), NVIDIA. callsign PD4TA
San Francisco, CA
Joined June 2010
We have the most epic planet-scale AI infra at Google DeepMind, but we are ambitious at making it even more awesome than it is. We're looking for seasoned, pragmatic, hands-on engineers as vanguards of the DeepMind codebase, and innovate on our AI infra with teams and legendary.
20
20
322
I got a violation posted on my front door in SF for remoddeling my own kitchen (myself) in-kind. I didn't know bc Holland, I can do whatever the f* I want in my house, bc. it's my house!. So the next day: i went to the SF building department. They hadn't processed the fine yet,.
NEW: Just issued an Executive Order that will allow victims of the SoCal fires to not get caught up in bureaucratic red tape and quickly rebuild their homes. We are also extending key price gouging protections to help make rebuilding more affordable.
405
445
7K
Tesla is best positioned to deliver on useful real-world AI:.It has the execution (team), the compute (dojo), platform+data (Bot+Fleet) the backing (Tesla) and the 0-to-1 (Factories) to do it. And all vertically integrated on a first-name basis. Join
114
378
4K
Tesla AI 10k H100 cluster, go live monday. Due to real-world video training, we may have the largest training datasets in the world, hot tier cache capacity beyond 200PB - orders of magnitudes more than LLMs. Join us!.
131
429
3K
At #Tesla Autopilot we are bringing up our 3rd cluster - one of the biggest supercomputers in the world. We're scaling fast and hiring, if interested please DM (ml/py/cpp/gpus/cuda/react/devops/sre)
94
398
2K
The practicalities of building a supercomputer are insane once you get down to it. Lets assume you have a datacenter location and ordered GPUs. You're probably thinking of these sleek orderly photos of racks of compute - just think about how those GPUs got there. I love this pic
@xDaily Tesla had no place to send the Nvidia chips to turn them on, so they would have just sat in a warehouse. The south extension of Giga Texas is almost complete. This will house 50k H100s for FSD training.
79
251
2K
Silicon valley is the only place where you can buy 96 core cpus at the coffee shop. Hmmmmm pcie lanes
107
128
2K
Excited for the big show today that many friends worked on tirelessly!.
The future will be streamed live. 10/10, 7pm PT
31
72
2K
Personal news -. Joined NVIDIA in 2016, and Tesla in 2019. Like family, I never thought I'd leave. My intuition pulls me to my next venture, that has the ingredients to change the world - again. I'm joining Google DeepMind this Monday. Excited to be a kid among legends!
121
42
2K
Insane time-to-LLM by the team. Earlier this year, there was literally nothing, no team, no gpus, no nothing. Excited for the future. Give it a try! The integration with X is quite nice, gives it a connected feel.
Just released Grok.
33
107
2K
Tesla FSD v12 is very impressive. I suggest anyone with a Tesla should give it a spin. Tricky driveway, intersection, offsetting, peeking, very natural and promising behaviour. I now wonder if they will solve self-driving before solving the auto-wiper 🤣.
119
103
2K
Azure/Microsoft released pics of OpenAI's next-gen AI datacenter. This supposedly powers [Chat]GPT* training (inference?). From the pics we can infer most components, power, network/ib topology, layouts, vendors and more. A thread 🧵[1/7]
41
361
2K
This is real: this is happening. This Train is going at max speed and cannot be stopped. (Pun intended). For context, today, our compute clusters have 0.3% of idle time; and 84% of jobs are high priority. We want to be in a place where we have an excess of compute.
31
142
2K
If you are at CVPR and want to work on the world's biggest supercomputers (nvidia gpu/amd gpu/dojo), DM @itsclivetime or drop by Tesla booth!.
The team is hosting a Happy Hour on Wednesday, June 21st, but please ask @philduan, @RamonaChandran, @aelluswamy or someone at their booth for tickets, not me :D Last year's Happy Hour is where I first met the team.
17
108
1K
Again, adding some color for the fans.- @MichaelDell showed off the racks he sold to X in a tweet last week. His racks are pretty lame compared to these. - @charlesliang (welcome to X!) pictured left is a true silicon valley legend, since 1993 founder-ceo of $SMCI (up 4200% in.
Thanks @elonmusk for leading the liquid cooling technology to large AI data centers! This may lead to preserving 20 billion trees for our planet❤️
38
90
1K
Would like to point out @O42nl was hired after reaching out over the Twitter dms. LinkedIn is dead. Twitter for Recruiting!.Also yes: we're always hiring excellent engineers, DMs open! We've got an insane amount of compute coming our way this year.
40
67
986
Tesla is sponsoring the @MLSysConf, come visit our booth for opportunities on the AI team and see our hardware. We have recently upgraded our GPU supercomputer (photo) to 7360 A-100(80GB) GPUs, making it Top-7 by gpu-count. Reach out to build #1:
33
127
828
@PrvnKalavai On prem all owned by Tesla. Many orgs say "We have" which usually means "We rented" few actually own, and therefore fully vertically integrate. This bothers me because owning and maintaining is hard. Renting is easy.
33
30
704
A few weeks post Twitter-acquisition in 2022 we found 700 V100 gpus (pcie, lol) in the datacenter. They were powered on and idle, and had been for ages: the forgotten remains of a honest attempt to make a cluster within Twitter 1.0. Haha how times have changed! 100k gpus on the.
31
41
605
Work on our vertically integrated in-house machine learning platform. Tiny team. Big compute.
23
65
574
Adding some color to what i see in these pics. Note @MichaelDell's photo from earlier this week was a close-up of the same rack: - $3M-$4M per rack pictured.- 6U Dell PowerEdge XE9680, Intel cpus (c'mon), aircooled (c'mon. ) with NVIDIA HGX baseboard (8x.
24
48
582
Google Trends identifies Tesla as 'Artificial Intelligence Company'. Same thing's happening as when I was at NVIDIA around 2016-ish: went from underrated gamer-company to AI juggernaut.
18
46
561
The "Tesla Autopilot" team is basically the "Tesla AI" team these days, given cross pollination and synergy between different tracks, like eg the TeslaBot. Esp true for AI Infra. If there is any company that will ship an actual humanoid robot it'd be Tesla.
Our TeslaBot team at Tesla is hiring across many disciplines! If you want to work with me on teaching humanoid robots to do anything, follow this link:. Follow this link to see all the types of roles we're looking for (many!):.
23
47
550
Looks like @xai s got the NVIDIA GB200 NVL72 up from @MichaelDell . Among first to turn on, they only started shipping these a month ago.
7
26
543
I feel current Tesla FSD in my car already captures 95% of the value i care about (highway nav), and don't rly care about 100% if I still need to pay attention. The real unlock is taking away the steering wheel - this is where you get my mom and everyone else on board.
Excited for the big show today that many friends worked on tirelessly!.
19
26
526
Congrats to SpaceX team, what a fantastic and inspirational achievement.
Watch the super hot plasma field grow as Starship re-enters the atmosphere!
8
17
468
@AravSrinivas @ylecun The nuance often kind of gets lost with E's one-liners. I think you need to distinguish between a Conv layer and a CNN. LeNet/AlexNet were obviously CNNs. But what Autopilot does today, you can probably hardly call a CNN, eventhough as u say - it prob must have conv layers.
17
39
462
My team's hiring for:.- ML Framework Engineer (py & c++).- ML Platform Engineer - Fullstack (py & js[React]).- ML Platform Engineer - Services (py & go).- ML Storage Engineer (py & c++).- ML Supercomputer Performance (py & c++).- (Germany) ML Platform SRE - Services (py & unix).
16
62
430
OpenAI finally joins team Vertical Integration. > You will design, build, and operate large-scale compute clusters to power our advanced AI research.
10
27
385
Managing a single supercomputer is hard. Now imagine having a lot - we're looking for fullstack engineers to help tie everything together into our machine learning platform (gui, tooling, job/model observability, health, etc). js+py. Interested? DMs open.
10
27
356
@DillonLoomis22 'Taking Delivery' sounds like receiving an Amazon package. For taking delivery, calculate back for 10k gpus.- gpus in a server.- servers in a box.- boxes in a truck.- amount of trucks.- amount of loading docks.- time it takes to unload a truck.
14
21
357
@sedielem Top hours wasted in my life.1) gaming.2) tv.3) batchnorm bugs. Probably closely followed by gpu driver updates.
18
10
332
Grok 1.5 announcement includes the examples that made me feel like self-driving can eventually be done (better) as a subset of a more generic AI, eg a multimodal LLM as shown here. When I was at Autopilot, this made me feel a bit worried. Such models can do examples like the
👀.
24
31
345
Proud to have engaged these two lovebirds. When we were building the first xAI cluster, it was hard to get vendors to move at our speed and be taken serious asking "what minute will the truck arrive". I asked for Charles to come over to X to chat and accelerate, and it took off.
Supermicro is here to support xAI's massive 10-fold expansion of the Colossus supercomputer in Memphis with over 1 million GPUs by establishing local operations/production, validation, service and support. With our optimized datacenter building blocks (DCBBS) and ambient.
12
19
330
[7/7] Closing thoughts - The AI tech tree is deep, and you could spend ur life in Pythonland. AI is expensive, and if you care about performance, everything below you is relevant, down to the bare metal. The future may be full of monolithic bare metal supercomputers.
17
31
281
Work on Tesla AI from Europe (on-site Germany), in role as SRE on our infra and supercomputers. Unique opportunity. I'd apply myself if I was in EU. Ninja skills *not* optional.
8
37
277
Tesla AI Infra. Where interns get more compute than their entire university. 🚀.
@philduan The infra is amazing: I constantly used more GPUs than I ever used at university and for some projects I used more GPUs than my whole university owns :o Also @tim_zaman and his team manage the infrastructure brilliantly. The code is nice and you can hack anything anytime!.
4
17
265
Hype. A joint timeline that once forked (Elon and Demis/DeepMind) now reattracting <3 <3. (Elon was early DM investor before merger w Google).
@elonmusk Thanks Elon! let's do an AI game together. .
6
11
264
Welcome to X @__eknight__ 👀. @Tesla_AI crowd should pay close attention. 🍿.
So excited that FSD customers were able to catch a glimpse of what we’ve been working on. Sincere thanks to my colleagues for the many long nights and hard work that led up to this point. Want to build end to end with us? Join the team @Tesla_AI!
3
17
245
+1, at Tesla and X my teams were so large I also had to adopt the much more superior "async 1:1". Please don't just drop your 1:1s, there is a lot of nuance:. With large teams, being a poor people manager is inevitable, but it's ok;.Having so many reports and avoiding 1:1s makes.
@StartupArchive_ Word. I had ~30 direct reports and didn't do 1on1s (as a scheduled, regular activity) at Tesla and imo it was great. Two meeting types that are a lot more useful:.1) The 4-8 person meeting where great ideas come from, and.2) The large meeting for broadcast. I went back to try.
4
15
248
Can someone get bike infra on the political agenda in the US please. Cheap. Easy. Healthy. Environmental. Social. Build it and they will come. Slow folks can use electric. Parents bucketbikes. Kids love it.
36
8
255
Can confirm, AMD MI250 is price competitive with basically zero code changes. MI300 series very appealing at the right price. Video hw decoder should be superior to NVIDIA's as it's an independent asic.
9
32
231
AI training native- vertically integrated storage filesystem. Tiny team, elegant code, gigantic storage.
3
22
228
A new (serious) AI startup doesnt need truckloads of dough. They would need gpus; by the 10ks. The only way to get that up in a <<12mo timeframe is to reallocate - so ironically, eg MSFT would need to take *from* OAI and give *to* NewStartup.
16
17
229
Meet the Tesla AI team at MLSys! Ask about model export, large scale training, running bare metal clusters, and dojo.
Come say hi at MLSYS! We’ve got a lot of cool things to talk about at our booth :)
6
16
217
[4/7] They are using the Azure ND A100 v4. Cost ~$150k/node. 8x A100 GPUs, 2x48 core CPU, 900GiB DDR4 ram. 8x IB HDR NICs provide a 200x8=1600Gb/s gpu-gpu fabric. The data fabric seems separate, using custom Azure stuff. Data ingest is a wimpy 50Gb/s: for LLMs totally fine.
4
28
203
@realGeorgeHotz Everybody gangsta until real world at-scale deployment. But it's a cool demo for sure.
3
2
200
@30SecondYou My sister in holand had an ADU made last year. I fell of my chair, look at these joins! Wtf!.I think the reasoning is, the unforgiving climate (compared to CA) and expensive energy means you want your home to be really well built to keep out the elements.
16
3
211
POV: getting married in 24 hours but ring has not arrived. There is something very visceral about executing your imagination in the real world. Wild nights with the blowtorch.
14
1
197
[3/7] Why is the datacenter in Quincy, WA?. AI eats compute which is fed by power and cooling (+=power). This is a massive Opex to cloud providers. But in Quincy you pay 3ct/kWh! (SF=30ct/kWH). This is bc they have hydro (green!) power. I guess that makes ChatGPT "green"?
4
25
178
@karpathy A nice pattern i learned ar nvidia, esp good for education, is add some ifdef's so you can either run the kernel on cuda or on cpu. This would only mean that you make your kernels block agnostic (only use on the gpu path) and for the cpu path you do a x,y double for loop.
2
7
176
My kid reinvigorated my love for stickers
8
4
169
Starlink at home depot. This is a commodity now.
BREAKING: @SpaceX's Starlink standard terminal kit is now available at Home Depot in the US for $600. Delivery is available immediately. Link:
3
4
142
3
5
137
@WholeMarsBlog Also - teleop is a hard requirement for a lot imitation learning. The quality of the 'imitation' is constrained by the quality and functionality of the teleop.
4
7
135
Taking away someone's compute is the ultimate humiliation in 2024
Over the past few months my team has been sailing against the wind. Sometimes we were struggling for compute and it was getting harder and harder to get this crucial research done.
5
6
126
Yep video went private I'll update the link when there's a new one.
2
6
120
[5/7] Fabric! A thin blue cable is optical/AOC (expensive) and a thick black one is copper/DAC (cheap). DAC only works short range ~5m. The data NICs flow into an underpopulated Arista 6070CX switch, and out-of-rack with 8x AOCs. gpu-gpu is all AOC, maybe to a core switch.
7
11
117
[2/7] The datacenter Microsoft/Azure pictures is in Quincy, WA, USA (47.23,-119.86). This is an old picture. Today, there's a datacenter in place of that parking lot. An old parking lot as the origin of this modern AI revival. Nice: I hate parking lots.
1
11
113
I was going to make fun of Dell, but I just realized my first GPU server was a Dell! It ran for 10 years without issues (pictured). Back then they were one of the few ones supporting FHFL double wides
5
6
118
[6/7] Their high-level layout is using cold aisle containment (this is the cold aisle), with amazing levels of standardization. All nodes depicted here are just for data storage and simple cpu compute. No HPC or AI happening in this aisle.
5
8
115
1
3
112
Personal news: I am starting PirateAI. The AGI LLM will be trained on a barge on open waters. Weights torrented through Starlink. Gpus by AMD. It will be maximally anti-establishment, deceptive and misaligned. Join us.
11
6
113
🚀 and happy 1st Googleversary to meeee.
What a way to celebrate one year of incredible Gemini progress -- #1🥇across the board on overall ranking, as well as on hard prompts, coding, math, instruction following, and more, including with style control on. Thanks to the hard work of everyone in the Gemini team and
8
1
114
Some might remember the 2015 "DIGITS DevBox" (I was hired to work on & why I moved to bay area). Basically, you bought a solid 4-GPU devbox (or made ur own), installed DIGITS, which shipped with multiple dl frameworks and had a builtin ai platform (scheduler, GUI, . ) 🧵.
Announcing NVIDIA Project DIGITS, a personal AI supercomputer that’s powered by the NVIDIA GB10 Superchip and based on #NVIDIAGraceBlackwell architecture. Preconfigured with the NVIDIA AI software stack, developers, researchers, data scientists and
5
13
112
@willdepue Not disjoined, just specialized. Mostly constrained by its realtime nature and in-car inference hw cost, flops and power costs. Roughly history:.1. End-to-end attempts since decades - all failed.2. ConvNet for eg laneline perception, controls w heuristics.3. Add more tasks.
4
7
108
@itsclivetime "No mom, these are not cup coasters.".True story - my mom once threw away a silicon wafer in the kitchen trash bin, bc she thought it was the bottom carton of a pie she bought that morning.
1
1
106
@alexandr_wang Why aren't we just feeding kids with candy only? The answer to this question and yours is the same. It's their addictive nature, and concerning to witness. And any healthy parent is totally fine their their kids being a "Normie". I am.
1
0
99
Google's coding ecosystem is truly fantastic. The way its components dovetail into each-other is quite remarkable, and oddly intuitive for a complex system. Makes GitHub feel like a git-wrapper, even though i love that too. Lots of opportunity for innovation here. Things like.
10
4
99
Updated link: here @karpathy also explains why anyone wanting to succeed in FSD would need large scale compute and storage.
4
10
90
@spacesudoer @elonmusk Ok lets limit amazon deliveries too. Incentivize local shopping, without all that transport and forests worth of packaging material. Seriously.
1
1
94
Calling in sick from work today. Can't believe it's been 22 years since 1.3.
Counter-Strike 2.
6
1
93
Origin of number 42 per Douglas Adams (1993):. > "The answer to this is very simple. It was a joke. It had to be a number, an ordinary, smallish number, and I chose that one. Binary representations, base thirteen, Tibetan monks are all complete nonsense. I sat at my desk, stared.
6
5
92
@karpathy @sinclanich Clip to int8 will speed things up drastically. If you do a top-k approach, you can recalc the top-k in fp32. Np can't do cosine distance fast though, but you can slap some instrinsics together in 20 loc. Will be bound by mem bw~ 200GB/s. Ur array is only 17MB, so 85us latency.
4
3
87
meta-comment: recruiting on X is real and works/got a hire from here before.
3
0
81
"Let me give you the money to buy NVIDIA" - Jensen and Masayoshi Son giggling about their business scheming with eachother is fun to watch - apparently Jensen could have fully owned NVIDIA in 2014, after Masayoshi bought ARM.
3
8
82
@WholeMarsBlog I recall @ARKInvest put a pricepoint at $1000 while the stock was $300. At the office we thought this was hilariously unrealistic. Estimating future value is so hard. People are bad at exponentials. Same when I was at NVIDIA when it hit $1000 pre splits, people got tatoos.
2
3
80
RIP TOP500. No one submits anymore, eg this 4.6k gpu (small) cluster is #9. Kind of sad bc it is such an awesome hw/sw systems challenge.
Ranked #9 in the TOP500 list of fastest supercomputers, Eos is the culmination of our ongoing commitment to pushing the boundaries of #AI technology and infrastructure. Learn more: #DataCenter #NVIDIADGX
6
3
76