Aleksander Holynski
@holynski_
Followers
1,683
Following
291
Media
35
Statuses
182
Explore trending content on Musk Viewer
jungkook
• 1162482 Tweets
برشلونة
• 222440 Tweets
بلد الوليد
• 200513 Tweets
Rice
• 171879 Tweets
Barca
• 166321 Tweets
Georgia
• 135910 Tweets
Brighton
• 109950 Tweets
Raphinha
• 102836 Tweets
Lamine
• 81679 Tweets
Everton
• 79168 Tweets
Hansi Flick
• 69651 Tweets
#WWEBash
• 61948 Tweets
Olmo
• 59806 Tweets
Haaland
• 57287 Tweets
#الاتحاد_الاحق_بالدعم1
• 41856 Tweets
Lewandowski
• 40775 Tweets
Bournemouth
• 31583 Tweets
McLaren
• 30188 Tweets
#RSAvNZL
• 28686 Tweets
Clemson
• 26644 Tweets
Iowa
• 25479 Tweets
Xavi
• 19841 Tweets
エネルギー補給
• 19361 Tweets
West Virginia
• 17569 Tweets
Penn State
• 17282 Tweets
Bianca
• 15186 Tweets
West Ham
• 14763 Tweets
All Blacks
• 13802 Tweets
Pinned Tweet
Videos are cool and all...but everything's more fun when it's interactive.
Check out our new project, ✨CAT3D✨, that turns anything (text, image, & more) into interactive 3D scenes!
Don't miss the demo!!
🌟 Create anything in 3D! 🌟 Introducing CAT3D: a new method that generates high-fidelity 3D scenes from any number of real or generated images in one minute, powered by multi-view diffusion models.
w/ lovely coauthors
@holynski_
,
@poolio
and an amazing team!
24
64
407
8
35
219
Check out our new paper that turns a (single image) => (interactive dynamic scene)!
I’ve had so much fun playing around with this demo.
Try it out yourself on the website:
Excited to share our work on Generative Image Dynamics!
We learn a generative image-space prior for scene dynamics, which can turn a still photo into a seamless looping video or let you interact with objects in the picture. Check out the interactive demo:
19
154
854
26
316
2K
Excited to show off our new project on single-image cinemagraphs. Our method automatically turns a _single image_ into a seamlessly looping video!
Website:
Video:
w/ Brian Curless, Steve Seitz, Rick Szeliski
More in thread! [1/5]
30
382
2K
Excited to share ReconFusion! 3D reconstruction of real-world scenes from only a few photos, powered by diffusion priors:
w/ amazing team
@ChrisWu6080
@BenMildenhall
@philipphenzler
@KeunhongP
@RuiqiGao
@watson_nn
@_pratul_
@dorverbin
@jon_barron
@poolio
9
60
355
And also some pretty cool failure cases -- if a class of objects aren't seen during training, but share similar textural properties to fluids...
[6/5]
15
40
294
We just posted a report on the state of the art in diffusion models for visual computing:
If you're new to diffusion models, or maybe just want a recap of everything that's been going on lately---this is a great place to start.
4
45
295
Happy to finally be able to share our
#CVPR2022
paper, InstructPix2Pix!
We taught a diffusion model how to follow image editing instructions — just say how you want to edit an image, and it’ll do it!
(w/ Tim Brooks & Alyosha Efros)
More on Tim’s site:
🧵
4
28
194
.
@QianqianWang5
's 🎉Best Student Paper🎉 is being presented at
#ICCV2023
tomorrow (Friday)!
▶️"Tracking Everything Everywhere All At Once"◀️ w/ Yen-Yu Chang,
@ruojin8
@zhengqi_li
@BharathHarihar3
@Jimantha
Friday Afternoon Oral & Poster! Come say hi!
1
21
170
We posted an updated version of Generative Image Dynamics to arXiv---the biggest change is to better contextualize our method with respect to prior work in image space motion analysis, especially the great work of
@AbeDavis
Check out our new paper that turns a (single image) => (interactive dynamic scene)!
I’ve had so much fun playing around with this demo.
Try it out yourself on the website:
26
316
2K
4
21
149
I'll be presenting CAT3D tomorrow at CVPR. Come say hi!
Monday 2:30pm at AI for 3D Generation (Summit Flex A)
🌟 Create anything in 3D! 🌟 Introducing CAT3D: a new method that generates high-fidelity 3D scenes from any number of real or generated images in one minute, powered by multi-view diffusion models.
w/ lovely coauthors
@holynski_
,
@poolio
and an amazing team!
24
64
407
3
12
115
some more fun CAT3D results ✨
tons more in the gallery:
5
10
92
It turns out images contain lots of useful cues about how things should be flowing -- like ripples in water, turbulent streams, motion blur. An image-to-image GAN learns a lot of these subtle cues, and can synthesize pretty complex motion.
[3/5]
Here's another result:
5
5
80
Check out
@xiaojuan_wang7
's new project!
🔎Generative Powers of Ten🔍
Use a pre-trained text-to-image model to generate deeeeep zoom videos!
(Excuse Twitter's terrible compression, check the webpage instead: )
Excited to share our work Generative Powers of Ten w/
@holynski_
@_pratul_
@BenMildenhall
@dorverbin
@kemelmi
Given a set of prompts describing a scene at varying zoom levels, our method creates a seamless zooming video. Check it out here:
4
16
110
3
6
73
We focus on fluids (flowing water, billowing smoke, clouds), i.e., things well approximated by particle motion. So, instead of predicting a sequence of flow fields for a video, we can predict a single Eulerian motion field (a particle velocity field).
[2/5]
2
6
70
Wow!! I've been a big fan of
@twominutepapers
for the longest time...it's such an incredible honor to have our paper featured.
3
5
57
To generate the video frames, we use a deep warping technique (encode-warp-decode). Since warping a single image usually leads to big holes, we use a novel symmetric splatting approach, which combines features from different points in time to produce more realistic images.
[4/5]
2
4
53
Thanks for the tweet!
Check out our project page:
Google presents CAT3D
Create Anything in 3D with Multi-View Diffusion Models
Advances in 3D reconstruction have enabled high-quality 3D capture, but require a user to collect hundreds to thousands of images to create a 3D scene. We present CAT3D, a method for creating
9
125
557
2
9
52
We've tried our method on a large collection of images, and found it to be surprisingly robust on a pretty wide variety of scenes!
[5/5]
1
1
49
Super neat! An interactive diffusion-based Photoshop.
A great example of how the right interfaces and controls can make a massive difference in the utility of these generative models.
1
7
40
🔮Readout Guidance🔮 is a neat way of controlling diffusion models (in pretty complex ways!)
See the site () for applications and interactive galleries.
Here's one favorite: where we guide the identity in a generated image to match a reference image.
Guidance on top of diffusion models can now be used to drag and manipulate images, create pose-conditioned images, and so much more! Check out Readout Guidance:
Work w/
@trevordarrell
,
@oliver_wang2
,
@danbgoldman
,
@holynski_
. More in thread 🧵.
5
48
240
1
6
41
We're hosting a CVPR workshop on AI-assisted art---a big focus is to understand how AI models are currently being used in artistic workflows (to help inspire the next generation of better, more useful AI tools).
We are excited to share that the AI for Creative Visual Content Generation Editing and Understanding Workshop
@cveu_workshop
has been accepted to
#CVPR2024
@CVPRConf
. See you in Seattle to meet art, tech, and creativity! It is the first time we come to the US🇺🇸
2
10
65
1
5
34
I'll be talking about our paper "Animating Pictures with Eulerian Motion Fields" this evening at Paper Session
#5
(10pm-12a ET, 7pm-9pm PT).
Come say hi!
Excited to show off our new project on single-image cinemagraphs. Our method automatically turns a _single image_ into a seamlessly looping video!
Website:
Video:
w/ Brian Curless, Steve Seitz, Rick Szeliski
More in thread! [1/5]
30
382
2K
0
2
33
Come hang out at this
#CVPR2024
workshop we're organizing!
Learn from researchers & artists about new creative applications, open technical challenges, & more.
The event is in-person only---no recording, no streaming! Don't miss out!
@CVPR
I'm co-organizing a CVPR workshop next Tuesday that is absolutely stacked with talent. If you're interested in anything related to art or generative video (eg Sora, Veo, Pika, Runway), be there.
8
9
123
0
4
29
Congrats to
@zhengqi_li
,
@Jimantha
, & Richard!!!
Congratulations to
@zhengqi_li
, Richard Tucker,
@Jimantha
, and
@holynski_
. Their paper “Generative Image Dynamics” received the
#CVPR2024
Best Paper Award. Read the paper:
8
28
178
0
0
26
Darn -- looks like twitter's encoding messed with the looping. Check the website for the full-quality results:
2
1
23
Very excited to mess around with this.
Our latest work on making Consistent Video Depth more ROBUST. Works great for casual phone videos that are really difficult for previous methods.
Another great collaboration with
@jastarex
and
@jbhuang0604
.
arXiv:
Project:
4
54
279
0
0
21
Happy to finally be able to share our
#CVPR2022
paper, InstructPix2Pix!
We taught a diffusion model how to follow image editing instructions — just say how you want to edit an image, and it’ll do it!
(w/ Tim Brooks & Alyosha Efros)
More on Tim’s site:
🧵
4
28
194
1
0
18
Seeing the world in a potato!
Introducing Eclipse, a method for recovering lighting and materials even from diffuse objects!
The key idea is that standard "NeRF-like" data has all we need: a photographer moving around a scene to capture it causes "accidental" lighting variations. (1/3)
5
34
328
1
0
12
check out dave's project!
automatically decomposes complex 3D scenes into individual objects (without relying on per-object text descriptions or annotations!)
a neat central insight: think of objects as "parts of a scene that can be moved around independently"
text-to-3d scenes that are automatically decomposed into the objects they contain, using only an image diffusion model & no other supervision:
work w/
@poolio
@BenMildenhall
Alyosha Efros and
@holynski_
3
21
162
1
0
12
Come check out our paper at 3DV today!
(6a PST oral / 8:30a PST poster)
We use vanishing points and planes to get rid of pose drift in SfM.
"Reducing Drift in Structure from Motion Using Extended Features"
Project page:
Video:
1
8
12
For those wondering, yes, we did try it on images from the original Powers of Ten 🙃
0
1
12
@jbhuang0604
I can't get enough of these advice threads. This needs to be a class!!
PHD101 "How to be a graphics+vision researcher", with Prof. Huang
0
1
10
Wonderfully trippy results!
View synthesis is super cool! How can we push it further to generate the world *far* beyond the edges of an image? We present Infinite Nature, a method that combines image synthesis and 3D to generate long videos of natural scenes from a single image.
19
458
2K
0
1
9
We trained the model on a massive dataset of generated editing examples, with triplets containing:
1. input image
2. text editing instruction
3. output image
How does one generate a dataset like this, you might ask?
1
0
8
@JPKopf
Sweet! If you can get it to loop, this could be like Casual3D + Panoramic Video Textures ()
1
0
9
If you’re interested in this stuff, I’d highly recommend reading Abe’s thesis, which includes a thorough and beautiful theory about the underlying frequency-space motion representation and how it connects to modeling object dynamics.
0
0
8
Check out more cool results on our website!
1
1
8
Diffusion models let you create amazing images given the right prompt.
But some things are hard to express in text, like where objects should go or exactly how big they should be.
How can we get this kind of control?
1
0
8
Self-guidance offers a new way to control the generation process:
Without any extra models or training, we can extract properties like object shape, size, and appearance from internal attention maps + activations.
We can then guide these properties to edit generated images.
1
1
8
Finally, fine-tune a text-to-image diffusion model to learn this transformation, conditioned on the input image and the instruction!
Here's one of our favorites, but you can try it for yourself with the demo on the website:
1
0
7
Robust Consistent Video Depth Estimation
@JPKopf
,
@jastarex
,
@jbhuang0604
Jointly estimates camera pose & dense depth for challenging video captures of dynamic scenes
0
1
7
Everything's better in 3D!
I was blown away by the incredible results of animating fluid motion from a single image last week.
I thought it would be fun to add a bit of 3D. Here are some results using 3D photo inpainting
3
31
221
0
1
7
I'm moderating the second session today---stop by!
I'm co-organizing a CVPR workshop next Tuesday that is absolutely stacked with talent. If you're interested in anything related to art or generative video (eg Sora, Veo, Pika, Runway), be there.
8
9
123
0
0
7
Follow-up idea: a network that makes me look good in pictures
We present nerfies! We use selfie videos to create 3d free-viewpoint portrait visualizations of yourself using Deformable NeRFs! More details at and below (1/8)
9
126
751
0
0
7
Self-guidance also works on real images, which allows you to "borrow" real objects and stick them in new contexts, sort of like a zero-shot DreamBooth.
1
0
7
No more chopped-off fingers!
Excited to release v2.0 of our Background Matting project, which is now REAL-TIME & BETTER quality: 60fps at FHD and 30fps at 4K! You can use this with Zoom, check out our demo! 👇
Webpage:
Video:
More in the thread! [1/6]
22
162
996
0
0
6
By combining a large language model (GPT-3) and a text-to-image model (Stable Diffusion)!
First, fine-tune the LLM on a small collection of human-written examples, and use it to generate a dataset of text triplets:
1. image caption
2. edit instruction
3. caption after the edit
1
0
6
For example, you can use self-guidance to move or resize an object (like this donut) — or even replace it with an item from a real image — all without changing the rest of the scene.
1
0
6
We can use self-guidance to edit entire images at once, too. For example, we can copy the appearance or inherit the layout of another scene — basically re-styling or re-composing any image!
2
0
6
@JPKopf
But I'm curious what the best way is to handle transient objects, like the kids in most of the RCVD videos.
Maybe something like action shots ()
Create a couple instances of the person, so there's always something interesting in every direction
1
0
6
Learning To Recover 3D Scene Shape From a Single Image
Wei Yin,
@oliverwang81
@simon_niklaus
& co
Uses learned priors to estimate the unknown focal and shift required for unprojecting monocular depth estimates to structurally-correct point clouds
1
0
6
Very cool! Check out the demos on the webpage.
We are happy to finally share our (David Griffiths and Tobias Ritschel) latest Relighting paper,
OutCast: Outdoor Single-image Relighting with Cast Shadows. To be presented next week at Eurographics 2022.
Paper, results and more at: . (1/4)
9
80
365
0
0
4
Then, use the input and output caption to generate a pair of corresponding images, using Prompt-to-Prompt:
1
0
4
@ericmchen1
Sure, here are a couple from a cool sculpture I saw at the MoMA.
The samples aren't 100% consistent, but definitely within the ballpark of what a well-tuned NeRF pipeline can handle (just as it might handle inconsistencies in real captures, eg minor dynamics & lighting changes)
1
0
3
Real-time NeRF in your browser -- even works on a laptop, no hefty GPUs needed!
Rendering a NeRF is slow, so we found a way to "bake" NeRFs into something more GPU-friendly. The result: real time NeRF rendering in your browser. Try it now before the internet hugs our server to death.
@PeterHedman3
@_pratul_
@BenMildenhall
@debfx
5
36
194
1
1
4
Abe’s work on Image-Space Modal Bases was an important inspiration for our paper, and the new version of the paper better reflects this —connecting some of the terms we used in our original draft with those used in prior work
1
0
4
Super cool stuff from
@yoni_kasten
@dolevofri
@oliverwang81
@talidekel
Decomposes a video into two canonical templates/atlases (represented by MLPs) that can be edited --- allowing edits to propagate through a whole video. Sort of like MLP Unwrap Mosaics. Awesome results!
Excited to share our work "Layered Neural Atlases for Consistent Video Editing".
Paper:
Project page:
(with
@dolevofri
,
@oliverwang81
and
@talidekel
, to appear in SIGGRAPH Asia'21) (1/9)
3
4
42
0
0
4
Our work uses these insights about converting dense 2D motion trajectories to the frequency domain, and shows that this spectral volume representation is also an efficient and effective one for __generating__ long-term motion from a single, still image.
1
0
3
This is a big collaboration with a wonderful team of experts in the field:
@Po_lhr
@toomanyyifans
@VGolyanik
@AbermanKfir
@jon_barron
@bermano_h
@ericryanchan
@talidekel
@akanazawa
@LingjieLiu1
@BenMildenhall
@MattNiessner
@peter_wonka
@GordonWetzstein
(& others off Twitter!)
0
0
3
@jbhuang0604
Wow, this is awesome! This was exactly what was missing. When I was collecting the training data, I was specifically looking for stationary shots, but this made things difficult, since the vast majority of nature videos online have this kind of slow pan.
2
0
3
Among the insights in Abe’s work is that the motion trajectories observed in a 2D video of an object exhibiting oscillations/vibrations, when converted to the Fourier domain, are — under certain assumptions — a projection of the 3D vibration modes of that object.
1
0
3
Challenge accepted! Here are a couple of my first-day favorites:
I dare a challenge for
#cvpr2021
twitter: for each paper of yours that you advertise on twitter, please share 1-3 interesting papers from other teams. Authors will appreciate it and the community will be stronger. Here are some champions:
@ducha_aiki
@CSProfKGD
@artsiom_s
3
4
60
1
0
3
Pulsar: Efficient Sphere-Based Neural Rendering
@chlassner
,
@MZollhoefer
Super cool visualizations of the optimization process -- definitely check out the video.
1
0
3
@jonathanfly
@jbhuang0604
@Feldman1Michael
Haven't tried it on pixel art yet, but I'll try that soon. I posted one example of non-nature in the original thread:
And also some pretty cool failure cases -- if a class of objects aren't seen during training, but share similar textural properties to fluids...
[6/5]
15
40
294
1
0
3
Modern photos of the past!
What did Abraham Lincoln really look like? Our new project simulates traveling back in time with a modern camera to rephotograph famous subjects.
Web:
Video:
w/
@ceciliazhang77
,
@RealPaulYoo
,
@rmbrualla
,
@jlawrence176
, Steve Seitz
5
63
248
0
0
3
@jonathanfly
Yeah, default parameters. The main training Colab will periodically save a checkpoint model, and you have to use a second Colab () to render the video given one of those checkpoints. I think I used the first or second checkpoint.
0
0
2
What a cool idea! Excited to see more complex totems like 3D printed refractive structures with unknown internal geometry / material.
Tomorrow at ECCV, we are presenting “Totems: Physical Objects for Verifying Visual Integrity”
Remember totems from Inception? We tried to make something *a bit* like that in reality.
website:
paper:
1/n
3
18
110
1
0
2
@JPKopf
Haha, yeah! It's improved a lot. You can catch some of our older results in the ablations ;-)
0
0
1
@jbhuang0604
So I've been hoping to find a way to do something like this, to make the shots look more like the type of stuff you'd find online -- you beat me to it! Thanks, I'll have to try it out myself :-)
1
0
1