📢📢📢𝐕𝐨𝐥𝐮𝐦𝐞𝐭𝐫𝐢𝐜 𝐑𝐞𝐧𝐝𝐞𝐫𝐢𝐧𝐠 𝐰𝐢𝐭𝐡 𝐁𝐚𝐤𝐞𝐝 𝐐𝐮𝐚𝐝𝐫𝐚𝐭𝐮𝐫𝐞 𝐅𝐢𝐞𝐥𝐝𝐬 at
#ECCV2024
Project led by Gopal Sharma (
@beakywings
)
TL;DR: extends MobileNerf to represent 𝐭𝐫𝐚𝐧𝐬𝐩𝐚𝐫𝐞𝐧𝐭 scenes... with meshes?🤯...🧵
📢📢📢 Thrilled to introduce "𝐌𝐨𝐛𝐢𝐥𝐞𝐍𝐞𝐑𝐅: exploiting the polygon rasterization pipeline for efficient neural field rendering on mobile architectures"
→ with 𝐥𝐢𝐯𝐞 𝐝𝐞𝐦𝐨𝐬
(Internship project lead by
@ZhiqinChen3
)
📢📢📢 Happy to introduce that "Kubric: a scalable dataset generator" was accepted at
#CVPR2022
.
– Do you work on video and/or 3D vision from images?
– Having a hard time applying the "scientific method" to your research?
Look no further 🧵👇
📢📢📢 𝗔𝘁𝘁𝗲𝗻𝘁𝗶𝗼𝗻 𝗕𝗲𝗮𝘁𝘀 𝗖𝗼𝗻𝗰𝗮𝘁𝗲𝗻𝗮𝘁𝗶𝗼𝗻 𝗳𝗼𝗿 𝗖𝗼𝗻𝗱𝗶𝘁𝗶𝗼𝗻𝗶𝗻𝗴 𝗡𝗲𝘂𝗿𝗮𝗹 𝗙𝗶𝗲𝗹𝗱𝘀
We ran A LOT of experiments to find the best way to make neural fields generalize... so you don’t have to!
📢📢📢 𝐒𝐮𝐧 𝐞𝐭 𝐚𝐥. "𝐂𝐚𝐧𝐨𝐧𝐢𝐜𝐚𝐥 𝐂𝐚𝐩𝐬𝐮𝐥𝐞𝐬: 𝐔𝐧𝐬𝐮𝐩𝐞𝐫𝐯𝐢𝐬𝐞𝐝 𝐂𝐚𝐩𝐬𝐮𝐥𝐞𝐬 𝐢𝐧 𝐂𝐚𝐧𝐨𝐧𝐢𝐜𝐚𝐥 𝐏𝐨𝐬𝐞"
A network design that realizes the concept of "𝐦𝐞𝐧𝐭𝐚𝐥 𝐩𝐢𝐜𝐭𝐮𝐫𝐞" for unsupervised 3D deep learning.
📢📢📢 𝐍𝐞𝐮𝐫𝐚𝐥 𝐒𝐞𝐦𝐚𝐧𝐭𝐢𝐜 𝐅𝐢𝐞𝐥𝐝𝐬 (NeSF) accepted to
#TMLR2022
–
In deep learning, we have tools to understand sequences, images, point clouds... we introduce the first field-to-field neural network design for scene understanding.
📢📢📢 Introducing "Vector Neurons"
Want a network (and latent space) that act by construction in an equivariant way w.r.t. SO(3) transformations?
All you need is to do is to generalize the scalar non-linearity to a vector one (e.g. Vector ReLU)
📢📢📢 Introducing 𝐍𝐞𝐮𝐫𝐚𝐥 𝐃𝐞𝐬𝐜𝐫𝐢𝐩𝐭𝐨𝐫 𝐅𝐢𝐞𝐥𝐝𝐬 (𝐍𝐃𝐅)
That's right, we 𝐭𝐞𝐚𝐜𝐡 𝐚 𝐫𝐨𝐛𝐨𝐭 to manipulate unseen objects, and unseen poses from 𝐣𝐮𝐬𝐭 𝟏𝟎 𝐞𝐱𝐚𝐦𝐩𝐥𝐞𝐬 🤯
Wanna know more? See this thread
Introducing “Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulation”!
(w/ video!)
NDFs are an object representation for robotic manipulation enabling imitation of pick-and-place tasks with pose generalization guarantees (1/n)
📢📢📢 thrilled to announce in Aug 2022 I will be 𝐣𝐨𝐢𝐧𝐢𝐧𝐠
@SFU
𝐚𝐬 𝐚𝐧 𝐀𝐬𝐬𝐨𝐜𝐢𝐚𝐭𝐞 𝐏𝐫𝐨𝐟𝐞𝐬𝐬𝐨𝐫 and visual computer research chair in
@SFU_CompSci
Looking for PhDs, applications due Jan 10th;
More info at →
Our paper won the
#CVPR2020
best student paper award!!!
Of course, many thanks to my amazing collaborators Zhiqin Chen, and Prof. Hao (Richard) Zhang.
p.s. this is the first paper I wrote with my PhD advisor after graduation :)
📢📢📢 thrilled to announce "𝟒𝐃-𝐟𝐲: 𝐓𝐞𝐱𝐭-𝐭𝐨-𝟒𝐃 𝐆𝐞𝐧𝐞𝐫𝐚𝐭𝐢𝐨𝐧 𝐔𝐬𝐢𝐧𝐠 𝐇𝐲𝐛𝐫𝐢𝐝 𝐒𝐜𝐨𝐫𝐞 𝐃𝐢𝐬𝐭𝐢𝐥𝐥𝐚𝐭𝐢𝐨𝐧 𝐒𝐚𝐦𝐩𝐥𝐢𝐧𝐠"
Way to start
@sherwinbahmani
🎉
PhD co-advised with
@DaveLindell
at
#UofT
.
Our new paper "𝐔𝐧𝐬𝐮𝐩𝐞𝐫𝐯𝐢𝐬𝐞𝐝 𝐩𝐚𝐫𝐭 𝐫𝐞𝐩𝐫𝐞𝐬𝐞𝐧𝐭𝐚𝐭𝐢𝐨𝐧 𝐛𝐲 𝐅𝐥𝐨𝐰 𝐂𝐚𝐩𝐬𝐮𝐥𝐞𝐬" is out
TL;DR: are newborns exposed to 14 million (i.e. ImageNet) labeled images? No! They learn by observing motion... in an unsupervised fashion
📢📢📢 𝐕𝐨𝐥𝐮𝐦𝐞 𝐑𝐞𝐧𝐝𝐞𝐫𝐢𝐧𝐠 𝐃𝐢𝐠𝐞𝐬𝐭 (𝐟𝐨𝐫 𝐍𝐞𝐑𝐅)
Tech-report:
Hope it's helpful, it's a self-contained step-by-step derivation of the volume rendering math (I needed to go carefully through it for another paper).
w/
@BenMildenhall
Do you like implicit functions to model 3D geometry?
We show that "Neural Articulated Shape Approximation" improves the approximation power by over 50% ABSOLUTE!!
Fast-forward of our
#ECCV
paper, full talk, additional material, and code comings soon.
📢📢📢 3D Gaussian Splatting brought you real-time rendering, but at slightly lower PSNR compared to mipNeRF360... 𝐚𝐬 𝐨𝐟 𝐭𝐨𝐝𝐚𝐲, 𝐭𝐡𝐚𝐭 𝐢𝐬 𝐧𝐨 𝐥𝐨𝐧𝐠𝐞𝐫 𝐭𝐫𝐮𝐞.
Introducing "3D Gaussian Splatting as
Markov Chain Monte Carlo"
I just published the cleaned-up official
#CVPR2022
latex template on
@overleaf
– might it save you some precious time :)
Best of luck to all your submissions! 🍀
📢📢📢 "Novel View Synthesis with Diffusion Models"
Single shot novel view synthesis results of quality never seen before. And, surprisingly, it does _not_ use NeRF to obtain a 3D consistent model, but rather 3D consistency is learnt during training.
📢📢📢 Have you ever wondered what "capsules" are in deep learning? For primary capsules, it is actually very very simple, and explained excellently by Weiwei in his upcoming talk at
#NeurIPS21
, linked below:
Thread 🧵⬇️
A
@CVPR
reviewer is asking a comparison against a black-box commercial package ($4k+) and I have nothing in the
#CVPR2022
policies to point out such ask is unacceptable!!! 😮😮
1) a dangerous path towards "pay-to-publish"?
2) for all I know, the reviewer could own the company?
Huge fan of this piece of research (i.e. wish it was mine). Time to bring geometry processing to the new era! And _plenty_ of ways to extend this paper...
"Geometry Processing with Neural Fields"
#CVPR2023
is too much about bold numbers. And it is WAY too easy to cheat on those (few keyboard strokes, right?)
We found two CVPR (orals) that mis-represented their results –– figuring out what to do in regards...
You WILL be found out.
Don't – just don't
If you have a friend that works at Google...
stop and give him/her a hug 🫂
It is "perf season", and generally a pretty painful moment of the year... especially for researchers.
📢📢📢 Today I'll be giving a talk at the
#CVPR2024
workshop on "Learning 3D with Multi-View Supervision"
See you at 16:20 in Summit 331
Get a preview of the upcoming
"3D Gaussian Splatting as Markov Chain Monte Carlo"
We can also jointly optimize geometry and appearance. In the example below, we determine the albedo and roughness of a Disney BSDF (shown under new viewing/illumination conditions). (6/8)
📢📢📢 𝐂𝐔𝐅 – 𝐂𝐨𝐧𝐭𝐢𝐧𝐮𝐨𝐮𝐬 𝐔𝐩𝐬𝐚𝐦𝐩𝐥𝐢𝐧𝐠 𝐅𝐢𝐥𝐭𝐞𝐫𝐬
Neural-fields beat classical CNNs in (regressive) super-res:
AFAIK, a first for
@neural_fields
in 2D deep learning?
Mostly their wins are in sparse, higher-dim signals ~NeRF
I truly don’t see the point in shaming a speaker for giving a bad talk… in front of 2k people during the Q&A session… why not in private and with constructive feedback?
Great reaction by the speaker.
(And the talk really wasn’t that bad… seen MUCH MUCH worse this week)
Considering a PhD? Interested in 3D deep learning, neural scene representations, NeRFs, and applications to robotics and graphics?
Come to Vancouver, 𝐒𝐢𝐦𝐨𝐧 𝐅𝐫𝐚𝐬𝐞𝐫 𝐔𝐧𝐢𝐯𝐞𝐫𝐬𝐢𝐭𝐲 (SFU) Deadline is 𝐉𝐚𝐧𝐮𝐚𝐫𝐲 𝟏𝟖 𝟐𝟎𝟐𝟑
Apply here:
📢📢📢 4DiM: Controlling Space and Time with Diffusion Models
This is NOT a video model
This is NOT a radiance field
This is a model that learns from space-time data (what your phone captures), and can generate “slices” of space-time.
📢📢📢 Introducing "𝐒𝐩𝐨𝐭𝐋𝐞𝐬𝐬𝐒𝐩𝐥𝐚𝐭𝐬: Ignoring Distractors in 3D Gaussian Splatting" lead by
@sabour_sara
and
@lily_goli
TL;DR: exploit pre-trained features to "recognize" what should be ignored.
Source code will be released in a few days.
📢📢📢 Thrilled to announce that nerf2nerf was accepted to ICRA 2023 – Huge congrats to
@lily_goli
on publishing her first PhD project (in her first year!)
ReLU Fields: The Little Non-linearity That Could
@AnimeshKarnewar
, Tobias Ritschel, Oliver Wang, Niloy J. Mitra
tl;dr: grid-based 3D representation + ReLU are almost NERFs, but much faster.
TL;DR: COLMAP completely fails in few shot settings, while neural bundle adjustment (SparsePose) regresses quite accurate camera poses.
Congratulations
@_sam_sinha_
on a flawless execution!
See you at
#CVPR2023
📢📢📢 𝐀𝐜𝐜𝐞𝐥𝐞𝐫𝐚𝐭𝐢𝐧𝐠 𝐍𝐞𝐮𝐫𝐚𝐥 𝐅𝐢𝐞𝐥𝐝 𝐓𝐫𝐚𝐢𝐧𝐢𝐧𝐠 𝐯𝐢𝐚 𝐒𝐨𝐟𝐭 𝐌𝐢𝐧𝐢𝐧𝐠 by Shakiba Kheradmand et al.
TL;DR: importance sampling for accelerating your novel view synthesis workloads (...yes, it should also work for 3DGS)
📢📢📢 CC3D: Layout-Conditioned Generation of Compositional 3D Scenes was accepted by
#ICCV2023
Congratulation
@sherwinbahmani
on your first AAA first-author paper, and very much looking forward to help you grow at
@UofT
with
@DaveLindell
We will be in
@siggraph
2023 with "3D Gaussian Splatting for Real-Time Radiance Field Rendering", have you ever seen radiance fields with 100+ FPS and MipNeRF360 quality?
Check out our website here:
Implicit functions meets kernel regression.
Excellent work by
@frncswllms
and
@NVIDIAAI
(had reviewed this work as part of Francis' PhD thesis at NYU):
Is the "era" of Poisson surface reconstruction officially over? MeshLab plugin
@ALoopingIcon
? :)
📢📢📢 𝐔𝐧𝐬𝐮𝐩𝐞𝐫𝐯𝐢𝐬𝐞𝐝 𝐊𝐞𝐲𝐩𝐨𝐢𝐧𝐭𝐬 𝐟𝐫𝐨𝐦 𝐏𝐫𝐞𝐭𝐫𝐚𝐢𝐧𝐞𝐝 𝐃𝐢𝐟𝐟𝐮𝐬𝐢𝐨𝐧 𝐌𝐨𝐝𝐞𝐥𝐬
TL;DR: semantic keypoints within a dataset emerge naturally from latent-diffusion model's text embeddings.
Wang et a. "SunStage: Portrait Reconstruction and Relighting using the Sun as a Light Stage"
Very cool work, i.e. using the sun as a light-probe (i.e. point light source in a light-stage) ... and yet another to scratch off the list of ideas :)
Rebain et al. "𝐃𝐞𝐜𝐨𝐦𝐩𝐨𝐬𝐞𝐝 𝐑𝐚𝐝𝐢𝐚𝐧𝐜𝐞 𝐅𝐢𝐞𝐥𝐝𝐬 (𝐃𝐞𝐑𝐅)"
That is, the render time of NeRFs can be improved (by +𝟑𝟎𝟎%!) by decomposing the scene representation into many lower-capacity networks.
Collaboration {UBC, Google, SFU}
Very neat paper to deal with equivariance in point clouds (
#CVPR2022
). Recommended read if you work on point clouds!
Core idea? Simple!
Learn a local coordinate frame by message passing.
@siggraph
@budmonde
@qisun0
"Instant Neural Graphics Primitives with a Multiresolution Hash Encoding", which can train neural graphics primitives in seconds and render them in milliseconds. 4/6
📢📢📢 Do you have verifiable research record in
#NeuralFields
/
#NeRF
and would like to come to
@GoogleAI
?
I collaborate with the SynthX team, that now have several Student Researcher positions available (eg. past project )
Apply → synthx-jobs
@google
.com
📢📢📢 𝐕𝐃𝟑𝐃: Taming Large Video Diffusion Transformers for 3D Camera Control
We extend video diffusion transformers with 3D camera control, enabling a new way towards 4D generation.
Project lead by
@sherwinbahmani
during his internship at
@Snap
Why is peer-reviews becoming more and more toxic? (i.e. trying to find negativity in an idea at all costs, and discount all but perfection).
The more toxic reviews you write, the more toxic reviews you are to expect for your own papers (i.e. what goes around, comes around).
📢📢📢 𝐁𝐀𝐍𝐅: 𝐁𝐚𝐧𝐝-𝐥𝐢𝐦𝐢𝐭𝐞𝐝 𝐍𝐞𝐮𝐫𝐚𝐥 𝐅𝐢𝐞𝐥𝐝𝐬 𝐟𝐨𝐫 𝐋𝐞𝐯𝐞𝐥𝐬 𝐨𝐟 𝐃𝐞𝐭𝐚𝐢𝐥 𝐑𝐞𝐜𝐨𝐧𝐬𝐭𝐫𝐮𝐜𝐭𝐢𝐨𝐧 lead by Ahan Shabanov (
@ahan_sh
) at
#CVPR2024
Wonder how to get frequency control in neural fields?
This is for you:
If you haven't had a change to read NeRFPlayer...
you should.
Been pondering whether this makes HyperNeRF _completely_ obsolete? Is there anything that HyperNeRF does better?
I don't think so, but feel free to argue otherwise!
Well done
@AutoVisionGroup
Suggestion for attending
#CVPR
2024.
Don’t hang out with the same people you hang out with every day, speaking a language that others can’t understand.
Go around! Mingle! Meet somebody new!
That’s what a conference is about.
Shouldn't this be fixed ASAP? This is 🤮 cringe-worthy 🤮for anybody with a signal processing background.
Also, this matters: "Impact of Aliasing on Generalization in Deep Convolutional Networks"
Our CVPR 2020 oral paper "Binary Space Partitioning Networks (BSP-Net)" has now source code available!
TL;DR: a neural network that generates low-polycount meshes
code → preprint →
Alright, name sadly scooped, although alas we moved away from fusion and towards semantics (i.e. NeSF).
... but, how well does it work off trajectory?
Anybody seen ScanNet results?
NeRFusion: Fusing Radiance Fields for Large-Scale Scene Reconstruction
The start date for NeurIPS2024 has been changed to Tuesday, December 10, allowing delegates to arrive on Monday for the Tuesday morning sessions. This adjustment is necessary due to limited hotel availability on Sunday night. The only corresponding change in the schedule is that
The more I AC/PC, the more I realize we need to show PhD students what happens during AC review consolidation.
This will contribute to significantly higher quality reviews, which will also make judging outcomes faster/easier.
📢📢📢 Excited to co-chair 3DV 2024 with Siyu Tang
@SiyuTang3
and Federico Tombari
@fedassa
Smaller conferences are where you create the networking that will last a lifetime.
📣 Exciting news! 📣
The call for papers is now open for 3DV2024, taking place in the stunning Davos, Switzerland!
📝 Submission: by July 31.
🔗 Details:
🌟 Don't miss the chance to share your research at
#3DV2024
🎉
#CallForPapers
#Davos
#Switzerland
Working on
#CVPR2023
? A simple trick to organize your floats. Store your figure data in the fig/ folder, and automate the rest of your life. Your filenames then become your \ref{...} keys
See snapshots👇👇
WARNING: controversial opinion incoming
it's BULLSHIT to put a blanket ban on social media ads, yet allow seniors present unpublished (to be submitted) work at major venues.
This only gives an unfair advantage to the big labs, w/ active senior members on the topic.
#ICCV
#CVPR
📢📢📢 𝐒𝐮𝐧 𝐞𝐭 𝐚𝐥. "𝐂𝐚𝐧𝐨𝐧𝐢𝐜𝐚𝐥 𝐂𝐚𝐩𝐬𝐮𝐥𝐞𝐬: 𝐔𝐧𝐬𝐮𝐩𝐞𝐫𝐯𝐢𝐬𝐞𝐝 𝐂𝐚𝐩𝐬𝐮𝐥𝐞𝐬 𝐢𝐧 𝐂𝐚𝐧𝐨𝐧𝐢𝐜𝐚𝐥 𝐏𝐨𝐬𝐞"
A network design that realizes the concept of "𝐦𝐞𝐧𝐭𝐚𝐥 𝐩𝐢𝐜𝐭𝐮𝐫𝐞" for unsupervised 3D deep learning.
Happy to share Gaussian Splatting SLAM
We show the first 3DGS-based Monocular RGB SLAM, the hardest SLAM setting.
Using 3D Gaussians as a unified representation, the method only requires RGB images - No need for SfM, depth sensor, or learned prior.
How did I miss this? 😀
Quite a change.
The CVPR’23 program will not have orals/spotlights. Only Best Paper Award nominees (~20-30 in total) will be given a plenary presentation at the conference.
"Attention Beats Concatenation" has been accepted (as is) at
#TMLR
– congratulations to Daniel (PhD student) for yet another successful project!
Now quite sure how to post it at
@TmlrPub
? ;)
)
📢📢📢 𝗔𝘁𝘁𝗲𝗻𝘁𝗶𝗼𝗻 𝗕𝗲𝗮𝘁𝘀 𝗖𝗼𝗻𝗰𝗮𝘁𝗲𝗻𝗮𝘁𝗶𝗼𝗻 𝗳𝗼𝗿 𝗖𝗼𝗻𝗱𝗶𝘁𝗶𝗼𝗻𝗶𝗻𝗴 𝗡𝗲𝘂𝗿𝗮𝗹 𝗙𝗶𝗲𝗹𝗱𝘀
We ran A LOT of experiments to find the best way to make neural fields generalize... so you don’t have to!
How to evaluate novelty fairly in a paper? 🧐
@CVPR
this might not be a bad article to broadcast to reviewers about to read rebuttals for
#CVPR2024
Thanks again to
@Michael_J_Black
for writing it.
📢📢📢 Urban Radiance Fields @
#CVPR2022
→ proof on how to integrate depth/lidar in NeRF
→ elegant modelling of exposure/balance
→ ... on Google StreetView data!
I am happy to announce that all our submissions to
#NeurIPS2020
were accepted! Special congratulations to the first (student!) authors Emre Aksan (ETHZ), Max Jiang (UCBerkeley), and Xiaogang Wang (SFU)
Fantastic work by
@akanazawa
's team
i.e. 32dB PSNR in 9minutes (vs. NeRF 1.6 days)
Angjoo... acronym way too long? It is not as friendly as NeRF {NeRFW, SNErG, DeRF, DNERF, ...}.
📢📢📢 Thrilled to announce that both our papers were accepted by
#NeurIPS2023
- Neural Fields with Hard Constraints of Arbitrary Differential Order ()
- Unsupervised Semantic Correspondence Using Stable Diffusion ()
How can you get a NeRF model that runs at interactive frame rate on most devices? (...CUDA is not an option)
The idea is to exploit the rasterization pipeline as much as possible, and employ (𝐧𝐞𝐮𝐫𝐚𝐥) 𝐝𝐞𝐟𝐞𝐫𝐫𝐞𝐝 𝐬𝐡𝐚𝐝𝐢𝐧𝐠.
.
@CVPR
many students from China won't be able to travel given the 30 days quarantine imposed upon re-entrance from the US.
Might be worth considering allowing for local events so that PhD students can (finally) have an in person experience?
Simply amazed that
@ICCV_2021
does not have 3D deep learning, geometric deep learning, 3D representations in their subject areas?
The closest are:
– 3D from a single image and shape-from-x
– Stereo, 3D from multi-view and other sensors
And they are barely appropriate...