I defended my PhD thesis last week 🥳 Thank you to everyone that made this possible, including my advisor
@mapo1
, examiners
@Jimantha
@quantombone
and Daniel Cremers, and the amazing
@cvg_ethz
. As per the tradition, I received a nice commemorative hat 🎓 Now time for vacations 😎
📢 Code release: a new toolbox for visual localization
➡️ try out SuperGlue for localization - CVPR'20 winner
➡️ run it with SfM on your own dataset within minutes
➡️ evaluate your local features/retrieval
➡️ take part in ECCV'20 challenges: 3 weeks to go!
Next week at
#CVPR2020
: our oral paper "SuperGlue: Learning Feature Matching with Graph Neural Networks" with
@ddetone
,
@quantombone
, A Rabinovich
Website:
Paper:
Video:
Winner of several CVPR competitions 👇🏻
You liked SuperGlue? You'll love ⚡️LightGlue⚡️, our new deep network for light-speed image matching!
➡️Faster, stronger, easier to train than SuperGlue
➡️Code:
➡️Paper:
Fantastic work by
@PhilippCSE
for
#ICCV2023
, with
@mapo1
1/
📢 Introducing LaMAR: a benchmark for Localization and Mapping with AR devices
Website:
Video:
➡️ Join our LaMAR tutorial at
#ECCV2022
on Monday 24/10 - intro by Marc Pollefeys
@mapo1
⬇️ hours of AR data, 3 locations, over 2 years
📢 We just released the code for OrienterNet, a deep net that learns to localize in 2D maps 🗺 like OpenStreetMap, just like humans do! Join us today at
#CVPR2023
poster PM-98.
Video:
Code:
Try our demo: take a picture and localize!
What makes a good
@CVPR
poster?
➡️ lots of visuals to guide your presentation
➡️ thick paper: serves as comfy nap mat 😴
➡️ glossy paper: serves as easy-to-clean tablecloth 🍽
➡️ little text: serves as bright windshield sunshade ☀️ 🚗
Tested & 💯 approved for camping in Canada!
Seeing so many empty posters & missing authors at
#CVPR2023
is heartbreaking - how many? 20%? Many PhD students worked hard but this absurd visa system jeopardized their chance to proudly present their work. I know that PCs
@CVPR
took action but this was largely insufficient… 1/
New release of our toolbox hloc: run 3D reconstruction and visual localization with COLMAP in Google Colab!
➡️ SfM, painless, with GPUs, for free
➡️ GitHub
➡️ Colab
Thanks
@PhilippCSE
@j___lambert
@mihaidusmanu
for the awesome work!
For
#NeurIPS2023
we're presenting our latest work SNAP - Self-Supervised Neural Maps for Visual Positioning and Semantic Understanding
Geo data at scale + self-supervision = 💥
Video:
Paper:
Code:
summary🧵⬇️
Yesterday the
#ICCV2023
workshop Quo Vadis Computer Vision was full of takes so hot I needed to let them cool down for a while not to burn myself - my favorites ⬇️
> We have a fanatic adherence to experiment, mostly hilariously poorly conducted e.g. no error bars – Forsyth
1/
📢Today we release the
@ETH
-
@Microsoft
visual localization dataset as part of the visloc
#ICCV2021
challenge! Focus on AR in changing conditions: in/outdoor, day/night,
@HoloLens
& phones
➡️
Demo with hloc & SuperGlue: setup in 5 minutes or your money back!
🚨 This week at
#ICCV2021
, check out "Pixel-Perfect Structure-from-Motion with Featuremetric Refinement" w/
@PhilippCSE
V. Larsson
@mapo1
➡️ oral & *best student paper award*
Website:
Paper:
Video:
thread ⬇️
I won the best presentation award at
#ICVSS2022
🥳 Many thanks to the school committee and to all attendees for voting! I very much enjoyed presenting and chatting in-person after 2 years of virtual posters sessions 😊
We have released COLMAP implementations of recent papers that make Structure-from-Motion robust to symmetries
Surprise: none of these SOTA algorithms works consistently across datasets with the same hyperparams. More work needed for this difficult problem!
Yesterday at
#ICVSS2022
: incredible talk by
@vincesitzmann
on neural scene representations, organizing many recent works into a unified taxonomy with great clarity.
Personal update: this week I joined the Microsoft MR&AI Zurich lab as a research intern :) I am excited to work on cool 3D geometry+learning problems with
@mapo1
and other inspiring researchers!
📢Join us tomorrow Friday morning at our
#ICCV2023
poster S-6 to learn more about LightGlue!
💥We have released the training code at
➡️You can now train LightGlue with SIFT, ALIKED, or your own local features, on your own dataset
⏩More models coming soon!
You liked SuperGlue? You'll love ⚡️LightGlue⚡️, our new deep network for light-speed image matching!
➡️Faster, stronger, easier to train than SuperGlue
➡️Code:
➡️Paper:
Fantastic work by
@PhilippCSE
for
#ICCV2023
, with
@mapo1
1/
Very cool work by Frank Fu &
@MauriceFallon
extending Back to the Feature to Camera-Lidar calibration. Learned image-lidar representations + differentiable geometry optimization FTW!
As expected, great generalization compared to black-box deep nets.
📢 NEW PAPER: Self-Supervised Camera/Lidar Calibration.
Appearing at
@corl_conf
'23: We've developed a self-supervised method for camera/LiDAR correspondence learning unlocking long-term sensor fusion in the wild.
We have now released the full raw data of LaMAR, which include 3D lidar point clouds & meshes, on-device ToF depth maps, IMU data, GPS signals, etc. Huge potential to benchmark and train models for different geometric vision tasks!
Check out to gain access
📢 Introducing LaMAR: a benchmark for Localization and Mapping with AR devices
Website:
Video:
➡️ Join our LaMAR tutorial at
#ECCV2022
on Monday 24/10 - intro by Marc Pollefeys
@mapo1
⬇️ hours of AR data, 3 locations, over 2 years
Really sad to miss
#CVPR2024
and its fantastic papers & workshops! But Puglia/Italy is treating me well in return 😊 I hope to catch up with everyone at ECCV in Milan - enjoy Seattle!
We have finally released the code for our
#ICCV2021
Pixel-Perfect Structure-from-Motion 🥳 Easy to use with COLMAP and hloc, more accurate/optimized/scalable than our original research code, can fit many use cases. Kudos to
@PhilippCSE
for the nice work
➡️
🚨 This week at
#ICCV2021
, check out "Pixel-Perfect Structure-from-Motion with Featuremetric Refinement" w/
@PhilippCSE
V. Larsson
@mapo1
➡️ oral & *best student paper award*
Website:
Paper:
Video:
thread ⬇️
Quick update: we have finally released the code for our CVPR 2021 paper Back to the Feature: It includes the evaluation & training as well as a 3D viewer, a dataset downloader, etc. ↘️
Yesterday was my last day interning at Facebook/Meta
@RealityLabs
. This was an incredible experience: I had fun working on cool problems with amazing people, from which I learned so much! Now I can't wait to soon tell you all about what I worked on 🔜💥
Paul-Edouard Sarlin is awarded the Graduate Award 2020 of the SEW-Eurodrive Foundation for his Master's thesis on "SuperGlue: Learning Feature Matching with Graph Neural Networks", supervised by Professor Roland Siegwart. Congratulations!
@pesarlin
On July 13th, I'll give a talk on how deep learning can supercharge 3D reconstruction and world mapping. I feel very lucky to present next to stellar world-class teams from Meta, Google, Microsoft, Pix4D. Join us if you're in Zurich!
Join us on July 13th at Meta Zurich for the 3rd Computer Vision Zurich Meetup on "Building a Digital World". Speakers from academia & industry will discuss about 3D reconstruction & mapping.
Seats are limited!
w/:
@FedericaBogo
,Margarita Grinvald,
@fedassa
I am excited to share that I joined
@mapo1
Prof Marc Pollefeys' CVG group at ETH as a PhD student! The next years will be a lot of fun & challenges, working on 3D vision & learning, with awesome colleagues & in the great city of Zurich 😁🇨🇭
I will give a total of 4 talks during
#CVPR2020
:
- @ 3D Scene Understanding workshop Monday
- SuperGlue for localization 2x talks Monday
- SuperGlue for two-view+SfM Friday 11:45a PST
+ live QA Wed 10:40a PST
Next week at
#CVPR2020
: our oral paper "SuperGlue: Learning Feature Matching with Graph Neural Networks" with
@ddetone
,
@quantombone
, A Rabinovich
Website:
Paper:
Video:
Winner of several CVPR competitions 👇🏻
My hloc toolbox with SuperGlue won 4 visual localization challenges at
#ECCV20
🔥 I will give two talks next week at two workshops
➡️for Autonomous Driving, Sun 23rd
➡️for Long-Term Changing Conditions, Fri 28th
See you "there" 🤩
📢 Code release: a new toolbox for visual localization
➡️ try out SuperGlue for localization - CVPR'20 winner
➡️ run it with SfM on your own dataset within minutes
➡️ evaluate your local features/retrieval
➡️ take part in ECCV'20 challenges: 3 weeks to go!
Our hierarchical localization paper won the visual localization challenge at
#CVPR2019
😀 Come to my talk at the visual localization workshop on Monday 2:15PM, and to our poster 3.2/213 on Thursday!
Paper:
@CVPR2019
@ETH_en
@ASL_ETHZ
@7Srobotics
What's the CO2 footprint of a conference like
@CVPR
? Including 1) the energy used by all GPUs for all experiments of all papers, and 2) flights for all in-person attendees. Has anyone ever looked into this? and for other academic conferences?
Just landed in Sicily to attend
#ICVSS
next week! This is my first in-person research event since 2019 - super excited to finally meet the real faces behind the names, make new friends, and learn from everyone! Let's meet if you're attending too 😀
Today I presented my Master thesis "SuperGlue: Learning Feature Matching with Graph Neural Networks"
done with
@ddetone
@quantombone
@ASL_ETHZ
@ETH_en
This marks the end of my time at Magic Leap - it was an amazing experience! 😁 Stay tuned for what's next
Are you attending
#CVPR2023
@CVPR
and interested in image matching? Then you should attend the Image Matching Workshop tomorrow morning! Bonus: team RMD-3DV, 8th place at the Kaggle challenge, will preview a pretty cool work in their talk at 12:15 😉
SuperGlue won 2 visual localization challenges on local features & indoor/outdoor loc ➡️ check out the workshop
It is also ranks 1st in the image matching challenge of
@ducha_aiki
@kwangmoo_yi
@FuaPv
by a large margin ➡️ leaderboard
#ECCV2022
was so fun! Great discussions, delicious food, wild parties, and many new friends. Finally got to meet the humans behind the names & papers - they’re indeed all real 😀
Our benchmark paper 𝗙𝗶𝘀𝗵𝘆𝘀𝗰𝗮𝗽𝗲𝘀 got accepted as an oral to the
#ICCV19
workshop on Autonomous Driving! 🙂 Join on Monday if you're interested in out-of-distribution detection for semantic segmentation!
Paper:
Benchmark:
"We must take the risks of AI as seriously as climate change. We can’t afford the same delay with AI."
This planet will be hell long before AI becomes smart enough to be an existential risk: ~30% of the population lives in areas unliveable by 2070. AI risk is a mere distraction.
tldr: PixLoc = end-to-end camera localization by learning deep features & uncertainties 😎 The code will be released in the next days! In the meantime:
➡️ Join our poster: Tues 6:8.30am CET
➡️ Play with our interactive viewer at
More viz coming soon 🤩
This will have a long-lasting impact on their career. Attending virtually is just pretty bad for networking. How many lost the chance to meet their future PhD/postdoc advisor? Start fruitful collaborations? Learn this new unwritten trick at poster xyz? Please never again. 2/2
📢 Introducing LaMAR: a benchmark for Localization and Mapping with AR devices
Website:
Video:
➡️ Join our LaMAR tutorial at
#ECCV2022
on Monday 24/10 - intro by Marc Pollefeys
@mapo1
⬇️ hours of AR data, 3 locations, over 2 years
LightGlue is already in
@kornia_foss
thanks to
@ducha_aiki
! It is also much easier to train - only 5 GPU-days. We hope that this will make it much easier to do research on image matching. We will soon release the full training code under Apache 2.0 license - STAY TUNED ⏩
3/3
#ICCV2021
nominated me as outstanding reviewer 🥳I enjoyed the time spent reviewing & learned a lot from the discussions with the ACs and other reviewers - thank you very much! It's a great way to learn about common issues in submissions and improve one's paper writing!
[PCs Update] We acknowledge 210 outstanding reviewers (top 5% experienced and top 5% student reviewers) online at:
amongst the many amazing reviewers this year.
We also ack. generous emergency reviewers:
@valzadrija
@Stephen_Hausler
@SattlerTorsten
Good question! ResNet has higher capacity + memory-efficiency but features not well-localized vs VGG because: 1) aggressive downsampling by 1st conv, 2) downsampling with strided conv instead of max-pool. This is fixable but with some network surgery, so VGG is a simpler choice
This. By writing many weak papers, everyone in our community is loosing out:
1) it overburdens the review system - reviewers handle too many papers and thus don’t have time to give useful feedback
2) it adds noise to arxiv/conferences and makes it easier to miss great papers
⬇️
This paper is a great example of deliberate misleading evaluation of baselines. Why? ⬇️
1/ To localize with local features we need a 3D sparse map. How it's done in the paper: lift 2D keypoints to 3D using depth from OpenCV stereo block matcher. Not joking.
This is a great read! I've also been thinking about this for a while: books (and the internet) are a summary of human knowledge; language models, even if mere approximations of it, encode statistics of the world useful to solve many vision and robotics tasks. thoughts ⬇️ 1/n
Here is the sequel to "Just ask for Generalization" - in this blog post I argue that Generalization *is* Language, and suggest how we might be able to re-use Language Models as "generalization modules" for non-NLP domains. Check it out!
My take: folks working on AI know that any AI existential risk is decades away and that we're not much closer to it now than pre-deeplearning. Those that signed the statement either seek attention (=funding) or want to block competitors from entering the juicy chatbot market. 1/
@pesarlin
Do you see a value in engaging in these discussions? Myself I do not participate but sometimes I'm worried these folks will end up overregulating the field and causing damage that could be avoided if I (we) spoke out more loudly. What's your take?
@jbhuang0604
Alternatively: because similar papers are grouped in the same poster session? Authors that are interested in one paper are busy presenting their own poster at the same time! This time conflict indeed made me miss several great papers today.
I’m attending
#ICCV2033
this week in Paris 🇫🇷 Looking forward to meeting old and new friends - please reach out if you have time for a chat! This year I’m super excited about the great line up of workshop speakers 🤩
@ICCVConference
Glad to be part of the
#CVPR2021
outstanding reviewers 🥳🥳 Thanks to the ACs for the support and congrats to everyone for your hard work! I really enjoyed the process and learned a lot regardless of the papers quality. Now off to
#ICCV2021
...
My thread yesterday could be seen as offensive and sparked concerns among many. Let me rephrase ⬇️
1/ My goal was not to publicly attack the authors of this paper, for which I have immense respect and sympathy, but instead to bring awareness on an incorrect evaluation protocol.
This paper is a great example of deliberate misleading evaluation of baselines. Why? ⬇️
1/ To localize with local features we need a 3D sparse map. How it's done in the paper: lift 2D keypoints to 3D using depth from OpenCV stereo block matcher. Not joking.
Yesterday the
#ICCV2023
workshop Quo Vadis Computer Vision was full of takes so hot I needed to let them cool down for a while not to burn myself - my favorites ⬇️
> We have a fanatic adherence to experiment, mostly hilariously poorly conducted e.g. no error bars – Forsyth
1/
🏎💨LightGlue is fast thanks to a mechanism that reduces the depth (# of layers) and width (# of points) *adaptively* for each input & layer. This means that easy image pairs are matched much faster than difficult ones - great for applications like SLAM & Structure-from-Motion
2/
@ducha_aiki
@wnlckwd
I disagree with "ML research is not about visualizations". Doing research requires understanding: we cannot explain ideas to others without first understanding them fully. ML theory alone is often not enough so visualizations are critical - good scientists need these skills.
PixLoc learns which features and objects matter for robust, long-term localization - a small step towards spatiotemporal understanding of appearance changes for human-level localization! 🤖
At the
#CVPR2023
CIVILS workshop,
@quantombone
is giving an overview of OrienterNet, our new work on localizing with OpenStreetMap - stay tuned for more details or come to our poster on Thursday afternoon!
The audience asking for clarification: “I’m not so familiar with Star Wars, but aren’t stormtroopers the bad guys? Then how can training to become a good Jedi prepare us to later be a stormtrooper?”
= industry vs academia, competing or complementary?
4/4
> We are experimentalists but we should feel lucky: unlike Monkeys, GPUs don’t need food, they just do the work.
> The problem with Putin, Musk, etc. is that they lack feedback from the real world – stay connected!
– Efros
2/
3) most importantly, it distracts us from writing great papers, which requires sustained, focused effort. Great papers have the highest impact on your career, while having a paper accepted to a conference is not special anymore - the review process is not sufficiently selective…
@BenMildenhall
The review process might be random but whether a paper has impact or not is much less random - citations/twitter decide so, not reviewers. Writing great papers takes sustained, focused effort and is harder if you get distracted by smaller, low-impact projects along the way.
The result: hundreds of AR unconstrained trajectories, aligned with ground-truth poses & scene geometry - perfect not just for localization & mapping but also for many other geometric vision tasks!
Then
@MattNiessner
told us about 3D reconstruction. I love this conclusion: combine deep nets, to learn priors and compressed representations, with geometry to enforce problem constraints. Use learning with parcimony!
#icvss2022
was incredible! Thank you
@GMFarinella
and the whole organizing team for your dedication, energy, and time! You're creating lifelasting memories and powerful positive reinforcement for generations of researchers. The computer vision community is truly amazing 👏
Last Social @ ICVSS 2022
Yes, ICVSSers are young and they have to go down with their “mind stress” after a week with 40 hours of scientific activities!
I already miss 2022 ICVSS Students! They will depart tomorrow morning ;-(
@SattlerTorsten
@ducha_aiki
Plugging in some experiments we did last year: none of the heuristics of the recent literature works consistently - it's just a hard problem. Still waiting for a deep net to solve it (doable but getting enough training data is hard)
We have released COLMAP implementations of recent papers that make Structure-from-Motion robust to symmetries
Surprise: none of these SOTA algorithms works consistently across datasets with the same hyperparams. More work needed for this difficult problem!
2/ The correct way: triangulation+BA. Every CV student learns that. Yet the paper deems this "too slow" vs an extra-engineered DSO. I already called this out in my talk in August + it is really so easy & fast with hloc+COLMAP
@dantkz
@zatxhi
@serizba
Yes, I should have made it clear - I would certainly change the default retrieval in hloc (and soon COLMAP!) to any approach that provides clear benefits over NetVLAD across the board :)
On top of the dataset, we also present:
2️⃣ a pipeline to build such ground truth fully automatically - no manual effort!
3️⃣ an extensive analysis of the state of the art, with great insights about using additional sensors like wifi/bluetooth, image sequences, VIO poses!
@ducha_aiki
@zhenjun_zhao
+ images have perfect GT poses
- queries are synthetic: occlusion artifacts, unrealistic lighting
- only 1 scene (3 in LaMAR)
- much fewer images (x10 in LaMAR)
- reference images from laser scanner with large FoV: easier to localize with, but too sparse to triangulate 3D points
"Could you please send us the paper that convincingly shows that a dumb rock has 0% chance of causing human extinction?" You're not doing science if your theory is not falsifiable.
@DrTechlash
To make that happen, could you please send us the paper that convincingly shows that human-level AI has a 0% chance of leading to human extinction? That the four independent ways that e.g. Dan Hendrycks found () that could lead to extinction are all wrong?
@chriswolfvision
Really cool, thanks for sharing! We also had a lot of fun visualizing attention patterns for SuperGlue, Fig 16 - highlights the importance of non-locality even for low-level vision tasks.
@chriswolfvision
Similar numbers for
#CVPR2020
, but different trend for the reviewers: Tsinghua U provides 3x fewer than Google at equal # of submitting authors. Since juniors don't review, do PIs there submit far more papers but actually review far fewer? Big player but not "community" player.
@dantkz
They are all evaluated only on outdoor datasets, right? I would love to see some evaluations with indoor data, for example on LaMAR, where retrieval is the main bottleneck
2/ I think that this paper is of great value to the community and I am glad that it got accepted to
#3DV2020
. But I am also a proponent of "good" science and want to discourage future works from adopting the evaluation as is, which is a common pattern in Computer Vision.
We improve the accuracy of Structure-from-Motion (= camera poses & 3D points), making COLMAP subpixel accurate & closer to Lidar performance. This enables:
➡️ accurate visual localization of new images
➡️ mapping with fewer images - critical for large-scale crowd-sourced AR 😎
2/
3/3 I apologize that my words could be deemed hurtful, and want to congratulate the authors on the acceptance! 🎉 Keep up the great work
@lukasvst
@PatrickWenzelML
@NanYang719
To scale and generalize to complex scenes, preserve shift-equivariance using local, distributed representations (e.g. grids) rather than global scene-level codes that lack this inductive bias. Great insights on
@songyoupeng
's and others' works.
@FabianFuchsML
For simple problems one can simply normalize by mean/std over the set - it is introduced in as Context Normalization (for 4D inputs). A follow-up weights the norm. with attention (O(n) vs O(n^2) for self-att)
@wsunid
@kwangmoo_yi