Johnny Lee famously hacked a Wii Remote to turn a 2D screen into a 3D window. We revived this trick in Chrome using a standard webcam.
Here’s how we did it. 🧵👇
1/ As soon as StableDiffusion landed we dropped everything to build a GENIE! 🧞♂️
🎙 Voice UI
🖼 AI Art
😎 AR previews
🤑 Instant purchasing
📦 On-demand production
Could AI make every shopper’s wish come true? 🤔👇
#ai
#aiart
#stablediffusion
#dalle
#dalle2
#vui
AI unlocks unlimited product photography. 🤯
Recent advances have made it possible for merchants to turn mobile photos into high quality product images + ad creative.
AI can now learn and reproduce a product on demand. More…👇[1/10]
#ai
#aiart
#stablediffusion
#dreambooth
AI is enabling some really magical experiences. 🪄
Originally planned to share this tomorrow,
@sergeyglkn
scooped us! 😜
Keep on eye on this space. It moves fast! 💨
Could you tell which room was virtual? 👀
Traditional AR can’t tell you how a couch feels. Our team
@Shopify
explored "Reverse AR" to wrap a virtual room around a real thing. Sit on a couch in-store while seeing it in your space—best of both worlds!
👇 How we built it… 🧵1/12
What if you could bring your home shopping?
Our latest prototype lets you teleport real furniture from the store into a 3D scan of your living room. It's amazing to experience first hand. 🤩
Next-gen pro headsets from
@Meta
and
@Apple
are dropping soon. Had us curious what the future of work might look like for Shopify merchants. 🤔
Could AR glasses revolutionize product photography?
🧵
#ar
#futureofwork
#quest
Been there—15 pages deep in cryptic instructions, surrounded by scattered parts feeling like 😵💫
Now with headsets like Quest Pro enabling hands-free AR, you can see 3D build instructions right in your space with helpful hints. 🤩
🧵👇
#XR
#AR
#FutureOfShopping
Meta's new
#QuestPro
tracks your hands and eyes to animate avatars. Our team
@Shopify
wondered if the same tech could give merchants insight into shopping patterns within their store…
Could VR give retailers a fresh perspective? 🧵👇
#vr
#passthrough
#futureofretail
Exploring Spatial Commerce opportunities
@Shopify
with Vision Pro on the horizon…
It's going to be fun pulling product demos right off the webpage and into the room 🤏✨🤯
Do you love perusing a store, seeing what leaps out as you go? 👀
Do you love the infinite scale of the internet, with powerful filters and categories? 🗂️
What if you could have the best of both worlds, plus instant AR try-on as a cherry on top? 🛋️
Every week that goes by we get closer to this. This is where we're is going. AI and AR coming together, not to block out the world, but to light it up 🌟
Our latest concept takes the
@shop
/
@openai
search integration one step further… 👇
#futureofshopping
Shopping with friends IRL is a great vibe, but challenging to replicate online. 👯
Spatial Commerce powered by Shopify with VisionOS and SharePlay will enable a new generation of virtual try-on experiences that are seamless and shared.
@Shopify
@Apple
AR and
#RoomPlan
are exciting tools for commerce. The ability to select and remove real objects from a scene brings us one step closer to a fully editable reality. Speaking of… time to bring back the dog ✌️[12/12]
@Shopify
@Apple
With an empty room, shoppers can now swipe through room sets which leverage the orientation of existing furniture to arrange themselves automatically… [4/12]
@Shopify
@Apple
AR makes it easy to try new couches, but the couch already in your space can get in the way. What if every room had a reset button? Space Eraser models spaces in high-fidelity—capturing room-defining objects, their size, position, orientation… [2/12]
Opened our first public 🤗HF Space under the official
@Shopify
account this morning: SDXL background replacement for product imagery. 🖼️
Creates a whole new reality around your product 🤯
It's so fun! Try the early demo 👇
@Shopify
@Apple
Live pixel data combines with LiDAR depth data to enable a lifelike digital twin of your room that can be overlaid onto your real space using AR and then edited digitally… [3/12]
@Shopify
@Apple
To determine if this would be possible today we explored two questions: how do we align the model produced by
#RoomPlan
with your room and how do we texture the model? [6/12]
It's my last week at
@Shopify
! Such a privilege to accelerate AI and spatial commerce efforts over the past few years. 🥲
Thrilled to share I'm re-joining
@Meta
to lead new GenAI product efforts. Excited to work again with Mark and some old friends new possibilities!
Tonight is
@jessicahische
and my anniversary. Cooked a low key dinner. Reminiscing about our favorite memories over the last 8 years. Kids are putting on a puppet show. So grateful for this woman. There never was a better human.
Since joining
@shopify
the
@primersupply
team has made seamless AR shopping available to millions of merchants and buyers in the Shop app.
We're democratizing the next major evolution in commerce and we’re just getting started. Come build with us 👇
Devices like Vision Pro from
@Apple
create exciting opportunities for product extensions. Imagine “waking” a physical product to unlock digital functionality. 🤯
#ARShopping
#SpatialComputing
All the nerdy details…
We used
#dreambooth
to train models for each product. 200-250 regularization images from the same product category, 20-25 training images of a specific product, 4000 iterations (~1–1.5hrs on an A100). ✌️🙏[10/10]
Anything behind the virtual window appears to have depth, and anything in front of it appears to pop out of the screen. The illusion truly shines when you close one eye.
@Shopify
@Apple
With a digital twin of your room and your furniture you could simply swipe your original sofa away to explore options that better match the rest of the room… [5/12]
Spatial Commerce powered by
@Shopify
will help brands immerse shoppers in a product story. ⛺🌲
With support for webXR, Vision Pro will help merchants deliver fully immersive VR experiences right from their web storefront…
4/ AR makes it possible to visualize the product in the real world in vivid detail and automatically size your order. Luckily we know a thing or two about this from our days at
@primersupply
😎
👋 We’ve been working on this one for a while at
@primersupply
…
💡Our AR paint intelligently blends in highlights and shadows
👨💻 Props to
@ericflo
for getting this to the finish line
📱 Live now:
3/ The patterns in our prototype were created using the prompts you hear in the video and the tiling technique we developed. Each frame of animation is a real step of the generation process. You’re watching StableDiffusion dream in real time 💭👀
We’re always looking at better ways to shop
@Shopify
.
Vision Pro’s intuitive eye, hand, and voice interactions could unlock instant, personalized insights when combined with the powers of conversational AI
👁️🤏💬✨🛍️
2/ Shopping is hard when you want something specific but aren’t sure where to find it. New models from
@stabilityAI
and
@OpenAI
can create a detailed image of anything you can imagine, so skip searching — ask for exactly what you want and print it on-demand! 🖨🌽
if any non-electronic item can have glasses-augmented smarts, really there's no limit to the things we could do: guitar tuning app without buying hardware. song tabs without pulling out a whole laptop...
you could choose to have as much or as little UI around 𝑎𝑛𝑦𝑡ℎ𝑖𝑛𝑔
@Shopify
@Apple
#RoomPlan
outputs an untextured USDZ model composed of unit cubes; one for each door, window, wall, and room-defining object—but, surprise! Models don’t contain floor or ceiling elements. We align the model with the real world using ARWorldMap. [7/12]
5/ Creativity drives commerce and AI is unlocking a lot more creative potential. 💥
It’s early days. The same AI tools we’ve leveraged here will soon enable people to create any form of media: literature, music, film, television, apps, even fashion.
@Shopify
@Apple
We reversed the graphics pipeline: instead of sampling from a texture and writing to the screen, we sample from the camera and write to a texture. RealityKit made this hard, so we used SceneKit instead. [9/12]
So proud of
@jessicahische
. Her second kid's book "Tomorrow I'll be Kind" is out in stores today. Jess has poured years (YEARS!) into this series. The message is so uplifting and kids are OBSESSED with the illustrations. Must have for every child library:
@Shopify
@Apple
This approach creates issues when furniture obscures a surface. In video above 👆, our couch obscured the wall.
#RoomPlan
provides bounding box references for room-defining objects. With this we can determine what parts of the texture to redact. [10/12]
@Shopify
@Apple
We can now use inpainting techniques to fill in the redacted texture. We were pleasantly surprised by the results produced by off-the-shelf tools like Photoshop and . [11/12]
The more I play with
#StableDiffusion
the more convinced I am that AI is going to be an accelerator for creativity without supplanting the economic value of human taste and perspective.
Photoshop did not kill art. It made art making easier and invited more people into it. Same.
@Shopify
@Apple
Since RoomPlan exports untextured models, we explored ways we might texture it automatically. The best way would be to localize and map the texture when capturing, but we make do with an aligned model and a real-time camera feed. [8/12]
But some products don't embed/reproduce well. StableDiffusion is currently challenged by…
1. Text, logos, and precise detail
2. Complex geometry
3. Counting
Excited to share SingSong, a system which can generate instrumental accompaniments to pair with input vocals!
📄
🔊
Work co-led by myself,
@antoine_caillon
, and
@ada_rob
as part of
@GoogleMagenta
and the broader MusicLM project 🧵
We transformed the monitor into a virtual window by recreating the real world in our game engine with precise dimensions. The webcam is the origin of the world, and the virtual window has the exact same dimensions as the screen.
AR/VR team
@Shopify
is playing with
@Apple
's new RoomPlan API. It outputs a USDZ model with an un-textured box primitive for each wall, door, window, and "room-defining" object—no ceiling or floor boxes. Origin is set to where the scan began.
Excited to share deep dives on our recent spatial commerce projects
@shopify
on our new Github repo! 👀
We’ll continue publishing research, playable prototypes, and open-source tools here to help accelerate the industry.
Enjoy! (and bookmark) ☺️🫶
Friday was my last day at Facebook. I’ve wanted to start something new for a long while, and now‘s the time. It’s impossible to sum up what being at Facebook has meant. I’m so grateful for my time and so excited about what’s next!
To help you prototype your own XR ideas we're open-sourcing a free tool: 👋Handy 🎉
Hands are difficult to animate manually, but Handy makes it as easy as putting on your headset and hitting record. Download:
Launched
@primersupply
1 year ago today! 🎂🥳
A few stats then + today…
Partners: 5… today: 50
Products: 250… today: 3500
Time to virtualize a product: 2-4hrs… today: 2-4 minutes
Team: 3… today: 8
So proud of everything this amazing team has accomplished so far. ❤️
The pace of progress around AI is staggering. Breakthrough research papers appear each week, enabling a new mind-bending wave of possibilities. It's both exciting and exhausting! 😅 [9/10]
It’s been shared by others with much bigger reach than me, but here it is for posterity: my talk from
@LDconf
about my journey from designer to manager, back to designer. 15m to learn what took me 6 years. Not bad 🖖
@Apple
@Polycam3D
Reverse AR unlocks fun in-store opportunities. In addition to shopper’s homes, merchants could transport shoppers and products anywhere: a Paris flat, a serene mountain cabin in Japan, or next to an infinity pool looking over a desert oasis. 🧵12/12 ✌️
AR and natural interactions will enable an explosion of opportunity that developers are only just beginning to unlock. Excited for
#Connect
next week and looking forward to playing around with color passthrough 🥽🌈🤩
Product designers: curious about working at an early-stage company? A founder friend of mine (Stripe Mafia) runs a well-funded co. and she's looking to hire their first designer. Contract or full-time. Remote or not. DM me if you're curious! ✌️
Today
@Apple
single-handedly reset the UX paradigms and quality bar for spatial computing to everyone's benefit. Hardware is still the constraint and will be for a while, but they nailed the interaction paradigms to unlock better software.
Pretty incredible.
If you're scratching your head a bit, this means computers can now see and understand what objects are around them in real time, enabling many of the object-as-app-platform interactions we've been exploring recently. 🤯
Today we're releasing the Segment Anything Model (SAM) — a step toward the first foundation model for image segmentation.
SAM is capable of one-click segmentation of any object from any photo or video + zero-shot transfer to other segmentation tasks ➡️
So excited to share more about what we've been building with you. We hope it brings a little creativity into your life and home during this stressful time. Stay safe! Stay well!
Excited to share what we’ve been working on. Our platform makes it easy visualize changes to your space instantly with products from curated brands. You can sign up for early access at: (1/4)
@letkma
Thanks for the all kind words and new follow y'all! 🤯
If you want to hear about more what our team
@Shopify
is up to vote for your SXSW panel on spatial computing and the future of commerce 🤓👇
AR is an incredible tool but right now, creating a 3D model is complicated, timely, expensive and not practical for many brands. We’re changing that
@Shopify
.
Today, with the release of
#iOS17
, we’re making it possible for all our merchants to create 3D models from the palm of
TFW the product you’ve been scheming on for a year and then actually building for six months gets submitted for app store review. Couldn’t have done it without
@heyadam
and an amazing crew of devs, advisors, and investors. 🤩
I would like to write a television series called “Florida Man...” where each episode is a vignette into one bat-shit crazy story based on real life. Buck wild stuff.
@Apple
@Polycam3D
In store, shoppers can instantly visualize their room around an object. Computer vision systems identify and segment furniture and people from the video stream and mask them into the virtual room. 🧵3/12
I freaking love this idea — with iOS 16, this guy is saving clips of every outfit he wears in the Notes app 🤯
With background removal, you can cleanly categorize almost anything 👌
Looking forward to sharing more about
@shopify
AR/VR/AI explorations and meeting everyone at
#AWE2022
. Keep an eye out for
@ericflo
and me, we’re friendly ☺️
@Apple
@Polycam3D
That’s when we realized that we could use depth-buffer segmentation to more cleanly cut objects out of the scene if we had a bounding box to constrain the depth buffer to world space. 🧵9/12
Metaverse today: vast spaces, mostly empty and “dead” to interaction.
Alternative: intimate spaces, densely populated with potential interactions.
I prefer the latter.