@rebeccayelin soooo cool!!

RT @dimadamen: 🛑📢 HD-EPIC: A Highly-Detailed Egocentric Video Dataset New collected videos��

RT @ego4_d: Ego-Exo4D is the world's largest source of egocentric body + hand pose estimates and eye gaze data exists across the dataset. W…

RT @maxxu05: My paper RelCon: Relative Contrastive Learning for a Motion Foundation Model for Wearable Data, from my @Apple internship, ha…

nice Gaze-LLE demo from @fffiloni!

@fffiloni @huggingface @moondreamai @vikhyatk moondream is so fast, super cool work! @vikhyatk

@BazaritoDorito @fffiloni @Gradio @huggingface @RemiCadene absolutely! there's uncertainty in all predictions, and our model is designed to produce a heatmap over different targets. in the GazeFollow test set, which we benchmark on, the images have multiple raters and in ambiguous cases the annotators often select different targets too.

@BazaritoDorito @fffiloni @Gradio @huggingface @RemiCadene the model is trained on images where human annotators estimated the gaze targets, including some cases where the eyes aren't visible. the model likely uses other cues like the general head direction and faces being a common gaze target to make this prediction

RT @dimadamen: Ego4D and Ego-Exo4D @ego4_d Consortium held a short online meeting last month sharing some of the members' works around usin…

@jacob_g_martin1 another thing is that these datasets are annotated by humans who are simply guessing where the person is looking. some have multiple annotators to capture uncertainty, but many only have one. would love to hear any thoughts you have on how to evaluate models for this task!

@jacob_g_martin1 the coordinates of the prediction and ground truth are normalized so that the height and width are equal to 1 prior to L2 calculation (i.e. x = x / width, y = y/ height). we do this to follow the established metric calculation protocol for the benchmarks we compare on

@Abhinav95_ thank you!🤗

@NeuralBets_ @tom_doerr I'm less familiar with the benchmarks in this area, and there are commercial solutions like Tobii that could be the best. this article compares some open source options like OpenGaze and WebGazer

@Sourabh85426135 check out tab 3. in the paper - there are many recent models for this task with nice results! ours has the benefit of using a frozen encoder + very few learnable params. however for on-device applications, the ViT encoder itself may pose challenges compared to smaller conv models

RT @TelepathicPug: @fionakryan This is great. I'm having too much fun.

@RigoTech1 @sangminlee777 @judyfhoffman @RehgJim that's my hypothesis, along with the results being quite close to the inter-rater agreement of the annotations. sweeping LoRA hyperparams (like which params to tune) could potentially get more gains. our main focus was on using the encoders out of the box.

@tom_doerr I didn't, but there are several works on webcam-based eye tracking that are designed specifically to predict the coordinate on the screen the viewer is looking at!

@vikhyatk thanks for sharing!

Big thank you to the Gaze-LLE team: Ajay Bati, @sangminlee777, Daniel Bolya, @judyfhoffman , @RehgJim for working on this project with me!