RT @WilliamRudmanjr: NOTICE uses Symmetric Token Replacement for text corruption and Semantic Image Pairs (SIP) for image corruption. SIP r…

RT @WilliamRudmanjr: We extend the generalizability of NOTICE by using Stable-Diffusion to generate semantic image pairs and find results a…

RT @WilliamRudmanjr: The finding that important attention heads implement one of a small set of interpretable functions boosts transparency…

RT @WilliamRudmanjr: How do VLMs like BLIP and LLaVA differ in how they process visual information? Using our mech-interp pipeline for VLMs…

RT @WilliamRudmanjr: Instead, LLaVA relies on self-attention heads to manage “outlier” attention patterns in the image, focusing on regulat…

@WilliamRudmanjr @CarstenEickhoff @health_nlp @BrownCSDept #nlproc #nlp #multimodal #interpretability #VLMs #mech_interp

RT @WilliamRudmanjr: The finding that important cross-attention heads implement one of a small set of interpretable functions helps boost V…

RT @WilliamRudmanjr: By visualizing cross-attention patterns, we've discovered that these universal heads fall into three functional catego…

RT @WilliamRudmanjr: Performing activation patching with NOTICE reveals a set of Universal Cross-Attention Heads that have a significant pa…

RT @WilliamRudmanjr: NOTICE uses Symmetric Token Replacement for text corruption and introduces Semantic Minimal Pairs (SMP) for image corr…

RT @WilliamRudmanjr: Mechanistic interpretability has advanced our understanding of LLMs, but what about multimodal models? Introducing NOT…

Lastly, we showcased the importance of structured clinical data. Using ablation analysis, we determined that structured clinical data (demographics, memory scores, balance testing, etc.) helps performance the most when combined with other modalities.

Through our experiments, we found that cross-modal attention in combination with self-attention contributes the most to performance, followed by cross-modal attention alone. This demonstrates the value of capturing cross-modal interactions through attention.

My work with @CarstenEickhoff and Ritambhara Singh, used attention-based deep learning to classify patients into control, moderate cognitive impairment, and Alzheimer’s disease. Our model achieved state-of-the-art performance on the ADNI dataset, at 96.88% accuracy.