#MRI simulations of images can only be done with spin simulations, but not with EPG? In the new paper Jonathan Endres and Co-Authors show a new MRI simulation called Phase Distribution Graphs which extends EPG for arbitrary timing and image encoding.

I took a brief look at the Harmonic Loss paper tl;dr: instead of dot-product with softmax, do euclid dist with normalized 1/d**n. I kinda want this to work. I've dabbled with preferring euclid many times throughout my career (eg triplet loss etc) However...

Since we're going back to the origins, let me add that Attention, which @karpathy correctly calls a "brilliant (data-dependent) weighted average operation", was not discovered in machine learning - in fact this dates back to data-dependent "filters" in image processing from the 90s. Perhaps the most well-known is the Bilateral Filter of Tomasi and Manduchi from 1998, and the SUSAN filter of Smith and Brady in 1997; followed by many variations like non-local means of Buades, Coll, and Morel in 2005, and more general kernel regression filters in 2007 by me and two of my students: A year or so before the 2014 time-frame mentioned in @DBahdanau's account of Attention, the kernel filters were generalized even further using the RKHS framework, and popularized even more by yours truly, in late 2012: Open access version here: Neither I - nor, I think, anyone in imaging - could reasonably claim any credit for the idea of attention in the ML context. But it is worth having the complete historical picture, which would be incomplete without understanding the importance and generality of "(data-dependent) weighted average operations" much more broadly.