Very pleased to confirm that our paper "Residual Stream Analysis with Multi-Layer SAEs" has been accepted to ICLR 2025!

@GoncaloSPaulo @nlp_ceo Well, it's definitely a feature of language modesl, and a problem in the sense that naïvely applying the same encoder/decoder at multiple layers isn't viable.

@nlp_ceo Oh, that's a nice idea! I guess you could reverse-engineer a lossy mapping between the representation spaces of adjacent layers that way. We tried applying 'tuned lens' transformations to similar effect, but with limited success

@nlp_ceo To clarify, 'surprisingly few' means 'fewer than we had expected' (not a majority). Our distributions of latent activations over layers are qualitatively similar to Anthropic's feature norms, and we concur that "feature drift" is a problem for this multi-layer/shared approach.

@nlp_ceo Thanks, I haven't read this thoroughly -- what do you mean by inter-layer dynamics, given that the method is data-free? At a glance, I can only see the LLM-based categorization of features as same/maybe/different, which look roughly equally distributed.

@nlp_ceo This is interesting, thanks. We actually found surprisingly few latents activated at multiple layers when training a single SAE on every layer of the residual stream:

@lucyfarnik See also

RT @lucyfarnik: @banburismus_ @leedsharkey Imo: - We published a paper that played into people's pre-existing beliefs - They started tweeti…

RT @a_karvonen: We find that SAEs on trained models have slightly higher autointerp scores than those trained on random models. Note howev…

RT @a_karvonen: @saprmarks @nabla_theta @aidanprattewart @ThomasEHeap @tslwn @lucyfarnik @laurence_ai Trenton had this to say: https://t.co…

RT @nabla_theta: @GoncaloSPaulo @aidanprattewart @ThomasEHeap @tslwn @lucyfarnik @laurence_ai like you'll find a feature that activates onl…

So, if you're using autointerp as a proxy measure to compare the 'interpretability' of SAEs, something like a randomised model is an important baseline.

FWIW, I agree with this analysis of our paper -- the takeaway is that you can get latents with good auto-interp scores without an 'interesting' underlying model (and single-token activation patterns are the most likely culprit).

@_MathAcademy_

Are computational models and processes increasingly analogous to the brain and cognition, or do we increasingly speak (think) in computational metaphors?

-- Hacker, P. M. S. (1990). Men, minds and machines. In Wittgenstein, meaning and mind. Cambridge, Mass., USA: Blackwell. pp. 89–111.