This is great, I've read multiple books and watched videos about the attention mechanism. Now that I understand it, this is the clearest example I've seen on how attention works.
I don't disagree with that. I did add an entire caveat paragraph there.
To me, it's more of a neat visualization, not something that can be used to interpret LLM behavior. Even with a lot of simplification, it can show some interesting patterns.
I mean, there so many headers and layers, it is tricky to make a choice that will resonate with our intuition . Is it some weighted average? Or maybe ablation test?
To me, it's more of a neat visualization, not something that can be used to interpret LLM behavior. Even with a lot of simplification, it can show some interesting patterns.
are you worried later-layer attention gets drowned out by earlier layers just because there are more of them contributing to the sum?
Right now only simple correlations are visible.
I mean, there so many headers and layers, it is tricky to make a choice that will resonate with our intuition . Is it some weighted average? Or maybe ablation test?
When I started, I expected I'd have to experiment a lot to find something comprehensible. But this simple computation can already show some patterns.