Does seeing the whole prefix make the last state a summary?
I was skeptical of treating the final token representation as a complete message about everything that came before it.
The intuition
Attention provides a path to earlier positions, not a guarantee that their useful information survives in a single vector. Next-token training rewards predictive representations, which need not preserve every fact a different agent might need.
A communication claim needs a receiver-side test: compare text, single-position vectors, and multiple-position representations; remove or shuffle messages. Access to information and a sufficient summary are different capabilities.
Related reading · Attention Is All You Need — architectural background