Research notes · September 2026
State-Aware Adaptation and Control of Language Models
Studying the boundary between parameter changes, activation interventions, and the state left behind by prior computation.
Independent research in progress. This page describes experimental infrastructure and open questions, not a published method or an established efficiency gain.
The question
A LoRA adapter changes the computation that reads a sequence, but it can also change the keys and values written while processing that sequence. Similarly, disabling an activation intervention does not reconstruct the state or tokens produced while it was active. These distinctions motivate controlled tests rather than an assumption of persistent behavioral change.
Separating parameters from cached history
I implemented a four-condition evaluation that crosses the current reader (base or adapted model) with the cache writer (base or adapted model). Matched token sequences and full-replay checks help isolate changes attributable to cached state from changes in current parameters.
- Base reader, base-written cache.
- Adapted reader, base-written cache.
- Base reader, adapter-written cache.
- Adapted reader, adapter-written cache.
What steering withdrawal taught me
My steering experiments compare interventions that are absent, continuously active, or active only over the prompt. Fixed-token scoring, free generation, random and orthogonal directions, and precision checks serve different purposes.
A crucial alignment detail: the first generated token is predicted from the final prompt position. If that position was still steered, the first token is not evidence of persistence after withdrawal.
In the settings examined, residual output changes did not by themselves establish persistent, direction-specific control. This negative result sharpened the evaluation: a changed cache, a changed trajectory, and a retained target behavior are different claims.
Where I want to take this
I am interested in inexpensive corrections that let a model reuse cached computation after a small adaptation. The next useful result would be a narrowly defined regime where correction improves over direct reuse at lower cost than recomputation, with explicit failures and quality–compute measurements.
Related interests include controllable latent reasoning and communication between models through learned representations.