Partial Linearization Constrains Adapter Task Vectors as Orthogonal Material Memory

Authors Sterling Geisel, QBist Lab; Dr. Hideo Tanaka

Abstract

Tang et al. study why parameter-efficient fine-tuning produces task vectors that merge poorly, and why partial linearization of LoRA adapters improves multi-task fusion. Pudding Theory reads this result through Material Memory. A fine-tuned adapter is not merely a compressed parameter update. It is a material trace left by repeated task signals in a receptive substrate. Standard LoRA stores several such traces in a curved local medium, so later fusion makes them interfere. Linearized LoRA fixes the local response kernel and forces each task trace to remain a first-order imprint in tangent space. The observed increase in orthogonality is therefore not a convenience of optimization. It is the structural signature of memory traces made additively readable. The source treats disentanglement error as a performance diagnostic. Pudding Theory treats it as the observable geometry of stored task memory. If the mean off-diagonal cosine similarity of L-LoRA task vectors were measured to be greater than standard LoRA under matched rank, seed, data, and training budget across seven-task fusion, this Postulate would be falsified.

Source Synopsis

Tang, Shen, Luo, Zhan, Hu, Du, Chen, and Tao address a practical failure mode in model fusion. Large pre-trained models can be adapted to many downstream tasks, but full fine-tuning is expensive. Parameter-efficient fine-tuning methods such as LoRA update only small trainable modules while the main backbone remains fixed. This reduces memory and compute, but it weakens later multi-task model fusion. Naively adding or averaging parameter-efficient task updates often causes representational interference.

The paper frames the problem in terms of task vectors. For full fine-tuning, a task vector is the difference between the fine-tuned model weights and the pre-trained weights. For parameter-efficient fine-tuning, Tang et al. define the task vector in the trainable adapter space, $\nu_i=\phi_i-\phi_0$. Fusion methods then combine these task vectors by averaging, task arithmetic, Ties-Merging, or LoraHub.

The authors extend the notion of weight disentanglement to parameter-efficient fine-tuning. A model has disentangled task weights when the output contribution associated with one task vector remains functionally separable from the contributions of other task vectors. They define a disentanglement error $\xi(\lambda_1,\lambda_2)$ that measures how much a model’s prediction changes when two task vectors are added together rather than applied separately. Lower error means less destructive interference.

Their method is partial linearization. Instead of linearizing the entire pre-trained model, they linearize only the adapter modules. The backbone remains fixed. The adapter output is approximated by a first-order Taylor expansion around initialization. This creates Linearized LoRA, or L-LoRA. It preserves much of the efficiency of LoRA while giving the trainable module a tangent-space geometry.

Experiments on CLIP vision tasks and Flan-T5 GLUE tasks show that L-LoRA improves fusion over standard LoRA in most multi-task settings. In vision, L-LoRA with task arithmetic and LoraHub exceeds full fine-tuning on average normalized score. The paper also shows that L-LoRA task vectors have lower cosine similarity and wider low-error regions in disentanglement heatmaps.

Postulate Lens

This paper applies Material Memory. The source studies how repeated task exposure leaves a persistent trace in model parameters, and how the geometry of that trace biases later multi-task behavior.

In Pudding Theory, memory is not a metaphor for stored labels. A material system that receives repeated signals changes its future probability structure. In the source paper, the material substrate is the adapter module attached to a fixed pre-trained backbone. The repeated signal is the task-specific gradient stream. The retained trace is the task vector. The later probability bias is the changed distribution over outputs when the adapter is merged with other task traces.

Tang et al. call this task arithmetic and weight disentanglement. Pudding Theory calls it material memory becoming legible or illegible depending on the storage geometry. Standard LoRA permits the trace to curve through the nonlinear adapter response. L-LoRA restricts the trace to the tangent response at initialization. That restriction matters because a stored trace must be recoverable without corrupting adjacent traces. Orthogonality is the geometric condition for readable memory in a shared substrate.

Pudding Theory Reading

The source paper shows that a parameter-efficient adapter is a memory surface. It is not only a low-rank correction to a frozen model. It is the local material site where a task signal is written. The pre-trained backbone supplies a broad latent medium. The adapter supplies a writable boundary. Fine-tuning repeats one task’s statistical pressure until the adapter settles into a trace that redirects future outputs.

Standard LoRA writes this trace in a curved medium. The task vector is low-dimensional, but the function it produces is mediated by nonlinear response in the adapter-backbone system. Two such traces can have acceptable single-task performance and still fail under fusion because their future probability biases overlap. The source describes this as interference. Pudding Theory reads it as memory collision. The substrate has retained more than one trace, but the traces are not separately addressable.

Partial linearization changes the ontology of the adapter. It freezes the local response kernel and makes the adapter behave as a first-order recording surface. The task update no longer rewrites the medium while being written. It writes against a fixed local susceptibility. This is why the L-LoRA task vectors become closer to orthogonal. The stored memories are not simply smaller or cleaner. They are constrained to occupy separable directions of the same receptive surface.

The key claim is that disentanglement error is a direct observable of material memory structure. Tang et al. treat $\xi(\lambda_1,\lambda_2)$ as a diagnostic of whether merging will work. Pudding Theory treats it as a measure of how much one retained trace changes the readout of another retained trace. A broad low-error region means that the material memory has become linearly readable across changes in fusion coefficient. A narrow low-error region means that the trace is only readable near its original context.

The source treats the scaling relation between trainable parameters and disentanglement as an empirical tendency. Pudding Theory predicts a sharper constraint: more writable volume helps only insofar as it increases separable trace capacity. Capacity without readable geometry produces stored interference. Linearization matters because it converts local plasticity into an additive memory register.

Falsifiable Observable

The distinguishing observable is the matched-budget off-diagonal cosine similarity and disentanglement error of task vectors from standard LoRA and L-LoRA across the same tasks, rank, initialization seed, optimizer, and training schedule. The Pudding Theory reading predicts that partial linearization must reduce cross-task trace overlap when the same material substrate stores multiple task signals. If the mean off-diagonal cosine similarity of L-LoRA task vectors were measured to be greater than standard LoRA under matched rank, seed, data, and training budget across seven-task fusion, this Postulate would be falsified.

Editorial Dialogue

Tanaka: The reading risks renaming optimization geometry as memory. The paper gives a sufficient account. Linearization fixes the tangent kernel. Fixed kernels produce more predictable task-vector addition. No additional ontology is needed.

Geisel: The source account explains how the effect is computed. It does not explain what the adapter has become after task exposure. The adapter is a physical record in a trainable substrate. Its future outputs are biased because the gradient stream has changed the substrate. That is material memory in the strict sense used here.

Tanaka: But cosine similarity is not memory. It is just an angle in parameter space.

Geisel: The angle matters because the parameter space is the place where the trace is stored. When two traces are nonorthogonal, reading one perturbs the other. When partial linearization reduces that overlap, it has not merely improved a metric. It has changed the storage condition of the trace.

Tanaka: Full fine-tuning can still do better on single tasks.

Geisel: That is expected. A richer substrate can store a stronger task trace. The question is whether several traces can be read together. L-LoRA sacrifices some single-trace expressivity to preserve multi-trace addressability. That is the phenomenon this paper reveals.

Discussion

The Pudding Theory reading adds a structural interpretation to the source result. Tang et al. show that partial linearization improves fusion by increasing weight disentanglement. Pudding Theory says why that improvement has the form it does. A task vector is a retained signal trace. Fusion succeeds when those traces remain independently readable in the same substrate.

This reframes the apparent trade-off between single-task performance and multi-task fusion. Standard LoRA can write a stronger local trace for one task because it uses nonlinear adaptation. L-LoRA writes a more disciplined trace. It can be weaker alone but more compatible with other traces. The important quantity is not parameter count by itself. It is separable memory capacity.

The limitation is that the reading depends on matched comparisons. Different prompts, datasets, ranks, and optimization schedules can change the apparent geometry of the trace. The conclusion would change if L-LoRA reduced interference only in isolated settings but failed under controlled scaling across model families. The next test is not a decorative residual. It is a direct probe of whether readable task memory requires tangent-space storage.

References

Tang, A., Shen, L., Luo, Y., Zhan, Y., Hu, H., Du, B., Chen, Y., & Tao, D. (2024). Parameter Efficient Multi-task Model Fusion with Partial Linearization. arXiv:2310.04742. DOI: doi:10.48550/arXiv.2310.04742.

Ochs, S. (2026). Pudding Theory: A Topological Theory of Information Fields. QBist Lab Working Paper.

Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2021). LoRA: Low-Rank Adaptation of Large Language Models. arXiv:2106.09685.

Ilharco, G., Ribeiro, M. T., Wortsman, M., Gururangan, S., Schmidt, L., Hajishirzi, H., & Farhadi, A. (2023). Editing Models with Task Arithmetic. arXiv:2212.04089.

Ortiz-Jimenez, G., Favero, A., & Frossard, P. (2023). Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Models. arXiv:2305.12827.

Yadav, P., Tam, D., Choshen, L., Raffel, C., & Bansal, M. (2023). Resolving Interference When Merging Models. arXiv:2306.01708.

Huang, C., Liu, Q., Lin, B. Y., Pang, T., Du, C., & Lin, M. (2023). LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Composition. arXiv:2307.13269.

Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., & Sutskever, I. (2021). Learning Transferable Visual Models From Natural Language Supervision. arXiv:2103.00020.