Training Vision Models for LLM-Driven Weight-Space Programming
Abstract
It has been demonstrated that, in certain settings, large language models can perform weight-space programming to directly modify the weights of neural networks. However, it is unknown how dedicated training of these networks can contribute to the successes of weight-space programming. We present a benchmark of LLM-driven weight-space programming on vision models, in which five LLMs edit convolutional networks and vision transformers on relabeling and error-correction tasks on Waterbirds, CIFAR-100, PACS and Entity-30. We report more than 2,600 such edit processes, whereby each edit is scored on held-out images for success, side effects and accuracy loss. We compare standard training with label smoothing, gradient-based editable training, and our Edit-Ready Training (ERT), a novel method that rehearses the direct, closed-form weight edits LLMs perform. We show that preparing networks for editing raises the success of LLM edits substantially where standard networks are hard to edit. ERT further achieves the highest mean score for GPT-6 Astra and Claude Opus 5.5 across all evaluated baselines. We will release our LLM testing framework alongside all tasks, editing trajectories and scores. Project page: https://viscom.uni-ulm.de/wsp
Community
Training Process to improve Edibility in Weight Space Programming by Timo Ropinski & Sebastian Kranz
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- Lost in Read-out: Locating the Image Quality Gap in Vision-Language Models with a Closed-Form Edit (2026)
- Can an LLM Actually Reason and Program about Neural Network's Weights or is it just memorization on steroids? Exploring"Weight-Space Programming" (2026)
- GradReason: LLMs as Meta-Learners that Replace Training Epochs with Training-Data-Free Weight Updates (2026)
- Weight Space Programming for Vision Language Model Repair (2026)
- BaseRSI: Recursive Self-Improvement of a Base Vision-Language Model via Training-Free Harness Internalization (2026)
- WeightBench: Evaluating Language Models as Weight-Space Programmers (2026)
- Can AI Mature a Model's Geometry? Training-Free Weight Baking in LLMs (2026)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend
Get this paper in your agent:
hf papers read 2611.26408 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 1
Collections including this paper 0
No Collection including this paper