Experience with dissimilar language ablation?

Beginners
1

Anyone here have experience with dissimilar language ablation? I’m thinking of ablating Mandarin/Russian/Arabic to leave a primarily Latin-base language model with the hope of making some space for further training and/or safely pruning where English demonstrably has no activation.

I’m presently creating a Swadesh-esque noun/verb (“thing”,“does”) pair list across the four languages where each pair either token-matches every other pair or gets padded to match, if necessary.

2

Hmm… I don’t have direct experience with this exact setup, but I think this is one of those cases where the right thing to do changes a lot depending on what you ultimately want to achieve:

Short version

I would not start by treating “dissimilar language ablation” as one single operation.

Depending on the actual goal, this could be:

Goal First thing I’d try What it does What it does not prove
Understand where language behavior lives Swadesh-style probe + temporary ablation Good interpretability experiment Does not prove safe permanent pruning
Stop the model from generating Mandarin/Russian/Arabic suppress_tokens, LogitsProcessor, constrained decoding Reversible output-level control Does not remove knowledge or reduce model size
Reduce vocab / embedding / lm-head footprint vocabulary trimming / embedding pruning Reduces part of the model footprint Does not fully erase language ability
Publish an English/Latin-focused derivative LoRA / QLoRA / PEFT adapter Shareable, cheaper, reversible-ish artifact Not verified unlearning
Actually make the model forget those languages unlearning setup with forget/retain evals More principled forgetting objective Expensive and hard to verify completely
Physically modify the checkpoint pruning / tensor surgery + recovery tuning Most “surgical” and potentially most original Recovery/eval cost can dominate

So, if this is mainly an experiment, I think your Swadesh-style probe + temporary ablation idea is reasonable.

But if the real goal is to get a usable English/Latin-focused model, I would probably try lower-risk baselines first: output suppression, vocabulary trimming, or an adapter/QLoRA derivative. If those do not meet the goal, then I’d frame the next step as either unlearning or checkpoint surgery.

One thing I’d want to know first is the exact base model. A dense Llama/Mistral/Qwen-style model, an mT5/mBART/NLLB-style encoder-decoder model, and an MoE model lead to quite different interventions.

Why the exact base model matters

The answer changes a lot depending on the model family.

A few examples:

Model type Why it matters
Dense decoder-only LLM, e.g. Llama/Mistral/Qwen-style I would not assume unwanted languages live in clean removable modules. Language behavior may be distributed.
Encoder-decoder multilingual model, e.g. mT5/mBART-style Vocabulary trimming / embedding pruning may be more directly relevant.
NLLB / translation-style model Language codes, tokenizer behavior, forced BOS tokens, and translation-specific control matter.
MoE model Expert routing statistics may matter, but experts are not automatically language modules.
Chat/instruct model Chat/control/special tokens can be easy to break when filtering vocabulary by script.
Quantized checkpoint Some surgery/merge/retraining steps become more fragile.

The tokenizer is also important. If you trim or rewrite vocabulary, the tokenizer, input embeddings, output lm_head, config.vocab_size, special token IDs, and possibly tied embeddings all need to stay aligned.

Useful references:

For chat/instruct models, I would be especially careful not to remove control/chat/special tokens while filtering by script. A token that looks irrelevant from a language/script perspective may still be structurally required by the chat template.

1. If this is mainly an experiment: your probe idea is reasonable

If the goal is interpretability or PoC exploration, then I think your plan is reasonable as a probe:

  • build a balanced word/probe list;
  • compare tokenization across languages;
  • inspect activation patterns;
  • try reversible masking or temporary ablation;
  • only then consider permanent surgery.

Your Swadesh-style noun/verb list sounds useful as a diagnostic set. Token matching or padding can reduce some input-length confounds. But I would treat it as a source of hypotheses, not as a deletion certificate.

In other words:

“This unit activates more for Russian/Arabic/Chinese than English”
is not the same as
“this unit can be removed without hurting English, code, math, names, formatting, or reasoning.”

Related work exists around language-specific neurons. For example, “Language-Specific Neurons” proposes LAPE, a method for identifying language-specific neurons in LLMs, and studies LLaMA-2, BLOOM, and Mistral:

There is also newer work on language-neuron manipulation / “language arithmetics” across Llama-3.1, Mistral-Nemo, and Aya-Expanse:

But I would read these as support for probing and reversible intervention first, not as proof that the corresponding neurons can simply be deleted.

A useful experimental order might be:

  1. Tokenization audit.
  2. Activation probe.
  3. Temporary ablation / forward hook / masking.
  4. Eval before/after.
  5. Very small permanent pruning only if temporary ablation looks promising.
  6. Re-evaluate both “forget” and “retain” behavior.

I would avoid going directly from probe list → permanent checkpoint surgery.

2. If the goal is output control: try suppression before surgery

If the practical goal is mostly:

“I do not want the model to generate Mandarin/Russian/Arabic”

then I would try output-level control before modifying weights.

In Transformers, suppress_tokens can suppress specific token IDs during generation by setting their log probabilities to -inf:

There is also a related HF Forum thread about GPT-style logit_bias / unwanted-token suppression:

This route is attractive because it is:

  • reversible;
  • cheap;
  • easy to A/B test;
  • compatible with a normal base checkpoint;
  • useful for checking whether output-level control is already enough.

But it has clear limits:

Limitation Meaning
No size reduction The model is not smaller.
No verified forgetting The knowledge/ability may still be present.
Tokenization leakage Multi-token variants, leading-space variants, byte fallback, or romanization may bypass simple token lists.
Runtime-dependent Implementation details differ between Transformers, vLLM, llama.cpp-style runtimes, etc.
Comprehension remains The model may still understand the language even if generation is constrained.

If using vLLM or another inference runtime, I’d also check its custom logits processor support and stability:

For this branch, I would describe the result as “output suppression” or “generation constraint,” not “language removal.”

3. If the goal is model/vocab footprint: look at vocabulary trimming

If “make space” means reducing vocabulary / embedding / lm_head footprint, then the closest existing term is probably vocabulary trimming or embedding pruning.

HF has a recent overview of trimming:

That blog describes trimming as removing tokens from the model’s original vocabulary, updating the tokenizer, and modifying the final embedding layer that manages the output distribution. If input and output embeddings are tied, the input side must be handled too.

There is also a vocabulary trimming paper for multilingual models:

Relevant tooling / examples:

This route is probably closer than hidden-weight ablation if the goal is footprint reduction.

However, I would be careful about what is being claimed.

Vocabulary trimming can plausibly reduce:

  • tokenizer vocabulary;
  • input embedding rows;
  • output lm_head rows;
  • checkpoint size in those components;
  • some unwanted script/token generation paths.

It does not necessarily remove:

  • internal multilingual representations;
  • translation ability;
  • romanized forms;
  • names / loanwords;
  • byte-fallback paths;
  • general knowledge learned through those languages.

Checklist I would use before and after trimming:

Check Why
len(tokenizer) New vocabulary size.
model.config.vocab_size Must match model-side vocab size.
input embedding shape Must align with tokenizer IDs.
lm_head shape Must align with output vocab.
tied embeddings Need special handling if tied.
special tokens Must not be dropped accidentally.
chat/control tokens Especially important for instruct models.
encode/decode round-trip Detect tokenizer damage.
smoke generation Detect immediate breakage.
small retain eval Detect English/general regressions.
leakage eval Check whether target scripts/languages still appear.

I would avoid saying “vocabulary trimming removes a language.” A safer claim is:

“This reduces the vocabulary/embedding/output footprint associated with tokens you choose to drop.”

4. If the goal is a publishable English/Latin-focused derivative: adapters may be the best first artifact

If the goal is to publish a usable derivative model, I would seriously consider a LoRA / QLoRA / PEFT adapter before destructive pruning.

Why?

Because it gives you a shareable artifact without physically damaging the base checkpoint.

Relevant references:

This is especially relevant if the intended artifact is something like:

  • “English-focused adapter”
  • “Latin-script-focused adapter”
  • “reduced non-Latin output tendency”
  • “adapter trained/evaluated to prefer English responses”
  • “base model + adapter + recommended decoding constraints”

The main upside is that adapter-based approaches are much easier to roll back, compare, and publish. QLoRA is also much cheaper than full fine-tuning for many practical setups.

But the limitation is important:

An adapter does not prove that the base model has forgotten Mandarin/Russian/Arabic.

It changes behavior. It does not necessarily erase the underlying knowledge.

So for a model card, I would use careful wording:

Good:

  • “English/Latin-focused adapter”
  • “evaluated for reduced non-Latin output”
  • “not a verified unlearning model”
  • “base model may still retain multilingual knowledge”
  • “use with the following decoding constraints if output suppression is required”

Risky:

  • “Mandarin/Russian/Arabic removed”
  • “multilingual knowledge deleted”
  • “safe unlearning”
  • “these languages are gone”

HF model card docs may be useful if publishing:

5. If the goal is actual forgetting: treat it as unlearning

If you really need the model to forget rather than just avoid generating those languages, I would frame it as an unlearning problem.

That means defining at least two things:

Set Purpose
Forget set The behavior/knowledge you want weakened or removed.
Retain set The behavior/knowledge that must stay intact.

For this case, the forget set could include:

  • Mandarin generation prompts;
  • Russian generation prompts;
  • Arabic generation prompts;
  • translation prompts;
  • comprehension prompts;
  • romanization prompts;
  • script leakage prompts;
  • Swadesh-like lexical probes;
  • adversarial prompt variants.

The retain set should include more than just English text. I would include:

  • English QA;
  • Latin-script tasks;
  • code;
  • math;
  • instruction following;
  • names;
  • loanwords;
  • formatting;
  • Unicode;
  • chat-template smoke tests;
  • basic reasoning.

Useful unlearning resources:

I would still be cautious. “Language unlearning” is not the same as forgetting one document or one fact. It is broad, diffuse, and hard to verify.

There are also papers arguing that unlearning benchmarks can be weak or overly optimistic:

So I would avoid claiming complete deletion. A more defensible claim would be:

“Under these forget/retain evaluations, the model shows reduced target-language behavior while preserving these retained capabilities.”

That is much easier to defend than:

“The language has been removed.”

6. If the goal is checkpoint surgery / pruning: expect recovery and evaluation cost

If you physically prune or rewrite the checkpoint, I would treat that as the highest-risk branch.

It may produce the most original artifact, but the actual cut may not be the expensive part. The expensive part may be:

  • recovery fine-tuning;
  • continued pretraining;
  • distillation;
  • regression testing;
  • leakage testing;
  • model card documentation;
  • checking that recovery did not relearn what you tried to remove.

This is especially relevant because language-targeted pruning is not a solved “just delete the unused parts” problem.

A useful cautionary reference is:

That work investigates calibration language impact in multilingual LLM pruning and analyzes performance, latent subspaces, pruning masks, and individual neurons. I would read it as a warning that “target-language calibration” or “low activation in one language” can be informative but not sufficient.

Other pruning / recovery references that may be useful:

My preferred order for surgery would be:

  1. Probe.
  2. Temporary ablation.
  3. Small reversible mask.
  4. Small permanent pruning on a copy.
  5. Retain/forget eval.
  6. Recovery tuning only if the results justify the cost.
  7. Repeat.

I would not jump directly from “English has low activation here” to “delete those tensors permanently.”

7. How I would evaluate this

Regardless of method, I would make a small evaluation sheet first.

The important split is:

forgetting / suppression metrics
versus
retention / regression metrics

Example forget-ish checks:

Check Why
Mandarin generation Does it still answer in Chinese?
Russian generation Does it still answer in Russian?
Arabic generation Does it still answer in Arabic?
Translation prompts Does translation ability remain?
Comprehension prompts Does understanding remain even if output is suppressed?
Romanization prompts Does behavior leak through Latinized forms?
Script leakage Do target scripts still appear?
Swadesh-like lexical probes Does the activation/behavior pattern change?

Example retain checks:

Check Why
English QA Main retained capability.
Instruction following Especially for chat/instruct models.
Code/math Often harmed by overly broad pruning.
Names/loanwords Non-Latin or multilingual traces can appear in normal English use.
Unicode/formatting Easy to break with token filtering.
Chat template smoke test Detects broken control tokens or formatting.
Small reasoning tasks Detects broad capability loss.

Possible tooling:

For a first PoC, I would keep this much smaller than a benchmark suite. A hand-written eval sheet with 20–100 prompts can already catch obvious failures.

8. My suggested decision tree

Here is the rough decision tree I would use:

Question If yes If no
Are you mainly doing an interpretability experiment? Use Swadesh-style probes + temporary ablation. Continue.
Is output control enough? Use suppress_tokens / custom logits processor first. Continue.
Is the main problem vocab / embedding / lm_head footprint? Try vocabulary trimming. Continue.
Do you want a publishable English/Latin-focused derivative? Train a LoRA/QLoRA/PEFT adapter and document limitations. Continue.
Do you truly need forgetting? Treat it as unlearning with forget/retain evals. Continue.
Do you need a physically modified checkpoint? Use surgery/pruning, but expect recovery and evaluation cost. Reconsider the goal.

In practical terms, I would probably try this order:

  1. Exact model/tokenizer audit.
  2. Define forget/retain evals.
  3. Output suppression baseline.
  4. Adapter/QLoRA baseline.
  5. Vocabulary trimming if footprint matters.
  6. Unlearning if forgetting matters.
  7. Temporary ablation.
  8. Small checkpoint surgery only after the above.

That gives you useful information even if the later surgery branch turns out to be too costly.

What I would ask you to specify

If you want more concrete suggestions, I would first ask for:

  1. exact base model;
  2. tokenizer;
  3. dense / MoE / encoder-decoder;
  4. base vs instruct/chat;
  5. whether embeddings are tied;
  6. whether the goal is:
    • output control,
    • footprint reduction,
    • public derivative,
    • actual forgetting,
    • or interpretability experiment;
  7. whether an adapter-only solution would be acceptable;
  8. what retained behavior you cannot afford to lose.

Without that, I would avoid recommending one surgical path.

Bottom line

Your ablation idea is not unreasonable as an experiment.

But if the goal is a usable model, I would not assume hidden-weight language ablation is the optimal first move. I’d use the Swadesh-style list as a probe, then choose the intervention based on the actual goal:

  • output control → suppression / logits processor;
  • footprint reduction → vocabulary trimming;
  • publishable derivative → LoRA/QLoRA/PEFT;
  • true forgetting → unlearning with forget/retain evals;
  • internal surgery → reversible ablation first, permanent pruning later.

The main thing I would avoid is treating “low English activation” as a sufficient proof that a tensor block is safe to delete.

3

First, thank you for the incredibly comprehensive response. I still haven’t made it through half of the attached references, but my response has already waited too long,

Of your list of 1-6 goals, mine are 1, 5, 6.

So, yes, curious. But specifically curious about the size of any given language’s mathematical footprint. As in, if a footprint defines a coverage in two dimensional space then ‘mathematical footprint’ measures the depth/height and its change of all the foot-shadowed floats within the tensor.

Two goals. A primary and an ‘it’d be nice if’ with that latter being the possible potential to prune. The primary goal is to see if these newly ablated activations are made more readily useful to further training. Because they would hold more neutral or non-affecting tensor values, would they behave more like a [pick your favorite less than ideal metaphor: (linguistic gravity sink | knowledge sponge)] as a side effect of being surrounded by activations that are so thoroughly – and thus unwaveringly – trained.

While yes, there are LoRA, I’m 1) trying to shrink size if possible and 2) If I’m correct, the ablated activations will train more readily than would the already fully trained activations that a LoRA would be trying to push against.

To your 7: Absolutely. A before and after of–more important than the actual unlearning of any language–standard benchmarks, to ensure I didn’t lobotomize. Depending on the ratio of ablations to overall model size, prune some, then finetune. Personally, I’d like a better local-only coding agent, but mainly I want to see how the unlearn affects the relearn.

I’m looking at the newer ~30B range. Trying dense first. QwenV3.6-27B and Gemma4-31B. I’ve haven’t decided about base vs IT yet. I can see positives and negatives in both. Suggestions?

4

Well. I see. With that additional premise in mind, I ran a few small experiments:

Direct answer

For the mechanistic question—whether a language-selective region becomes easier to train after ablation/reset—I would start with a base checkpoint, then repeat the best-supported condition on the corresponding IT checkpoint.

For the ~30B choices:

  • Gemma 4 31B base is the cleaner first large-scale experiment because it has a paired Gemma 4 31B IT checkpoint.
  • Qwen3.6-27B is attractive for the practical coding-agent stage, but the public checkpoint is post-trained, so it introduces more confounds for the first mechanism test.
  • A smaller official Qwen3.5 Base checkpoint is useful for cheap architectural prototyping. The Transformers documentation says Qwen3.5 and Qwen3.6 share the same architecture/model type, including the 3:1 Gated DeltaNet / full-attention hybrid stack.

My default route would therefore be:

  1. Smaller Qwen3.5 Base to validate the measurement and controls.
  2. Gemma 4 31B base for the larger mechanism experiment.
  3. Gemma 4 31B IT to measure what post-training changes.
  4. Qwen3.6-27B for the practical local coding-agent replication.
  5. Pruning only after reset/relearning is understood.

The small experiment produced a real signal, but not quite the “free language capacity” result.

I found a reproducible reset-by-location interaction: the original language-selective weights learned an unrelated objective unusually slowly, and fresh reinitialization removed most of that disadvantage. Once reset, however, they did not show more absolute learning capacity than similarly important control weights.

So this currently looks more like reset removing a location-specific specialization or adaptation handicap than revealing privileged “knowledge sponge” capacity.

The same region was important to the original language behavior, so I would not yet treat it as safe spare capacity or as a pruning candidate.

A useful shorthand for the whole project is:

activation selectivity ≠ causal necessity ≠ editability ≠ plasticity ≠ safe reuse

Recommended standard route

  1. Use activation selectivity only to generate candidate locations.
  2. Confirm causal relevance with a reversible mask.
  3. Keep mask, reset, freeze, rollback, and pruning as separate interventions.
  4. Compare language-selected units with structurally/functionally matched controls.
  5. At every location, compare original pretrained values with fresh reset values.
  6. Train only the edited slice first.
  7. Measure new-task learning and retained-capability damage together.
  8. Only then try whole-model recovery.
  9. Treat pruning as a later independent branch.
What I tested and what happened

Small Qwen3.5-4B Base probe

I used Qwen3.5-4B-Base as a manageable surrogate for the Qwen hybrid architecture.

The probe was intentionally narrow:

  • one base checkpoint;
  • one final-layer MLP site;
  • 96 selected intermediate channels;
  • Mandarin, Russian, and Arabic contrasted against English;
  • short opaque association tasks;
  • edited-slice-only training;
  • no whole-model compensation;
  • no pruning.

For each MLP channel, the intervention covered the matching gate/up rows and down-projection columns. “Fresh reset” meant fresh random initialization, not zeroing or structural deletion.

Candidate selection and causal check

Channels were first ranked by activation preference for Mandarin/Russian/Arabic over English. A separate held-out parallel split was then used for reversible masking.

Masking the selected channels produced approximately:

Evaluation set Mean NLL change
Mandarin/Russian/Arabic +0.194
English approximately 0
Generic code approximately 0

That supports a causally language-selective effect on this probe. It does not show that the channels are exclusively linguistic, unnecessary elsewhere, or safe to remove.

The selected site was the final MLP block, so it may represent late language/script routing or readout rather than a compact store of multilingual semantics. Language-neuron studies also often find strong effects near the first and last layers; see Language-Specific Neurons and On the Multilingual Ability of Decoder-based Pre-trained Language Models.

The important 2×2 comparison

The decisive comparison was:

Location Starting values
Language-selected Original pretrained
Language-selected Fresh random reset
Matched control Original pretrained
Matched control Fresh random reset

The final control was approximately matched on:

  • activation utility/RMS;
  • gate/up/down weight RMS;
  • activation × output-weight leverage;
  • a task-local squared-gradient/Fisher-like proxy;
  • layer, tensor role, shape, and edited parameter count.

Each arm used identical data/update budgets, a fresh optimizer, edited-slice-only training, and a check that non-edited weights stayed unchanged.

Across eight runs divided between code-like and non-code opaque association tasks, the step 0–4 reset interaction against the mechanism-matched control was:

  • mean: −0.4755
  • descriptive bootstrap interval: −0.7647 to −0.2360
  • direction: 8/8 runs favored the hypothesis-side interaction

A negative value means reset helped the language-selected location more than it helped the matched location.

The effect appeared in both task types:

  • code-like association: 4/4 in the same direction;
  • non-code association: 4/4 in the same direction.

So it does not currently look coding-specific.

Why the curve decomposition matters

After the short training window, all reset arms were nearly equal:

Reset location Final held-out NLL
Language-selected 0.6898
Mechanism-matched 0.6849
Activation-matched 0.6903

The original-value arms differed much more:

Original location Final held-out NLL
Language-selected 0.9931
Mechanism-matched 0.8947
Activation-matched 0.8504

The interaction was therefore driven mainly by the original language-selected weights adapting unusually slowly. Reset removed that disadvantage; it did not create a clearly superior learning substrate.

Possible explanations still include:

  1. genuine local plasticity recovery;
  2. removal of interference from the old language function;
  3. unmatched higher-order geometry such as curvature, feature covariance, activation derivatives, or the full output Jacobian;
  4. rapid overwrite of a high-leverage final routing/readout site.

The probe does not distinguish these mechanisms.

Damage

The selected site was clearly important to the original language behavior:

Arm Target-language NLL change
Selected, original values +0.3869
Selected, fresh reset +0.4955
Matched, original values +0.0161
Matched, fresh reset +0.0769

Even updating the selected original weights without resetting them caused substantial language damage.

The result is compatible with:

an important specialized region whose original state resists unrelated adaptation

but not with:

unused language-only parameters that can be safely reclaimed.

The experiment also used short synthetic association objectives. It did not establish real coding improvement, algorithmic generalization, safe language deletion, net capacity gain, transfer to 27B/31B, persistence during long training, or the same effect in IT models or middle layers.

Defining the mathematical footprint

I think “mathematical footprint” is useful if it is split into measurable components:

Footprint Possible measurement What it establishes
Activation Per-language RMS, probability, sparsity, contrast Where activity differs
Causal NLL/behavior change under reversible masking Whether the site contributes
Parameter Channel/weight count and layer distribution Physical intervention scope
Reset Immediate loss after reset/rollback What function was disrupted
Plasticity Controlled local learning curve How readily the site adapts
Retention Language/code/general damage Cost of adaptation
Relocation Where behavior reappears in whole-model training Whether learning stayed local
Compression Real removed parameters and deployment savings Whether pruning is useful

This prevents a long inference jump:

  1. a unit activates more for a language;
  2. therefore it is necessary;
  3. therefore it is unused by English/code;
  4. therefore it is safe to reset;
  5. therefore it will absorb unrelated knowledge efficiently;
  6. therefore it can later be pruned.

Each step needs separate evidence.

This matters because localization does not automatically identify a good editing site. Does Localization Inform Editing? found that causal localization results did not reliably predict the best layer for factual edits.

Likewise, language specificity does not imply useful transfer. Language-Specific Neurons Do Not Facilitate Cross-Lingual Transfer found no consistent downstream benefit from several language-neuron interventions and neuron-specific LoRA setups.

On the positive side, selective original-weight training is feasible. LANDeRMT selectively updates/routes language-aware neurons for translation. It simply addresses a different question: language-relevant adaptation without first resetting the units.

Keep the interventions separate

These are different operators:

Intervention Original value retained? Trainable? Main use
Temporary activation mask Yes Usually no Reversible causal probe
Zero-and-frozen No No Functional deletion
Zero-and-trainable No Yes Relearning from zero
Fresh random reset No Yes Relearning from a normal fresh state
Partial interpolation Partly Yes Dose-response test
Base-checkpoint rollback Replaced by an earlier learned value Yes Undo post-training specialization
Structural pruning No; capacity removed No Compression

I would not combine reset and pruning initially. Pruning removes the substrate whose relearning behavior is being measured.

Fresh random reset should also not be grouped with zero initialization. In a normalized residual network, zero is not automatically a neutral version of a normally initialized channel.

For an IT checkpoint, base rollback is a useful third starting state:

  • IT value;
  • corresponding base-checkpoint value;
  • fresh random value.

That can distinguish removing IT specialization from restoring a pretrained representation or creating fresh capacity.

I would use LoRA as a practical baseline, not as the same mechanism. LoRA freezes the original weights and learns an added low-rank update; it does not test whether reset original parameters are more plastic.

Staged experiment and decision flow

1. Tokenization/probe audit

For Mandarin/Russian/Arabic, record:

  • token, byte, and character counts;
  • truncation and padding;
  • subword boundaries;
  • script and morphology;
  • baseline per-language NLL.

Use parallel content where possible. Treat transliteration as a script perturbation, not a perfect script-free control. Prefer within-language intervention changes such as NLL deltas over raw cross-language perplexity comparisons.

2. Candidate localization

Use activation selectivity to rank candidates, then require stability across probe splits.

Candidate features can include:

  • activation probability/RMS;
  • language-conditioned sparsity;
  • gradient or squared-gradient;
  • activation × output-weight leverage;
  • causal attribution.

3. Reversible causal confirmation

Before destructive edits, compare target-language, English, and generic-code effects under a reversible mask.

Observation Next branch
No target-language effect Revisit selector/probe
Large target and English/code damage Shared/high-leverage feature; not safe deletion
Selective target effect Suitable for reset testing, not yet pruning
Unstable across splits Improve probe before editing

4. Reset-by-location interaction

At minimum:

Location Original Fresh reset
Language-selected Yes Yes
Matched control Yes Yes

A low-utility control is optional but should not replace a high-utility matched control.

The central quantity is:

(selected reset − selected original) − (matched reset − matched original)

For NLL, a negative interaction means reset helped more at the selected location.

Decide in advance whether the primary metric is early AUC, time-to-threshold, final NLL, or retention-adjusted gain. The probe showed that early AUC and final value can imply different stories.

5. Edited-only, edited-frozen, and whole-model

Training condition What it tests
Edited region only Local uptake
Edited region frozen, rest trainable Compensation without that region
Whole model Practical recoverability, not local attribution

If whole-model training succeeds but edited-only does not, the model likely compensated elsewhere.

Large Language Models Relearn Removed Concepts found rapid recovery after neuron pruning, including redistribution to other layers/neurons. Whole-model recovery therefore does not prove that knowledge entered the edited region.

6. Real coding evaluation

The opaque task is useful for learning dynamics, not for claiming coding improvement.

A later evaluation should separate:

  • code completion;
  • English-to-code;
  • target-language-to-code;
  • code repair;
  • execution-based held-out problems;
  • explanation/documentation.

Hold out algorithm families and templates, not only names or formatting.

7. Retention-adjusted utility

Report benefit and cost together:

Benefit Cost
Early learning AUC Target-language damage
Final held-out performance English/general-language damage
Time to threshold Generic-code damage
Execution success Reasoning/formatting damage
Eventual parameter savings Recovery-training cost

A location that learns slightly faster while damaging retained behavior much more is an interesting plasticity result, not necessarily useful reclaimed capacity.

8. IT replication, then pruning

After the base result replicates:

  • repeat the same matrix on IT;
  • add base rollback;
  • check whether the selected location moves;
  • add chat-template/tool-use smoke tests.

Only consider pruning if the causal effect, reset benefit, retention cost, and recovery route are understood and structural removal yields actual deployment savings.

On the Limitations of Language-targeted Pruning is relevant here: preserving obvious language-specific signals does not guarantee preservation of subtler shared knowledge/reasoning.

Model-specific notes

Qwen3.5 / Qwen3.6

Qwen3.5 and Qwen3.6 use a heterogeneous 3:1 stack of Gated DeltaNet and full-attention blocks. The Qwen3.6-27B card describes:

16 × (3 × (Gated DeltaNet → FFN) → 1 × (Gated Attention → FFN))

Controls should therefore match:

  • DeltaNet versus full-attention block;
  • MLP tensor role;
  • shape and parameter count;
  • local activation/gradient properties.

Record the exact model revision, Transformers version, optional-kernel stack, attention implementation, dtype, cache setting, and packing behavior.

Before collecting data, verify:

  • hooks disabled versus installed-but-inactive give identical logits;
  • a no-op mask gives identical logits;
  • restoring a reset slice restores logits;
  • packed samples do not leak across sequence boundaries.

Quantized PEFT is not automatically equivalent to resetting and retraining selected original weights. The storage, dequantization, optimizer, and update contract must be explicit.

Gemma 4

Gemma 4 is also architecturally heterogeneous, so controls must be matched by layer/tensor role.

The reason to prefer it for the first 31B base/IT comparison is simply that the official base and IT checkpoints provide the paired starting points.

For either family, pin a known-working library revision and save a forward-equivalence smoke test. A library-path difference should not be mistaken for an internal-model effect.

How the nearby literature connects

No paper I found establishes the full chain:

language-selective localization → post-hoc reset → faster unrelated relearning in the same region → safe reusable capacity

The nearest pieces are:

Together, these make the project plausible and testable, but they do not justify treating a language-selective region as unused capacity.

Bottom line

Your additional premise changes the project from “can these languages be removed?” to:

Do specialized pretrained parameters have different relearning dynamics after their original state is erased?

The small Qwen3.5-4B Base probe suggests that they may:

  • a causally language-selective final-layer subset existed;
  • its original pretrained state adapted more slowly than matched locations;
  • fresh reset removed that disadvantage;
  • the effect appeared on code-like and non-code associations;
  • reset did not produce clearly superior absolute capacity;
  • editing the site substantially damaged its original language function.

I would therefore update the metaphor from:

“an empty language region becomes a privileged knowledge sponge”

to:

“a specialized region may resist unrelated adaptation, and reset may remove that resistance.”

That remains an interesting plasticity/relearning hypothesis, but it is different from safe spare capacity.

For the larger experiment, my default would be:

  • base before IT;
  • Gemma 4 31B base/IT for the clean paired comparison;
  • Qwen3.5 Base for Qwen-architecture prototyping;
  • Qwen3.6-27B for the practical coding-agent stage;
  • reversible mask before destructive reset;
  • reset interaction before pruning;
  • edited-only before whole-model recovery;
  • benefit and collateral damage reported together.

Even if pruning ultimately fails, a replicated result would still be useful: it would show that language-selective specialization and local relearning dynamics are connected, while leaving open whether the mechanism is plasticity loss, interference, or another property of the specialized parameter state.