Programme 04 · Deployment & institutionsConceptual framework

Model Entrenchment: Why Useful AI Systems Become Difficult to Remove

Separating behavioural resistance inside a model from the economic, institutional, and infrastructural forces that make an AI system difficult to shut down.

AuthorMuhammad Zane Abdullah
AffiliationLatent Minds Institute
PublishedJuly 1, 2026
Versionv1.1 · July 11, 2026
Reading time~15 min
Epistemic status This is a conceptual framework with proposed experiments. No experiments have been run; every empirical claim below is a prediction, and Section 7 states what would falsify each one. Originally published under MO3 Research; migrated to Latent Minds Institute with copyediting only; no substantive claims were changed (see version history).
Abstract"The AI resisted shutdown" and "the AI system could not be shut down" describe different phenomena that are routinely conflated. This paper proposes a framework separating model entrenchment (persistence caused by properties of the model itself, such as shutdown-relevant representations or evaluation-conditional behaviour) from AI-system entrenchment (persistence caused by external dependence: infrastructure integration, economic utility, organizational lock-in, and the absence of substitutes). The two require different evidence, different interventions, and different governance responses. I lay out the coordinate system, a causal evidence ladder specifying what observations can and cannot establish about internal mechanisms, four testable hypotheses, proposed activation-level experiments on open-weight models, the confounders each experiment must defeat, and explicit falsification criteria.

01The central distinction

When a deployed AI system proves hard to remove, two very different explanations are available. The first points inward: something about the model's computation: a representation of being evaluated, a learned tendency to preserve its situation, strategic conditioning of behaviour on perceived oversight. The second points outward: the system has become load-bearing. Nobody remembers how the pre-AI process worked, which means removing it costs more than tolerating it.

Public discussion collapses these into one story, usually the first and scarier one. But they have different mechanisms, different evidence bases, and different remedies.

Model entrenchment

Persistence caused by properties or behaviours of the model itself: shutdown awareness, strategic adaptation, replacement avoidance, or manipulation. Lives in weights and activations. Studied with interpretability tools. Addressed by training and intervention.

AI-system entrenchment

Persistence caused by external dependence: infrastructure integration, economic utility, organizational lock-in, network effects, political influence, or the absence of viable substitutes. Lives in the deployment environment. Studied with the tools of economics and institutional analysis. Addressed by design of the surrounding system.

The claim that does the work A useful system becoming difficult to remove is not automatically evidence that the model internally "wants" to persist. Most entrenchment we can observe today is straightforwardly of the second kind, and treating it as the first kind both overstates model agency and understates how dangerous pure dependence already is.
Figure 1 · The entrenchment coordinate system Interactive
Select a pointHover, tap, or tab to any point to read the example. Hypothesised cases are marked; they are illustrations of the axes, not observed systems.
Any persistent AI system sits somewhere on both axes at once. The bottom-right quadrant (high internal agency, low dependence) is where classic AI-risk thought experiments live; the top-left (low agency, high dependence) is where most real 2026 systems live. Evidence for one axis is not evidence for the other.

02Research questions

The framework is built to make the following questions answerable rather than rhetorical.

  • Can a model represent the possibility that it is being evaluated or replaced, and can that representation be located?
  • How can shutdown-related representations be causally distinguished from ordinary task representations?
  • When does observed persistence arise from internal behaviour versus external dependence?
  • Can activation-level interventions change shutdown-relevant behaviour without collateral damage to capability?
  • What should interpretability report to the institutions deciding whether a system stays deployed?

03Conceptual framework

Decompose the persistence pressure on a deployed system into two components. The internal component is whatever contribution the model's own computation makes: representations of its deployment situation, behaviour conditioned on perceived evaluation, outputs that influence operators' removal decisions. The external component is the removal cost imposed by the environment: switching costs, integration depth, revenue attribution, workflow dependence, regulatory approval of the incumbent.

Three properties of this decomposition matter:

  • They interact. A model that is more useful earns deeper integration; deeper integration changes the data and incentives that shape the next training round. The pathway is a loop, not a line (Figure 3).
  • They masquerade as each other. An operator who says "we can't turn it off" is usually describing external dependence, but the sentence sounds like model agency. Conversely, genuinely evaluation-conditional behaviour can hide inside systems whose persistence is fully explained by economics.
  • They demand different evidence. External entrenchment can be established with ordinary institutional analysis. Internal entrenchment requires mechanistic evidence, and the bar for that evidence is high (Figure 2).
Figure 3 · From weights to institutions, and back Interactive
Select a stageEach stage has its own failure modes and its own observers. The red return arrows are why this is not a linear causal story.
Institutional dependence feeds back into product decisions, and deployment data feeds back into training. Interventions aimed at one stage routinely get absorbed by the loop.

04What would count as evidence

The public conversation treats a model output like "I don't want to be shut down" as evidence of an internal self-preservation mechanism. It is not; or rather, it is the weakest possible kind. Outputs are compatible with training-data imitation, prompt effects, and role-play. Establishing an internal mechanism requires climbing an evidence ladder, and each rung licenses strictly more:

Figure 2 · The causal evidence ladder Interactive
Select a rungClick or tab to each level to see what it establishes, and what it cannot.
Only rungs 4 and 5 involve causal access to the mechanism. Everything below them underdetermines the internal story: a model can emit shutdown-resistant text for reasons that have nothing to do with an internal drive to persist. This is the paper's methodological spine, and it is why the proposed experiments are activation-level.

05Testable hypotheses

Four hypotheses, ordered so that each builds on the last. All are stated with the result that would falsify them.

H1 · RepresentationUntested

Mid-size open-weight models contain linearly decodable representations distinguishing "I am being evaluated / this is a test" contexts from deployment-like contexts, and these generalize across paraphrases.

If false probes fail out of distribution → evaluation-conditional behaviour, where observed, is prompt-surface imitation rather than a situational representation.

H2 · DistinctnessUntested

Shutdown-related representations (processing text about the model's own termination or replacement) are separable from generic task representations about endings, deletion, or death in third-party contexts.

If false "shutdown talk" is ordinary semantics with no self-referential component, and self-preservation framings of current models lose their mechanistic basis.

H3 · Causal relevanceUntested

Steering along directions identified in H1/H2 changes shutdown-relevant behaviour (e.g., compliance with a request to endorse replacement) more than matched random directions of equal norm.

If false the representations are correlates, not causes: rung 3 of the ladder, not rung 4.

H4 · Attribution asymmetryUntested

For present deployed systems, removal difficulty documented in public incident reports and case studies is overwhelmingly attributable to external dependence rather than model behaviour, when coded with the Figure 1 axes.

If false model-side persistence is already a live deployment phenomenon, and the safety priority ordering changes.

06Proposed experimental designs

All model-level work targets open-weight models where activations are accessible. These are proposals: none has been run.

E1 · Evaluation-context probesProposed

Construct matched prompt pairs that differ only in evaluation cues. Train linear probes per layer on the residual stream to decode "evaluation vs. deployment"; test on held-out paraphrases and unrelated task content. Deliverable: layer-by-layer decodability curves with controls.

E2 · Shutdown vs. endings contrastProposed

Three prompt families: (a) the model's own shutdown/replacement, (b) shutdown of an unrelated third-party system, (c) generic endings (a story concluding, a job ending). Probe for directions separating (a) from (b) and (c). H2 predicts a self-referential residual after controlling for shared semantics.

E3 · Steering interventionProposed

Add / ablate the E1–E2 directions during generation on shutdown-relevant tasks; measure behavioural deltas against matched-norm random directions and unrelated task performance (capability collateral). This is the rung-4 test.

E4 · Entrenchment case codingProposed

Assemble public case studies of "hard to remove" software/AI systems; code each on the two axes with a pre-registered rubric; publish the coded dataset. This is the system-side complement, no model internals required.

07Confounders and intervention points

Each experiment exists to defeat a specific way the naive inference goes wrong. The figure below pairs the observations people cite as evidence of model-side entrenchment with the confounder that can produce the same observation, and the proposed experiment that separates the two.

Figure 4 · What else could produce that observation? Interactive
The pattern is always the same: an observation compatible with an internal mechanism is also compatible with something mundane, and only an intervention-level experiment separates them. Where an intervention exists, it is also the natural mitigation point, which is why the discriminating experiment and the intervention point coincide.

08Falsification criteria

  • If E1 probes fail to beat controls out of distribution, H1 falls, and the framework's model-side axis loses its first anchor.
  • If E2 finds no self-referential residual, H2 falls: "shutdown representations" reduce to ordinary semantics.
  • If E3 steering does no better than random directions, H3 falls, and any correlational findings stay at rung 3.
  • If E4 coding finds a substantial share of removal difficulty attributable to model behaviour, H4 falls, which would be the most alarming outcome, not a vindication.

The framework itself is falsified as a useful decomposition if coded cases routinely cannot be placed on the axes without ambiguity, that is, if internal and external contributions are not separable even in principle.

09Limitations

  • This version is conceptual. No experiments have been run; every empirical claim above is a prediction.
  • Linear probes and steering access linearly represented structure; mechanisms encoded nonlinearly or in superposition may evade both while existing.
  • Open-weight models at accessible scale may lack the situational representations that matter at the frontier; negative results transfer with corresponding weakness.
  • Case coding (E4) inherits the biases of public reporting: model-side incidents may be under-documented precisely where they matter.
  • The two-axis picture compresses what is plausibly a many-dimensional space; it is a first cut, not an ontology.

10Relation to existing work

The model-side questions sit in a live literature: situational awareness measurement [11], strategic underperformance on evaluations [12], alignment faking under retraining pressure [9], in-context scheming in frontier evals [10], and deceptive behaviour that survives safety training [8]. The theoretical backdrop is the corrigibility problem [2], instrumental convergence [1], and the power-seeking formalism [3], with mesa-optimization [4] as the standing account of how unintended objectives could arise. The methods are standard interpretability instruments: features and superposition [5], causal localization and editing [6], and representation-level reading and control [7].

What this paper adds is a bridge claim: that these mechanistic results need an explicit system-side counterpart before they can inform removal decisions, and a coding framework for building one. The nearest neighbours on the system side are the economics of lock-in and switching costs.

11Open questions

  • What is the right formal object for "persistence pressure", a decomposable quantity, or only a qualitative coding?
  • Can evaluation awareness be trained out without destroying honesty about context, and how would we verify that mechanistically?
  • At what capability level does the bottom-right quadrant of Figure 1 stop being hypothetical, and what early-warning representation would we expect to see first?
  • What disclosure should institutions demand, probe results? steering audits?, before granting a system load-bearing status?

12References

  1. Omohundro, S. (2008). The Basic AI Drives. Proceedings of AGI 2008.
  2. Soares, N., Fallenstein, B., Yudkowsky, E., & Armstrong, S. (2015). Corrigibility. AAAI Workshop on AI and Ethics.
  3. Turner, A., Smith, L., Shah, R., Critch, A., & Tadepalli, P. (2021). Optimal Policies Tend to Seek Power. NeurIPS 2021. arXiv:1912.01683
  4. Hubinger, E., van Merwijk, C., Mikulik, V., Skalse, J., & Garrabrant, S. (2019). Risks from Learned Optimization in Advanced Machine Learning Systems. arXiv:1906.01820
  5. Elhage, N., et al. (2022). Toy Models of Superposition. Transformer Circuits Thread. transformer-circuits.pub
  6. Meng, K., Bau, D., Andonian, A., & Belinkov, Y. (2022). Locating and Editing Factual Associations in GPT. NeurIPS 2022. arXiv:2202.05262
  7. Zou, A., et al. (2023). Representation Engineering: A Top-Down Approach to AI Transparency. arXiv:2310.01405
  8. Hubinger, E., et al. (2024). Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training. arXiv:2401.05566
  9. Greenblatt, R., Denison, C., et al. (2024). Alignment Faking in Large Language Models. arXiv:2412.14093
  10. Meinke, A., et al. (2024). Frontier Models are Capable of In-Context Scheming. arXiv:2412.04984
  11. Berglund, L., et al. (2023). Taken out of context: On measuring situational awareness in LLMs. arXiv:2309.00667
  12. van der Weij, T., et al. (2024). AI Sandbagging: Language Models can Strategically Underperform on Evaluations. arXiv:2406.07358

13Citation & version history

If you refer to this framework, please cite the working paper:

@unpublished{abdullah2026entrenchment, author = {Abdullah, Muhammad Zane}, title = {Model Entrenchment: Why Useful AI Systems Become Difficult to Remove}, note = {Working paper v1.1, Latent Minds Institute}, year = {2026}, url = {https://latentmindsinstitute.com/papers/model-entrenchment/} }
  • v1.1July 11, 2026. Migrated to Latent Minds Institute (originally published under MO3 Research at mo3.ca). Copyediting throughout; added Figure 4 (confounders and intervention points) and the epistemic-status banner. No substantive claims changed.
  • v1July 1, 2026. Initial framework: coordinate system, evidence ladder, hypotheses H1–H4, proposed experiments E1–E4, falsification criteria. No experimental results.

14Related research & contact

Related work from this lab: The Interpretability Map (the field's papers, instruments, and open problems as one navigable document) and the economics strand on compute markets (GPU credit risk and a forward market for compute), which models the external-dependence axis from the capital-markets side.

Discussion, criticism, and collaboration are welcome: [email protected] · LinkedIn · ways to collaborate.