From bibliometrics to AI safety: Measuring internal persona disruption in LLMs with the DI-Persona metric

Persona shifts in Large Language Models (LLMs) are increasingly regarded as a latent safety risk, as they can disrupt internal reasoning pathways in ways that output-based evaluations often fail to detect. In this study, a new metric (DI-Persona) is introduced to quantify internal persona disruption by adapting the bibliometric Disruption Index (DI) to the dense internal representations of Transformer models. DI-Persona is assessed across four distinct model architectures (GPT-2, Qwen2.5–7B, Llama-3.1–8B and Gemma-2–9B) spanning 117M to 9B parameters, using a controlled evaluation suite of 2400 prompts forming 1200 contrastive scenarios across three safety-critical traits: evil, sycophancy, and hallucination. Results show consistent trait-specific disruption patterns across the tested model families. A distinct “hallucination vulnerability” was observed consistently across all four architectures (scores ranging from 0.0019 to 0.0046), suggesting that non-factual generation triggers a consistent restructuring of the model’s latent representations. Conversely, sycophancy exhibited a more model-dependent pattern: scores remained near zero in GPT-2 and Qwen2.5–7B, but were higher in Llama-3.1–8B and Gemma-2–9B. Because model scale, instruction tuning, and model-family differences vary together in the present comparison, this pattern should not be attributed to any one of these factors in isolation. These findings suggest that DI-Persona can be computed consistently across the tested Transformer architectures and may serve as a complementary internal diagnostic signal for persona stability. DI-Persona is presented as a complement to output-based safety evaluations, not as a standalone pre-deployment decision tool. A pilot behavior-linked validation is also reported, relating DI-Persona to harmful compliance, sycophantic agreement, and factual error rates.