PROBING THE HIDDEN STATES OF MEDICAL LANGUAGE MODELS: A LATENT-SPACE ANALYSIS OF FACTUALITY, JUDGMENT, AND STABILITY

Large Language Models (LLMs) are increasingly being explored for medical applications, yet their tendency to produce plausible but factually incorrect information raises important concerns about their reliability. While factuality is commonly assessed through generated responses, recent research suggests that information related to truthfulness may also be encoded in the models' internal representations. However, the interpretation of such signals remains uncertain: do hidden-state probes recover objective factuality, or do they primarily reflect what the model itself judges to be true? This thesis investigates this distinction by examining the latent representations of MedGemma 1.5 4B, a language model specialized in the medical domain. To support a controlled evaluation, a benchmark of factual and counterfactual medical statements is constructed from expert-annotated clinical knowledge. Counterfactual statements are generated through a near-miss procedure that substitutes medically salient entities with clinically plausible but incorrect alternatives, while preserving linguistic structure and sentence length. The resulting benchmark undergoes physician validation to strengthen the reliability of its factual annotations. The experimental analysis addresses three complementary research questions. First, it investigates whether factuality representations remain stable across different prompt-induced processing contexts, termed mindsets. By evaluating hidden-state probes across multiple network layers and semantic prompting families, the study demonstrates that factuality decodability depends on the conditioning context. Geometric analyses further reveal that representations induced by paraphrased prompts within the same semantic family tend to exhibit greater consistency than those associated with different semantic framings, suggesting that prompting influences the organization of factual information in the latent space. Second, the thesis examines whether factuality probes distinguish objective truth from the model's own judgment. Logistic regression, mass-mean, and nonlinear SAPLMA probes are evaluated separately on statements where the model's predictions agree or disagree with ground truth. The results reveal a systematic reversal in probe behavior when the model is incorrect: despite strong performance on statements where model judgment and factual labels coincide, probe predictions tend to follow the model's erroneous assessment rather than recover objective factuality. The consistency of this phenomenon across different probing methods suggests that aggregate performance can conceal a substantial dependence on the model's internal judgment. Third, the study investigates whether hidden representations encode the stability of the model's judgments across repeated stochastic generations. A Flip Predictor is introduced to anticipate response instability directly from pre-generation hidden states, without requiring factual labels. The findings demonstrate that behavioral instability can be predicted from internal representations, providing a complementary signal for characterizing model reliability. However, stable judgments are not necessarily correct, and models may consistently reproduce factually incorrect answers. Overall, the findings highlight the distinction between objective factuality, model-relative judgment, and behavioral stability. Although hidden-state representations encode information about these properties, their decodability does not guarantee factual correctness or establish a causal role in the model's predictions. This thesis provides a framework for critically evaluating representation-based monitoring methods and emphasizes the limitations of treating hidden-state probes as standalone truth detectors in medical applications.