CIRCUIT-LEVEL ANALYSIS OF FACTUALITY ENCODING IN TRANSFORMER ARCHITECTURES

Large Language Models (LLMs) have demonstrated exceptional capabilities in lots of domains and tasks but are fundamentally plagued by their propensity to "hallucinate" or generate non-factual information with a high degree of confidence. This unreliability is the greatest barrier to their widespread adoption in high-stakes, truth-critical domains. Recent foundational work has provided strong evidence that LLM’s internal hidden states do, in fact, encode a representation of a statement’s truthfulness and that this information can be extracted by a simple external classifier. However, these findings are correlational; they show that factuality is represented, but not how the model’s internal components compute this representation. This thesis bridges the gap between detection and explanation. We perform an additional step and move from external probing to applying principles of mechanistic interpretability to discover the specific computational mechanisms that underpin factuality assessment within a Transformer. Using mechanistic intepretability methods on a pretrained transformer model, we perform several circuit discovery experiments and extract a minimal sub-network, or "circuit" of attention heads and MLP neurons responsible for this computation. We then apply probing techniques to the internal hidden states extracted from this circuit, to demonstrate that the components defining it, are the ones responsible for factuality.