AUTOMATIC PROMPT OPTIMIZATION FOR DISCOVERING RECOMMENDER SYSTEM DATASETS IN LARGE LANGUAGE MODELS

Large language models (LLMs) have been widely adopted for many machine learning tasks. However, a growing concern is that these models may rely on memorizing datasets encountered during training rather than learning generalizable patterns. Dataset memorization can lead to misleading experimental evaluations because strong performance may stem from data leakage rather than true generalization. In this study, we present an evolutionary-based methodology using GEPA (Genetic Pareto Algorithm) to investigate dataset memorization in LLMs. Our approach uses GEPA to automatically generate input queries and prompt structures that maximize the extraction of potentially memorized information. Framing memorization detection as a search and optimization problem, GEPA explores a space of candidate expressions designed to elicit specific data instances, attributes, or relationships from the target dataset. We evaluate memorization by measuring the overlap between the information generated by the model and the original dataset using coverage and similarity-based metrics. The evolutionary process discovers non-trivial query patterns that would be difficult to design manually and provides a systematic, model-agnostic probing mechanism. Our experimental results show that GEPA effectively reveals varying degrees of dataset memorization across different models and configurations. Overall, this study introduces GEPA as a novel, automated tool for auditing memorization in large models and contributes to more reliable evaluation practices and a deeper understanding of how training data influences model behavior.