DEVELOPMENT OF THE IMAGE ANALYSIS MODULE OF A PERSONAL ASSISTANT WITH KNOWLEDGE-AWARE REASONING CAPABILITIES

Current personal assistants are unable to answer complex queries that require crossreferencing user data fragmented across different apps and databases, such as photos, events, notes, etc. This thesis work proposes an architecture to overcome this limitation, centered on a Personal Knowledge Graph (PKG) that dynamically unifies personal data sources into a single database and can be queried very simply. Furthermore, an advanced Visual Analysis module is proposed to enable complex visual questions about images. An orchestrator equipped with an LLM populates the graph with a single user’s personal data, while a system of agents can query it using GraphRAG to provide a structured context to the LLM, anchoring responses to real data and mitigating hallucinations. If the user’s question necessitates advanced reasoning over one or more images, the Visual Analysis module would be invoked to improve the system’s performance. The evaluation demonstrates that visual analysis effectively fills informational gaps (over 82% positive responses) and that the multimodal integration is robust.