EXPLORING THE IMPACT OF MULTI-MODALITY IN CONVERSATIONAL RECOMMENDER SYSTEMS AND SEARCH

This thesis investigates the impact of multi-modality on conversational recommender systems (CRSs) and search clarification tasks. Specifically, we built a food CRS, MMFCRS, that utilizes multi-modal signals, conversational strategies, and nutritional labels to encourage users to make healthy food choices. We introduce the concept of multi-modality to user-system interactions and recommendation explanations and investigate whether they impact user behavior. Our results show that using multi-modal explanations has a significant impact on conversation time and system effectiveness evaluation. The version of our CRS with multi-modal explanations drives users towards healthier choices without significant differences from other versions. In addition, we collect a novel multi-modal clarification dataset, MIMICS-MM, to explore the application of multi-modality in search clarification tasks. We investigate whether combining textual and visual content can assist the process of clarifying users search intent. We also address the challenge of automating the generation of multi-modal clarifications using text-to-image generation models. Our results show that users rely on multi-modal content to better contextualize search clarification panes, and image quality and relevance directly impact user behavior. Overall, our work promotes the use of multi-modality in CRSs and search clarification tasks, providing researchers with a better understanding of its power and potential impact.