A NOVEL DATA QUALITY ASSESSMENT FRAMEWORK: THE CASE STUDY OF DEVELOPING AN AGENT-BASED ONTOLOGY METHODOLOGY FOR VALIDATION ROBUSTNESS OF DIGITAL TWINS

With the vast amount of data being generated nowadays, data quality (DQ) has become a critical success factor for any organisation. Additionally, the digitalisation of processes, offers the possibility to implement intelligent automation practices to dynamically conduct repetitive tasks. The automation of processes aided by Artificial Intelligence (AI) is a powerful tool to build intelligent systems that can make reasoning, find patterns, structure knowledge based on insights that would otherwise remain unrevealed, and finally support decision-making processes. Even though data quality the literature on DQ is extensive at a conceptual level, from a practical perspective, this topic is usually considered as series of tasks to be performed in data pre-processing steps. Currently, the primary requirement is to understand the data available and detect DQ issues. To address these challenges, this research work presents a structured framework to standardise the process of assessing the quality of data and modelling the knowledge obtained from such an assessment by implementing an ontology. The main steps of the framework are: (i) identify user’s requirements; (ii) perform a data quality assessment (DQA) through measurement of the level of quality of data considering DQ issues, dimensions and their metrics, and evaluation of this level with respect to an acceptability threshold; and (iii) capture the knowledge from the DQA results using an ontology that models the DQA insights in a standard reusable way. Following the proposed framework, an Excel-based tool is utilised to capture data user’s requirements. Python is used to design specific DQ dimension-based scripts, and as key role of core component to streamline the developed step-by-step process into an integrated DQ information pipeline. The concept of Agents is introduced in DQ and an ontology about DQ, manipulated with owlready2 in Python, provides a standard structure to model the DQ insights obtained from the assessment, while it is frequently updated to enrich captured knowledge, reducing time and costs for future projects. An industrial case study, using operational data of a Digital Twin simulation, is employed to validate the proposed ontological framework and tool; the results show a well-structured guide that can effectively assess DQ and model knowledge providing a digital representation of DQ issues.