A carregar...
Publicação
Discord, conflict and contradiction in AI : a benchmark for learning with contradictory facts
| Nome: | Descrição: | Tamanho: | Formato: | |
|---|---|---|---|---|
| 1.32 MB | Adobe PDF |
Autores
Orientador(es)
Resumo(s)
The amount of information present in online sources has been growing rapidly over the preceding decades. This growth in available information has led to contradictory statements becoming more prevalent in digital resources, but their true incidence and impact is unknown. Knowledge Graphs are commonly used as sources to augment Language Models since they can provide a semantic background, but can also contain an unknown quantity of contradictory information. Artificial Intelligence models trained on these sources usually resolve contradictions by adopting the most common viewpoint, which can hide other perspectives or even preserve misconceptions. Benchmarks that consider contradictory facts are essential to foster the development of more robust artificial intelligence models. This thesis introduces benchmarks for evaluating how language models handle contradictory information derived from structured sources. Two datasets from the biomedical domain and general knowledge, the Gene Ontology and Wikidata, respectively, were analyzed to detect contradictions using reasoning. These contradictions were then used to build benchmark datasets (using three contradiction types and building both explicit and implicit contradiction versions of the datasets) for a binary classification task that evaluated the ability of four language models (Qwen2-7b-Instruct, Qwen3-32b, Deepseek-R1 and Gemini 2.5 Flash) to determine if a set of statements were contradictory or not. Experimental results demonstrate that while language models perform reasonably well in detecting explicit contradictions, their performance drops significantly when handling implicit contradictions, highlighting key limitations in current language models reasoning abilities. This work provides an important step in improving the reliability and inclusivity of language models when presented with contradictory information, supporting the development of more contradiction-aware artificial intelligence architectures.
Descrição
Tese de Mestrado, Ciência de Dados, 2025, Universidade de Lisboa, Faculdade de Ciências
Palavras-chave
Knowledge Graphs Contradiction Language Models Benchmark Datasets Artificial Intelligence