M3D: Multimodal MultiDocument Fine-Grained Inconsistency Detection
| dc.contributor.author | Tang, Chia-Wei | en |
| dc.contributor.committeechair | Thomas, Christopher Lee | en |
| dc.contributor.committeemember | Lourentzou, Ismini | en |
| dc.contributor.committeemember | Huang, Lifu | en |
| dc.contributor.department | Computer Science and#38; Applications | en |
| dc.date.accessioned | 2024-06-11T08:01:10Z | en |
| dc.date.available | 2024-06-11T08:01:10Z | en |
| dc.date.issued | 2024-06-10 | en |
| dc.description.abstract | Validating claims from misinformation is a highly challenging task that involves understanding how each factual assertion within the claim relates to a set of trusted source materials. Existing approaches often make coarse-grained predictions but fail to identify the specific aspects of the claim that are troublesome and the specific evidence relied upon. In this paper, we introduce a method and new benchmark for this challenging task. Our method predicts the fine-grained logical relationship of each aspect of the claim from a set of multimodal documents, which include text, image(s), video(s), and audio(s). We also introduce a new benchmark (M^3DC) of claims requiring multimodal multidocument reasoning, which we construct using a novel claim synthesis technique. Experiments show that our approach significantly outperforms state-of-the-art baselines on this challenging task on two benchmarks while providing finer-grained predictions, explanations, and evidence. | en |
| dc.description.abstractgeneral | In today's world, we are constantly bombarded with information from various sources, making it difficult to distinguish between what is true and what is false. Validating claims and determining their truthfulness is an essential task that helps us separate facts from fiction, but it can be a time-consuming and challenging process. Current methods often fail to pinpoint the specific parts of a claim that are problematic and the evidence used to support or refute them. In this study, we present a new method and benchmark for fact-checking claims using multiple types of information sources, including text, images, videos, and audio. Our approach analyzes each aspect of a claim and predicts how it logically relates to the available evidence from these diverse sources. This allows us to provide more detailed and accurate assessments of the claim's validity. We also introduce a new benchmark dataset called M^3DC, which consists of claims that require reasoning across multiple sources and types of information. To create this dataset, we developed a novel technique for synthesizing claims that mimic real-world scenarios. Our experiments show that our method significantly outperforms existing state-of-the-art approaches on two benchmarks while providing more fine-grained predictions, explanations, and evidence. This research contributes to the ongoing effort to combat misinformation and fake news by providing a more comprehensive and effective approach to fact-checking claims. | en |
| dc.description.degree | Master of Science | en |
| dc.format.medium | ETD | en |
| dc.identifier.other | vt_gsexam:40024 | en |
| dc.identifier.uri | https://hdl.handle.net/10919/119382 | en |
| dc.language.iso | en | en |
| dc.publisher | Virginia Tech | en |
| dc.rights | In Copyright | en |
| dc.rights.uri | http://rightsstatements.org/vocab/InC/1.0/ | en |
| dc.subject | multi-modality reasoning | en |
| dc.subject | fine-grained reasoning | en |
| dc.subject | multi-document understanding | en |
| dc.subject | text | en |
| dc.subject | image | en |
| dc.subject | video | en |
| dc.subject | audio | en |
| dc.title | M3D: Multimodal MultiDocument Fine-Grained Inconsistency Detection | en |
| dc.type | Thesis | en |
| thesis.degree.discipline | Computer Science & Applications | en |
| thesis.degree.grantor | Virginia Polytechnic Institute and State University | en |
| thesis.degree.level | masters | en |
| thesis.degree.name | Master of Science | en |
Files
Original bundle
1 - 1 of 1