AI Aided Annotation

Bishop, Jonah B. M.; David, Isaac; Lubana, Ishaandeep

AI Aided Annotation

dc.contributor.author	Bishop, Jonah B. M.	en
dc.contributor.author	David, Isaac	en
dc.contributor.author	Lubana, Ishaandeep	en
dc.date.accessioned	2022-05-13T14:40:21Z	en
dc.date.available	2022-05-13T14:40:21Z	en
dc.date.issued	2022-05-11	en
dc.description.abstract	Human annotation of long documents is a very important task in training and evaluation in NLP. The process generally starts with the human annotators reading over the document in its entirety. Once the annotator feels they have a sufficient grasp on the document, they can begin to annotate it. Specifically, annotators will look for questions that can be answered, and then write down the question and answer. In our client’s case, the chosen long documents are electronic theses and dissertations (ETDs) which are often 100-150 pages minimum, thereby making it a time consuming and expensive process to annotate. The ETDs are annotated on a chapter by chapter basis as content can vary significantly in each chapter. The annotations generated are then used to help evaluate downstream tasks such as summarization, topic modeling, and question answering. The system aids the annotators in the creation of a Knowledge Base that is rich with topics/keywords and question-answer pairs for each chapter in ETDs. The core of the system revolves around an algorithm known as the Maximal Marginal Relevance. By utilizing the MMR algorithm with a changeable lambda value, keywords, and a couple of other elements, we can identify sentences based on their similarity or diversity relative to a collection of sentences. This algorithm would greatly enhance the annotation process in ETDs by automating the process of identifying the most relevant sentences. Thus, annotators do not have to sift through the ETDs one sentence at a time, instead making a comprehensive summary as fast as the MMR algorithm can work. As a result, annotators can save many hours per ETD, resulting in more human generated annotations in a shorter amount of time. The final deliverables are the project, a final slideshow presenting our work throughout the semester, a final report, and a video demonstrating exactly how to use our platform. All of this is available here on VTechWorks in this report. Additionally, the project is being built using GitHub, making it free and available to the public to fork and modify in any way they see fit.	en
dc.description.notes	The deliverables available for download include a copy of the presentation in both .pdf and .pptx form, a copy of the report in .pdf and .docx form, and an MP4 of the video walkthrough. Another version of the video is online at: https://www.youtube.com/watch?v=C5N5sKn4a2E. The following is a link to the GitHub available for use/modification: https://github.com/jonahbishop/AI-Aided-Annotation.	en
dc.identifier.uri	http://hdl.handle.net/10919/110085	en
dc.language.iso	en_US	en
dc.publisher	Virginia Tech	en
dc.rights	Attribution 4.0 International	en
dc.rights.uri	http://creativecommons.org/licenses/by/4.0/	en
dc.subject	Maximal Marginal Relevance	en
dc.subject	Annotation	en
dc.subject	Website	en
dc.subject	AI Aided Annotation	en
dc.subject	Chapter Annotation	en
dc.title	AI Aided Annotation	en
dc.type	Presentation	en
dc.type	Report	en
dc.type	Video	en