Object Detection

Electronic theses and dissertations (ETDs) contain valuable knowledge that can be useful in a wide range of research areas. To effectively utilize the knowledge contained in ETDs, the data first needs to be parsed and stored in an XML document. However, since most of the ETDs available on the web are presented in PDF, parsing them is a challenge to make their data useful for any downstream task, including question-answering, figure search, table search, and summarizing. For information search and extraction, contextual information is needed to perform these tasks. However, such semantic information is hidden in PDF documents. In contrast, XML can explicitly share semantic information. The structure within XML documents can enforce semantic continuity within the tag elements. Accordingly, knowledge graphs can be more easily built from XML, rather than PDF, representations. The goal of this project was to extract different elements of scholarly documents such as metadata (title, authors, year), chapter headings and subheadings, equations, figures (and captions), tables (and captions), and paragraphs, and then package them into an XML document. Subsequently, a pipeline responsible for the conversion and a dataset to support the object detection step was developed. Over the semester, 200 ETDs, both born-digital and scanned, were annotated using a online tool called RoboFlow. A model based on Facebook’s open-sourced object detection model, Detectron2, was trained with the created dataset. Besides that, a pipeline that utilizes the model has been built that converts an ETD in PDF into an XML document, which can then be used for future downstream tasks and HTML for visualization. A dataset consisting of 200 annotated ETDs and a working pipeline were delivered to the client. From the project, the Object Detection Team learned numbers of libraries related to the task, built a sense of the importance of version control, and understood how to split a large task into smaller and more approachable pieces.