Team 4: Language Models, Classification and Summarization

Naleshwarkar, Kanad; Bhatambarekar, Gayatri; Desai, Zeel; Kumaran, Aishwarya; Haque, Shadab; Srinivasan Manikandan, Adithya Harish

Team 4: Language Models, Classification and Summarization

dc.contributor.author	Naleshwarkar, Kanad	en
dc.contributor.author	Bhatambarekar, Gayatri	en
dc.contributor.author	Desai, Zeel	en
dc.contributor.author	Kumaran, Aishwarya	en
dc.contributor.author	Haque, Shadab	en
dc.contributor.author	Srinivasan Manikandan, Adithya Harish	en
dc.date.accessioned	2024-04-25T02:40:14Z	en
dc.date.available	2024-04-25T02:40:14Z	en
dc.date.issued	2023-12-17	en
dc.description	Team4ClassificationSummarizationLanguageModelsReport.pdf: Final report by Team 4; Team4ClassificationSummarizationLanguageModelsPresentation.pptx: Presentation slides for the final presentation of Team 4; Team4ClassificationSummarizationLanguageModelsPresentation.pdf: Presentation slides in PDF format; Team4ClassificationSummarizationLanguageModelsReportSource.zip: LaTeX source code for the report	en
dc.description.abstract	The CS5604 class at Virginia Tech has been tasked with developing an information retrieval and analysis system that can handle the collection of data of at least 500,000 Electronic Theses and Dissertations (ETDs), under the direction of Dr. Edward A. Fox. This program should function as a search engine with a variety of capabilities, including browsing, searching, giving suggestions, and rating search results. The class has been split into six teams to execute this job, and each team has been given a specific task. The goal of this report is to provide an overview of Team 4's contribution, which focuses on classification, summarization, and language models. Our prime tasks were testing out various models for classification and summarization. During the course of this project, we evaluated models developed by the previous team working on this task and explored various strategies to improve them. For the classification task, we fine-tuned the SciBERT model to get standardized subject category labels that are in accordance with ProQuest. We also evaluated a large language model, LLaMA 2, for the classification task, and after comparing its performance with the fine-tuned SciBERT model, we observed that LLaMA 2 was not efficient enough for a large-scale system that the class was working on. For summarization, we evaluated summaries generated by various transformer, non-transformer, and LLM-based models. The five models that we evaluated for summarization were TextRank, LexRank, LSA, BigBirdPegasus, and LLaMA 2 7B. We observed that although TextRank and BigBirdPegasus had comparable results, the summaries generated by TextRank were more comprehensive. This experimentation gave us valuable insight into the complexities of processing a large set of documents and performing tasks such as classification and summarization. Additionally, it allowed us to explore the deployment of these models in a production environment to evaluate their performance at scale.	en
dc.identifier.uri	https://hdl.handle.net/10919/118663	en
dc.language.iso	en_US	en
dc.publisher	Virginia Tech	en
dc.rights	Attribution-NonCommercial 4.0 International	en
dc.rights.uri	http://creativecommons.org/licenses/by-nc/4.0/	en
dc.subject	Classification	en
dc.subject	Summarization	en
dc.subject	Language Models	en
dc.subject	LLMs	en
dc.subject	LLaMA	en
dc.subject	Subject Categories	en
dc.title	Team 4: Language Models, Classification and Summarization	en
dc.type	Presentation	en
dc.type	Report	en