Generating an Intelligent Human-Readable Summary of a Shooting Event from a Large Collection of Webpages

Chandrasekaran, Arjun; Sharma, Saurav; Sulucz, Peter; Tran, Jonathan

Generating an Intelligent Human-Readable Summary of a Shooting Event from a Large Collection of Webpages

dc.contributor.author	Chandrasekaran, Arjun	en
dc.contributor.author	Sharma, Saurav	en
dc.contributor.author	Sulucz, Peter	en
dc.contributor.author	Tran, Jonathan	en
dc.date.accessioned	2014-12-14T02:10:01Z	en
dc.date.available	2014-12-14T02:10:01Z	en
dc.date.issued	2014-12	en
dc.description	Filename and description of all included files: ShootingsReportPdf.pdf - PDF version of our project report. ShootingsReportDoc.docx - MS Word version of our project report. ShootingsPresentationPpt.pptx - MS PowerPoint version of our project presentation. ShootingsPresentationPdf.pdf - PDF version of our project presentation. Source code is included in folder "shooting_summary_code": shooting_summary_code/mapper.py - The map script used on the Hadoop cluster. This file contains the regular expressions which are used to extract the data from the collections. shooting_summary_code/parse.py - Standard frequency based reducer on the cluster. Takes sorted input from the mapper, and reduces based on frequency. shooting_summary_code/reducer.py - Python which parses the output from the mapper and reducer. Does not run on Hadoop cluster. Relies on data being sorted by frequency (most frequent at top of file). Contains the implementation of the regular grammar, along with filtering techniques. shooting_summary_code/TrigramTagger.pkl - A Python pickled version of our trigram tagger, which is used in parse.py.	en
dc.description.abstract	We describe our approach to generating summaries of a shooting event from a large collection of webpages. We work with two separate events - a shooting at a school in Newtown, Connecticut and another at a mall in Tucson, Arizona. Our corpora of webpages are inherently noisy and contain a large amount of irrelevant information. In our approach, we attempt to clean up our webpage collection by removing all irrelevant content. For this, we utilize natural language processing techniques such as word frequency analysis, part of speech tagging and named entity recognition to identify key words about our news events. Using these key words as features, we employ classification techniques to categorize each document as relevant or irrelevant. We discard the documents classified as irrelevant. We observe that to generate a summary, we require some specific information that enables us to answer important questions such as "Who was the killer?", "Where did the shooting happen?", "How many casualties were there?" and so on. To enable extraction of these essential details from news articles, we design a template of the event summary with slots that pertain to information we would like to extract. We designed regular expressions to identify a number of 'candidate' values for the template slots. Using a combination of word frequency analysis and specific validation techniques, we choose the top candidate for each slot of our template. We use a grammar based on our template to generate a human readable summary of each event. We utilize the Hadoop MapReduce framework to parallelize our workflow, along with the NLTK language processing library to simplify and speed our development. We learned that a variety of different methods and techniques are necessary in order to provide an accurate summary for any collection. It is seen that cleaning poses an incredibly difficult yet necessary task when attempting to semantically interpret data. We found that our attempts to extract relevant topics and sentences using the topic extraction method Latent Dirichlet Allocation and k-means clustering did not result in topics and sentences that were indicative of our corpus. We demonstrate an effective way of summarizing a shooting event that extracts relevant information by using regular expressions and generates a comprehensive human-readable summary utilizing a regular grammar. Our solution generates a summary that includes key information needed in understanding a shooting event such as: the shooter(s), date of the shooting, location of the shooting, number of people injured and wounded, and the weapon used. This solution is shown to work effectively for two different types of shootings: a mass murder, and an assassination attempt.	en
dc.description.sponsorship	NSF DUE-1141209 and IIS-1319578	en
dc.identifier.uri	http://hdl.handle.net/10919/51137	en
dc.language.iso	en_US	en
dc.rights	Creative Commons CC0 1.0 Universal Public Domain Dedication	en
dc.rights.uri	http://creativecommons.org/publicdomain/zero/1.0/	en
dc.subject	document summarization	en
dc.subject	computational linguistics	en
dc.subject	natural language generation	en
dc.subject	natural language processing	en
dc.subject	shooting summary	en
dc.subject	news summarization	en
dc.subject	nlp	en
dc.subject	Research Subject Categories::TECHNOLOGY	en
dc.subject	computational linguistics	en
dc.title	Generating an Intelligent Human-Readable Summary of a Shooting Event from a Large Collection of Webpages	en
dc.title.alternative	Generating a Summary for a Shooting Event	en
dc.type	Presentation	en
dc.type	Software	en
dc.type	Technical report	en