Hadoop Project for IDEAL in CS5604

Simple item page
dc.contributor.authorCadena, Joseen
dc.contributor.authorChen, Mengsuen
dc.contributor.authorWen, Chengyuanen
dc.date.accessioned2015-05-15T04:06:38Zen
dc.date.available2015-05-15T04:06:38Zen
dc.date.issued2015-05-11en
dc.description.abstractThe Integrated Digital Event Archive and Library (IDEAL) system addresses the need for combining the best of digital library and archive technologies in support of stakeholders who are remembering and/or studying important events. It leverages and extends the capabilities of the Internet Archive to develop spontaneous event collections that can be permanently archived as well as searched and accessed. IDEAL connects the processing of tweets and web pages, combining informal and formal media to support building collections on chosen general or specific events. Integrated services include topic identification, categorization (building upon special ontologies being devised), clustering, and visualization of data, information, and context. The objective for the course is to build a state-of-the-art information retrieval system in support of the IDEAL project. Students were assigned to eight teams, each of which focused on a different part of the system to be built. These teams were Solr, Classification, Hadoop, Noise Reduction, LDA, Clustering, Social Networks, and NER. As the Hadoop team, our focus is on making the information retrieval system scalable to large datasets by taking advantage of the distributed computing capabilities of the Apache Hadoop framework. We design and put in place a general schema for storing and updating data stored in our Hadoop cluster. Throughout the project, we coordinate with other teams to help them make use of readily available machine learning software for Hadoop, and we also provide support for using MapReduce. We found that different teams were able to easily integrate their results in the design we developed and that uploading these results into a data store for communication with Solr can be done, in the best cases, in a few minutes. We conclude that Hadoop is an appropriate framework for the IDEAL project; however, we also recommend exploring the use of the Spark framework.en
dc.description.sponsorshipNSF grant IIS - 1319578, III: Small: Integrated Digital Event Archiving and Library (IDEAL)en
dc.identifier.urihttp://hdl.handle.net/10919/52342en
dc.language.isoenen
dc.publisherVirginia Techen
dc.rightsCreative Commons Attribution-NonCommercial-ShareAlike 3.0 United Statesen
dc.rights.urihttp://creativecommons.org/licenses/by-nc-sa/3.0/us/en
dc.subjectIDEALen
dc.subjectHadoopen
dc.subjectBig Dataen
dc.subjectInformation Retrievalen
dc.titleHadoop Project for IDEAL in CS5604en
dc.typePresentationen
dc.typeSoftwareen
dc.typeTechnical reporten

Files

Original bundle
Now showing 1 - 5 of 5
Thumbnail Image
Name:
HadoopFinalReport.pdf
Size:
2.5 MB
Format:
Adobe Portable Document Format
Description:
Hadoop Team Final Report (PDF version)
Download
Name:
HadoopFinalReport.docx
Size:
1.77 MB
Format:
Microsoft Word XML
Description:
Hadoop Team Final Report (Word version)
Download
Thumbnail Image
Name:
HadoopPresentation.pdf
Size:
1.21 MB
Format:
Adobe Portable Document Format
Description:
Hadoop Team Final Presentation (PDF version)
Download
Name:
HadoopPresentation.pptx
Size:
1.46 MB
Format:
Microsoft Powerpoint XML
Description:
Hadoop Team Final Presentation (PowerPoint version)
Download
Name:
hadoop_team_code.tar.gz
Size:
36.52 KB
Format:
Unknown data format
Download
License bundle
Now showing 1 - 1 of 1
Name:
license.txt
Size:
1.5 KB
Format:
Item-specific license agreed upon to submission
Description:
Download

Collections

CS5604: Information Retrieval