MOON: MapReduce on Opportunistic eNvironments

dc.contributor.authorLin, Heshanen
dc.contributor.authorArchuleta, Jeremyen
dc.contributor.authorMa, Xiaosongen
dc.contributor.authorFeng, Wu-chunen
dc.contributor.authorZhang, Zheen
dc.contributor.authorGardner, Mark K.en
dc.contributor.departmentComputer Scienceen
dc.date.accessioned2013-06-19T14:36:38Zen
dc.date.available2013-06-19T14:36:38Zen
dc.date.issued2009en
dc.description.abstractMapReduce offers a flexible programming model for processing and generating large data sets on dedicated resources, where only a small fraction of such resources are every unavailable at any given time. In contrast, when MapReduce is run on volunteer computing systems, which opportunistically harness idle desktop computers via frameworks like Condor, it results in poor performance due to the volatility of the resources, in particular, the high rate of node unavailability. Specifically, the data and task replication scheme adopted by existing MapReduce implementations is woefully inadequate for resources with high unavailability. To address this, we propose MOON, short for MapReduce On Opportunistic eNvironments. MOON extends Hadoop, an open-source implementation of MapReduce, with adaptive task and data scheduling algorithms in order to offer reliable MapReduce services on a hybrid resource architecture, where volunteer computing systems are supplemented by a small set of dedicated nodes. The adaptive task and data scheduling algorithms in MOON distinguish between (1) different types of MapReduce data and (2) different types of node outages in order to strategically place tasks and data on both volatile and dedicated nodes. Our tests demonstrate that MOON can deliver a 3-fold performance improvement to Hadoop in volatile, volunteer computing environments.en
dc.format.mimetypeapplication/pdfen
dc.identifierhttp://eprints.cs.vt.edu/archive/00001089/en
dc.identifier.sourceurlhttp://eprints.cs.vt.edu/archive/00001089/01/moon.pdfen
dc.identifier.trnumberTR-09-21en
dc.identifier.urihttp://hdl.handle.net/10919/20350en
dc.language.isoenen
dc.publisherDepartment of Computer Science, Virginia Polytechnic Institute & State Universityen
dc.rightsIn Copyrighten
dc.rights.urihttp://rightsstatements.org/vocab/InC/1.0/en
dc.subjectAlgorithmsen
dc.subjectData structuresen
dc.titleMOON: MapReduce on Opportunistic eNvironmentsen
dc.typeTechnical reporten
dc.type.dcmitypeTexten

Files

Original bundle
Now showing 1 - 1 of 1
Loading...
Thumbnail Image
Name:
moon.pdf
Size:
602.17 KB
Format:
Adobe Portable Document Format