Big Data Processing in the Cloud: a Hydra/Sufia Experience

Brittle, Collin; Xie, Zhiwu

Big Data Processing in the Cloud: a Hydra/Sufia Experience

dc.contributor.author	Brittle, Collin	en
dc.contributor.author	Xie, Zhiwu	en
dc.date.accessioned	2016-06-25T21:38:35Z	en
dc.date.available	2016-06-25T21:38:35Z	en
dc.date.issued	2014-06-10	en
dc.description.abstract	Presentation video available at https://connectpro.helsinki.fi/p1txjdy74ts/ This presentation addresses the challenge of processing big data in a cloud-based data repository. Using the Hydra Project’s Hydra and Sufia ruby gems and working with the Hydra community, we created a special repository for the project, and set up background jobs. Our approach is to create the metadata with these jobs, which are distributed across multiple computing cores. This will allow us to scale our infrastructure out on an as-needed basis, and decouples automatic metadata creation from the response times seen by the user. While the metadata is not immediately available after ingestion, it does mean that the object is. By distributing the jobs, we can compute complex properties without impacting the repository server. Hydra and Sufia allowed us to get a head start by giving us a simple self deposit repository, complete with background jobs support via Redis and Resque.	en
dc.identifier.uri	http://hdl.handle.net/10919/71460	en
dc.language.iso	en_US	en
dc.relation.haspart	http://urn.fi/URN:NBN:fi-fe2014070432268	en
dc.relation.haspart	https://connectpro.helsinki.fi/p1txjdy74ts/	en
dc.relation.ispartof	Open Repositories 2014	en
dc.rights	Creative Commons Attribution 3.0 United States	en
dc.rights.uri	http://creativecommons.org/licenses/by/3.0/us/	en
dc.subject	Digital library	en
dc.subject	Big data	en
dc.subject	Institutional repository	en
dc.subject	Fedora	en
dc.subject	Hydra	en
dc.title	Big Data Processing in the Cloud: a Hydra/Sufia Experience	en
dc.type	Article	en