Show simple item record

dc.contributor.authorBrittle, Collin
dc.contributor.authorXie, Zhiwu
dc.date.accessioned2016-06-25T21:38:35Z
dc.date.available2016-06-25T21:38:35Z
dc.date.issued2014-06-10
dc.identifier.urihttp://hdl.handle.net/10919/71460
dc.description.abstractPresentation video available at https://connectpro.helsinki.fi/p1txjdy74ts/ This presentation addresses the challenge of processing big data in a cloud-based data repository. Using the Hydra Project’s Hydra and Sufia ruby gems and working with the Hydra community, we created a special repository for the project, and set up background jobs. Our approach is to create the metadata with these jobs, which are distributed across multiple computing cores. This will allow us to scale our infrastructure out on an as-needed basis, and decouples automatic metadata creation from the response times seen by the user. While the metadata is not immediately available after ingestion, it does mean that the object is. By distributing the jobs, we can compute complex properties without impacting the repository server. Hydra and Sufia allowed us to get a head start by giving us a simple self deposit repository, complete with background jobs support via Redis and Resque.en_US
dc.language.isoen_USen_US
dc.relation.ispartofOpen Repositories 2014en_US
dc.relation.hasparthttp://urn.fi/URN:NBN:fi-fe2014070432268
dc.relation.hasparthttps://connectpro.helsinki.fi/p1txjdy74ts/
dc.rightsAttribution 3.0 United States*
dc.rights.urihttp://creativecommons.org/licenses/by/3.0/us/*
dc.subjectDigital libraryen_US
dc.subjectBig dataen_US
dc.subjectInstitutional repositoryen_US
dc.subjectFedoraen_US
dc.subjectHydraen_US
dc.titleBig Data Processing in the Cloud: a Hydra/Sufia Experienceen_US
dc.typeArticleen_US


Files in this item

Thumbnail
Thumbnail
Thumbnail

This item appears in the following Collection(s)

Show simple item record

Attribution 3.0 United States
License: Attribution 3.0 United States