Parquet Containers

Abstract

Archives preserve content and support various types of analysis, study, and use. Archiving the Web and social media (e.g., Twitter) content involves use of representations like Web ARChive (WARC) and JavaScript Object Notation (JSON). Parquet is a newer representation that yields better performance with large collections for many data processing tasks. In support of archive-related research at Virginia Tech in University Libraries and the Digital Library Research Laboratory (DLRL), this project involved the development of software (tested in Linux and Windows, as well as on local and container cluster environments), and its packaging in containers, for converting WARC to Parquet as well as JSON to Parquet. Images of the resulting containers are available to support the research of others.