LucidWorks Vectorize Module for the Digital Library Curriculum Initiative

Abstract

The goal of our project was to create a learning module for students who are interested in converting a large number of documents of data into a usable form for machine learning, information retrieval, and related purposes. In order to complete this task, we wrote a module that gives information about how LucidWorks Big Data software handles the task of vectorizing documents using a workflow. This module details the approach that LucidWorks implements, and gives detailed instructions on how to create a collection, start the workflow, check the status of the workflow, and finally access the results after the workflow completes. Upon completion of our module, users will be able to test their understanding using the example documents provided by the LucidWorks software, and be familiar with Hadoop’s distributed file system. After users are familiar with how the software works, they will be able to create their own vectorized representations of documents. Our module also provides information about the installation of LucidWorks software on a virtual machine; if the users have no access to the software they will then be able to create their own instance of it. The module will be available also through http://en.wikiversity.org/wiki/Curriculum_on_Digital_Libraries.