LucidWorks Vectorize Module for the Digital Library Curriculum Initiative

TR Number
Date
2013-05-18
Journal Title
Journal ISSN
Volume Title
Publisher
Abstract

The goal of our project was to create a learning module for students who are interested in converting a large number of documents of data into a usable form for machine learning, information retrieval, and related purposes. In order to complete this task, we wrote a module that gives information about how LucidWorks Big Data software handles the task of vectorizing documents using a workflow. This module details the approach that LucidWorks implements, and gives detailed instructions on how to create a collection, start the workflow, check the status of the workflow, and finally access the results after the workflow completes. Upon completion of our module, users will be able to test their understanding using the example documents provided by the LucidWorks software, and be familiar with Hadoop’s distributed file system. After users are familiar with how the software works, they will be able to create their own vectorized representations of documents. Our module also provides information about the installation of LucidWorks software on a virtual machine; if the users have no access to the software they will then be able to create their own instance of it. The module will be available also through http://en.wikiversity.org/wiki/Curriculum_on_Digital_Libraries.

Description
This is a module for students wishing to learn about the LucidWorks Vectorize workflow. It walks them through the steps to use this tool and teaches them the purpose of the vector space model. There are also suggestions for discussion upon completion of the module. The LucidWorks Overview Module should be completed prior to starting this module. The ModuleIntro.(pdf/pptx/mp4) is an introduction to the module. It gives an overview of the concepts as well as learning goals for students. The FinalReport.pdf is an informative piece about the creation of this module. It includes user and developer manuals, as well as descriptions of some of the problems we encountered. The VectorizeModule.(pdf/docx) is the LucidWorks module itself. It includes all of the information concerning the module, as well as prerequisites and references. User's should either watch the introductory video or PowerPoint prior to starting on the Vectorize Module pdf file.
Keywords
LucidWorks, vectorize, module, workflow
Citation