This is an introduction to the LucidWorks Vectorize Workflow
Module. I'm David Kniphuisen. And I'm Alan Tran. Upon
completion of this module, users will be able to
generate a Vectorize workflow in LucidWorks, send queries
to LucidWorks for progress on the workflow and formulate
the data into vectors for machine learning. This module
should require about three hours out of class to complete
the module and then one hour in class to complete
the exercises in groups. This module follows after the
LucidWorks overview module that can be done in parallel
with the clustering and classification modules. The prerequisites for
this module is familiarity with UNIX, having an overview of
LucidWorks, Big Data Software and an understanding of
basic cURL commands. The bulk of our module gives
details to the background of the workflows and LucidWorks
as well as the background as to why we need
vectorized documents. This is presented in the description of
the of the Vector Space Model, which explains the
importance of converting documents to vectors as well as the
limitations of this model. The other half of our
body of knowledge includes a detailed guide on how to
run a Vectorize workflow that is to take the data
in a collection provided by Hadoop distributed file system,
send it through LucidWorks, extract, transform and
load workflows, then send it through a sub-workflow
in order to vectorize these documents. cURL commands are
used as queries to interact with the LucidWorks framework
and we will and we provide an explanation of the
significant parameters that are required during the workflow. Hopefully
after the completion of our module, students would understand
the LucidWorks structure. They will also understand the
importance and limitations of the Vector Space Model, be
able to send simple queries that will create collections,
start sub-workflows through a distributed file system and be
able to check the status of jobs and finally verify
the results.