\chapter{Requirements}

To provide the teams working on ingesting data into Elasticsearch with a certain subset (Phase I) and a complete set (Phase II) of text data with accompanying metadata information, the following requirements and conditions should be taken into consideration.

\begin{enumerate}
    \item \textbf{Text data from the documents should be provided in its entirety and with completeness.}
    
    In other words, all the text information contained in processed documents is supposed to be saved and transferred to the other teams without any loss, unless otherwise indicated.
    In Phase I of the project, only documents of the "Deposition" type were considered.
    Therefore, this needed to be extended to support the other document types as well.
    Phase II of the project involved processing all the documents from the database.
    
    \item \textbf{Only requested meta-information fields are to be provided for further processing.}
    
    This means that the final metadata file should only comprise two sets of fields: the ones which are referred to by the ELS team and necessary for the search engine, and the fields that are not involved in the search process directly but can be used by the FEK team for visualization and demonstration purposes.

    \item \textbf{The metadata files should follow the format requirements.}
    
    Depending on what restrictions the ELS team imposes on the format of the to-be-ingested files, it was our responsibility to follow these restrictions.
    For example, certain metadata files, describing a similar topic or several related ones (such as articles, questionnaires, etc.), might be incorporated into a single file for potential improvement of search engine capabilities.
    
    \item \textbf{The CMT team should communicate with the ELS team and other teams on a regular basis.}
    
    This was necessary in order to understand current updates in constraints imposed on the file structure and size so that the CMT team is able to follow the 2nd and 3rd requirements. This was also necessary to provide other groups with updates regarding our progress.
    
    \item \textbf{The CMT team must pre-process file content for the TML team appropriately}
    
    As various intelligent text analysis algorithms require data to be ingested in different formats, we also provided the TML team with full-text data in desired formats. The reader may find further details in Section \ref{Section:preprocessing}.
    
    \item \textbf{Potential storage of the data on cloud.cs.vt.edu for future project organization.}
    
    Although it is not required to store all the documents and metadata tables in the cloud storage for the purpose of pre-processing, this could be a potential direction for the sake of persisting files to be utilized by future project contributors.
\end{enumerate}