%-------------Approach, Design, Implementation
\chapter{Design}

\section{Approach}

We broke down our research objectives into the following research questions:

\textbf{RQ1}: \textit{How do we structure the metadata in a format that ElasticSearch can ingest?}

\textbf{RQ2}: \textit{What text pre-processing should be done on the OCR'ed documents to make them useful for the TML team?}

\textbf{RQ3}: \textit{How does UCSF's proprietary OCR tool compare with open-source OCR tools on a subset of the tobacco documents?}
\newline

We initially started by investigating these research questions on a subset of the tobacco documents. More specifically, we used the roughly 8,000 deposition documents in this work. We later expanded the scope to cover all 14 million documents.

In order to be able to provide other teams with the necessary information in the required format, the following steps were followed:


1. Download data from the UCSF database onto the tobacco.cs.vt.edu virtual machine, including files and corresponding metadata. This is illustrated in Figure \ref{filesUCSF}.
\begin{figure}[H]
\centering
 \includegraphics[width=\textwidth]{figures/pictrue1.png}
 \caption{Files downloaded from the UCSF site}
 \label{filesUCSF}
\end{figure}

2. Import metadata into a database and analyze the structure of data.

3. Process a small collection of data first. Specifically, locate deposition files based on file type, then process the deposition files.

4. Convert metadata and files into JSON format, as shown in Figure \ref{processData}.
\begin{figure}[H]
\centering
 \includegraphics[width=\textwidth]{figures/pictrue2}
 \caption{Process data to be formatted}
 \label{processData}
\end{figure}

5. Pre-process texts of deposition files, as shown in Figure \ref{textProcessing}. This involved the following steps:

\begin{itemize}
\item Text cleaning: Delete the invalid characters which can not be decoded to Unicode. Delete the blank characters, like CRLF line terminators, and so on. 
\item Tokenization: Split the given text into smaller pieces. Words, numbers, punctuation marks, and others are considered as tokens.
\item Lemmatization: Reduce inflectional forms to get the correct base forms of words by using lexical knowledge.
The aforementioned processes are described in Chapter \ref{implementation} in detail.
\item Concatenation of Q’s and A’s in one line for easy reading. This process is described in Chapter \ref{implementation} in detail.
\end{itemize}
\begin{figure}[H]
\centering
 \includegraphics[width=\textwidth]{figures/preprocess}
 \caption{Text preprocessing}
 \label{textProcessing}
\end{figure}

6. Provide the relevant JSON files to the ElasticSearch team.

7. Migrate preprocessed deposition files and JSON files to Ceph.

The next step was to apply the above-stated process to all types of documents.
We have extracted the metadata and content for all the documents and sent them to the ELS team for processing, and to the TML team for summarization.

\section{Tools}
\begin{itemize}
    \item MariaDB: MariaDB Server, with its continual open source innovation, is a modern relational database.
    We store and view all the metadata of files in MariaDB 5.5.
    \item Python: We used Python 3 to develop scripts to process files, including JSON packages, Codec packages, and so on.
    \item NLTK: Natural Language ToolKit\cite{nltk} is a leading platform for building Python programs to work with human language data.
    We used NLTK to do text tokenization and lemmatization, and to provide high-quality text to other teams.
\end{itemize}

