\chapter{Implementation} \label{implementation}

To manage our work, we identified three major tasks:

\begin{enumerate}
    \item Identifying different document types
    \item Processing metadata
    \item Processing document text
\end{enumerate}

These tasks are described in detail below.
In order to achieve our goals in a reasonable amount of time, along with preparing and executing MySQL queries, we focused on developing code in Python because of its popularity and ease-of-use.
We started by working with only the deposition documents, and then we transitioned to working with all document types.

\section{Identifying Different Document Types}

The tobacco documents within the dataset are categorized by type, such as memo, letter, and deposition.
Some of the documents belong to several types.
The information about each document’s type is encoded in the MySQL database (further details can be found in Section~\ref{section:meta}).
We identified different document types and decided to process deposition documents.
The number of depositions documents is relatively small: 7995 files overall as opposed to the entire collection size (about 14 million files).
We developed a Python script to produce a file containing the record keys (unique identifiers) of all the deposition documents. 
The record keys were then used to retrieve the deposition files from the database directly, as the record keys correspond to file names.

\section{Processing Metadata}
\label{section:procmeta}

The objective of the metadata processing task is to transform the metadata from a MySQL database to a JSON file that ElasticSearch can ingest. Because ElasticSearch could not ingest the metadata of all 14 million documents in one file, we have transformed the metadata into a collection of 31 files, each containing the metadata of at most half a million documents.

As shown in Figure \ref{fig:metadata}, we encode a document’s metadata as two JSON objects: a header and a body. The header object identifies the document with its unique ID. The body object contains the document’s metadata as a collection of key-value pairs, where the key identifies the metadata field, and the value specifies the corresponding value for the given document.
\begin{figure}[H]
    \centering
    \caption{JSON object representing metadata}
    \label{fig:metadata}
\begin{minted}[linenos=false,showspaces=false,breaklines=true]{json}
{"index": {"_id": 1, "_index": "tobacco"}}
{"url": "https://s3-us-west-2.amazonaws.com/edu.ucsf.library.iddl.
 artifacts/f/z/w/d/fzwd0000/fzwd0000.pdf", 
 "Legacy_(LTDL2)_Tobacco_Id": "gbu00a00", 
 "Title": "RISING MEDICAL COSTS REQUIRE G.H.C. DUES INCREASE IN 1967  VIEW",
 "Document_Date": "1967-02-28 00:00:00",
 "Author": "NEWMAN HF;SIEGAL A",
 "Case": "MNAG",
 "Description": "DISCUSSES DUES INCREASE",
 "Date_Added_UCSF": "2002-02-01 00:00:00",
 "Document_Type": "article", 
 "availablility": "public", 
 "availablilitystatus": "no restrictions",
 "Mentioned": "GROUP HEALTH COOPERATIVE OF PUGET SOUND;...
  "Attached_Artifacts": 
    [{"name":"fzwd0000.ocr",
	  "mediaType":"text/plain",
	  "size":7730},
	 {"name":"fzwd0000.pdf",
	  "mediaType":"application/pdf",
	  "size":308035},
	 {"name":"fzwd0000.tif",
	  "mediaType":"image/tiff",
	  "size":314038},
	 {"name":"fzwd0000_thumb.png",
	  "mediaType":"image/png",
	  "size":100478}],
   "Page_Map": "60025743/5745",
   "Numeric_start_bates": "60025743",
   "Numeric_end_bates": "60025745",
   "Page_Count": "3"
}
\end{minted}
\end{figure}
The following steps were applied to transform metadata into the required JSON format.
First, connect to the MySQL database in order to retrieve metadata. Next, execute a MySQL query to retrieve all metadata entries, ordered by document ID, with document IDs between the given minimum and maximum ID values. Then, retrieve the result of this query. The result is a list of metadata entries, each consisting of a document ID, a metadata field name, and a metadata value. Extract the metadata entries for each document in an iterative fashion. Generate the header JSON object followed by the body obejct for each document.

We later included a text field, which is provided in a page-wise fashion: an additional key "text content" consists of two key-value pairs, where the keys are "content" and "page".
This would facilitate full-text search by the search engine.

\section{Processing Document Text}
The task of processing the content of the documents can be divided into 2 parts based on the requests from the teams:
\begin{enumerate}
    \item Preparing documents for the ELS and FEK teams
    \item Pre-processing documents for applying of intelligent algorithms for TML team
\end{enumerate}

These sub-tasks are described in Sections \ref{subsection:prep} and \ref{Section:preprocessing}.

\subsection{Preparing document text for the ElasticSearch and Front-End Kibana Teams}
\label{subsection:prep}

All the files stored in the virtual machine have a .ocr format.
In order to make them digestible by the ElasticSearch engine, we developed and subsequently updated a Python script to convert each of the deposition .ocr files into a JSON file.
The typical structure of a deposition file is shown in Figure~\ref{fig:deposition_structure}.

\begin{figure}[H]
    \centering
    \includegraphics[width=\textwidth]{figures/deposition_structure.PNG}
    \caption{The structure of the first page of the deposition document}
    \label{fig:deposition_structure}
\end{figure}

We structured the data to extract page contents separately as shown below:

\begin{minted}[linenos=false,showspaces=false,breaklines=true]{json}
{
"Page No." : "Page content"
}
\end{minted}

The structure of the final file generated by the Python script is shown below:


\begin{minted}[linenos=false,showspaces=false,breaklines=true]{json}
{"index": {"_id": 0, "_index": "tobacco"}}
{"Title": "Deposition of THOMAS RICHARD ADAMS, March 15, 2000, WHITELEY v. RAYBESTOS-MANHATTAN INC.", 
"Document_Date": "2000-03-15",
"text_content":[{"page":"1","content":"IN THE SUPERIOR COURT OF THE STATE OF CALIFORNIA"}, 
{"page": "2","content":"DEPOSITION OF THOMAS RICHARD ADAM"}]}
\end{minted}

Besides the text itself, the file represents a page-wise distribution of the content, as well as the document date, as requested by the ELS and FEK teams.

\subsection{Pre-processing documents for text analytics and machine learning (TML)}
\label{Section:preprocessing}
To ensure the correct operation of different automatic text analysis and ML approaches, we follow the specific requirements for the document text preprocessing provided by the TML team.
The tasks include the following steps:
\begin{enumerate}
    \item Cleaning documents
    
The documents stored in the virtual machine were received as a result of preliminary performing of the Optical Character Recognition (OCR) procedure.
A common issue about automatic extraction of characters from document scans is that the latter may contain a lot of specialized information, such as line numbers and handwritten inscriptions. Although some of them might be valuable for retrieval purposes, a significant portion of these characters and strings can hinder the performance of ML algorithms, so cleaning the documents is important.
Moreover, the recognizing algorithm may produce service symbols, which can become additional noise for ML algorithms.
In order to reduce the negative impact of such characters, we prepared a cleaning algorithm in Python and used it to pre-process a set of deposition documents.
The examples of the comparative file contents (before and after the pre-processing accordingly) are demonstrated in Figures~\ref{fig:clean_numbers1}, \ref{fig:remove_garbage1}, and \ref{fig:remove_garbage2}.
\begin{figure}[H]
    \centering
    \includegraphics[width=\textwidth]{figures/clean_numbers_swapped.PNG}
    \caption{Cleaning page and line numbers from depositions (before vs. after)}
    \label{fig:clean_numbers1}
\end{figure}
\begin{figure}[H]
    \includegraphics[width=\textwidth]{figures/remove_garbage_characters.PNG}
    \caption{The results of cleaning garbage characters, excess spaces, and service page numbers}
    \label{fig:remove_garbage1}
\end{figure}

\begin{figure}[H]
    \includegraphics[width=\textwidth]{figures/remove_garbage_characters1.PNG}
    \caption{The results of cleaning garbage characters, excess spaces}
    \label{fig:remove_garbage2}
\end{figure}

It should be mentioned that some files may contain line numbers not only in the beginning of the line, but within the text as well.
An example of such a file is given in Figure~\ref{fig:clean_numbers2}.
Also, garbage characters that cannot be identified automatically exist, such as recognized handwritten phrases in certain documents, which might not appear in any other file.

\begin{figure}[H]
    \includegraphics[width=\textwidth]{figures/nums_within.png}
    \caption{Line numbers within the document}
    \label{fig:clean_numbers2}
\end{figure}

\item Tokenization and stemming

Some natural language processing approaches require the input text to be represented in the form of separate instances, i.e., tokens.
For other tasks, it might be necessary to bring the tokens into their initial form.
This operation is also known as stemming or lemmatization.
This topic is discussed in detail in Section~\ref{section:toklem}.
Following the received requests and anticipating potential needs, we prepared a Python script that breaks down a single .ocr file into tokens and returns a file with the original lemmas of the tokens.
A screenshot with the processing outcome is shown in Figure~\ref{fig:lemma_results}.

\begin{figure}[H]
    \includegraphics[width=\textwidth]{figures/lemma_results.PNG}
    \caption{Results of tokenization and lemmatization (before vs. after)}
    \label{fig:lemma_results}
\end{figure}

\end{enumerate}

\section{Deliverables and Timeline}
In order to keep track of our progress, we developed a list of deliverables.
The deliverables are summarized in Table~\ref{tab:deliverables}.

\begin{table}[H] 
\caption{Deliverables}
\label{tab:deliverables}
\centering
\begin{tabular}{| m{10cm} | m{3cm} |}
\hline
           Task & Completion date \\ 
\hline
Sample of metadata JSON files and document text JSON files to be consumed by the ELS team &09.17.19     \\ 
\hline
Complete structure of the MySQL metadata database &09.24.19 \\
\hline
JSON text files for deposition documents (ELS) &10.02.19 \\
\hline 
JSON metadata files for deposition documents (ELS) &10.03.19 \\
\hline 
Tokenization and lemmatization done for the TML team &10.08.19 \\
\hline 
Cleaned content of deposition documents (TML) &10.24.19 \\
\hline
Content of article documents extracted for TML team &10.24.19 \\ 
\hline
Different OCR approaches tested on a single PDF &10.28.19 \\
\hline 
Extracted metadata for all the document types for the ELS team &11.07.19 \\
\hline 
Cleaned and pre-processed article-style documents (TML) &11.15.19 \\
\hline
Generated JSON metadata files for 1M documents &11.18.19\\
\hline
Different OCR approaches tested on a sub-set of PDFs &11.20.19 \\
\hline
Generated JSON data files for 1M documents &11.20.19 \\
\hline
Recommendations on choosing OCR tools, including comparative table &12.02.19 \\
\hline
Script for generating JSON metadata files with the field containing cleaned document content &12.03.19 \\
\hline
Script for UNIT testing and validating of JSON files &12.03.19 \\
\hline
JSON metadata files for all 14M documents &12.06.19 \\
\hline
JSON metadata+text files for 100K documents as requested by the ELS team &12.07.19 \\
\hline
All the scripts uploaded to the GitLab repository &12.09.19 \\
\hline
\end{tabular}
\end{table}

