%-------------Literature review, coordinated with the references section
\chapter{Literature Review}
The prescribed textbook \textit{"An Introduction to Information Retrieval"} \cite{info_ret_book} has been immensely useful during the course of this project. The book helped us understand the basics of tokenization and lemmatization, key steps in pre-processing.

\section{Overview and Expectations}
We closely interacted with Dr. Townsend to understand his requirements.
His research focuses on understanding the organizational strategies and corporate tactics utilized by tobacco companies during the second half of the 20\textsuperscript{th} century to fight through the tobacco settlement cases.
The aim of his research is to construct a timeline and uncover the most prominent characters and analyze the roles companies have played in these cases.
Dr. Townsend was expecting an easily navigable search engine that would return noise-free results, i.e., an interface that facilitates mass retrieval of files based on certain criteria as opposed to them being fetched individually.
Examples include downloading all documents on a page filtered by conditions, and seeing results that bundle documents around a certain case.
It should also be able to identify cases with a common witness.
During subsequent meetings with Dr. Townsend, he expected the teams to provide him with some features in the user interface provided by the front-end team. This included  export functionality and visualization for the tobacco documents showing the data features and the document clusters.

\section{Tobacco Settlement Documents}
UCSF’s Industry Documents Library (IDL) hosts an archive of 14 million documents created by tobacco companies about their advertising, manufacturing, marketing, scientific research, and political activities~\cite{ucsf_truth_tobacco_documents}.
Truth Tobacco Industry Documents were created in 2002 by the UCSF Library.
The main intention was to provide access to the tobacco industry documents produced during litigation involving 39 states and the seven major tobacco industry organizations.
These documents come in various types and could be any of the following: company memos, letters, depositions, etc. The system we build needs to cater to all these variants.

UCSF also provides a list of exhaustive field names~\cite{ucsf_tobacco_dataset_info} that depicts the metadata of the documents. This is being used as a reference point, and we are using this to parse and get relevant information out of the data since it is not well-documented.

The tobacco settlement document report~\cite{vtechworks:tobacco_settlement} by the team that worked on this project for CS 4624 (Multimedia, Hypertext, and Information Access) during the spring 2019 semester, has also been a point of reference.
Previous work in this area had been done in June 2019 and we have been studying the team reports and presentations as well, in order to get a better understanding of the effort in this space.
It contains information about various methods incorporated, such as creating scripts for parsing the various documents and coming up with the metadata and doc2vec model implementation. 

We had used these resources as a starting point for analyzing the data to come up with a proper knowledge base for the metadata and text extraction from the 14M documents.


\section{Tokenization and Lemmatization}
\label{section:toklem}
Tokenization refers to chopping a given sentence or document unit into pieces, called tokens, perhaps throwing away certain characters such as punctuation.
A token is an instance of a sequence of characters in some particular document that are grouped together as a useful semantic unit for processing.
Tokenization accomplishes a very crucial task in pre-processing, to enable the handling of semantic issues in subsequent stages of machine processing.
It also contributes to a structural description of an input sentence.
The tokens become the input for other processes like parsing and text mining.
Tokenization plays a large part in the process of lexical analysis~\cite{tokens_info}.
Figure~\ref{fig:tokens} shows the result of the tokenization undertaken for a sample deposition file.

\begin{figure}[H]
    \centering
    \includegraphics[width=\textwidth, height=2.5cm] {figures/tokens.png}
    \caption{Tokens of fghb0000.ocr}
    \label{fig:tokens}
\end{figure}

For grammatical reasons, documents tend to use different forms of a word, such as \textit{organize}, \textit{organizes}, and \textit{organizing}.
Additionally, there are families of derivationally related words with similar meanings, such as \textit{democracy}, \textit{democratic}, and \textit{democratization}.
In many situations, it would be useful for a search for one of these words to return documents that contain another word in the set.
Stemming and lemmatization refer to reducing inflectional forms and sometimes derivationally related forms of a word to a common base form~\cite{info_ret_book}. 
The difference between stemming and lemmatization is that a stem might not be an actual word, whereas a lemma is an actual language word or root form.
Stemming follows an algorithm with steps to perform on the words, which makes it faster. Figure~\ref{fig:stemming} shows the result of the stemming on a sample set.

\begin{figure}[H]
    \centering
    \includegraphics[width=0.5\textwidth]{figures/stem_sample.PNG}
    \caption{Stemming sample\cite{diff_stem_lemm}}
    \label{fig:stemming}
\end{figure}

In contrast, lemmatization uses a lexicon, such as in the WordNet corpus, to produce lemma, which makes it slower than stemming.
If we are building a language application in which language is important one should use lemmatization as it uses a corpus to match root forms~\cite{lemma_info}.
If confronted with the token \textit{saw}, poor stemming might return just \textit{s}, whereas lemmatization would attempt to return either \textit{see} or \textit{saw} depending on whether the use of the token was as a verb or a noun.

Lemmatization takes into consideration the morphological analysis of the words.
To do so, it is necessary to have detailed dictionaries that the algorithm can look through to link the form back to its lemma~\cite{diff_stem_lemm}. Lemmatization is used in text mining, which involves analyzing the texts written in natural language, and extracting high-quality information.
Text mining includes text categorization, text clustering, sentiment analysis, document summarization, and entity relation modeling (i.e., learning relations between named entities).
For document clustering, lemmatization reduces the number of tokens that contain the same information and hence speeds up the whole process.
Figure~\ref{fig:lemmatization} shows the result of the lemmatization on a sample set.
\begin{figure}[H]
    \centering
    \includegraphics[width=\textwidth]{figures/lemma_sample.PNG}
    \caption{Lemmatization sample\cite{diff_stem_lemm}}
    \label{fig:lemmatization}
\end{figure}

Examples of lemmatization include:

am, are, is => be

car, cars, car’s, cars’ => car

been had done languages cities mice => be have do language city mouse

\section{PDF Processing Techniques}
PDF is one of the most important and widely used digital formats, used to present and exchange documents.
The tobacco dataset thus is shared as 14M PDF files.
The current dataset also contains OCR files, but the existing ones have garbage characters and line numbers included.
Some .ocr files have text in the format of JSON data.
Hence, we explored many PDF processing techniques mentioned below to improve the PDF processing step.
Figure~\ref{fig:original_pdf} is the original PDF file: jydp0228.pdf that has been processed by the following techniques.

\begin{figure}[H]
    \centering
    \includegraphics[width=13cm]{figures/Original_pdf.PNG}
    \caption{Sample scanned document fetched from the UCSF site: jydp0228.pdf}
    \label{fig:original_pdf}
\end{figure}

\subsection{PDFMiner}
 PDFMiner focuses entirely on getting and analyzing text data from PDF documents.
 It allows obtaining the exact location of text on a page, as well as other information such as fonts or lines.
 It includes a PDF converter that can transform PDF files into other text formats (such as HTML) \cite{pdf_process}.
 This method tries to maintain the data format, for example, tabular format, but data in different columns gets merged into a single column.
 It also takes into consideration the structure of the data; this is shown in Figure~\ref{fig:pdfminer}.
 \begin{figure}[H]
    \centering
    \includegraphics[width=\textwidth]{figures/PDFMiner.PNG}
    \caption{Sample scanned document processed by PDFMiner}
    \label{fig:pdfminer}
\end{figure}
 
\subsection{PyPDF2}
PyPDF2 is capable of splitting, merging together, cropping, and transforming the pages of PDF files.
It can also add custom data, viewing options, and passwords to PDF files.
It can retrieve text and metadata from PDFs as well as merge entire files together~\cite{pdf_process}.
This technique extracts just the text from the PDF but does not maintain the format of the data, for example, tabular format.
It does not take into consideration the field structure, for example, the title is merged with the rest of the text as shown in Figure~\ref{fig:pypdf2}.

\begin{figure}[H]
    \centering
    \includegraphics[width=\textwidth]{figures/PyPDF2.PNG}
    \caption{Sample scanned document processed by PyPDF2}
    \label{fig:pypdf2}
\end{figure}

\subsection{Abbyy Cloud OCR SDK}
It is a web-based AI-powered cloud OCR SDK (Software Development Kit) which provides excellent text recognition, PDF conversion, and data capture functionalities, enabling it to convert scans into searchable PDF documents \cite{abbyy}.
It maintains the data format and structure as shown in Figure~\ref{fig:abbyy_ocr}.
\begin{figure}[H]
    \centering
    \includegraphics[height = 7 cm, width=\textwidth]{figures/Abbyy_OCR.PNG}
    \caption{Sample scanned document processed by Abbyy Cloud OCR SDK}
    \label{fig:abbyy_ocr}
\end{figure}

Table~\ref{tab:ocr_method} summarizes the performance of the above mentioned OCR methods for different document types.
\begin{table}[H] 
\caption{OCR methods comparison}
\label{tab:ocr_method}
\centering
\begin{tabular}{ *5{>{\raggedright\arraybackslash}p{3cm}}}
\hline
& \textbf{UCSF} & \textbf{PyPDF2} & \textbf{PDFMiner} & \textbf{Abby Cloud OCR} \\
\hline
Two Column PDF's & Poor to Average
(comparable to pdfminer) & Poor & Average (comparable to UCSF) & Good \\
\hline
Newspaper Articles & Very poor & Extremely poor / No result & Very poor & Average \\
\hline
Tables & Average (comparable to pdfminer) & Poor & Average (comparable to UCSF) & Excellent \\
\hline
Plain texts, letters & Excellent & Good & Excellent & Excellent \\
\hline
Handwritten Texts & Very Poor & Extremely poor & Very Poor & Poor to Average \\
\hline
\end{tabular}
\end{table}

