\documentclass{article}

\usepackage[utf8]{inputenc}
\usepackage{graphicx}
\usepackage{xcolor}
\usepackage{cite}
\usepackage{float}
\usepackage{listings}
\usepackage{minted}
\usemintedstyle{monokai}
\usepackage[toc,page]{appendix}
\usepackage{indentfirst}
\usepackage{hyperref}
\hypersetup{
    colorlinks=true,
    linkcolor=blue,
    filecolor=magenta,      
    urlcolor=cyan,
}
 
\urlstyle{same}

\renewcommand{\listingscaption}{Code}
\newcommand{\fixme}[1]{\textcolor{red}{#1}}

\title{Fall 2019 CS5604 Information Retrieval  Final Report\\
Front-end and Kibana (FEK)}
\author{Edward Powell \and Han Liu \and Yanshen Sun \and Chao Xu \and Rong Huang}



\begin{document}

% @article{manning2010introduction,
%   title={Introduction to information retrieval},
%   author={Manning, Christopher and Raghavan, Prabhakar and Sch{\"u}tze, Hinrich},
%   journal={Natural Language Engineering},
%   volume={16},
%   number={1},
%   pages={100--103},
%   year={2010},
%   publisher={Cambridge university press}
% }

\maketitle
\setcounter{page}{0}
\thispagestyle{empty}

\begin{center}
Instructed by Professor Edward A. Fox\\
Assisted by Ziqian Song\\
\vspace{1cm}
\begin{figure}[hbt]
  \includegraphics[width=\linewidth]{figures/vt_logo.jpg}
\end{figure}

\vspace{1cm}
Virginia Polytechnic Institute and State University\\
Blacksburg, VA 24061
\end{center}


\newpage

\begin{abstract}
During the last two decades, web search engines have been driven to new quality levels due to the continuous efforts made to optimize the effectiveness of information retrieval.
More and more people are becoming satisfied during their information retrieval processes, and web search has gradually replaced older methods, where people obtained information from each other or from libraries.
Information retrieval systems are in constant interaction with users and help users interpret and analyze data. 
Currently, we are building the front end of a search engine, where users can explore information related to Tobacco Settlement documents from the University of California, San Francisco, as well as the Electronic Theses and Dissertations (ETDs) of Virginia Tech (and possibly other sites).
This report introduces the current work of the front-end team to build a functional user interface, which is one of the key components of a larger project to build a state-of-the-art search engine for two large datasets.
We also seek to understand how users search for data, and accordingly provide the users with more insight and utilities from the two datasets with the help of the visualization tool Kibana.
Already, a search website, where users can explore the two datasets, Tobacco Settlement dataset and ETDs dataset, has been created.
A series of functionalities of the searching page have been realized, for instance, the login system, searching and filter functions, a Q\&A page, and a visualization page.



\end{abstract}

\newpage

\tableofcontents
\newpage
\listoffigures

\newpage

%%%%%%%%%% Overview %%%%%%%%%%%
\section{Overview}
Our team designed a website \footnote{This is the present website link: \url{http://2001.0468.0c80.6102.0001.7015.b2eb.3731.ip6.name:3000}. Currently, the website can only be accessed on Virginia Tech's network. Furthermore, one can \href{https://www.nis.vt.edu/ServicePortfolio/Network/RemoteAccess-VPN.html}{VPN} into Virginia Tech's network if one needs access remotely. In the future, there will be a different link where the project is stored, and the plan is to make it accessible from anywhere.} for user interaction that allows the exploration and analysis of the Electronic Theses and Dissertations (ETD) dataset from Virginia Tech, and the tobacco dataset from the University of California, San Francisco.

The front-end and Kibana (FEK) team aimed to provide an interface between users and an accurate search system, which comprises intelligent retrieval, a complex recommendation system, and visualizations and analysis tools for both users and developers to better comprehend the data.
In addition, there should be a small user study and customer discovery layer to help with later iterations of the website.
\subsection{Project Introduction}
The full-text search project includes a web application that provides full-text search of Virginia Tech Electronic Theses and Dissertations (ETDs), and of Tobacco Settlement documents.
Aside from basic searching, the application also provides personalized recommendations and data visualizations to assist users in discovering the trends and patterns of their field of interest.
\subsection{How Was It  Managed?}

The main task of the front end team is to connect with the data provided by the Elasticsearch team, and to produce a user-friendly interface for searching, filtering, summarizing, and visualizing the data. 
Aside from the web pages, the team should also produce a ``bridge'' for customized communications between UI and Elasticsearch.
Considering that this is a small application with only a few pages in the front-end, and communication and user authority functions in the ``bridge,'' the team decided to make the application as light-weight as they could. 

As a JavaScript library, React\cite{react} is now one of the most popular front-end development tools \cite{jetbrainreact}\cite{stateofjsreact}.
The team made a comparison between Vue.js (a popular front-end framework) and React.
Although Vue.js provides more ready-to-use function modules, the structure of React code is easier to understand.
Therefore, React was eventually chosen as the front-end development tool for this project. 

For ``bridge" development, the team chose Flask\cite{flask} for the same reason as they chose React. 
Again, the major data source and processing functions of the website are provided by Elasticsearch, where the functions of the ``bridge" include only data communication between front-end and Elasticsearch and a sign in / sign up system.
Flask would be a good tool for construction of a simple application that does not necessarily follow an MVC structure.

With regard to functionality, we roughly divided our functions into two parts.
One is the searching and browsing of the documents, which includes basic searching and filtering of data, document detail pages, and a login system for personalized recommendations.
The other is visualization of data, which aims to assist users exploring distributions and correlations of attributes of documents, and developers analyzing user behavior and monitoring performance of clusters.

The main tool employed for visualization is Kibana, a tool that is designed specifically for Elasticsearch data visualization.
Both our basic searching and visualization pages are able to exchange information with the Elasticsearch container through a RESTFul API.
Furthermore, basic searching in our visual pages will also allow filters. Once a filter is applied, all the charts will be updated to only show the information of remaining data.
\subsection{Problem/Challenges}
\begin{itemize}
	\item How to exchange data between our web application and Elasticsearch through Python, JavaScript, and Kibana?
	\item How to generate queries with a user's input?
	\item How can the system fully support both searching activities of both amateur users and professional users?
	\item How to generate and persist searching logs and user clicking logs?
	\item How to generate a secure and fully functional user login system?
	\item How are admin users different from general users?
	\item How to produce visualizations to help users understand our datasets? 
	\item How to acquire and visualize system monitoring data?
	\item How to produce a self-checking system based on unit tests?
	\item How to evaluate our product?
\end{itemize}
\subsection{Solution Developed}
For the search interface part, the team decided to make use of an open-source library: React (Reactive Search).
The library provides functions that can easily send requests to, and pull data from, Elasticsearch.
Developers can generate Elasticsearch queries simply by filling in determined fields in a query template.
It can also automatically generate decent CSS and animations for HTML elements.
This enables the team to develop a user-friendly website with only a little knowledge of JavaScript and HTML programming.

Furthermore, there are users with different familiarity of the datasets as well as Elasticsearch.
For example, first-time-users (amateur user) usually don't know about existing fields and searching tricks in Elasticsearch.
For this kind of user, we provide auto-completion and a series of filters with unique value lists, so that amateur users can use our application just like a common search-engine interface.
For those who know the field names in the datasets and are familiar with Kibana and Elasticsearh searching tricks, we also support ``:" and ``\&\&" operators in search queries, which work just the same way as in Kibana.


Elasticsearch also provides an API for Python.
With the API, Python can communicate with Elasticsearch in the manner of a general HTTP request clause.
To customize the format of user logs, actions of users (searching, setting filters, choosing a determined article, etc.) are first sent to Python for processing, and then saved to Elasticsearch, as well as to files in Ceph (for persistence). 
For the user login system, the team provides ``register" and ``login" interfaces for general users, and ``delete user", ``change username" and ``change email" for admin users.
For the sake of security, passwords are encoded before sending to a MySQL database.
Email addresses are also requested during registration for identity validation. 

After discussions with the data collection teams (CME and CMT), Pie chart, bar chart, sun burst chart, tag-cloud, heat-map, and line chart were chosen for article features.
A system monitoring page is also built based on the flask monitoring dashboard, so that admin users can acquire more information about the performance of the web application. 

Unit test is implemented with ``unittest", a framework provided by the Python core library.
More unit test functions should be produced for INT’s CI/CD, so that we can guarantee that no push to GitHub brings errors to the project.
\subsection{Future Work}
\begin{itemize}
	\item Complete unit tests for INT’s CI/CD.
	\item Design evaluation module for our system.
	\item Build user behavior and system monitoring visualization pages.
	\item Add more features by working with the TML team, such as the recommendation system that they have developed with the use of the logs.
	\item Welcome to our Github to report issues and give feedbacks in the future.
\end{itemize}

%%%%%%%%%% Literature Review %%%%%%%%%%%
\newpage
\section{Literature Review}
\subsection{The Importance of Information Retrieval Systems}
The world is changing rapidly, not only in terms of dress, eating habits, transportation, etc., but also the way people search for information.
In this so-called Information Age, hundreds of millions of people use web search engines to retrieve information every day.
An information retrieval system can be considered as a system that reads the users' questions and performs a search through an internally stored collection of documents to present the most relevant information to the users \cite{manning2010introduction}. Information retrieval was originally a part of librarianship.
However, it has expanded into a variety of fields such as business management, office automation, genomic databases, fingerprint identification, medical image management, knowledge finding in databases, and multimedia management.
\subsection{The Importance of Tobacco Settlement Dataset and Virginia Tech ETDs Dataset}
\paragraph{Tobacco Settlement Dataset:}
In 1998, several million once secret documents, over 35 million pages, became available to the public as the result of legal action~\cite{tobacco}.
The documents include letters, memos, telexes, emails, and research reports; strategic, political, and organizational plans; organizational charts, lists of consultants, invoices, and copies of cheques paid; testimony in courts and before legislatures; advertising, marketing, media, and public relations strategies; and several other categories.
Since 1998 the documents have been used in many ways to support the right of citizens to know how products, even legal products, affect their health, and to gain relief from past and continuing injury.
The documents so used become a powerful weapon for protecting the public’s health. 
These documents can continue to influence the public and are also used to show governments what the tobacco companies know about their products—and tried to hide—and so can lead to legislation intended to reduce the harm tobacco causes.
Therefore, it is important to have more high quality platforms that people can search for these documents and understand the risks of smoking and what the tobacco companies want to hide~\cite{tobacco}.
\paragraph{Virginia Tech ETDs Dataset:}
An ETD is an electronic version of a thesis or dissertation.
An ETD is formatted just like a traditional thesis or dissertation (with pagination, tables, figures, references, etc.), but is saved as a PDF file and submitted electronically to the university.
An ETD makes a student's research immediately accessible to a broad audience, while reducing both printing and binding costs for the researcher. 
The ETDs of Virginia Tech benefit people in a variety of ways.
For instance, research in VT's ETD system is widely disseminated and free to anyone, including scholars at under-resourced institutions and in developing nations, as long as there is an Internet connection.
Also, these works will be preserved for decades or even centuries, unlike research posted on a personal web page or published by a company that might go out of business.
In a word, the ETD dataset is a great resource for researchers or other people to learn useful knowledge~\cite{ETDs}.

\subsection{Realization of a Web-based Information Retrieval System}
Elasticsearch, which is an open-source, distributed,  and RESTful search engine, makes it significantly easier to build a search engine that can analyze a great amount of information effectively and deliver relevant results to users \cite{divya2013Elasticsearch}.
Elasticsearch is built to handle large amounts of data while maintaining user-friendly and accessible methods to retrieve data.
There are a handful of APIs that allow applications from any language or framework to access an Elasticsearch index.
Postman is the platform we use to guarantee the efficiency of API calls between our interface and Elasticsearch. 
 
To ease the users' searching processes, multiple filters should be applied on the searching website for users to access the information, and present the most relevant information to the users~\cite{niu2015analyzing}.
To help users to digest the information faster, the design of the interface should be easy and appealing to use, and allow users to effectively find the answers they desire.

In addition, Kibana is a good way to search, view, and perform advanced data analysis  with  data  stored  in  Elasticsearch  indices using a REST API, and to create graphs and charts to help present aggregations for users to visualize in the field that they want~\cite{robles2017using}.
It  is  built on  top  of Elasticsearch and provides a user-friendly interface that can easily handle a large amount of data, and helps create dashboards and query data in real-time.

%%%%%%%%%%% Requirements %%%%%%%%%%%%%
\newpage
\section{Requirements}
\subsection{Introduction}
Our goal is to build a user-friendly web interface, which provides several basic functionalities, clear and well-formatted layouts, and a feedback window for user support.
\subsection{Functionalities}
Below is the list of functionalities our team will provide:
\begin{itemize}
    \item ETD searching
    \item Tobacco searching
    \item Recommendations for searching
    \item Content filtering by author, date, location, etc.
    \item List of content sorted by relevance score
    \item Detailed content found by clicking on any content anchor in the list 
    \item Login page and sign up page
    \item Log system to record user activities 
    \item Project introduction page with project overview, all six teams’ profiles, and descriptions
    \item Pie charts  
\end{itemize}
\subsection{Quality/Performance}
The quality of a front-end interface can be measured by considering appearance and functionalities.
For appearance, the interface should be both concise and informative.
``Concise'' here means that the level and partitions of sections should be identifiable, so that users can easily find the information they are looking for.
``Informative'' indicates that the interface should provide enough messages for (1) operational guidance, (2) connections among records, and (3) connections between a record and the group containing it.
For functionalities, the aim is to reduce the time when searching for a useful item.
For example, a recommendation should be provided so that they can find related documents without thinking of a new query.

The performance of front-end pages can be assessed from three aspects: 
\begin{enumerate}
    \item basic elements loading time.
    \begin{itemize}
        \item text loading time.
        \item image loading time.
        \item front-end functions running time.
\end{itemize}
    \item front-end response time.
    \item back-end response time.
\end{enumerate}
The priorities of these three elements should be: texts$>$functions$>$images.
Front-end response time includes the time consumed by a front-end function processing a user action.
Back-end response time is the time interval between sending a request and getting a response from the back-end. 
Besides, the total time consumption and the changing rate of response time with the increment of concurrency should also be examined. 
All of the above measurements will be checked with online tools such as Siege (front-end time consumption) and Locust.io (server response time).
The system will be modified until the total delay is less than 300ms.

\subsection{User Support}
\subsubsection{User Manual}
To make users have a positive navigating experience on our website, we will provide a user manual.
The user manual provides detailed instructions about how to utilize functionalities on the website. 

\subsubsection{Articulatory and Semantic Feedback}
Articulatory and semantic feedback will be given once users encounter any possible problems or errors such as those listed below:
\begin{itemize}
    \item Invalid password or username on login page.
    \newline Solution: printing out an error message ``Invalid password or username.''
    \item An email address has been registered by previous users.
    \newline Solution: printing out an error message ``This email address has been used.''
    \item Connection error.
    \newline Solution: navigate to the 404 page.
\end{itemize}




%%%%%%%%%%% Design %%%%%%%%%%%%%%
\newpage
\section{Design}\label{sec:design}
\subsection{Kibana Utilities}
Kibana is a tool that we can use to search, view, and perform advanced data analysis with data stored in Elasticsearch indices, and visualize the data in a variety of tables, charts, histograms, tables, and maps \cite{esrankingapis}.
It is built on top of Elasticsearch and leverages the functionalities of Elasticsearch.
It is a part of the ELK stack, which consists of Elasticsearch, Logstash, and Kibana.
It connects to Elasticsearch using the REST API.
It provides a user-friendly interface that can easily handle a large amount of data and helps create dashboards that are easy to build.
It aids advanced users who query data in real-time.
In our project, we will utilize Kibana to create a series of charts and graphs, including pie charts, bar graphs, line graphs, etc.
A collection of these visualizations related to the ETD dataset and tobacco dataset will be provided on our website for users to explore.
Users can search across all documents with any filters or text.
Kibana also supplies many other features.
Inspection of the network traffic on Kibana also will be conducted to see how it works with Elasticsearch.

\subsubsection{Kibana Aggregations}
Aggregations represent a collection of documents from a specific search query, and can lead to desired visualizations in Kibana.
Some of the important aggregations include metric aggregation and bucket aggregation. Metric aggregations compute metrics based on values extracted in a specified way from the fields of documents.
Bucket aggregations build buckets of documents depending on certain criteria.
All types of aggregations are listed below.
In the visualization builder, we can choose metric aggregation for the visualization’s Y-axis, and bucket aggregation for the X-axis.

\subsubsection{Charts}
To create a Kibana visualization, a variety of choices are available to choose from, such as charts, data, metrics, maps, and time series.
Each of them has its own focus on how to present extracted data.

\paragraph{Basic Charts:} 
\begin{itemize}
    \item Pie charts are used to display the parts of some whole.
    In Kibana, they are determined by metrics aggregation.
    \item Vertical bar charts are used for clarifying trends with a large dataset.
    Kibana determines it by metrics aggregation. 
    \item A heat map is a graphical representation of data where the individual values contained in a matrix are represented as colors.
    The color for each matrix position is determined by the metric aggregation.
\end{itemize}

\paragraph{Data:} 
\begin{itemize}
    \item A data table is used to display the raw data of a composed aggregation.
    \item Goal and gauge visualization enables displaying the progress toward an identified goal.
\end{itemize}

\paragraph{Maps:}
\begin{itemize}
    \item A coordinate map associates the results of aggregation with geographic locations. 
    \item Thematic maps use shapes' color intensities to represent a metric’s value.
\end{itemize}

\paragraph{Time Series:} 
\begin{itemize}
    \item Timelion is a visualization tool for time series in Kibana.
    It has the ability to add multiple time series to one chart.  
    \item Time Series Visual Builder allows you to combine an infinite number of aggregations and pipeline aggregations to display complex data in a meaningful way.
\end{itemize}


\subsection{Postman Utilities}
Postman is a collaborative platform for API development. 
Users can quickly and easily send REST requests directly within Postman.
In our case, we use Postman to quickly check Elasticsearch APIs and try to display the response data in a helpful manner.
Figure \ref{fig:postmanrequest} shows the request demo for accessing Elasticsearch via Postman.
Postman helps with understanding the requests to Kibana and Elasticsearch, allowing us to structure our back-end system more quickly and efficiently.
Users also can use Postman directly to search for data on Elasticsearch \footnote{This \href{https://blog.red-badger.com/2013/11/08/getting-started-with-elasticsearch}{link} provides examples for how to create and get from an Elasticsearch index using Postman \cite{postelastic}.}.


\begin{figure}
  \includegraphics[width=\linewidth]{figures/postman_request_demo.png}
  \caption{Postman request}
  \label{fig:postmanrequest}
\end{figure}

\subsection{Build Our Own User-friendly Website}
Our main goal is to build a user-friendly website for users to retrieve data from Elasticsearch.
The website architecture should include back-end and front-end systems.
Our back-end system is mainly for the user system and log system in our website infrastructure.
Elasticsearch is also the back-end for us to get data and collaborate with other teams.

\subsubsection{Infrastructure}
\begin{figure}[h]
  \includegraphics[width=\linewidth]{figures/infrastructure.jpg}
  \caption{Infrastructure}
  \label{fig:infrastructure}
\end{figure}

As shown in Figure \ref{fig:infrastructure}, we depict the whole infrastructure of our website.
It consists of 5 main modules; in each module there are several parts.
We would also have connections to MySQL, Elasticsearch, and Kibana for storing and retrieving data.


\subsubsection{Back-end}
For the back-end of this system, Python and MySQL will be primarily used.
Data and documents are retrieved from the datasets by using the Elasticsearch API.

\paragraph{Connection to Kibana and Elasticsearch:}
The connection to Kibana and Elasticsearch is mainly executed by the Python request package and REST APIs on Kibana and Elasticsearch.
This is direct and simple, but it could not provide us a convenient way to build a whole searching page with fields and filters.
Accordingly, we use ReactiveSearch, which is an Elasticsearch UI components library for React and React Native.
This allows us to use CORS (Cross-Origin Resource Sharing) HTTP requests to connect to Elasticsearch.
Basically, it still works on the original REST APIs of Elasticsearch.
For example, we send our searching queries to Elasticsearch via API call: `` POST /index/\_msearch? ''.
The details are discussed in Section \ref{sec:dev}.


\paragraph{Connection to Front-end:}
We use Python Flask to connect our back-end system to the front-end system.
We use Flask Render to build our website on the targeted URL path.
Plus, some data processing would be done in the back-end system and then transferred to the front-end to be displayed.
Then, its JavaScript code would further process the received data and display the expected format.
Additionally, we write the POST API to receive log data posted from the front-end system.

\paragraph{User login system:}
Users may register on the website, and the user information will be stored in MySQL.
We require standard input for usernames such as a unique name in the database that has a minimum of 4 characters and a maximum of 20.
We run a MySQL container on the cloud server and use HTTP to connect to MySQL.
Considering it would lose user data once the machines shut down, we are going to persist user data with Ceph.
The user validation should also be processed in this system.
We save and retrieve data in MySQL to maintain the user system.

\subsubsection{Front-end}
Utilizing the Flask framework, our website runs on Python, JavaScript, HTML, and CSS. 
% There are five main modules on the website: user login, searching bar, filters block, field block, and result display block. 
Our homepage contains navigation bars and a login/register button for users. Also, considering we have two different datasets, we offer two searching pages for users to choose from. There are two links for two searching pages on the homepage. In each searching page, there are three parts presented to the user.
There is a search bar, filters with fields, and a results block.

\paragraph{User login:}
The entry offers users the ability to register and login to our system.
From their profile page, users can see their searching history on the website.
There will also be a logout button.
Users who do not want to create an account will still be able to take advantage of many of the features the website will offer.
\\
There are two kinds of users in our system: normal users and admin users.
Normal users would have the ability to search, and to look at visualization pages.
Admin users would have access to the admin page where they can be CRUD users in MySQL.
Also, admin users can upload new files to two datasets and they can employ the monitoring dashboard where they can monitor all APIs in this project. 

\paragraph{Searching bar:}
Users send queries by putting some text in the search bar.
Possible suggestions are displayed as the user enters characters.
The auto-complete part is also based on the response from Elasticsearch.
The auto-completion would start from the third character that a user types in.
Additionally, we support searching in specific fields via ``fields'' + ``text'', like ``title: march''.
We also set different weights for various searching fields, e.g., we recognize that title is more important than other fields when users search for something.

\paragraph{Filters:}
Users will be able to self-define filters to request specific results.
For example, users could limit their searching fields by selecting some target fields.
This part would also show details of each field.
For example, it would show the number of related results in such fields.


\paragraph{Results display:}
The searching results will be displayed here.
It would automatically be updated as long as searching requests change.
When the user clicks one of the filters, then the results block would refresh immediately.
At the top of the results block, it shows the total number of results and the time cost.
This will give users an intuitive feeling of searching speed and quantity.
For users' convenience, we add a hyperlink on each title of results.
Users could click on the title and access the original PDF file.
Once a user clicks on the title and redirects to the original PDF file, we will record those activities in the log. 



\subsection{Website Glance}
% \fixme{website demo;  Realized functionalities}

\subsubsection{Homepage}
The homepage is shown as Figure \ref{fig:home}.
 \begin{figure}[H]
  \includegraphics[width=\linewidth]{figures/UI_main.png}
  \caption{Home Page}
  \label{fig:home}
  \end{figure}
  
\subsubsection{User Login and Register}
The login page is shown as Figure \ref{fig:login}, and the registration page is shown as Figure \ref{fig:register}.
\begin{figure}[H]
  \includegraphics[width=\linewidth]{figures/UI_register.png}
  \caption{Registration Page}
  \label{fig:register}
  \end{figure}
   \begin{figure}[H]
  \includegraphics[width=\linewidth]{figures/UI_login_1.png}
  \caption{Login Page}
  \label{fig:login}
  \end{figure}
\subsubsection{Searching Page}
There are two searching pages.
One is for searching the ETD dataset; the other is for searching the tobacco dataset.
The ETD searching page is shown as Figure \ref{fig:search_etd}.
The tobacco searching page is shown as Figure \ref{fig:search_tobacco}. 
 \begin{figure}[H]
  \includegraphics[width=\linewidth]{figures/searchingpage_etd.png}
  \caption{ETD Searching Page}
  \label{fig:search_etd}
  \end{figure}
  
 \begin{figure}[H]
  \includegraphics[width=\linewidth]{figures/searchingpage_tobacco.png}
  \caption{Tobacco Searching Page}
  \label{fig:search_tobacco}
  \end{figure}
 
Figures \ref{fig:search_etd_example} and \ref{fig:search_tobacco_example} show some examples for searching in the two datasets.
 \begin{figure}[H]
  \includegraphics[width=\linewidth]{figures/searchingpage_etd_example.png}
  \caption{ETD Searching Page}
  \label{fig:search_etd_example}
  \end{figure}
  
 \begin{figure}[H]
  \includegraphics[width=\linewidth]{figures/searchingpage_tobacco_example.png}
  \caption{Tobacco Searching Page}
  \label{fig:search_tobacco_example}
  \end{figure}
  
  
% \newpage 
\subsubsection{Visualization}\label{sec:visualization}
We have built two visualization pages.
One is for ETD settlement documents and the other is for tobacco settlement documents.
Different types of visualization have been applied, such as charts, data tables, and maps.

The visualization of ETD data is an interactive, comprehensive, and graphic page for users to have a better understanding of the data distribution by time, departments, degree level, and degree name.
Visualizations are shown in Figure \ref{fig:etd_visual}.

The visualization of tobacco data contains several types of graphs such as Data Tables, Tag Graph, Pie Chart, Area Graph, and Gauge.
These are shown in Figure \ref{fig:tobacco_visual}.
The keywords utilized mainly include brands, cases, languages, and topics.

Several steps are included in the process of creating visualizations.
They are shown below, as well as in Figure \ref{fig:visual_process}.
\begin{itemize}
    \item Select the proper type of graph for visualizing the data.
    \item Pick up a data set.
    \item Add buckets or aggregations for the graph.
    \item Add the created visualization into the dashboard.
\end{itemize}


\begin{figure}[H]
  \includegraphics[width=\linewidth]{figures/etd_visual.png}
  \caption{ETD Visualization Page}
  \label{fig:etd_visual}
  \end{figure}

\begin{figure}[H]
  \includegraphics[width=\linewidth]{figures/tobacco_visual.png}
  \caption{Tobacco Visualization Page}
  \label{fig:tobacco_visual}
  \end{figure}
  
 \begin{figure}[H]
  \includegraphics[width=\linewidth]{figures/process_visual.png}
  \caption{The Process of Creating a Visualization}
  \label{fig:visual_process}
  \end{figure}
% \begin{figure}[H]
%   \includegraphics[width=\linewidth]{figures/websitedemo.png}
%   \caption{Initial Web Design}
%   \label{fig:webdesign}
% \end{figure}

% Figure \ref{fig:webdesign} depicts our initial design of the front end website.


% \begin{figure}[H]
%   \includegraphics[width=\linewidth]{figures/search_results.png}
%   \caption{Searching results demo}
%   \label{fig:searchresults}
% \end{figure}
% Figure \ref{fig:searchresults} depicts the searching demo for ``breathe.''
% It is matching a text entry of source data.
% The results are ranked by scores in Elasticsearch.

% \newpage
\subsection{Log System}
% \fixme{To record user activities }
Each time the user searches something in the searching bar, it will send a request to Elasticsearch.
We jsonify the requests data and send it to the back-end API we created for processing requests data.
As the request data are poorly summarized and do not completely overlap with what should be in the log record, we re-format the JSON data by combining the JSON and user-session information, and merging query items.
Finally, the re-formatted JSON data is sent to Elasticsearch for persistence and further analysis.

% \fixme{Next Step is that we are going to record documents that are clicked by users. Users might click links to documents, we will record these activities.}
To determine what exact documents users are searching, we also record click activities when users select a hyperlink associated with a title.
The JSON log data will contain user information and document information.
Those kinds of logs should be used for better user recommendation. 

These two types of logs are stored in separate indices in Elasticsearch.
The logs are also saved in JSON files in Ceph as backups. 

\subsection{Recommendation}
Search quality evaluation starts with looking at the users of our search application and taking into account the things for which they are searching.
The challenge is to tweak this translation process from user entries to a concrete query, leading to search results that contain the most relevant information with respect to a user's information need.
There are two alternatives.

\subsubsection{Ranking System in Elasticsearch}
Basically, we use the default ranking of response data from Elasticsearch.
The ranking evaluation API \cite{esrankingapis} allows users to evaluate the quality of ranked search results over a set of typical search queries.
It supports various evaluation metrics for ranking. 
Currently, the following metrics are supported.

\paragraph{Precision at K (P@k):}
This measures the number of relevant results in the top k search results.
It is a form of the well-known precision metric that only looks at the top k documents.
It is the fraction of relevant documents in those first k results.

\paragraph{Mean Reciprocal Rank:}
For every query in the test suite, this metric indicates the reciprocal of the rank of the first relevant document.
For example, finding the first relevant result in position 3 means the reciprocal rank is 1/3.
The reciprocal rank for each query is averaged across all queries in the test suite to give the mean reciprocal rank.

\paragraph{Discounted Cumulative Gain (DCG):}
In contrast to the two metrics above, discounted cumulative gain takes into account both the rank and the rating of the search results.
The assumption is that highly relevant documents are more useful for the user when appearing at the top of the result list.
Therefore, the DCG formula reduces the contribution that high ratings for documents on lower search ranks have on the overall DCG metric.

\paragraph{Expected Reciprocal Rank (ERR):}
Expected Reciprocal Rank (ERR) is an extension of the classical reciprocal rank for the graded relevance case \cite{chapelle2009expected}.
It is based on the assumption of a cascade model of search, in which a user scans through ranked search results in order, and stops at the first document that satisfies the information need.

\subsubsection{Recommendation Data from TML Group}
% This needs further discussion with the TML group.
The TML group is still working on this part.
We have not been notified about data content so we have not collaborated yet.
Basically, it could play a role similar to the internal ranking system in Elasticsearch but with improvements and greater accuracy.
Additionally, we could list some specific recommendations for specific users.

%%%%%%%%%%%%% Implementation %%%%%%%%%%%%%%%
\newpage
\begin{figure}[h]
  \includegraphics[width=\linewidth]{figures/timeline.png}
  \caption{Timeline}
  \label{fig:timeline}
\end{figure}
\newpage
\section{Implementation}
% \fixme{This part should be with a timeline, list of milestones and deliverables, and discussion of evaluation that led to selection of the best methods}

The tools that we utilized for our implementation were primarily: Kibana, Python, JavaScript, HTML, CSS, and Elasticsearch.

\subsection{Milestones}
\subsubsection{Completed}
\begin{itemize}
  \item System running with Shakespeare data
  \item System running with a subset of both datasets
  \item System running with the full ingestion of both datasets
  \item Shakespeare data running with Kibana and testing its various modules and visualizations
  \item Shakespeare data tested with Postman
  \item Login and Register system
  \item Administrator has standard CRUD operations
  \item Filters that are specific to the ETD and Tobacco datasets are shown as facets in our interface
  \item Put our system into the CS testing container cluster
  \item Connected Kibana to Tobacco and ETD datasets
  \item Searching pages connected to two datasets: Tobacco and ETD
  \item Hyperlinking to original documents implemented
  \item Logs generated for TML team
  \item ETD dataset has an editable dashboard with a set of visualizations connected with Kibana.
  \item Tobacco dataset has an editable dashboard with a set of visualizations connected with Kibana.
  \item Built a framework for a FAQ page

\end{itemize}
\subsubsection{To Be Completed}
\begin{itemize}
  \item Implement an improved recommendation system based on the TML team's work
  \item Administrator upload for a new file for both datasets is ready but a back-end endpoint is needed from the INT group.
  \item A profile page that users can utilize and navigate should be developed.
  This page should also allow users to recover and change their password.
  \item A small user study to test the feasibility and usability of the website's functionality and UX
\end{itemize}
\subsection{Deliverable}
Our main deliverable is the front end website itself.
This deliverable is paramount.
It enables access to the entire class project and will help display all of the teams' work. 
\subsection{Intra-Evaluation Process}
For Elasticsearch request/response simulation, we have tested out Postman \cite{Postman} and dejavu\cite{dejavu}. 
Postman was eventually chosen as it had a concise interface and JSON could help us understand the structure of data. Elasticsearch-GUI\cite{esgui} and ElasticHQ\cite{elastichq} were also considered for the template of our searching page. 
However, the team finally decided to develop the pages by themselves, as it would be more flexible and the underlying functionalities could be easier to understand.
Initially, our group was going to use and had an early version of our website, that utilized PHP, but we decided to use Python because most of the group members are familiar with it, potential visualizations are easily obtainable, and there is an abundance of documentation for the language. 
\subsection{User Log Persistence}
There are mainly three functions to be considered regarding user log persistence.
The first one is passing the request information and user information from the front-end to the back-end (the ``bridge'').
Then queries generated by the front-end instead of user activity need to be discarded, while the remaining requests should be reformatted.
Finally, the reformatted JSON data is sent to Elasticsearch, as well as to Ceph (for persistence). 

In the first function, searching queries and article choosing requests are sent separately.
The former are sent during rendering, while the latter are sent by an "onClick" function through AJAX.

Although only searching and article choosing actions need to be recorded in the logs, there are many different types of requests sent to the back-end when a user interacts with the searching page.
For example, to load the initial list of articles, the front end needs to send a ``match all'' request to Elasticsearch.
To achieve ``auto-completion'' while a user is typing in the searching bar, the front-end is asked to keep sending "prefix" queries to Elasticsearch.
Therefore, a filter is supposed to be set so that the log could only contain a completed search or article choosing action. 

After acquiring the request bodies, the useful information is extracted and reformatted as log objects in JSON format.
The log is then stored in Elasticsearch through its API and in a file in Ceph through an OS operation. 

%%%%%%% User Manual %%%%%%%%
\newpage
\section{User Manual for Website}
The user manual is intended to help users navigate the website and understand the interface. The figure below depicts the basic workflow for an end user on the website. 

\begin{figure}[h]
\centering\includegraphics[width=100mm,scale=0.8]{figures/userworkflow.png}
  \caption{User Workflow}
  \label{fig:8}
\end{figure}
\subsection{A User Interested in the Tobacco Dataset}
\subsubsection{Scenario 1}
\begin{enumerate}
  \item A user will register for an account.
  \item A user will login to their account.
  \item From the homepage, a user will click on ``Search for Tobacco''.
  \item A user can begin to type in a keyword such as ``Reynolds'' to help narrow their search.
  \item Note: A user could click on a document from the auto-complete search bar.
  \item A user enters their search query and reviews the list of documents.
  \item A user clicks on a document to review the original PDF version.
\end{enumerate} 

\subsection{A User Interested in the ETD Dataset}
\subsubsection{Scenario 2}
\begin{enumerate}
  \item A user will register for an account.
  \item A user will login to their account.
  \item From the homepage, a user will click on ``Search for ETD''.
  \item A user can begin to type in a keyword such as ``Doctoral'' to help narrow their search.
  \item Note: A user could click on a document from the auto-complete search bar.
  \item A user enters their search query and reviews the list of documents.
  \item A user clicks on a document to review the original PDF version.
\end{enumerate} 

\noindent 
Figure \ref{fig:14} shows the basic database schema for our users table at the moment.
There are plans to add more fields that will be useful for the TML team as well as help log the system status. 
\begin{figure}[h]
\centering\includegraphics[width=50mm,scale=0.5]{figures/dschema.png}
  \caption{Database Schema for Users Table}
  \label{fig:14}
\end{figure}

\subsection{User Manual for Kibana}
\begin{figure}[h]
\centering\includegraphics[width=50mm,scale=0.5]{figures/discover.png}
  \caption{Kibana Discover}
  \label{fig:15}
\end{figure}
\noindent
To access Kibana that is connected to our Elasticsearch indices, a user must be on the VT network to access our website \footnote{This is the present website link: \url{http://2001.0468.0c80.6102.0001.7015.b2eb.3731.ip6.name:3000}. Currently, the website can only be accessed on Virginia Tech's network. Furthermore, one can \href{https://www.nis.vt.edu/ServicePortfolio/Network/RemoteAccess-VPN.html}{VPN} into Virginia Tech's network if one needs access remotely. In the future, there will be a different link where the project is stored, and the plan is to make it accessible from anywhere.}
The Discover tab is the easiest way for a novice to interact with the datasets by taking advantage of the graphical user interface of Kibana with preset filters.
Once on the Discover tab, a user needs to select a dataset as is shown in Figure \ref{fig:15}.
From here, users can filter their results and use the drop-down bar to see all the fields in the index.
Additionally, multiple fields can be filtered at one time. 

\begin{figure}[h]
\centering\includegraphics[width=\linewidth]{figures/devtools.png}
  \caption{Kibana Dev Tools}
  \label{fig:16}
\end{figure}
For a more robust and fine-tuned way to control the data, a user can click on the tool gear and manipulate the indices with more complex queries.
The page is displayed in Figure \ref{fig:16}.
These queries have a lot more potential, power, and flexibility than the defaults provided in the Discover page.
Weights of various fields can also be ``boosted'' here. 

Logs that show the usage of Elasticsearch can be found through Kibana.
These are automatically generated.
This could also be useful for the TML team.
Kibana has some visualizations that help to aggregate the data.
More details of this were explained in the Visualization Section \ref{sec:visualization}.
\clearpage

\subsection{User Manual for FAQ Page}
A frequently asked questions (FAQ) page has been developed on the website to help users understand the significance of this website and how to use it. 
\begin{figure}[h]
\centering\includegraphics[width=130mm,scale=1]{figures/FAQ.png}
  \caption{FAQ Page}
  \label{fig:11}
\end{figure}

%%%%%%%%%%%%%% Dev Manual %%%%%%%%%%%%%%%
\newpage
\section{Developer Manual}\label{sec:dev}
\subsection{Workflow }
\begin{figure}[h]
  \includegraphics[width=\linewidth]{figures/architecture.png}
  \caption{FEK Team Workflow}
  \label{fig:architecture}
\end{figure}
The workflow of our team is shown in Figure \ref{fig:architecture}.

\subsection{Data Exploration}
Data exploration for an index can be discovered using Elasticsearch. The documentation can be found \href{https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started-explore-data.html}{here}. 

Kibana has even better tools to explore an Elasticsearch index. Documentation for the data exploration can be found \href{https://www.elastic.co/guide/en/kibana/current/discover.html}{here} and documentation for visualizations can be found \href{https://www.elastic.co/guide/en/kibana/current/visualize.html}{here}.   
\subsection{Kibana Aggregations}
We focus on metric aggregations and bucket aggregations currently.
\paragraph{Metric Aggregation:} Metric aggregations compute metrics based on values extracted in a specified way from the fields of documents.
Metrics enable a variety of functionalities.
The count, average, sum, min, max, cardinality, percentiles, and percentile rank aggregations are shown in Figures \ref{fig:metricA_count} to \ref{fig:metricA_percenRank_}, respectively.  

\begin{figure}[H]
    \centering
    \includegraphics[width=\linewidth]{figures/metricA_count.png}
    \caption{Count: return a raw count of the elements in the selected index pattern}
    \label{fig:metricA_count}
\end{figure}

\begin{figure}[H]
    \centering
    \includegraphics[width=\linewidth]{figures/metricA_avg.png}
    \caption{Average: return the average of a numeric field}
    \label{fig:metricA_avg}
\end{figure}

\begin{figure}[H]
    \centering
    \includegraphics[width=\linewidth]{figures/metricA_sum.png}
    \caption{Sum: return the total sum of a numeric field}
    \label{fig:metricA_sum}
\end{figure}

\begin{figure}[H]
    \centering
    \includegraphics[width=\linewidth]{figures/metricA_min.png}
    \caption{Min: return the min value of a numeric field}
    \label{fig:metricA_min}
\end{figure}

\begin{figure}[H]
    \centering
    \includegraphics[width=\linewidth]{figures/metricA_max.png}
    \caption{Max: return the max value of a numeric field}
    \label{fig:metricA_max}
\end{figure}

\begin{figure}[H]
    \centering
    \includegraphics[width=\linewidth]{figures/metricA_cardinality.png}
    \caption{Cardinality: return the number of unique values of a field}
    \label{fig:metricA_cardinality}
\end{figure}

\begin{figure}[H]
    \centering
    \includegraphics[width=\linewidth]{figures/metricA_percentiles.png}
    \caption{Percentiles: divide the values in a numeric field into the specified percentile bands}
    \label{fig:Percentiles}
\end{figure}

\begin{figure}[H]
    \centering
    \includegraphics[width=\linewidth]{figures/metricA_percenRank_.png}
    \caption{Percentile Rank: returns the percentile rankings for the specified values in the numeric field}
    \label{fig:metricA_percenRank_}
\end{figure}


\paragraph{Bucket Aggregation:} Bucket aggregations build buckets of documents depending on certain criteria.
All types of these aggregations are listed below.

\begin{figure}[H]
    \centering
    \includegraphics[width=\linewidth]{figures/BucketA_Range.png}
    \caption{Range: specify ranges of values for a numeric field}
    \label{fig:BucketA_Range}
\end{figure}

\begin{figure}[H]
    \centering
    \includegraphics[width=\linewidth]{figures/BucketA_Histogram.png}
    \caption{Histogram: specify intervals for a numeric field}
    \label{fig:BucketA_Histogram}
\end{figure}

\begin{figure}[H]
    \centering
    \includegraphics[width=\linewidth]{figures/BucketA_Term.png}
    \caption{Terms: specify the top or bottom n elements of a given field to display}
    \label{fig:BucketA_Term}
\end{figure}

\begin{figure}[H]
    \centering
    \includegraphics[width=\linewidth]{figures/BucketA_Filter.png}
    \caption{Filter: specify a set of filters for the data}
    \label{fig:BucketA_Filter}
\end{figure}

\begin{figure}[H]
    \centering
    \includegraphics[width=\linewidth]{figures/BucketA_IPv4Range.png}
    \caption{IPv4 Range: specify ranges of IPv4 addresses}
    \label{fig:IPv4 Range: specify ranges of IPv4 addresses}
\end{figure}


\subsection{Deployment}
We have made careful and detailed documentation in our \href{https://github.com/BourneXu/CS5604InfoStorageFEK}{repository} for the project that explains how to deploy our website both locally and also remotely using the Kubernetes \& Docker platforms.  


\subsection{Requests Sample}
The Code \ref{code:searchreq} is the request demo for searching.
The request is sent to Elasticsearch (for retrieving searching results) and the back-end (for persistence of searching logs) separately.
The back-end will receive the jsonfy data through ``/emitlogs" and then send a re-formatted JSON (Code \ref{code:searchlog}) with the es.index() function.
\begin{listing}[H]
\begin{minted}[
frame=lines,
framesep=2mm,
baselinestretch=1.2,
bgcolor=black,
fontsize=\footnotesize,
linenos
]
{json}
POST /index/_msearch?
{
   "preference":"List"
}{
   "query":{
      "bool":{
         "must":[
            {
               "bool":{
                  "must":[
                     {
                        "bool":{
                           "should":[
                              {
                                 "multi_match":{
                                    "query":"asd",
                                    "fields":["text_entry"],
                                    "type":"best_fields",
                                    "operator":"or",
                                    "fuzziness":0
                                 }
                              },
                              {
                                 "multi_match":{
                                    "query":"asd",
                                    "fields":["text_entry"],
                                    "type":"phrase_prefix",
                                    "operator":"or"
                                 }
                              }
                           ],
                           "minimum_should_match":"1"
                        }
                     }
                  ]
               }
            }
         ]
      }
   },
   "highlight":{
      "pre_tags":["<mark>"],
      "post_tags":["</mark>"],
      "fields":{
         "text_entry":{}
      },
      "require_field_match":false
   },
   "size":20
}
\end{minted}
\caption{Searching Request}
\label{code:searchreq}
\end{listing}

\begin{listing}[H]
\begin{minted}[
frame=lines,
framesep=2mm,
baselinestretch=1.2,
bgcolor=black,
fontsize=\footnotesize,
linenos
]
{json}
/emitlogs
{
   "status":200,
   "message": "Success",
    {
       "data":{
          "user":{
             "username": "username",
             "email": "email"
             },
            "activity":{
                "url": "http://url_Elasticsearch:9200/index/_msearch?",
                "search_text": "asd",
                "filters": {"text_entry": "asd"},
            }
        },
        "dataset": "dataset",
        "time": "time",
        "ip": "user_ip"
    }
}
\end{minted}
\caption{Searching Log}
\label{code:searchlog}
\end{listing}

\newpage
\bibliographystyle{acm}
\bibliography{sample}

\end{document}