\section{Design}
%This section should explain the approach, tools, methodology, conceptual background, and deliverable

The class project development and production phases will leverage the CS cloud infrastructure that is running Rancher which is based on the Kubernetes container management platform. Since all the class teams will be initially working on the \textit{testing} cluster on the CS cloud, we have created projects, one per team, to provide a level of organization in the \emph{testing} cluster. The six projects are CS5604-CME, CS5604-CMT, CS5604-ELS, CS5604-FEK, CS5604-INT, and CS5604-TML. Under each project, we have deployed (1) a basic CentOS container and (2) another CentOS container that is connected to Ceph storage by mounting an ephemeral volume (Ceph File System) to the CentOS container. In order to be able to do that, a Ceph storage server must be running. The CS Technical Staff has already set up the Ceph storage system and it is available as a Kubernetes persistent storage, i.e., part of the CS Cloud cluster.


\subsection{Docker Containers}
Containers are a lightweight alternative to virtual machines. They are an operating-system-level virtualization whereby the operating system isolates the application and limits its resources (see Figure~\ref{fig:container}). In contrast to hardware-level virtualization, containers ensure fast deployment with low performance overhead.

Containers enable users to run software applications in a reliable and portable way by encapsulating an entire execution environment (e.g., application, dependencies, libraries, and needed configuration files) into one package. 
This facilitates fast and consistent application deployment regardless of the deployment environment. That has led to the widespread adoption of container technology~\cite{everything}. 
Containers' impact on data-center efficiency is significant because of the resulting reduction in operational costs due to higher consolidation density and power minimization.

\begin{figure}[h]
    \centering
    \includegraphics[width=0.55\textwidth]{figures/containerVM.png}
    \caption{The difference between a container and virtual machine}
    \label{fig:container}
\end{figure}

Containers leverage Linux kernel features\footnote{Docker can run atop other operating systems by leveraging a Linux-based hypervisor.} -- control groups (i.e., cgroups) and namespaces -- to achieve the desired isolation and portability features~\cite{LinuxContainers}.
Docker is currently the world's leading container management platform, followed by CoreOS rkt, Mesos, and LXC~\cite{report}.
Docker simplifies the containerization and distribution of applications by providing a tool for packing, shipping, and running applications.

Containers are created from images that are stored in an online store called a container registry such as Docker Hub~\cite{dockerhub}, Quay~\cite{quay}, and Google Container Registry~\cite{GoogleContainerRegistry}.
Users access container registries by pulling and pushing images using a Docker engine on end hosts that process user commands through a daemon process.

The INT team is responsible for creating and deploying containers according to other teams' requirements. However, because of the portable nature of containers, all of the teams have the option to deploy Docker containers on their own machines and refine them according to their project needs. The simplicity of Rancher and kubectl allows us to work expediently with containers in the CS cloud cluster. From there all the team containers can be integrated into one operating container cluster.

\subsection{Kubernetes, Rancher, kubectl, and Containers}
\label{Kubernetes-Rancher-kubectl}
Large-scale applications often involve many containers that are associated in clusters.
Kubernetes (K8s) is an open-source system that automates the deployment, scaling, and management of container clusters. It simplifies the management of container clusters by combining the containers that form an application into one unit~\cite{kubernetes}. 
%Kubernetes has the following features that make our projects more deliverable: Service discovery and load balancing, Automatic bin packing, Storage orchestration, Self-healing, Automated rollouts and rollbacks, Secret and configuration management, Batch execution and Horizontal scaling.

%\subsection{Rancher}
Rancher is an open-source platform that provides services on top of Kubernetes~\cite{Rancher}. In addition to providing Kubernetes-as-a-Service, Rancher has the following capabilities:
\begin{itemize}
    \item
    Rancher provides a centralized management of clusters and the containers running in them. Since every team has its own project and namespace, operational and security challenges can be managed efficiently.
    
    \item
    Rancher is resource-agnostic; thus we can bring up clusters from different providers and migrate resources~\cite{RancherOverview}. Rancher unifies the individual deployments as a single Kubernetes Cloud.
    
    \item
    Rancher has easy authentication policies for users and groups. Admins can enable self-service by delegating the administration of Kubernetes clusters or projects and namespaces directly to individual users or groups~\cite{RancherWithKubernetes}. This is especially useful since it allows us to control memberships, ownerships, permissions, and capabilities of the six different team projects.
\end{itemize}

Kubectl~\cite{kubectl} is the Kubernetes command-line tool to control the Kubernetes cluster manager. Kubectl can be used to deploy and manage pods (made up of a container, or several closely related containers) in the Kubernetes cluster.

Figure~\ref{fig:kube-docker-rancher} shows the relationship between Kubernetes, Rancher, kubectl, Kubernetes pods, Docker containers, and Kubernetes-managed physical nodes.
Rancher generally provides a user interface and API for users to interface with the Kubernetes cluster.
Kubectl on the other hand provides a command line interface to manage and work with the Kubernetes cluster.
Kubernetes manages nodes (i.e., worker machines).
A node may be a VM or a physical machine.
Each node contains the services necessary to run Pods including the container runtime (i.e., Docker), kubelet, and kube-proxy~\cite{kube-node}.
Pods are the smallest deployment unit in Kubernetes which host the application instance.
Each Pod represents a single container or multiple tightly coupled containers that share resources.
The Pod is an environment in which containers run; it exists until the container is terminated and the pod is deleted~\cite{kube-pod}.


 
 \begin{figure}[h!]
        \centering
        \includegraphics[width=0.75\textwidth]{figures/Kubernetes-Rancher-kubectl-Docker.png}
        \caption{Container administration hierarchy}
        \label{fig:kube-docker-rancher}
\end{figure}

\subsection{Ceph}
The storage system used in our project is Ceph~\cite{weil2006ceph,ceph}.
Ceph basically is a shared storage system that is mounted on different components of our storage and retrieval system including containers and Tobacco and ETDs VMs.
Ceph is a distributed \textit{object}, \textit{block}, and \textit{file system} storage that is widely used for its scalability, flexibility, and reliability.
Ceph is supported by many cloud computing vendors and is widely used in modern data centers.

\subsection{System Architecture}
Figure~\ref{figure:arch} shows the structure and the components of the information system that is designed in this project.

\begin{figure}[h!]
\includegraphics[width=15cm]{figures/System-diagram.png}
\caption{System diagram for information retrieval and analysis system}
\label{figure:arch}
\end{figure}

The main metadata of the current system are for ETDs and Tobacco Settlement Documents, which are sourced respectively from Universities (Virginia Tech, for now) and the UCSF Deposition Documents.

Both of the sets of documents are pre-processed to extract useful information like text data and metadata.
Kibana and Elasticsearch, in conjunction, provide the framework for data analysis.
Furthermore, the system works on analyzing user logs to provide efficient recommendation. 


\subsection{Data Ingestion}
\label{kafka}
Because both data collections in our system, ETDs and Tobacco Settlement Documents, are continually expanding, there is a need to handle the ingestion of new documents.
There are both front-end and back-end components to implementing the ingestion system.  The discussion that follows describes the ideal system that we pursued to implement.

The front-end component is to be a web interface created by the FEK team that will be specific to administrators for each collection. This user interface ideally will support two options. The first option facilitates uploading a single document for a specified collection to be queued for ingestion. The second option allows specifying a source for a batch of documents to be queued for ingestion. The queued documents will be processed by the back-end component of the ingestion system.

The back-end of the ingestion system centers around the use of Apache Kafka~\cite{kreps2011kafka}. 
% Intro about Apache Kafka ... producers, consumers, broker, topics, ZooKeeper, Kafka, and Schema Registry..
Apache Kafka is a distributed publish-subscribe messaging system. It is a fast, scalable, design-integrated, distributed, partitioned, and replicated commit log service~\cite{Kafka-doc}.

Apache Kafka's architecture includes the following components:

\begin{itemize}
    \item[1.] Topic: A specific type or category of message flow; %The message is the payload of the byte, and the topic is the class name or seed name of the message; 

    \item[2.] Producer: Any object that can post/publish messages to any topic;

    \item[3.] Broker: To buffer unprocessed messages and to decouple data processing from data producers and consumers, published messages are stored in a Kafka cluster, which is a group of servers that are called brokers;

    \item[4.] Consumer: It can pull data from/subscribe to one or more topics, that are maintained by the Broker, to consume published messages.
\end{itemize}

The back-end of our ingestion system will be configured such that the front-end will feed or link the source of the provided documents to Kafka as the producer for queuing into \textit{Raw Tobacco/ETDs} topics and/or have Kafka connectors send new records to these topics once added to the databases that host raw ETD and tobacco data as shown in Figure~\ref{fig:kafka-topics}.
The CME and CMT data processing containers will be configured as consumers for the queued documents for their respective data collections.

The CME and CMT data processing containers will also serve as a second layer of producers for a second level of queuing of newly processed documents for indexing into Elasticsearch.
The processed documents are stored in \textit{Processed Tobacco/ETDs} topics.
The Elasticsearch container will be configured as a consumer of those topics.
The TML data clustering container will consume raw data for clustering.
The TML clustering container then becomes a producer and sends the clustering results to Kafka's \textit{Clustering IDs} topic to be consumed by Elasticsearch.
TML also performs text summarization on the tobacco documents only as ETDs already have an abstract that summarizes the full text.
TML sends summarization results into the \textit{summaries} topic to be consumed by ELS.
Finally, as was planned if time permits, the FEK containers will send logs such as user logs to the \textit{Logs} topic to be consumed by TML to present recommendations. 

For the case that a user specifies a very large queue of documents to be processed, Kafka will be able to scale and throttle both levels of queue processing such that system resources are not adversely impacted.

\begin{figure}[h!]
\centering
\includegraphics[width=0.9\textwidth]{figures/kafka-topics.png}
\caption{Kafka topics (blue) and producers and consumers}
\label{fig:kafka-topics}
\end{figure}

Using Kafka for queuing of documents also allows for scalability with respect to adding new collections to our system.
Ultimately, as the number of collections grows, indexing of documents into Elasticsearch would be the bottleneck of the ingestion system.
Queuing the documents through Kafka allows for the most efficient means of prioritizing and scheduling how new documents from all collections are to be ingested.
Consumers consume produced topics in real time.

\subsubsection{Changes to System Architecture After Including Apache Kafka}
\label{changes-kafka}

Apache Kafka glues together our system's microservices.
It serves as a changelog when processing data in real-time and at scale. 
Kafka can automate the process of adding new documents by monitoring a timestamp column in the source table to identify new and modified rows to facilitate real-time processing.
To connect data sources with Kafka, we will integrate the ETD data VM as a source using a MongoDB Connector~\cite{mongodb-kafka}, and integrate the tobacco data VM as a source using Java Database Connectivity (JDBC) connector~\cite{jdbc-kafka} to import (new) data from MongoDB and MySQL, respectively, into Apache Kafka topics. 

Kafka (and ZooKeeper~\cite{zookeeper} which maintains the Kafka cluster nodes and keeps track of Kafka topics, etc.) must be configured to store data on Ceph for persistence as Kafka and Zookeeper containers are ephemeral~\cite{kafka-docker}.

At every point in the containerized system, data producers must send the data to the respective topics, and consumers must subscribe to the respective topic as well.
Kafkacat~\cite{kafkacat} is an open-source command line tool that enables such interaction with Kafka topics.
It is used to produce, consume, and list topic and partition information for Kafka. 

\subsubsection{Monitoring new additions to directory using inotify-tools}

The ELS team explored an alternative --- a simpler way of automatically ingesting new documents by continuously monitoring the input directory via inotify-tools~\cite{love2005kernel,inotify-tools}.

inotify-tools is a C library and a set of command-line programs for Linux providing a simple interface to inotify~\cite{inotify}.
It can be used to monitor and act upon file system events.

A Python script continuously listens to inotify-tools, which is configured to output the name of the new document been uploaded into the monitored directory.
This way new documents can be referenced and ingested into the Elasticsearch pipeline once produced. 

\subsection{Continuous Integration \&  Continuous Deployment (CI/CD)}
\label{section:cicd}
%\todo{Add Gitlab into the options and discuss benefits}
Continuous Integration and Continuous Deployment (\textit{CI/CD}) lays out practices to follow in order for \textit{unpushed} but committed code to more quickly be integrated while maximizing safety to result in the best experience for the end users.
It is a software development methodology which allows developers to push code multiple times a day instead of waiting till the end of a typical release cycle.

The advantage of this method is that users get small features released very quickly.
From the perspective of developers, it is the management of small lines of code changes which helps minimize the instability of the production environment.

The need for \textit{CI} in \textit{CI/CD} can be better explained with the view that a developer can push all the changes to their code in the same place, with all the same processes being run on it.
This makes the output code to be more coherent with the system. 

Once all the code and the changes are in the same place, the same processes will run on the code repository. These \textit{processes} could be: 

\begin{itemize}
\item
Run automatic code quality scans on it and generate a report of how well your latest changes adhere to good coding practices.
\item
Build the code and run any automated tests that you might have written to make sure your changes did not break any functionality.
\item
Generate and publish a test coverage report to get an idea of how thorough your automated tests are.
\end{itemize}

The process could be explained diagrammatically as shown in Figure~\ref{fig:ci}.

\begin{figure}[h!]
\centering
\includegraphics[width=0.85\textwidth]{figures/CI.png}
\caption{Continuous Integration \& Continuous Deployment}
\label{fig:ci}
\end{figure}

For the \textit{CI/CD} purposes of the project, we have explored three tools: Jenkins~\cite{jenkins}, Travis CI~\cite{travis}, and GitLab~\cite{gitlab}.
Table~\ref{comparison-table} provides a comparison between them.

\begin{center}
\begin{longtable}{|p{.185\textwidth}|p{.23\textwidth}|p{.23\textwidth}|p{.26\textwidth}|}
\caption{Comparison of CI tools} \label{comparison-table} \\

\hline \multicolumn{1}{|c|}{\textbf{Feature }} & \multicolumn{1}{c|}{\textbf{Jenkins}} & \multicolumn{1}{c|}{\textbf{Travis CI}} & \multicolumn{1}{c|}{\textbf{GitLab}} \\ \hline 
\endfirsthead

\multicolumn{3}{c}%
{{\tablename\ \thetable{} -- continued from previous page}} \\
\hline \multicolumn{1}{|c|}{\textbf{Feature }} & \multicolumn{1}{c|}{\textbf{Jenkins}} & \multicolumn{1}{c|}{\textbf{Travis}} & \multicolumn{1}{c|}{\textbf{GitLab}}\\ \hline 
\endhead

\hline \multicolumn{4}{|r|}{{Continued on next page}} \\ \hline
\endfoot

\hline \hline
\endlastfoot

\textbf{Ease of Setup}
&
Jenkins needs elaborate setup. Long wait time is required for the complete installation. &
Setting up is as easy as creating a config file. & GitLab provides easy setup with a \textit{yaml} file.\\
\hline
\textbf{Hosted service} &
No &
Yes &
Yes\\
\hline
\textbf{Performance}&
Has unlimited customization options. &
Best choice for open-source project because of ease of use and setup. &
Supports public and private repositories along with a lot of other options like supporting container registry. \\
\hline 
\textbf{Tool Type} & 
Open-source & 
Commercial &
Open-Source\\
\hline
\textbf{Usage}& 
Free& 
Free for Open Source Project and paid for Enterprise  &
Free for public projects and offers different kinds of membership to organizations such as educational institutions.
GitLab is thus free to use for this project.\\
\hline
\textbf{Server Machine} &
Server-based & 
Cloud-based &
Cloud-based\\
\hline
\textbf{Support for cloud services} &
Yes & 
Yes &
Yes\\
\hline
\textbf{Notification services} &
Yes & 
Yes &
Yes\\
\hline

\end{longtable}
\end{center}

\subsubsection{Introduction to GitLab CI/CD}
We initially started working with Travis CI because of the ease of use, but then due to its large number of limitations we shifted to GitLab. 

GitLab CI/CD is a powerful tool built into GitLab that allows you to apply all the continuous methods (Continuous Integration, Delivery, and Deployment) to your software with no third-party application or integration needed.

To use GitLab CI/CD, all you need is an application codebase hosted in a Git repository, and for your build, test, and deployment scripts to be specified in a file called .gitlab-ci.yml, located in the root path of your repository.

In this file, you can define the scripts you want to run, define include and cache dependencies, choose commands you want to run in sequence and those you want to run in parallel, define where you want to deploy your app, and specify whether you will want to run the scripts automatically or trigger any of them manually.

\subsubsection{Design of our GitLab CI/CD pipeline}

Once you’ve added your \texttt{.gitlab-ci.yml} configuration file to your repository, GitLab will detect it and run your scripts with the tool called GitLab Runner, which works similarly to your terminal.

We installed our GitLab runner on a dedicated VM (cicd.vt.edu) using the official documentation~\cite{gitlab-runner-docs}. Whenever there is a commit on any repository that contains a \texttt{.gitlab-ci.yml} file located in the root path of the repository, it triggers a CI/CD pipeline on the GitLab runner.
The Runner then performs the sequence of operations as listed in the configuration file.
A basic example would be to create an environment to run the test suite and once all the tests passes, build and deploy the application according to the specified set of instructions.

Figure~\ref{fig:gitlab-pipeline} shows a workflow diagram of the system CI/CD pipeline.

\begin{figure}[h]
    \centering
    \includegraphics[width=0.95\textwidth]{figures/gitlab-pipeline.png}
    \caption{GitLab CI/CD pipeline}
    \label{fig:gitlab-pipeline}
\end{figure}
