\newpage
\section{Overview}
% an overview of the project effort, including how it was managed, problems/challenges faced, solutions developed, further work needed, etc.

\subsection{Project Management}
There are five members in our team, so team management is crucial to accomplish the project goal. 
Including class sessions, we schedule Zoom meetings~\cite{zoom} two times a week on average to discuss tasks and issues, and to plan ahead for future work to ensure progress. 
Because the INT team's role is to ensure an operating portable container environment by integrating all the teams' efforts, 
%as the interface between the data processing teams, the elasticsearch team and the front end developers, 
we communicate with the other teams in class sessions and on designated Slack~\cite{slack} channels (e.g., \#int-service-requests) to discuss system design requirements and resolve integration and implementation issues that hinder the overall project progress.
%\HA{Should we add links to our pages in Slack and Trello here?}
 
As mentioned above, we have adopted several tools 
%within the group 
to achieve efficient collaboration. We use Slack and Zoom for team communication, Trello~\cite{trello} for task management, GitHub~\cite{github} for code collaboration, and Docker Hub~\cite{dockerhub} for container distribution.

\subsection{Problems and Challenges}
\label{problemsandchallenges}
% problems where we needed Chris's help
We have faced a few challenges including: 
\begin{itemize}
\item[a.]
Acquiring specific container requirements from the teams at an early stage of development. 
\item[b.]
Transferring large processed datasets into Ceph File System storage. It is not simple to copy large data from VMs that store the tobacco and ETD datasets over the internet to Ceph storage; a connection between the VMs and Ceph must be established. This problem is solved. The steps to follow are in \S\ref{ceph_tutorial}.
\item[c.]
The time limitation of containers' shell on the Computer Science (CS) cloud (i.e., 1-hour timeout), making all processes running shut down after that time interval. This problem is solved. The steps to follow are in \S\ref{kubectl_tutorial}.
\item[d.]
The ephemeral execution environment provided by the containers' shell on the Computer Science (CS) cloud. This meant that all installations and executions are deleted after closing the shell. This problem is solved. The steps to follow are in \S\ref{kubectl_tutorial}.
\item[e.]
The inability to save and package a modified container as a new container image by using kubectl or Rancher (see \S\ref{literature-review} and \S\ref{Kubernetes-Rancher-kubectl}). Tools and dependencies installed after deploying the container will not become part of the container image. This is because containers are ephemeral (short lived) and Kubernetes, the underlying container orchestration for Rancher, does not allow committing changes to a container. This problem is solved. The solution is explained in~\S\ref{dockerfile-tutorial}. 

\item[f.] 
Some of the Docker container images that are created and built (e.g., via Dockerfiles~\cite{dockerfile}) are very large. This will slow down the deployment and the scaling of the services that these containers provide. A solution to this problem is under development. We propose to shrink the size of Docker images by using tools such as Docker-Slim~\cite{docker-slim} or by creating very efficient Dockerfiles. To facilitate the latter, other project teams must finalize task requirements.
This will guide us toward removing unused and unneeded packages and libraries.

\item[g.]
We have faced some challenges while deploying Kafka on our Kubernetes cluster. We investigated several approaches. We tried using Kafka's Helm charts but this approach failed because it requires Helm to be installed on Kubernetes \texttt{default} namespace for which we do not have access. We also tried using Rancher's catalogs, specifically Confluent Kafka Chart, which was deployed, however, we could not configure Kafka topics and metadata to successfully push data. Another approach was by deploying a single \textit{zookeeper} Docker image and a single \textit{broker} Docker image. However, we were unable to connect to Loadbalancer Ingress possibly because Kafka does not support IPv6. We finally tried Spotify's Single Docker packaged Kafka, however, it was only supported on Docker-swarm (Docker's built-in orchestration support).

\item[h.]
In order to set up a CI/CD pipeline on GitLab~\cite{gitlab-cicd}, we are required to deploy a GitLab \textit{Runner} instance, a service used to run jobs like tests and send the results back to GitLab. The way to do it is to deploy a GitLab Runner instance into our Kubernetes cluster by using the gitlab-runner Helm chart~\cite{gitlab-runner}. However, that option requires administrator permissions and Helm installed on Kubernetes \texttt{default} namespace. We tried integrating our Kubernetes cluster by using the kubeconfig file, but got a \texttt{403 forbidden} error that is probably due to not having access to the default namespace. 

\item[i.] After setting the Kafka broker cluster\footnote{In our case we only deployed one broker and exposed it on \texttt{testing133.cs.vt.edu:9092}.} and zookeeper containers and then creating all the topics we need for automating the data pipeline, it was challenging to decide where to exactly place Kafkacat producer and consumer commands within our system, especially since we don't have all of our system components containerized. Nevertheless, knowing where to place the Kafkacat commands, it is straightforward to send/receive data to/from Kafka topics by using the Kafkacat command in producer or consumer mode and specifying the topic and directory path to the location of the new data. 
%In order to have an automated data pipeline supporting new data ingestion into our information retrieval system, we need to run a kaf between

\item[j.] The production cluster (Teaching cluster) does not have any physical nodes, therefore, until the cluster has some nodes, no containers can be deployed.


%Will containerizing CMT processing be important in our system, since the container will/might not be used?

%Every team has a unique set of requirements regarding what should be installed in their containers. So building those docker containers, testing and managing them is challenging.
\end{itemize}

\subsection{Solutions Developed}

In order to deal with some of the challenges mentioned in \S\ref{problemsandchallenges}, we often contacted Chris Arnold, of the CS technical staff at Virginia Tech. Chris's significant efforts to help us are much appreciated. 
We were able to solve the problem of transferring large processed datasets into Ceph by installing a Ceph client on the Virtual Machines that host the tobacco and ETD datasets and mounting Ceph as a volume for direct interaction. We have created a tutorial (see \S\ref{ceph_tutorial}) that explains the above process for reference.

To address the 1-hour timeout limit on the CS cloud containers' shell, %when accessed via Rancher~\cite{Rancher},
we leveraged the kubectl tool by installing it on our local machines and configuring it to connect to the CS cloud cluster. 
The kubectl tool facilitates direct control and management of all the containers and connected services (e.g., Ceph) in the cluster. 
The detailed process of installing and configuring kubectl has been documented in \S\ref{kubectl_tutorial}. 

To solve the inability to save and package a modified container, we can utilize Docker's command line interface, specifically the \texttt{Docker commit}~\cite{docker-commit} command, to save a running container's current state as an image for portability purposes.
With that said, building a Docker container from a Dockerfile is how we will do it. Moreover, since we will be deploying multiple containers, we will be leveraging a YAML file which will describe all the containers to be deployed in our system.
Such a YAML file describes a multi-container application system, and is deployed either via \texttt{docker-compose} on hosts where Docker is installed or \texttt{Kompose} (discussed in \S\ref{migration-to-production}) on Kubernetes infrastructures.

In pursuit of a solution for a continuous integration and continuous deployment (CI/CD) pipeline, we performed a comprehensive study on which tool is most suitable for the job (refer to Table~\ref{comparison-table} in~\S\ref{section:cicd}).  We laid out a foundation that can be used as a basis for complete CI/CD integration. On GitLab there is a CS5604 project that we created where teams (analogous to those in our project) can push their work.  We also provided a \textit{Hello World} style demonstration on how to configure a CI/CD pipeline.

Part of the process of setting up our GitLab pipeline involved the challenge of deploying a GitLab \textit{Runner}. We tackled this by installing the Runner on a dedicated virtual machine instead of connecting GitLab to our Kubernetes cluster. We followed a procedure \cite{gitlab-runner-install} for installing GitLab Runner that was archived in the GitLab Docs repository.

Another aspect of our system that we pursued was a solution to ingest new documents into our collections. This ingestion system was designed to be based on Apache Kafka to serve as the underlying data pipeline (See \S\ref{kafka}). In the development process we encountered a problem of deploying a single zookeeper container and a single Kafka broker, but we were able to fix the problem by following an article~\cite{kafka-deploy} on DZone. The only change we had to make was to assign a static IPv4 address as the LoadBalancer Ingress of the Kafka-service since our Kubernetes cluster could not handle IPv6 for internal communications\footnote{
    For more details follow this article:~\cite{kafka-ipv6-challenge}}.
Finally, the communication of the Kafka-service and the Kafka \textit{broker} was established by assigning the same IPv4 as the environment variable in kafka-broker.yml.

\subsection{Potential Development}
%rename this section to "Potential Development" in the Final version of this report 
%instead of Development Needed

Building Docker~\cite{docker} containers is a big part of our team's objective. To get the teams started, we have deployed a few basic Docker containers on Rancher for development and testing purposes. The deployed containers include a base image of CentOS and another CentOS container with a Ceph Volume mounted for each team. We have equipped the teams with different containers that package different tools and libraries in the testing cluster based on an initial set of requirements the teams provided. Some teams have not finally provided a clearly established and evaluated requirements and scripts for their data processing, machine learning models, etc., for us to create customized and dedicated Docker containers for their tasks.

We wanted to support the ingestion of new data, specifically ETDs, into our information retrieval system. Therefore, we started exploring Kafka (explained in \S\ref{kafka}) and have successfully deployed Kafka and created the needed topics. However, we have not integrated Kafkacat (explained in \S\ref{changes-kafka}) producer and consumer commands into the components of our system since they are not completely containerized. With that said, we have provided a diagram connecting all the producers and consumers to all the topics in our system.

We had also planned to create a CI/CD pipeline for each of the services being deployed on our cluster. But due to time constraints and dependency on other teams to provide us with the unit tests, we were not able to integrate the CI/CD pipeline with their solutions.
Future development to fully implement CI/CD could continue using the CS5604 GitLab project that we set up.

Finally, we were expected to deploy everything on the production cluster, but we were unable to achieve that due to resource constraints and also because other teams were not ready with their production level code.
However, we did research how we can leverage \texttt{Kompose} \cite{use-kompose} to deploy multiple dependent containers together.

