\chapter{Components} 
\label{components}
\section{Docker and Containers}
For this project, we are leveraging the CS cloud Infrastructure~\cite{cloud}. Docker is an open-source platform that enables developers to build, deploy, run, update, and manage containers—standardized, executable components that combine application source code with the operating system (OS), libraries, and dependencies required to run that code in any environment (Figure~\ref{fig:container_arch}). It benefits both system administrators and software developers, which is a crucial part of DevOps toolchains. 

\begin{figure}[h!]
	\centering
	\includegraphics[scale = 0.5]{img/container_arch.png}
	\caption{Container is on top of OS \cite{containers}}
	\label{fig:container_arch}
\end{figure}

% Containers are made possible by process isolation and virtualization capabilities built into the Linux kernel. These capabilities—such as control groups (cgroups) for allocating resources among processes, and namespaces for restricting process access or visibility into other resources or areas of the system—enable multiple application components to share the resources of a single instance of the host operating system in much the same way that a hypervisor enables multiple virtual machines (VMs) to share the CPU, memory and other resources of a single hardware server. This facilitates fast and consistent application deployment regardless of the deployment environment. That has led to the widespread adoption of container technology. 

Containers are created from customized images that are stored in a container registry both privately and publicly such as Docker Hub, and the Google Container Registry~\cite{google_container_registry}. Virginia Tech also has its container registry. 

We will create, deploy and maintain containers based on the requirements of each team. To meet each team's specific requirements, we also require (and help) each team to write a Dockerfile for fast deployment. A Dockerfile is a text document that contains all of the commands a user could call on the command line to assemble an image~\cite{dockerfile_2019}. In the VT CS cloud~\cite{vt_cs_cloud}, we use a cluster called \textit{Discovery} on which we deploy all containers. 


\section{Jupyter Notebooks}
The Jupyter Notebook is an open-source web application that can be used to create and share documents that contain live code, equations, visualizations, and text. Jupyter Notebook is maintained by the people at Project Jupyter~\cite{jupyter_intro}.

Jupyter Notebooks are a spin-off project from the IPython project, which used to have an IPython Notebook project itself. IPython is a growing project, with increasing language-agnostic components (Figure~\ref{fig:ipy}).
\begin{figure}[h!]
	\centering
	\includegraphics[scale = 0.5]{img/ipy.png}
	\caption{IPython Structure \cite{ipython_internals}}
	\label{fig:ipy}
\end{figure}
The name, Jupyter, comes from the core supported programming languages that it supports: Julia, Python, and R. Jupyter ships with the IPython kernel, which allows one to write programs in Python. 

We offer each team containers with Jupyter Notebook embedded that allow them to access a remote server and modify files easily, instead of deploying and testing locally. We use a Docker image file from Docker Hub~\cite{dockerhubJupyter} to quickly spin up containers inside the CS cloud. Detailed steps will be shown in Section~\ref{ch:summary}. Jupyter Notebooks are used as computational documents by the teams.

\section{PostgreSQL}
We use PostgreSQL (Postgres)~\cite{postgresql} as our database for the services available as well as holding the conﬁguration and data for Airﬂow. As is described in Section ~\ref{subsection:postgres_lr}, PostgreSQL is an advanced, enterprise-class, and open-source relational database system that supports both SQL (relational) and JSON (non-relational) querying. Postgres acts as the central database for ETD profiling. It holds the metadata for ETDs such as title, date issued, authors, types, and URLs uploaded by the curator team. The experimental teams are using APIs to fetch the metadata and analysis, segment, train models, and save profiled results back to Postgres. Teams will also create their databases to ensure that each service only has access to the appropriate database. 

Same as Jupyter Notebook, Postgres is set up on a container using the official Docker image~\cite{postgres_dh}. To access Postgres services, we forward the hostname and port of Postgres, which can be accessed by other Jupyter Notebook containers within the same cluster.

\section{Ceph}
Ceph is an open-source software-defined storage system that acts as an object, block, and file storage in one unified system. It is a reliable and scalable architecture that decouples data from physical storage hardware. This is done by introducing software abstraction layers which then help with fault management. It provides interfaces for multiple storage types, thus eliminating the need for multiple storage solutions. Hence, Ceph's best applications are for cloud, Openstack, Kubernetes, and other microservices and container-based workloads, to manage large data storage.

For this project, we are using Ceph as our storage solution for managing code repositories of the various teams. These repositories consist of programs that run the various services that are the crux of this project, hence their storage on Ceph is crucial for the efficient running of the system on the cloud.

\section{Camelot}
Camelot provides the functionality to manage multiple isolated collections of data within a single running Camelot. There are situations where you want to keep data isolated from each other, for example, as a contractor using Camelot to conduct research for several clients. For this, Camelot offers datasets where each dataset can be thought of as a separate Camelot
%database. 
data environment.
Each dataset, as per their documentation \cite{camelot}, is an ``isolated collection of data" within a single running Camelot.

For our project, Camelot is being used as the main storage system for the Electronic Theses and Dissertations.

\section{Kubernetes}
In this project, we use Kubernetes (K8s) which is embedded in~\url{cloud.cs.vt.edu} to help us manage and orchestrate Docker containers. While we offer each team several containers to build their services, K8s comes to play its role: (1) whenever there is a need, to scale one service up/down, and (2) to monitor and manage the health of the containers (abstracted as pods in K8s). 

% Kubernetes~\cite{kubernetes} is also known as K8s and is an open-source container orchestration system for automating software deployment, scaling, and management of containerized applications. A Kubernetes cluster is a set of nodes that run the containerized applications thus acting like a lightweight and flexible virtual machine. K8s containerizes applications packages of an app with its dependencies and necessary services. 

% Containers can run across multiple machines and environments: virtual, physical, cloud-based, and on-premises within Kubernetes clusters. Clusters consist of one master node and worker nodes, all of which can either be physical computers or virtual machines. The master node controls the state of the cluster, manages the task assignment, schedules, and scales applications. Worker nodes are those that run the applications and perform the tasks assigned by the master node.

% A Kubernetes cluster has the following main components -
% 1. API server - exposes REST interface to all Kubernetes resources
% 2. Scheduler - container placement based on requirements
% 3. Controller manager - runs controller processes and manages cluster's actual state
% 4. Kubelet - ensures containers are running in a Pod by interacting with the Docker engine, run by worker nodes
% 5. Kube-proxy - manages network connectivity, run by worker nodes
% 6. Etcd - stores all cluster data

\section{Rancher}
Rancher~\cite{rancher} is an open-source multi-cluster orchestration platform that addresses the operational and security challenges of managing multiple Kubernetes clusters across any infrastructure. It deploys Kubernetes clusters and also unites them with central authentication and access control. In simple terms, it helps manage Kubernetes clusters and provides a useful UI for monitoring and managing.

The major features that make Rancher incredibly useful are as follows.
\begin{enumerate}
    \item \textbf{Authentication and Authorization} - Teams need user management to ensure that administrative access is restricted to those qualified. This makes sure that developers don't perform operations outside of their scope, for example deleting a worker or modifying a cluster.
    \item \textbf{Security Compliance} - Kubernetes clusters in production need to ensure that the right form of security is provided. Rancher provides a unique feature that scans the deployed clusters and runs tests for assessing if they meet the benchmarks.
\end{enumerate}

\section{GitLab and CI/CD}
As discussed in Section~\ref{section:gitlab_lr}, GitLab is an open-source code repository and collaborative software development platform which provides free open and private repositories. It provides Continous Integration and Continuous Deployment capabilities which automate the build, integration, and verification of the code.

For this project, we are using GitLab as our primary Version Control System to ensure a faster, more reliable, and more collaborative software development process.