\newpage
\section{Developer Manual}
\label{developermanual}
% a developers manual that covers:
%   how the project software was developed, including architecture, data flow, module overviews, etc.
%   how the project data was obtained and processed, with examples
%   how the project can be installed, operated, maintained, and possibly enhanced or extended
%   an inventory of all the files going into VTechWorks, describing each file and file grouping
%Note: This should be written for people in the future who might have to understand your data and your code, and be able to handle a new type of dataset, or make enhancements to the code.
\subsection{Providing Access to Ceph Persistent Storage From ETD and Tobacco VMs}
\label{ceph_tutorial}
In order to be able to transfer large processed datasets from the virtual machines, that store the ETD and tobacco datasets, into Ceph storage, a Ceph client must be installed on the VMs.
Then, Ceph is mounted onto the VMs, facilitating direct access.
The login information and secret key are examples (they’re not real).
This is a reference that explains the process.
No action is required from any team. 

\begin{itemize}
\item[1.] Using the terminal on your machine, connect to the VM (a CentOS machine); must have root access:

\texttt{ssh user@tobaccovm}
\item[2.] Enter the password: 

\texttt{Password}

The following steps are executed in the VM:
\item[3. ] Install the Ceph tools and Ceph client on the VM:

\texttt{sudo yum -y install centos-release-ceph-nautilus}

\texttt{sudo yum -y install ceph-common}

\item[4.] To have permissions to connect to and access the CS cluster, you will have to first:

Store the cluster’s secret key into a file (we named the file here: cephfs.secret)

\texttt{echo ABCDEFGHIJKLMNOPQRSTUVWXYZ > cephfs.secret}

Assign read and write permissions for cephfs.secret

\texttt{chmod 600 cephfs.secret}

\item[5.] Create a directory that we’ll use to mount to Ceph File System

\texttt{mkdir /mnt/ceph}

\item[6.] Edit /etc/fstab file (which is the OS’s file system table) to add Ceph File System info

\texttt{vi /etc/fstab}

Add the following line right under the last line:

\texttt{101.102.1.10:/courses/cs5604   /mnt/ceph                 ceph    name=cs5604,}

\texttt{secretfile=/root/cephfs.secret,\_netdev,noatime 0 0}

\item[7.] Mount all filesystems mentioned in fstab as indicated 

\texttt{mount -a}

\item[8.] To check all mounted filesystems

\texttt{df -h}

\item[9.] To check content of Ceph Storage (after adding content using Kubectl)

\texttt{ls /mnt/ceph -l}   

\item[10.] To exit the VM and go back to your machine

\texttt{exit}
\end{itemize}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{Kubectl Installation and Introduction}
\label{kubectl_tutorial}
In order for teams to be able to access the container cluster including Ceph storage (and avoiding Rancher limitations such as the 1 hour shell limit), kubectl needs to be installed on their machines.
Using Rancher (CS cloud), we have set up containers for all of the other teams to mount data to Ceph storage.
Each container is named \texttt{team-ceph}, so for the CME team, the container (and pod) is \texttt{cme-ceph}.
Mounting Ceph File System allows us to persist and share data regardless of the ephemeral life of a container.

The following steps show how to install and work with Kubectl.

Steps 1-3 are for the CME, CMT, ELS, and INT groups -- and possibly for TML.
\begin{itemize}
    \item[1.] Install Kubectl:
    
    For Linux: (Download the latest release, make the kubectl binary executable, move the binary into your PATH) 
    
    \url{https://kubernetes.io/docs/tasks/tools/install-kubectl/#install-kubectl-on-linux}
    
    For MacOS:
    
    \url{https://kubernetes.io/docs/tasks/tools/install-kubectl/#install-kubectl-on-macos}
    
    For Windows:
    
    \url{https://kubernetes.io/docs/tasks/tools/install-kubectl/#install-kubectl-on-windows}

	
    \item[2.] kubectl looks for a file named config in the \$HOME/.kube directory
    \begin{itemize}
        \item[a.] Create a directory that is named ``.kube''.
        
        \texttt{mkdir ~/.kube}
        \item[b.] Create an empty file and call it ``config''.
        
        \texttt{vi config}
        \item[c.] Go to cloud.cs.vt.edu. Under the testing cluster, click on the Kubeconfig File (see Figure~\ref{fig:pic1}).
        \begin{figure}[h]
        \centering
        \includegraphics[width=0.80\textwidth]{figures/pic1.png}
        \caption{Kubectl configuration}
        \label{fig:pic1}
        \end{figure}
        \item[d.] You’ll see the following page; click on ``Copy to Clipboard'' (see Figure~\ref{fig:pic2}).
        \begin{figure}[h]
        \centering
        \includegraphics[width=0.8\textwidth]{figures/pic2.png}
        \caption{Kubeconfig File}
        \label{fig:pic2}
        \end{figure}
        
        \item[e.] Return back to your computer’s terminal, paste what you copied into the config file we created in step (b), and then save and close the file.

    \end{itemize}
    
    \item[3.] Move the config file we just created into the .kube directory.

    \texttt{mv config ~/.kube/}
    
    \item[4.] Display the pods that are within the namespace cs5604-team-db. For example, for the CME team, the namespace is \texttt{cs5604-cme-db}.

    \texttt{kubectl get pods --namespace cs5604-cme-db} (see Figure~\ref{fig:pic3} for the output)
    \begin{figure}[h]
        \centering
        \includegraphics[width=0.65\textwidth]{figures/pic3.png}
        \caption{List of pods in the namespace \texttt{cs5604-cme-db}}
        \label{fig:pic3}
    \end{figure}
    
    \item[5.] Run a shell in the \texttt{cme-ceph-758b7ffccc-sn52n} pod to access Ceph File System.

    \texttt{kubectl exec -it -n cs5604-cme-db cme-ceph-758b7ffccc-sn52n /bin/bash}

    \item[6.] Once in the container’s shell, check disk space usage info of the container’s file systems.

    \texttt{df -h}

    You can see Ceph is mounted on \texttt{/mnt/ceph} in Figure~\ref{fig:pic4}.
    \begin{figure}[h]
        \centering
        \includegraphics[width=0.8\textwidth]{figures/pic4.png}
        \caption{List mounted volumes on the container cme-ceph-758b7ffccc-sn52n}
        \label{fig:pic4}
    \end{figure}
    
    \item[7.] To access Ceph via the mount point, change to that location.

    \texttt{cd /mnt/ceph}
    
    \item[8.] List the contents of Ceph (see Figure~\ref{fig:pic5}).

    \texttt{ls} 
    \begin{figure}[h]
        \centering
        \includegraphics[width=0.65\textwidth]{figures/pic5.png}
        \caption{Listing the contents of Ceph}
        \label{fig:pic5}
    \end{figure}
    \item[9.] You can store files in Ceph, e.g., we’ll create a file called test.txt.

    \texttt{touch test.txt}

    \item[10.] Don’t forget to exit the pod shell once you’re done.

    \texttt{exit}

\end{itemize}
The following steps are for the ElasticSearch Group:
\begin{itemize}
    \item[1.] To get the cluster IP address and External-IP for elasticsearch, list all resources under the namespace elasticsearch.

    \texttt{kubectl get all -n elasticsearch}

    \item[2.] Get a shell into the \texttt{els-ceph-dc4799796-d6wvj} pod.

    \texttt{kubectl exec -it -n elasticsearch els-ceph-dc4799796-d6wvj /bin/bash}

    \item[3.] To connect Ceph with elasticsearch (using the ClusterIP)

    \texttt{curl 10.43.38.7:9200}

    \item[4.] To check all the indices that are currently on Elasticsearch from the \texttt{els-ceph} pod (see Figure~\ref{fig:pic6})
    
    \texttt{curl 10.43.38.7:9200/\_cat/indices?v} 
    \begin{figure}[h]
        \centering
        \includegraphics[width=0.9\textwidth]{figures/pic6.png}
        \caption{List of indices that are currently on Elasticsearch from the \texttt{els-ceph} pod}
        \label{fig:pic6}
    \end{figure}
    \item[5.] To connect Elasticsearch outside the testing cluster (using the External-IP) (see Figure~\ref{fig:pic7})
    \begin{figure}[h]
        \centering
        \includegraphics[width=0.8\textwidth]{figures/pic7.png}
        \caption{Connect Elasticsearch outside the testing cluster (using the External-IP)}
        \label{fig:pic7}
    \end{figure}
    \item[6.] Exit the pod shell once you’re done.

    \texttt{exit}

\end{itemize}

%\input{container_management_manual.tex}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{Deploying Containers from Rancher Catalogs}
\label{deploying-ranchercatalogs}
In this section, we will describe how to leverage Rancher's Catalogs and Apps to deploy containers seamlessly. To demonstrate this feature, we deploy a MySQL container as an example. With that said, Rancher hosts a wide variety of other applications including Elasticsearch, Kibana, Kubeflow, etc.

Catalogs are GitHub repositories or Helm Chart repositories filled with applications that are ready-made for deployment. Applications are bundled in objects called Helm charts~\cite{catalogs}.

\begin{itemize}
    \item[1.] Login to \url{cloud.cs.vt.edu} and navigate to the namespace where the application/container has to be deployed.
    \begin{figure}[!ht]
    \centering
    \includegraphics[width=0.9\textwidth]{figures/namespace.png}
    \caption{Navigating to CS5604-INT namespace}
    \label{fig:namespace}
    \end{figure}
    
    \item[2.] Under the Apps tab click on launch.
    \begin{figure}[!ht]
    \centering
    \includegraphics[width=0.9\textwidth]{figures/launch.png}
    \caption{Launching a Rancher App}
    \label{fig:launch}
    \end{figure}
    
    \item[3.] Search for a ready-made container that you want to deploy (see Figure~\ref{fig:mysql}).
    \begin{figure}[!ht]
    \centering
    \includegraphics[width=0.9\textwidth]{figures/mysql.png}
    \caption{Launching MySQL App}
    \label{fig:mysql}
    \end{figure}
    
    \item[4.] Configure your App with required parameters like namespace, environment variables, volume, etc. (see Figure~\ref{fig:configure}).
    \begin{figure}[!ht]
    \centering
    \includegraphics[width=0.9\textwidth]{figures/configure.png}
    \caption{Configuring the App}
    \label{fig:configure}
    \end{figure}
    
    \item[5.] Execute the shell of the newly deployed App and type the command to test if it works properly or not (see Figure~\ref{fig:sqlstart}). 
    \begin{figure}[!ht]
    \centering
    \includegraphics[width=0.9\textwidth]{figures/sqlstart.png}
    \caption{Testing if MySQL App has been launched successfully}
    \label{fig:sqlstart}
    \end{figure}
    
\end{itemize}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{Deploying Containers from Docker Hub}
\label{deploying-dockerhub}
In this section, we will describe how to create a new container from Docker Hub on Rancher. To demonstrate this feature, we deploy a Python container as an example. With that said, we can find almost all of the containers we need in \url{hub.docker.com}. We can specify the version for any container.

\begin{itemize}

    \item [1.]Open \url{hub.docker.com}, search for the container that you want to deploy, confirm the full name of the container and the version tag, for example, \textit{python:latest} (see Figure~\ref{fig:dockerhub}). 
    \begin{figure}[!ht]
    \centering
    \includegraphics[width=0.9\textwidth]{figures/dockerhub.png}
    \caption{Search for container on Docker Hub}
    \label{fig:dockerhub}
    \end{figure}
    
    \item [2.]Login to \url{cloud.cs.vt.edu} and navigate to the namespace where the application/container has to be deployed (see Figure~\ref{fig:namespace}).
    
    \item [3.]Under the Workloads tab click on Deploy.
    
    \item [4.]Configure your Container with required parameters like namespace, environment variables, volume, etc. (see Figure~\ref{fig:configure}).
    It should be noted that the full name of the container and the version tag should be accurately typed into the \textbf{Docker Image} text-box (see Figure~\ref{fig:pythonimage}). Click \textbf{Launch} to finish your deployment.
    \begin{figure}[!ht]
    \centering
    \includegraphics[width=0.9\textwidth]{figures/pythonimage.png}
    \caption{Search container on Docker Hub}
    \label{fig:pythonimage}
    \end{figure}
    
    \item[5.] Execute the shell of the newly deployed App and type the command to test if it works properly or not (see Figure~\ref{fig:shell}). 
    
\end{itemize}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{Jupyter Notebooks Installation and Exposing an External IP}

In this section we will describe how to deploy a custom Docker container in the CS cloud cluster and expose it as a service for external access by using a static IP address and basic authentication.

\subsubsection*{What is Jupyter Notebook?}

The Jupyter Notebook~\cite{jupyter} is an open-source web application that allows you to create and share documents that contain live code, equations, visualizations, and narrative text.
Uses include: data cleaning and transformation, numerical simulation, statistical modeling, data visualization, machine learning, and much more. It’s the most popular tool for interactively developing and presenting data science projects.

%This tutorial is divided into three subsections:

\subsubsection*{Build and upload Docker images on Docker Hub}
\begin{itemize}
    \item[1.] Create a Dockerfile in an new empty directory. The following commands should be inserted in an empty file named Dockerfile with no file extension:
    
    \texttt{FROM python:3}\\
    \texttt{RUN pip install jupyter}\\
    \texttt{WORKDIR /mnt/}\\
    \texttt{ENTRYPOINT ["jupyter", "notebook", "--ip=0.0.0.0", "--no-browser","--allow-root"]}
   
   Note that the \texttt{ENTRYPOINT} instruction allows you to configure a container that will run as an executable.
   It is similar to the \texttt{CMD} instruction, because it also allows you to specify a command with parameters.
   The difference is that the \texttt{ENTRYPOINT} command and parameters are not ignored when a Docker container runs with command line parameters. 
    
    \item[2.] To run the following command, Docker must be installed.
    In the directory where you created the Dockerfile above, run the following:
    \begin{lstlisting}[language=bash]
    $ docker build -t jupyter-docker . 
    \end{lstlisting}
    
    \item[3.] You can run the container using the following command on your local system.
    We also need to mount the volume to store our notebooks outside the Docker directory.
    \begin{lstlisting}[language=bash]
    $ docker run -p 8888:8888 -v <path_to_your_local_directory>:
      <path_inside_docker> -it jupyter-docker
    \end{lstlisting}
    \item[4.] To publish the image to use in any deployment environment (CS cloud in our case), create an account on Docker Hub and login to it via the terminal using the command:
    \begin{lstlisting}[language=bash]
    $ docker login --username=<username>
    \end{lstlisting}
    \item[5.] Tag the newly created Docker image.
    \begin{lstlisting}[language=bash]
    $ docker tag 754a7069962a <username>/jupyter:first
    \end{lstlisting}
    Note that “754a7069962a" is the image ID. To list images use the command: \texttt{docker images}. “First” is the  tag/version.
    
    \item[6.] Push your container’s image to Docker Hub: 
    \begin{lstlisting}[language=bash]
    $ docker push <username>/jupyter:first
    \end{lstlisting}

\end{itemize}

\subsubsection*{Deploy on Kubernetes Cluster (CS Cloud)}
\begin{itemize}
    \item[1.] Login to CS cloud. Go to your namespace and click deploy. The remaining fields should be filled as shown in Figure~\ref{fig:arch}.

    \begin{figure}[ht!]
    \centering
    \includegraphics[width=0.90\textwidth]{figures/deploy.png}
    \caption{Deploying a container on Rancher}
    \label{fig:arch}
    \end{figure}
    
    Note that "rahulavt" is a Docker hub username from where the Docker image can be pulled. Add a port 8888 because the application will run on 8888.
    
    \item[2.] Attach a Ceph volume to where the Notebooks will be stored.
    
    \item[3.] Once successfully deployed, Jupyter runs on localhost with some token. But we want to expose that to the outside world with the token as a security filter.

\end{itemize}

\subsubsection*{Create and expose a service using kubectl}
\begin{itemize}
    \item[1.] In the terminal on your local system where you have installed and configured kubectl, and type \footnote{Here we are creating a service of type LoadBalancer.
    For more details follow this article:~\cite{exposeExternalIP}.}:
    \begin{lstlisting}[language=bash]
    $ kubectl expose deployment tml-jupyter 
      --type=LoadBalancer --name=jupyter-service -n cs5604-tml-db
    \end{lstlisting}
    \item[2.] Type the following command to see a list of all the services:
    \begin{lstlisting}[language=bash]
    $ kubectl get services -n cs5604-tml-db
    \end{lstlisting}
    \begin{figure}[ht]
    \centering
    \includegraphics[width=0.9\textwidth]{figures/services.png}
    \caption{List of running services for a given namespace}
    \label{arch}
    \end{figure}
    
    \item[3.] Now copy the external IP with port on your browser to open the Jupyter notebook.
    \begin{figure}[ht]
    \centering
    \includegraphics[width=0.9\textwidth]{figures/jupyter.png}
    \caption{Jupyter Notebook's Interface}
    \label{arch}
    \end{figure}
    
    \item[4.] Once being asked for access token you can get it from the logs generated by the Jupyter pod in the CS cluster.
    \begin{figure}[ht]
    \centering
    \includegraphics[width=0.9\textwidth]{figures/jupyterToken.png}
    \caption{Accessing logs of a pod in Kubernetes}
    \label{arch}
    \end{figure}
    
\end{itemize}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{Committing Changes to a New Container Image}
\label{commit-tutorial}
The purpose of this tutorial is to demonstrate how to incrementally develop Docker images in a local setting, specifically to save a running container’s current state as an image by committing a container’s modifications and changes into a new image. 

The following instructions can run on any host system with Docker installed. To push an image to Docker Hub, you must have a Docker Hub account. To login to Docker Hub via the command line interface (CLI), use the following command: 

\texttt{docker login --username=Dockerhub-USERNAME --email=yourEmail@domain.com}

Once you modify a Docker container using Docker CLI on a host machine, you can follow the steps to come.

\begin{itemize}
    \item Save modifications made in a container into a new container image. Use the ID of the container that was modified and name the new container image. 

\texttt{docker commit CONTAINER\_ID NEW-IMAGE-NAME}

The previous command will return the SHA256 hash of the image (see Figure \ref{fig:commit}).
\begin{figure}[h]
    \centering
    \includegraphics[width=0.7\textwidth]{figures/commit.png}
    \caption{The commit command}
    \label{fig:commit}
\end{figure}

To get the new image information:

\texttt{docker image ls}

See Figure~\ref{fig:ls} for a sample output. Notice the change in size in the new image compared to the original image.
\begin{figure}[h]
    \centering
    \includegraphics[width=0.8\textwidth]{figures/ls.png}
    \caption{Displaying the docker images}
    \label{fig:ls}
\end{figure}


\item To push the image to Docker Hub for fast distribution, you must tag the image as the following:

\texttt{docker tag NEW-IMAGE-NAME Dockerhub-USERNAME/NEW-IMAGE-NAME}

Push the image to the registry using the image ID.

\texttt{docker push Dockerhub-USERNAME/NEW-IMAGE-NAME}

After that, the container image will be stored in Docker Hub with the repository name ``\texttt{Dockerhub-USERNAME/NEW-IMAGE-NAME}''. The new modified image can now be pulled and deployed on any host machine by using a \texttt{docker pull} command, or deployed on Rancher.




\end{itemize}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


\subsection{Building a Docker image from a Dockerfile}
\label{dockerfile-tutorial}
In this tutorial we explain how to build an image from a Dockerfile for container development.
Running the built image will start a Docker container.
A Dockerfile is a description of the software that constitutes an image.
Dockerfiles contain a set of instructions that specify what environment to use and which commands to run.
For demonstration, we will dockerize a simple Hello World Python application that uses Flask, an HTTP server for Python applications.

To follow this tutorial, you can use a local host and should have Docker installed. 

\begin{itemize}
    \item[1.] Create a fresh directory (e.g., hello-world/). This directory defines the context of a Docker build.
    It will contain all of the things needed to build the Docker image.
    
    \item[2.] In the hello-world directory, create a file named “Dockerfile” with no extension required.
    To the Dockerfile, write what is shown in Figure~\ref{fig:dockerfile}.
    \begin{figure}[h!]
        \centering
        \includegraphics[width=0.45\textwidth]{figures/dockerfile.png}
        \caption{Dockerfile of Hello World Python application that uses Flask}
        \label{fig:dockerfile}
    \end{figure}
    
    The \texttt{FROM} instruction specifies the Base Image from which we are building.
    The \texttt{RUN} instruction specifies which additional commands to execute.
    The commands are in shell form (they will be run in a shell).
    The \texttt{CMD} specifies which command to execute when the image loads.
    There can only be one \texttt{CMD} instruction in a Dockerfile, and that can be in three different forms (explained in~\cite{build-dockerfile}).
    In this demonstration, we use the shell form.

    \item[3.] In the hello-world directory, create another file named “requirements.txt”.
    This file will contain the names of packages and dependencies needed to run the application.
    In this case, we need Flask (see Figure~\ref{fig:flask}).
    \begin{figure}[h!]
        \centering
        \includegraphics[width=0.55\textwidth]{figures/flask.png}
        \caption{Contents of requirements.txt file}
        \label{fig:flask}
    \end{figure}
    
    \item[4.] In the hello-world directory, create another file named “hello.py”.
    This file will contain the Python code (see Figure~\ref{fig:python-hello}).
    The IP address of the application is specified to be the localhost and the port is 5000.
    \begin{figure}[h!]
        \centering
        \includegraphics[width=0.45\textwidth]{figures/python-hello.png}
        \caption{Python code of the Hello World! application that uses Flask}
        \label{fig:python-hello}
    \end{figure}
    
    \item[5.] In the hello-world directory run the following command:
    
    \texttt{docker build -t flask-hello:1 .}
    
    Here flask-hello:1 is the name:tag of the Docker image that is being created.
    
    \item[6.] The application is running now.
    Browse to \url{http://localhost:5000/}, see Figure~\ref{fig:flask-hello}.
    \begin{figure}[h!]
        \centering
        \includegraphics[width=0.45\textwidth]{figures/flask-hello.png}
        \caption{Hello World!}
        \label{fig:flask-hello}
    \end{figure}

\end{itemize}

\subsection{Implementing CI/CD with Travis CI}

 In the sections that follow, we will demonstrate step-by-step how to integrate an existing project with \textit{Travis CI}~\cite{travis} and set up a simple CI/CD process.
 We will create a simple hello world Python application for the purpose of this demonstration. 
 
\begin{itemize}
\item[1.] Connect Travis CI to your project.
Most CI tools integrate seamlessly with Git services -- especially GitHub.
Thus, if you already have a repository on GitHub \cite{github}, add that to Travis CI, otherwise create a new repository. 
Your repository might look something like what is shown in Figure~\ref{fig:git_repo}.
    \begin{figure}[h!]
        \centering
        \includegraphics[width=0.8\textwidth]{figures/github_repo.png}
        \caption{A sample GitHub repository}
        \label{fig:git_repo}
    \end{figure}

\item[2.] Most CI tools expect a configuration file to exist in the project.
Travis CI expects a .travis.yml file to exist in the project root.
The .travis.yml \footnote{This has language: which is Python in this case, install: dependencies, and script: the set of unit tests to run.} file being used for demonstration would look like what we show in Figure~\ref{fig:travis.yml}. 

    \begin{figure}[h!]
        \centering
        \includegraphics[width=0.45\textwidth]{figures/travisFile.png}
        \caption{A sample .travis.yml file}
        \label{fig:travis.yml}
    \end{figure}
 \item[3.] Once the repository is ready, you do a git commit.
 The moment you push a change, Travis will trigger the build process (see Figure~\ref{fig:travis_build}).
 \begin{figure}[h!]
        \centering
        \includegraphics[width=0.9\textwidth]{figures/travis_trigger.png}
        \caption{Travis build process}
        \label{fig:travis_build}
    \end{figure}
 \item[4.] Once all test cases pass you will see a result as shown in Figure~\ref{fig:travis_result}.
 \begin{figure}[h!]
        \centering
        \includegraphics[width=0.9\textwidth]{figures/travis.png}
        \caption{Travis build results}
        \label{fig:travis_result}
    \end{figure}
 
 Testing is just one step in the CI process.
 You can add more lines to the script stage, including processes like linting, E2E tests, performance tests, and any other metric you can think of to test new code before you merge it to the main code base~\cite{travisBlog}.
 
\end{itemize}
 