\chapter{Literature Review} 
    \label{LR}
    \section{Cloud-based Service Provisioning}
        Cloud-based computation resources are offered in a vertical model, with each layer in the model operating at a different level of abstraction ~\cite{inproceedings}. The layers in this model are: 
        
        \begin{enumerate}
            \item Infrastructure as a Service (IaaS)
            \item Platform as a Service (PaaS)
            \item Container as a Service (CaaS)
            \item Software as a Service (SaaS)
            \item Function as a Service (FaaS)
        \end{enumerate}
        
        We decided to choose CaaS as our level of abstraction. This involves spinning up the containers and their orchestration. We chose Docker and Kubernetes for this job, respectively (discussed more in Section \ref{components}).
        
        \subsection{Selecting the Level of Abstraction}
        We have chosen ``Container as a Service" as our desired level of abstraction.
        The reasoning behind this is as follows. Our requirements, as discussed in Section~\ref{ch:require }, have us supporting other teams in the way of offering a platform for their applications. We must give them autonomy in choosing their technological stack, while at the same time minimizing the amount of effort across all teams. Unfortunately, IaaS and PaaS, despite their flexibility, usually entail more work on the part of the team in the form of initial set-up. For example, with IaaS, each team would get a compute node hosted on the cloud, but would then have to set up networking, storage, security, and other aspects associated with the compute node. On the other hand, using SaaS or FaaS would limit the flexibility in terms of choosing the technological stack which the teams might want to work with. For these reasons, we conclude that CaaS is situated in a sweet spot of the trade-off range between flexibility and effort.
        
        \subsection{Containers as a Service}
        Containers-as-a-Service (CaaS) is a cloud service that helps manage and deploy apps using container-based abstraction. CaaS can be deployed on-premises or in a cloud~\cite{redhat_caas}. CaaS is especially useful to developers in building containerized apps that are more secure and also scalable. Users can buy only the resources they want (scheduling capabilities, load balancing, etc.), saving money and increasing efficiency. Containers create consistent environments to rapidly develop and deliver cloud-native applications that can run anywhere. 
        
        Compared to IaaS platforms that use virtual machines or bare metal hardware as fundamental resources, CaaS platforms package up applications and all their dependencies into containers more lightweight than virtual machines. For this reason, it is possible to host more containers on a single host than full-fledged virtual machines~\cite{caas_iaas}. CaaS is a suitable platform for developers who want more control over container orchestration. Using CaaS allows developers to deploy applications on containers without worrying about the limitations on container orchestration provided by the typical Platform-as-a-Service (PaaS).
        % In the range of cloud computing services, CaaS is considered a kind of subset of Infrastructure-as-a-Service (IaaS) and is found between IaaS and Platform-as-a-Service (PaaS).
        
        The basic resources of CaaS are containers, which are a popular deployment mechanism for cloud-native apps and microservices. CaaS also increases portability between environments, whether hybrid or multi-cloud.
        
        Using containers has many benefits:
        \begin{itemize}
            \item \textbf{Efficiency}: Containers require fewer resources than virtual machines (VMs) since they don’t need a separate operating system. One can run several containers on a single server. They require less bare-metal hardware, which means lower costs.
            \item \textbf{Portability}: Apps developed in containers have everything they need to run and can be deployed in multiple environments, including private and public clouds. Portability means flexibility because one can more easily move workloads between environments and providers. 
            \item \textbf{Scalability}: Containers can scale horizontally, meaning a user can multiply identical containers within the same cluster to expand when needed. 
            \item \textbf{Increased security}: Containers are isolated from each other, which means if one container is compromised, others won’t be affected. A container allows incorporating specific security requirements and practices into golden images to maintain the security posture even in different cloud environments and different footprints.
            % \item \textbf{Speed}: Because of their autonomy from the operating system, starting and stopping a container is a matter of seconds. This also allows for faster development and operational speed, as well as a faster, smoother user experience.
        \end{itemize}
        
        \subsection{Kubernetes}
        Kubernetes is regarded as one of the most trending container orchestration services~\cite{7922500} and is used by the CNCF (Cloud Native Computing Foundation) as their orchestration technology~\cite{k8s_cncf}. Kubernetes can give the orchestration container management capabilities required to deploy containers at scale across multiple server hosts with multiple layers of security while managing the health of those containers over time.  In Kubernetes, a node is a worker machine; it may be a virtual machine or a physical machine, depending on the cluster. Each node contains the services necessary to run pods and is managed by the master components of Kubernetes. 
        
        In this project, we use \url{cloud.cs.vt.edu} which offers us a Discovery cluster to build the ETD project. We offer each team several containers to build their services, and we use Kubernetes for orchestration.
        
        % \section{Elasticsearch}
        % Elasticsearch is a distributed search and analytics engine built on Apache Lucene~\cite{elastic_aws}. Since its release in 2010, Elasticsearch has quickly become the most popular search engine and is commonly used for log analytics, full-text search, security intelligence, business analytics, and operational intelligence use cases. Elasticsearch offers simple REST-based APIs, and a simple HTTP interface, and uses schema-free JSON documents, making it easy to get started and quickly build applications for a variety of use cases. It allows the processing of large volumes of data in parallel, quickly finding the best matches for specific queries. That data can be things like numbers, strings, and dates. Each document has a unique ID and a given data type, which describes what kind of entity the document is. For example, a document can represent an encyclopedia article or log entries from a web server~\cite{elastic_abueg}. 
    
    \section{Database Technologies}
    \subsection{Criteria for Choice of Database Management System}
    The primary user of the database is Team-1, as they're responsible for the central persistence of the results of all other teams. As per their requirements, we required a database management system that is: 
    
    \begin{enumerate}
        \item Relational in nature 
        \item ACID compliant~\cite{acid_compliance}
        \item Well-tested, mature, and with community support
        \item Scalable (as we are going to be storing a considerable amount of ETD data
        %relating to ~30K ETDs
        )
    \end{enumerate}
        
    Some of the candidates that were considered are PostgreSQL~\cite{postgresql}, MySQL~\cite{mysql}, MS SQL~\cite{mssql}, and Oracle DB~\cite{oracle_db}, out of which we decided to go with PostgreSQL for the reasons given in Section~\ref{subsection:postgres_lr}.
    
    \subsection{PostgreSQL}
    \label{subsection:postgres_lr}
        PostgreSQL is a free and open-source object-relational database management system that is used as one of the chief databases across various web and mobile applications~\cite{postgres_market_share}. It is well known for its reliability, stability, and performance. The tool, along with providing support for a variety of advanced data types, also provides well-optimized performance on par with other DBMS tools like MS SQL and Oracle SQL. 
        
        There are multiple benefits to using PostgreSQL:
        
        \begin{itemize}
            \item \textbf{Open source}: PostgreSQL is a completely free and open source tool, which provides the freedom and flexibility to modify and implement as per the particular application requirements.
            \item \textbf{Reliable}: PostgreSQL is ACID (Atomicity, Consistency, Isolation, and Durability) compliant, thereby ensuring that all transactions are atomic and the data consistency is retained in case of failures by rolling back all the changes made. Further, due to massive contributions to its maintenance from various companies and individuals across the world, any bugs found are resolved immediately.
            \item \textbf{Scalable}: PostgreSQL scales well vertically and performs better as more computing resources are added.
        \end{itemize}
        
        % \subsection{PgAdmin}
        % PgAdmin is a web-based Graphical User Interface (GUI) management application used to communicate with PostgreSQL and derivative relational databases on both local and remote servers. It is a very popular open-source platform fully dedicated to PostgreSQL and has graphical user interface administration tools to manage your relational databases. Some features include a query tool for SQL statements and importing/exporting CSV files~\cite{pgadmin_1}. An example of query results in PgAdmin is shown in Figure~\ref{fig:query_tool}.
        % \begin{figure}[h!]
        % 	\centering
        % 	\includegraphics[scale = 1]{img/query_tool.png}
        % 	\caption{Example of PgAdmin Query Tool}
        % 	\label{fig:query_tool}
        % \end{figure}
        
        In this project, we relieve the query workload of both end-users and developer-users by allowing them to access and manage the PostgreSQL database on a web browser, instead of writing tedious SQL statements. 
    
    \section{Workflow Automation}
    
    We aim to achieve workflow automation to help other teams automate various tasks as services and workflows, to increase the overall productivity of the group. 
    
    \subsection{Workflows as Directed Acyclic Graphs}
        A directed acyclic graph (DAG), is a finite directed graph with no directed cycles. That is, it consists of finitely many vertices and edges, with each edge directed from one vertex to another, such that there is no way to start at any vertex `v' and follow a consistently-directed sequence of edges that eventually loops back to `v' again. Equivalently, a DAG is a directed graph that has a topological ordering, a sequence of the vertices such that every edge is directed from earlier to later in the sequence~\cite{dag_wiki}. We show an example in Figure~\ref{fig:dag_sample}, which is one of our workflow automation designs.
        
        \begin{figure}[h!]
        	\centering
        	\includegraphics[scale = 0.5]{img/workflow_DAG.png}
        	\caption{A DAG Sample}
        	\label{fig:dag_sample}
        \end{figure}
        
        A workflow is a series of steps taken to achieve a goal from a set of inputs. Workflows usually have an end goal, for example, creating visualizations for sales numbers of the last day.  Thus, workflows can be imagined as DAGs, where each node is an executable step. For example, to create your visualization from the past day’s sales, the following steps may need to be performed:
        
        \begin{enumerate}
            \item Read data from a relational database. 
            \item Transform the data by clipping irrelevant columns or attributes. 
            \item Load this data into a data warehouse tool.
            \item Trigger the fetching of this data from the visualization tool.
        \end{enumerate}

        Thus, if we were to represent this as a DAG, it would look as depicted in Figure~\ref{fig:example_dag}.
        
        \begin{figure}[h!]
        	\centering
        	\includegraphics[scale = 0.4]{img/example_dag.png}
        	\caption{A workflow represented as a DAG}
        	\label{fig:example_dag}
        \end{figure}
        
        \subsection{Apache Airflow as a Workflow Automation Tool}
        
        Apache Airflow~\cite{apache_airflow_about} is a platform that lets you build and run workflows. A workflow is represented as a DAG, and contains individual pieces of work called Tasks, arranged with dependencies and data flows taken into account ~\cite{airflow_official}.
        
        An Airflow installation generally consists of the following components:
        \begin{enumerate}
            \item A scheduler, which handles both triggering scheduled workflows, and submitting Tasks to the executor to run.
            \item An executor, which handles running tasks. In the default Airflow installation, this runs everything inside the scheduler, but most production-suitable executors push task execution out to workers.
            \item A web server, which presents a handy user interface to inspect, trigger, and debug the behavior of DAGs and tasks.
            \item A folder of DAG files, read by the scheduler and executor (and any workers the executor has).
            \item A metadata database, used by the scheduler, executor, and web server to store state.
        \end{enumerate}
        
        In Airflow, an operator encapsulates the operation to be performed in each task in a DAG. Airflow has a wide range of built-in operators that can perform specific tasks, some of which are platform-specific. For example, a `BashOperator' executes a command inside a Bash shell on the machine that is running Airflow. Additionally, it is possible to create customized operators. The dependencies of operators could be defined as follows: first\_task >> [second\_task, third\_task]. To execute a DAG, users can either operate through CLI or a DAG user interface.
        
        % Apache airflow can be used in scenarios where there is a requirement for backup from DevOps tasks and storing the results into a Hadoop cluster after the execution of a Spark job. Airflow can be used for training the machine learning models, and also triggering jobs like a SageMaker. Also, airflow works well on pipelines that get data from multiple sources or perform data transformation. When the organizing, and scheduling of data pipeline workflows is pre-scheduled for a specific time interval airflow can be used efficiently~\cite{airflow_practice}.
        
        In this project, to accomplish workflow automation, we use Airflow to execute DAGs generated by the Knowledge Graph and Reasoner. We explain this further in Section~\ref{ch:Airflow_interactions}.
    \section{Continuous Integration and Continuous Deployment in GitLab}
    \label{section:gitlab_lr}
        Continuous Integration and Continuous Deployment or `CI/CD', is \say{a method to frequently deliver apps to customers by introducing automation into the stages of app development} \cite{what_is_cicd}, put simply.
	       CI/CD is defined declaratively as a pipeline. This pipeline takes in code,  performs some commands, and deploys it. 
	           \begin{figure}[h!]
    	            \centering
    	            \includegraphics[scale = 0.5]{img/ci-cd-flow-desktop.png}
    	            \caption{The General CI/CD Pipeline Workflow}
    	            \label{fig:ci-cd-flow-desktop}
    	        \end{figure}
    	   Each of the ‘Continuous’ steps has certain stages as explained below. Refer to \ref{fig:ci-cd-flow-desktop} for a Visual Representation.
    	        \begin{enumerate}
    	            \item \underline{Continuous Integration}: Continuous Integration focuses on automating code integration by automatically testing the code, ensuring that it is bug-free, and integrating it with a Version Control System, typically hosted in the Cloud. This speeds up the code merge process as it no longer needs to be merged and built on a developer's machine and is instead built in the cloud.
    	            \item \underline{Continuous Delivery}: Continuous Delivery is the automatic delivery of code to various testing environments such as development and staging for quality assurance (QA)  purposes. 
    	            \item \underline{Continuous Deployment}: This is the final step in the DevOps process wherein the code, after testing and building, is pushed to the server in the form of build artifacts.
    	        \end{enumerate}
    	   These stages need to be in the same order but a typical workflow may not see all the stages. For example, we can compare the concept of ‘Merge Request’ builds with a typical workflow. In it, the merge is tested with the help of automated testing before it can be merged into another branch.
    	   
    	   The following are some of the advantages of CI/CD \cite{benefits_of_cicd}.
        	   \begin{itemize}
                    \item Automated testing enables continuous delivery, which ensures software quality and security, and increases the profitability of code in production.
                    \item CI/CD pipelines enable a much shorter time to market for new product features, creating happier customers and lowering strain on development.
                    \item The great increase in overall speed of delivery enabled by CI/CD pipelines improves an organization’s competitive edge.
                    \item Automation frees team members to focus on what they do best, yielding the best end products.
                    \item Organizations with a successful CI/CD pipeline can attract great talent. By moving away from traditional waterfall methods, engineers and developers are no longer bogged down with repetitive activities that are often highly dependent on the completion of other tasks. 
        	   \end{itemize}
        	
        	GitLab has an inbuilt solution for CI/CD called GitLab Runner which runs and deploys the code. The GitLab Runner can be installed on our infrastructure which means that the runner should run on an isolated VM which is only dedicated to the runner~\cite{gitlab_runner}. This is due to performance issues.
        	
        	The runner is open source and hence, the source code is publicly accessible at \url{https://gitlab.com/gitlab-org/gitlab-runner/}.
        	
        	Pipeline code can be executed by various ‘methods’ that GitLab calls as executors. The following is the list of executors \cite{list_of_gitlab_executors}:
        	    \begin{itemize}
        	        \item SSH
        	        \item Shell
        	        \item Parallels
                    \item VirtualBox
                    \item Docker
                    \item Docker Machine (auto-scaling)
                    \item Kubernetes
                    \item Custom
        	    \end{itemize}
        	 
        	For our implementation, we have used `Docker' as our executor which means it provides a clean build environment without having to install any tools on the host machine.



         \section{Grafana}
Grafana \cite{grafana} is an open-source solution for running data analytics, pulling up metrics that make sense of the massive amount of data & to monitor our apps with the help of customized dashboards. Grafana, being an open-source solution, also enables us to write plugins from scratch for integration with several different data sources. The tool helps us study, analyze, and monitor data over some time, technically called time series analytics. It helps us track the user behavior, application behavior, frequency of errors popping up in production or a pre-prod environment, type of errors popping up, and contextual scenarios -- by providing relevant data.
    	   