\chapter{Design} \label{ch:discussion}
	   \section{System Architecture}
	        % Content Here %
	        To design the architecture, we first proceeded with defining the overall architecture and inter-component relationships, to gain a holistic understanding of the system. We then concluded defining system boundaries and using domain-driven design, separating the concerns that we will facilitate as a team. 
	        
	        The whole system revolves around the idea of containers. Containers are executable units of software in which application code is packaged, along with its libraries and dependencies, in common ways so that it can be run anywhere, whether it be on a desktop, traditional IT, or the cloud.
	        
	        Each team writes code which is then packaged into containers, tagged with the git short commit, and then deployed onto the CS Cloud with the help of CI/CD. 

                Figure \ref{fig:overall_system_vision} shows the overall system with each team's responsibilities. We define them here briefly.
                
                \begin{figure}[h!]
    	            \centering
    	            \includegraphics[width=\textwidth]{img/system_arch.png}
    	            \caption{Overall system vision}
    	            \label{fig:overall_system_vision}
	            \end{figure}
	            
                \begin{enumerate}
                    \item Team-1 needs to be able to expose access and operations on data stored in persistent storage layers, namely the Camelot and Ceph filesystem~\cite{what_is_ceph}, and a Relational DataBase Management System (RDBMS) like PostgreSQL~\cite{about_postgresql}, via an Application Programming Interface (API)~\cite{what_is_an_api}. 
                    \item Team-2 needs to be able to query multiple data sources, process the data, and expose those search results through a search engine like Elastisearch~\cite{elastic_aws}.
                    \item Team-3's goals are twofold. They first need to access the ETD data exposed by Team-1, and train Machine Learning models on this data. Secondly, they need to store this data back into the storage layer exposed by Team-1.
                    \item Team-4's goals are quite similar to Team-3, from our perspective. They too need to access data, train models on it, and store the results in a storage layer.
                    \item Common goals: Each team needs version-control~\cite{what_is_vcs}, maintenance, and testing their code, seamlessly. 
                \end{enumerate}
                After understanding the holistic view of the system and identifying the different components and their interactions, we proceeded to design an architecture that would facilitate these requirements. Figure \ref{fig:system_architecture_overview} shows our system architecture, and identifies four different areas.
                
                \begin{figure}[h!]
    	            \centering
                    \includegraphics[width=16cm]{img/team_5_system_arch.png}
    	            % \includegraphics[scale = 0.4]{img/team_5_system_arch.png}
    	            \caption{System Architecture}
    	            \label{fig:system_architecture_overview}
	            \end{figure}
	            
	            
	            
	            \begin{enumerate}
	                \item Continuous Integration, Continuous Delivery: The teams will be able to maintain, test, and deploy their code via GitLab~\cite{why_gitlab} as the Version Control System (VCS).
	                \item The Camelot and Ceph filesystems: The Camelot and Ceph filesystems are where all the Electronic Theses and Dissertations (ETDs), knowledge graphs, machine learning Jupyter notebooks, PostgreSQL static files, and workflow automation temporary files will be stored. They are managed by Virginia Tech's Computer Science department's cloud infrastructure team and our team will be provided an interface to use them via static internal network IPs and credentials for passing access control.
	                \item Kubernetes and cloud infrastructure: We identified four different classes of containers, namely: storage (PostgreSQL), service, training, and reasoning. 
	                    \begin{enumerate}
	                        \item storage containers: These containers are responsible for the RDBMS and other services that will eventually store data, metadata, and trained models.
	                        \item service containers: These containers are responsible for serving business logic in the form of web payloads such as Javascript Object Notation (JSON)~\cite{what_is_json} or Extensible Markup Language (XML)~\cite{what_is_xml} and serving front-end web pages.
	                        \item training containers: These containers are utilized to train Machine Learning models on the ETD data.
	                        \item Workflow automation containers: These containers are used to serve the knowledge graph and workflow manager that will be developed by Team-5.
	                    \end{enumerate}
	                \item Apache Airflow: Orchestrating all these service containers will be done by Apache Airflow which provides a programmatic way to define workflows. 
	            \end{enumerate}
	            
	        
       \section{CI/CD Pipeline}
            One of the main aims of CI/CD is automating certain Developer tasks which tend to be repetitive and time-consuming. These tasks include deployment, unit testing, packaging, etc. To automate these, we can take advantage of DevOps pipelines which are built to execute code defined in an instructions file. GitLab Runner is one such tool that executes commands as defined in a .gitlab-ci.yml file. 
            
            We have identified certain cases wherein code can be containerized and deployed on the CS Cloud using Docker. 
            
            Each such use case has the following 3 files defined:
                \begin{enumerate}
                    \item Dockerfile: This is the instruction set that defines how to dockerize aka containerize the code and also defines a command or script to start the container when run with no dependencies needed after containerizing it. The Dockerfile is also immensely useful for creating Developer Environments which are portable custom software environments that have all code dependencies installed.
                    \item .dockerignore: This file instructs Docker to ignore paths defined in it during the build process. Its usefulness comes from the fact that there are certain files/folders which are not needed to run an application and can therefore cause unnecessary bloat in Docker images.
                    \item .GitLab-ci.yml: The GitLab CI/CD YAML file is the instruction file which in simple terms, instructs the GitLab Runner what to do with the code. Because it takes bash scripts as inputs, the Runner can do anything with the code. For logical division, jobs can be divided into stages such as build, test, and deploy with inter-job dependencies, artifact publishing, and much more. 
                \end{enumerate}
                
            The Runner uses the bitnami/kubectl image for running kubectl commands which can update images after they have been pushed to the code.vt.edu Container Registry. Due to timing constraints during development, some images are in Docker Hub while the rest are in code.vt.edu Container Registry. For an uncomplicated design, it is recommended for all images to be in code.vt.edu Container Registry.
            
            These images are tagged with the Git Short Commit and then updated in Cloud CS.
        \section{Workflow Automation Pipeline}
        \label{ch:Airflow_interactions}
        How does one efficiently run workflows without much infrastructure cost? A Workflow Manager is a solution to this problem and Apache Airflow is one such Workflow Manager. Airflow is based upon the concept of DAGs or Directed Acyclic Graphs, which means tasks that do not have circular dependencies. Each DAG consists of multiple tasks which are defined using Operators. These Operators perform some operations on data and then return results. 
        Let us take the example of PDF chapter summarization. When someone tries to upload a PDF and selects Chapter Summarization, only the appropriate service which is built to handle Summarization should be triggered and others should not. The advantage of Airflow is that these workflows are programmatic and defined using code written in Python.
        For our current model, we have defined Goals, Workflows, and Services. Following are their definitions:
            \begin{itemize}
                \item Goal: A Goal is an overall output that one wishes to achieve. Goals can include text summarization, segmentation, etc.
                \item Workflow: A Workflow describes what needs to be done to achieve a particular goal. It is important to note that each Workflow has exactly one goal but a goal can have many different workflows.
                \item Services: Lastly we have services which are the atomic tasks needed to complete a workflow. A workflow can thus have many different services, especially if it is complex. It can also have different combinations and linear/nonlinear services. 
            \end{itemize}

        \subsection{Overall Workflow Design}
            \label{section:airflowOverallDesign}
            The overall workflow design has been shown in Figure ~\ref{fig:af-design.png}. We used the Python script from the 2020 INT team to launch the workflow automation service. Developers can specify goals, and services, using the Flask UI which will then be populated into the database. Then they have to re-generate the generated context-free-grammar file referencing the workflow tables (details of the tables structure can be found in the next subsection). Then, end users can call the \textit{generate\_workflow} API to generate DAGs. They can also pass in variables such as \textit{ETD\_ID} as environment variables for each service. After that, they can trigger the DAGs through the Flask UI.
            % The Reasoner is a decision-making process that will decide based on factors which workflow is the most optimum.
            
            \begin{figure}[h!]
            	\centering
            	\includegraphics[scale = 0.56]{img/workflow_arch.png}
            	\caption{Workflow Automation Design}
            	\label{fig:af-design.png}
            \end{figure}
        
        \subsection{Workflow Database Design}
        \label{section:registryDatabasePrototype}
        Besides other metadata tables required for ETDs, we mainly have four tables that together provide the data for workflow functionality. We have designed our current schema keeping in mind the relationships between goals, workflows, and services. The table descriptions are as follows. The UML diagram for the same is shown in Figure~\ref{fig:uml_diagram_airflow}.
         
        \begin{figure}[h!]
        	\centering
        	\includegraphics[scale = 0.55]{img/ER.png}
        	\caption{ER diagram for Workflow Automation}
        	\label{fig:uml_diagram_airflow}
        \end{figure} 
        
            \subsubsection{Goal Metadata Table}
            This table stores goal-related information that will be used as inputs and outputs to the services metadata. It also includes the environment variables each goal requires. The schema contains the following columns:
            \begin{enumerate}
                \item goal\_id: (Integer) This primary key of the table uniquely identifies a goal.
                \item goal\_name: (String) A human-readable string that represents each goal’s functionality.
                \item goal\_description: (String) A brief description of the goal, that helps users understand its functionality.
                \item goal\_format: (String) Represents the format of the output represented by the goal.
                \item file\_location: (String) Represents the directory/filename associated with the goal.
                \item envs: (String) Some environment variables that can be used inside the container.
                \item owned\_by: (String) The owner of the goal.
            \end{enumerate}

            \subsubsection{Service Metadata Table}
                The Service Metadata table describes all services required by workflows. Each service picks up its input from a file location and deposits the output of its operations at another file location. This is how we achieve communication between services in a workflow. Note that a service might be an API defined by team 1 or any of the general services provided by teams 3 or 4. The schema contains the following columns:
                \begin{enumerate}
                    \item service\_id: (Integer) This primary key of the table uniquely identifies a service.
                    \item service\_name: (String) A human-readable string that represents each service’s functionality.
                    \item service\_description: (String) A brief description of the service, that helps users understand its functionality better.
                    \item image\_url: (String) Stores the Docker image URL on Docker Hub\footnote{While ideally all images should be in code.vt.edu Container Registry, due to Time constraints, some Docker Images are on Docker Hub while the Rest are in code.vt.edu Container Registry.}, for example, ``lyuze/postgres:latest''.
                    \item cluster\_namespace: (String) Represents the cluster namespace, e.g., etd.
                    \item owned\_by (String): (String) The owner of the service.
                \end{enumerate}
        
            % \subsubsection{Reasoner Metadata Table}
            %     The reasoner table contains the required `workflow' defined by the Knowledge Graph. Currently for simplicity, we only have 1 workflow per goal. The purpose of this table is for future continuous development when one goal may contain several workflows or other smart optimization to be done. The schema contains the following columns:
            %     \begin{enumerate}
            %         \item workflow\_id: (Integer) The primary key of the table that represents the workflow ID.
            %         \item workflow\_name: (String) Represents the workflow name.
            %         \item workflow\_description: (String) A human-readable string that represents each workflow’s functionality.
            %         % \item service\_input\_id: (Integer) For this step of the workflow, the service should be the input. The foreign key from the service metadata table.
            %     \end{enumerate}
                
                % Note that since each workflow may require multiple services and goals, but in the reasoner table, each row only contains one of the services required by this workflow, thus the workflow\_id should \textbf{\textit{not}} be unique/primary key.
                
            \subsubsection{Reasoner Table}
                The reasoner table contains the task order for the workflow. The reasoner service will reference this table to generate the context-free-grammar (cfg) file.
                \begin{enumerate}
                    \item goal\_id: (Integer) The foreign key referenced from the primary key of the Goal Metadata table. This column indicates the Output Goal of a particular service.
                    \item input\_goal\_id: (Integer) The foreign key referenced from the primary key of the Goal Metadata table. Certain services need Input Data/files, etc. That is indicated by this column.
                    \item service\_id: (Integer) The foreign key referenced from the primary key of the Service Metadata table.
                \end{enumerate}