\chapter{Developer Manual} \label{ch:developer_manual_}
	    \section{Deploying a Container} \label{ch:developer_manual_rancher}
	    
	        Requirements: Owner-level role access to the DLRL cluster on CS Cloud
	        
	        \emph{Additional steps on how to expose this container using a Kubernetes Ingress can be found 
         in a video guide we prepared
         \cite{video_guide}.}
	    
    	    \begin{enumerate}
    	        \item Navigate to https://cloud.cs.vt.edu (Figure~\ref{fig:8_1}).
        	        \begin{figure}[h!]
        	            \centering
        	            \includegraphics[scale = 0.2]{img/cloudcslogin.png}
        	            \caption{Cloud CS Login Page}
        	            \label{fig:8_1}
        	        \end{figure}
    	        \item Use your CS username and password to log in
    	        \item Click on the “Discovery” cluster and navigate to the DLRL project (Figure~\ref{fig:8_2}).
    	        \newpage
        	        \begin{figure}[h!]
        	            \centering
        	            \includegraphics[scale = 0.2]{img/Screenshot 2022-09-15 at 23-10-55 Computer Science Cloud.png}
        	            \caption{CS Cloud Homepage}
        	            \label{fig:8_2}
        	        \end{figure}
    	        \item Click on deploy (Figure~\ref{fig:8_3}).
        	        \begin{figure}[h!]
        	            \centering
        	            \includegraphics[scale = 0.2]{img/activeworkloads.png}
        	            \caption{Workloads}
        	            \label{fig:8_3}
        	        \end{figure}
    	        \item Figure~\ref{fig:8_4} shows the configuration for the container. The name is just the name of the container. The Docker Image can be any image from the Docker Hub. A namespace can be thought of as a logical folder space wherein workloads/services can be placed. Ports can be mapped. A node port is a port that can be used to explicitly expose the port of a pod on a node. Hostport, like the standard Docker port, is used to expose the container port. A Node port has a value between 30,000 to ~33,000 and is generally used along with a load balancer.
    	        \newpage
        	        \begin{figure}[h!]
        	            \centering
        	            \includegraphics[scale = 0.2]{img/workloadconfig.png}
        	            \caption{Workload deployment page}
        	            \label{fig:8_4}
        	        \end{figure}
        	   \item After that we see the environment variables configuration options along with node scheduling which can be used to deploy to specific nodes (Figure~\ref{fig:8_5}).
        	        \begin{figure}[h!]
        	            \centering
        	            \includegraphics[scale = 0.2]{img/env.png}
        	            \caption{Environment Variables and Scheduling}
        	            \label{fig:8_5}
        	        \end{figure}
        	   \item The most important section is the volumes. To have persistent storage across restarts and failures, we need to use a certain persistent mount point or storage. The CS cloud has a managed Ceph container setup that is ready to use. Input the supplied folder and credentials, and monitor IPs to gain access (Figure~\ref{fig:8_6}).
    	            \begin{figure}[h!]
        	            \centering
        	            \includegraphics[scale = 0.2]{img/attachingceph.png}
        	            \caption{Attaching a Ceph filesystem}
        	            \label{fig:8_6}
        	        \end{figure}
        	   \newpage
        	   \item After attaching Ceph, we define the mount point (Figure~\ref{fig:8_7}).
        	        \begin{figure}[h!]
        	            \centering
        	            \includegraphics[scale = 0.2]{img/afterattachceph.png}
        	            \caption{Mount configurations}
        	            \label{fig:8_7}
        	        \end{figure}
        	  \item Once we click on Launch, the system will spin up the image. If that succeeded, a pod will be created successfully as seen in Figure~\ref{fig:8_8}.
            	  \begin{figure}[h!]
        	            \centering
        	            \includegraphics[scale = 0.2]{img/successpods.png}
        	            \caption{Running Pods}
        	            \label{fig:8_8}
        	        \end{figure}
        	   \newpage
        	   \item We expose various containers that we create using the Kubernetes resource definition of the Ingress, which is internally implemented as a Load balancer in Level-7 of the OSI model. We define this in the `Load Balancer' tab. Click on Add and a configuration page will open up (Figure~\ref{fig:8_9}).
        	   \begin{figure}[h!]
        	            \centering
        	            \includegraphics[scale = 0.2]{img/ingress.png}
        	            \caption{Load Balancer Config}
        	            \label{fig:8_9}
        	        \end{figure}
        	   \item These deployments which contain pods are not publicly accessible. They can be used by other pods in other deployments to exchange data; this exposed IP is a local IP. To access, for example, a Web Server, a service called Ingress is required. This ingress acts as a connection between the outside internet and the deployment. Add a user-friendly name and assign it a namespace. Click on `Specify hostname’ to use a hostname of the form xxxx.discovery.cs.vt.edu. Use `/’ as the path, and the workload will be the container that we just set up. The port will be the one that we choose to expose. Click on save and if it succeeds, the newly created container will be available on xxxx.discovery.cs.vt.edu if it happens to be a web server-based container.
        	   \subsection{Troubleshooting}
        	   \begin{enumerate}
        	        \item Frequently, you might encounter the error \textbf{CrashLoopBackOff: “Back-off 10s restarting failed container=web pod=xxxxxxx(461c937d-d870-11e6-98de-005056040cc2)”}. All this error means is that your container failed to initialize. This could be due to various errors such as incorrect storage credentials, entry point script errors, insufficient resources, etc. The following 2 things might be useful for diagnosis:
            	        \begin{enumerate}
            	            \item \underline{Container Logs}: This is the most useful way to find the reason why the container is crashing. To access the logs of a container, click on a pod and then click on the 3 dots near the top right to see a small menu (see Figure \ref{fig:container_options}). Click on ‘View Logs’. This will open up a pop-up with all the logs displayed as seen in Figure \ref{fig:container_logs}
            	            
            	            \begin{figure}[h!]
                	            \centering
                	            \includegraphics[scale = 0.2]{img/logoptions.png}
                	            \caption{Option to see Container Logs}
                	            \label{fig:container_options}
                	        \end{figure}
                	        \begin{figure}[h!]
                	            \centering
                	            \includegraphics[scale = 0.2]{img/logsviewer.png}
                	            \caption{Container Logs}
                	            \label{fig:container_logs}
                	        \end{figure}
        	            \item \underline{Pod Events}: Once you click on the pod, you can view the events such as image pull, container initialization, etc. in the ‘Events’ section as seen in Figure  \ref{fig:pod_events}. Although not the most useful as it does not state reasons why a Pod crashed, it can still output errors during image pull, for example a 401 error when attempting to pull from a private registry whose access tokens have not been set up in Rancher.
        	        \end{enumerate}
        	        \begin{figure}[h!]
        	            \centering
        	            \includegraphics[scale = 0.2]{img/podevents.png}
        	            \caption{Pod Events}
        	            \label{fig:pod_events}
        	        \end{figure}
        	   \end{enumerate}
    	    \end{enumerate}
    	    
	    \section{Granting Access to Team Members on CS Cloud and GitLab}
    	    \begin{enumerate}
    	        \item Students from the CS department can directly set up their account on cloud.cs.vt.edu by setting up a password
    	        \item Students from other departments need specified access permissions to be able to access their accounts for cloud.cs.vt.edu
    	        \item The INT team is given Owner Access to be able to create and modify containers, adding other members
    	        \item Other team members are given access by assigning Project Member status
    	    \end{enumerate}
    	 % \section{Apache Airflow}
      %       In this section, we show the procedure of running DAG with Apache Airflow using a web server. 
      %       \begin{enumerate}
      %        %    \item Firstly, developers need to define a DAG workflow. For example, here we want to create a task in a DAG to spawn a container with a Jupyter Notebook environment, using “rahulavt/jupyter:first” image (Figure~\ref{fig:af-python}). \textbf{Note that this is only for demonstration purposes. For the final implementation, DAGs are automatically generated through knowledge graph and metadata tables so there is no need for users to hard-code them manually.}
      %        %    \begin{figure}[h!]
      %   	    %         \centering
      %   	    %         \includegraphics[scale = 0.5]{img/af-python.png}
      %   	    %         \caption{Example Python Script for a DAG}
      %   	    %         \label{fig:af-python}
      %   	    % \end{figure}
      %           \item Users can access Airflow main web server page, click into the DAG they want to execute (Figure~\ref{fig:af-dagspec}), and enable the DAG if previously not enabled. 
      %           \begin{figure}[h!]
      %   	            \centering
      %   	            \includegraphics[scale = 0.2]{img/af-dagspec.png}
      %   	            \caption{DAG's Specification Page}
      %   	            \label{fig:af-dagspec}
      %   	    \end{figure}
      %           \item Then click the “arrow” button on the right-hand side, hit “Trigger DAG”, and refresh the page. After that, You should be able to see a green indicating the DAG is successfully executed, click that (1) and click on “log”(2), see Figure~\ref{fig:af-afterexec}.
      %           \begin{figure}[h!]
      %   	            \centering
      %   	            \includegraphics[scale = 0.2]{img/af-afterexec.png}
      %   	            \caption{DAG Executed Successfully}
      %   	            \label{fig:af-afterexec}
      %   	    \end{figure}
      %           \item Inside the log page, you can see the DAG was executed and the Jupyter Notebook container is running(Figure~\ref{fig:af-log}).
      %           \begin{figure}[h!]
      %   	            \centering
      %   	            \includegraphics[scale = 0.2]{img/af-log.png}
      %   	            \caption{DAG Log in Airflow Webserver}
      %   	            \label{fig:af-log}
      %   	    \end{figure}
      %           \item To confirm, we go to \url{cloud.cs.vt.edu} and inside the discovery cluster, we can observe the container we just spawned (Figure~\ref{fig:af-jupyter-rancher}).
      %           \begin{figure}[h!]
      %   	            \centering
      %   	            \includegraphics[scale = 0.25]{img/af-jupyter-rancher.png}
      %   	            \caption{Confirm with Rancher}
      %   	            \label{fig:af-jupyter-rancher}
      %   	    \end{figure}
      %       \end{enumerate}
      \section{Using the Reasoner APIs}
            In order to use the Reasoner and Airflow APIs, there are 2 ways: use the existing functions in the Frontend Repository, or use the bare APIs. Following are the ways to use each.
            \subsection{Functions}
                These functions are available only if you are integrating any code into the Frontend code. The file with the following URL \href{https://code.vt.edu/aaron2000/cs5604-front-end/-/blob/main/src/api/AirflowReasoner.js}{https://code.vt.edu/aaron2000/cs5604-front-end/-/blob/main/src/api/AirflowReasoner.js} contains the functions. We will now give a description for each, and explain their order of execution.
                \begin{enumerate}
                    \item generateWorkflow: This functions takes 2 parameters as input: workflowID and the payload. The Workflow ID is the ID used to run a Workflow. For example, for summarization and classification, the workflowID is 124. Payload is the JSON object that contains the environment variables to be passed. The Response Body contains the generatedWorkflowID that needs to be passed to the runWorkflow function.
                    \item runWorkflow: As stated above, it takes the generatedWorkflowID as input and an array of service names that can be used to provide user understandable names for various services run. The result of this is a status link that provides the status of the run along with the run key.
                    \item getWorkflowStatus: The input to this function is the run key which is obtained from the runWorkflow function. The output gives the status. When the workflow has succeeded or failed, the logs are returned.
                \end{enumerate}
                Now, the order to run these is just 1, 2, and then 3. 
            \subsection{Bare APIs}
                For projects not using the Frontend UI, the APIs will need to be integrated directly. Section 6.1.2 has a detailed explanation for each API along with the order to run each.
    	 \section{GitLab CI/CD} \label{gitlab_cicd_setup}
    	    To set up Continuous Integration and Continuous Deployment, a DevOps Pipeline needs to be set up. This pipeline takes code, performs some tasks such as building or testing, and then deploys it to some location. This is the gist of what a pipeline does. Pipelines can also be increasingly complex. 
    	    Since we are using GitLab as our main repository for storing all the code, we will also make use of GitLab's own DevOps, also called the GitLab Runner. Following are the ways to set up the GitLab runner and also configure it to make use of the Docker executor.
    	    
    	    As mentioned before, you need to have Docker installed on the VM. It is sufficient to install the Docker Engine since the only interface will be the command line. To install Docker, the following steps need to be performed which are for Ubuntu and are hence, OS-dependent.
        	    \begin{enumerate}
        	        \item Go to \url{https://docs.docker.com/engine/install/} and select your Operating System. Follow the steps to install Docker.
        	        \item Once installed, start the Docker service using the command 
            	        \begin{verbatim}
                            sudo service docker start
                        \end{verbatim}
                    \item Verify that the Docker install and startup were successful by running a test image for example
                        \begin{verbatim}
                            sudo docker run hello-world
                        \end{verbatim}
        	    \end{enumerate}
        	
        	Now that the Docker Engine is installed, the GitLab Runner can be installed. We recommend using the Linux Distribution so that the local Docker Engine can be used without the need for the installation of any specific Programming Language libraries.
        	    \begin{enumerate}
        	        \item Go to \url{https://docs.gitlab.com/runner/install/linux-repository.html} and install the runner for the specific distribution.
        	        \item Verify the installation by running 
        	            \begin{verbatim}
                            sudo gitlab-runner status
                        \end{verbatim}
                    \item We now need to Register the Runner. In your Repository (here we are using a Repository in \url{https://code.vt.edu}), Go to Settings (Left Menu) -> CI/CD -> Runners (see Figure \ref{fig:gitlab_runner_view})
                        \begin{figure}[h!]
            	            \centering
            	            \includegraphics[scale = 0.4]{img/gitlab_runner_view.png}
            	            \caption{GitLab Runner Configuration}
            	            \label{fig:gitlab_runner_view}
            	        \end{figure}
            	   \item  Note the URL and Registration token.
            	   \item Now run the following command 
        	            \begin{verbatim}
                            sudo gitlab-runner register
                        \end{verbatim}
                    \item Enter all the details as asked and select Docker as the executor and docker:stable as the default image.
                    \item Run the following command to start the runner
                        \begin{verbatim}
                            sudo gitlab-runner start
                        \end{verbatim}
                        and once started, it can be seen in the ‘Specific Runners’ section as shown in Figure \ref{fig:gitlab_runner_success}.
                        \begin{figure}[h!]
            	            \centering
            	            \includegraphics[scale = 0.4]{img/gitlab_runner_success.png}
            	            \caption{Available GitLab Runners}
            	            \label{fig:gitlab_runner_success}
            	        \end{figure}
            	   \item Unfortunately, the pipeline will fail due to the Docker executor not having access to the host's Docker engine. To fix this, the config.toml file needs to be modified. This file is located at 
            	        \begin{verbatim}
                            /etc/gitlab-runner/
                        \end{verbatim}
                    \item Once opened, the privileged option should be set to true and the volumes should include the Docker socket as in the host Docker socket should be mounted to the Containers. 
                        \begin{verbatim}
    privileged = true
    volumes = [“/var/run/docker.sock:/var/run/docker.sock”, “/cache”]
                        \end{verbatim}
                    Figure \ref{fig:gitlab_config_file} has the working config file with the changes highlighted. 
                        \begin{figure}[h!]
            	            \centering
            	            \includegraphics[scale = 0.7]{img/gitlab_config_file.png}
            	            \caption{GitLab Runner's Config file}
            	            \label{fig:gitlab_config_file}
            	        \end{figure}
            \subsection{Troubleshooting}
                One of the main problems arises when one tries to deploy the GitLab Runner onto Kubernetes, for example Rancher. This happens when Docker is selected as the executor. Since the Container within the Pod does not have full root access to the underlying file system, various permission errors get thrown. 
                \newline
                \newline
                Another problem arises when Podman is used instead. Podman is a daemonless container management system similar to Docker. Unfortunately, Podman fails to execute as well within a pod since it requires additional subuids and subgids.
                If you are trying to deploy your instance of the GitLab Runner, we, therefore, recommend running the GitLab Runner inside a dedicated VM with full root access. 

                
            	   

                    
        	    \end{enumerate}
        \section{Cloud CS and GitLab Runner Resilience}
            As with a system, both Cloud CS and the Virtual Machine upon which the Runner is hosted can experience outages. It is good to note that \href{https://code.vt.edu}{code.vt.edu} is not expected to have significant issues since it is hosted on Amazon Web Services.
            The following 2 sections talk about what measures have been put in place and what more can be done.

            \subsection{GitLab Runner}
                Both Docker and the GitLab Runner have been configured to run during OS initialization. This prevents manual intervention. Moreover, because the Runner uses Docker as the executor, no additional software needs to be installed as all software needed can be used from Docker images.
            \subsection{Cloud CS}
                Since Cloud CS runs Kubernetes, in case of Pod failure, they get restarted automatically. This restart only happens if a deployment is created. A deployment can manage pods and their scaling. 

                Although Kubernetes automatically manages pods, their individual file system is ephemeral. It thus becomes pertinent to mount specific data folders to persistent storage such as Ceph.
                
                \subsubsection{PostgreSQL}
                    PostgreSQL, which is the main Database of the class, needs to be mounted.
                    Otherwise, after a restart, all data will be lost. As seen in Figure \ref{fig:postgres_mount}, the env PGDATA is defined which specifies where the PostgreSQL data should be stored, and we mount Ceph to /var/lib/postgresql/data.
                     \begin{figure}[h!]
            	            \centering
            	            \includegraphics[scale = 0.5]{img/postgres_mount.png}
            	            \caption{Team 1s Database volume configuration}
            	            \label{fig:postgres_mount}
            	        \end{figure}
                    \newline
                    If we now view the contents of the directory, /team1/postgres/data, we can see all PostgreSQL database data. See Figure \ref{fig:ceph_postgres_view}.
                     \begin{figure}[h!]
            	            \centering
            	            \includegraphics[scale = 0.5]{img/ceph_postgres_view.png}
            	            \caption{Contents of the /team1/postgres/data directory in Ceph}
            	            \label{fig:ceph_postgres_view}
            	        \end{figure}
                     \newline
                \subsubsection{Elasticsearch}
                    Similar to PostgreSQL, Team 2's Elasticsearch is also mounted in Ceph. This provides persistence for the indexes. See Figure \ref{fig:elastic_mount}.
                    \begin{figure}[h!]
            	            \centering
            	            \includegraphics[scale = 0.5]{img/elastic_mount.png}
            	            \caption{Team 2's Elasticsearch volume configuration}
            	            \label{fig:elastic_mount}
            	        \end{figure}
                \newline
            
%             \section{APIs from all Teams}
% \begin{table}[]
% % \rotatebox{90}
% \begin{tabular}{m{\dimexpr0.333\textwidth-23\tabcolsep-\arrayrulewidth\relax}|
%                 m{\dimexpr0.333\textwidth-8\tabcolsep-\arrayrulewidth\relax}|
%                 m{\dimexpr0.333\textwidth-8\tabcolsep-\arrayrulewidth\relax}|
%                 m{\dimexpr0.333\textwidth-15\tabcolsep-\arrayrulewidth\relax}|
%                 m{\dimexpr0.333\textwidth-15\tabcolsep-\arrayrulewidth\relax}|
%                 m{\dimexpr0.333\textwidth-15\tabcolsep-\arrayrulewidth\relax}|
%                 m{\dimexpr0.333\textwidth-10\tabcolsep-\arrayrulewidth\relax}|
%                 m{\dimexpr0.333\textwidth-10\tabcolsep-\arrayrulewidth\relax}|
%               }
% \textbf{Team} & \textbf{Container}                            & \textbf{Container role}                                 & \textbf{API is Defined/Called} & \textbf{Request Method} & \textbf{API}                                                                                                                                                                                                                                           & \textbf{API description}                                                                                                                                                                                                                             & \textbf{Report section} \\
% \hline
% 4             & https://team4-container-2.discovery.cs.vt.edu & generating summaries                                    & Called                         & GET                     & https://team-1-flask.discovery.cs.vt.edu/v1/etds/\textless{}etd-id\textgreater{}/objects?type=cleaned\_text used in https://code.vt.edu/harishbabu/cs5604-f22-team-4/-/tree/Summarization/driver.py to get the object ID                               & Getting Object IDs and Object metadata                                                                                                                                                                                                               & 7.2                     \\
% 4             & https://team4-container-2.discovery.cs.vt.edu & generating summaries                                    & Called                         & GET                     & https://team-1-flask.discovery.cs.vt.edu/v1/objects/\textless{}object-id\textgreater{}/file used in https://code.vt.edu/harishbabu/cs5604-f22-team-4/-/tree/Summarization/driver.py to get the object file using object ID                             & Getting Object File using Object ID                                                                                                                                                                                                                  & 7.2                     \\
% 4             & https://team4-container-2.discovery.cs.vt.edu & generating summaries                                    & Called                         & POST                    & save\_summary in https://code.vt.edu/harishbabu/cs5604-f22-team-4/-/tree/Summarization/driver.py                                                                                                                                                       & save the summary JSON into the database                                                                                                                                                                                                              & 7.3.3                   \\
% 4             & https://team4-container-1.discovery.cs.vt.edu & generating classification labels                        & Called                         & GET                     & https://team-1-flask.discovery.cs.vt.edu/v1/etds/\textless{}etd-id\textgreater{}/objects?type=cleaned\_text used in https://code.vt.edu/ano22/cs5604-team4-classification/-/blob/main/Final\_Classification\_Store\_\_1\_.ipynb to get the object ID   & Getting Object IDs and Object metadata                                                                                                                                                                                                               & 7.2                     \\
% 4             & https://team4-container-1.discovery.cs.vt.edu & generating classification labels                        & Called                         & GET                     & https://team-1-flask.discovery.cs.vt.edu/v1/objects/\textless{}object-id\textgreater{}/file used in https://code.vt.edu/ano22/cs5604-team4-classification/-/blob/main/Final\_Classification\_Store\_\_1\_.ipynb to get the object file using object ID & Getting Object File using Object ID                                                                                                                                                                                                                  & 7.2                     \\
% 4             & https://team4-container-1.discovery.cs.vt.edu & generating classification labels                        & Called                         & POST                    & save\_classification in https://code.vt.edu/ano22/cs5604-team4-classification/-/blob/main/Final\_Classification\_Store\_\_1\_.ipynb                                                                                                                    & save the classification JSON into the database                                                                                                                                                                                                       & 7.2                     \\
% 5             & https://reasoner.discovery.cs.vt.edu          & Reasoner-related container service                      & Called                         & GET                     & https://reasoner.discovery.cs.vt.edu/generateGrammar                                                                                                                                                                                                   & read reasoner table from postgres database, then generate context-free-grammar file                                                                                                                                                                  & 6.1.2                   \\
% 5             & https://reasoner.discovery.cs.vt.edu          & Reasoner-related container service                      & Called                         & POST                    & https://reasoner.discovery.cs.vt.edu/generateWorkflow/:workflowId                                                                                                                                                                                      & API to generate workflow based on the end goal Id                                                                                                                                                                                                    & 6.1.2                   \\
% 5             & https://team2020-airflow.discovery.cs.vt.edu  & Airflow-related container service                       & Called                         & POST                    & https://team2020-airflow.discovery.cs.vt.edu/workflow/run                                                                                                                                                                                              & API to run the workflow based on given workflow Id                                                                                                                                                                                                   & 6.1.2                   \\
% 5             & https://team2020-airflow.discovery.cs.vt.edu  & Airflow-related container service                       & Called                         & GET                     & http://team2020-airflow.discovery.cs.vt.edu/workflow/run?key=\{\{key\}\}                                                                                                                                                                               & API to get running workflow status                                                                                                                                                                                                                   & 6.1.2                   \\
% 5             & https://team2020-api.discovery.cs.vt.edu      & Workflow-CRUD-operation (goals, services, or reasoners) & Called                         & GET                     & https://team2020-api.discovery.cs.vt.edu/\{\{object\}\}/                                                                                                                                                                                               & Get all metadata from a specific resource table(goals, services, or reasoners)                                                                                                                                                                       & 6.1.3                   \\
% 5             & https://team2020-api.discovery.cs.vt.edu      & Workflow-CRUD-operation (goals, services, or reasoners) & Called                         & GET                     & https://team2020-api.discovery.cs.vt.edu/\{\{object\}\}/\{\{resource\_id\}\}                                                                                                                                                                           & Get a spefiic resource metadata from either goal, service, or reasoners                                                                                                                                                                              & N/A                     \\
% 5             & https://team2020-api.discovery.cs.vt.edu      & Workflow-CRUD-operation (goals, services, or reasoners) & Called                         & DELETE                  & https://team2020-api.discovery.cs.vt.edu/\{\{object\}\}/\{\{resource\_id\}\}/                                                                                                                                                                          & delete a specific resource from either goal, service, or reasoners                                                                                                                                                                                   & N/A                     \\
% 5             & https://team2020-api.discovery.cs.vt.edu      & Workflow-CRUD-operation (goals)                         & Called                         & POST                    & https://team2020-api.discovery.cs.vt.edu/goals/                                                                                                                                                                                                        & add a new goal, body should be goal\_metadata                                                                                                                                                                                                        & 7.2                     \\
% 5             & https://team2020-api.discovery.cs.vt.edu      & Workflow-CRUD-operation (goals)                         & Called                         & PUT                     & https://team2020-api.discovery.cs.vt.edu/goals/\{\{goal\_id\}\}                                                                                                                                                                                        & udpate a goal metadata, body should be goal\_metadata                                                                                                                                                                                                & N/A                     \\
% 5             & https://team2020-api.discovery.cs.vt.edu      & Workflow-CRUD-operation (services)                      & Called                         & POST                    & https://team2020-api.discovery.cs.vt.edu/services/                                                                                                                                                                                                     & add a new service, body should be service\_metadata                                                                                                                                                                                                  & 7.2                     \\
% 5             & https://team2020-api.discovery.cs.vt.edu      & Workflow-CRUD-operation (services)                      & Called                         & PUT                     & https://team2020-api.discovery.cs.vt.edu/services/\{\{service\_id\}\}                                                                                                                                                                                  & update a service metadata, body should be service\_metadata                                                                                                                                                                                          & N/A                     \\
% 5             & https://team2020-api.discovery.cs.vt.edu      & Workflow-CRUD-operation (reasoners)                     & Called                         & POST                    & https://team2020-api.discovery.cs.vt.edu/goals/                                                                                                                                                                                                        & add a new reaonser (link goals and services), body should contain service\_id, input\_id, and output\_id                                                                                                                                             & 7.2                     \\
% 5             & https://team2020-api.discovery.cs.vt.edu      & Workflow-CRUD-operation (reasoners)                     & Called                         & PUT                     & https://team2020-api.discovery.cs.vt.edu/goals/?input\_id=\{\{input\_id\}\}\&service\_id=\{\{service\_id\}\}\&output\_id=\{\{output\_id\}\}                                                                                                            & update a resoner,  body should contain service\_id, input\_id, and output\_id                                                                                                                                                                        & N/A                     \\
% 5             & https://team2020-api.discovery.cs.vt.edu      & Workflow-CRUD-operation (reasoners)                     & Called                         & GET                     & https://team2020-api.discovery.cs.vt.edu/goals/data?input\_id=\{\{input\_id\}\}\&service\_id=\{\{service\_id\}\}\&output\_id=\{\{output\_id\}\}                                                                                                        & get reasoner metadata                                                                                                                                                                                                                                & N/A                     \\
% 5             & https://team2020-api.discovery.cs.vt.edu      & Workflow-CRUD-operation (reasoners)                     & Called                         & DELETE                  & https://team2020-api.discovery.cs.vt.edu/goals/?input\_id=\{\{input\_id\}\}\&service\_id=\{\{service\_id\}\}\&output\_id=\{\{output\_id\}\}                                                                                                            & delete a reaonser                                                                                                                                                                                                                                    & N/A                     \\
% 3             & https://team3-container-1.discovery.cs.vt.edu & Object Detection                                        & Called                         & GET                     & https://team-1-flask.discovery.cs.vt.edu/v1/etds/\textless{}etd-id\textgreater{}/pdf                                                                                                                                                                   & Get ETD PDF by ETD ID                                                                                                                                                                                                                                & 7.2.2                   \\
% 3             & https://team3-container-1.discovery.cs.vt.edu & Object Detection                                        & Called                         & POST                    & https://team-1-flask.discovery.cs.vt.edu/v1/objects/\textless{}etd-id\textgreater{}/\textless{}type\_name\textgreater{}                                                                                                                                & Save detected object (ETD page images, XML, text and image objects)                                                                                                                                                                                  & 4.4.1                   \\
% 3             & https://team3-container-1.discovery.cs.vt.edu & Object Detection                                        & Called                         & GET                     & https://team-1-flask.discovery.cs.vt.edu/v1/etds/\textless{}etd-id\textgreater{}/objects?type=\textless{}type\_name\textgreater{}                                                                                                                      & Get Objects by ETD ID                                                                                                                                                                                                                                & 5.3.1                   \\
% 1             & https://team-1-flask.discovery.cs.vt.edu      & Back-end server                                         & Called                         & POST                    & https://team-1-flask.discovery.cs.vt.edu/v1/etds                                                                                                                                                                                                       & Add new ETD to the database and upload the file to the file system                                                                                                                                                                                   & 7.8.1                   \\
% 1             & https://team-1-flask.discovery.cs.vt.edu      & Back-end server                                         & Called                         & POST                    & https://team-1-flask.discovery.cs.vt.edu/v1/topic\_model/object/\textless{}object-id\textgreater{}                                                                                                                                                     & Save topic modeling results. Example: \{  "related\_topics": {[}    "word\_1",    "word\_2",    "word\_3",    "word\_4",    "word\_5"  {]},  "related\_document": {[}   23,    55,    66,    100,    98  {]},  "topic\_model\_used": "model\_name"\} & 7.8.1                   \\
% 1             & https://team-1-flask.discovery.cs.vt.edu      & Back-end server                                         & Called                         & GET                     & https://team-1-flask.discovery.cs.vt.edu/v2/digitalobjects/all?page=5\&per\_page=50                                                                                                                                                                    & Get ETD and its objects v2                                                                                                                                                                                                                           & 7.8.1                   \\
% 1             & https://team-1-flask.discovery.cs.vt.edu      & Back-end server                                         & Called                         & GET                     & https://team-1-flask.discovery.cs.vt.edu/v1/report/school-contribution                                                                                                                                                                                 & Currently it return the school contribution ratio.                                                                                                                                                                                                   & 7.8.1                   \\
% 1             & https://team-1-flask.discovery.cs.vt.edu      & Back-end server                                         & Called                         & GET                     & https://team-1-flask.discovery.cs.vt.edu/v1/etds/\textless{}etd-id\textgreater{}                                                                                                                                                                       & Get ETD by ETD id                                                                                                                                                                                                                                    & 7.8.1                   \\
% 3             & https://team3-container-2.discovery.cs.vt.edu & Topic modeling                                          & Called                         & GET                     & https://team-1-flask.discovery.cs.vt.edu/API/get\_etd\_embeddings/\{model\_name\}/\{num\_topics\}/\{etd\_id\}                                                                                                                                          & Function to take in dataset etd\_id and return Topic vector                                                                                                                                                                                          & 4.4.2                   \\
% 3             & https://team3-container-2.discovery.cs.vt.edu & Topic modeling                                          & Called                         & GET                     & https://team-1-flask.discovery.cs.vt.edu/API/get\_etd\_relateddocs/\{model\_name\}/\{num\_topics\}/\{etd\_id\}/\{topk\}                                                                                                                                & Function to take in dataset etd\_id and return the related documents                                                                                                                                                                                 & 4.4.2                   \\
% 3             & https://team3-container-2.discovery.cs.vt.edu & Topic modeling                                          & Called                         & GET                     & https://team-1-flask.discovery.cs.vt.edu//API/get\_etd\_relatedtopics/\{model\_name\}/\{num\_topics\}/\{etd\_id\}/\{topk\}                                                                                                                             & Function to take in dataset etd\_id and return the related Topics                                                                                                                                                                                    & 4.4.2                   \\
% 2             & https://search.discovery.cs.vt.edu            & Search service backend                                  & Called                         & POST                    & https://search.discovery.cs.vt.edu/api/v1/search                                                                                                                                                                                                       & API to take in search query, search method and search by and return ETD metadata as search result                                                                                                                                                    & 4.2.1                   \\
% 2             & https://search.discovery.cs.vt.edu            & Search service backend                                  & Called                         & GET                     & https://search.discovery.cs.vt.edu/api/v1/documents/\textless{}etd\_id\textgreater{}                                                                                                                                                                   & API to fetch ETD metadata by its ID, used by front-end in the Doucment View                                                                                                                                                                          & 4.2.1                   \\
% 2             & https://search.discovery.cs.vt.edu            & Search service backend                                  & Called                         & GET                     & https://search.discovery.cs.vt.edu/api/v1/logs/\textless{}user\_id\textgreater{}                                                                                                                                                                       & API to fetch logs of a specific user by the user id, Used by recommendation service                                                                                                                                                                  & 4.2.3                   \\
% 2             & https://recommendation.discovery.cs.vt.edu    & Recommendation service Backend                          & Called                         & POST                    & https://recommendation.discovery.cs.vt.edu/api/v1/recommendation/infer                                                                                                                                                                                 & Takes in user\_id to infer recommendations for the user.                                                                                                                                                                                             & 4.2.4.6                 \\
% 2             & https://recommendation.discovery.cs.vt.edu    & Recommendation service Backend                          & Called                         & POST                    & https://recommendation.discovery.cs.vt.edu/api/v1/recommendation/train                                                                                                                                                                                 & Takes in ETDs read of all user and ETD data to train a model for recommendation                                                                                                                                                                      & 4.2.4.6                 \\
% 2             & https://search.discovery.cs.vt.edu            & Search service backend                                  & Called                         & POST                    & https://search.discovery.cs.vt.edu/api/v1/experiment/create                                                                                                                                                                                            & Creates a search experiment with given parameters from front-end search experimenter page                                                                                                                                                            & 4.2.1.3                 \\
% 2             & https://search.discovery.cs.vt.edu            & Search service backend                                  & Called                         & POST                    & https://search.discovery.cs.vt.edu/api/v1/experiment/delete                                                                                                                                                                                            & delete a search experiment by taking its name as input                                                                                                                                                                                               & 4.2.1.3                 \\
% 2             & https://search.discovery.cs.vt.edu            & Search service backend                                  & Called                         & POST                    & https://search.discovery.cs.vt.edu/api/v1/experiment/search                                                                                                                                                                                            & Searches over a created experiment index by taking in query parameters from search experimenter page                                                                                                                                                 & 4.2.1.3                 \\
% 2             & https://search.discovery.cs.vt.edu            & Search service backend                                  & Called                         & POST                    & https://search.discovery.cs.vt.edu/api/v1/index                                                                                                                                                                                                        & Indexes the ETD metadata into Elasticsearch, used by workflow automation.                                                                                                                                                                            & 4.2.1                   \\
% 2             & https://search.discovery.cs.vt.edu            & Search service backend                                  & Called                         & GET                     & https://search.discovery.cs.vt.edu/api/v1/chapters/\textless{}chapter\_id\textgreater{}                                                                                                                                                                & Gets the respective chapter by chapter id from Elasticsearch                                                                                                                                                                                         & 4.2.1.1                 \\
% 2             & https://search.discovery.cs.vt.edu            & Search service backend                                  & Called                         & POST                    & https://search.discovery.cs.vt.edu/api/v1/chapters/search                                                                                                                                                                                              & Searches over all the chapters stored in Elasticsearch                                                                                                                                                                                               & 4.2.1.2                 \\
% 2             & https://search.discovery.cs.vt.edu            & User management backend                                 & Called                         & POST                    & https://search.discovery.cs.vt.edu/api/v1/signup                                                                                                                                                                                                       & API to create a new user and store user details in Elasticsearch                                                                                                                                                                                     & 4.2.6.1                 \\
% 2             & https://search.discovery.cs.vt.edu            & User management backend                                 & Called                         & GET                     & https://search.discovery.cs.vt.edu/api/v1/users/\textless{}user\_id\textgreater{}                                                                                                                                                                      & API to fetch a user profile by user id from Elasticsearch                                                                                                                                                                                            & 4.2.6.1                 \\
% 2             & https://search.discovery.cs.vt.edu            & User management backend                                 & Called                         & PUT                     & https://search.discovery.cs.vt.edu/api/v1/users//\textless{}user\_id\textgreater{}                                                                                                                                                                     & API to update a user profile by user id                                                                                                                                                                                                              & 4.2.6.1                 \\
% 2             & https://search.discovery.cs.vt.edu            & User management backend                                 & Called                         & POST                    & https://search.discovery.cs.vt.edu/api/v1/google/login                                                                                                                                                                                                 & API to login using google ID                                                                                                                                                                                                                         & 4.2.6.1                
% \end{tabular}
% \end{table}