\chapter{Overview} \label{ch:overview}

Our entire system consists of the following components.
\begin{enumerate}
            \item Developer Operations: We have created containerized environments for each team. Continuous Integration and Continuous Deployment have also been a major focus of our DevOps tasks. We have developed a pipeline that automatically deploys the latest image for any code that has been pushed to the master branch of the repository. It also has the provision of adding testing that can be done before deployment. 
            % Following are the Cluster Statistics by the end of the semester -
            % \begin{enumerate}
            %     \item 45 Deployments 
            %     \item 40 Ingresses
            %     \item 39 Pods
            %     \item 180 Services 
            %     \item 13 Secrets
            %     \item Volumes - CEPH: etd-cephfs-cs5604-pv: Total: 20TB, Used: 12T,  Available: 8.3T, Used: 59%
            % \end{enumerate}

            \item Workflow Management: The Workflow Management consists of 3 parts - the Registry UI (Section ~\ref{ch:registry_ui}), the Registry Backend (Section ~\ref{ch:registry_backend}), and the Airflow workflow automation system (Section ~\ref{ch:implementation_workflow_automation}). The Registry UI is aimed to allow the different teams to view, modify, or add more services, goals, and workflows to our Registry database. The Registry backend handles making these API calls and correctly registering this data in the database. The reasoner table, which holds the relationships between Goals and Services, is used to create a context-free grammar. This grammar, which contains production rules, is used to create workflows. The reasoner contains all the code to convert the reasoner table to a context-free grammar, and then from production rules, which are generated as a result of the context-free grammar, to actual workflows. The workflows are generated in the form of DAG files. The Airflow System brings the workflow to life by running the DAG file and spinning up all the required services mentioned within the file to accomplish the results of the workflow.
            \item DevOps Support: Alongside setting up the above, the INT team members have created guidelines and documentation to support other teams. We have also connected each of our team members to other teams to set up their containerized environments along with any other setup requirements and troubleshooting they may require.
            
\end{enumerate}
        
        \section{Collaboration and Communication}
        We have set up efficient collaboration and communication structures that involve all five people in our team. Our internal collaboration model is described in detail, in Section \ref{section:projectManagement}, which talks more about how we picked up a Human-Computer Interaction related monograph~\cite{user_centered_agile} that prescribes user-centered design techniques using Agile methods to define our project management strategy. With regards to communication, we use a Discord~\cite{discord} server, called ‘DevOps-Team’ with dedicated channels to have all our conversations and provide any context. We usually meet in person but have accommodations for people who may not be able to make it to the venue via Zoom~\cite{zoom}. We have dedicated liaisons for each team, details regarding which are communicated via our 
%\href{https://canvas.vt.edu/courses/156101/pages/f2022-team5}{
team page on Canvas. We use the class sessions as synchronous liaisoning sessions to communicate and collaborate with other teams. We have used these sessions to get requirements and also offer support to other teams. We use email, the class Discord channel, and our Canvas page for all asynchronous communication. 

        \section{Problems Faced}
        We faced the following problems in the first and second phases of implementation. Their solutions are discussed in Section \ref{ch:solutions_developed}:
        \begin{enumerate}
            \item Understanding which style of project management we must use, user requirements, and gaining those requirements from the respective teams.
            \item We were informed that the storage that was initially provisioned for us (Ceph~\cite{what_is_ceph}) is not going to be enough to store all the ETD data and will significantly fall short of it.
            \item We were informed by Teams 3 and 4 that they were unable to install dependencies on the Docker containers that we had spun up for them and were facing storage issues. This happens because the disk space on the container is provisioned at the time of creation and when the dependencies are installed after the container has been spun up, this results in disk space issues.
            \item Two of our original team members, who were liaisons to Team-3 and Team-4, stopped taking the course. Unfortunately, this happened after the team liaisons were allocated and some work had been scheduled and assigned. 
            \item Some of our team members did not have prior experience working with cloud technologies such as Kubernetes, Docker, etc. 
            \item We faced problems with deploying GitLab onto Kubernetes, as within a container it cannot gain root access to the filesystem, leading to various permission errors.
            \item We faced problems with detecting GPU nodes of the CS cloud infrastructure from the clusters of Team-3 and Team-4, which are necessary for training and running machine learning models.
            \item We faced issues with understanding and integrating the reasoner implementation of Team INT-2020 into our work due to a lack of sufficient documentation of the design choices they made.
            \item Downtime of the Cloud Environment sometimes led to a pause in all team's work. Since this is managed by the CS Department, we
            %have been communicating 
            had to communicate
            with them to keep it up and running.
            
        \end{enumerate}
        
        \section{Solutions Developed}\label{ch:solutions_developed}
        %At this juncture in time, we have ideated over many details of the project, which are part of this report, and others that we are yet to finalize. We are still identifying a few problems and 
        We have proceeded to approach problems as described above and as instructed by Professor Edward Fox, by gaining user requirements, identifying their goals, and designing our solution around them. We have worked along with our subject matter expert Mr. Dhanush Dinesh, as well as the teaching assistant for the course, Mr. Hemayet Ahmed Chowdhury. They have helped us outline and provide guardrails for the process. They in turn put us in touch with Mr. Chris Arnold, who manages both Ceph and Camelot. We also build on the work done by the team before us, who took the course in Fall 2020~\cite{INT2020}, as they have provided some directions to us in terms of the solution and architecture design. We worked to improve upon their solution. Specifically, in response to some of the problems mentioned above, we came up with the following solutions, numbered to match the list of problems from the previous section. 
        
        \begin{enumerate}
            \item The solution to this problem is described in detail in Section \ref{section:projectManagement}. However, to mention it briefly, we decided to use Scrum as the chosen framework for project management. Scrum also details how to identify user requirements.
            \item To get around the problem of limited storage on Ceph, we were provided with Camelot~\cite{camelot} as an alternative storage solution. 
            \item We solved the problem of low disk space on Docker containers by specifying Docker files that contained the published image with pre-installed dependencies so that at the time of image pulling, Docker will allocate enough space for the base binary as well as any dependencies. 
            \item We solved the problem of the reduced headcount by re-assigning one of the liaisons for Team-4 over to Team-3. We accounted for the loss of headcount during October by dividing up the work between Team-3 and Team-4 liaisons by putting these teams under the same umbrella, given the similarity in the teams' nature of work.
            \item We got around the problem of missing knowledge by taking onboarding sessions and studying suggesting readings.
            \item We overcame the problem with deploying GitLab onto Kubernetes by deploying GitLab on a dedicated Virtual Machine. Further, the parameters ‘privileged’ and ‘volumes’ in the GitLab runner were updated as below.
            
            \begin{verbatim}
    privileged = true
    volumes = [“/var/run/docker.sock:/var/run/docker.sock”, “/cache”]
            \end{verbatim}
            \item For the containers to detect the GPUs of CS cloud infrastructure which is under the label ‘gpu’, the nodeAffinity constraint is set in the Kubernetes YAML file to the label ‘gpu’ so that the deployments of Team-3 and Team-4 containers are tied to the nodes under the particular label.
            \item To understand and integrate Team INT-2020's reasoner into our system we sought the help of Mr. Mohit Thazhath, from Team INT-2020, and Dr. Prashant Chandrasekar at UMW, to help us understand their design choices.
        \end{enumerate}