\chapter{Workflow Automation} 
\label{Airflow_interactions}

% \section{Initial Workflow Automation Prototype}
% \subsection{Prototype Design}
% We have decided to create a simple, low-fidelity implementation of the overall workflow automation. Apache Airflow allows running a Directed Acyclic Graph (DAG) in which, each node in the graph is an operator that performs an action. Each DAG in Airflow is essentially a workflow composed of operators that are run after one another. Every workflow that is defined is essentially a Python script, which can be run using the Python binary. An example of an operator is the `BashOperator', which executes a `bash' command on the machine that the script is running. 

% We are working with the concepts of goals, services, knowledge graph, registry and optimizer as defined in Section~\ref{section:airflowOverallDesign} above. 

% We will be working with a singular goal, to `Detect objects within an image'. We will be creating a DAG that is composed of querying the registry (fetching the details about the goals and services), then using these details to spin up a Docker container on the CS Cloud server that runs Object Detection on a given file (the service) and finally storing the output of the service in the JSON format.

% Figure~\ref{fig:whole_pipeline} shows the overall workflow. In Figure~\ref{fig:detailed_pipeline}, we show the technical overview of each component, including how data flows among them. 
% \begin{figure}[h!]
% 	\centering
% 	\includegraphics[scale = 0.55]{img/detailed_pipeline.png}
% 	\caption{Workflow Control Plane}
% 	\label{fig:detailed_pipeline}
% \end{figure}

% The light blue box represents the Postgres database, which contains some ETD metadata and our three metadata tables used for workflow functionality. When a workflow is triggered by end user, an SQL query is sent to Postgres, which:

% \begin{enumerate}
%     \item Gets all rows that matches the workflow\_id
%     \item Uses the foreign keys of service\_id, goal\_id\_input, and goal\_id\_output to query service table and goal table
%     \item Fetches image\_url(s) and file\_location(s), respectively
% \end{enumerate}

% After we have fetched all the data via the `PostgresOperator', the Reasoner uses the query results to generate DAG(s) intelligently, that is used to execute inside Airflow. 

