\newpage
\section{System Evaluation}
\label{evaluation-section}

\subsection{Performance Evaluation}
To measure the system's performance we perform a stress test for different components of the system.
Such testing requires the collaboration of all the CS5604 teams.
One form of stress test that we considered is to replicate logged sessions at higher speeds.
Through trial and error we can measure the maximum throughput of our system to process user sessions.
To do that, we can perform a stress test with the most recent user sessions and/or archived sessions that represent typical user activity. 
Another form of stress test can be performed when the system is deployed on a new cluster.
For that, we can simulate archived user sessions at typical user load and/or at maximum user load to gauge the capabilities of the underlying system hardware.
After evaluating our system and the resources with respect to time and learning curve, we decided to go ahead with load testing our system which we will be discussing in depth in~\S\ref{stress-testing}.


\subsection{Automatic Failure Recovery}
Ideally, if a component of our information system fails we will be able to recover from the situation automatically. We can recover from application crashes that occur within containers provided that persistent data for the application is located on Ceph (or another persistent storage).
In this case,
the  application can be restarted without significant problems as long as no critical data was held in memory or in temporary files at the time of  failure.

One of the core features of Kubernetes is that it’s designed to maintain the desired state defined by operators and app admins. A series of reconciliation loops constantly works to find the optimal path from the current state to the desired state for all components of the cluster \cite{kube-auto_recovery}.

From the system's perspective, handling a container failure is easy because several simple loops are always running.
To handle a single failure of a highly available app, each loop
\begin{itemize}

\item ensures that the correct number of containers is running and creates any that aren't; 
\item ensures that all unassigned containers are assigned to nodes that are healthy;
\item ensures that each node is running containers assigned to it;
\item ensures that traffic is load balanced only to healthy back-end containers.
\end{itemize}


\subsection{Failures That Require Manual Recovery}
We cannot automatically recover from failures that occur due to systemic problems.  The following are examples of systemic issues that may lead to failure:

\begin{itemize}
    \item Problems with Pushed Updates: Servers within the cluster may fail as a result of activity from system updates.  System updates can disrupt the system configuration.  They can also break a library dependency chain that affects whether an installed application can run. 
    \item Consumed System Resources:  Cluster activity that consumes too many resources may cause cascading failures of uncoupled applications and/or containers.  If the system resources are not freed, crashed applications and containers will not have the resources to be restarted.
\end{itemize}

Since systematic failures require manual intervention, the most that can be done is that the system may notify a system administrator, if the failure is such that the system is left partially running with the capability of detecting failures within its subsystems.

\subsection{Stress Testing} \label{stress-testing}

As discussed before, Stress Testing is a type of performance testing which validates the highest limit of a system; which could be a computer, a device, a program or a network -- with heavy load.

Stress testing tries to break the system under test by taking away the resources from it.
Hence, it is also called as \textit{negative} testing.
The main purpose of doing a stress test is to see that the system fails and recovers gracefully -- a quality which is known as \textit{recoverability}.

A stress test helps verify the stability and reliability of the system.
It makes sure that the system does not break under extreme conditions by forcing those conditions and in turn preparing the system for the worst.
A stress test prepares the system to be robust, available at all times, and handle the errors.

Load testing is a type of stress testing which checks how systems function under a large number of concurrent simulated users accessing the system over a period of time. This is used to verify how systems handle heavy load volumes.

Load testing can be used to determine the number of users the system can handle. Load testing tools allow to create different user scenarios which is helpful to test different parts of the system, like the login page.
 
\subsubsection{ Stress Testing Tools}
For our project, we have explored two stress testing tools: Locust\cite{locust} and JMeter\cite{jmeter}. See Table~\ref{stress-testing-comparison-table} for a comparison of the tools. Because of Locust's ease of setup and its support for Python, we decided to adopt it for load testing.


\begin{center}
\begin{longtable}{|p{.25\textwidth}|p{.32\textwidth}|p{.32\textwidth}|}
\caption{Comparison of stress testing tools} \label{stress-testing-comparison-table} \\

\hline \multicolumn{1}{|c|}{\textbf{Feature }} & \multicolumn{1}{c|}{\textbf{JMeter}} & \multicolumn{1}{c|}{\textbf{Locust}} \\ \hline 
\endfirsthead

\multicolumn{3}{c}%
{{\tablename\ \thetable{} -- continued from previous page}} \\
\hline \multicolumn{1}{|c|}{\textbf{Feature }} & \multicolumn{1}{c|}{\textbf{JMeter}} & \multicolumn{1}{c|}{\textbf{Locust}} \\ \hline 
\endhead

\hline \multicolumn{3}{|r|}{{Continued on next page}} \\ \hline
\endfoot

\hline \hline
\endlastfoot

\textbf{Scripting}
&
Supports GUI and Scripting &
Supports Python coding\\
\hline
\textbf{Best For} &
Performance testing of Web Applications &
It provides a functionality to check the simultaneous number the system can handle\\
\hline
\textbf{Capability} &
It works for Web applications, Servers, Group of servers, and network &
It can perform load testing on multiple distributed machines\\
\hline 
\textbf{Pricing} & 
Free & 
Free \\
\hline
\textbf{Open-Source?}& 
Yes& 
Yes \\
\hline
\textbf{Easy to use with VCS?} &
No & 
Yes \\
\hline
\textbf{Concurrency} &
JMeter is thread-based so it requires a separated thread to simulate a user. JMeter relies more on the machine’s performance, and it is more expensive when simulating the same amount of users.
 & 
Locust is based on co-routine, and uses the async approach. It is possible to simulate large amount of users in one machine. \\
\hline
\textbf{Support for distributed testing} &
Yes & 
Yes \\
\hline

\end{longtable}
\end{center}


\subsubsection{Locust Stress Test Results}

We used Locust to implement load testing. The general idea behind load testing is to see the number of maximum requests the front-end website could support and further implement load balancing techniques to manage the load.

The steps that we followed were:

\begin{enumerate}

    \item Locust provides use of 2 parameters -- number of users to be simulated and the \textit{Hatch Rate} -- which is the number of users spawned per second.
    In our experiment, we keep both the number of simulated users and the hatch rate to be the same number. Figure~\ref{fig:load-test-1} shows the result of load testing when the website is simulated with 100 users with a hatch rate of 100 users per second.
    
    \begin{figure}[ht!]
        \centering
        \includegraphics[width=0.90\textwidth]{figures/100u1c.PNG}
        \caption{Load testing of front-end website by simulating 100 users with 100 users spawned per second on one container.}
        \label{fig:load-test-1}
    \end{figure}
    
    \item If the number of fail requests is zero, increase the number of simulated users. Since the front-end website was able to handle the initial 100 users as seen in Figure \ref{fig:load-test-1}, we increased the number until the website failed to return some of the responses. We found that the website started sending fail requests at 500.
    
    \begin{figure}[ht!]
        \centering
        \includegraphics[width=0.90\textwidth]{figures/500u1c.PNG}
        \caption{Load testing of front-end website by simulating 500 users with 500 users spawned per second on one container.}
        \label{fig:load-test-2} 
    \end{figure}
    
    \item We then increased the number of container instances to two and carried on the load test with the last simulated number of users, i.e., 500.
    
    \begin{figure}[ht!]
        \centering
        \includegraphics[width=0.90\textwidth]{figures/500u2c.PNG}
        \caption{Load testing of front-end website by simulating 500 users with 500 users spawned per second on two containers.}
        \label{fig:load-test-3}
    \end{figure}
    
    \item As seen from Figure \ref{fig:load-test-3}, the number of fail requests is zero which means that the container was successfully able to balance the number of requests (500) amongst the two containers.
    
    \item To test the limits of our system, we decided to further increase the number of users to 1000 and the number of fail requests still remained at zero. Thus, we can safely say that the front-end website can successfully handle requests from 1000 users, spawned per second if the number of containers is increased to two to balance the load.
    
    \begin{figure}[ht!]
        \centering
        \includegraphics[width=0.90\textwidth]{figures/500u2c.PNG}
        \caption{Load testing of front-end website by simulating 1000 users with 1000 users spawned per second on two containers.}
        \label{fig:load-test-3}
    \end{figure}
    
\end{enumerate}


