%-------------Lessons Learned (a history, allowing you to reflect on your experience, giving an overview of the project effort, including how it was managed)
\chapter{Lessons Learned}
\section{Data Pre-processing}

While developing the project we faced some challenges due to the size of the data and complexity of the tasks involved.

For the supervised learning task to predict web page changes, we used 14 days of data corresponding to \url{cnn.com}.
Although we would get some trends from this limited data, a larger time series dataset spanning multiple web pages is needed to properly evaluate the model and recognize the trends.
But in the case of sitemap change, we require all the original webpages crawled during these 14 days since the idea is to catch the trend of the sitemap change during these hourly crawls. 

We initially tried loading the Parquet files into a Pandas dataframe before extracting the information from them.
Each Parquet file had roughly 4000 payloads.
When trying to load each one sequentially using Pandas, the system continuously ran out of memory.
We fixed this by using the PySpark SQL dataframe instead, that just loaded the dataframe schema in memory and collected the results in memory based on the queries made.
This was much more memory efficient as it did not load each Parquet file in memory completely.
In case of sitemap predictions, saving the payload while building the sitemap was appearing to be an onerous task considering the payloads required a lot of space in memory.
We decided to build a sitemap without saving the payload since it is not a feature in sitemap prediction, thereby decreasing the complexity of building sitemap and its features for prediction. 

The timestamps in the Parquet files were in the \textit{YYYYMMDD} format, but to properly capture the trends in the data we required timestamps up to the second.
We extracted the timestamp values for each snapshot based on values from multiple columns in the Parquet files.
Ideally, there should be a standardized format for storing the timestamp values in the Parquet files such that all the necessary information pertaining to the timestamp is available in a single column.


\section{Constructing Baselines}
To generate the dataset for the supervised learning task for predicting web page changes, we used thresholds based on cosine and overall HTML structure similarities. 
The values of these thresholds were decided by us based on web page differences exhibited on the cnn.com dataset.
Ideally, we should conduct human evaluations to prepare ground truth data and evaluate the threshold values from this data.

The \url{cnn.com} web page payload comprises of only the static skeleton of the homepage.
The articles on the page are loaded dynamically using JavaScript.
To properly label the dataset, we extracted this list of articles from the <script> tag and used it in our feature modelling.
Although this is useful to model the data on \url{cnn.com}, this is a not a generalised technique to extract web page content on the internet.

\section{RL model design}
When designing the RL model, all the essential elements -- environment, agent, action, and reward -- could make vast differences with small changes. 
We should organize the code to be prepared for various changes and version tracks. 
The code should also be written in a good object-oriented manner.
In this way, the model logic will be clearer to understand and easier to modify.
In our current stage, there remain many parts in the model that need to be tested and validated carefully.


Training a deep RL model can take a very long time, depending on the training steps. 
It is useful to use tools such as the Tensor Board to track the training execution.
In many cases, we would get an intuition about the training result from the early stages of training.
Figure \ref{fig:tr1} shows an example of two training processes when we design our model. 
The red line is not working well with the training steps, and we should not waste more time on this training.
By tracking the training process, we can reduce the time for testing the model.

\begin{figure}[!ht]
    \centering
  \includegraphics[width=0.8\textwidth]{figures/TR_1.png}
  \caption{An example that shows two training processes. A higher positive reward score indicates that the model is working towards a potential solution.}
  \label{fig:tr1}
\end{figure}

Considering the evaluation of our RL model, at the current stage, we only look at the rewarding score designed by our logic. 
This evaluation is not sufficient to evaluate the actual performance of the model for our task.
In a typical RL application, an agent usually reaches a deterministic state for each training loop: wins, or loses a game, for example.
Thus the final results of the RL model can be evaluated easily.
In our task, the state of ``winning'' is more complicated, compared with the simple gaming environment.
In this case, more consideration should be taken into account for evaluating the RL model.