%-------------Developer's Manual (so someone can pick up where you left off, easily)
\chapter{Developer's Manual}

\section{Project Architecture and Inventory}
Our source code is hosted in a GitHub repository at \url{https://git.cs.vt.edu/xw0078/cs-6604-webarchive}.

\section{File Inventory}
\begin{enumerate}
    \item dataExtract.ipynb is the container notebook for the script to extract the payload and timestamp data for the \textit{cnn.com} homepage from the Parquet files.
    The root directory and the URL for which the data is to be extracted are given as input.
    The script outputs the timestamp and payload information for the corresponding URL from all Parquet files found in the root directory by recursively traversing all folders.
    The result is then stored into a .pkl file \textit{archiveData.pkl}
    \item classifyArchives.ipynb is the container notebook for training the supervised learning models and making predictions.
    The \textit{archiveData.pkl} file generated by the previous script is provided as input and the results are the predictions made by the different supervised learning models along with the evaluations of the predictions.
    
    \item ParquetfileProcessor.ipynb is a container notebook for the script to extract the timestamp and URL data for \textit{hourly crawls} from the Parquet files.
    
    \item  Trie.py is a site map creating package.
    
    \item ResponseParser.py is a utility file created to parse the payload saved in Parquet files. It extracts HTML data from the HTTP response saved in the payload.
    
    \item NewCNN\_1hour2levelUpdated.py is a main package created which calls Trie.py and ResponseParser.py to build the sitemap, and contains functionalities implemented as explained in the previous section related to the sitemap.  
    
    \item CNN\_1hour2levelMainNew.py is the main file explaining the use of packages, and generates the evaluation result. 
    
    
    
\end{enumerate}

\section{Method Descriptions}
\begin{enumerate}
    \item tag\_visible method returns all the HTML tags that contribute towards visible text content on the web page.
    The method takes a BeautifulSoup object element for a web page as input and returns \textit{True} or \textit{False} based on whether that particular DOM element contributes to the visible content on the web page or not.
    \item extract\_script method returns whether the BeautifulSoup object element for a web page is the \textit{script} element or not.
    The method takes a BeautifulSoup object element for a web page as input and returns \textit{True} or \textit{False} based on whether that particular DOM element is the script element or not.
    \item  script\_from\_html method extracts the entire content within the <script> tag for an HTML page.
    The method takes an HTML page as input and returns the entire content inside the <script> tag as a single string.
    \item text\_from\_html method extracts the visible text content on a web page.
    The method takes an HTML page as input and returns the complete visible text on the web page as a single string.
    \item filter\_text method filters the input text by removing all the stopwords, punctuation, and white spaces more than one space from it.
    The method takes a string as input and returns the filtered text as its output.
    \item extract\_articles method extracts the \textit{articleList} object from within the <script> tag of an HTML page.
    The method takes the content within the <script> tag as a string as input and returns the contents of the \textit{articleList} object as a string.
    \item classifyProphet method uses the Facebook Prophet tool to model the time-series data from the dataset and make predictions for the crawling schedule for a specified number of hours in the future.
    The method takes the data from the \textit{archiveData.pkl} file as input, transforms the data into a suitable training data vs. label format for Prophet, and then returns the predictions as output.
    \item classiyWindowSVM\_RF method uses scikit-learn's SVM and Random Forest algorithms to model the time-series data from the dataset using a sliding window of a specified size and make predictions for the crawling schedule for a specified number of hours in the future.
    The method takes the data from the \textit{archiveData.pkl} file as input, transforms the data into suitable training data vs. label format for the algorithms, splits the data into training and testing data, and then returns the predictions and their evaluations as output.
    
    \item buildDataDayWise method takes foldername as input and builds the dataset from the Parquet files.
    This method was designed to experiment with Parquet files and build some test data.
    To build a dataset from a whole dataset of Parquet file, use ParquetfileProcessor.ipynb.
    It generates `cnnNodeData.pkl' which will be processed to generate a sitemap. 
    
    \item cleanDataSet method is used to clean and convert cnnNodeData.pkl and make it usable to build a sitemap.  
    
    \item makingSitemapTree method is used to build the sitemap from the dataset obtained from `cnnNodeData.pkl'.
    It is also used to build the training dataset used for the classification and regression models.
    It returns `changeNodesMatrix' containing training data. 
    
    \item results method returns the evaluation done by training and predicting using SVM and Random Forest models. 
    
\end{enumerate}
