%-------------Users' Manual (how to run your software)
\chapter{User Manual}

\section{ArchiveOrgCollectionScraper}
Gitlab repo: \newline
\url{https://git.cs.vt.edu/xw0078/archiveorgcollectionscraper}

ArchiveOrgCollectionScraper is a web scraper that can fetch all the downloadable links of collection data from the collection page.
An example of usage can be found in the ``GetArchiveOrgDownloadLinks.ipynb'' Jupyter Notebook.
The code takes the URL of the target collection.
Then it will generate a text file that contains all the downloadable links to WARC and CDX files in the collection.
The collection can be downloaded through Wget:
\begin{verbatim}
wget -i urls.txt
\end{verbatim}

Note:
You need to have proper user authentication to get access to non-public collections.

\section{Zeppelin}
Official website: \newline
\url{https://zeppelin.apache.org/}

Zeppelin is a notebook environment (similar to Jupyter Notebook) that supports multiple languages and frameworks.
In this project, we primarily take advantage of the Spark integration and Scala programming language.
We use Zeppelin to work with the Web2Warc library and Archive Unleashed Toolkit. 

\section{Web2Warc}
Github repo: \newline
\url{https://github.com/helgeho/Web2Warc}

Web2Warc is a simple and easy crawler that can produce WARC/CDX output.
We modified Web2Warc to collect data that are compatible with AUT.
We run Web2Warc under the Zeppelin environment.
Please check the example code under ExampleNotebooks in our project repo:
\url{https://git.cs.vt.edu/xw0078/cs-6604-webarchive}

\section{Archive Unleashed Toolkit (AUT)}
Github repo: \newline
\url{https://github.com/archivesunleashed/aut/}

AUT is a web archive big data processing tool under the Spark environment.
We modified AUT to support extracting more index information as we need it.
We run AUT data conversion under the Zeppelin environment.
Please check the example code under ExampleNotebooks in our project repo:
\url{https://git.cs.vt.edu/xw0078/cs-6604-webarchive}

\section{Heritrix}
Github repo: \newline
\url{https://github.com/internetarchive/heritrix3}

Heritrix is an open source web crawler that is designed by the Internet Archive.
A core concept about Heritrix is that it is designed to work with the "robots.txt" exclusion directives and META robots tags.
In other words, the crawling jobs fully respect the rules that are defined by the web sites creator. 



\section{HTML-Similarity}
Github repo: \newline
\url{https://github.com/matiskay/html-similarity}

This package provides a set of functions to measure the similarity between web pages.
All the similarity metrics take values between 0 and 1.

We use the package to compute the style-similarity, structural-similarity, and the joint similarity, to compare the webpages.


\section{TextDistance}
Github repo: \newline
\url{https://github.com/life4/textdistance}

TextDistance is a Python library that compares two or more text sequences based on more than 30 comparison techniques.
We use this library to compare the plaintext extracted from the two web pages to be compared and then perform comparisons based on Jaccard Similarity, Cosine Similarity, Edit Distance, and Sorensen-Dice Similarity.


\section{BeautifulSoup}
Github repo: \newline
\url{https://github.com/waylan/beautifulsoup}

BeautifulSoup is a Python library that extracts data out of HTML and XML files and provides methods to navigate, search, and modify the parse tree.

\section{PySpark}
Package Docs: \url{https://spark.apache.org/docs/2.2.0/api/python/pyspark.html}

PySpark is the Python API for Spark that lets us harness the simplicity of Python and the power of Apache Spark in order to tame Big Data. To work with the massive data, we configured the session with following settings:

\begin{minted}{bash}
SparkSession.builder 
    .master("local[*]")
    .config("spark.executor.memory", "70g")
    .config("spark.driver.memory", "50g")
    .config("spark.memory.offHeap.enabled", "true")
    .config("spark.memory.offHeap.size", "14g")
    .appName("sampleCodeForReference")
    .config("spark.driver.cores", "12")
    .config("enableVectorizedReader","false")
\end{minted}

\section{Prophet}
Official Documentation: \url{https://facebook.github.io/prophet/}

Prophet is a procedure for forecasting time series data based on an additive model where non-linear trends are fit with yearly, weekly, and daily seasonality, plus holiday effects. Prophet is robust to missing data and shifts in the trend, and typically handles outliers well. It is open source software released by Facebook’s Core Data Science team. It is available for download on CRAN and PyPI.

\section{scikit-learn}
Official Documentation: \url{https://scikit-learn.org/stable/}

scikit-learn provides simple and efficient tools for predictive data analysis. The open source library is built on NumPy, SciPy, and matplotlib.

\section{OpenAI Gym}
Official Documentation:
\url{https://gym.openai.com/docs/}

Gym is a toolkit for developing reinforcement learning problems.
Gym is compatible with TensorFlow or Theano for deep reinforcement learning architecture.
Gym also provides the environment super class where you can design your problem and your own model, which can be later trained with different reinforcement learning algorithms.

\section{Stable Baselines}
Official Documentation:
\url{https://stable-baselines.readthedocs.io/en/master/}

Stable Baselines is a set of reinforcement learning algorithms based on OpenAI's Baselines.
Through Stable Baselines, you can easily apply the stable training policy algorithms to the OpenAI's Gym envrionment.
