\chapter{Future Work}
This section discusses the future tasks with respect to processing the tobacco dataset.

\section{Improve the Text Cleaning Process}
We intend to improve the exisitng text cleaning process. The current process is good at identifying line numbers that are at the beginning of the line but fails to detect line numbers occurring in between sentences.
We would like to improve this process to account for line numbers within the text as well as increase the precision of identifying  garbage characters.

\section{Finalize OCR Method}
We have explored alternate OCR implementation techniques and have received preliminary results.
The current OCR'ed documents have multiple non-decodable characters that require removal.
We are removing such characters using a Python script in the pre-processing stage in order for data to be used for processing.
Hence, we revisited the OCR process for alternative approaches that would result in better quality documents.
A decision needs to be taken with respect to moving forward with an alternate OCR method, and this needs to be finalized and applied to documents.

\section{Test Coverage}
We intent to add more unit tests for the scripts written.
This would help ensure that updates to the scripts do not break the existing functionality.
This would also aid the initiative of building a CI/CD pipeline for the project.

\section{Pipeline for Ingesting New Documents}
We need to focus on automating the ingestion of new tobacco documents as and when they become available.
This would involve coordinating with the INT team to set up a Kafka pipeline and prepare automation scripts to trigger the ingestion of new documents.
We presently have a proof of concept that needs to be extended to build a full-fledged solution.

