% NSF proposal generation template style file.
% based on latex stylefiles written by Stefan Llewellyn Smith and
% Sarah Gille, with contributions from other collaborators.
%

\documentclass{proposalnsf}
\usepackage{comment}
\usepackage{float}
\usepackage{url}
%\usepackage[square,sort,comma,numbers]{natbib}
\usepackage{multirow}
\usepackage{graphicx}
\graphicspath{ {images/} }

% See this file for a set of pre-defined journal abbreviations
%\input{journal-abbreviations.tex} 

\newcommand{\degrees}{$\!\!$\char23$\!$}
\renewcommand{\refname}{\centerline{References cited}}

% This handles hanging indents for publications
\def\rrr#1\\{\par
\medskip\hbox{\vbox{\parindent=2em\hsize=6.12in
\hangindent=4em\hangafter=1#1}}}

\def\baselinestretch{1}

\begin{document}

% Title Page
\begin{center}
\thispagestyle{empty}
\parskip=1pt%
\vspace*{5\parskip}%

{\LARGE \textbf{CS 6604 Digital Libraries}}

{\Large Final Term Project Report\\ETDseer: Concept Paper}

\vspace{2in}
{\large
Team Members:

Yufeng Ma

Tingting Jiang

Chandani Shrestha
}

\vspace{1.5in}
{\large
5/3/2017

Virginia Tech

Blacksburg, VA 24061
}

\vspace{1.5in}
{\large
Instructor: Professor Edward A. Fox\\
Copyright 2017 Yufeng Ma, Tingting Jiang, Chandani Shrestha, and Edward A. Fox
}
\end{center}



\newpage
\pagenumbering{arabic}
\renewcommand{\thepage} {B--\arabic{page}}

\begin{center}
{\Large{\bf Draft Project Summary}}\\*[3mm]

\end{center}
ETDseer (electronic thesis and dissertation digital library connected with SeerSuite) will build on 15 years of collaboration between teams at Virginia Tech (VT) and Penn State University (PSU), which both have been leaders in the worldwide digital library community. VT helped launch the national and international efforts for ETDs more than 20 years ago, which has been led by the Networked Digital Library of Theses and Dissertations (NDLTD, directed by PI Fox); its Union Catalog has increased to 5 million records. PSU hosts CiteSeerX, which co-PI Giles launched almost 20 years ago, and which is connected with a wide variety of research results under the SeerSuite family.\\
       
\noindent ETDs, typically in PDF, are a largely untapped international resource. Digital libraries with advanced services can effectively address the broad needs to discover and utilize ETDs of interest. Our research will leverage SeerSuite methods that have been applied mostly to short documents, plus a variety of exploratory studies at VT, and will yield a “web of graduate research”, rich knowledge bases, and a digital library with effective interfaces. References will be analyzed and converted to canonical forms, figures and tables will be recognized and re-represented for flexible searching, small sections (acknowledgments, biographical sketches) will be mined, and aids for researchers will be built, especially from literature reviews and discussions of future work. Entity recognition and disambiguation will facilitate flexible use of a large graph of linked open data. \\

\noindent
{\bf Keywords:} CiteSeerX, deep learning, domain independent digital library (DL), information extraction (IE), information retrieval (IR), machine learning (ML), natural language processing (NLP), NDLTD\\


\noindent
{\bf Intellectual Merit}

\noindent This research will lead to fundamental contributions in the DL, IE, IR, ML, and Web crawling/archiving areas, as well as contributions in HCI and AI. For example, we will develop and apply new methods to find all references (from footnotes, ends of chapters, and the bibliography), extract unambiguous values in spite of thousands of different reference styles employed, and build end-user services atop an accurate citation graph including distinct works as well as entities for graduate students, advisors, and other scholars.  
\ \\

\noindent
{\bf Broader Impacts}

\noindent This research will tackle many difficult challenges faced by the AI, DL, IE, IR, ML, and NLP communities, at scale, advancing big data efforts in these domains. It will have broad impact on national research and education by opening up ETD resources for broad and helpful use.\\

%\renewcommand{\thepage} {B--\arabic{page}}

\newpage

% reset page numbering to 1.  This is helpful, since the text can only
% be 15 pages, and reviewers will want to believe we've kept within
% those limits

\pagenumbering{arabic}
\renewcommand{\thepage} {D--\arabic{page}}

\newpage

\noindent{\Large \bf PROJECT DESCRIPTION}\\

\noindent Using advanced IE and NLP techniques, coupled with machine learning and information retrieval methods, we will research ETDseer, a tailored DL for large English ETD collections, which would offer special services: review the literature, identify hypotheses, list research questions, explain approaches, describe methods, summarize results, discuss findings, present conclusions, and provide insights on open problems.  We can provide these services by processing references and citations, as well as information extracted from chapters, sections/subsections, tables, and figures.  Though ETD collections cover many disciplines, a suitable domain independent digital library can be prototyped now, using advanced natural language processing and information extraction techniques, coupled with machine learning and information retrieval methods. The resulting system would enable stakeholders to engage in advanced scenarios that go well beyond conventional searching and browsing. Section 1 summarizes the statement of need for this research with goals and objectives. Section 2 covers results from prior NSF support. Section 3 describes our approach, including connections with real life scenarios for the scholarly community, methodologies applied, and system components. Section 4 addresses project management and timing. Remaining sections give Related Work, Expected Significance, and Broader Impacts.
\section{Introduction}

{\bf Statement of Need }\\
ETDs can be a valuable aid to learning and scholarship. They are a largely unused international resource.Tapping this resource requires research first on effective ways to discover ETDs of interest, and then support for better using identified works to aid a highly diverse community of researchers and educators. At present, there is no good way to address these needs for large heterogeneous collections of long documents like ETDs, typically in PDF, but also available in other formats. 
 
% [2, 6, 7, 11, 12, 15]
Though there are widely available DLs that support indexing, searching, and browsing of short articles or papers,  book length objects typically are searched using limited and often misleading metadata records, with minimal help from classification efforts (especially for multidisciplinary works). While full-text search is used to extend faceted searching, it is not effective when processing large files in natural language. A preferable approach could be applying Natural Language Processing (NLP), e.g., information extraction (IE). Currently available resources  like Google Scholar and CiteSeerX \cite{caragea2014citeseer, giles2006pkdd, giles1998citeseer, li2006citeseerx, ororbia2015www,teregowda2010hotcloud}, that extract, analyze, and link references in short publications, do provide additional capabilities, but they do not work well with  ETDs (due to length, complexity, and domain variations). 

In this research, we address the need for a tailored DL focusing on ETDs based on advanced concepts from information retrieval (IR), Web crawling/archiving, information extraction, machine learning (ML), and deep learning.\\\\
{\bf Goals }
\begin{enumerate}  
\item Aid effective dissemination of valuable information from years of collections of ETDs, to the scholarly community, including a whole spectrum of researchers from an inexperienced student researcher to a well-established faculty member.
\item Prototype a domain independent digital library that provides advanced services to a vast collection of book-length long documents.				
\item Advance the state-of-the-art in DL and NLP with regard to long document retrieval and utilization, for both individual documents and groups of documents.
\item Advance the state-of-the-art in DL and NLP with regard to analyzing long documents; processing tables, figures, and references; extracting citations and key concepts from chapters, sections, and subsections (including special parts of ETDs like literature reviews, acknowledgements, and bio-sketches); and expanding the capabilities of DL beyond indexing, searching, and browsing.
\item Advance the state-of-the-art in information integration, synthesis, and generation, including when generating lists (e.g., of references, literature reviews, approaches, methods, and findings).
\end{enumerate}
{\bf Objectives	}
\begin{enumerate}
\item Develop methods to extract semantic content and surrounding context from long documents such as research questions and their corresponding hypotheses.	
\item Aid stakeholders through interfaces tailored for exploration, to discover interesting content, review literature, and evaluate references as well as ETDs.
\item Assist stakeholders as they study and research with ETDs by intelligently providing advanced services through stating hypotheses, listing research questions, explaining approach, describing methods, summarizing results, discussing findings, drawing conclusions, and providing insights about open problems.
\end{enumerate}

% \section{Results from Prior NSF Support}

% This section is intentionally left blank, as we do not have access to the  necessary relevant information.

\section{Approach}
\subsection{Advanced services in discovering and utilizing ETDs} \label{advanced_services}
ETDseer is aiming to improve the utilization of valuable ETD resources by providing specialized services to a spectrum of scholarly community users in a fine-grained manner. Based on the specific research and study needs of different stakeholders, we identify some of the representative real-life scenarios and describe them as follows.  Table \ref{table:1} summarizes the service requirements generated from the envisioned scenarios and their corresponding anticipated outputs. \\

\noindent\textit{Scenario 1. Identify a reading list}\\
NS, a new graduate student who enters the research arena with vague interests, needs to study published works to gain understanding of key research questions. NS can search and find a suitable set of ETDs. Results are given in a tabular form, indicating the quality of the selected works (based on citation counts and other criteria), and showing clusters of related research questions. After NS reviews this data and selects portions of particular interest, the DL extends its analysis. Relevant references are extracted, converted to a canonical form, and presented as a reading list. Additionally, the figures, tables, and equations of the selected ETDs are summarized and presented as a supplement. Optionally, social/bibliographic networks, and other helpful visualizations, are provided.\\

\noindent\textit{Scenario 2. Collect approaches to a research problem}\\
SR, a student researcher, has come across a challenging research problem and is interested in the discussions in journal and conference papers he has reviewed so far, which indicate that three different methods have been employed, but without details and comparative studies. ETDseer helps SR identify the ETDs that are related to each of the methods as well as corresponding involved datasets. Then ETDseer creates a site that has descriptions of the source code, as well as the training and testing data associated with each method. A well-formatted summarization table is generated.\\

\noindent\textit{Scenario 3. Create award-winning paper template} \\
Student researcher, SR, with an almost completed ETD, wants to win the best paper award at a prestigious conference. Based on deep learning analysis of other award winning papers in that area, and their corresponding ETDs, an outline of a paper derived from the ETD is constructed for SR, including tables, figures, equations, and references.\\

\noindent\textit{Scenario 4. Identify Collaborators} \\
Faculty researcher, FR, who has identified a specific research problem that necessitates collaboration, seeks a list of different approaches used to tackle this problem as well as a timeline view of the evolution of associated research studies. FR describes the problem, and receives a list of selected ETDs. Documents listed in the related work sections, proposed approaches/solutions in the middle of ETDs, and open problems mentioned in the conclusion or the future work sections, are identified. A summary table categorizing the details is presented.

FR studies the summary provided and provides feedback about preferences and priorities. The DL prepares a tailored summary, and a shortlist of potential collaborators, along with their contact information and brief bio-sketches, that is supplemented with notes on how they might complement FR's background.\\

\noindent\textit{Scenario 5. ETD quality evaluation} \\
University administrator, UA, would like a rough assessment of the quality of an ETD submitted from one of the local departments. ETDseer provides a table related to the selected ETD that contains: counts of elements (references, equations, figures, and tables), a histogram of citations to key prior works of the author, degree of match between proposed approach and research problem, and a summary of experimental results. \\

\noindent\textit{Scenario 6. Prepare course syllabus and lecture slides} \\
Graduate instructor, GI, is teaching a new advanced course. GI prepares course related materials on a specific research topic and receives a list of related ETDs in ETDseer.  Based on the ETDs of interest, ETDseer uses clustering, topic analysis, and summarization methods to construct a draft course syllabus. Included in the syllabus is a hierarchical topical outline, with summaries for each entry, linked with a suitable reading list, which includes the ETDs as well as the most important other open source publications that were discussed in those ETDs. 

GI also wants to focus on a specific problem and discuss the most promising solutions. GI gives ETDseer a description of the problem, and receives related ETDs that are neatly categorized in terms of their various problem statements, research questions, and provided solutions. Furthermore, ETDseer creates drafts for class including properly sequenced slides and lecture notes, with helpful examples, illustrations, and summary tables.\\

\noindent\textit{Scenario 7. Organize a conference} \\
A conference organizer, CO, wants to identify a list of Technical Program Committee members for a conference. CO gives ETDseer the list of topics from the announcement. ETDseer searches through the related ETDs and returns a list of advisor research faculty names that appear in the metadata for the ETDs, along with names of ETD authors from at least five years earlier who have highly cited ETDs. To provide CO with more detailed information, ETDseer generates a table that ranks those advisors based on h-index, the weight of the ETDs in each advisor's research group, citation counts, etc.
CO also wants to identify the potential participants of the conference. CO queries ETDseer with a list of keywords, related to the theme of the conference. ETDseer presents a subgraph from the relevant portion of the ETD-derived citation graph, extracts authors of those works, and returns their names and contact information as a CSV file, which can be used to send a general conference announcement for submissions and/or participation.\\

\noindent\textit{Scenario 8. Manage a journal}\\
A journal editor, JE, seeks to identify peer reviewers for a journal paper submission. JE queries ETDseer using keywords from the submission. ETDseer responds with several author names, indicating research interest closely related to the submission. This would be based on their published ETDs and their recent publications that can be extracted by ETDseer.
JE also needs to check if a follow up paper submission has at least 30\% original content relative to previous publications. JE queries ETDseer with the author names for an originality check. ETDseer identifies previous publications belonging to the authors, and uses a cloud service to return the estimated percentage of new content/work in the submitted paper.

\begin{table}[!htbp]
\centering
\caption{ETDseer Scenario Table}
\label{table:1}
\begin{tabular}{||l | l | l||} 
 \hline
 \textbf{Stakeholder Group(s)} & \textbf{Requirements} & \textbf{Expected Outputs}  \\ 
 \hline\hline
 \multirow{2}*{\textbf{Cross}} & Faceted Browsing &  Categorized exploration of ETDs\\\cline{2-3}
 & Filtered Searching & Metadata-based discovery of ETDs  \\\cline{2-3}
\multirow{2}*{\textbf{Cutting}} & Summarization & Synthesis of search results \\\cline{2-3}
               & Visualization & Linking of related content\\
 \hline
  & Aspect-specific&  Specific ETDs, e.g., within a date \\
 & access & range or with an advisor name\\\cline{2-3}
 \multirow{2}*{\textbf{Student}}& Match research & Desired ETDs with quality scores \\
    & question interests & Research questions/hypotheses highlighted \\\cline{2-3}
             & \multirow{2}*{Reference} & Related ETDs/books/articles/papers \\
                    & & Tabular/Canonical representations \\
                    & \multirow{2}*{extraction} & Downloadable package of related work \\
                    & & Lists of journals and conferences \\\cline{2-3}
\multirow{2}*{\textbf{Researcher}}    & \multirow{2}*{ETD analysis,} & ETD content summarizations \\
    &  & Figures, tables, and equations \\
    & \multirow{3}* {Generation} & Key sections and list of related problems\\
    &  & Visualizations (social/bibliometrics networks) \\
    & {of study aids} & Timeline overview of evolutionary work \\\cline{2-3}
    & Linking of problems & Different methods for a problem \\
    & \multirow{2}*{with methods} & A site with detailed resources \\
    & & An award winning paper (outline/draft)\\
 \hline
 \textbf{Faculty}  & Research problem & Synthesis of related ETDs\\
 \multirow{2}*{\textbf{Researcher}} & \multirow{2}*{exploration aid} & Proposed approaches and solutions \\
 & & Future works summarization \\
 \hline
 \multirow{2}*{\textbf{Graduate}} & Advanced topics & Slides cover research question/ problems\\
 & lecture preparation & Synthesis of provided potential solutions \\\cline{2-3}
 \multirow{2}*{\textbf{Instructor}} & Graduate course & Draft with a hierarchical topical outline\\
 & syllabus formulation & Link to each topical entry with a reading list\\
 \hline
 \multirow{2}*{\textbf{Conference}} & TPC member & List of advisor research faculty names \\
 & identification & Ranking table of advisors\\\cline{2-3}
  \multirow{2}*{\textbf{Organizer}} & Potential participants & Subgraph of the ETD-derived citation graph \\
 & identification & CSV file of author names, contact info.\\
 \hline
 \multirow{2}*{\textbf{Journal}} & Peer-reviewer  & \multirow{2}*{Research interest-based reviewer list} \\
 & identification & \\\cline{2-3}
 \multirow{2}*{\textbf{Editor}} & Content originality &  Previous publications of the authors\\
 & check & Estimated percentage of new content/work\\
 \hline
\end{tabular}
\end{table}

\subsection{System Design} \label{system_design}
We begin our discussion by describing the architectural design of ETDseer that extends CiteSeerX \cite{caragea2014citeseer} to provide advanced services on the ETD collections that enhance learning and research. An outline of the ETDseer architecture that is based on the 5S framework \cite{gonçalves20045S} is shown in Figure~\ref{fig:architecture}. 
\begin{figure}[ht]
\centering
\includegraphics[width=120mm]{ETD_diagram}
\caption{ETDseer System Architecture}
\label{fig:architecture}
\end{figure}
\subsubsection{Data Source}
ETDseer will be based on ETD submissions to NDLTD \cite{oclc_ndltd, fox1997nrin}. The NDLTD initiative, which was started in the 90s, seeks to expand electronic publication of student research, and make ETDs accessible from around the world. The Union Catalog consists of metadata records for ETDs from contributing institutions and universities. As of March 2010, the Union Catalog consisted of metadata records for 820,000 ETDs in various languages. By 2013 the number exceeded three million and by April 2017, five million. It contains ETDs in at least 12 languages. It is the single largest cumulative source of information on ETDs available on the Internet. Figure \ref{fig:NDLTD_stat} shows the top 12 NDLTD participating members ranking by the number of ETD submissions as of this writing.  \\
\begin{figure}[ht]
\centering
\includegraphics[width=120mm]{NDLTD_stat}
\caption{Top 12 NDLTD participating institutions and universities}
\label{fig:NDLTD_stat}
\end{figure}

\noindent These book-sized ETDs are global scholarship assets, which often have higher value than shorter scholarly  publications (e.g., conference and journal papers) in terms of rich information content and reproducibility of science based on detailed descriptions of research work. Easier access and practical services enabling full usage of this vast collection of rich documents will be of invaluable aid to the scholarly community.

\subsubsection{Existing ETD-related DL Technologies}

ETDseer will hold the ability to access, analyze, and synthesize from long ETD documents, with many more advanced services than are available in DLs like NDLTD \cite{oclc_ndltd}. NDLTD provides basic functionalities that support collecting, disseminating, and searching/browsing of ETDs. In particular, it provides filtered searches with customizable fields like subject, publication date, and language as well as faceted browsing with language and keyword  predefined as the two facet fields. Figure \ref{fig:NDLTD_facets} shows the filtered/faceted navigation capabilities in NDLTD. As we can see, it misses quite a few fields such as discipline, research questions/problems, and citations that are crucial to the ETD content space. \\
\begin{figure}[ht]
\centering
\includegraphics[width=150mm]{NDLTD_facets}
\caption{Filtered and faceted navigation in NDLTD}
\label{fig:NDLTD_facets}
\end{figure}

On the other hand, a closely related DL, CiteSeerX \cite{li2006citeseerx}, does a much better job in terms of analyzing and presenting the scholarly information from large collections of research works, with features like automatic metadata extraction, citation indexing, and reference linking. However, CiteSeerX deals only with short length documents, thus excluding its use of many features in long ETD documents.

A proper integration of these two DLs, where the datasets (metadata and long documents) are mostly from NDLTD, combined with the metadata and technical features from CiteSeerX, will guide the initial development of ETDseer.  ETDseer will leverage most of the DL tools that CiteSeerX has provided. Nevertheless a simple integration of features is neither feasible nor sufficient with ETDs. Although CiteSeerX can work as a technological foundation for building ETDseer, the target documents that CiteSeerX serves are short documents. To process and analyze much longer documents in ETDseer and to achieve similar goals, heuristic approaches such as regular expressions and knowledge rule-based approaches adopted by CiteSeerX will not be sufficient. Furthermore, unlike CiteSeerX, most of the functional requirements in ETDseer, such as extracting research questions and hypotheses, cannot be addressed via keyword and fulltext only searching and browsing. 

CiteSeerX handles documents from several specific scientific disciplines, mostly from computer science and a few from chemistry. The writing format and styles of these domain specific documents are fairly consistent, so a knowledge base can be built and used accordingly. However, ETDs can be from multiple disciplines, with various formats and writing styles, which further adds to the complexity of how they are analyzed and presented. Apart from the main content and format, the multidisciplinary ETDs will vary in the construction and frequency of other multimedia aspects like tables, equations, graphs, and images. Additionally, depending on the domain of interest, references can appear in the final section, at the end of chapters, or as footnotes. These unique characteristics of ETDs have to be taken into consideration, and machine learning approaches, such as neural networks-based deep learning methods, can potentially be used to achieve the advanced functionalities of ETDseer.

Besides leveraging existing DL technologies, we will research how to address the broad and deep requirements brought by ETDseer stakeholders along with ETD-related challenges.

\subsubsection{Structured Data Extraction}
Since ETDs are long documents, a passage retrieval, combined with a document retrieval, approach is appropriate \cite{salton1993approaches, tellex2003quantitative, mittendorf1994document}. 
Furthermore, ETDseer should extract key information from ETDs and present that to end users. 
However, structural complications associated with ETDs make these retrieval and extraction tasks much more difficult than when dealing with representative scientific papers. In a scientific paper, the authors typically arrange contents into clearly distinct sections, such as introduction, related work, conclusions, and references. In contrast, ETDs have highly varied and unpredictable structures. Some ETDs have book-like structures where each chapter has its designated role in the book, while others are presented as a composite of scientific papers where each chapter represents a separate paper. In the latter case, front matter (i.e., table of contents, table of figures, and table of tables) can appear in multiple chapters. The location of references can vary: some appear at the end of the document, some are at the end of chapters, and some can be seen as footnotes \cite{park2011isetd}. Existing SeerSuite segmentation and extraction tools are not effective with such varying formats, or with unpredictable structures. Herein we propose potential solutions so ETDseer can address these challenges.\\

\noindent\textit{Research Questions}
\begin{itemize}
\item How can ETDseer identify all hierarchically arranged sections in a long document? 
\item How can we extract (or infer) from the ETDs the table of figures, table of tables, table of equations, as well as each of the figures, tables, and equations, along with their content/captions?
\item For references that appear in various places within the ETDs, how can we locate and extract them, and present them in a canonical format?
\end{itemize}


\noindent\textit{Research Plan}
\begin{itemize}
\item \textit{Task 1:} {\bf Segmentation} of ETD documents can be achieved through the combination of  heuristic-based strategies \cite{srinivasan2011isetd} and deep learning approaches. 
Heuristic methods will focus on ETD specific knowledge like font and semantic information of chapter titles and numberings. 
To deal with ambiguous chapter segmentations, deep learning approaches will treat the whole document as one picture, with which segmentation or detection approaches like Mask R-CNN \cite{he2017mask} can be applied. 
With the resulting structured data, key extraction methods used in CiteSeerX \cite{caragea2014citeseer,giles1998citeseer} can be applied to obtain the hierarchically structured subsections from the ETDs.
\item \textit{Task 2:} {\bf Tables and figures} are effective devices for presenting research results. 
To extract these types of structured data, Dr. Giles (Co-PI) and his research group have made good progress on dealing with scientific data in computer science and chemistry related domains. 
This has led to TableSeer \cite{liu2007tableseer} and research results from the doctoral work on figures by Sagnik Ray Choudhury \cite{williams2014bigdata}, along with other related publications \cite{choudhury2016curve, choudhury2016scalable, ray2015automatic, choudhury2015architecture, choudhury2013figure}.

With discipline-independent ETDs, the content in figures and tables can be domain specific. 
Figure \ref{fig:FigTableEqua} shows figures and tables from various disciplines. 
In Computer Science and Mathematical disciplines they have more data-intensive content, while disciplines such as Education and Sociology tend to have table and figure content with more descriptive words. 
To extract these style-free figures and tables, machine-learning based approaches should be applied in addition to existing CiteSeerX methods.
\begin{figure}[ht]
\centering
\includegraphics[width=150mm]{FigureTableEqua}
\caption{Sample figures, tables and equations in ETDs from various disciplines}
\label{fig:FigTableEqua}
\end{figure}
\item \textit{Task 3:} Existing {\bf citation extraction} tools from CiteSeerX perform well on extracting references at the end of documents and thus they can be directly adopted by ETDseer. 
For references occurring at the end of each chapter, or some other random locations like placed in tables or as footnotes, we choose to use machine learning-based approaches. 
In particular, we will first conduct studies of ETDs to gather reference placement information from a wide range of disciplines. 
This step will ensure that a variety of features will be extracted. 
Then we can perform fine-grained feature selection to improve accuracy and efficiency. 
A classifier will be trained to identify references appearing at unusual locations. 
Extracted references will be ultimately converted into some canonical format like BibTex so they can be neatly presented in any user-desired style.
\end{itemize}

\subsubsection{Text Generation}
While NDLTD has made access to ETDs much easier than before, its keyword-based searching and topic-based faceted browsing greatly limit its usage. 
A large number of ETDs will be returned on a particular topic, but it will be difficult for users to identify which ETDs to read. 
Besides, each of these book-length documents generally contains numerous chapters, within which there are multiple sections and subsections.
It is time consuming for readers to browse through these length documents with their numerous chapters and sections, when only interested in the key ideas. 
A good summary can become a hypertext hub for jumping to parts of interest.

This can be done through passage retrieval \cite{wade2005passage, salton1993approaches, liu2002passage}, probabilistic graphical models, and topic analysis techniques like Latent Dirichlet Allocation (LDA) \cite{blei2003latent}. 
To specifically deal with discipline-independent ETDs, topic modeling's generalization capacity can be applied in ETDseer. 
However, some ETDs are assembled as a collection of scientific papers, which sometimes have relatively loose connections in terms of topics. Such can complicate the use of topic modeling in such ETDs. 
To overcome this hurdle, we plan to develop new models that will rely on  better representations, derived through deep learning.\\

\noindent\textit{Research Questions}
\begin{itemize}
\item How will stakeholders determine which ETD, or chapter of an ETD, or section within a chapter, to read, so that the desired information on a specific topic can be obtained? 
With ETDs from diverse disciplines, to what extent can we improve the generalization capability of the current techniques in use, and how?

\item To summarize key information from different sections of ETDs, how can we improve quality and utility, and from what kinds of aspects? 
How can we achieve the goal of generating both precise and concise summaries for all hierarchical levels, i.e., in a chapter of an ETD, in an ETD, and in a group or cluster identified from the entire ETD collection?
\end{itemize}

\noindent\textit{Research Plan}
\begin{itemize}
\item \textit{Task 1:} To extract key information from text documents, traditional {\bf probabilistic graphical modeling} techniques have been explored and applied extensively. 
In these models, words are represented so that each contributes equally in the vocabulary set. 
These inefficient representations cause generation of imprecise and often highly flawed key information, especially in the context of long ETD documents. 
We propose to extend the probabilistic graphical modeling approach with distributed representations of words as adopted by word2vec \cite{mikolov2013distributed} and GloVe \cite{pennington2014glove}. 
This would produce more efficient representations of words, and result in more meaningful phrases and sentences. 
Moreover, we will integrate attention models to our modeling mechanisms in order to learn salient words and phrases automatically. 
Accordingly, we will design a representation learning-based probabilistic graphical model to extract topics and key information from individual sections of the document.
\item \textit{Task 2:} To extract a set of {\bf topics from different hierarchical levels} of ETDs, such as in a specific chapter, in a particular ETD, or in an sub-collection of the ETDs, deep learning models such as encoder-decoder or sequence-to-sequence (seq2seq) based approaches will be researched; they have shown promising results in various applications such as in image captioning and machine translation \cite{goodfellow2016deep}. Furthermore, with additional support of the attention mechanism \cite{olah2016attention}, salient keywords can be learned effectively, which not only ensures high quality of generated multi-topic summarizations but in turn further facilitates our goal in \textit{Task 1} of learning key information. 
Figure~\ref{fig:atten} illustrates the conceptual results addressing text summarization problems demonstrated by one of the state of art attention model-based approaches. 
The general concept is that when generating new words for a summary, at each time step, determine whether to turn to source text or the vocabulary set through a generator gate. 
If turning to source text for fact generation, the attention model can help decide which words should be paid more attention to at the current time step. 
We further propose to combine seq2seq and attention mechanisms to train a supervised learning model in ETDseer. 
Then, we plan to use Amazon Mechanical Turk to evaluate the summaries for ETD documents from different hierarchical levels of content.

\begin{figure}[ht]
\centering
\includegraphics[width=\textwidth]{GetToPoint}
\caption{Architecture of state of the art text summarization model \cite{see2017get} }
\label{fig:atten}
\end{figure}

\end{itemize}


% Visualization 
\subsubsection{Network Visualization}
References contained in scholarly publications are used to cite and give credit to related works. 
The citation links between "citing" and "cited" research works allow easy navigation among related works. 
Reference networks consisting of citation links not only intuitively present the relationships between research works but also provide a unique way  to trace through the evolution of research ideas. Furthermore, different research groups often work on similar research problems. 
Social networks consisting of author names can suggest potential collaborations between different groups. 
Readers often want to see such relationships in research communities so that in the future they can pay more attention to these groups. 
Given the fact that the total number of references in an ETD is significantly larger than in scientific articles, building social and bibliometric networks between papers and papers, papers and ETDs, ETDs and ETDs, as well as their authors, becomes more necessary. \\

\noindent\textit{Research Questions}
\begin{itemize}
\item How can we build a huge network among all the people and references in ETDseer? 
What are the attributes required for better visualization of the network?
\item Considering the resulting social network, how can we characterize the relationships between research groups? 
What do we need to show in the resulting visualizations?
\end{itemize}

\noindent\textit{Research Plan}
\begin{itemize}
\item \textit{Task 1:} Observing that one reference can cite another but rarely vice versa, we will {\bf build directional reference networks}. 
We propose to adopt the force-directed graph \cite{fruchterman1991graph} approach for visualization. 
In terms of what to show in the network, we will focus on presenting numerical data such as citation counts and paper quality scores. 
\item \textit{Task 2:} {\bf Social networks} consisting of research groups can reflect their collaboration strengths. 
These networks can include users like co-authors of the same work, attendees of the same workshop, panelists serving on same panel, and also the research students sharing the same advisor. 
However, there is no readily populated information indicating the relationships between groups except their mutual citations. 
To visualize their connections, we need to extract key features such as research similarity. 
First, we plan to cluster such research groups in terms of their research interests. 
After the high-level grouping, fine-grained metrics such as direct citations and research topic similarity based on the particular research problems they work on can be used in visualization.
\end{itemize}


\subsection{Expected Deliverables and their Impacts}
We explained the scenarios of how and in what context ETDseer is expected to be used in Section \ref{advanced_services}, and also discussed the system design required to achieve the services described in the scenarios in Section \ref{system_design}. 
In this section we connect the dots and present how our specifically designed system as well as the involved technologies can achieve the desired end results.
Figure \ref{fig:workflow_uml} shows an example of the workflow of the scenario involving a student researcher and the relevant system components we designed for it to be functional. 
We further list a few other examples of the expected end
results of the services we provide by integrating the discussed technologies.\\
\begin{figure}[ht]
\centering
\includegraphics[width=150mm]{UML}
\caption{A workflow diagram of a student researcher involved scenario}
\label{fig:workflow_uml}
\end{figure}

The following subsections summarize expected deliverables and their impact on some of the key stakeholders.

\subsubsection{List of Open Problems and Methods}

Using machine learning based text extraction and summarization techniques, ETDseer auto-generates a list of open problems and associated methods for the end users, matching their desired research interests based on the search query. 
ETDseer provides the functionality of extracting open problems from the {\textit{Future Work}} and {\textit{Conclusion}} sections of various ETDs. 
This service can save user time and effort as critical similarity between their research topic and the methods used in previous works, can lead to waste of time and resources that the researcher invested. 
Similarly, ETDseer also extracts methods, approaches, and solutions from  {\textit{Methodology, Design}}, or related sections. 
This gives the researchers ample options to direct their own approaches in their research. 
Basically, a customized list of open problems and methods can enable users of ETDseer to explore the novelty of the research problems (through the generated list of open problems), and provide a basis for localizing potential approaches (through the list of the relevant methods) to the problem.

\subsubsection{List of ETD metadata}

Applying automatic metadata extraction techniques, ETDseer can auto-store a list of metadata fields, and the users can then retrieve the values via searching and browsing. 
This list consists of features including but not limited to: links to related documents, author information, citation statistics for documents in the provided list, and related reference lists. 
Related documents would also have the bibliography provided for each corresponding document. 
A user can utilize it to filter the ETDs that would be most relevant to her interest, while saving significant time and effort to go through all the ETDs without any filter. 
Extracted author information from the relevant documents can be used to find researchers to collaborate on projects, or contact relevant authors for various purposes like conference invitations for paper submissions and presentations. 
Citation statistics could be used to measure the quality of the ETDs, and users interested in more popular ETDs can use this feature as well. 
For researchers and students, finding conferences precisely related to their fields can be a painstaking process, as the list of conferences provided by external websites can be misleading, and not even relevant or of expected quality. 
ETDseer helps with assessing the quality of the resources it provides, e.g., conference lists, by adding metrics like rank, publication citations, and the percentage of accepted papers. 

\subsubsection{List of tables, figures, equation, acknowledgement, references}

After applying our text extraction and synthesis techniques, a synergistic list of essential parts of ETDs can be generated by ETDseer and presented to the stakeholders in a format conforming to users’ specific requests. 
These synergistic lists of information are often very useful to end users. 
When dealing with hundreds and thousands of files and documents, the summarization part is vital, as it improves efficiency of the system by saving user time. 
Furthermore, the visualization aspect of summarizing the ETDs is essential, as users can spend minimal time in deciding which of the ETDs and articles in the provided list are worth further exploration. 
Therefore, ETDseer takes into consideration the visual features of ETDs like tables, figures, and equations, and provides an organized list of summaries accordingly. 
ETDseer also has the ability to extract information from undervalued sections like acknowledgments, which hold valuable information regarding the organization and people involved in the particular research. 
Additionally, a comprehensive list of references is provided for the users to explore related works.

\begin{comment}
\subsection{Evaluations}
\subsubsection{Component by component evaluation}
\subsubsection{End to end evaluation}
\subsubsection{User Studies}
Section 3.4, including subsections (3.4.1, 3.4.2 and 3.4.3) are left blank, as we are yet to create a prototype and also lack sufficient data required to evaluate the proposed system. 

\section{Project Management and Timeline}
\end{comment}

\section{Related Work}
\subsection{Academic Citation and Search System}
CiteSeerX \cite{caragea2014citeseer} has been developed at Penn State University by the group led by Dr. Giles, as a public search engine and digital library, and is gaining popularity in scientific and academic paper archiving. 
Considered to be the first automated citation indexing system (with a patent on this topic), it is the predecessor of academic search tools such as Google Scholar and Microsoft Academic Search, which also focus on short documents. 
However, it mainly focuses on computer and information science.

\subsection{Natural Language Processing}
\begin{itemize}
\item \textit{Named Entity Recognition} \cite{lam2007named, etzioni2005unsupervised}. 
State-of-the-art NER systems have been developed for well-formed English text. 
However, due to the varied writing styles among ETDs, it's hard to extend such models to all disciplines. 
We plan to build an ETD NER dataset and train models for ETDs from different domains to improve NER performance on ETDs.

\item \textit{Segmentation}.
Most previous segmentation techniques mainly employ heuristic-based strategies \cite{srinivasan2011isetd}. 
As ETDs are scanned line by line, we can detect the changes of font style or font size, which indicates the start of new sections in a document. 
Based on such heuristics, hierarchical sections can be segmented. 
But success depends on how well the ETD is formatted, and ETDs from different disciplines do bear different formatting, which forces development of more generalized segmentation techniques. 
There are statistics-based models for text segmentation \cite{beeferman1999statistical}, but they rely on the feature representation of the text data. 
We plan to use recent deep models for feature representations \cite{he2017mask} and combine them with general heuristics in all ETDs to boost performance.

\item \textit{Topic Analysis}. 
There have been much work on extracting key texts from paragraphs. 
Traditional approaches typically use probabilistic graphical models, e.g., extended Latent Dirichlet Allocation \cite{blei2003latent}. 
Recent deep learning models, like recurrent neural networks based on attention \cite{olah2016attention}, can help locate salient words/phrases that readers should pay attention to. 
We will build upon such models with the goal of extracting important texts that can be indexed as metadata to aid user searching.

\item \textit{Summarization}. 
In order to catch the key ideas from long documents, it's essential for the system to summarize them. 
State-of-the-art methods on text summarization in the NLP field provide good tools that can be used within the framework of the ETDseer project. 
Our plan is to rely on recent powerful sequence-to-sequence learning \cite{sutskever2014sequence} approaches combined with attention mechanisms \cite{olah2016attention} and memory networks \cite{sukhbaatar2015end} to generate the necessary text summaries. 
We will build on extensive local work developed by our team, along with other existing methods.

\end{itemize}

\subsection{Image Processing}
\begin{itemize}
\item \textit{Figure Extraction}. 
Identifying and extracting figures along with their captions from long  documents like ETDs is important both as a way for summarization, and to gain deeper insights into the work. 
Recent work on Mask R-CNN \cite{he2017mask} and its original Region-based CNN \cite{girshick2014rich} for image instance segmentation should be of high relevance to figure extraction. 
We anticipate building upon these state-of-the-art architectures and incorporating ETD-specific domain knowledge, which will result in a robust figure extraction component. 
\item \textit{Image Captioning}. 
Another key aspect for figures in scientific papers is the adjacent captions. 
Although authors will provide short descriptions about the figures, users want a more elaborate summary about the images. 
This leads to a requirement for image captioning -- an open problem  in computer vision. 
Sequence to sequence learning based models like \cite{xu2015show} and \cite{lu2016knowing} are currently exemplifiers of the best approaches. 
These can help us produce better summaries combined with surrounding texts.
\end{itemize}

\subsection{Table Extraction}
Tables are ubiquitous in scientific documents like ETDs. 
They present experimental results or statistical data in a condensed fashion. 
However, problems with automatic table extraction from untagged documents, and the lack of a universal table metadata specification,  have hindered the success of table processing in digital libraries. 
TableSeer \cite{liu2007tableseer} can crawl digital libraries, detect tables from documents, extract table metadata, and build indexes. 
We plan to incorporate TableSeer's key components for table manipulation into ETDseer, together with other models \cite{marinai2009metadata, etzioni2008open, witten2002importing}.

\subsection{Visualization}
Leveraging the very large social network and reference network produced, visualization is an integral part of the ETDseer system. 
It can help users gain deep insights about how different groups are working together. 
Christopher North, a professor at Virginia Tech, is expert on network and digital library visualization \cite{wang2002enhancing, yost2006perceptual, gansner2000open}. 
His StarSpire project \cite{bradel2014multi} and V2PI approach for dynamically updating parameters are of high relevance. 
This will allow more interaction with users as they navigate through the visualized network and learn from it.

\section{Expected Significance}
Each of the planned series of more functional prototypes of ETDseer will meet the boarder and deeper requirements of various users of the DL system in various scenarios. 
Documents and references will be analyzed and converted to canonical forms.
Figures and tables will be  automatically extracted and summarized. 
The limitations in flexible searching, browsing, visualizing, and exploring of ETDs, and as well as subsections within ETDs, due to their length and domain complexities, will be significantly minimized, if not eliminated. 
This research will lead to fundamental contributions in the digital library domain in terms of advanced services discussed in Section \ref{advanced_services}, by developing and applying techniques from information retrieval, information extraction, machine learning, HCI, and AI.

\section{Broader Impacts of the Proposed Work}
ETDseer, apart from providing the services discussed above, will impact the academic community in a much broader way. It will compute a variety of quality metrics like h-index, citation counts, and conference ranking, to guide users to content that is both relevant and of high value.  
Negative results, that rarely are published, will become available for the first time. 
Details of research -- given in ETD text, figures, and tables that rarely appear elsewhere -- will become accessible, and ensure a deeper understanding of methods and findings. 
While this advanced visualization of data can enhance the overall user experience, it can also be highly beneficial for users who have difficulty reading and interpreting works in general. 
A rich range of services connected with hundreds of thousands of (multimedia) ETDs, will dramatically advance research by graduate students (who make frequent use of others' theses) and the broader scholarly community, which requires an efficient system that enables more research and helps produce important results in less time.
\newpage
\pagenumbering{arabic}
\renewcommand{\thepage} {E--\arabic{page}}
\nocite{*}
\bibliography{csxref}
\bibliographystyle{plain}

\newpage
\pagenumbering{arabic}
\renewcommand{\thepage} {F--\arabic{page}}
% \noindent{\Large \bf BUDGET JUSTIFICATION}


\end{document}
