fact, I think we'll go ahead and get started
. So I'm Julie Speer, Associate Dean for research and
informatics. Your university library and I have the honour
of welcoming you all to Newman for the text and
data mining forum offered as part of the Library's Open
Data Week celebration. Um, and Open Data Week
is something we celebrate each year in hopes of bringing
together members of the Virginia tech community, uh,
to consider what access and open means in the context
of their own data intensive research projects and also to
help raise awareness resources at the university and beyond that
can support and facilitate access to data for research and
creative purposes. Um, we're truly honored to have
such a distinguished group of panelists here today prepared to
share their own experiences. So thank you all.
Um, and before I turn things over to Ginny
for panelists introductions, I just want to briefly mention
that today's events in this week's content. Mine programs
receive support from the Dean's Beyond Boundaries Innovation Fund,
which is an initiative designed specifically to advanced innovative information
and data ideas that explore concepts addressed to the universities
beyond boundaries vision. So thank you all for coming
and I hope you enjoyed today's morning. Oh,
the good afternoon. My name is Ginny Pannabecker
and it helps sciences research support coordinator here
at the V. T. University Libraries. Um
and I've been really excited to work with Tom Arrow
and Stefan Cost Berger who are here from ContentMine
um to come this week and provide workshops throughout the
week um on text and data mining, particularly for
scholarly and research literature and for today's forum, I'm
really excited to have so many researchers here from VT
who do all kinds of different text and data mining
projects. Um, so to start out with,
I'd like to introduce our panelists. We have Brent
Huang associate professor in the department of Computer Science whose
work done. Um I apologize I'm an assistant professor
. I just want to appreciate automatic connection. Yes
joined a panel and you can also move out quickly
. Um So assistant professor thank you in the department
of Computer Science. Um So I was really excited
to invite Dr Huang because of his work related to
social media mining projects. Um And then we have
um Dr Leanna House associate professor in the department of
Statistics. Um And we've seen that Leanna Hous has
a research in text and data mining for education methods
and also looking at statistical methods for reliability in results
of mining. And we have Dr Ed Fox um
a professor in the department of Computer Science. And
also I want to thank Dr Fox for his help
also in arranging the events this week. So it's
been great. We went to a couple of his
classes with our content mind speakers to talk with them
about text and data mining. So I'm excited to
hear more about his projects related to these methods.
And then we have Dr Patrick Fan, professor and
Pamplin College of Business. Um also someone who works
with mining, social media, text data and um
dr Ewing uh Dr Tom Ewing, Associate Dean College
of Liberal Arts and Human Sciences. So um Dr
Ewing has done many projects related to extend data money
for historical and archival documents, which is humanities and
social sciences perspective, although also in ways related to
the sciences, such as Um the flu pandemic in
the early 1900s. Right? Um so our last
uh Panelist here is Tom Arrow, so software developer
from ContentMine and uh tom will start us off
with a brief introduction to what is text and data
mining and kind of set us up for our discussion
today. Thank you very much Ginny. Yes,
so I'm just gonna give you a really quick run
through what text and data mining is, just sort
of engage the audience. How many of you have
heard that text and data mined before? How did
you come to our talks? Also? Quite a
lot of that's very good. So basically uh text
and data, Michael's using machines to extract information from
text that we either didn't have access to already.
All that was uh sort of hidden within the text
itself. So maybe you're digging out relationships that are
not possible for a human reader to see. Maybe
you're looking at trends of hundreds of different social media
things, for example, or you're trying to analyze
lots of different scientific papers over a really broad time
span. So this is a a very important thing
for people to do because as as the world is
sort of growing larger and larger and it's amount of
data it has, then it becomes completely impossible even
for big organized groups of humans to look through all
the information and to actually and get knowledge from it
and to benefit from that information. So we have
to rely on machines to be able to help us
to do that. And an example of where this
might be very, very useful is one we've been
bringing up during our talks is in the case of
something like a pandemic. So if you think about
the zika virus last year, around the time of
the olympics and then people very quickly needs to be
able to get up to speed on that topic.
And they could actually get up to speed from a
wide variety of sources. But if you wanted to
find out from the academic literature a lot about zika
, then you need to find all those papers related
to zika and you would need to find out things
about those papers. You you might want to find
out how much they can be transmitted and we know
it's transmitted by mosquitoes. But maybe you need to
know more about the species of mosquitoes and I'm how
you might then control those. And so it's not
feasible if you're in a big hurry to be able
to do that entirely manually, you want to have
some assistance. So this is where something like text
and data mining comes in and you can pluck out
the species, for example from a paper and then
look at the most relevant species to your particular time
. Or you might want to pluck out locations and
find out where these kind of pandemics have happened before
and maybe we'll happen again I think gives a pretty
good summary of what you can do with text and
data mining. There are a number of different things
you have to be able to do um in order
to carry it out and I guess they can be
broken down into three phases. So first you have
to get the raw information to mine and so in
the case of analyzing tweets and then you have to
have a supply of raw tweets from twitter somehow.
Maybe you you pay for them or maybe you get
them for free. Um in the case of academic
papers you either have to download them from the publisher's
or get them from a repository I guess in the
case of old style print media, you have to
find some leaders willing to scan those in or even
has the source of them in the first place.
And that's kind of the first total you have to
overcome. And and with that also comes some kind
of step where you have to, if you're getting
from a variety of different sources, you have to
normalize them in some way so that you can kind
of compare all those documents without, you know,
without suffering from the fact that some came from one
source and some came for another. You then have
a step where you're actually extracting the information from the
text. So this might be very simple. It
might just be looking up particular words named entities that
you're interested in. It might be doing things like
NLP natural language processing. You might be doing things
like trying to analyze the sentiment of tweet something like
that is what you can do. And then once
you actually got all this information out and you'd like
this to be in some kind of structure form,
you then want to do something downstream of it.
So you probably want to do some analysis. Maybe
you want to see how the species you've extracted from
your papers about a pandemic connect together or you want
to see about how sentiment over a particular topic changed
over time. And then the really the end goal
of having done this analysis is then to help influence
humans make a decision about some policy area or where
to do more research or where we might be able
to solve a problem. So I think that sums
up kind of a good pipelines for how you might
do that. Hey, happy about thank you very
much. So, um I thought one of the
most interesting things about having you all here would be
to hear more about a particular project or two that
you've worked on related to text and data money.
So, um, if you can each give us
a description of an example project um or interest that
you have in text and data mining and let us
know what what led you to use text and data
many methods in that project. What did TDM let
you do or find that other methods might not have
? Um so we could go ahead and start with
you again, I'm sure. So, a content
line, but we're basically entirely based around doing text
and data mining and we're focused particularly on mining the
academic literature. So were heavily in the business of
doing things like downloading papers from publishers from repositories and
solving the technical problems and the social problems related to
this, which involves looking at advocacy in terms of
copyright law, and also involves technical measures and trying
to work out how to actually get hold of papers
. Um we do a little bit of providing tools
for people to do the analysis themselves, um and
also pointing people in the right direction of other tools
that might help them out and things that I'm already
available, but people people don't know about, I
guess, a good example of the sort of project
we've been going through recently, as we've been doing
this sort of scoping project on zika and on pandemics
. And this is very interesting because and it was
very topical last year and it's very difficult to to
quickly clean all that information without without having to to
download it all and read it by yourself. So
, um without using text mining tools, you wouldn't
know for certain what papers to download and you would
not be certain how those papers maybe connect to one
another. So this is something that you definitely would
be unable to do. For example, we looked
at the co occurrence of different species within the Zika
papers. One thing that probably a specialist researcher already
knew about but we didn't know about but we're not
specialists in pandemics but keenan giving these people who are
specialists was um that I'm there was a lot of
mentions of some species that we didn't know related to
the mosquitoes that spread the zika virus. So that's
eighties Egypt is the mosquito that spreads it and other
viruses that are similar to like dengue fever and other
fluffy viruses. And we found for example that a
species that kept cropping up we didn't know about was
this species called Wolbachia. And this turns out actually
is a bacterium that infects the mosquito itself and makes
it less susceptible to zika transmission, which is obviously
actually very interesting. And maybe we could then think
about designing research to take advantage of this. Or
maybe we could think about hoping to propagate this bacterial
find areas where this bacteria is very prevalent. Which
we can also do by text mining and finding what
regions this bacteria is mentioned as being found in.
Um and we can realize that maybe this area is
less likely to be affected by the pandemic and so
resources might want to be directed somewhere else. Yeah
, that's a nice example of where text and data
mining can have a really big impact on something that's
actually a problem. Thanks. Yeah. Then if
maybe each of you just don't. The table would
like to share an example also. Okay, thank
you here at the tom end of the table.
So thank you, jenny, appreciate the invitation to
come and talk and thank you all of you for
coming. Um My approach to this, this brief
talk changed actually over the weekend. Um I spent
sunday morning in Washington touring the new african American Museum
, the Smithsonian Institute, which is really a remarkable
place and I would encourage all of you to go
. Um but I'm thinking back on it. I
think one of the things that was so remarkable is
the museum makes you slow down and read. Um
if you go through the history exhibit, you spend
about an hour just reading the exhibit, the captions
, the text, the examples and everyone is doing
this and they're all doing it quietly. Um and
so what I think at a time when it seems
like everything we do is faster. This museum is
really quite striking and then it tells you to slow
down and pay attention. Um and that was that
was useful and and a million people have gone there
in the last six months and you can't get a
ticket through august, it's in very high demand.
Um The other reason I wanted to talk about that
is when you Go through the ground floor and you
look up on this three storey wall, um they
have several quotations um in this amazing, you know
, size front, um one of which comes from
from the novelist James Baldwin 1965. This is what
the quote on the wall says. Uh the great
force of history comes from the fact that we carry
it within us are unconsciously controlled by its history is
literally present in all that we do unquote. And
it was the last part that that really kind of
connected with what I was thinking about in terms of
text and data mining, because history is very present
in text and data mining. Um, it really
needs to be acknowledged and addressed and I would say
more generally, um, the humanities, you know
, everything has a history, nothing is completely new
or unprecedented. Everything is changing all the time.
And so something like a pandemic or the zika virus
um, is new, but it's also historical.
There's their origins and their roots and their causes that
come back to that. And we need to be
able to think about that. Um, as we
approach those kinds of problems and those kinds of topics
, history and more generally, humanities are very much
about complexity and context. And so one of the
issues with text and data mining is the extent to
which they de contextualized knowledge or simplify it, you
know? And so you want to be aware of
that tension between complexity and context on the one hand
and the volume and the speed of the information that
you're gathering through these other methods. And it's also
about scalability. You know, there's a real attraction
in a real value, in being able to look
at information at a very, very large scale,
there's Australia's value, being able to look at a
small scale and to move between those levels. Um
as you do this analysis. So, so as
jenny mentioned, um I've been involved in two projects
and when you said earlier, 19 hundreds, I've
kind of nodded because that's actually halfway between. So
the first was a project on digging into data from
the spanish flu of 1918. Um and that involved
um Naren Ramakrishnan from the, from the environment of
computer science, Professor Bernie's house, private english and
and Bruce Spencer sitting there in the middle From university
libraries. And the new project is on the Russian
flu of 1889. That's why it's actually bang right
in the middle, which is less well known certainly
than the spanish flu, but was a project that
was particularly well suited to a text mining approach because
This era, the 1880s 1890s are in some ways
kind of pivotal moment in worldwide transmission of information.
Um You have because of the international telegraph and cable
system, the opportunity for information to travel at a
high speed all around the world at great distances.
Um and for information to travel at a pretty high
volume, you know, so if you were to
think about obviously where we are now in terms of
any information age, Um this period 1880s 1890s are
one of the transformative moments that helped bring us to
where we are now. Um, it's also a
very good era for doing text mining because so many
materials have been digitized and you mentioned earlier this problem
of scanning and, and using ocr optical character recognition
to turn these pages into into text. A lot
of that has been done and it's been done with
newspapers and it's been done also with periodical. So
this project is looking at um, the relationship in
some ways between different kinds of information in popular media
, which would be newspapers at the time, but
also more specialized publications and medical journals and so on
. And probably some of the same ones you're looking
at now that go back that far. The british
Journal of Medicine and General American Medical Association and so
on. Um, you know, and for a
historian, this is, this is big data.
This is, this is high volume. So um
, if you look at just one newspaper collection,
the one the Library of Congress has, you know
, over the one year and you search for time
like influenza or grip, which was a synonym at
the time. You know, there's 13,000 pages that
have either those two words on them the six weeks
at the peak of the epidemic. Um, from
middle of December to early January, more than 4000
pages. So a quarter of the whole years is
concentrated in the whole years, concentrated just in these
In the six weeks. And so what this project
has has taught me to do. Um, and
then working with students and learning from them, you
know, is to think about how being able the
capacity to look across large volumes of text lets you
not only focus your research questions but also shape them
, you know, and ask about a different set
of questions that I might have using more traditional humanity's
kinds of methods. Um and to use that capacity
for volume two point at certain kinds of issues and
certain kinds of questions. But then it does involve
going back to the texts and reading them closely,
you know, and you really think about interpretation of
the meaning that's conveyed within those texts and how they
relate to each other. So in some ways what
I end up with in terms of text and data
mining is very much a dialectical relationship. You know
, one of them is very important advances the other
. But then as you move back into that close
reading, it suggests a different set of questions that
you can then take back to the text and data
mining and maybe frame the search differently your frame that
comparisons and so on. And so I see these
these tools. I'm not only closely related but necessary
for each other. Thank you. Thank you.
All right. My name is Patrick Fan. I'm
a family member in the accounting and information system in
the College of Business. My background is uh primarily
in computer science and also information science. And I
had this training information retrieval for a number of years
. And then I joined Virginia Tap in the business
school to work in this uh, accounting and information
system department. So when I came to Virginia Tech
, you know, my background in this text in
this information, the treatment area is really starting to
transform into the text mining area. This is where
you see Back in the 2000, you know,
you know, there's a lot of applications on the
application of data mining on the text. So the
project that I've been working on, uh, since
the work at Virginia Tech has been a lot of
different projects, but the one that I want to
mention to you guys today is the one that we
used in the text mining method to mind these online
public forums to discover the product quality issues. So
we call the term called quality and genetics. Alright
. So you can potentially look that up online.
We have published a number of papers in that area
. The reason we are doing that is because we
are in the business school. We need to understand
what the product, especially the quality issues uh,
made by the manufacturers. We need to understand the
issues. We need to basically try to address those
issues and then we can make the customer happy.
However, I think it turns out if you look
at the news news media, there's a lot of
scandals going on, like for example, GM General
Motors, they have done a number of recalls in
the past about the vehicles and if you look at
those vehicles, many of those vehicles, recalls are
done after many, many years, right has been
passed, passed. And but however, if you
look at this online social media space, there have
been numerous discussions, talk about the issues with the
vehicles, right? And so people chat about it
in the social media, people complain about in the
online forums and then the other people offer the solutions
for those kind of problems. So for us,
you know, we, we feel like, you
know, this could be an interesting data source for
us looking into and see how can we mind the
social media space, especially the public available data source
and then try to mind these qualities, you know
, defect issues with with the product. So we
basically, you know uh I work with a colleague
B. E. I. T. In the
same uh college business and we work together, you
know try to mind this online forums. We started
with the product category called vehicles, the cars.
And then we mind thousands of car forums and we
just call the data down and then we go through
the data. We do some human annotations and to
label which discussions related in the car defects. And
then we build classifiers and that can automatic classify those
threats discussions into like defect related on defect not related
, right? So once you have all of these
and then you can extract all these different entities,
different defect categories components involved. And then you can
aggregate all this information you can present to begin a
dashboard dashboard fashion and then show to the manufacturing,
show to the managers, you know, these are
the things you need to look into right for your
product. So vehicle turned out to be a great
example for us to work around and then we have
a great success on that. So the research has
been featured in new york times and then the dealers
, the manufacturers has contacted us in terms of,
you know, trying to work with us in terms
of get these tools into use by the manufacturers to
help them to do a better job in terms of
uncovering the potential issues with the product. So now
we have uh, branch, I'll try to do
more comprehensive product quality analytics, uh, you know
, surveillance, all kinds of products. We are
now using the data from amazon reviews. You can
find tons, tons of online reviews. We analyzed
the data from the forums and basically we try to
come with different categories of product and then we try
to help the manufacture out to do the quality controls
. So this is one of those examples I want
to give to you. Of course there's a lot
of other things that we are working on right now
. In terms of for example, the firm disclosures
, you guys you guys have ever heard about United
Airlines, Right? The story about the United Airlines
, it's all the news media all the time.
So this is what we call them as a bad
pr Okay, so bad public relations, you know
, press release. So when you talk about this
kind of news, if the news is very negative
, it's a crisis and then the firm has to
respond very quickly. Now, how do you respond
? Wasn't strategy going to respond? Right. And
then those kind of actions may cause some different social
media responses by the users. So if this thing
was not handled properly right, this could go into
a turn out to be like a disaster, you
know, for for the company. And then if
you look at the stock price for the for the
United Airlines, you know that bad pr really has
heard the stock tremendously one single day. Right?
So these are the things we we we analyze a
lot in the business school. You know, we
try to understand the impact on some of the news
, impact on the stock price, on the equity
price. We also try to see how can we
come with a better strategy to deal with this kind
of crisis, deal with this kind of uh issues
. So crisis management, how to do the proper
P. R. E. Ds and how to
do the proper disclosures about about your major incident you
may have. And these are all these things we
try to understand. So there's a lot of opportunities
in the business school fee or try to analyze using
this text data mining. Thank you. Mhm.
Mhm. Thanks. I'm at Fox in Computer Science
. I direct the digital Library research Laboratory. So
it's nice being in a library, grateful for being
invited and for all the people who are here.
Um We're very close to the 10th anniversary of the
shooting at Virginia Tech. So the project that I've
been working on has been connected to that mm When
that happened, I thought what can I do that
I can bring my knowledge and expertise to try to
be of help. So the series of projects really
all one, but For like 10 years now we've
had funding for the National Science Foundation and ongoing for
several more years to try to remember and try to
record what happens in the world and all the important
events and to understand them into analyze them. To
help people who want to understand particular events and also
trends. So I want to invite all of you
if you're interested, we have a project called Guitar
, I'm not very musical. It stands for global
event trend archive research. So if you want to
collect information that anything that you think is an important
event in the world or try to understand some particular
trend, we'd love to work with you. We
have now 1.5 billion tweets that we've been collecting and
we have millions and millions of web pages. And
so the idea here is to make sure when any
important event occurs that we immediately begin to capture as
much as we can about this because it gets lost
. Otherwise tweets have a fairly short lifespan and they're
easily disappear. Web pages and half a year and
many of them have changed or gone. If no
one records them, then we won't be able to
do this. And there are some groups that try
to do this like the insurance archive which we partner
with, but no one has figured out how to
really use those archives that keep storing them. And
it's great to kind of pile up all this information
. But unless you can help people make use of
them and to do research with them And to do
it over long periods. The Internet archive began in
1996 96, mostly early 97. So we now
have 20 years' worth of collection of information from the
Web. It's a bit spotty because they don't have
resources to do this. And they have a lot
of partners Virginia tech and others. We're helping collect
even more things. So how can we look at
this information? How can we find people and organizations
and concepts and events and activities and see how things
change over time and how people talk about things in
different ways. one of the students helping in this
was interested in hiking. A lot of people in
this area. I'm sure interested in hiking and so
on. So the Appalachian Trail, the continental divide
trail, the pacific Coast Trail. How do people
talk about things differently as they, as the years
progress in each of those? And does that mirror
the mission of those? So she's trying to extract
information from all of these things. So I've been
doing this kind of thing for about 40 years now
. I guess I'm the old timer here in the
room. But this is really an ancient need.
Mm Humanity has always needed information to find food,
find um places of arable to get water other people
to work with and do things with. So,
extracting information from the streams of text that we have
all around us is a fundamental problem. So this
is the latest version of this. And now we
use a lot of technologies. We use statistics,
we use machine learning, we use artificial intelligence,
we use language processing these a lot of different approaches
to that. And so These kind of methods let
us serve the needs of users. And about 30
people around campus have agreed to be helping in the
guitar project because they want to do research about sustainability
. Energy use, population politics, war, violence
, all kinds of different trends and things that are
going on. And by the way, I should
invite you. The lyrics tonight is having a pre
showing For free. So you can get a free
movie at 7:00 on between Earth and Sky. So
it's all about the environment and trends and so on
there there. So how do we understand this and
and not forget about that and not ignore it.
Like so many things get ignored in our world.
So I think these techniques let us get the information
out. But then we can search them. We
can browse them, we can summarize them. We
can do lots of different things that people want to
have and to aid their research. So I invite
you all to become part of this dr house.
Sure. Uh, yeah, so my name is
Lana House. I am a faculty member in the
Department of Statistics. And first again, thank you
jenny for inviting inviting us here. Um second,
I'd like to say that everything every project to which
I refer is certainly a group effort, part of
a collaborative team. So if I use the pronoun
I I really mean we there's a group of us
who work on problems related to data mining and others
that being said as a statistician um collaborate with some
computer scientists. We were really initially in the game
of trying to develop methods to be able to learn
from high dimensional data and specifically how we use these
methods and in collaboration with just expert judgement so that
as we explore high dimensional data, we want to
include the human in this analytical loop. Right?
And text mining became a specific example or application of
these methods. So what do I mean by that
? First of all, uh modern day datasets are
extremely high dimensional and we as human beings can only
visualize three dimensions. Right? So you have this
fundamental question of exploring high dimensional data in exploring high
dimensional data, you might be seeking patterns, you
might be trying to find anomalies, who knows?
But the approach that I take to explore high dimensional
data, it might be very different from the approach
that it takes, some very different from the approach
that he takes. So is that wrong? Absolutely
not. Right. But we want to be able
to develop a method that parallels a personal sense making
process as they explore high dimensional data set. Well
, for example, um the original, Gosh,
eight years ago, one of the original problems we
had was motivated by simulated data set from the Department
of Defense. One could imagine that you're trying to
identify where a terrorist attack might take place. And
the kinds of data that we're using might be twitter
data might be articles, might be newspaper articles,
might be airline tickets, who knows need. The
idea was that an expert in human in terrorist attacks
might not have all of the different analytical methods that
his or her fingertips. But if we can present
the data in a way for him or her to
explore and then subsequently detect a pattern and then explore
that further and then possibly see an anomaly and explore
that further. That's what we wanted to do was
to try to develop an analytical approach that is including
the human as he or she detects something that he
or she wants to know more, right, and
then provide the relevant data associated. So um yeah
, so I'm kind of glazed over some sort of
high level of what I mean by uh developing an
analytical approach that's paralleling uh since making process and then
using text data with that, but ultimately that was
one of our original goals and it can apply to
any high dimensional data set. Mhm. Uh So
in the spirit of being briefed, let's see if
they're oh, so uh with with text mining being
an application, it does present some challenges for us
, Right? So ultimately our analytical methods work well
when you have quantified the text documents, Right?
But that in itself is a hard task. You
can parse it out by word, right? And
then you might have frequency of that word in a
given document. But then what if the document is
super long or what if it's a word that appears
in all the documents, that's not necessarily going to
be helpful? And there are some standard ways by
which if I could use my own words sort of
quantify these text documents, but each standard way will
present its own issues and comes into the analysis and
you have to eventually make make some choices. That
can be hard to do. Um So saying that
when it comes to a pattern, like what what
we're looking for possibly patterns in these text documents or
possibly anomalies in the text document that in its own
right is a big term. What defines a trend
? Are we gonna use clustering? Are we going
to use some sort of time trend? Who knows
? Right. So again, it's gonna be application
specific and these are problems that we address similarly.
What defines an anomaly? Gosh, if you have
millions of documents, how the heck are you going
to find that one document that might lead us to
the next policy that we have to create? You
know? So these again are pretty hard to find
. And again, we're trying to use expert judgement
to help us define trend and to help us define
an anomaly. So that's more or less what we
do in this research group. We refer to the
research group as Beijing visual analytics sort of combining.
So Beijing statistics uh by definition allows you to conclude
expert judgment right? And then the visual analytics part
. This allows us to convey information in high dimensional
data visually so that experts aren't bogged down by uh
, analytical methods. Although even though we call ourselves
Beijing visual analytics, we found some challenges so we
might not formally use bays. But the spirit of
bays remains and all that we do. I'll end
with this sort of a side project in terms of
text mining. Because you did mention that I have
I have research interest in education and education pedagogy.
And if one can imagine if we're developing a technology
that a non technical statistician will be able to use
and it might then rely very well or fall very
well within education forum and then educating a class.
But then we want to evaluate our approach is and
we'll have open ended questions when we're trying to analyze
whether or not our our approach is working and those
are hard to grade. So can we use text
mining to help us evaluate if our our new uh
approaches in the classroom or are doing well or not
? Right. So who knows? We're on we're
working on that particular problem. Uh So without that
, with that I'll pass it on. Thank you
. All right. So I'm Burt Hong this isn't
professor in computer science. My research is on machine
learning and the way that interfaces with text and data
mining is that essentially you can think of machine learning
as trying to uh yes. Automate text and data
mining to some extent it's done in the way.
So there's some of the other panelists already mentioned examples
of this where rather than deciding on keywords you should
search for there, they sort of you sort of
give the computer examples of what you want to find
and what you don't want to find. And the
machine learning tools will then attempt to extrapolate from that
and figure out, you know, what are the
search terms I should include in my query or you
know, what patterns of language usage should I be
including? Um And this technology has been developed a
lot in the past, let's say two decades.
Um to the point where now it's becoming really useful
and there's a lot of opportunities to apply it to
new ideas and new new challenges. Um, what
I think about as a researcher and machine machine learning
is the cusp of what is not yet fully possible
. And these are examples where people would maybe would
try to use machine learning and the results don't really
, they don't really answer the question you want to
answer. And so there's something missing in what current
techniques are enabling right now. An example of this
is one of my favorite applications I'm looking at is
looking for, looking, detecting, cyberbullying and online
harassment in social media. So this is an example
of something where the language underlying online harassment is changing
rapidly, because it mostly involves young people who are
essentially inventing their own language every year. You know
, you get new new people coming up with new
slang terms all the time. Um, it's very
personal. So it's not just the language thing.
You can't just really search for language. You have
to look at context. You have to look at
, you know, what, what the social norms
of the, of the communities that these people are
part of our, like, maybe they're part of
a community that likes to use swear words a lot
. And, and, and so then it's not
really abnormal for them to be swearing at their friends
all the time. But then if you detect that
somebody is all of a sudden using swear words out
of, uh, abnormal square words for that person
, uh, directing at an individual, maybe that's
a sign of harassment or or some kind of conflict
like these, these little subtleties are things that we
don't really have a great machine learning tools to handle
, or text mining tools to handle. It's the
kind of thing where text mining would either get you
too much stuff that's not relevant or or would essentially
miss all of the examples of the things we're looking
for. So my goal as a researcher is going
to try to come up with how to fix this
. How do you come up with a workflow where
a researcher, maybe a social scientist might be able
to try to narrow down um, the the data
mining process to be more precise about the the idea
they're looking for. So again, in this example
, the the technique we're working with is something we're
calling weak supervision. So rather than um, what
rather than a having to have an expert look at
a bunch of examples and decide, is this an
example of harassment? Or is this not an example
of harassment? We're trying to give them just uh
, an interface where they provide general rules of thumb
of, you know, we think words that include
uh messages that include these racial slurs are likely to
be harassment, messages that are where lots of messages
that look like they're harassment are directed at one individual
are likely to be bullying. Like these kind of
general rules which may not hold all the time,
but are essentially you can think of them as search
terms get then extrapolated by an algorithm to new search
terms to new ideas, new new features that may
describe this phenomenon. Um And then there's this person
in the loop where as the, you know what
the machine learning algorithm and text mining algorithm returns.
Um the person can assess whether it's, you know
, it needs more rules or maybe this rule let
it down the wrong path. And then they can
remove those rules or or add more rules depending on
what they want to adjust. So that this is
a kind of a different inter interface for text and
data mining than we typically see nowadays. Um But
it's it's the way I think as a as a
researcher is, I'm trying to come up with these
new things. They're not ready for people to use
for the most part. Um But it's I want
to, my research agenda tends to look at the
problems for which your instinct as an expert will probably
think, oh, text mining is not really ready
for that, it's not gonna work. But then
the question is, why won't it work? And
can we build the tools that will address those issues
? Okay, so that's my story, I guess
that's great. Thank you very much. Well,
um I definitely have some questions I can ask here
, but I'd like to open it up to the
to our audience here and ask if any of you
have questions for our panelists that you like to know
more about their projects were about these methods. Don't
be shy. Mhm. Mhm. Um I guess
I'm just kinda curious what you see is the biggest
challenges for your projects, whether it's it's access or
legal issues or technical challenges, human creativity challenges or
what what? Yeah, those are the biggest challenges
. A Sure. So so for for me,
especially for this project, I mentioned the social media
online harassment project. Um right now, what we're
doing is we're analyzing public social media, people who
leave their privacy settings, everybody, you know,
everyone can read it. Um and that allows us
to see what's happening and but um but a lot
of what's happening, a lot of actual interpersonal,
you know, online detrimental behavior happens in hidden behind
privacy locks so we're not able to access that the
companies are able to access that, but then even
if they have access to it, there are privacy
um ethical issues about doing a lot of science with
that people's private information. So um yeah, that's
a big challenge. I don't have a good answer
for that right now. I'm sort of covering my
eyes and years and just saying this is public data
. You know, the users accidentally left their privacy
settings, you know, off and they're they're having
these very personal conversations but it's their own fault for
leaving it open, but hopefully I'm doing something good
for them, not bad for them, but it's
there's a lot of really tricky ethics going on here
, especially with interpersonal communication. So I guess I
should speak up for open data week. Right?
So this is open data week and I think it's
important that as much as we can we share with
each other and help each other and communicate in an
open fashion. So, um because I want to
work with this kind of information, I've tried to
focus on places where I could get to the data
I want without having a lot of hassle. I'm
not a very argumentative person, so I'm not going
after the journal publishers and all the others that are
worth going after. But it's not my personality.
So and I focused over the years, initially on
technical reports and theses and dissertations and more recently web
pages which are out there for people and tweets which
are fairly open. At least we get a good
sample for those things. So there's enormous amounts of
these things out there and plenty to be able to
test the different hypotheses and methods and techniques. So
it would be nice to do others. And certainly
the intelligence communities want to have special information worked with
. Mm I worked in a project in Qatar and
we were trying to we propose we would help them
um organize the information digital libraries in the country.
A lot of people really didn't care about that and
they didn't want to share things. We had wonderful
technology to work with chemists and the petrochemical industry,
but they didn't want to let anyone see their stuff
. So too bad for them. But there is
some open information and I'm sure the other time you
will tell us about some of that. So it
depends what you're looking for. It depends what your
goals are for my purposes. It's enough to get
to the open information, which means that then we
have other challenges, right? She were asking to
me the biggest challenges. Language when we talk about
text, we're talking about language. Language is a
stream and it's an evolving um method of communication.
We change the way we use words, we change
the meaning of words. We add words, anyone
here know how many words are in the english language
, roughly? About 500 billion. 500,000. I'm
sorry about 500,000. But if you if you think
of all the different names of people and names of
cities and roads and locations and organizations then probably my
500 million is actually a a good number. So
language is generative and it's creative, so understanding it
, it is difficult if we get a lot of
text and you know, google has trillion word collections
now that you can play with and work with.
But there's always new things and the way those words
are changing. And if you are looking at archiving
things over a period of time, the use of
those things change. Which is why we have historians
and social scientists and others who help us study this
. So working with others who are in linguistics and
history and sociology and other field I think is important
. So the partnerships are really important. Yeah,
I think if I can chip in about their open
data stuff too. So this is a big problem
that we have particularly looking at the scientific literature because
although obviously people who write tweets, do you hold
the copyright on those tweets? Um they've kind of
signed away a lot of that to twitter and twitter
then very happy for you to get a bit of
it for free or buy more of it for the
money. Um But academic publications in the most part
still owned by publishers and it's a multibillion dollar industry
to control these publications and then resell them to libraries
. Um Water individuals and libraries like Virginia Tech Library
pay for them to allow the students have access.
But people who aren't affiliated with an institution like that
maybe don't have access. And even if you do
have access, there are still legal problems, because
in most places in the world, just because you
have the right to read, it doesn't necessarily mean
you have the right to copy it for the purposes
of text and data mining. And if you want
to, uh if you want to text data mining
at some point, you're copying it right, you've
got to get it from this over to your server
and you got to get it from uh you know
, one form to another. And and in many
cases this is prohibited by publishers terms of use.
I'm and so you're left in a position where you
have kind of kind of three choices. You can
, you can go to each publisher. Um,
and you can say, I would really like to
text data, mine, your publications and, and
in some cases this might be possible. But in
others, let's say you want all the papers on
a particular topic area. They're gonna be published across
many journals across many publishers. That's a lot of
letters you have to write, begging for access,
essentially a lot of access fees. They might try
and charge you extra to be able to text and
data line them. It's potentially a lot of lawyers
you might need to hire to read those contracts.
And if you're a PhD students are starting a project
and you want to do some text and data mining
, you don't have six months to years to write
to publishers and beg for access. And in the
UK. And in JAPAn at the moment, there
is a uh, explicit noncommercial text and data mining
exception in copyright law, which is really good.
And it's very promising. It's still not been very
heavily tested, facts, not been tested in the
UK courts, really, nobody's challenged anybody over yet
. Um, and you're still not off the hook
because at the end of the day, you're getting
the papers from the publishers. And if you want
to download 10,000 papers from a publisher, then they
can stop you from doing that by disabling access to
the server. They can cut off your university if
they want to. And this means that can be
very difficult to go ahead. We have a colleague
who works for content mine who's in the Netherlands who
downloads papers for the purposes of trying to identify academic
misconduct. And so he wants to download lots of
papers and search through lots of statistics that represents the
newspapers. And uh, yeah, he got his
university shut off from some publishers temporarily. Um,
and it was quickly reactivated. But then, you
know, the publishers right to the library and say
we don't want him doing this even though the library
pay for it. And the university right at the
library, right to university and the university right to
their researcher. And they say, please stop.
Um, and so that can make things very difficult
. And so the third way is you can use
open access material, which is what we typically bend
towards just because it really helps clarify the legal situation
. There are more and more open access journals out
there. And there's still a lot of very good
quality open access journals and their hybrid, um,
open access journals with papers that we can get hold
of. And this makes the legal situation a lot
clearer and it's a lot less stressful for researchers and
for people like us researching it because we know we're
not going to accidentally trip purpose in these terms of
use to commit copyright infringement. And then you have
a problem that's going to take us months to sort
out and so on. So for us, the
main problem is is actually access to the papers kind
of legal and in some cases then technical as well
. So where you get them from can be better
or worse. You know, not all publishers are
happy to get their papers and sometimes they come as
horrible, horrible formats. And so I think from
the history side, um much of the same um
, issues hold true. Um I think what's a
little different about the history side is that these materials
were not born digital. They existed in a print
version, Published version. And so what matters is
actually how they get digitized and by whom you know
. So so we could go back to 1890,
you know and look at 10 different newspaper titles.
And so in 18 90 they were all print,
they all look the same. But now some of
those may have ended up in the chronicling America collection
Library of Congress which is not only free and public
to anyone in the world but actually gives you access
to the complete text files through ocr of every single
one of these newspapers. But the same newspaper may
have ended up in the gale collection of the Proquest
collection or read X. Where um your your access
is limited by which libraries you subscribe to, you
know which which libraries what your library subscribes to.
And then the format is very constrained. So you
can't get the text files from from any of those
collections unless you go and buy them separately. And
so what you end up with is the kind of
political economy of the last 20 years of who has
digitized collections affecting what kind of source materials you can
look at for something that happened 140 years ago.
Um and obviously that changes if you chronologically move up
to the to the same time period and so echoing
ed, I'll make a very strong case for for
using open data. So again, Library of Congress
chronicling America in terms of newspapers. Enemy for medical
periodical is the Medical Heritage Library, which is kind
of a gateway to get into internet archive um gives
you remarkable access to. Again not only the pdf
versions of journals, but also to to um text
files and other kinds of materials. And so if
there are things in that collection, You know,
you have all kinds of possibilities to do things that
you can through others, although of course, in
the United States, the 1922 copyright date, you
know, it really affects how you're doing things.
And so, again, just in terms of challenges
, you know, there's a kind of what I'd
say, the technical side, in terms of what
kind of tools do you have and what can they
do? But there's a different source, sought source
question. And and what I've seen in some historical
text money project is they end up looking at sources
from a particular historical period, not because they're the
most interesting or important or significant sources at the time
, but because they're the ones people have access to
. You know, and so that becomes, it's
not that they're wrong. It's just that is affecting
the kind of judgment they're making about what happened in
the historical past, which is not normally what a
historian would do, historian would say. Well,
You know, what was the most influential newspaper?
That's the one I look at as opposed to which
one is full text searchable. Now. That's the
one I look at. I mean from the business
side, I think one of the challenges we're facing
right now is how to access the data that we
need. Uh, there's a lot of uh disclosure
data is a public available. You can crawl them
online. But if you try together the comprehensive coverage
, all the news has reported about a particular incident
. You have to rely on some commercial databases.
And if you're using the commercial databases for this news
, uh, like effective a database. For example
, if you try to cross the data from Factiva
, you're gonna shut you down immediately. Pretty much
you can detect that your chronic, so you have
to be very sensitive in terms of, you know
, the, what's the copyright issues there? And
then if you're gonna crawl instead of for you to
manually download all the news articles, you have to
crawl, you know, you know, meaningful fashion
, which means that, you know, you probably
want to desire crawler very creatively. Okay. So
which means you probably the pause a little bit like
maybe like a human being to really try to download
in some news articles. And so, you know
, if you can get all the data you need
, I think, uh, that's only the first
step. Okay. How to clean the data.
All right. How to pre processed data into the
format and how you can drag all these entities that
you want to you want to have. And if
you're going to engage in the classification kind of task
and how to label, you know, you have
to hire some human beings. Do the labelling,
right? What? This is like a negative events
. This is like a positive event, right?
This is like a fraud. This is non fraud
. So doing this kind of labelling is also very
, very labor intensive. Um, and so once
you get all the data pre processed into the right
format and then it's ready for the analytics. Data
analysis. They did analysis part is pretty, pretty
straightforward in my mind. There's tons tons of tools
out there that you can use. You can use
open source Softwares, you can use commercial Softwares.
And sometimes in Virginia Tech. We also have some
uh, education, you know, sata license right
to some of the the package that we can use
most of time that we in the business school we
use we use some open source, you know,
packages. Vika is the one we use a lot
. And then we also use uh python, you
know, there's a lot of pre build, you
know, packages are is another language you can use
if you do this kind of data mining, data
analysis. So you know, try to leverage those
and then um talking your model and then try to
see whether they really the data can tell a story
, you know, for your field research. I
think that's what we're trying to do. Uh you
know, having the data is one thing, but
I think you make sure that you can tell a
good story using the data you have. I think
it's another thing. So that's the uh things I
try to teach my students and make sure that you
can tell a good story behind the data. So
yes, uh complete the table when it comes to
answer this question, I don't really have any good
answers with the ones that I'm going to pose and
I wouldn't again, there are technical issues that you
have to deal with first. But ultimately, um
, I feel many people forget that even in this
land of big data, when you have access to
so many documents, there's still potential for sampling bias
. Right? And oftentimes people forget that thinking that
you're hiding behind the fact that you've got millions of
documents. But really at the end of the day
, that's a sub that is a step, potentially
a small sample in the larger population of all possible
documents you could be analyzing and you are very vulnerable
to biasing your sample if you don't think hard about
where you're collecting the data. But that saying that
um depending upon your goal of what you're trying to
do at every step of an analytical process, there
is going to be some source of uncertainty. And
if you're not taking on a probabilistic approach, which
most people are not because the data are so large
, it's hard to keep track of that uncertainty and
to really understand what that means with any implication of
whatever story you come up with. Right? So
, if you come up with the story of your
data, which is ideal right at the end of
the day, there's there's almost a choose your own
adventure kind of story. And then what is the
probability or association or however you want to define it
with each particular outcome. And so then that completely
feeds into these ethical questions about implications of the predictions
that you make or how to use those predictions right
and so forth in the line. So so too
, I think challenging but no really good answer necessarily
would be to keep in mind that even in the
land of big data that you are sampling still and
then also like how you quantify uncertainty and any implications
associated with that uncertainty if you can, you know
, what do you do? Thank you very much
. Um well, just ask us uh one more
question um to kind of wrap up a little and
then we've opened for you to add other questions after
that. So if you could maybe briefly touch on
what you think about TDM in the future, um
what's next for you and um what's left to explore
in TDM? How might you plan on using that
to support growth areas at VT. Such as the
new destination areas? Or it sounds like you're taking
a lot of interdisciplinary approaches all of you. So
that's certainly one area that seems promising. Sure.
So, so I'm not really sure about supporting players
that because I don't really know these growth areas are
. But first we were an organization that's entirely based
around text data mining. We see it is very
important for the future. Um it looks like economically
, it's very important. It's a huge chunk of
so projected revenue in the big data sector and it's
also going to be and very important for solving lots
of these problems, particularly the amount of data we
have to manage, grows and grows and grows.
And most of this data is unstructured text. It's
not mostly nice. Pretty done tables. I think
what what, what we hope to see in the
future is we really like to see more and more
open tools for for doing text and data mining,
not just the the tools for getting hold of data
, but also the tools are actually doing the analysis
. And we'd like to see these tools since they
open in the hands of more researchers, we'd like
to see them in the hands of people who are
, you know, first PhD students would like to
see high school students perhaps beginning to use them.
We'd like to see just citizen scientists having a go
and seeing what they can do with the data they
can get hold of. Um, and so that's
where we put a lot of our time and effort
into. We do our best to develop software that
we think will be useful and and make it openly
available. And also we come into outreach and things
like this where we come and talk to people and
we're running a workshop tomorrow where hopefully we're going to
be encouraging more people who don't quite know how to
get started in text and data mining of away in
and give them instruction to some of the tools that
might help solve some of their problems. So it's
always tricky to ask a historian about the future,
but three things, I think very quickly. One
I would I would absolutely eco um what time had
to say in terms of the possibility of creating more
tools? Um and I think here again, as
a as a as a teacher, um you know
, the more tools that are accessible to students and
that can be used by students um in their own
work on projects, the better, you know,
and and to have to be a situation where they're
learning to use tools, um but they're learning to
use tools as a way to think about information because
the tools will change. Um and actually many of
the online tools that I've used a year ago,
23 years ago are gone now. And so you
have to be able to think creatively about what you're
trying to do. Obviously, if you can do
your own coding and programming someone, you're you're not
quite in that situation. Um so that's one just
in terms of the tools, um you know,
I think text and data mining is a way to
to empower students and citizens scholars and the public more
generally. Um you know, there's a lot of
discussion now about algorithms and and and and the kind
of hidden ways in which, you know, really
important decisions about our lives are being made um without
us really knowing that's what's happening. And so I
think Texan data mining as a way to think about
, you know, how to process large amounts of
information, makes sense of it and extract meaning,
you know, is a really useful life skill um
that we're all gonna need um increasingly and I think
the last piece and this is kind of where I
started um you know, is to think about the
future, but also moving back and forth between the
past and the present. You know, and learning
as we look ahead. What is it we want
to understand? Um you know, there are historical
examples in our historical references that that can be informative
. Um I couldn't help thinking while you were talking
about ralph Nader. Um you know, and and
before he threw the florida election to George bush um
as a presidential candidate, you know, was transformative
with his attacks or his criticism of the american auto
industry. Um, you know, which to my
mind was part of a real transformation in terms of
how we think about individual rights and what are your
rights vis a vis a huge corporation. And so
in some ways, you know, what you see
, what you see in these, you know,
product reviews and online forums, to me is is
a direct continuity from that, you know, breakthrough
1960s and 1970s, which then brings you back to
the question of our the Israeli citizens who are writing
these at one point is the corporation come in and
start writing their own reviews. And how do you
you sort that out? And so I think there
are ways in which um knowing, you know,
the continuity and the and the movement between these different
periods can really help us be more educated users and
consumers of text and data mining in the future.
Um I think for me, uh one of the
things that I'm trying to do right now is the
trying to incorporate and uh some new tools, new
methods into the business domain applications, for example,
like there's a lot of a deep learning new tools
coming up already to analyze the images, analyze the
videos, right? So in the business domain we
have a lot of potential applications, especially in the
health care domains. And so currently we are working
with the Caribbean Clinic, try to using this deep
learning to analyze the lung cancer images, cT scans
, brain tumor scans e F D F G.
So this from the brain uh um these uh MRI
scans. So all of these can potentially leverage some
new data mining, takes money tools, methods that
we can use. Another thing we can think about
is, you know in terms of we need,
you know how I think about how we can apply
and this method in a very creative way to solve
some you know uh solving problems that you have not
been done before. Okay. Uh so if you
think about in the business disciplines, we have operation
management, we have capital market research, we have
marketing, you know social media analytics is very diverse
. You know application areas and some of the areas
are fully developed. For example in the financial areas
people have using this uh text money tools to analyze
the news and tweets and then try to do the
stock trading automatically. Right? Those are very advanced
already. But in certain other areas like for example
in the hotel and travel, tourism management areas,
not a lot of research has been done in this
area. That's why, you know, there's,
there's a lot of potential, uh, you can
work with the H. T. M. Faculty
members, you know, this is what I'm doing
right now. I'm trying to work with one of
the HTM Hotel and Tourism management faculty members and then
try to apply the text mining, data mining to
their online travel side reviews, Tripadvisor, yelp reviews
and then try to look at this data. You
know, are there any bias in this online reviews
? You know, are there any conclusions in terms
of the dimension between the review ratings, which they
think what it said on the, on the text
? Right. And then also, you know,
how those reviews really help the consumer to make decisions
. There's a lot of potential issues that could confound
this impact on this online reviews on the consumer decision
making. So we try to basically try to collaborate
more engaged, more collaborations across the college, within
the college and also across the college. And they
try to do this kind of a broader applications of
text and data mining. A lot of computer scientists
or information scientists try to build an information system so
people can use it. And I think people are
familiar with google and others that are widely used for
many purposes. So the question is, how do
we do that on a large scale to serve a
diverse set of needs that we've heard express here from
, from lots of people. So, the guitar
project I mentioned again, I want to invite people
if if you have interest in global events or trends
and want to play with text in various forms,
help us figure out what the needs are of different
groups and help us build a system that will adjust
to that and help us build good interfaces for different
communities. So we're proposing to have interfaces for school
Children. We have a group that's trying to do
virtual reality interfaces. We want to help librarians have
good interface is uh we want to have specific scholars
in different disciplines, be able to do their own
work. So how can we build some kind of
infrastructure, some kind of system that will help support
this. So the way we do this is by
thinking about it and by trying to have some theory
and some formalization under underneath this and to explore it
with people, uh to see how it can work
. So I want to end with one of the
things that we've tried to do to organize these information
systems concepts is to have a theory which we call
five S, which stands for stream structure, spaces
, scenarios, and societies. So here we've been
hearing about text which is essentially a stream and we've
talked about different structures of different levels of structuring.
Um We've heard about high dimensional spaces and probability spaces
and we certainly have talked about lots of scenarios,
things that people would do with this kind of information
, but ultimately it's for the societies, for all
the different people who are going to work with this
. How can we bring those together into a nice
framework in this system? So that, to me
is the big problem. Mm. Uh so we
definitely have some some ideas about where we're gonna extend
uh in terms of Virginia tech and research. So
so with that I'll mention the education of the curriculum
. I'm very much related to the destination area data
and decision sciences. And we're talking about these high
level analytics. But the real question is how do
we get it? So that people who don't pride
themselves and being quantitatively gifted can also have access to
information gained from text analytics. So, so I
think that there's a huge agenda that's there from a
pedagogical perspective of how can we make these high level
methods available to 18 year old students who don't even
know how to download a paper, let alone parse
it out and understand patterns and and even money and
even realize that text is a source of data.
Right? So so that would be where I would
say Virginia tech can be a leader. Okay,
so 11 I guess other aspects of the future that
I think about a lot is um we're in this
space right now where these tools are becoming more democratized
everyone down this line. That's basically thinking of ways
to improve, to create better tools, more powerful
tools, more publicly available tools and the maturity of
the software has been unprecedented lately. Even these deep
learning methods that are some of the more complex ideas
coming out of research the Softwares and incredibly well written
. So people, anyone with some coding background can
download google's deep learning software and start playing with it
. Um The danger with this or I guess it's
not really dangerous. There's two sides of this that
need to be considered. One is that more people
are going to be doing text and data mining and
other types of data mining. Um That's wonderful because
it's taking a lot of decision making out of this
sort of human intuition view and looking at real data
. But on the other hand, like Diana mentioned
, the data is not, the data is not
perfect and the models aren't perfect. So what can
happen is people make decisions and they say, oh
the data is backing me up on this decision and
you end up doing something that is it's really bad
for society. Really unfair to certain sub groups in
society. So there's a lot of challenges there that
some researchers are starting to take very seriously and I
think Virginia Tech can be a leader in that.
Yeah, So just to springboard off of that for
sure as we develop these, these ways to teach
uh data mining and text analytics, we need to
also develop true critical thinkers that can balance their personal
. They might not even be aware of personal biases
, biases, personal judgment with the information gained.
You know, that is a that is a hard
line to toe and important. Yeah, well thank
you very much. Um, so I feel like
we I had a really excellent overview of text and
data mining overall and some great examples and I really
appreciate hearing all of you talk about the role of
humans and society in relation to these, what we
consider as technical methods for approaching problems. Um,
well, I just want to add one thing here
for the VT libraries for all of you, our
panelists and our audience. Um, please keep us
in mind as a partner if this is the kind
of project that you'd like to work on and you
are having trouble accessing sources that may require some mediation
between a publisher or other entity or you're just wondering
how to get a particular type of thing. Um
, please let us know, reach out and we'll
see how we can work with you and beyond that
. I'd like to say thank you very much to
all our panelists. Really enjoyed hearing all your stories
and your, your methods for approaching these problems.
Thank you very much. Yeah.