talking. Uh, since the clock says one Oh
, one. Um, welcome to the second session
, second session, and the additional humanities and social
sciences. Uh, digital discussions, um, looking
around the room and see where the screen is.
Uh, my name is Tom Ewing, and I'm
the associate dean for research and graduate studies in the
College of Liberal Arts and Human Sciences. And I'm
also a professor in the history department. Um,
and, um, this is a moderately informal discussion
. Um, so we're gonna save plenty of time
at the end, Um, for questions. I've
got a couple announcements to make before we begin.
And perhaps the most important, if also unfortunate one
is, um there's actually a class that begins in
this room at two o'clock. So we will need
to be out by 1. 50. Um,
so we'll keep track of the time, and And
if we need to continue discussion out there in the
in the construction zone, um, we can certainly
do that. Uh, just in terms of upcoming
events in this series. Um, I was asked
to remind you that the registration for that camp at
University of Virginia November 8th and ninth is still open
, although there on the waiting list. So you
should contact Quinn or per team if you're interested.
The next session in this series is October 8th.
Michal Kosinski. Um, we'll be talking here and
then the following Friday, October 25th. There will
be a session as part of open access week.
Um, and there's more information about those events on
the library website. Okay, so wrong button.
There we go. Um, this is a project
that we've been working on, um, 2.5 years
, I guess. On and off, it's hard
to keep track. Um, there are four of
us here who are principal investigators on this project.
Three who will be presenting today are discussing today Bernie's
house Professor, Department of English, and Aaron Rama
Krishna, Professor of department computer science. Bruce Bencic
from University of Libraries is also principal investigator. But
he is, um I believe teaching a class this
afternoon. Um, we have had four gs work
on this project over the course of over the whole
time. Michelle. Sarah, Who's here? Um
, left us for greener pastures. Kathleen Kerr,
who is in Florida, uh, has been working
on this for just about a year now, Um
, and she'll be presenting as part of this session
. Also, in some God who was, uh
, G in the computer science department is a conference
, so she will not be here in presenting,
but this just gives some sense of who's involved.
Um, we do have funding for this project from
the National Endowment for the Humanities As part of the
digging into Data Challenge. It's a two year grant
and ends this December. Um, we've also received
cost share, uh, contributions from the Institute for
Society Culture Environment Institute for Critical Technology, Advanced Science
here at Virginia Tech, as well as the departments
of English Computer Science and History. The focus of
our work going button is the 1918 Spanish flu epidemic
. Um, and I particularly like this map because
, um, it's taken from a 1919 publication,
but it shows very nicely the spread of the influenza
, um, over several periods of time. Um
, and it's been one that I've used, you
know frequently and refer to we're going to talk a
lot about data visualizations today. This is a good
example of data visualizations in 1919, the red titles
on There are some weekly newspapers that I'm gonna be
talking about later. Um, it did get a
little squished in the movement around from place to place
. Um, I'm going to start with this example
. This is from the Iron County Record, which
is a weekly newspaper in Cedar City, Utah.
Um, I like this editorial a lot. Um
, because of the title, I like the title
. Influenza is serious. It's a nice reminder of
the of the kind of way that this disease was
perceived at the time. It's also unusual in the
context middle of October, because it's saying that the
the costs of the epidemic are actually higher than the
costs of the war. Um, only, um
, two iron County people have died in at the
front and already to have died from the influenza.
And this is a kind of recurring question among historians
about how the influenza was perceived, given the wartime
context. And so this is a good primary source
from the middle of the influenza, truly addressing that
question quite directly. Whereas Iron County, Utah.
Oh, sorry. This is this is the full
page showing you where that editorial shows up. Uh
, iron County. Cedar City is the capital.
It's in the southwest corner of Utah. Um,
as you can see, the population in 1918 wasn't
very much. Is about a third of the size
of Montgomery County at that time. It's still not
very large. It's a fairly sparsely populated Um,
if you've ever flown from, uh, the East
Coast to Los Angeles and about the last hour of
the flight, you're looking out the window wondering if
there's anybody alive down there. That's that's the part
of the country we're talking about. Really? Excellent
. I'll stop saying disparaging things about it. Why
does it matter? Other than the 47,000 people who
live there? Um, it's one of the newspapers
that's available, um, in digitized format from the
chronicling America Collection at the Library of Congress. Um
, and this is a free, publicly available website
. Um, as long as the government is functioning
, I'm not sure right now, Uh, library
Congress is closed, so their website still seems to
be working. But that may not last. Um
, and it's very easy to find articles about the
influenza. You go in the database and you select
that newspaper you type in a keyword like influenza.
You set the date range, Um, and you
come back with about 50 results. Again, this
is a weekly newspaper. So each of the issues
during the height of the epidemic from mid early October
through December had in most cases several pages that had
articles about our mentions of influenza. And this database
is nice because if you look at the images up
there, all those little orange blocks spots are places
where the word influenza up here is on the page
. So it's very easy to look at this and
say OK, that third one over from the left
on the top row is a front page, and
those are all mentions of influenza, so it's a
pretty quick, easy way to look at that.
One of the things we've done in this project called
Topic modeling with Segmentation, is actually doing very much
the same thing. It's looking across this title all
of the text for a certain period of time,
and it's identifying the words that appear in proximity to
the keywords we identified, which, in this case
, the influenza epidemic I think disease, a few
others, and then it's clustering those based on their
frequency and proximity. So and then it divides it
up into segments. So this is one that goes
from late August through October 18th issue I just mentioned
. And then the next one goes from middle of
October through December. I'm not gonna talk a lot
more about this now because Kathy's gonna talk about that
, um, in her session. But it's a
very good way to kind of pull out key terms
and quickly see not only what they are about how
they connect with each other. The part I'm going
to talk about is, um, related. But
it's a very different method that we're looking at,
which is tone classification. So if we go back
to my example initially, of those results that come
up and you can see where the orange boxes are
if you look at the same issue that I had
up earlier, there is no orange box in that
editorial that had the headline, Influenza is serious.
There's a little orange box over there in the second
column, but there isn't one. And the reason
for that is the OCR in the chronicling America Collection
is not very good. It is, in fact
, bad. Um and so the headline that should
read influenza is serious, actually reads infants or something
. And the next line, the first line of
editorial, there's an end instead of an A.
And so a search for influenza spelled correctly doesn't find
it anywhere in this article. Now, if we
look for epidemic, that would have shown up a
few places, but it doesn't show up here.
And what you actually need to do is clean up
the text in order to actually find these terms as
they should be spelled. And so the text on
the right is one that I have manually retyped by
looking at the image of the newspaper. Um,
and this was a recurrent problem going to come back
to this. But you can see there's an awful
lot of misspelled truncated garbage words, Um, in
this and there are others that are a whole lot
worse. Okay, so what? We were interested
in this project and we got interesting fairly early on
was finding some way to automatically classify the sentences that
we were finding in these articles. We came up
with these four categories. We actually started with eight
and we kind of worked our way backwards down to
four. That to us provided a pretty good range
of the kinds of tones that we were seeing as
we read these articles and looked at the text.
And so what we wanted was an automated way to
go through this text and sort sentences out into these
categories. We started by doing a lot of manual
coding because we were trying to train a classifier to
do this. And Kathy, maybe grimacing at the
moment because she was the one who did all the
cleaning up of this text from one newspaper, the
Washington Herald, which is available from chronicling America.
This is just one example of what we did.
We had four classifiers, um, and we each
went through these and marked them by one of those
four categories, and then we compare them. This
is actually a little bit of a misrepresentative sample because
it looks like we were in agreement. We were
in agreement. Sometimes we were in disagreement a lot
of the time, and there were actually some sense
. There are many senses where three of us classified
in one way, one classified at a different.
There were many that were split two and two.
There were actually a handful of sentences that the four
of us each classified differently. Um, I don't
have those up there, but we can talk about
what those were. And this was the distribution we
came up with. About half of the sentences that
we looked at were what we would consider explanatory.
They were essentially neutral. They were conveying information.
Um, about a quarter were reassuring. Okay?
They provided some sense of things were getting better.
Um, and the smaller percentage about 15% were warning
. And then about 10%. We classified as alarmist
. They had some sense of this is really bad
. And we have to do something about it.
Influences serious. That kind of a message. Um
, this is the same information, just in a
slightly different version. And those were the four coders
. So Kathleen, Bernice, Bruce and I did
this. Um, what we then did is we
took the classifier and we applied it to sentences from
a sample of eight weekly newspapers. And if you
think back to that map I had at the beginning
, those with the eight titles I went through and
identify texts. I picked articles about the impact of
influenza at a local level. Um, editorials.
If there were any, um, statements from local
public health people, whether they were municipal or county
. Um, I did not include any kind of
descriptions of individual victims were obituaries. Although where there
were a lot of those and through this process,
I came up with about 700 sentences from these eight
newspapers, and you can see the distribution there.
We then ran. Well, not we Soma,
who knows how to do this? Ran the classifier
on these sentences, and this was the distribution she
came up with. It's similar to what we found
with manual coding. Uh, slightly different. Little
bit more than half come up as explanatory. Um
, just under, uh, right. About 20%
come up is reassuring. A much smaller percentage,
less than 2% come up as alarmist than 16% come
up as warning and you can see the distribution there
for each of those titles. And this is similar
information just showing you across the the bar graph.
Um and these were some sentences also from the Iron
County records that the automatic classifier identified by these categories
. Okay, so you can see some warnings.
You can see some explanatory. You can see some
reassuring. Um, I have not checked. Compared
the accuracy of this classifier with emanuel coding. Um
, for a variety of reasons. When we did
this previously, we were getting a rate of about
70% which is against something we can, um,
come back and talk about. There are a lot
of issues with this method. Um, the one
I mentioned early in terms of the disagreement among the
coders. Um, certainly the poor text derived from
the OCR and the substantial time needed to prepare the
text to make it clean is a real issue.
One of the things we're trying to figure out is
what's the right level of analysis? Do you really
want to look at sentences? It Would it be
better to look at phrases that are pulled out from
those are longer, more contextualized sentences in the whole
number of articles? Um, we can only do
the tone classification on newspapers in the chronicling America collection
. Um and then this whole question of the accuracy
rate of the of the classifier and what this has
brought me back to is something that's a sent a
passage in a recent publication called Big Data, which
really calls into question what we're doing here in terms
of kind of expert analysis. I'm gonna come back
. We might come back to this later on the
discussion, Um, in terms of what's going on
. But it really calls and really asks whether,
as you scale up the level of analysis, you
no longer need kind of the precision in understanding that
the context and the meeting All right, I'm going
to stop there and turn it over to Kathy.
Kathy, if you will say something, I will
advance the slides. Hey, what if I will
One. If you give me a wave or say
next slide, I will do that. Okay.
All right. So I'll get started with this,
but you need me to stop. You can't hear
me or some such thing just because I can pretty
much anybody else. All right. All right.
So what, actually, is I'm going to talk
to you a little bit about our case, which
initially we designed to answer some content related questions.
Um, using the data mining 15 daily newspapers from
the chronicling American database and one newspaper from the prairie
provinces. Can you switch to the next place?
I can't see the list of papers up there.
I don't really data. Do you want me to
turn it? So you're not facing yourself? No
, no, no, no. That's okay.
Alright. Anyway, we selected these papers in this
table based on several criteria, we considered, uh
, representation of geographic diversity of, um, game
publication, urban circulation. And then, of course
, the availability of digitized versions of the newspaper.
Essentially what? We wanted to know how newspapers reported
on the influenza nationally, regionally and locally. And
we also wanted to understand what characterized of epidemic related
reporting, which we identified as the period when the
most, um, word clusters in a segment which
will show you here in a second, Um,
were clearly about influenza. However, as we went
about our analyses, what we found were some very
interesting methods. Related questions quickly emerged remaining the topic
modeling and segmentation output. That's what I'm going to
focus on on just read little discussion and specifically on
our reporting at the aggregate level differs from reporting in
the individual newspapers in terms of what, President?
If the segments that word cluster. So to generate
these topic mildly and segmentation outlets, I'm trying to
go to the next slide. Uh, we extracted
tests from newspapers based on our search terms in Lanza
rib flu epidemic. And then we ran topic modeling
and segmentation algorithm on each of the papers that were
in at that table that will show you. And
then we ran the algorithm on all the papers together
, which we'll call the office team papers out boat
should be like they had their. So the task
was that you see in these segments represent the 20
most frequent words in a cop it and you'll notice
that some words are much larger than others. Um
, this is because those were her with greater frequency
in that particular segment or that particular topic. Sorry
. Private segment. Um, so with that in
court, you know, here is that the frequency
, the art norms within a segment or across the
segments are only normally within those, um, specific
discrete and clouds. So this this particular supply shows
what we identified as the peak of reporting for the
all six papers output as well as the segment immediately
following here. Um, and we identified this peak
as October 9th through October 23rd, 1918. And
as you can see, if you look at that
red cloud in the upper left hand quarter, um
, you'll see tone words like seven die, Um
, as well as these terms like kill pneumonia,
Spanish and influenza You can at but not quite so
. What we identify are a number of very interesting
phenomena and our analyses various out differences in peak 40
for influenza and the intensity of revolution. We noticed
the appearance of triple I characters, that football reference
at the aggregate level and the absence of quarantine weight
forces at that level. So, as previously noted
, that the peak reporting at the aggregate level approximately
two weeks in duration. However, for most of
the individual newspapers reporting extended for 30 days and sometimes
longer. Although there were some exceptions, The Bismarck
Tribune, for example, um, at a in
its probably millions segmentation output shows that he can influence
our reporting from November 7th 1918 to November 21st.
But as it turns out, peak reporting was actually
in weight. Five power. Um, if you
see there, That advertisement on the right hand side
of the screen. The newspaper ran a series of
these advertising for subscription throughout the month of November,
and it appears that what they did was to shift
, um, what appears to be the peak?
So it looks like it's in November wins, actually
in October. All right, you could forward to
the next slide. All right, So our analysis
also shows that the topic modeling and segmentation algorithm is
able to detect differences and shifts in the reporting of
information about the academic sometimes, Um, bullshit.
For example, this segment which represents the recording of
the influenza and Evanston the morning bulletins, suggests that
even the epidemic peace, even as keep the reporting
in this Canadian paper, was not as dominant as
it was in the U. S. Papers.
Salmon, uh, Michigan. The only three of
the five clusters are clearly about influenza, while during
keep reporting in the U. S. Paper the
terms the search drugs are frequent enough that they appeared
in various combinations and usually the words were quite large
. Um, in all of the five clusters in
the segment, if you look forward to the next
slide, please. Okay, So the algorithm also
suggests that U S newspapers reported on different topics and
to varying degrees. Um, for example, while
, uh, football related terms here in the all
15 papers, you can see here in the lower
right hand corner, we're about them. But red
at the height of influenza, uh, football trade
, according was not necessarily a national phenomenon. Uh
, outputs for nine out of the individual papers don't
contain any terms related to football reporting, but it
still made at us to aggregate level. That voice
, okay, and a very similar phenomena here with
this triple I character set is really very interesting.
Um, we thought that at first this relates to
the very bad of yards almost talking about earlier,
um, and specifically in the Albuquerque Evening Herald.
Um, there were there were a lot of these
characters that showed up in the output. Um,
and you see here this is a search I did
in chronicling America just on this site. Hopefully characters
that throughout the over the period of analysis and all
of those red glasses obviously are occurrences of the characters
that but as it turns out, there are quite
a number of other papers that this character that shows
up, for example, lines that the that the
OCR doesn't know what to do with, um,
they show up at this character set. And so
what we end up with in the aggregate outlet is
, uh, these characters that showing up in almost
50% of the 20 Okay, you can look in
the next life. Okay, So quarantine is a
little bit of a contrary example. We expected it
to appear in the all 15 ounce papers out,
but But it didn. So we assume that this
them that it didn't rise to the level of prominent
influence related this courses. But if you look at
these, um, these newspaper articles, um,
they are from the busy daily review and the output
of that particular paper, um, frequent occurrences of
the term quarantine. Um, and they show up
in the segments both before, during and after the
influenza epidemic. Which suggests, of course, that
the that paper was recording to a public. It's
already very familiar with, uh, public health measures
such as quarantine when the epidemic, I guess like
Okay, so what we decided is the attitude is
The most important findings of this particular study really involves
the differences scale mates in the analysis of our data
mining output, it's very clear that some papers influenced
the output at the aggregate level more than others,
uh, some recording at the local level right to
significant at the national level. However, other recording
doesn't appear in the office behaviors out. But even
though we might expect that it was because these activities
were reported and across the country, so what?
We expected the supply lines absent from the outputs of
the case of the quarantine dentures and some topics that
we wouldn't expect that the average level such as,
uh, recording on football, actually do appear.
Uh, So what this suggests to us is that
any analysis of topic modeling and outputs for the aggregated
sources have to take into account the possibility of recording
that only appears to be condemned for it. But
in fact may not be, um, go,
in short, the scale of data mining matter that
affects our generated by the topic. Traveling in segmentation
in what we thought were very interesting and sometimes very
surprising way. All right. Thank you. Cathy
. Do you want me to do this? You
can do that. Yeah. Okay. So I'm
just going to read this talk. Visualization choices are
not neutral. They are meant to convey the data
outputs of the computer computational algorithms. But they necessitate
an understanding of computational conventions and specific data input decisions
as well as tacit knowledge of their own conventions to
be legible to readers and researchers. In addition,
the visualizations are explicitly rhetorical persuading the researcher towards particular
interpretive decisions. So in thinking about visualization, were
mostly in this particular study. We're not thinking about
making visualizations for, um, for the public.
We're talking about the visualizations as renderings of the data
mining outputs that are used by researchers to create interpretations
. The 1918 influenza pandemic occurred at an important juncture
in the history of vaccine development before it was possible
to create vaccines for influenza viruses. But after some
vaccinations have been developed for other diseases, numerous vaccines
were created during the deadly second wave of in the
fall of 1918, although there's no definitive evidence that
any were actually effective. Um, positive newspaper reporting
notwithstanding, Nevertheless, there was significant reporting, perhaps
because vaccines at the time represent developing confidence in the
public that medicine could in fact provide that benefits in
terms of health. After the war, vaccines became
an element of medical practice evident in advertising. And
that's this particular advertisement here from the ST Joseph's Observer
, where this doctor, um, advertises vaccines as
part of the services that he provides. And and
this is part of demonstrating how modern a doctor he
is, right? So he has vaccines available.
So it's important to note that we did not set
out to ask or answer any questions about visualization and
the research project overall. Rather, these questions arose
during the analysis of data mining outputs, by which
point decisions related to data mining algorithms had already been
made as a result are aimed for the project expanded
to include the analysis of visualization conventions as they relate
to data mining outputs. Generally, we seek to
understand the would you seek to better understand the persuasive
effects of visualization conventions, the underlying assumptions that influence
or interfere with researchers, interpretations of visualizations and how
different design choices suggest different interpretive possibilities for humanists.
The general thrust of this particular study is not to
consider how given visualization can be modified to meet a
user specific requirements, but to move toward an analysis
of the inevitably persuasive effects of any visualization. So
next life, please. There's a wealth of scholarship
relating to data visualization, visual and digital rhetoric and
big data. But there is little research that brings
these disciplines together or examines the implications of big data
visualization. So one important distinction raised by our review
of the literature is the rationalist social constructionist divide.
A rationalist approach suggests that the goal is to represent
data credibly and accurately. The ideal is achievable with
the right format. The social constructionist approach, on
the other hand, suggests the right format is negotiated
in relation to the needs of both producer and user
, emphasizing design conventions and their naturalization. From the
social constructionist perspective, Normalizing practices often results in an
uncritical and unquestioning acceptance of visualizing conventions without an exploration
of their persuasive effects. So restorations will almost always
fall into the social constructionist camp, and this causes
interdisciplinary contentions because predictably, computer scientists tend to be
rationalists. So if rhetoric shins are by default and
social constructionist right treating, we treat the data mining
conventions as naturalizations computer scientists want to adjust the algorithms
to display the data better. So I think of
this as the let's fix it approach. So if
if the restorations bring up an issue of persuasiveness or
or the way in which conventions are naturalized, computer
scientists often respond and say we can fix that.
We can change that. Um, so that's been
an interesting experience on our, uh, in our
research group. We applied the integrated topic Model Topic
modeling and segmentation algorithm to 90 titles from January 1
1918 to December 31st, 1919. We extracted text
chunks from weekly and daily papers, right? So
these 90 papers in the chronicling America database, using
search terms, vaccination vaccine and inoculation and their roots
, we ran two extractions first, excluding text chunks
containing the terms black leg or cholera, because those
at the time referred to Black leg was a disease
in cattle for which there was a vaccine and there
was a hog cholera epidemic in the Midwest. So
we excluded those terms because we didn't really want to
know about. They actually were really overrepresented in the
data. Then we ran another extraction. We also
blocked the term serum. Um, and the outputs
of the first extraction we labeled no black leg,
no cholera. And the second one we labeled,
no one wanted terms. So next slide, please
. We found out that the decision to exclude serum
um, uh, basically introduced problems. It was
based on an initial observation that it also seemed overrepresented
in the data output. Technically, serum and vaccines
are different things. A serum is created by separating
blood into its solid and liquid components, so the
antibodies to microbes can be utilized for their immunizing properties
. So Thomas, sick with influenza, he gets
better. We withdraw some of his blood, we
make a serum out of it. And we injected
into me with the hopes that it will make me
better because his antibodies will help my immune system.
That's a serum. A vaccine is different. Vaccine
is some kind of inactivated or killed form of the
actual microbe that's injected in me to prevent disease.
However, at the time, serum was sometimes used
as a synonym for vaccine, although not all the
time. So we ran into problems thinking we could
we could just exclude serum. So this slide includes
an example of the distinction between serum and vaccine being
upheld in reporting. So this there's two discussion here
that talks about the difference between vaccine and serum.
But if you go to the next slide, we
see an article that uses two terms indiscriminately. So
our decision to exclude serum affected the data outputs because
we lost some reporting on the influenza vaccine that,
like this article in which the first part refers to
a vaccine developed at the Mayo Clinic as a serum
. Um, significant. So there's this Doctor Edward
Rosenau developed a vaccine at the Mayo Clinic that was
used in Chicago, and it was reported on through
a lot of these newspapers. But when we lost
reporting on that word cluster, we lost that word
cluster in the outputs that excluded serum. Um,
on the other hand, when we excluded serum,
we actually saw more mentions of smallpox, which seems
to have a negative relationship or an inverse relationship to
serum in this period. Okay, next slide.
So So anyway, So that was some of the
discussion to some of the impact we had about sort
of input decisions that we sort of realized over the
course of doing the analysis, that of the significance
of the effects that they actually had on what we
were seeing in the tag in the word clusters.
So we looked at, um and this is an
example here. This so the list of words right
there The yellow list is the word cluster that refers
to the Mayo Clinic vaccine and the use of it
in Chicago. That's from the no black, like
no collar extraction because we didn't have that word cluster
at all in the other extraction. Alright, next
slide. So we use three different forms of visualization
. The three visualizations each represent the same topic modeling
and segmentation outputs. Um, they're all indexes in
some way. And I thank Chris wherever he is
for pointing that out to me. Um, so
these are all different ways of indexing the reporting,
so I'm just gonna give you a brief overview.
This is the typical output of topic modeling in tag
clouds. We see this all the time on the
Internet and in other places. Um, the size
of each word represents its relative frequency within the cluster
, and the words are arrayed in a non purposeful
manner. The fact that two words are next to
each other means nothing in these tag clouds, they
facilitate it. Now this is our sort of our
initial gambit. At an interpretation, the tag clouds
facilitated an analysis that is bounded and oriented toward narrative
. Within the segments, each tag cloud seems to
tell a particular story. The researcher is motivated to
search within the cloud for thematically related words and to
identify the topic that the cluster of the words in
the cloud represents, that is, the researcher works
to narrate a coherent story about the reporting that's represented
in each tag cloud. Next slide, please.
So this is the novel, um, visualization themed
Delta that, um, Salama developed with walkers Javad
who and his advisor at Purdue when he was a
graduate student. There. Um, this is this
represents topic modeling results over time. So the segments
are actually a raid on the vertical cluster. The
vertical lines that's that's the segment in terms of time
and then the lie that these trend lines are variable
with lines that branch and merge, and they follow
the term through contiguous segments, demonstrating the persistence or
discrete nous of particular word clusters over time, the
thickness of the line renders the frequency of the term
within its cluster. But because the data is not
normed across segments, the changing thickness of the line
can be deceiving. Just because the line is thicker
at one segment than a different one doesn't mean it's
actually more frequent there. It only means it's more
frequent within its cluster there than other terms that are
not as thick in that same cluster. So the
theme Delta Visualization highlights the flow of discrete words across
an entire period of analysis. It makes more visible
how a particular word moves through groupings of words across
sentence segments. In addition, it draws the eye
across this visualization to emphasize the temporal aspects of words
. Interpretation of this form of visualization thus traces the
connection across time, but de emphasizes narrative since the
trend lines become the focus, not the clusters of
words. And the researcher starts with a macro level
. Look at the trend lines, which is what
I've offered you here, so you're kind of back
. You can't really see the words, but you
can see how the groups of terms move either together
or separately, across the segments um The next slide
then shows you how that it's actually interactive. So
on the when you're using theme Delta, when you
take your cursor and put it over a term,
it will turn that trend line with that turn another
with that term in in another colour. So you
can actually see how that individual term trends across the
segments. So the last visualization choices the word frequency
list, and that's on the next slide. Um
, and I have to say, this is sort
of the one that I didn't really invent this one
, but I insisted that we look at it because
I can't stand tag clouds. I don't know what
to do with them. I would go to the
meetings and they would, you know, Cathy and
Tom would be saying, Oh, you know,
I can see this and this and I'll be like
, What? How am I supposed to read this
damn thing? So this is the one that I
liked because I felt like I could manipulate it in
ways that I could actually use it and created analysis
by by reading through it in a more, um
, I don't know. For me, the word
frequency list of the ones that I go to first
that I like. And Kathy doesn't like them.
So So this in these lists, the most frequent
word in a cluster is at the top. These
are the most index ical, right? They look
most like an index. They really, um your
you pay attention to to the words, um,
the frequency of words and distribution is very clear in
the tag clouds. Of course, if words are
of similar size, you can't really tell necessarily which
one is more frequent than the next one. This
one makes it very clear what the most frequent word
is and on down they allow. The research could
easily identify the key words in each cluster. They'll
also allow you mean if you array them in a
regular old word processing program like Microsoft Word. You
can do what I did here, which is to
highlight words in categories according to color. And then
you can actually just look through the your list and
see what I discovered. We're not showing it here
, but in this one, blue terms are symptom
terms and yellow terms, our treatment or prevention terms
. And if you have a cluster that is all
blue and yellow. That's an ad, because it
all it has is symptoms and responses. Red ones
here are administrative terms. And so when you've got
those in there with the yellow, it's usually about
public health measures. So, um what? So
these and these kind of allow us to focus again
quickly once you once you've actually coded them? And
there probably is a way somehow to write an algorithm
to do that, to look through and see what
kind of, um of, uh, reporting that
you're looking at. And that's where we are with
our analysis so far in understanding the rhetorical impacts of
different ways of visualizing the same data outputs. All
right, so let's I just want to finish with
this. We have a research symposium scheduled in two
weeks at the Research Center in Arlington. Um,
and it's actually gonna be broadcast through BB I here
on campus. If anyone's interested, going over there
and watching you're certainly welcome to I would highlight our
keynote speakers. Uh, David Morin's Jeffery Taubenberger,
who are two leading virologists who we learned yesterday are
exempt from the furloughs, so they actually will be
working, unlike a bunch of other people who may
not be able to come. Um, but we'll
open up now for questions. Um, like I
said, we need I need to be out of
here before too, but open up to your questions
, suggestions, anything you want. Yeah, Findings
. Or I'm holding mhm like the first clue cow
. No. So the one thing it would be
easier to do a lot of the analysis because most
of the reporting on those things is already digital,
right? So we wouldn't. So, in fact
, you could do You could actually train a classifier
and do the tone analysis far more effectively because we
wouldn't be dealing with the bad OCR. So ostensibly
you could like New York take a newspapers, or
you could do other forms of, you know,
social media, but I think that okay, pictures
. Yeah, right. Oh, other people are
doing I mean, it's called what? Well,
now infant ideologies Gunthers turn. But there's an epidemic
intelligence. There's a whole field where they're looking at
, like twitter feeds and social media to try and
figure out, and Google searches to try and figure
out locate where people are um, looking for information
about certain illnesses and then trying to see if that
matches up with actual outbreaks. Yeah, yeah,
13%. Right. But they're but they're really looking
in epidemic intelligence. They're really trying to, like
, locate sort of newly emerging diseases like and I
don't know if they've done it with, like,
the Middle Eastern respiratory syndrome, which is right now
going on in in Saudi Arabia. But they're also
very much interested in, you know, h five
n one. Of course, they're interested in the
emerging ones they're afraid of. Yeah. Question the
actress don't have 30. Yeah, problem. But
can I hear? So I can't hear you all
very well. I'll repeat the questions. The question
is how one tries to cope with mystical counts based
on dirty OCR. Uh huh. What I've observed
in working is that accuracy is not inaccuracy is not
evenly distributed. There's some words that it almost always
gets right. There are some words that it almost
always wrong and words with capital letters, words with
ligatures, and so on and so forth are real
candidates for inaccuracy. I was wondering if there might
be an algorithm one could used to try and correct
for you. Look at a string of characters and
you try and come up with a coefficient of how
likely you're unlikely. Is it? Yeah, yesterday
or to be God. And one could use that
to try and balance this sort of, you know
, if anyone has worked with that kind of project
, Yeah, so there's a lot of work in
in statistical spelling correction, right? I mean,
it happens when you use Google, right that we're
all work. It will tell him mean this,
right? It's not really having any deep knowledge of
language of anything, but it's basically knows that it's
more likely that people search for this than that right
tonight. And it's asking you if you want to
make the connection. So there's a lot of words
that take a reference document that is known to have
much better, you know, because geography, the
news that correct something else that as much for her
performance like you said accents. And, you know
, some nonstandard usage is in English. Those are
pretty difficult, correct. But over time these to
start addressing, they start chipping away at different pieces
of the problem, and they keep getting better.
But I don't think there's one universal solution fully traps
that. The one thing that these tools open up
is that you don't have access to to a large
database of text documents, so you could get away
with doing a lot of shadow level analysis and basically
compensating for that by the sheer volume of data that
so you know, you don't know good understanding of
grammar or language structures and so on. It's just
that, you know, this word seems much more
likely to be used in conjunction with this other world
. That's probably what it meant, right? So
you could people do those types of sensical models,
and there's a lot of research in that space competition
. Linguistics. Is there a sense of how wide
the corrections would be or how accurate or inaccurate a
normed sets with? I don't know, but but
people do research in the space they deliberately introduced on
Earth, right? So they take a document and
they artificially, you know, commute or change some
characters. Let's see whether you'll be able to retrieve
those you know, tell us how they do it
. I don't know exactly in terms of politicians software
they wanted, but they're working. Yeah. Yeah
. What are you looking at? Yeah, go
ahead. Uh huh. Have a question about maybe
start with the comment. I feel like you may
be submerged or backgrounded fundamental theories and models that you
have possibly to facilitate, You know, the interdisciplinary
collaboration. And but I'm interested, right? Like
, uh, what were your models for any of
you individually disciplinary or otherwise, that informed how you
go about it. Seems like there was a good
deal of pragmatic in stream, in some cases,
ad hoc decision making. But then these models that
you use and we operate in our disciplines To what
extent did those come out in conversations and what to
prioritize and how to advance? Well, I think
in a couple days I mean, I think 11
real problem is none of us were experts on the
1918 influenza when we proposed to grant, or six
months later, when we got it. So we
actually had to learn a lot of this as we
were starting the process and at the same time figuring
out these methods. And we probably spent a year
trying to figure out how to clean up the OCR
, you know, pursuing kind of different options before
ultimately deciding not to do any of them. You
know, I think from my perspective as a historian
, the question I keep coming back to is how
do these methods help me to understand the spread of
this disease, how people reacted? You know what
the kind of impact was on society. And I'm
I'm coming down kind of with a mixed reaction.
You know, part of me thinks if we spent
this money in this much time just reading the damn
newspapers, we would actually know more about 1918.
But, you know, we wouldn't have developed these
other kinds of methods in ways that Nathan suggested that
might then apply to other kinds of diseases and other
kinds of information networks. So that's kind of how
I came at it from a history background. I
want to say I think that one of the one
of the things that happens is that we tend to
think it's like the same conversation about online, right
? Let's scale up, and we're going to have
economies of scale, right? I think that I
was just asking Kathy if she could hear. Now
, uh, here's a little bit better, so
we have to use the mix. That's what Lily
So, just like with online, like so there's
not actually a huge economy of scale here, like
in order to actually do this work well, we
have to have a phalanx of graduate students to be
transcribing the OCR with this particular project and the bad
OCR to actually be reading the actual articles and creating
our own digitized text to then data mine. So
that's super expensive, and you can't really do that
. And so I think that I think I think
these methods hold are interesting and hold a lot of
promise. But they may not be in terms of
person hours. They're not a substitute for the individual
research. I mean, it's not like you get
you can you can affect study a lot more texts
, but it actually takes a lot more people.
So it's not like you can just replace the people
doing the analysis with the computers. And I think
that's something that the, you know, the cultural
mix people. The question is, how much do
you? Can you really, you know, replace
and, you know, art no rain is always
telling us Well, you know, topic modeling segmentation
is not meant to replace. Analysis is meant to
tell you where to go to analyze. So that's
one way to think about it. The other reason
is, you know, we're studying historical newspapers,
and whatever you want to do with this historical data
is very difficult, even even if it's manual.
Right? So on the other hand, you take
any modern text which is obviously already going to be
available in text form. So you're going to avoid
some of these initial problems that we're talking about.
Then you can get down to the details of this
is really helping us with this. You know the
distinction between close reading and open reading, right?
So I think it's best to not think of any
of this as a substitute for humanists and scholars.
But you can employ you the big picture, all
of your you know, the corpus. So one
of them help you to narrow down on what to
look for next. I can't even help you figure
out what are the questions that you should be asking
then. Then you start looking at them much closer
and so on. Michelle. Yeah, I had
a question actually trying to relate the pathology from the
tone analysis to what's going on. So the tone
analysis methodology it makes sense, uh, interpreter reliability
and how you're trying to classify it. And you're
trying to really categorize a phrase or a sentence.
When you do the word clouds and the word frequency
West and the theme Delta, it feels like what
you're interpreting is word rather than me. And so
, if the interpretive goal is to in the end
, have the highlighted list like you have Bernie's where
you're looking at stories of conversation, then it may
be interesting to see if you can apply the tone
analysis methodology to become a category analysis methodology where you're
thinking of the categories of the words rather than the
word frequencies themselves. Because in the end, I
think that's what came out of your highlight list is
we're smashing a list of 20 words and to really
two categories. Station one of the, um,
when I would say one of the things that one
of the things we want to do next with,
we would have to do next to demonstrate our points
in the vaccination visualization study. There are two things
one would be to study expert researchers and actually have
them consciously talk about what they do as they use
different forms of visualization in their research. So because
all of the evidence from this comes from Cathy looking
through these three different forms of the data and creating
an analysis from them and then sort of reflecting on
that, what was she actually doing? How was
she actually relate to doing that? Um and then
so that would be one piece of the study would
be to actually use people who actually are researchers in
the field. How do they use these methods?
Are these visualizations? And the next thing would be
to do a non expert study where you'd actually give
these two people who don't know anything about it and
say, What are these different things to different forms
of visualization of the same day to tell you about
what's hap? Um but I also want to say
something with respect to your question, to one of
the things I had last year. I had to
undergraduate researchers look at read through 14 newspapers on the
chronically in America and then America's Struggle Newspapers database,
which is a read except propriety. The library has
it, and they read through all of the issues
. All of the newspapers from 1950 like one has
618 newspapers from 1915 to 1922 looking at articles on
vaccination to find out if there's a shift in reporting
about vaccination from before and after the pandemic. Now
these were undergraduates, so not necessarily terribly sophisticated researchers
, but they couldn't tell any kind of real shift
in the reporting. But our look at the two
years 1918 and 1919 actually does show a shift of
an initial kind of reporting that then changes during the
pandemic and then seems to go back to that,
that kind of that initial reporting. And I think
of it as a forest versus the trees thing that
when you're looking at the forest, you can kind
of like, see the shift. But if you're
looking at the individual trees, you don't see it
as much. You do all that close reading of
individual articles. It's hard to actually see the difference
. So that's one way that this kind of analysis
, it's like that the notion of distance reading that
Bill Hart Davidson was talking about, whatever he was
here, um, this idea. So somehow this
is getting us farther back to see a pattern that
we might not be able to see when we do
all that close reading really close up. So I
was told there was a classroom, too, But
I'm looking out the door there and not seeing a
large group of students eager for knowledge. So I'm
happy to keep having questions and comments. And then
if they do show up, we'll all run.
If there are more questions or comments to leave,
yeah, I can hear something is really quite funny
. But I'm trying and you're exactly what? Yeah
, I have a question about the theme Delta and
what surprised me there was the lack of Connecticut.
Am I understanding this correctly that in each vertical structure
there are words? And when those words we occur
in other vertical structures, they're linked. Um,
so it surprises me. How many words aren't linked
? Did you hear that question, Kathy? No
, I hear parts of it, but there's a
lot of it Can you hear me Now? Can
you hear me now? Yeah. So the question
was, um why are the why are so many
words and clean delta not linked two words in contiguous
segments, so I can explain that, Um,
so if you look at, they should be Can
you hear me? Cathy, I wish to go
on. Okay, so if you look at if
you look at a vertical column that only aboard some
20 words and they're not all they don't constitute all
the words that have written about during that time frame
. So these words in some sense of the most
prominent words will be featured in the topics that were
discovered during the time segment how they chose, um
, so we didn't get into all the algorithmic details
. So there are a topic modeling algorithms that basically
model the distribution of words within the time segment,
and they're breaking the distribution down into a few key
topics. So each topic has certain words to have
a higher propensity of factoring, in other words.
So to make this picture fit on a single screen
, only the top I don't know, five or
six words were chosen in each topic. No,
this is the same. This is the same outputs
as all of the other 20 words, but I'm
telling you what it does. Visualization never shows all
the words just too many words. What's the meaning
of anyone? Line joints to groups. It is
the same word, if you if you look closer
, you can't quite see here. But if you
see a line horizontal line going across from left to
right there using the same words so kidney in this
visualization kidney is blue, so you can follow kidney
shows up in other segments. So the vertical line
is a segment time segment, like a two week
period or a three week period or four week period
. And the word kidney's gonna with respect to vaccine
vaccine that's going to show up in an ad,
right? It's like it's like an ad for a
for a pharmacy or a doctor's office is going to
have kidney and vaccine and the same, that's what
that's why that word shows up. But remember,
this is the same data is visualized in the other
ones. Can it? Only it only it only
shows a line connecting them. If it appears in
the next state. If it appears in the next
segment here. Still, segments over. It doesn't
. Yeah, so what? It shows if they're
if they're not, if it's if it's broken up
, it shows. Kathy called Granular reporting right that
there's not a lot of continuity. But look at
the look at the clusters that actually reappear in the
next segment. That's where the whole band of lines
goes together to the next group. That means that
there's a cluster in or a topic in one segment
that's actually reappearing in other segments. Often that is
an ad. So sometimes there's an ad that appears
often in a particular, um, newspaper. Or
it could be that for whatever reason, um,
there's a set of articles that is on the same
issue. That means that the topic persists in that
next segment software. Did you use for this?
This is something that was developed within the group.
It's not an optician software. Is there any reasoning
to the order being the top? Is that the
most frequent word? So there's no they're not looking
for a straight line? No, you're just looking
for the connections, and there's some discussion in the
about whether or not you know how to minimize crossing
the lines, which makes it confusing. I mean
, you know, one of the things one of
the if you don't this we didn't in another iteration
, we would norm the frequencies across all the segments
because one of the things that this visualization does is
it makes you think that if the line is thicker
over here and thinner here, that it's less frequent
here than here. But it doesn't work that way
. You have to Norman across the segments in order
to make that, But since that's a normal thing
, that's a natural thing that a human is going
to do. It's gonna say, Oh, it's
more frequent here because it's thicker. So in the
next generation, we would definitely want to do that
. In order to make. To align the visualization
with the reading practices of the users may be able
to give an example of an interpretation of a graph
like this. You're trying to say Kidney has talked
about consecutively for these periods, but then it seems
to drop out of the conversation, and that's an
example of interpretation of reading this like what's the,
uh Well, the so So let me give an
example from from a different application. So we applied
this on the campaign speeches made by Obama and Romney
last year. And so you'll have a picture like
this for Obama. You'll have a different picture for
me. And what we found was that the word
cut right was used by Romney in the sense of
tax cut. Right? But as Obama talks about
pay cuts, right, so the word cut was
they're both figures. But if you see cut,
which was each other word does it join with?
They're very different, right? And the cut went
on for a while and then suddenly they started talking
about mhm Syria, right, The embassy bombing.
Right. So at that point, the cluster stop
. The new cluster started emoji, right? So
you could take, you know, two sets of
figures like that and make this qualitative comparisons. So
here it's the same thing. So I really don't
know what the cluster is. Take one. You
can see that there on the left, there's one
cluster that basically stays together from through a number,
and then it stops and another one dominates and So
the question is, what are those clusters and why
are they doing that? And you would you have
to see what the words were. Figure out if
you could tell from this what kind of a topic
it is. Is it an ad or is it
a consistency in reporting? And then perhaps So we
found one that was at the very end, and
I can't remember which extraction it was. But it
had all these weird words and it had this word
Varicose seal. Go back to the Saint Joseph Observer
. Add if you could and I was like,
What the hell is a varicose seal? It's it's
a varicose vein of the scrotum. Okay? And
it's in this ad. So this ad ran,
um, in these. So all five of the
of the word clusters in this one period were basically
different, like this ad. And so it dominated
all the reporting. It was the only thing on
vaccination in all the papers and this one particular period
at the end of 1919. But so that's one
of the so the fact that it predominant. I
saw that in the word frequency list, but I
was like, you know. Oh, my goodness
, What is this? But then it leads you
back to the paper. Or then you can say
, Oh, look at how Vaccine is being articulated
in this ad as part of modern medical care in
1919. It's something that Dr Ste. Francis offers
to his, you know, to his patients.
I don't know what, 606 and 9 14.
So question figure those out. Other questions? Yeah
. Be interested to him or about the ultimate goals
and the outcomes of the project. Do you have
a particular finish line in mind? There, Obviously
. Lots of ways to present this great data that
you you're coming up with. What is your goal
? To present the data in multiple ways and allow
this to interpret? Are you reaching towards new interpretations
of the influence epidemic of what do you have in
mind for the finish line? Yeah, that's a
good question. I mean, in some ways,
the symposium is the finish line. Or, uh
, right. Sports analogy would be, um we
see, we we actually just finished, and Bernice
and Kathy were up very late. Um, doing
about 55 page report that's going out to the other
participants in the symposium as well. We have a
board of external advisors, Um, the only outcome
that this digging into data challenge grant requires as a
white paper, which will be basically what we've done
. Um, I think at that point it's It's
kind of it's going to move in different directions,
depending on the kind of conventions of the participants for
computer scientists, as I understand that they will publish
the algorithms and make them generally available, which is
probably the most functional piece of this. You know
, from my perspective as a historian, and come
back to this question of you know what have I
how the methods have been using the findings in this
project, going to lead to publications that can inform
the field in terms of history of medicine. And
we've got some ideas on that, Um, you
know, I think the direction Bernice and Kathy have
been going in terms of the the multiple forms of
visualization responds to a different kind of field. And
then I guess, Nathan's point. How does this
relate to epidemiology that many of the participants at the
symposium that's their background and we're hoping to get not
only get this to them but also get feedback from
them in terms of how does this help us not
to analyze the spread information about contemporary disease but potentially
to anticipate what's coming? Um, you know,
so it's it's not. Creating a database is not
creating an online project. It really is creating methods
that will help can be disseminated in some different ways
. And I want to answer that question to that
. I think that one of the what are things
you find out if you do any research on that
, well, any kind of historical research on epidemic
diseases that a lot of the a lot of the
research is done using newspapers. So in others,
people will like, you know, talk about what
the sources of information that historians use or newspapers.
So what's interesting about this study is that we're actually
analyzing the newspapers so and analyzing the kind of information
and one of the things that Tom I'm going to
highlight. One of the pieces that Tom worked on
, which is the weekly newspaper study, is to
look at the kind of reporting that is evident in
these rural weekly newspapers that most of the time when
the historian has when historians have worked on this particular
, um, pandemic, they're going to look at
large newspapers in cities like Philadelphia, Washington and Roanoke
. Times is the paper that Nancy Bristowe looks at
in her book American Pandemic. But one of the
things we found in looking at the weekly newspapers was
that the reporting in those is actually often user generated
, like social media today. So there are these
sex, whole sections of local news. You know
, Mrs Mr and Mrs Smith went to Chicago last
weekend to visit their son. You know, John
Stewart is ill at home with influenza. I mean
, that would be like a little line that would
be in this local reporting. And so what you
find is is that, um so actually looking at
the newspapers as data rather than just using the newspapers
. This data and this project has helped us do
that both at a macro level and at a very
close analysis level. And I think that Tom has
been spearheading this work on the on the weekly newspapers
, and I think that actually is a tremendous contribution
to the scholarship because you just don't see it.
Yeah. Innovation, Micro. And you're reality closer
, just like that. Anything else? All right
. Thank you very much for coming. Yeah.