[Philip Young] Good evening and welcome to the Open Access Week keynote address.
I'm Philip Young, the scholarly communications librarian here at Virginia Tech.
I want to mention that tonight's keynote address is sponsored by the University Libraries,
Computational Modeling and Data Analytics, the Department of
Computer Science, the Department of Statistics, the Laboratory for
Interdisciplinary Statistical Analysis (LISA) and the Virginia Bioinformatics Institute.
As I mentioned earlier, we have a sign-up sheet for those who would
like NLI credit for attending tonight, and there's also an option on the sheet
for joining our Open@VT email list so you can get announcements about open
related events at the university. There's also an evaluation form and we'd greatly
appreciate your feedback on this event. There's also some information about
library services and support of open access, such as our digital repository
VTechWorks, where this video will be housed, our Open Access Fund which is
provided by the Provost's office and the Libraries, and also our journal publishing services.
So tonight's keynote address is the fifth and final event of
Open Access Week this year, but also for the second year, part of Open Access Week
has been sponsoring a scholarship to OpenCon and this scholarship, this
travel scholarship has been sponsored by the University Libraries and the Graduate School.
OpenCon is an international conference for students and early career
researchers on open access, open data, and open educational resources and last year
in Washington DC, Victoria Stodden presented at OpenCon.
This year, OpenCon 2015 will be held in Brussels, Belgium
from November 14th to the 16th. After our selection committee reviewed a
very strong pool of applications, the travel scholarship was awarded to
Sreyoshi Bhaduri, a PhD student in Engineering Education.
Could you stand up, and let's give you a hand. [Applause]
I'd also like to announce an upcoming event of particular note given tonight's address.
LISA, the University Libraries and the Center for Open Science will
collaborate during Research Week, November 30th through December 4th,
to provide services for researchers to promote reproducible research practices
at Virginia Tech. Researchers can schedule appointments to learn how to
improve their own reproducible research practices or can just drop in to get
advice about their research, so please put research week on your calendar beginning November 30.
Tonight's speaker, Dr. Victoria Stodden, is an
associate professor in the Graduate School of Library and Information
Science at the University of Illinois at Urbana-Champaign.
She completed both her PhD in statistics and her law degree at Stanford University.
Her research centers on the multifaceted problem of enabling
reproducibility in computational science. This includes studying adequacy
and robustness in replicated results, designing and implementing validation systems,
developing standards of openness for data and code sharing, and resolving
legal and policy barriers to disseminating reproducible research.
Her Open Access Week keynote address is titled "Scholarly communication in the
era of big data and big computation." Please welcome Dr. Stodden. [Applause]
[Victoria Stodden] Thank you so much for the introduction Philip, and it's of course a pleasure to be here.
A couple of things before I start talking. So the first one is I put my slides on
my website so lots of stuff is hotlinked on there if there's something you wanted
to go back to you feel free to just jump onto my website this same slide deck is
there and the other thing I wanted to say and this is a really big room but I
think it would be great especially since this is an evening talk and maybe a
little more casual if you have questions just feel free to shoot your hand up and
we can do that in the middle of the talk like whatever you want to talk about we
can be it doesn't need to be quite so formal even though the room makes it
seem very formal but I'll also leave questions time for questions at the end
too so if you don't feel like jumping in but if you want to just feel free
okay as Philip said scholarly communication in the era of big data so
what I would make computation what I'd like to do is focus on how I think
scholarly communication is changing and needs to change as researchers and
others start to use technologies that are based around the computer so when
I'm talking about computation sometimes people kind of think of this is sort of
the big high performance computing simulations or I'm thinking of
computation as anytime you're using a computer in your research so this would
involve someone who had say done a survey and has entered their data on
their laptop and is using say R on their laptop to do sort of different
tests between different groups they've used a computer they're falling in my
bin all the way up to all the types of more sophisticated and sort of advanced
computation so having said that let's get started so my agenda today I wanted
to review some of these technological changes and try to ground our thinking
around reproducibility and thinking about how these changes impact the
scholarly record in different forms of reproducibility trying to understand
what reproducibility really might mean and think about where these sources of
changes are actually coming from so we can start to think about how they're
actually impacting dissemination and impacting how we actually do research
i'd like to then say ok so now that we understand or have an idea of how these
changes are actually happening in research what can we rely on to think
about what should be happening so what I'm going to do is present a framework
grounded in scientific norms you know feel free to push back on that
but I'm thinking then maybe we can use that framework to then start thinking
about what should be happening in scholarly communication and in the
scholarly record and my punchline is sort of hidden in there I'm going to
bring it all back to access at the end okay so technological change the title
gives some of it away so everybody in here I'm sure has heard about big data I
mean the New York Times is reporting on big data and all the jobs and so on so
for some reason and this is this is all from the technology since the late
1990s or into 2000 as a society we've been obsessed with collecting data on
all different levels so as researchers industries collecting data government's
collecting data it's become easy disks are cheap we're awash in data so that's
one paradigm that's actually changing the type of research we do we no longer
think of data as something scarce as a researcher something that's falling out
of our experimental design where I need to go out and like almost like mining or
chip the data out and you end up maybe with like 10 observations if you're
lucky now everything is completely different we're just have masses of data
so we have the opposite problem how do we sort of start thinking about dealing
with the data so we also have another change that is structurally impacting
the type of research that we do and that's the speed of the computer so we
have very very fast machines what this starts to do even outside the changes
that are happening from big data is we end up being able to do things like
simulate an entire physical system change parameters run the system again
and ask questions about our world using that computational power this isn't
something that was really available to us in any kind of systematic way even 20
years ago 30 years ago so those are two changes you probably have noticed and
thought about yourselves the third one I find people recognize it less so I'd
also be interested in your your impressions of this when we use all
these computational technology you have to use software to access the data you
have to use software to run these simulations and use the big
computational power or even just you know your own CPU on your own laptop
you're doing this all in software you're doing analysis and software you're doing
things like inference extraction of information from the data in the
software and this is all new we never did that before we did things like
discovery at the bench with our hands in a lab notebook we did things like we
went out and we did surveys and calculations and so we never had that
sort of centrality to software that we have today associated with these
technologies so my argument is real scientific discoveries are
appearing now in the software to the new modality for discovery maybe you think
of for example statistical methodology or machine learning and how you actually
implement this is in the software and only in the software so we've got this
series of unique vehicles now that are occurring because of the technology that
I'm going to try and convince you we need to incorporate into the scholarly
record data everyone sort of talks about data we know about this I think software
is a first-class object scholarly object just like data and and I'll talk more
about it but but all this sort of encumbrances that go with that - like
workflows and so on and what you actually did on the machine so I put
this picture up here this is Laura patch sir he's a professor I believe in
mathematics and also in biology at Berkeley and I was just you know
watching these lectures on YouTube when I was really bored and I stumbled across
this one so he it's actually a terrific lecture he's giving a keynote and at
this point in the lecture he goes he starts to expound how software contains
ideas that enable biology so it's software it's not in the publication
it's not in the conference presentation it's in the software okay the second
impact I'll just note these or the second type of impact I wanted to note
these more for completeness and I'll discuss them more as we go along I'm not
going to spend much time on changes in communication digitization internet this
is sort of probably it's sort of undergirds everything we're discussing
today and has a lot more around it to do with things like open access and
dissemination and accessibility but I'll leave that as a subtext to our entire
conversation um the other one that I'll get back to later on in the talk is
intellectual property law so this becomes something that hits us in the
face as soon as we do things like put something on the web
scholars never really used to have to worry that much about intellectual
property law you wanted a preprint in 1972 you would
write to you know that author and say can you send me the preprint and that
was like normal the department would send you know in the Manoa
ve'lo preference to your department and so on and you kind of read the preprint
and you didn't have to think too much about interactions with journals or what
happens if someone else happens to read that preprint it just was it's a
completely different world and so as a community that the scholarly community
is now having to deal with intellectual property issues that attach to things
like when I put data online when I put software online when I put other types
of scholarly objects online I have to confront all these issues so I'll come
back to that but that's also something new that's falling out from all the
technological changes okay so how do we start to make sense of these changes
that we all kind of knew about on some level but how do we contextualize this
in the context of what it means to do science or to do research okay the first
thing to think about is pulling apart this idea of reproducibility you've
probably realized or noticed even discussions in the popular press around
reproducibility I think the Economist had a special issue dedicated to this LA
Times talks about it and New York Times has been talking about it using the word
reproducibility I think at this point in the discussion is unclear and what we
need to do is sort of be more clear about what we mean when we mention
reproducibility so I have found it helpful to really think about
reproducibility in three facets and I'll dig into these a little bit more and
explain them in a little more detail but I set aside empirical reproducibility so
what we've been doing for hundreds of years
reproducibility has been part of what we've called science for about four
hundred years we've always had this notion of reproducibility so that's what
I kind of call empirical reproducibility I compare that with computational and
reproducibility so how would this notion of reproducibility change now that we've
inserted this technology the computer into our research workflow somewhere the
final form of reproducibility that I found it useful to label is statistical
reproducibility so I'll get to discussing all of those
a little more depth okay empirical reproducibility so I put two examples on
here one on the Left this was an article published I think in 2014 and it's a
really cool I give if you're interested it's worth reading it's about two pages
long and what they're doing is they're highlighting the difficulties in
reproducibility between two labs that had a joint grant one at Berkeley one at
Harvard and they had set aside in their proposal two months to just make sure
that the actual production of cells in each of the labs were identical and so
you couldn't if you were just sort of given one of the one of the cell outputs
you couldn't identify which lab it came from and so they basically started
undoing and unpacking their chains of research in each of the two labs and it
ended up taking them two years before they could actually produce cells that
were not I didn't have a signature associated with each of the lab they
just didn't know why they were coming out so different and the part that I
highlighted here they actually found out that one lab was sort of stirring the
beaker another lab was throwing in a centrifuge and that was causing the
signature differences between the cells and you know and it sort of highlights
this other question around tacit knowledge so both labs thought that was
obvious and that's what everybody did and so they didn't report it right and
they really had to dig in to see why they couldn't reproduce each other's
results so that's an example of empirical reproducibility nothing
particularly new around computational technology there on the right I put this
picture up here because it says it raises an interesting ethical question
I'm just sort of throwing in as we go along so this is a round table that was
done by National Academies last year and it's called reproducibility issues in
research with animals and animal models and they had brought this workshop
together because you think about reproducibility is a good thing right if
I'm able to reproduce your results that lends extra credibility and evidence to
them right so what if you're getting those actual results causes you like you
have to kill mice along the way so maybe now that there
not like it's not an obvious good that I'm going around trying to reproduce
these results if I'm killing animals as I'm doing it so their discussion was
really how do we interpret reproducibility I still think that
classifies as empirical reproducibility that you know they're in and out doing
sort of doing these physical experiments and so on but it's an interesting
question okay that's empirical reproducibility
okay computational reproducibility the second flavor that I'm identifying so I
think it's it's helpful to give a little bit of context on computational
reproducibility so if you think back to the scientific method traditionally
we've had two branches of the scientific method so we started with the deductive
branch mathematics formal logic and deductive reasoning and this one clearly
doesn't reach all the questions that we're interested in like when should I
plant my crops for example deductive logic is not going to help you that much
there but being able to take some measurements and start to do inductive
logic might give you a better guess of when to when to plant your crops as
opposed to just sort of flipping a coin here what not so in the second branch
the empirical branch we have this in
statistical analysis of controlled experiments and a sort of more more
formal encumbrance about how we actually do research okay so there's a lot of
chatter about new branches of scientific method because of the use of these
computational tools in research and you may have heard some of these discussions
too the third branch of science arising from computation and simulation fourth
branch of the scientific method data driven discovery data enabled research I
put a sneaky question mark there after a third and fourth and so you can probably
even hear the skepticism in my voice okay so here's here's a couple of
examples so if you look at the panel on the left here the website so you don't
really need to read it to know that that's from 1995 it's got a really
old-school look to it it is from 1995 and I just and there's nothing
particularly special about these examples I just sort of clicked around
on the web a bit and found them so on the left
modeling and simulation workshop put together by NIST so that happened in
1995 I pulled out the little piece that I thought was interesting there and
their blurb it is there but it's hard to read it's common now to consider
computation as a third branch of science besides theory and experiment it's
common in 1995 you know we know that that's old hat on the other side is a
book you've probably seen so I think this came out in 2009 that was Microsoft
that put together this edited edition and lots of different authors and
chapters in there and writing the preface it goes this book is about a new
fourth paradigm for science based on data intensive computing so there's this
idea that these new branches the scientific method or in a sense of fait
accompli okay so let's take a step back and think about what the scientific
method is therefore in the first place so what we use a scientific method for
is - it recognizes that everything we're doing is researchers in the discovery of
new knowledge is fraught with mistakes we're humans we're just trying to figure
out a new mathematical proof we're trying to figure out a new relationship
in a cell and we don't know what we're doing by definition we're trying to
uncover new discoveries so what we want to do is the best we can do is try to
convince members of the community that we've done everything we can to route
error out of our process that led to that discovery that I'm now presenting
to the community so we recognize this ubiquity of error and we use the
scientific method as a tool to route error out of the discovery process and
out of the scholarly record that's our goal I mean we never know if we really
achieved you know capital T truth right that's part of discovery we just kind of
try to get more and more certain of what we're doing ok so what does that mean
for the two branches of the scientific method for the deductive branch that
means we have a well-defined concept of a proof right so you may have a new
mathematical discovery but you can't submit it as a new discovery to
the community unless you can show them the steps of your reasoning along the
way and why you think that proof is right and that result is right and so
we've got this notion of the proof you wouldn't think of taking a mathematical
result and just publishing it about the proof right like you get laughed at I
mean I even I wouldn't try that okay and then in the empirical branch we've got
similar standards so we have an entire machinery of hypothesis testing use of
appropriate statistical method in the very structured way that you communicate
those findings so similarly you are trying to publish a result in the
empirical sciences and you just leave the methods section blank again you're
not going to have much luck publishing this no matter how good your result is
the idea is telling other people what you've done so we can get it as right as
possible so here's my claim that I'll come back to at the end computation so
the all this talk about third branch fourth branch presents only a potential
new branch of the scientific method until we start to develop similar
standards for how we disseminate a computationally based finding what's the
notion of the proof when you've used a computer in your work or what's the
notion of the same structured communication of all these steps that
have gone into finding that result and so this is something like it's a little
bit of an unfair comparison because these first and second branches they've
had literally hundreds of years to think about what their standards would be
we've been sort of at this for maybe 20 years in the computational branch so of
course we're a little behind but I think that's the framework to think about
what's appropriate for dissemination and what should accompany a computational
result when it's forwarded to the community as a discovery okay there's a
couple of ways people have been thinking about this so in 1992 there was a
professor and he's still a professor at Stanford in geophysics Department just
emeritus now John Claire bot John Clervaux inspired my adviser around
reproducible research and my adviser was David Donoho and he paraphrased
paraphrased Claire bouts approach as far this is the approach I was sort of
raised with some a grad student so he goes the idea is so the idea of really
reproducible research an article about computational science in a scientific
publication is not the scholarship itself it's advertising of a scholarship
the actual scholarship is the complete set of instructions and data that
generated your results so the scholarship is that thinking remember
we're talking about software embedding ideas and data being part of the
discovery process so that thinking is really where the scientific innovation
is happening it's not in sort of the end result and that thinking all the what's
happening in the software what's happening in the computational aspects
and in the data is largely unavailable when you read that PDF because the PDF
came from empirical reproducibility with the methods section so we've got this
gap now I wanted to just make a quick note basically to preamp the question
which I always get which is yeah but you say reproducible and who cares if I can
take your code and produce the same tables and figures and I can use your
data it doesn't make it right so that's true of course it doesn't make
it right and we can reproduce terrible software and bad inferences all day if
we wanted to I mean hopefully we're not feeling nice when we do that but the but
if you actually go through and do that independent regeneration of the results
I go say I recode your methods instead of using your code I write my own and
maybe I recollect the data maybe it's a year later and I can go out and get like
a new dataset that's roughly comparable but just when you're later in time and
then I get some results and they're not like the old original results so if I
can't sort of hit return and regenerate your results from your code in your data
I can't compare where the two are different right so it's true that the
second one where you do that independent reimplementation of the experiment is
where you start to add new scientific knowledge you start to bolster our
confidence or sort of erode our confidence in the results
but I can't compare I really need that level of transparency in both to be able
to understand why they're coming out different I guarantee you they're gonna
come out different so okay the third in the last form of reproducibility is what
I'm calling statistical reproducibility and so this is this is hit the press a
reasonable amount so this this to me comprises things like poor experimental
design so you have some results that maybe your experiment is underpowered
and you wouldn't actually expect you just fluke ly got some what looked to be
significant results and and with an underpowered experiment you wouldn't
expect another attempt if that experiment to produce results if you
just sort of had a one-off fluke phase you couldn't tell when you were
publishing it things like sort of pee hacking running lots of tests to try and
sort of mine results that really aren't there and using techniques that actually
amplify significance things like how do you if you've ever worked with data this
is one of the first things you run into so how do you deal with outliers how do
you clean the data what's a mistake what should you keep in how do you deal with
missing values do you have an imputation method all of these things can
dramatically change your results and when was the last time you read a paper
where any of that was clearly explained in there so how do you know really what
you're what you're reading when you read the results all the classic statistics
issues maybe you have misspecified the model or you've left pieces out of the
model that you need to include maybe the model is itself very fragile so
parameters change a little bit and predictions or outputs change wildly
that would seem like not a good characteristic of your model and then
there are these human elements too like we tend to want to extend previous
findings that tends to be easier to publish then some finding is
diametrically opposed to something that's established in the literature
right those need to be harder to kind of get out into the community we may have
our own conflicts of interest maybe we've already published and received
some success as a scientist for findings you kind of don't want to go against
them you know all of these sort of human aspects and bias you start to enter into
it - okay so lots of changes going on different
ways of thinking about reproducibility and how do we contextualize these
changes so it's clear that technology is impacting and accelerating scholarly
research so it's it's not easy to find research that doesn't use a computer
anymore and you can do it but it's it's showing up everywhere
it's showing up in English departments where they do things like machine
learning over texts to say compare authorship of different Shakespearean
plays and so on and so all of these tools and questions are being asked and
the tools are being used in different areas and new areas and people are
finding it fascinating and useful so these computational and technological
changes are near ubiquitous across the scientific and research landscape okay
so how does that match up with the norms that we would expect from a researcher
from a scientist I mean I just went through sort of this description of the
scientific method that had certain behavioral attributes associated with it
things like oh it's good to reproduce it's good to convince the community
through transparency and clarity that you've done what you can to make this
result as correct as it can be so how can we sort of match up this those two
types of thoughts so how can the norms in the scientific community that we
would ascribe to help guide appropriate responses to all of these changes that
are happening around technology so I have a few answers for you I don't have
one clear definitive really nice answer one of the places when I started
thinking about this as I went back to a Robert K Merton snorum so these are well
known and they're from 1942 they are controversial like as you can imagine
pretty much anything anyone tried to say and this area would be controversial but
there's a couple in there that I found resonated and I found useful and sort of
framing some thinking so I wanted to share them with you so Merton
promulgated five norms so the first one communalism so scientific results are
the common property of the community the second one universalism all
scientists can contribute to science regardless of race nationality culture
it's about the results it's in a sense it's supposed to be a meritocracy so by
the way if you're already sort of struggling with the reality versus
what's being written in the north so we're talking those four norms in an
ideal sense disinterestedness as a scientist or researcher you're acting
for the benefit of the common scientific enterprise you're not trying to
advantage yourself or get personal gain from this you're not trying to get rich
for example originality is the scientific claims are supposed to
contribute something new and the last one skepticism scientific claims must be
exposed to critical scrutiny before being accepted so I lean on skepticism
in particular and also communalism in framing the next part of my talk
scepticism okay so that's Robert Boyle that I put there so he is as many know
some people call him father of chemistry he was a very influential scientist in
the sixteen hundreds in the 1660s he was embroiled in trying to create a vacuum
in an air pump and this was apparently a very competitive thing to be doing among
the scientists in those days and they didn't all live in the same geographic
area and so they would sort of write letters to each other and Boyle found
this really frustrating because you would sort of get insufficient evidence
or insufficient description to really replicate discoveries that other
scientists were claiming that they made so you had two options either the
discovery was wrong or your implementation was wrong so you really
had to get that implementation right so you knew what was going on so he wrote
down our first instantiation of how the report on the discovery you know telling
the community about the discovery should have reproducibility built in and it
should stand on its own you shouldn't have to actually write the author and
ask for more clarification so that was Boyle skepticism requiring that the
claim can be independently verified so we need transparency in the
communication of these discoveries and as I said this has happened in the
1660s also at the beginning of the Royal Society and first Journal transaction of
the Royal Society which is kind of amalgamating all those letters okay so
now how does that inter and start to impact the scholarly record okay so a
couple of things so I think this really requires to some degree a rethinking of
the notion of the scholarly record I have started to think about this as a
vehicle that gives us access or and/or the ability to regenerate items that
were relied on in the generation of results or giving us access to items
that were required for independent replication and reproducibility so those
are almost saying the same thing the difference in my mind is whether or not
we've been able to capture in that research process all the sort of dead
ends and avenues that scientists went down in their discovery process and
usually don't get reported so really that's that's coming back to the idea of
the pea hacking and how we actually understand the significance of these
findings okay so what do I mean by items so I think it's worth distinguishing a
few of them so digital scholarly objects articles tax code software data workflow
information research environment details details about the state of the machine
that you're actually running the experiments on and all the hardware
details and so lots of different things could be considered items and I would
also consider things like material objects items so these material objects
go to empirical reproducibility the digital objects tend to go to
computational reproducibility so material objects like can you get your
hands on reagents lab equipment instruments texts all that that actually
go to empirical reproducibility so in this context thinking about things like
what version of the data you have what version of the software you have what
particular characteristics with reagent you have all of this is really important
to be able to recreate that chain that that original scientist went
okay so um this is certainly isn't something that I'm the only one who's
noticed that there are issues many scientists have been worried about the
fact that it's difficult to unpack computational research to understand and
find out what actually happened and different community members have been
coming up with different solutions so I put together this is an incomplete list
but I put together some of these sort of software infrastructure responses and
platforms that people have been coming up with to try and help solve this
problem so if you're accessing my slides from my website these are hotlinked I
highly encourage you to click on some of these or some of this stuff is just
really fantastic and really interesting okay so I think these these efforts sort
of loosely in an overlapping way sort of grouped themselves into three groups so
one is post publication so dissemination platforms how do you get extra objects
out there with the publication after publication another sort of this is
bright line publication so another group is pre publication so how do you do
things like use software tools to better track all those tests that you did as a
statistician or all those avenues you went down and it kind of didn't pan out
and never kind of made it into the final narrative so how do you manage workflows
in a way that helps people share and track the information that underlies
their results and then there seems to be this other bin where people are trying
to do things like augment the document that we actually use to communicate our
results so embedded publishing coming up with maybe instead of the pedia static
PDF you could end up with a PDF that allows you to click on a figure drill
into code recreate that figure run this and so so people are doing all sorts of
innovative work I'll just mention that almost all of these like I said are just
researchers putting stuff together outside their day job because they think
it's an important problem that needs to be solved very few of these are
commercial or industry Venturi there's a couple of exceptions on there like gene
pattern is a collaboration with Microsoft for example there's a few
but generally speaking these are just sort of people putting stuff together to
try and solve the problem and I think is important so I've been working on our
research compendium and run my code so these dissemination platforms so I'll
just I'll mention research compendium quickly because it ties together some of
these threads in a more concrete way that I've been talking about so far so
research compendia you can actually just go to it research
compendium or --g and it's a prototype it's a research testbed and we put this
together I had a postdoc working on this Jennifer
Siler and also a developer Sheila migas who together the three of us built this
what we wanted to do was understand better how people would share or what
tools and platforms I needed to share reproducible research we wanted to also
understand whether we could put extra aspects that made reproducibility easier
along with those objects so we tried to do things so we we wanted to link data
in code to published articles enabling reuse of the data in code as well as
reproducing the original results we wanted to start to understand how
researchers actually shared and come up with sort of guidelines or thinking
about effective ways to share data and code and workflow we're actually able to
run the code and data that's given to us in research camp India so we could
verify the results the other people don't have to I mean unless you do it
your software is trusting us but we're kind of a third party at least you're
not only trusting the original researcher and the idea that once we
start sharing and collating data we could actually do things like validate
findings on much larger datasets like related datasets for example and as I
mentioned earlier around statistical models that may or may not be stable or
be sensitive to small changes in calibration with larger data than the
ability to run this in bigger platforms we could actually do these stability and
sensitivity checks so the idea is how can we end up with a more robust
scholarly record so this what a research compendium page would
look like well it does look like so the blue at the top is linking to the
published article so any published article we would set up a permanent page
here and the abstract isn't the abstract from the paper it's talking about the
code it's talking about the data you can click on buttons here download code
download data and you can download the article this is open access so I think
this one actually is an open access articles so you can just go ahead and
download it and you can see the different licensing and so on that we've
got here and here's an example of where we actually run it so you spot on the
right of those four buttons we can get code and data article we can also verify
so if you wanted you could just click verify and get the results that we ran
back on the author's code and data so you can see whether or not that matches
the paper and this one does by the way and you can also see a couple of
interesting things here at least I think they're interesting do I on code do I on
data do I on the the compendium age itself and because those are those are
fixed because this is a publication in the scholarly record so the code may
evolve however this is the code that got those results so that code gets frozen
in that data set gets frozen mistakes and all and it's sitting there on github
that's all open-source if you want to poke into it okay and we use the MIT
license on our software so that means pretty much you can grab it and do what
you want with it we're interested in attribution so that's why we use the MIT
license because of course we lean back on those norms with the scholarly
community okay so there are community responses at the same time so now we've
seen these individual efforts of researchers who are putting together
tools or doing individual efforts there's also a few instances where
communities have started to engage each other as an effort to say this is really
serious we need to start taking these standards around computational research
and publication of computational findings really seriously
in 2009 I was at Yale Law School and one of my jobs there was to have a
roundtable and so that was a number of years ago and I'd only just graduated
and so there's kind of a terrifying idea but I brought 30 people together who are
32 people together who are different stakeholders in different fields we had
some people from funding agencies many people from academia who were
researchers and they had this interest in reproducibility and in what was
happening with code and data and their dissemination and that was 2009 and we
still had enough to bring people together we put together sort of I put a
wiki up at the last session and we collaboratively put together a set of
principles and things that we would like to see or that we felt we needed and
then so we created this kind of manifesto or document it's the thinking
has evolved a lot since 2009 but it's still interesting is that some of the
things that we were sort of laying out in that document we laid out things that
are happening and then we laid out dreams and things we would really like
to see in there there was a much larger gathering in 2012 so this was at Brown
they run what's called an ice room series so you get a week and they do
sort of different topics every week and so this one reproducibility and
computational experimental mathematics we had December 10 to 14 in 2012 so that
was a lot bigger we brought a lot of different people together had a somewhat
similar discussion however of course the community had advanced and we had many
more solutions at that point we understood the problem better and so we
had a wide-ranging discussion again we also came up with a workshop report
these are all you know openly available this one was a lot longer and we had we
went through all these different aspects in a sort of concrete way and we had two
publications out of here and our goal setting the default to open or setting
the default to reproducibility the idea being sharing code sharing data sharing
these computational artifacts like workflows and so on would be the normal
or expected way to publish computational findings and the exceptions would be
things like you run into privacy issues you ran
into the proprietary issues or other problems like that and you make an
exception for them rather than the situation we have today where it's just
normal not to make that those extra things available extra artifacts and
exceptionally once in a while people do so we sort of have exactly the opposite
to the situation that we're trying to describe as important here okay so that
was ice term again community coming together these are some of the issues in
the interest of time I won't go through these all in detail but these are some
of the things that were flagged as I said slides on my website if you're
interested um please feel free to dig in or ask me any questions we can go back
to it there was another workshop so this was
last summer no summer 2014 two summers ago there's a National
Science Foundation project that's a really big project called exceed and
what exceed does is it builds software and access points for researchers who
may not be specialized in using sort of very high-performance computing machines
and systems to make it easier for them to use it so they can use it in their
and their work so these resources become available to a much larger community
other than sort of specialists you can run it on these sort of though they are
quite delicate machines and so the exceed folks and the leadership started
thinking that maybe that's a place where reproducibility can be thought about and
for example supercomputing contexts so what does it mean to reproduce a result
that's been generated using one of these HPC resources maybe the solution is in
that middleware that sort of bridges that gap between the researcher and the
actual resource and it can do things like what I was talking about earlier
like save the important pieces of code this the order that you ran that did
your different functions in what the parameter settings were what the Machine
state was at the time you're running in different libraries you needed to run it
all of these sort of pieces of information that help you understand
that output so maybe that's something we can automate in
in that sort of middleware exceed layer and because that's kind of a unique
opportunity usually there isn't that mediation I would just run stuff on my
laptop right but that sort of opportunity allows us to sort of push on
what it would mean to really have reproducibility and what we need to
gather and sort of understand that better so that's an ongoing discussion
because that was only summer of 2014 but I think it's interesting the way these
different communities are starting to take on and think about these issues
okay in concert with that at the same time and again many of you probably are
familiar with some of these but there's been a number of directives coming
primarily from the White House and from the Office of Science and Technology
Policy and other areas in in Washington there's one that just happened this
summer 2015 the National strategic computing initiative I'll just highlight
their strategic objective for if you're interested they say what they want is a
computing ecosystem that embraces access they actually use the word and software
and workflows and they don't use the word reproducibility which i think is
really unfortunate but they're kind of getting close right they're starting to
think about what it means in in this sort of high performance computing
context what it means to do work that's actually subsequently available to the
community I'm probably everyone knows about the
2013 open data open access executive memorandum and the executive order to
the federal agencies making data available and making articles available
and there are there things that are happening like different data management
plans or emerging NIST is developing their common access platform so things
are kind of happening I don't know of anyone who's mandating reproducibility
but there are these little kind of nibbles and steps that are happening now
I think it's really important to have these types of activities happening at
the funding agency level and happening at the OSTP level at the White House
because as you saw there like on the page around infrastructure of these
scientists for examples and researchers had been coming up with their own
Lucian's two problems and yet with the problems not solved right because any
one researcher can take these extra efforts to produce for example
reproducible research if they're not rewarded in the community then it's
really not something that you can kind of encourage for example junior faculty
postdocs or students to do they really need to focus on doing things that are
rewarded in their career and that advance them so we have this collective
action problem where none of us can kind of act individually to fix this what
solves the problem is when we all act together and so those are really really
hard problems something like nudges from OSTP help move us all together so they
help to solve this collective action problem in general I'm not a fan
personally I've sort of sort of top-down management of the scientific community
and that's been something that's been resisted for its entire existence but
that collective action problem is serious enough where it just won't be
broken by internally everyone just waking up one morning and going up I
think I'm gonna do it reproducibly today it's just not gonna
happen so there's sort of this grassroots
emergence of solutions I think it also needs her to this top-down kind of
nudging us all together okay I'll just mention and I won't go into detail again
in the interest of time but it's not just funding agencies in the White House
and those stakeholders that are discussing reproducibility it's not just
researchers journals are also doing this and I looked into this in a paper there
are requirements that are popping up in journals all over the place around data
availability requirements code availability requirements are coming
science has required code and data sharing since 2011 nature requires data
sharing for example so there's sort of other kind of breaks to this collective
action problem that that are occurring again one journal isn't going to change
the landscape so okay I just wanted to mention too about how this availability
of code availability of data opens up opportunities outside the ivory tower
and it used to be for almost all of science that most of
the discussions around discovery around verifiability around integrity and
discovery really happened with in the ivory tower within communities who sort
of knew each other and knew the problems and so on now we share things more
openly on the web we have this opportunity to really engage many more
brains so crowdsourcing public engagement in
science these things are sort of happening and emerging when I looked
into this largely around just data collection so those sort of donating
data to scientific experiments I didn't see much in the way of say you know I
was I was talking in the beginning about discovery is embedded in software and so
on I haven't seen that yet from crowdsourcing community how are that
engagements there and it's starting to grow so that's something I'm very
optimistic about it requires I think access to I put it in quotes coherent
digital scholarly objects a lot going on in there like can you understand what
the data are saying are they correctly documented is there metadata how do you
know what versions you're using apply all that to software as well ever it has
all of those same things have a different meaning in the software
context and workflow context for example but can you actually figure out what the
person was doing when they actually shared those objects we need a mechanism
for bringing in ingesting or evaluating new findings that might come from
non-traditional sources it's not it's sort of an insulin insider's game to
submit to a journal like how to actually write it the way that it gets taken
seriously and all these sort of more political aspects so how do we how do we
deal with findings that came from the source we didn't expect or an amateur
maybe you might call it even though they might be just a skill but they just have
a different style lots of legal issues reuse privacy using digital scholarly
objects and so on and you see this everywhere so you you like Washington is
all abuzz with evidence-based whatever evidence based policy evidence-based
medicine evidence-based decision making and and so that's something that I think
is a new topic how do you actually know that this is reliable evidence do I need
to get the code do I do the data to be a voting citizen for
example okay my last topic access and intellectual property so I mentioned at
the outset that intellectual property was something reasonably new that the
scientific community was having to deal with
I'll principally talk about copyright although I can talk about patents a
little bit copyright presents a very important barrier to reproducibility and
to reuse of shared digital scholarly objects it's embedded right in our
Constitution so here's the language so copyright to promote the progress of
science and the useful arts or any useful arts by securing for limited
times to authors and inventors the exclusive right to their respective
writings and discoveries so the idea is you created and owned the copyright you
also own how it gets subsequently used belongs to you so that's the idea of
copyright so any original expression of an idea so for example writing a paper
writing a piece of software making it making that idea manifest somehow
automatically falls under copyright so you don't need to register or do
anything it's copyright to you so think of that in the software context you've
written a script that produces a really beautiful figure that's going to go in
your paper script this copyright to you applies to not just code papers figures
tables and so on so what copyright does is it says the copyright holder is the
only one who's allowed to reproduce the work make a copy you've got the right to
copy it and the other thing that copyright says is you are the only one
is a copyright holder who can prepare derivative works based on the original
work so for example standing on the shoulder of giants no this is copyright
so both of these things work completely against our norms as scientists I want
people to reproduce my work I would love it if they took my work and built on it
and cited me that is directly impactful for my career right that's how all right
Center system is set up and copyright works right
against both of those and you might think it would disappear it's basically
in perpetuity 70 years plus the life of the author so if you think about the
value of software and how quickly it deprecates really is just talking about
infinity there so that's a sketch there are some exceptions to copyright like
fair use for example but they certainly don't stretch to what I've been
describing about reusing code reusing data within the scientific context of
reproducibility okay so probably most of you recognize Richard Stallman here who
arguably kicked off the open source software movement with the invention of
a rider or a piece of text that would go with a copyright object and just pre
permission in advance how you could use that object so having something under
copyright so I can't reproduce your worst I can't create derivative works
based on your work but I could have I called you and got permission or asked
you and so Salman's genius was to say well
I'll just pretend you asked and I'll just put the answer out there and you
can reuse this in these according to my particular Terms of Use so that's open
licensing and he of course created the new public license that way since then
hundreds and hundreds and hundreds of licenses are written and created so you
sort of take your pick whatever you want it's probably out there and it sort of
is an end run around the sort of restrictions a copyright places on the
work the work is still copyright to the author but the author has pre
permissioned the use okay so in 2001 Larry Lessig who was inspired by Richard
Stallman with two co-founders created Creative
Commons and what he did or what they did was develop sort of four aspects of
licenses that they thought were important for artistic works so an
artist might want to share say a video or an image and they might not want all
the protection of copyright look it's okay if you use my image and say Oh
collage or something and you might want to do the same thing that Stallman did
and pre permission to use so that's not Creative Commons data sort of printed or
created licenses that allowed you to sort of do that for artistic works
Stallman was only four code Creative Commons is never for code so they sort
of filled in two different gaps there no one had thought about what do you do in
the scientific context cuz our norms are different they're different from say the
artistic context and they're different from the open source software context
we're not doing that in science either of those we have relationships and
commonalities with what they might consider their norms Creative Commons or
open-source software communities but we have a slightly different set of norms
in the scientific community so the idea of the reproducible research standard
was guidelines around license use for researchers and for scientists releasing
code releasing data and and their articles and so the idea was use open
licenses that match our scientific norms pretty much say you can do whatever you
want with my stuff I just want you to attribute me that's pretty much what we
do is scientists if you go back to communalism things become property of
the community but we want to be recognized and cited for our efforts so
releasing our media components non code components under Creative Commons
Attribution license code components under MIT license or modified BSD
license which was equivalent to the attribution-only license for software
and data is complicated but I recommend that you dedicate data to the public
domain or maybe you can attach an appropriate attribution license to it
and the idea then is when you share objects the copyright holder and a
researcher you do it in such a way that the community recognizes what they can
do with it right they can reproduce your work they can build on your work insight
you just the same way our norms have always been I'll mention data quickly in
the u.s. we have no copyright on raw facts interestingly in Europe they do
have copyright on raw facts and as everyone knows I'm sure it's very
frequent to have international collaborations or collaborations across
different legal systems and science stuff that happens all the time what we
have in the US in court case it says raw facts are not
copyrightable however original selection and arrangement of the raw facts are
copyrightable so what does that mean in the scientific context it's like totally
unclear this was the this Court came out of a fight over phone books right oh
it's nothing to do with science there haven't been cases subsequent to that
that flush this out in the scientific context so we don't know what that means
I don't know exactly what a raw fact is I could take some guesses
however original selection the arrangement of our effect you can make
an argument around that maybe some kind of copyright is creeping in if I take
sort of some from this data set or from some mashup from that data set and then
sort of maybe the container is copyright to me so that's why I say to the public
domain dedication it just gets rid of all those residual copyright issues that
may be floating around with your data a quick note on patents so bayh-dole act
was passed in 1980 what the bayh-dole act did is it charged universities with
ownership over inventions that were happening on their campus and using
their resources and then said you should seek the patent system to actually
capitalize on these inventions and you can generate a revenue stream and the
idea being the inventions then actually made their way out into the larger
community and could start companies and help industry and make us all richer and
so on so that if you notice the nineteen eighty right we weren't using computers
then in research so now there's this interesting thing that's happening where
algorithm in code software's can be patentable universities charge through
bayh-dole have this stance towards patentability so I've just spent
probably what felt like about three hours telling you that I think we need
to release code and make it available on I even gave you some normative structure
for how we should release it and then we run smack into bayh-dole that says oh
actually if that's patentable maybe the university should be making a patent on
this and making it available through the tech transfer office for a fee so we are
starting to see these kind of you know different kind of norms budding against
each other so I think that's about all that's coming actually okay a little
bit more on ownership of research codes just to say that universities do it very
differently and the rights you have over your inventions differ from institution
to institution and people are starting to also push with ideas around sharing
code similar to how we saw movements around sharing data so code seems to be
kind of picking up the steam the same way data actually did okay so my very
last slide we have stakeholders researchers funding agency folks people
in the white house journal editors acting to some degree independently now
remember they're all subdivided by different fields right they are probably
not even talking to each other and there many of them are trying to take on and
solve these problems of reproducibility so the idea that coordination can help
us solve that collective action problem
so I shouldn't have probably called it conservative what I meant was sort of
minimalistic proposal scholarly record comprises access to ability to
regenerate items relied on in the generator in the generation of the
results that were actually published so it includes things like getting your
hands on the software getting your hands on how the software was actually
implemented what were the parameter settings what data did you start with
what data did you produce as an output and and this idea of access can unify
our approach and instantiate these notions of reproducibility in the
scholarly record so I'll stop there and take questions
yeah that's a great question yeah that's a really great question so
so I cleverly sidestepped any concrete examples notice because they get very
difficult and what you mean by reproducibility very much depends not
just on the actual nature of the problem and so that is true that some people and
some research progresses by stages and so you might think of reproducibility as
sort of stage wise checks for example throughout your progress it depends also
when you publish so if you're publishing in different parts in different stages
maybe there's a more holistic way of thinking about reproducibility makes
sense in that context and also in different communities the norms are just
really different and so you would expect culturally different ways of sort of
moving towards reproducibility so one of the things that I've been thinking about
is what does it mean to do software testing in science and in the
open-source software community you don't contribute a piece of code unless you
have tests that show how it works and so on so I would think also it might be
like maybe these immediate intermediary stages that you're hitting on could act
like tests in the open-source software world where they're sort of checks where
you know that the software is behaving how it should and then that's how you
start to instantiate reproducibility it's just an idea I mean the the
different communities are going to solve the problems I think in different ways
so it's soon I won't be able to give a high level talk like this anymore
because people are just gonna ask me questions like that other questions yes
this is a descendant of that thinking
yeah no it hasn't no I think we just haven't had the pressure to do it I
don't I'm not convinced it's necessarily that hard but it does unfurl itself into
a more complicated problem as soon as you start thinking about it so it's one
thing to take short MATLAB scripts maybe to demonstrate an image compression
algorithm and sort of check that they're working your tests could even actually
be certain images that you've just embedded in your paper it's a different
thing to say well what if I have you know I've joined a research group they
have codes that have been around and people have been working on them for 25
years there's even sort of old Fortran codes embedded in the middle no one
wants to touch it right no one's gonna it kind of works so we're not gonna open
that door and so what is testing mean in that context and and arguably it's
incredibly important because I'm certainly not reading that code there
was kind of this assumption all the way through that you could look at the code
and somehow that was helpful you're not always gonna be able to look at the code
or run it for example so then how do we sort of modular eyes and test pieces or
reuse pieces we just haven't because it hasn't been part of the scholarly record
and our normal communication and dissemination mechanism we just haven't
even sort of faced those problems and we're starting to and I'm starting to
see things like do we put out calls for interoperability and checks before they
can get on to big systems to make sure that they've sort of you know they've
got modularity in there that it's going to be to some degree as platform
independent as they can make and so this this thinking is starting to emerge but
it's very very young and I think this is a whole new kind of field where we can
really kind of use that the heritage of Knuth to make our work a lot better I
mean I think we're gonna get it all and software and that's where we gotta go
and got to think about yeah
that's great yep yep there so that I have a multi-faceted response to that so
the first thing is in my experience students are very excited by these
issues in my experience students come into technical or computational research
and they're floored by the fact that we're not acting like an open source
group like they're used to before they got here
and I have seen it where this actually causes a lot loss of confidence in what
we're producing in the scientific community and they start to doubt what
we're doing because it just sort of seems like it's not prioritized like is
the code right nobody checks what and so so I think there's a role there for
really supporting those students because I think we actually are in danger of
losing people who are initially very motivated around science and then sort
of see things that are less than optimal and how we're carrying it out and that
we haven't adapted to these computational methods however I don't
think it's on them even though they're the most excited group to implement it
because of that reward structure and I don't think there's anything I this is
my personal opinion I don't think there's anything wrong with a young
student publishing in such a way or doing work in such a way that is
currently rewarded so they can advance their career I don't think it's on them
to take the time to do all this stuff however I do think it is on the older
more established faculty to sort of lead the way and start doing that so things
like establishing where you might put the code that you develop as part of
your group that happened to me when I was a research group so we made
platforms ourselves like I don't recommend that we should have made our
own webpages to do it you have like a whole ivory system here they can support
this this is something that they shouldn't be done on your own personal
webpage that everyone moves institutions and so on
but that was part of what I learned as a graduate student and part of that was so
my advisor could see what we were doing and knew what we were doing and knew
what we were publishing so I think those types of methods and faculty could have
maybe guidance around what tools are they're gonna use github for doing this
are they gonna use something here for housing code or data how do versions
attached to this and just sort of having some very lightweight steps and and I my
sense of students will just sort of inhale it like that's not an adoption
problem if this lightweight structures get set up and and I think there's lik
sort of mini collective action problems at this level too so things like a Dean
or a chair could say well here's our recommendation we're putting code out
we're putting data out that's strong but if they just sort of make an
encouragement around it it can help set a tone with it I'm not necessarily
saying they need to make requirement for all the reasons actually that you
mentioned that what that means in different research groups to release
code data reproducibility it's wildly different and it's much more difficult
for some people and for other people some people have industry collaborations
they have to untangle you know like there's there's a lot of details that
come on there so I don't know if that's helpful I really think it's the older
folks that gotta lead the way on this yeah yeah but so does providing the
proof right of math it may I think it could it could make it a lot slower and
a lot more expensive yeah yeah yeah I know so even though I'm saying it's the
older folks that need to lead I think it's younger people that are really
going to adopt this stuff one of my colleagues says to teach the puppies
well I think a lot of this is tools and again it's it's so granular for the
particular application but I feel like if we were using infrastructure tools
that capture these things more automatically and made things easier it
wouldn't be this necessarily this huge burden in some situations to really
produce reproducible research for the computational aspects so that's why I'm
sort of talking like you're saying the tool development is also part of it
that's what I so I think tools are a big part of the solution how they get
developed and funded and promulgated is another question but you're raising all
the real problems I mean you saw my Yale roundtable in 2009 I think a reasonable
question would be if so what's changed since you did that in 2009 there's been
some changes but not seven years worth of changes right that you would expect
it's very slow so these are real problems but I don't think that would
stop us from sort of chipping away and there is some evidence that sharing code
sharing data and so and increases citation rates for articles now that may
be confounded because in my opinion I see very good scholars releasing data
releasing Co so maybe there's a confounding factor
there however it sort of starts to set that that example and I certainly have
more confidence in work where I can get a hold of code and then when I can't
when it's computational so I think there's a natural advantage to doing it
as well as the sort of cultural advantages but ya know I if I thought I
could sit there and change every Nobel you know want to be mind about how they
need to do their research I mean that's yeah yeah no that's not naive it you're
starting to bring in an interesting link so I haven't talked about open access at
all and you can see there's sort of philosophical relationships between the
issues and one of the reasons is because there's lots of discussion around open
access not as much around these issues so I tend to sort of focus on these
issues but some of the barriers to open access are also barriers in the sort of
world that you just described because we start now interacting with publishers
with the established journals holding impact factor and holding sort of a more
conservative walk over the way we do it more innovative journals may perhaps be
newer journals and then it's harder to attract because they don't have the
impact factor and they're doing things like certifying code for example so
those are open questions and again I don't have easy solutions for those
however I do know that we make you know forward progress little steps by little
stuff and so there are there are journals like biostatistics for example
and there's some other journals that will run code and they'll put a kite
mark on the big R or something on your article when you're published in their
journal and so that's still ongoing open experiment to the impact but it starts
to at least the community can take advantage of these things and they are
voluntarily taking advantage of it to try and you know display the integrity
of the work that they're publishing
other questions yeah
not that I know of no not that I know however I've heard that before so in
research compendium run my code where we're gathering code that's associated
with publications a couple of people have said you don't will be really
useful like you got the code in our give me the state of version of the code or
vice versa or something it's really hard they're
really hard for all so I don't I don't know how to solve that in an easy way I
think that is a burden that it's going to be really difficult to push back to
reviewers but that's really interesting and and where I thought you were going
to go with your comment was if I have a mathematical sort of a pseudo code for
the the code itself do I need to actually then release the code right
like if I can actually lay out the structure yeah maybe that's enough and I
think I don't think it's enough well first of all the code sitting there so
why wouldn't you release it right the pseudo code is very helpful however hat
I think there are crucial details just even in that implementation like how
you've actually you know implemented that on the machine like what have you
done in terms of digitization and you know what type of storage of use for the
numbers is that creating rounding issues or whatever it is maybe there's just a
plain old bug in there and so I think those are both helpful but sometimes I
do get the question list if I have sort of well thought through pseudo code
maybe that's enough but I think that extra step of putting it on the machine
is non-trivial and lots of mistakes can introduce there
yeah
yeah yeah that's really interesting so so again you're you're bringing in these
sort of existing structures like that we have to interact with an existing
journal system that's been there a long time that it still holds a lot of the
keys to think like promotion and so I think I think you're right that there
that that type of social pressure can come about I think it's I think it so I
you know I just don't have easy answers on you know how to interact with these
these establish systems but but yeah we do have to do it
you