WEBVTT
Kind: captions
Language: en

00:00:10.610 --> 00:00:15.059
[Philip Young] Good evening and welcome to the Open
Access Week keynote address.

00:00:15.059 --> 00:00:19.710
I'm Philip Young, the scholarly communications
librarian here at Virginia Tech.

00:00:19.710 --> 00:00:22.830
I want to mention that tonight's keynote
address is sponsored by the University Libraries,

00:00:22.830 --> 00:00:27.330
Computational Modeling and
Data Analytics, the Department of

00:00:27.330 --> 00:00:31.109
Computer Science, the Department of
Statistics, the Laboratory for

00:00:31.109 --> 00:00:36.000
Interdisciplinary Statistical Analysis
(LISA) and the Virginia Bioinformatics Institute.

00:00:36.000 --> 00:00:41.370
As I mentioned earlier, we have
a sign-up sheet for those who would

00:00:41.370 --> 00:00:45.000
like NLI credit for attending tonight,
and there's also an option on the sheet

00:00:45.000 --> 00:00:49.770
for joining our Open@VT email list so
you can get announcements about open

00:00:49.770 --> 00:00:54.239
related events at the university.  There's
also an evaluation form and we'd greatly

00:00:54.239 --> 00:00:59.699
appreciate your feedback on this event.
There's also some information about

00:00:59.699 --> 00:01:04.379
library services and support of open
access, such as our digital repository

00:01:04.379 --> 00:01:10.860
VTechWorks, where this video will be
housed, our Open Access Fund which is

00:01:10.860 --> 00:01:14.849
provided by the Provost's office and the
Libraries, and also our journal publishing services.

00:01:14.849 --> 00:01:21.869
So tonight's keynote
address is the fifth and final event of

00:01:21.869 --> 00:01:27.569
Open Access Week this year, but also for
the second year, part of Open Access Week

00:01:27.569 --> 00:01:34.289
has been sponsoring a scholarship to
OpenCon and this scholarship, this

00:01:34.289 --> 00:01:38.040
travel scholarship has been sponsored by
the University Libraries and the Graduate School.

00:01:38.040 --> 00:01:42.509
OpenCon is an international
conference for students and early career

00:01:42.509 --> 00:01:48.599
researchers on open access, open data, and
open educational resources and last year

00:01:48.599 --> 00:01:52.739
in Washington DC, Victoria Stodden
presented at OpenCon.

00:01:52.739 --> 00:01:55.709
This year,
OpenCon 2016 will be held in Brussels, Belgium

00:01:55.709 --> 00:02:01.679
from November 14th to the 16th.
After our selection committee reviewed a

00:02:01.679 --> 00:02:05.849
very strong pool of applications, the
travel scholarship was awarded to

00:02:05.849 --> 00:02:10.709
Sreyoshi Bhaduri, a PhD student in
Engineering Education.

00:02:10.709 --> 00:02:18.040
Could you stand up, and let's give you a hand.  [Applause]

00:02:18.040 --> 00:02:22.969
I'd also like to announce an upcoming
event of particular note given tonight's address.

00:02:22.969 --> 00:02:27.650
LISA, the University Libraries
and the Center for Open Science will

00:02:27.650 --> 00:02:32.450
collaborate during Research Week,
November 30th through December 4th,

00:02:32.450 --> 00:02:36.500
to provide services for researchers to
promote reproducible research practices

00:02:36.500 --> 00:02:40.310
at Virginia Tech. Researchers can
schedule appointments to learn how to

00:02:40.310 --> 00:02:44.000
improve their own reproducible research
practices or can just drop in to get

00:02:44.000 --> 00:02:47.930
advice about their research, so please
put research week on your calendar beginning November 30.

00:02:47.930 --> 00:02:54.019
Tonight's speaker, Dr. Victoria Stodden, is an

00:02:54.019 --> 00:02:56.599
associate professor in the Graduate
School of Library and Information

00:02:56.599 --> 00:03:00.019
Science at the University of Illinois at
Urbana-Champaign.

00:03:00.019 --> 00:03:03.680
She completed both her PhD in
statistics and her law degree at Stanford University.

00:03:03.680 --> 00:03:08.389
Her research centers
on the multifaceted problem of enabling

00:03:08.389 --> 00:03:14.209
reproducibility in computational science.
This includes studying adequacy

00:03:14.209 --> 00:03:18.290
and robustness in replicated results,
designing and implementing validation systems,

00:03:18.290 --> 00:03:22.849
developing standards of openness
for data and code sharing, and resolving

00:03:22.849 --> 00:03:28.400
legal and policy barriers to
disseminating reproducible research.

00:03:28.400 --> 00:03:32.329
Her Open Access Week keynote address is
titled "Scholarly communication in the

00:03:32.329 --> 00:03:43.069
era of big data and big computation."
Please welcome Dr. Stodden.  [Applause]

00:03:43.069 --> 00:03:46.220
[Victoria Stodden]  Thank you so much for the introduction Philip, and
it's of course a pleasure to be here.

00:03:46.220 --> 00:03:50.900
A couple of things before I start talking.
So the first one is I put my slides on

00:03:50.900 --> 00:03:55.400
my website so lots of stuff is hotlinked
on there if there's something you wanted

00:03:55.400 --> 00:03:58.639
to go back to you feel free to just jump
onto my website this same slide deck is

00:03:58.639 --> 00:04:04.699
there and the other thing I wanted to
say and this is a really big room but I

00:04:04.699 --> 00:04:08.120
think it would be great especially since
this is an evening talk and maybe a

00:04:08.120 --> 00:04:12.159
little more casual if you have questions
just feel free to shoot your hand up and

00:04:12.159 --> 00:04:16.370
we can do that in the middle of the talk
like whatever you want to talk about we

00:04:16.370 --> 00:04:20.120
can be it doesn't need to be quite so
formal even though the room makes it

00:04:20.120 --> 00:04:23.990
seem very formal but I'll also leave
questions time for questions at the end

00:04:23.990 --> 00:04:27.470
too so if you don't feel like jumping in
but if you want to just feel free

00:04:27.470 --> 00:04:31.580
okay as Philip said scholarly
communication in the era of big data so

00:04:31.580 --> 00:04:36.370
what I would make computation what I'd
like to do is focus on how I think

00:04:36.370 --> 00:04:41.810
scholarly communication is changing and
needs to change as researchers and

00:04:41.810 --> 00:04:46.580
others start to use technologies that
are based around the computer so when

00:04:46.580 --> 00:04:51.740
I'm talking about computation sometimes
people kind of think of this is sort of

00:04:51.740 --> 00:04:55.700
the big high performance computing
simulations or I'm thinking of

00:04:55.700 --> 00:05:00.020
computation as anytime you're using a
computer in your research so this would

00:05:00.020 --> 00:05:04.430
involve someone who had say done a
survey and has entered their data on

00:05:04.430 --> 00:05:09.680
their laptop and is using say R on
their laptop to do sort of different

00:05:09.680 --> 00:05:12.980
tests between different groups they've
used a computer they're falling in my

00:05:12.980 --> 00:05:17.920
bin all the way up to all the types of
more sophisticated and sort of advanced

00:05:17.920 --> 00:05:24.920
computation so having said that let's
get started so my agenda today I wanted

00:05:24.920 --> 00:05:29.570
to review some of these technological
changes and try to ground our thinking

00:05:29.570 --> 00:05:33.620
around reproducibility and thinking
about how these changes impact the

00:05:33.620 --> 00:05:37.820
scholarly record in different forms of
reproducibility trying to understand

00:05:37.820 --> 00:05:43.100
what reproducibility really might mean
and think about where these sources of

00:05:43.100 --> 00:05:46.040
changes are actually coming from so we
can start to think about how they're

00:05:46.040 --> 00:05:49.630
actually impacting dissemination and
impacting how we actually do research

00:05:49.630 --> 00:05:54.710
i'd like to then say ok so now that we
understand or have an idea of how these

00:05:54.710 --> 00:05:59.090
changes are actually happening in
research what can we rely on to think

00:05:59.090 --> 00:06:03.110
about what should be happening so what
I'm going to do is present a framework

00:06:03.110 --> 00:06:08.090
grounded in scientific norms you know
feel free to push back on that

00:06:08.090 --> 00:06:11.600
but I'm thinking then maybe we can use
that framework to then start thinking

00:06:11.600 --> 00:06:14.750
about what should be happening in
scholarly communication and in the

00:06:14.750 --> 00:06:19.790
scholarly record and my punchline is
sort of hidden in there I'm going to

00:06:19.790 --> 00:06:24.950
bring it all back to access at the end
okay so technological change the title

00:06:24.950 --> 00:06:30.110
gives some of it away so everybody in
here I'm sure has heard about big data I

00:06:30.110 --> 00:06:34.160
mean the New York Times is reporting on
big data and all the jobs and so on so

00:06:34.160 --> 00:06:40.980
for some reason and this is this is all
from the technology since the late

00:06:40.980 --> 00:06:46.480
1990s or into 2000 as a society we've
been obsessed with collecting data on

00:06:46.480 --> 00:06:50.770
all different levels so as researchers
industries collecting data government's

00:06:50.770 --> 00:06:55.660
collecting data it's become easy disks
are cheap we're awash in data so that's

00:06:55.660 --> 00:06:59.380
one paradigm that's actually changing
the type of research we do we no longer

00:06:59.380 --> 00:07:04.210
think of data as something scarce as a
researcher something that's falling out

00:07:04.210 --> 00:07:08.889
of our experimental design where I need
to go out and like almost like mining or

00:07:08.889 --> 00:07:13.180
chip the data out and you end up maybe
with like 10 observations if you're

00:07:13.180 --> 00:07:17.020
lucky now everything is completely
different we're just have masses of data

00:07:17.020 --> 00:07:20.050
so we have the opposite problem how do
we sort of start thinking about dealing

00:07:20.050 --> 00:07:27.010
with the data so we also have another
change that is structurally impacting

00:07:27.010 --> 00:07:30.880
the type of research that we do and
that's the speed of the computer so we

00:07:30.880 --> 00:07:36.729
have very very fast machines what this
starts to do even outside the changes

00:07:36.729 --> 00:07:40.690
that are happening from big data is we
end up being able to do things like

00:07:40.690 --> 00:07:45.729
simulate an entire physical system
change parameters run the system again

00:07:45.729 --> 00:07:50.169
and ask questions about our world using
that computational power this isn't

00:07:50.169 --> 00:07:54.100
something that was really available to
us in any kind of systematic way even 20

00:07:54.100 --> 00:07:58.630
years ago 30 years ago so those are two
changes you probably have noticed and

00:07:58.630 --> 00:08:04.090
thought about yourselves the third one I
find people recognize it less so I'd

00:08:04.090 --> 00:08:10.900
also be interested in your your
impressions of this when we use all

00:08:10.900 --> 00:08:14.950
these computational technology you have
to use software to access the data you

00:08:14.950 --> 00:08:17.650
have to use software to run these
simulations and use the big

00:08:17.650 --> 00:08:21.810
computational power or even just you
know your own CPU on your own laptop

00:08:21.810 --> 00:08:27.970
you're doing this all in software you're
doing analysis and software you're doing

00:08:27.970 --> 00:08:32.110
things like inference extraction of
information from the data in the

00:08:32.110 --> 00:08:36.339
software and this is all new we never
did that before we did things like

00:08:36.339 --> 00:08:41.320
discovery at the bench with our hands in
a lab notebook we did things like we

00:08:41.320 --> 00:08:44.910
went out and we did surveys and
calculations and so we never had that

00:08:44.910 --> 00:08:49.209
sort of centrality to software that we
have today associated with these

00:08:49.209 --> 00:08:55.540
technologies so my argument is
real scientific discoveries are

00:08:55.540 --> 00:09:00.700
appearing now in the software to the new
modality for discovery maybe you think

00:09:00.700 --> 00:09:05.710
of for example statistical methodology
or machine learning and how you actually

00:09:05.710 --> 00:09:10.150
implement this is in the software and
only in the software so we've got this

00:09:10.150 --> 00:09:16.030
series of unique vehicles now that are
occurring because of the technology that

00:09:16.030 --> 00:09:19.450
I'm going to try and convince you we
need to incorporate into the scholarly

00:09:19.450 --> 00:09:23.230
record data everyone sort of talks about
data we know about this I think software

00:09:23.230 --> 00:09:27.670
is a first-class object scholarly object
just like data and and I'll talk more

00:09:27.670 --> 00:09:31.540
about it but but all this sort of
encumbrances that go with that - like

00:09:31.540 --> 00:09:35.860
workflows and so on and what you
actually did on the machine so I put

00:09:35.860 --> 00:09:41.530
this picture up here this is Laura patch
sir he's a professor I believe in

00:09:41.530 --> 00:09:46.660
mathematics and also in biology at
Berkeley and I was just you know

00:09:46.660 --> 00:09:51.310
watching these lectures on YouTube when
I was really bored and I stumbled across

00:09:51.310 --> 00:09:55.270
this one so he it's actually a terrific
lecture he's giving a keynote and at

00:09:55.270 --> 00:10:00.700
this point in the lecture he goes he
starts to expound how software contains

00:10:00.700 --> 00:10:04.839
ideas that enable biology so it's
software it's not in the publication

00:10:04.839 --> 00:10:10.240
it's not in the conference presentation
it's in the software okay the second

00:10:10.240 --> 00:10:15.790
impact I'll just note these or the
second type of impact I wanted to note

00:10:15.790 --> 00:10:20.350
these more for completeness and I'll
discuss them more as we go along I'm not

00:10:20.350 --> 00:10:25.270
going to spend much time on changes in
communication digitization internet this

00:10:25.270 --> 00:10:29.230
is sort of probably it's sort of
undergirds everything we're discussing

00:10:29.230 --> 00:10:33.339
today and has a lot more around it to do
with things like open access and

00:10:33.339 --> 00:10:36.940
dissemination and accessibility but I'll
leave that as a subtext to our entire

00:10:36.940 --> 00:10:42.000
conversation um the other one that I'll
get back to later on in the talk is

00:10:42.000 --> 00:10:45.730
intellectual property law so this
becomes something that hits us in the

00:10:45.730 --> 00:10:48.900
face as soon as we do things like put
something on the web

00:10:48.900 --> 00:10:53.170
scholars never really used to have to
worry that much about intellectual

00:10:53.170 --> 00:10:57.670
property law
you wanted a preprint in 1972 you would

00:10:57.670 --> 00:11:02.920
write to you know that author and say
can you send me the preprint and that

00:11:02.920 --> 00:11:06.090
was like normal the department would
send you know in the Manoa

00:11:06.090 --> 00:11:09.930
ve'lo preference to your department and
so on and you kind of read the preprint

00:11:09.930 --> 00:11:14.550
and you didn't have to think too much
about interactions with journals or what

00:11:14.550 --> 00:11:18.630
happens if someone else happens to read
that preprint it just was it's a

00:11:18.630 --> 00:11:23.910
completely different world and so as a
community that the scholarly community

00:11:23.910 --> 00:11:27.390
is now having to deal with intellectual
property issues that attach to things

00:11:27.390 --> 00:11:31.620
like when I put data online when I put
software online when I put other types

00:11:31.620 --> 00:11:35.100
of scholarly objects online I have to
confront all these issues so I'll come

00:11:35.100 --> 00:11:38.850
back to that but that's also something
new that's falling out from all the

00:11:38.850 --> 00:11:45.630
technological changes okay so how do we
start to make sense of these changes

00:11:45.630 --> 00:11:49.440
that we all kind of knew about on some
level but how do we contextualize this

00:11:49.440 --> 00:11:56.190
in the context of what it means to do
science or to do research okay the first

00:11:56.190 --> 00:12:00.630
thing to think about is pulling apart
this idea of reproducibility you've

00:12:00.630 --> 00:12:06.870
probably realized or noticed even
discussions in the popular press around

00:12:06.870 --> 00:12:12.330
reproducibility I think the Economist
had a special issue dedicated to this LA

00:12:12.330 --> 00:12:17.340
Times talks about it and New York Times
has been talking about it using the word

00:12:17.340 --> 00:12:24.330
reproducibility I think at this point in
the discussion is unclear and what we

00:12:24.330 --> 00:12:27.870
need to do is sort of be more clear
about what we mean when we mention

00:12:27.870 --> 00:12:31.080
reproducibility so I have found it
helpful to really think about

00:12:31.080 --> 00:12:34.800
reproducibility in three facets and I'll
dig into these a little bit more and

00:12:34.800 --> 00:12:42.720
explain them in a little more detail but
I set aside empirical reproducibility so

00:12:42.720 --> 00:12:44.970
what we've been doing for hundreds of
years

00:12:44.970 --> 00:12:50.280
reproducibility has been part of what
we've called science for about four

00:12:50.280 --> 00:12:55.740
hundred years we've always had this
notion of reproducibility so that's what

00:12:55.740 --> 00:13:00.150
I kind of call empirical reproducibility
I compare that with computational and

00:13:00.150 --> 00:13:03.480
reproducibility so how would this notion
of reproducibility change now that we've

00:13:03.480 --> 00:13:09.690
inserted this technology the computer
into our research workflow somewhere the

00:13:09.690 --> 00:13:14.220
final form of reproducibility that I
found it useful to label is statistical

00:13:14.220 --> 00:13:19.649
reproducibility so I'll get to
discussing all of those

00:13:19.649 --> 00:13:25.829
a little more depth okay empirical
reproducibility so I put two examples on

00:13:25.829 --> 00:13:32.249
here one on the Left this was an article
published I think in 2014 and it's a

00:13:32.249 --> 00:13:35.879
really cool I give if you're interested
it's worth reading it's about two pages

00:13:35.879 --> 00:13:39.990
long and what they're doing is they're
highlighting the difficulties in

00:13:39.990 --> 00:13:44.100
reproducibility between two labs that
had a joint grant one at Berkeley one at

00:13:44.100 --> 00:13:49.499
Harvard and they had set aside in their
proposal two months to just make sure

00:13:49.499 --> 00:13:54.869
that the actual production of cells in
each of the labs were identical and so

00:13:54.869 --> 00:13:59.369
you couldn't if you were just sort of
given one of the one of the cell outputs

00:13:59.369 --> 00:14:03.420
you couldn't identify which lab it came
from and so they basically started

00:14:03.420 --> 00:14:08.040
undoing and unpacking their chains of
research in each of the two labs and it

00:14:08.040 --> 00:14:12.660
ended up taking them two years before
they could actually produce cells that

00:14:12.660 --> 00:14:15.899
were not I didn't have a signature
associated with each of the lab they

00:14:15.899 --> 00:14:19.589
just didn't know why they were coming
out so different and the part that I

00:14:19.589 --> 00:14:26.910
highlighted here they actually found out
that one lab was sort of stirring the

00:14:26.910 --> 00:14:31.050
beaker another lab was throwing in a
centrifuge and that was causing the

00:14:31.050 --> 00:14:34.649
signature differences between the cells
and you know and it sort of highlights

00:14:34.649 --> 00:14:37.470
this other question around tacit
knowledge so both labs thought that was

00:14:37.470 --> 00:14:41.129
obvious and that's what everybody did
and so they didn't report it right and

00:14:41.129 --> 00:14:43.740
they really had to dig in to see why
they couldn't reproduce each other's

00:14:43.740 --> 00:14:47.370
results so that's an example of
empirical reproducibility nothing

00:14:47.370 --> 00:14:53.189
particularly new around computational
technology there on the right I put this

00:14:53.189 --> 00:14:57.120
picture up here because it says it
raises an interesting ethical question

00:14:57.120 --> 00:15:02.399
I'm just sort of throwing in as we go
along so this is a round table that was

00:15:02.399 --> 00:15:07.980
done by National Academies last year and
it's called reproducibility issues in

00:15:07.980 --> 00:15:12.870
research with animals and animal models
and they had brought this workshop

00:15:12.870 --> 00:15:17.879
together because you think about
reproducibility is a good thing right if

00:15:17.879 --> 00:15:23.970
I'm able to reproduce your results that
lends extra credibility and evidence to

00:15:23.970 --> 00:15:29.639
them right so what if you're getting
those actual results causes you like you

00:15:29.639 --> 00:15:33.420
have to kill mice along the way so maybe
now that there

00:15:33.420 --> 00:15:36.810
not like it's not an obvious good that
I'm going around trying to reproduce

00:15:36.810 --> 00:15:41.040
these results if I'm killing animals as
I'm doing it so their discussion was

00:15:41.040 --> 00:15:44.310
really how do we interpret
reproducibility I still think that

00:15:44.310 --> 00:15:47.820
classifies as empirical reproducibility
that you know they're in and out doing

00:15:47.820 --> 00:15:51.030
sort of doing these physical experiments
and so on but it's an interesting

00:15:51.030 --> 00:15:55.130
question
okay that's empirical reproducibility

00:15:55.130 --> 00:16:02.700
okay computational reproducibility the
second flavor that I'm identifying so I

00:16:02.700 --> 00:16:06.480
think it's it's helpful to give a little
bit of context on computational

00:16:06.480 --> 00:16:11.850
reproducibility so if you think back to
the scientific method traditionally

00:16:11.850 --> 00:16:17.160
we've had two branches of the scientific
method so we started with the deductive

00:16:17.160 --> 00:16:24.150
branch mathematics formal logic and
deductive reasoning and this one clearly

00:16:24.150 --> 00:16:27.360
doesn't reach all the questions that
we're interested in like when should I

00:16:27.360 --> 00:16:31.080
plant my crops for example deductive
logic is not going to help you that much

00:16:31.080 --> 00:16:34.470
there but being able to take some
measurements and start to do inductive

00:16:34.470 --> 00:16:39.780
logic might give you a better guess of
when to when to plant your crops as

00:16:39.780 --> 00:16:43.620
opposed to just sort of flipping a coin
here what not so in the second branch

00:16:43.620 --> 00:16:47.330
the empirical branch we have this in

00:16:48.260 --> 00:16:52.770
statistical analysis of controlled
experiments and a sort of more more

00:16:52.770 --> 00:16:57.690
formal encumbrance about how we actually
do research okay so there's a lot of

00:16:57.690 --> 00:17:02.880
chatter about new branches of scientific
method because of the use of these

00:17:02.880 --> 00:17:06.960
computational tools in research and you
may have heard some of these discussions

00:17:06.960 --> 00:17:14.520
too the third branch of science arising
from computation and simulation fourth

00:17:14.520 --> 00:17:20.990
branch of the scientific method data
driven discovery data enabled research I

00:17:20.990 --> 00:17:25.830
put a sneaky question mark there after a
third and fourth and so you can probably

00:17:25.830 --> 00:17:30.420
even hear the skepticism in my voice
okay so here's here's a couple of

00:17:30.420 --> 00:17:36.270
examples so if you look at the panel on
the left here the website so you don't

00:17:36.270 --> 00:17:40.530
really need to read it to know that
that's from 1995 it's got a really

00:17:40.530 --> 00:17:43.920
old-school look to it it is from 1995
and I just and there's nothing

00:17:43.920 --> 00:17:46.350
particularly special about these
examples I just sort of clicked around

00:17:46.350 --> 00:17:49.400
on the web
a bit and found them so on the left

00:17:49.400 --> 00:17:55.440
modeling and simulation workshop put
together by NIST so that happened in

00:17:55.440 --> 00:17:59.250
1995 I pulled out the little piece that
I thought was interesting there and

00:17:59.250 --> 00:18:02.610
their blurb it is there but it's hard to
read it's common now to consider

00:18:02.610 --> 00:18:08.280
computation as a third branch of science
besides theory and experiment it's

00:18:08.280 --> 00:18:16.080
common in 1995 you know we know that
that's old hat on the other side is a

00:18:16.080 --> 00:18:19.650
book you've probably seen so I think
this came out in 2009 that was Microsoft

00:18:19.650 --> 00:18:23.130
that put together this edited edition
and lots of different authors and

00:18:23.130 --> 00:18:27.150
chapters in there and writing the
preface it goes this book is about a new

00:18:27.150 --> 00:18:31.650
fourth paradigm for science based on
data intensive computing so there's this

00:18:31.650 --> 00:18:35.730
idea that these new branches the
scientific method or in a sense of fait

00:18:35.730 --> 00:18:41.700
accompli okay so let's take a step back
and think about what the scientific

00:18:41.700 --> 00:18:47.610
method is therefore in the first place
so what we use a scientific method for

00:18:47.610 --> 00:18:55.110
is - it recognizes that everything we're
doing is researchers in the discovery of

00:18:55.110 --> 00:18:59.940
new knowledge is fraught with mistakes
we're humans we're just trying to figure

00:18:59.940 --> 00:19:03.510
out a new mathematical proof we're
trying to figure out a new relationship

00:19:03.510 --> 00:19:07.799
in a cell and we don't know what we're
doing by definition we're trying to

00:19:07.799 --> 00:19:12.540
uncover new discoveries so what we want
to do is the best we can do is try to

00:19:12.540 --> 00:19:16.440
convince members of the community that
we've done everything we can to route

00:19:16.440 --> 00:19:20.070
error out of our process that led to
that discovery that I'm now presenting

00:19:20.070 --> 00:19:26.309
to the community so we recognize this
ubiquity of error and we use the

00:19:26.309 --> 00:19:30.540
scientific method as a tool to route
error out of the discovery process and

00:19:30.540 --> 00:19:35.400
out of the scholarly record that's our
goal I mean we never know if we really

00:19:35.400 --> 00:19:39.630
achieved you know capital T truth right
that's part of discovery we just kind of

00:19:39.630 --> 00:19:44.730
try to get more and more certain of what
we're doing ok so what does that mean

00:19:44.730 --> 00:19:48.270
for the two branches of the scientific
method for the deductive branch that

00:19:48.270 --> 00:19:52.890
means we have a well-defined concept of
a proof right so you may have a new

00:19:52.890 --> 00:20:00.080
mathematical discovery but you can't
submit it as a new discovery to

00:20:00.250 --> 00:20:04.540
the community unless you can show them
the steps of your reasoning along the

00:20:04.540 --> 00:20:08.200
way and why you think that proof is
right and that result is right and so

00:20:08.200 --> 00:20:11.770
we've got this notion of the proof you
wouldn't think of taking a mathematical

00:20:11.770 --> 00:20:16.630
result and just publishing it about the
proof right like you get laughed at I

00:20:16.630 --> 00:20:22.390
mean I even I wouldn't try that okay and
then in the empirical branch we've got

00:20:22.390 --> 00:20:28.510
similar standards so we have an entire
machinery of hypothesis testing use of

00:20:28.510 --> 00:20:32.410
appropriate statistical method in the
very structured way that you communicate

00:20:32.410 --> 00:20:41.110
those findings so similarly you are
trying to publish a result in the

00:20:41.110 --> 00:20:44.920
empirical sciences and you just leave
the methods section blank again you're

00:20:44.920 --> 00:20:49.480
not going to have much luck publishing
this no matter how good your result is

00:20:49.480 --> 00:20:53.620
the idea is telling other people what
you've done so we can get it as right as

00:20:53.620 --> 00:20:59.590
possible so here's my claim that I'll
come back to at the end computation so

00:20:59.590 --> 00:21:04.210
the all this talk about third branch
fourth branch presents only a potential

00:21:04.210 --> 00:21:08.230
new branch of the scientific method
until we start to develop similar

00:21:08.230 --> 00:21:13.480
standards for how we disseminate a
computationally based finding what's the

00:21:13.480 --> 00:21:17.350
notion of the proof when you've used a
computer in your work or what's the

00:21:17.350 --> 00:21:20.530
notion of the same structured
communication of all these steps that

00:21:20.530 --> 00:21:24.370
have gone into finding that result and
so this is something like it's a little

00:21:24.370 --> 00:21:27.430
bit of an unfair comparison because
these first and second branches they've

00:21:27.430 --> 00:21:30.400
had literally hundreds of years to think
about what their standards would be

00:21:30.400 --> 00:21:35.110
we've been sort of at this for maybe 20
years in the computational branch so of

00:21:35.110 --> 00:21:38.020
course we're a little behind but I think
that's the framework to think about

00:21:38.020 --> 00:21:43.030
what's appropriate for dissemination and
what should accompany a computational

00:21:43.030 --> 00:21:49.150
result when it's forwarded to the
community as a discovery okay there's a

00:21:49.150 --> 00:21:55.510
couple of ways people have been thinking
about this so in 1992 there was a

00:21:55.510 --> 00:21:59.410
professor and he's still a professor at
Stanford in geophysics Department just

00:21:59.410 --> 00:22:05.680
emeritus now John Claire bot John
Clervaux inspired my adviser around

00:22:05.680 --> 00:22:10.710
reproducible research and my adviser was
David Donoho and he paraphrased

00:22:10.710 --> 00:22:15.520
paraphrased Claire bouts approach as far
this is the approach I was sort of

00:22:15.520 --> 00:22:21.010
raised with some a grad student so he
goes the idea is so the idea of really

00:22:21.010 --> 00:22:26.470
reproducible research an article about
computational science in a scientific

00:22:26.470 --> 00:22:31.210
publication is not the scholarship
itself it's advertising of a scholarship

00:22:31.210 --> 00:22:35.320
the actual scholarship is the complete
set of instructions and data that

00:22:35.320 --> 00:22:39.250
generated your results so the
scholarship is that thinking remember

00:22:39.250 --> 00:22:44.980
we're talking about software embedding
ideas and data being part of the

00:22:44.980 --> 00:22:50.740
discovery process so that thinking is
really where the scientific innovation

00:22:50.740 --> 00:22:56.679
is happening it's not in sort of the end
result and that thinking all the what's

00:22:56.679 --> 00:22:59.230
happening in the software what's
happening in the computational aspects

00:22:59.230 --> 00:23:02.830
and in the data is largely unavailable
when you read that PDF because the PDF

00:23:02.830 --> 00:23:07.150
came from empirical reproducibility with
the methods section so we've got this

00:23:07.150 --> 00:23:13.660
gap now I wanted to just make a quick
note basically to preamp the question

00:23:13.660 --> 00:23:19.870
which I always get which is yeah but you
say reproducible and who cares if I can

00:23:19.870 --> 00:23:24.309
take your code and produce the same
tables and figures and I can use your

00:23:24.309 --> 00:23:27.910
data it doesn't make it right
so that's true of course it doesn't make

00:23:27.910 --> 00:23:33.010
it right and we can reproduce terrible
software and bad inferences all day if

00:23:33.010 --> 00:23:38.260
we wanted to I mean hopefully we're not
feeling nice when we do that but the but

00:23:38.260 --> 00:23:44.710
if you actually go through and do that
independent regeneration of the results

00:23:44.710 --> 00:23:49.960
I go say I recode your methods instead
of using your code I write my own and

00:23:49.960 --> 00:23:53.980
maybe I recollect the data maybe it's a
year later and I can go out and get like

00:23:53.980 --> 00:23:57.880
a new dataset that's roughly comparable
but just when you're later in time and

00:23:57.880 --> 00:24:04.750
then I get some results and they're not
like the old original results so if I

00:24:04.750 --> 00:24:09.040
can't sort of hit return and regenerate
your results from your code in your data

00:24:09.040 --> 00:24:14.770
I can't compare where the two are
different right so it's true that the

00:24:14.770 --> 00:24:20.260
second one where you do that independent
reimplementation of the experiment is

00:24:20.260 --> 00:24:23.080
where you start to add new scientific
knowledge you start to bolster our

00:24:23.080 --> 00:24:26.919
confidence or sort of erode our
confidence in the results

00:24:26.919 --> 00:24:31.389
but I can't compare I really need that
level of transparency in both to be able

00:24:31.389 --> 00:24:33.700
to understand why they're coming out
different I guarantee you they're gonna

00:24:33.700 --> 00:24:40.480
come out different so okay the third in
the last form of reproducibility is what

00:24:40.480 --> 00:24:45.850
I'm calling statistical reproducibility
and so this is this is hit the press a

00:24:45.850 --> 00:24:50.769
reasonable amount so this this to me
comprises things like poor experimental

00:24:50.769 --> 00:24:55.179
design so you have some results that
maybe your experiment is underpowered

00:24:55.179 --> 00:24:59.499
and you wouldn't actually expect you
just fluke ly got some what looked to be

00:24:59.499 --> 00:25:02.769
significant results and and with an
underpowered experiment you wouldn't

00:25:02.769 --> 00:25:06.820
expect another attempt if that
experiment to produce results if you

00:25:06.820 --> 00:25:09.879
just sort of had a one-off fluke phase
you couldn't tell when you were

00:25:09.879 --> 00:25:16.169
publishing it things like sort of pee
hacking running lots of tests to try and

00:25:16.169 --> 00:25:22.690
sort of mine results that really aren't
there and using techniques that actually

00:25:22.690 --> 00:25:29.289
amplify significance things like how do
you if you've ever worked with data this

00:25:29.289 --> 00:25:32.080
is one of the first things you run into
so how do you deal with outliers how do

00:25:32.080 --> 00:25:35.980
you clean the data what's a mistake what
should you keep in how do you deal with

00:25:35.980 --> 00:25:39.609
missing values do you have an imputation
method all of these things can

00:25:39.609 --> 00:25:43.690
dramatically change your results and
when was the last time you read a paper

00:25:43.690 --> 00:25:47.769
where any of that was clearly explained
in there so how do you know really what

00:25:47.769 --> 00:25:52.570
you're what you're reading when you read
the results all the classic statistics

00:25:52.570 --> 00:25:56.679
issues maybe you have misspecified the
model or you've left pieces out of the

00:25:56.679 --> 00:26:01.570
model that you need to include maybe the
model is itself very fragile so

00:26:01.570 --> 00:26:05.669
parameters change a little bit and
predictions or outputs change wildly

00:26:05.669 --> 00:26:10.090
that would seem like not a good
characteristic of your model and then

00:26:10.090 --> 00:26:15.820
there are these human elements too like
we tend to want to extend previous

00:26:15.820 --> 00:26:19.899
findings that tends to be easier to
publish then some finding is

00:26:19.899 --> 00:26:22.629
diametrically opposed to something
that's established in the literature

00:26:22.629 --> 00:26:26.470
right those need to be harder to kind of
get out into the community we may have

00:26:26.470 --> 00:26:30.730
our own conflicts of interest maybe
we've already published and received

00:26:30.730 --> 00:26:34.269
some success as a scientist for findings
you kind of don't want to go against

00:26:34.269 --> 00:26:37.809
them you know all of these sort of human
aspects and bias you start to enter into

00:26:37.809 --> 00:26:44.680
it - okay
so lots of changes going on different

00:26:44.680 --> 00:26:49.330
ways of thinking about reproducibility
and how do we contextualize these

00:26:49.330 --> 00:26:57.790
changes so it's clear that technology is
impacting and accelerating scholarly

00:26:57.790 --> 00:27:02.410
research so it's it's not easy to find
research that doesn't use a computer

00:27:02.410 --> 00:27:05.560
anymore and you can do it but it's it's
showing up everywhere

00:27:05.560 --> 00:27:08.680
it's showing up in English departments
where they do things like machine

00:27:08.680 --> 00:27:13.960
learning over texts to say compare
authorship of different Shakespearean

00:27:13.960 --> 00:27:19.180
plays and so on and so all of these
tools and questions are being asked and

00:27:19.180 --> 00:27:23.050
the tools are being used in different
areas and new areas and people are

00:27:23.050 --> 00:27:26.950
finding it fascinating and useful so
these computational and technological

00:27:26.950 --> 00:27:32.530
changes are near ubiquitous across the
scientific and research landscape okay

00:27:32.530 --> 00:27:37.480
so how does that match up with the norms
that we would expect from a researcher

00:27:37.480 --> 00:27:42.430
from a scientist I mean I just went
through sort of this description of the

00:27:42.430 --> 00:27:47.260
scientific method that had certain
behavioral attributes associated with it

00:27:47.260 --> 00:27:50.850
things like oh it's good to reproduce
it's good to convince the community

00:27:50.850 --> 00:27:56.040
through transparency and clarity that
you've done what you can to make this

00:27:56.040 --> 00:28:01.630
result as correct as it can be so how
can we sort of match up this those two

00:28:01.630 --> 00:28:06.370
types of thoughts so how can the norms
in the scientific community that we

00:28:06.370 --> 00:28:10.720
would ascribe to help guide appropriate
responses to all of these changes that

00:28:10.720 --> 00:28:18.520
are happening around technology so I
have a few answers for you I don't have

00:28:18.520 --> 00:28:22.960
one clear definitive really nice answer
one of the places when I started

00:28:22.960 --> 00:28:28.210
thinking about this as I went back to a
Robert K Merton snorum so these are well

00:28:28.210 --> 00:28:33.970
known and they're from 1942 they are
controversial like as you can imagine

00:28:33.970 --> 00:28:38.500
pretty much anything anyone tried to say
and this area would be controversial but

00:28:38.500 --> 00:28:42.970
there's a couple in there that I found
resonated and I found useful and sort of

00:28:42.970 --> 00:28:47.190
framing some thinking so I wanted to
share them with you so Merton

00:28:47.190 --> 00:28:52.240
promulgated five norms so the first one
communalism so scientific results are

00:28:52.240 --> 00:28:57.370
the common property of the community
the second one universalism all

00:28:57.370 --> 00:29:01.660
scientists can contribute to science
regardless of race nationality culture

00:29:01.660 --> 00:29:05.110
it's about the results it's in a sense
it's supposed to be a meritocracy so by

00:29:05.110 --> 00:29:09.010
the way if you're already sort of
struggling with the reality versus

00:29:09.010 --> 00:29:11.980
what's being written in the north so
we're talking those four norms in an

00:29:11.980 --> 00:29:18.460
ideal sense disinterestedness as a
scientist or researcher you're acting

00:29:18.460 --> 00:29:22.330
for the benefit of the common scientific
enterprise you're not trying to

00:29:22.330 --> 00:29:27.520
advantage yourself or get personal gain
from this you're not trying to get rich

00:29:27.520 --> 00:29:31.300
for example originality is the
scientific claims are supposed to

00:29:31.300 --> 00:29:36.670
contribute something new and the last
one skepticism scientific claims must be

00:29:36.670 --> 00:29:42.880
exposed to critical scrutiny before
being accepted so I lean on skepticism

00:29:42.880 --> 00:29:49.470
in particular and also communalism in
framing the next part of my talk

00:29:49.470 --> 00:29:58.350
scepticism okay so that's Robert Boyle
that I put there so he is as many know

00:29:58.350 --> 00:30:02.620
some people call him father of chemistry
he was a very influential scientist in

00:30:02.620 --> 00:30:09.370
the sixteen hundreds in the 1660s he was
embroiled in trying to create a vacuum

00:30:09.370 --> 00:30:14.890
in an air pump and this was apparently a
very competitive thing to be doing among

00:30:14.890 --> 00:30:19.060
the scientists in those days and they
didn't all live in the same geographic

00:30:19.060 --> 00:30:23.950
area and so they would sort of write
letters to each other and Boyle found

00:30:23.950 --> 00:30:26.800
this really frustrating because you
would sort of get insufficient evidence

00:30:26.800 --> 00:30:31.690
or insufficient description to really
replicate discoveries that other

00:30:31.690 --> 00:30:34.960
scientists were claiming that they made
so you had two options either the

00:30:34.960 --> 00:30:38.050
discovery was wrong or your
implementation was wrong so you really

00:30:38.050 --> 00:30:42.280
had to get that implementation right so
you knew what was going on so he wrote

00:30:42.280 --> 00:30:50.260
down our first instantiation of how the
report on the discovery you know telling

00:30:50.260 --> 00:30:53.650
the community about the discovery should
have reproducibility built in and it

00:30:53.650 --> 00:30:56.980
should stand on its own you shouldn't
have to actually write the author and

00:30:56.980 --> 00:31:01.480
ask for more clarification so that was
Boyle skepticism requiring that the

00:31:01.480 --> 00:31:06.460
claim can be independently verified so
we need transparency in the

00:31:06.460 --> 00:31:11.649
communication of these discoveries
and as I said this has happened in the

00:31:11.649 --> 00:31:16.929
1660s also at the beginning of the Royal
Society and first Journal transaction of

00:31:16.929 --> 00:31:22.690
the Royal Society which is kind of
amalgamating all those letters okay so

00:31:22.690 --> 00:31:28.450
now how does that inter and start to
impact the scholarly record okay so a

00:31:28.450 --> 00:31:34.419
couple of things so I think this really
requires to some degree a rethinking of

00:31:34.419 --> 00:31:39.700
the notion of the scholarly record I
have started to think about this as a

00:31:39.700 --> 00:31:46.690
vehicle that gives us access or and/or
the ability to regenerate items that

00:31:46.690 --> 00:31:52.480
were relied on in the generation of
results or giving us access to items

00:31:52.480 --> 00:31:55.720
that were required for independent
replication and reproducibility so those

00:31:55.720 --> 00:32:00.340
are almost saying the same thing the
difference in my mind is whether or not

00:32:00.340 --> 00:32:04.659
we've been able to capture in that
research process all the sort of dead

00:32:04.659 --> 00:32:09.340
ends and avenues that scientists went
down in their discovery process and

00:32:09.340 --> 00:32:13.840
usually don't get reported so really
that's that's coming back to the idea of

00:32:13.840 --> 00:32:17.799
the pea hacking and how we actually
understand the significance of these

00:32:17.799 --> 00:32:25.779
findings okay so what do I mean by items
so I think it's worth distinguishing a

00:32:25.779 --> 00:32:32.470
few of them so digital scholarly objects
articles tax code software data workflow

00:32:32.470 --> 00:32:36.519
information research environment details
details about the state of the machine

00:32:36.519 --> 00:32:39.100
that you're actually running the
experiments on and all the hardware

00:32:39.100 --> 00:32:44.049
details and so lots of different things
could be considered items and I would

00:32:44.049 --> 00:32:48.669
also consider things like material
objects items so these material objects

00:32:48.669 --> 00:32:51.940
go to empirical reproducibility the
digital objects tend to go to

00:32:51.940 --> 00:32:56.049
computational reproducibility so
material objects like can you get your

00:32:56.049 --> 00:33:00.850
hands on reagents lab equipment
instruments texts all that that actually

00:33:00.850 --> 00:33:09.190
go to empirical reproducibility so in
this context thinking about things like

00:33:09.190 --> 00:33:14.200
what version of the data you have what
version of the software you have what

00:33:14.200 --> 00:33:17.889
particular characteristics with reagent
you have all of this is really important

00:33:17.889 --> 00:33:23.260
to be able to recreate that chain that
that original scientist went

00:33:23.260 --> 00:33:29.660
okay so um this is certainly isn't
something that I'm the only one who's

00:33:29.660 --> 00:33:34.250
noticed that there are issues many
scientists have been worried about the

00:33:34.250 --> 00:33:38.840
fact that it's difficult to unpack
computational research to understand and

00:33:38.840 --> 00:33:43.160
find out what actually happened and
different community members have been

00:33:43.160 --> 00:33:47.570
coming up with different solutions so I
put together this is an incomplete list

00:33:47.570 --> 00:33:52.160
but I put together some of these sort of
software infrastructure responses and

00:33:52.160 --> 00:33:55.670
platforms that people have been coming
up with to try and help solve this

00:33:55.670 --> 00:34:00.800
problem so if you're accessing my slides
from my website these are hotlinked I

00:34:00.800 --> 00:34:04.070
highly encourage you to click on some of
these or some of this stuff is just

00:34:04.070 --> 00:34:12.080
really fantastic and really interesting
okay so I think these these efforts sort

00:34:12.080 --> 00:34:15.740
of loosely in an overlapping way sort of
grouped themselves into three groups so

00:34:15.740 --> 00:34:21.409
one is post publication so dissemination
platforms how do you get extra objects

00:34:21.409 --> 00:34:27.800
out there with the publication after
publication another sort of this is

00:34:27.800 --> 00:34:31.070
bright line publication so another group
is pre publication so how do you do

00:34:31.070 --> 00:34:35.359
things like use software tools to better
track all those tests that you did as a

00:34:35.359 --> 00:34:39.260
statistician or all those avenues you
went down and it kind of didn't pan out

00:34:39.260 --> 00:34:43.159
and never kind of made it into the final
narrative so how do you manage workflows

00:34:43.159 --> 00:34:46.909
in a way that helps people share and
track the information that underlies

00:34:46.909 --> 00:34:52.760
their results and then there seems to be
this other bin where people are trying

00:34:52.760 --> 00:34:58.520
to do things like augment the document
that we actually use to communicate our

00:34:58.520 --> 00:35:03.500
results so embedded publishing coming up
with maybe instead of the pedia static

00:35:03.500 --> 00:35:07.609
PDF you could end up with a PDF that
allows you to click on a figure drill

00:35:07.609 --> 00:35:12.260
into code recreate that figure run this
and so so people are doing all sorts of

00:35:12.260 --> 00:35:19.760
innovative work I'll just mention that
almost all of these like I said are just

00:35:19.760 --> 00:35:23.869
researchers putting stuff together
outside their day job because they think

00:35:23.869 --> 00:35:28.150
it's an important problem that needs to
be solved very few of these are

00:35:28.150 --> 00:35:32.810
commercial or industry Venturi there's a
couple of exceptions on there like gene

00:35:32.810 --> 00:35:35.660
pattern is a collaboration with
Microsoft for example there's a few

00:35:35.660 --> 00:35:40.100
but generally speaking these are just
sort of people putting stuff together to

00:35:40.100 --> 00:35:44.180
try and solve the problem and I think is
important so I've been working on our

00:35:44.180 --> 00:35:49.700
research compendium and run my code
so these dissemination platforms so I'll

00:35:49.700 --> 00:35:54.350
just I'll mention research compendium
quickly because it ties together some of

00:35:54.350 --> 00:35:58.250
these threads in a more concrete way
that I've been talking about so far so

00:35:58.250 --> 00:36:00.590
research compendia
you can actually just go to it research

00:36:00.590 --> 00:36:08.120
compendium or --g and it's a prototype
it's a research testbed and we put this

00:36:08.120 --> 00:36:11.600
together
I had a postdoc working on this Jennifer

00:36:11.600 --> 00:36:17.980
Siler and also a developer Sheila migas
who together the three of us built this

00:36:17.980 --> 00:36:25.580
what we wanted to do was understand
better how people would share or what

00:36:25.580 --> 00:36:29.960
tools and platforms I needed to share
reproducible research we wanted to also

00:36:29.960 --> 00:36:36.230
understand whether we could put extra
aspects that made reproducibility easier

00:36:36.230 --> 00:36:42.050
along with those objects so we tried to
do things so we we wanted to link data

00:36:42.050 --> 00:36:47.390
in code to published articles enabling
reuse of the data in code as well as

00:36:47.390 --> 00:36:52.520
reproducing the original results we
wanted to start to understand how

00:36:52.520 --> 00:36:56.000
researchers actually shared and come up
with sort of guidelines or thinking

00:36:56.000 --> 00:37:01.610
about effective ways to share data and
code and workflow we're actually able to

00:37:01.610 --> 00:37:06.140
run the code and data that's given to us
in research camp India so we could

00:37:06.140 --> 00:37:10.340
verify the results the other people
don't have to I mean unless you do it

00:37:10.340 --> 00:37:14.420
your software is trusting us but we're
kind of a third party at least you're

00:37:14.420 --> 00:37:19.910
not only trusting the original
researcher and the idea that once we

00:37:19.910 --> 00:37:23.510
start sharing and collating data we
could actually do things like validate

00:37:23.510 --> 00:37:28.760
findings on much larger datasets like
related datasets for example and as I

00:37:28.760 --> 00:37:34.190
mentioned earlier around statistical
models that may or may not be stable or

00:37:34.190 --> 00:37:38.750
be sensitive to small changes in
calibration with larger data than the

00:37:38.750 --> 00:37:41.930
ability to run this in bigger platforms
we could actually do these stability and

00:37:41.930 --> 00:37:45.740
sensitivity checks so the idea is how
can we end up with a more robust

00:37:45.740 --> 00:37:50.150
scholarly record so this
what a research compendium page would

00:37:50.150 --> 00:37:54.589
look like well it does look like so the
blue at the top is linking to the

00:37:54.589 --> 00:37:57.650
published article so any published
article we would set up a permanent page

00:37:57.650 --> 00:38:02.300
here and the abstract isn't the abstract
from the paper it's talking about the

00:38:02.300 --> 00:38:06.200
code it's talking about the data you can
click on buttons here download code

00:38:06.200 --> 00:38:10.490
download data and you can download the
article this is open access so I think

00:38:10.490 --> 00:38:13.970
this one actually is an open access
articles so you can just go ahead and

00:38:13.970 --> 00:38:18.109
download it and you can see the
different licensing and so on that we've

00:38:18.109 --> 00:38:26.329
got here and here's an example of where
we actually run it so you spot on the

00:38:26.329 --> 00:38:29.990
right of those four buttons we can get
code and data article we can also verify

00:38:29.990 --> 00:38:34.970
so if you wanted you could just click
verify and get the results that we ran

00:38:34.970 --> 00:38:39.020
back on the author's code and data so
you can see whether or not that matches

00:38:39.020 --> 00:38:43.579
the paper and this one does by the way
and you can also see a couple of

00:38:43.579 --> 00:38:47.720
interesting things here at least I think
they're interesting do I on code do I on

00:38:47.720 --> 00:38:57.589
data do I on the the compendium age
itself and because those are those are

00:38:57.589 --> 00:39:01.280
fixed because this is a publication in
the scholarly record so the code may

00:39:01.280 --> 00:39:06.740
evolve however this is the code that got
those results so that code gets frozen

00:39:06.740 --> 00:39:12.260
in that data set gets frozen mistakes
and all and it's sitting there on github

00:39:12.260 --> 00:39:18.020
that's all open-source if you want to
poke into it okay and we use the MIT

00:39:18.020 --> 00:39:22.880
license on our software so that means
pretty much you can grab it and do what

00:39:22.880 --> 00:39:27.650
you want with it we're interested in
attribution so that's why we use the MIT

00:39:27.650 --> 00:39:33.020
license because of course we lean back
on those norms with the scholarly

00:39:33.020 --> 00:39:39.980
community okay so there are community
responses at the same time so now we've

00:39:39.980 --> 00:39:45.680
seen these individual efforts of
researchers who are putting together

00:39:45.680 --> 00:39:50.530
tools or doing individual efforts
there's also a few instances where

00:39:50.530 --> 00:39:54.920
communities have started to engage each
other as an effort to say this is really

00:39:54.920 --> 00:39:59.300
serious we need to start taking these
standards around computational research

00:39:59.300 --> 00:40:03.500
and publication of computational
findings really seriously

00:40:03.500 --> 00:40:09.500
in 2009 I was at Yale Law School and one
of my jobs there was to have a

00:40:09.500 --> 00:40:13.730
roundtable and so that was a number of
years ago and I'd only just graduated

00:40:13.730 --> 00:40:19.250
and so there's kind of a terrifying idea
but I brought 30 people together who are

00:40:19.250 --> 00:40:23.540
32 people together who are different
stakeholders in different fields we had

00:40:23.540 --> 00:40:26.810
some people from funding agencies many
people from academia who were

00:40:26.810 --> 00:40:31.010
researchers and they had this interest
in reproducibility and in what was

00:40:31.010 --> 00:40:35.390
happening with code and data and their
dissemination and that was 2009 and we

00:40:35.390 --> 00:40:43.100
still had enough to bring people
together we put together sort of I put a

00:40:43.100 --> 00:40:47.420
wiki up at the last session and we
collaboratively put together a set of

00:40:47.420 --> 00:40:52.340
principles and things that we would like
to see or that we felt we needed and

00:40:52.340 --> 00:40:57.170
then so we created this kind of
manifesto or document it's the thinking

00:40:57.170 --> 00:41:01.220
has evolved a lot since 2009 but it's
still interesting is that some of the

00:41:01.220 --> 00:41:04.340
things that we were sort of laying out
in that document we laid out things that

00:41:04.340 --> 00:41:07.430
are happening and then we laid out
dreams and things we would really like

00:41:07.430 --> 00:41:14.570
to see in there there was a much larger
gathering in 2012 so this was at Brown

00:41:14.570 --> 00:41:18.590
they run what's called an ice room
series so you get a week and they do

00:41:18.590 --> 00:41:23.000
sort of different topics every week and
so this one reproducibility and

00:41:23.000 --> 00:41:27.680
computational experimental mathematics
we had December 10 to 14 in 2012 so that

00:41:27.680 --> 00:41:32.330
was a lot bigger we brought a lot of
different people together had a somewhat

00:41:32.330 --> 00:41:37.700
similar discussion however of course the
community had advanced and we had many

00:41:37.700 --> 00:41:41.240
more solutions at that point we
understood the problem better and so we

00:41:41.240 --> 00:41:44.450
had a wide-ranging discussion again we
also came up with a workshop report

00:41:44.450 --> 00:41:50.450
these are all you know openly available
this one was a lot longer and we had we

00:41:50.450 --> 00:41:54.200
went through all these different aspects
in a sort of concrete way and we had two

00:41:54.200 --> 00:41:59.810
publications out of here and our goal
setting the default to open or setting

00:41:59.810 --> 00:42:04.040
the default to reproducibility the idea
being sharing code sharing data sharing

00:42:04.040 --> 00:42:08.330
these computational artifacts like
workflows and so on would be the normal

00:42:08.330 --> 00:42:16.010
or expected way to publish computational
findings and the exceptions would be

00:42:16.010 --> 00:42:18.349
things
like you run into privacy issues you ran

00:42:18.349 --> 00:42:20.989
into the proprietary issues or other
problems like that and you make an

00:42:20.989 --> 00:42:25.219
exception for them rather than the
situation we have today where it's just

00:42:25.219 --> 00:42:30.439
normal not to make that those extra
things available extra artifacts and

00:42:30.439 --> 00:42:34.339
exceptionally once in a while people do
so we sort of have exactly the opposite

00:42:34.339 --> 00:42:39.140
to the situation that we're trying to
describe as important here okay so that

00:42:39.140 --> 00:42:44.359
was ice term again community coming
together these are some of the issues in

00:42:44.359 --> 00:42:47.809
the interest of time I won't go through
these all in detail but these are some

00:42:47.809 --> 00:42:50.839
of the things that were flagged as I
said slides on my website if you're

00:42:50.839 --> 00:42:55.069
interested um please feel free to dig in
or ask me any questions we can go back

00:42:55.069 --> 00:42:59.239
to it
there was another workshop so this was

00:42:59.239 --> 00:43:06.769
last summer no summer 2014
two summers ago there's a National

00:43:06.769 --> 00:43:12.650
Science Foundation project that's a
really big project called exceed and

00:43:12.650 --> 00:43:18.619
what exceed does is it builds software
and access points for researchers who

00:43:18.619 --> 00:43:24.170
may not be specialized in using sort of
very high-performance computing machines

00:43:24.170 --> 00:43:27.439
and systems to make it easier for them
to use it so they can use it in their

00:43:27.439 --> 00:43:31.609
and their work so these resources become
available to a much larger community

00:43:31.609 --> 00:43:37.699
other than sort of specialists you can
run it on these sort of though they are

00:43:37.699 --> 00:43:43.549
quite delicate machines and so the
exceed folks and the leadership started

00:43:43.549 --> 00:43:47.959
thinking that maybe that's a place where
reproducibility can be thought about and

00:43:47.959 --> 00:43:55.189
for example supercomputing contexts so
what does it mean to reproduce a result

00:43:55.189 --> 00:44:02.719
that's been generated using one of these
HPC resources maybe the solution is in

00:44:02.719 --> 00:44:07.369
that middleware that sort of bridges
that gap between the researcher and the

00:44:07.369 --> 00:44:11.390
actual resource and it can do things
like what I was talking about earlier

00:44:11.390 --> 00:44:15.979
like save the important pieces of code
this the order that you ran that did

00:44:15.979 --> 00:44:19.429
your different functions in what the
parameter settings were what the Machine

00:44:19.429 --> 00:44:22.670
state was at the time you're running in
different libraries you needed to run it

00:44:22.670 --> 00:44:26.809
all of these sort of pieces of
information that help you understand

00:44:26.809 --> 00:44:30.470
that output so maybe that's something we
can automate in

00:44:30.470 --> 00:44:35.240
in that sort of middleware exceed layer
and because that's kind of a unique

00:44:35.240 --> 00:44:39.380
opportunity usually there isn't that
mediation I would just run stuff on my

00:44:39.380 --> 00:44:46.369
laptop right but that sort of
opportunity allows us to sort of push on

00:44:46.369 --> 00:44:51.079
what it would mean to really have
reproducibility and what we need to

00:44:51.079 --> 00:44:54.800
gather and sort of understand that
better so that's an ongoing discussion

00:44:54.800 --> 00:44:59.390
because that was only summer of 2014 but
I think it's interesting the way these

00:44:59.390 --> 00:45:02.500
different communities are starting to
take on and think about these issues

00:45:02.500 --> 00:45:07.670
okay in concert with that at the same
time and again many of you probably are

00:45:07.670 --> 00:45:10.970
familiar with some of these but there's
been a number of directives coming

00:45:10.970 --> 00:45:13.819
primarily from the White House and from
the Office of Science and Technology

00:45:13.819 --> 00:45:18.800
Policy and other areas in in Washington
there's one that just happened this

00:45:18.800 --> 00:45:23.200
summer 2015 the National strategic
computing initiative I'll just highlight

00:45:23.200 --> 00:45:29.720
their strategic objective for if you're
interested they say what they want is a

00:45:29.720 --> 00:45:37.430
computing ecosystem that embraces access
they actually use the word and software

00:45:37.430 --> 00:45:41.599
and workflows and they don't use the
word reproducibility which i think is

00:45:41.599 --> 00:45:45.079
really unfortunate but they're kind of
getting close right they're starting to

00:45:45.079 --> 00:45:49.569
think about what it means in in this
sort of high performance computing

00:45:49.569 --> 00:45:54.319
context what it means to do work that's
actually subsequently available to the

00:45:54.319 --> 00:45:57.440
community
I'm probably everyone knows about the

00:45:57.440 --> 00:46:01.640
2013 open data open access executive
memorandum and the executive order to

00:46:01.640 --> 00:46:05.420
the federal agencies making data
available and making articles available

00:46:05.420 --> 00:46:10.910
and there are there things that are
happening like different data management

00:46:10.910 --> 00:46:15.800
plans or emerging NIST is developing
their common access platform so things

00:46:15.800 --> 00:46:20.240
are kind of happening I don't know of
anyone who's mandating reproducibility

00:46:20.240 --> 00:46:24.770
but there are these little kind of
nibbles and steps that are happening now

00:46:24.770 --> 00:46:31.760
I think it's really important to have
these types of activities happening at

00:46:31.760 --> 00:46:36.200
the funding agency level and happening
at the OSTP level at the White House

00:46:36.200 --> 00:46:41.030
because as you saw there like on the
page around infrastructure of these

00:46:41.030 --> 00:46:44.120
scientists for examples and researchers
had been coming up with their own

00:46:44.120 --> 00:46:50.570
Lucian's two problems and yet with the
problems not solved right because any

00:46:50.570 --> 00:46:54.620
one researcher can take these extra
efforts to produce for example

00:46:54.620 --> 00:46:59.180
reproducible research if they're not
rewarded in the community then it's

00:46:59.180 --> 00:47:03.800
really not something that you can kind
of encourage for example junior faculty

00:47:03.800 --> 00:47:07.220
postdocs or students to do they really
need to focus on doing things that are

00:47:07.220 --> 00:47:12.290
rewarded in their career and that
advance them so we have this collective

00:47:12.290 --> 00:47:18.950
action problem where none of us can kind
of act individually to fix this what

00:47:18.950 --> 00:47:23.000
solves the problem is when we all act
together and so those are really really

00:47:23.000 --> 00:47:31.070
hard problems something like nudges from
OSTP help move us all together so they

00:47:31.070 --> 00:47:34.640
help to solve this collective action
problem in general I'm not a fan

00:47:34.640 --> 00:47:40.730
personally I've sort of sort of top-down
management of the scientific community

00:47:40.730 --> 00:47:45.620
and that's been something that's been
resisted for its entire existence but

00:47:45.620 --> 00:47:48.590
that collective action problem is
serious enough where it just won't be

00:47:48.590 --> 00:47:52.100
broken by internally everyone just
waking up one morning and going up I

00:47:52.100 --> 00:47:55.610
think I'm gonna do it
reproducibly today it's just not gonna

00:47:55.610 --> 00:47:58.370
happen so there's sort of this
grassroots

00:47:58.370 --> 00:48:02.420
emergence of solutions I think it also
needs her to this top-down kind of

00:48:02.420 --> 00:48:07.700
nudging us all together okay I'll just
mention and I won't go into detail again

00:48:07.700 --> 00:48:10.640
in the interest of time but it's not
just funding agencies in the White House

00:48:10.640 --> 00:48:14.690
and those stakeholders that are
discussing reproducibility it's not just

00:48:14.690 --> 00:48:18.890
researchers journals are also doing this
and I looked into this in a paper there

00:48:18.890 --> 00:48:25.640
are requirements that are popping up in
journals all over the place around data

00:48:25.640 --> 00:48:28.790
availability requirements code
availability requirements are coming

00:48:28.790 --> 00:48:34.190
science has required code and data
sharing since 2011 nature requires data

00:48:34.190 --> 00:48:38.600
sharing for example so there's sort of
other kind of breaks to this collective

00:48:38.600 --> 00:48:42.770
action problem that that are occurring
again one journal isn't going to change

00:48:42.770 --> 00:48:51.860
the landscape so okay I just wanted to
mention too about how this availability

00:48:51.860 --> 00:48:56.630
of code availability of data opens up
opportunities outside the ivory tower

00:48:56.630 --> 00:49:00.890
and it used to be
for almost all of science that most of

00:49:00.890 --> 00:49:04.880
the discussions around discovery around
verifiability around integrity and

00:49:04.880 --> 00:49:09.410
discovery really happened with in the
ivory tower within communities who sort

00:49:09.410 --> 00:49:14.360
of knew each other and knew the problems
and so on now we share things more

00:49:14.360 --> 00:49:18.020
openly on the web we have this
opportunity to really engage many more

00:49:18.020 --> 00:49:21.500
brains
so crowdsourcing public engagement in

00:49:21.500 --> 00:49:26.660
science these things are sort of
happening and emerging when I looked

00:49:26.660 --> 00:49:29.750
into this largely around just data
collection so those sort of donating

00:49:29.750 --> 00:49:35.990
data to scientific experiments I didn't
see much in the way of say you know I

00:49:35.990 --> 00:49:41.210
was I was talking in the beginning about
discovery is embedded in software and so

00:49:41.210 --> 00:49:44.600
on I haven't seen that yet from
crowdsourcing community how are that

00:49:44.600 --> 00:49:47.450
engagements there and it's starting to
grow so that's something I'm very

00:49:47.450 --> 00:49:53.540
optimistic about it requires I think
access to I put it in quotes coherent

00:49:53.540 --> 00:49:57.440
digital scholarly objects a lot going on
in there like can you understand what

00:49:57.440 --> 00:50:01.070
the data are saying are they correctly
documented is there metadata how do you

00:50:01.070 --> 00:50:04.670
know what versions you're using apply
all that to software as well ever it has

00:50:04.670 --> 00:50:07.100
all of those same things have a
different meaning in the software

00:50:07.100 --> 00:50:10.730
context and workflow context for example
but can you actually figure out what the

00:50:10.730 --> 00:50:15.920
person was doing when they actually
shared those objects we need a mechanism

00:50:15.920 --> 00:50:20.140
for bringing in ingesting or evaluating
new findings that might come from

00:50:20.140 --> 00:50:25.220
non-traditional sources it's not it's
sort of an insulin insider's game to

00:50:25.220 --> 00:50:28.730
submit to a journal like how to actually
write it the way that it gets taken

00:50:28.730 --> 00:50:33.050
seriously and all these sort of more
political aspects so how do we how do we

00:50:33.050 --> 00:50:37.670
deal with findings that came from the
source we didn't expect or an amateur

00:50:37.670 --> 00:50:40.550
maybe you might call it even though they
might be just a skill but they just have

00:50:40.550 --> 00:50:48.680
a different style lots of legal issues
reuse privacy using digital scholarly

00:50:48.680 --> 00:50:54.080
objects and so on and you see this
everywhere so you you like Washington is

00:50:54.080 --> 00:50:58.100
all abuzz with evidence-based whatever
evidence based policy evidence-based

00:50:58.100 --> 00:51:04.430
medicine evidence-based decision making
and and so that's something that I think

00:51:04.430 --> 00:51:10.430
is a new topic how do you actually know
that this is reliable evidence do I need

00:51:10.430 --> 00:51:14.990
to get the code do I do
the data to be a voting citizen for

00:51:14.990 --> 00:51:24.110
example okay my last topic access and
intellectual property so I mentioned at

00:51:24.110 --> 00:51:27.920
the outset that intellectual property
was something reasonably new that the

00:51:27.920 --> 00:51:30.530
scientific community was having to deal
with

00:51:30.530 --> 00:51:34.220
I'll principally talk about copyright
although I can talk about patents a

00:51:34.220 --> 00:51:41.480
little bit copyright presents a very
important barrier to reproducibility and

00:51:41.480 --> 00:51:47.930
to reuse of shared digital scholarly
objects it's embedded right in our

00:51:47.930 --> 00:51:52.970
Constitution so here's the language so
copyright to promote the progress of

00:51:52.970 --> 00:51:58.250
science and the useful arts or any
useful arts by securing for limited

00:51:58.250 --> 00:52:01.700
times to authors and inventors the
exclusive right to their respective

00:52:01.700 --> 00:52:08.180
writings and discoveries so the idea is
you created and owned the copyright you

00:52:08.180 --> 00:52:13.670
also own how it gets subsequently used
belongs to you so that's the idea of

00:52:13.670 --> 00:52:18.800
copyright so any original expression of
an idea so for example writing a paper

00:52:18.800 --> 00:52:23.270
writing a piece of software making it
making that idea manifest somehow

00:52:23.270 --> 00:52:28.310
automatically falls under copyright so
you don't need to register or do

00:52:28.310 --> 00:52:33.290
anything it's copyright to you so think
of that in the software context you've

00:52:33.290 --> 00:52:36.530
written a script that produces a really
beautiful figure that's going to go in

00:52:36.530 --> 00:52:42.980
your paper script this copyright to you
applies to not just code papers figures

00:52:42.980 --> 00:52:50.360
tables and so on so what copyright does
is it says the copyright holder is the

00:52:50.360 --> 00:52:55.910
only one who's allowed to reproduce the
work make a copy you've got the right to

00:52:55.910 --> 00:53:00.200
copy it and the other thing that
copyright says is you are the only one

00:53:00.200 --> 00:53:03.590
is a copyright holder who can prepare
derivative works based on the original

00:53:03.590 --> 00:53:08.450
work so for example standing on the
shoulder of giants no this is copyright

00:53:08.450 --> 00:53:14.180
so both of these things work completely
against our norms as scientists I want

00:53:14.180 --> 00:53:18.110
people to reproduce my work I would love
it if they took my work and built on it

00:53:18.110 --> 00:53:22.880
and cited me that is directly impactful
for my career right that's how all right

00:53:22.880 --> 00:53:25.370
Center system is set up and copyright
works right

00:53:25.370 --> 00:53:30.470
against both of those and you might
think it would disappear it's basically

00:53:30.470 --> 00:53:35.750
in perpetuity 70 years plus the life of
the author so if you think about the

00:53:35.750 --> 00:53:40.520
value of software and how quickly it
deprecates really is just talking about

00:53:40.520 --> 00:53:45.710
infinity there so that's a sketch there
are some exceptions to copyright like

00:53:45.710 --> 00:53:49.160
fair use for example but they certainly
don't stretch to what I've been

00:53:49.160 --> 00:53:53.540
describing about reusing code reusing
data within the scientific context of

00:53:53.540 --> 00:54:00.670
reproducibility okay so probably most of
you recognize Richard Stallman here who

00:54:00.670 --> 00:54:04.990
arguably kicked off the open source
software movement with the invention of

00:54:04.990 --> 00:54:11.900
a rider or a piece of text that would go
with a copyright object and just pre

00:54:11.900 --> 00:54:16.070
permission in advance how you could use
that object so having something under

00:54:16.070 --> 00:54:20.210
copyright so I can't reproduce your
worst I can't create derivative works

00:54:20.210 --> 00:54:24.530
based on your work but I could have I
called you and got permission or asked

00:54:24.530 --> 00:54:27.620
you
and so Salman's genius was to say well

00:54:27.620 --> 00:54:30.470
I'll just pretend you asked and I'll
just put the answer out there and you

00:54:30.470 --> 00:54:35.270
can reuse this in these according to my
particular Terms of Use so that's open

00:54:35.270 --> 00:54:40.090
licensing and he of course created the
new public license that way since then

00:54:40.090 --> 00:54:44.000
hundreds and hundreds and hundreds of
licenses are written and created so you

00:54:44.000 --> 00:54:48.350
sort of take your pick whatever you want
it's probably out there and it sort of

00:54:48.350 --> 00:54:53.990
is an end run around the sort of
restrictions a copyright places on the

00:54:53.990 --> 00:54:56.750
work the work is still copyright to the
author but the author has pre

00:54:56.750 --> 00:55:03.470
permissioned the use okay so in 2001
Larry Lessig who was inspired by Richard

00:55:03.470 --> 00:55:07.280
Stallman
with two co-founders created Creative

00:55:07.280 --> 00:55:14.210
Commons and what he did or what they did
was develop sort of four aspects of

00:55:14.210 --> 00:55:17.750
licenses that they thought were
important for artistic works so an

00:55:17.750 --> 00:55:23.840
artist might want to share say a video
or an image and they might not want all

00:55:23.840 --> 00:55:26.840
the protection of copyright look it's
okay if you use my image and say Oh

00:55:26.840 --> 00:55:30.410
collage or something and you might want
to do the same thing that Stallman did

00:55:30.410 --> 00:55:34.640
and pre permission to use so that's not
Creative Commons data sort of printed or

00:55:34.640 --> 00:55:39.230
created licenses that allowed you to
sort of do that for artistic works

00:55:39.230 --> 00:55:43.700
Stallman was only four code Creative
Commons is never for code so they sort

00:55:43.700 --> 00:55:48.980
of filled in two different gaps there no
one had thought about what do you do in

00:55:48.980 --> 00:55:53.119
the scientific context cuz our norms are
different they're different from say the

00:55:53.119 --> 00:55:56.600
artistic context and they're different
from the open source software context

00:55:56.600 --> 00:56:00.380
we're not doing that in science either
of those we have relationships and

00:56:00.380 --> 00:56:06.020
commonalities with what they might
consider their norms Creative Commons or

00:56:06.020 --> 00:56:10.790
open-source software communities but we
have a slightly different set of norms

00:56:10.790 --> 00:56:14.630
in the scientific community so the idea
of the reproducible research standard

00:56:14.630 --> 00:56:21.230
was guidelines around license use for
researchers and for scientists releasing

00:56:21.230 --> 00:56:26.780
code releasing data and and their
articles and so the idea was use open

00:56:26.780 --> 00:56:32.570
licenses that match our scientific norms
pretty much say you can do whatever you

00:56:32.570 --> 00:56:36.830
want with my stuff I just want you to
attribute me that's pretty much what we

00:56:36.830 --> 00:56:40.790
do is scientists if you go back to
communalism things become property of

00:56:40.790 --> 00:56:44.859
the community but we want to be
recognized and cited for our efforts so

00:56:44.859 --> 00:56:48.920
releasing our media components non code
components under Creative Commons

00:56:48.920 --> 00:56:54.830
Attribution license code components
under MIT license or modified BSD

00:56:54.830 --> 00:56:58.880
license which was equivalent to the
attribution-only license for software

00:56:58.880 --> 00:57:05.150
and data is complicated but I recommend
that you dedicate data to the public

00:57:05.150 --> 00:57:08.390
domain or maybe you can attach an
appropriate attribution license to it

00:57:08.390 --> 00:57:12.470
and the idea then is when you share
objects the copyright holder and a

00:57:12.470 --> 00:57:16.640
researcher you do it in such a way that
the community recognizes what they can

00:57:16.640 --> 00:57:20.060
do with it right they can reproduce your
work they can build on your work insight

00:57:20.060 --> 00:57:26.930
you just the same way our norms have
always been I'll mention data quickly in

00:57:26.930 --> 00:57:34.790
the u.s. we have no copyright on raw
facts interestingly in Europe they do

00:57:34.790 --> 00:57:41.240
have copyright on raw facts and as
everyone knows I'm sure it's very

00:57:41.240 --> 00:57:44.680
frequent to have international
collaborations or collaborations across

00:57:44.680 --> 00:57:51.260
different legal systems and science
stuff that happens all the time what we

00:57:51.260 --> 00:57:54.630
have in the US
in court case it says raw facts are not

00:57:54.630 --> 00:57:59.490
copyrightable however original selection
and arrangement of the raw facts are

00:57:59.490 --> 00:58:07.320
copyrightable so what does that mean in
the scientific context it's like totally

00:58:07.320 --> 00:58:11.850
unclear this was the this Court came out
of a fight over phone books right oh

00:58:11.850 --> 00:58:15.690
it's nothing to do with science there
haven't been cases subsequent to that

00:58:15.690 --> 00:58:20.460
that flush this out in the scientific
context so we don't know what that means

00:58:20.460 --> 00:58:25.740
I don't know exactly what a raw fact is
I could take some guesses

00:58:25.740 --> 00:58:28.830
however original selection the
arrangement of our effect you can make

00:58:28.830 --> 00:58:32.940
an argument around that maybe some kind
of copyright is creeping in if I take

00:58:32.940 --> 00:58:36.480
sort of some from this data set or from
some mashup from that data set and then

00:58:36.480 --> 00:58:40.020
sort of maybe the container is copyright
to me so that's why I say to the public

00:58:40.020 --> 00:58:43.890
domain dedication it just gets rid of
all those residual copyright issues that

00:58:43.890 --> 00:58:53.280
may be floating around with your data a
quick note on patents so bayh-dole act

00:58:53.280 --> 00:58:59.300
was passed in 1980 what the bayh-dole
act did is it charged universities with

00:58:59.300 --> 00:59:04.650
ownership over inventions that were
happening on their campus and using

00:59:04.650 --> 00:59:08.490
their resources and then said you should
seek the patent system to actually

00:59:08.490 --> 00:59:12.720
capitalize on these inventions and you
can generate a revenue stream and the

00:59:12.720 --> 00:59:16.170
idea being the inventions then actually
made their way out into the larger

00:59:16.170 --> 00:59:19.770
community and could start companies and
help industry and make us all richer and

00:59:19.770 --> 00:59:24.600
so on so that if you notice the nineteen
eighty right we weren't using computers

00:59:24.600 --> 00:59:29.330
then in research so now there's this
interesting thing that's happening where

00:59:29.330 --> 00:59:35.940
algorithm in code software's can be
patentable universities charge through

00:59:35.940 --> 00:59:40.050
bayh-dole have this stance towards
patentability so I've just spent

00:59:40.050 --> 00:59:44.040
probably what felt like about three
hours telling you that I think we need

00:59:44.040 --> 00:59:47.850
to release code and make it available on
I even gave you some normative structure

00:59:47.850 --> 00:59:51.930
for how we should release it and then we
run smack into bayh-dole that says oh

00:59:51.930 --> 00:59:55.500
actually if that's patentable maybe the
university should be making a patent on

00:59:55.500 --> 01:00:00.210
this and making it available through the
tech transfer office for a fee so we are

01:00:00.210 --> 01:00:05.100
starting to see these kind of you know
different kind of norms budding against

01:00:05.100 --> 01:00:10.320
each other so I think that's about
all that's coming actually okay a little

01:00:10.320 --> 01:00:14.280
bit more on ownership of research codes
just to say that universities do it very

01:00:14.280 --> 01:00:19.650
differently and the rights you have over
your inventions differ from institution

01:00:19.650 --> 01:00:24.329
to institution and people are starting
to also push with ideas around sharing

01:00:24.329 --> 01:00:28.170
code similar to how we saw movements
around sharing data so code seems to be

01:00:28.170 --> 01:00:34.230
kind of picking up the steam the same
way data actually did okay so my very

01:00:34.230 --> 01:00:41.640
last slide we have stakeholders
researchers funding agency folks people

01:00:41.640 --> 01:00:47.430
in the white house journal editors
acting to some degree independently now

01:00:47.430 --> 01:00:51.089
remember they're all subdivided by
different fields right they are probably

01:00:51.089 --> 01:00:54.930
not even talking to each other and there
many of them are trying to take on and

01:00:54.930 --> 01:00:58.380
solve these problems of reproducibility
so the idea that coordination can help

01:00:58.380 --> 01:01:02.660
us solve that collective action problem

01:01:03.290 --> 01:01:07.020
so I shouldn't have probably called it
conservative what I meant was sort of

01:01:07.020 --> 01:01:12.420
minimalistic proposal scholarly record
comprises access to ability to

01:01:12.420 --> 01:01:16.800
regenerate items relied on in the
generator in the generation of the

01:01:16.800 --> 01:01:20.849
results that were actually published so
it includes things like getting your

01:01:20.849 --> 01:01:24.390
hands on the software getting your hands
on how the software was actually

01:01:24.390 --> 01:01:26.970
implemented what were the parameter
settings what data did you start with

01:01:26.970 --> 01:01:33.030
what data did you produce as an output
and and this idea of access can unify

01:01:33.030 --> 01:01:36.900
our approach and instantiate these
notions of reproducibility in the

01:01:36.900 --> 01:01:47.409
scholarly record so I'll stop there and
take questions

01:01:49.760 --> 01:02:11.820
yeah that's a great question
yeah that's a really great question so

01:02:11.820 --> 01:02:17.190
so I cleverly sidestepped any concrete
examples notice because they get very

01:02:17.190 --> 01:02:21.000
difficult and what you mean by
reproducibility very much depends not

01:02:21.000 --> 01:02:25.980
just on the actual nature of the problem
and so that is true that some people and

01:02:25.980 --> 01:02:30.510
some research progresses by stages and
so you might think of reproducibility as

01:02:30.510 --> 01:02:33.930
sort of stage wise checks for example
throughout your progress it depends also

01:02:33.930 --> 01:02:36.990
when you publish so if you're publishing
in different parts in different stages

01:02:36.990 --> 01:02:40.230
maybe there's a more holistic way of
thinking about reproducibility makes

01:02:40.230 --> 01:02:45.150
sense in that context and also in
different communities the norms are just

01:02:45.150 --> 01:02:48.630
really different and so you would expect
culturally different ways of sort of

01:02:48.630 --> 01:02:51.930
moving towards reproducibility so one of
the things that I've been thinking about

01:02:51.930 --> 01:02:57.000
is what does it mean to do software
testing in science and in the

01:02:57.000 --> 01:03:00.180
open-source software community you don't
contribute a piece of code unless you

01:03:00.180 --> 01:03:03.930
have tests that show how it works and so
on so I would think also it might be

01:03:03.930 --> 01:03:07.230
like maybe these immediate intermediary
stages that you're hitting on could act

01:03:07.230 --> 01:03:10.650
like tests in the open-source software
world where they're sort of checks where

01:03:10.650 --> 01:03:13.770
you know that the software is behaving
how it should and then that's how you

01:03:13.770 --> 01:03:16.800
start to instantiate reproducibility
it's just an idea I mean the the

01:03:16.800 --> 01:03:20.070
different communities are going to solve
the problems I think in different ways

01:03:20.070 --> 01:03:23.490
so it's soon I won't be able to give a
high level talk like this anymore

01:03:23.490 --> 01:03:30.350
because people are just gonna ask me
questions like that other questions yes

01:03:32.780 --> 01:03:37.190
this is a descendant of that thinking

01:03:40.520 --> 01:03:49.320
yeah no it hasn't no I think we just
haven't had the pressure to do it I

01:03:49.320 --> 01:03:54.060
don't I'm not convinced it's necessarily
that hard but it does unfurl itself into

01:03:54.060 --> 01:03:56.970
a more complicated problem as soon as
you start thinking about it so it's one

01:03:56.970 --> 01:04:00.540
thing to take short MATLAB scripts maybe
to demonstrate an image compression

01:04:00.540 --> 01:04:03.540
algorithm and sort of check that they're
working your tests could even actually

01:04:03.540 --> 01:04:07.500
be certain images that you've just
embedded in your paper it's a different

01:04:07.500 --> 01:04:12.210
thing to say well what if I have you
know I've joined a research group they

01:04:12.210 --> 01:04:16.620
have codes that have been around and
people have been working on them for 25

01:04:16.620 --> 01:04:19.920
years there's even sort of old Fortran
codes embedded in the middle no one

01:04:19.920 --> 01:04:23.520
wants to touch it right no one's gonna
it kind of works so we're not gonna open

01:04:23.520 --> 01:04:29.160
that door and so what is testing mean in
that context and and arguably it's

01:04:29.160 --> 01:04:32.190
incredibly important because I'm
certainly not reading that code there

01:04:32.190 --> 01:04:34.830
was kind of this assumption all the way
through that you could look at the code

01:04:34.830 --> 01:04:37.800
and somehow that was helpful you're not
always gonna be able to look at the code

01:04:37.800 --> 01:04:42.390
or run it for example so then how do we
sort of modular eyes and test pieces or

01:04:42.390 --> 01:04:46.260
reuse pieces we just haven't because it
hasn't been part of the scholarly record

01:04:46.260 --> 01:04:50.400
and our normal communication and
dissemination mechanism we just haven't

01:04:50.400 --> 01:04:53.430
even sort of faced those problems and
we're starting to and I'm starting to

01:04:53.430 --> 01:05:00.360
see things like do we put out calls for
interoperability and checks before they

01:05:00.360 --> 01:05:03.540
can get on to big systems to make sure
that they've sort of you know they've

01:05:03.540 --> 01:05:08.850
got modularity in there that it's going
to be to some degree as platform

01:05:08.850 --> 01:05:12.810
independent as they can make and so this
this thinking is starting to emerge but

01:05:12.810 --> 01:05:16.890
it's very very young and I think this is
a whole new kind of field where we can

01:05:16.890 --> 01:05:21.510
really kind of use that the heritage of
Knuth to make our work a lot better I

01:05:21.510 --> 01:05:25.080
mean I think we're gonna get it all and
software and that's where we gotta go

01:05:25.080 --> 01:05:31.880
and got to think about yeah

01:05:46.350 --> 01:05:57.910
that's great yep yep there so that I
have a multi-faceted response to that so

01:05:57.910 --> 01:06:04.150
the first thing is in my experience
students are very excited by these

01:06:04.150 --> 01:06:08.740
issues in my experience students come
into technical or computational research

01:06:08.740 --> 01:06:12.910
and they're floored by the fact that
we're not acting like an open source

01:06:12.910 --> 01:06:15.250
group like they're used to before they
got here

01:06:15.250 --> 01:06:21.040
and I have seen it where this actually
causes a lot loss of confidence in what

01:06:21.040 --> 01:06:24.580
we're producing in the scientific
community and they start to doubt what

01:06:24.580 --> 01:06:28.240
we're doing because it just sort of
seems like it's not prioritized like is

01:06:28.240 --> 01:06:34.060
the code right nobody checks what and so
so I think there's a role there for

01:06:34.060 --> 01:06:37.150
really supporting those students because
I think we actually are in danger of

01:06:37.150 --> 01:06:41.860
losing people who are initially very
motivated around science and then sort

01:06:41.860 --> 01:06:45.580
of see things that are less than optimal
and how we're carrying it out and that

01:06:45.580 --> 01:06:49.240
we haven't adapted to these
computational methods however I don't

01:06:49.240 --> 01:06:53.100
think it's on them even though they're
the most excited group to implement it

01:06:53.100 --> 01:06:57.010
because of that reward structure and I
don't think there's anything I this is

01:06:57.010 --> 01:06:59.440
my personal opinion I don't think
there's anything wrong with a young

01:06:59.440 --> 01:07:04.200
student publishing in such a way or
doing work in such a way that is

01:07:04.200 --> 01:07:08.320
currently rewarded so they can advance
their career I don't think it's on them

01:07:08.320 --> 01:07:13.000
to take the time to do all this stuff
however I do think it is on the older

01:07:13.000 --> 01:07:17.740
more established faculty to sort of lead
the way and start doing that so things

01:07:17.740 --> 01:07:24.640
like establishing where you might put
the code that you develop as part of

01:07:24.640 --> 01:07:28.690
your group that happened to me when I
was a research group so we made

01:07:28.690 --> 01:07:33.820
platforms ourselves like I don't
recommend that we should have made our

01:07:33.820 --> 01:07:38.140
own webpages to do it you have like a
whole ivory system here they can support

01:07:38.140 --> 01:07:41.650
this this is something that they
shouldn't be done on your own personal

01:07:41.650 --> 01:07:44.500
webpage that everyone moves institutions
and so on

01:07:44.500 --> 01:07:47.980
but that was part of what I learned as a
graduate student and part of that was so

01:07:47.980 --> 01:07:51.010
my advisor could see what we were doing
and knew what we were doing and knew

01:07:51.010 --> 01:07:54.970
what we were publishing so I think those
types of methods and faculty could have

01:07:54.970 --> 01:07:57.880
maybe guidance around what tools are
they're gonna use github for doing this

01:07:57.880 --> 01:08:03.130
are they gonna use something here for
housing code or data how do versions

01:08:03.130 --> 01:08:07.000
attached to this and just sort of having
some very lightweight steps and and I my

01:08:07.000 --> 01:08:10.599
sense of students will just sort of
inhale it like that's not an adoption

01:08:10.599 --> 01:08:17.680
problem if this lightweight structures
get set up and and I think there's lik

01:08:17.680 --> 01:08:21.190
sort of mini collective action problems
at this level too so things like a Dean

01:08:21.190 --> 01:08:24.850
or a chair could say well here's our
recommendation we're putting code out

01:08:24.850 --> 01:08:28.900
we're putting data out that's strong but
if they just sort of make an

01:08:28.900 --> 01:08:32.650
encouragement around it it can help set
a tone with it I'm not necessarily

01:08:32.650 --> 01:08:35.380
saying they need to make requirement for
all the reasons actually that you

01:08:35.380 --> 01:08:39.670
mentioned that what that means in
different research groups to release

01:08:39.670 --> 01:08:43.719
code data reproducibility it's wildly
different and it's much more difficult

01:08:43.719 --> 01:08:46.750
for some people and for other people
some people have industry collaborations

01:08:46.750 --> 01:08:49.960
they have to untangle you know like
there's there's a lot of details that

01:08:49.960 --> 01:08:53.319
come on there so I don't know if that's
helpful I really think it's the older

01:08:53.319 --> 01:09:13.180
folks that gotta lead the way on this
yeah yeah but so does providing the

01:09:13.180 --> 01:09:18.940
proof right of math it may I think it
could it could make it a lot slower and

01:09:18.940 --> 01:09:59.590
a lot more expensive yeah yeah yeah I
know so even though I'm saying it's the

01:09:59.590 --> 01:10:01.989
older folks that need to lead I think
it's younger people that are really

01:10:01.989 --> 01:10:08.160
going to adopt this stuff one of my
colleagues says to teach the puppies

01:10:08.160 --> 01:10:13.480
well I think a lot of this is tools and
again it's it's so granular for the

01:10:13.480 --> 01:10:17.020
particular application but I feel like
if we were using infrastructure tools

01:10:17.020 --> 01:10:20.170
that capture these things more
automatically and made things easier it

01:10:20.170 --> 01:10:24.489
wouldn't be this necessarily this huge
burden in some situations to really

01:10:24.489 --> 01:10:28.900
produce reproducible research for the
computational aspects so that's why I'm

01:10:28.900 --> 01:10:32.950
sort of talking like you're saying the
tool development is also part of it

01:10:32.950 --> 01:10:37.960
that's what I so I think tools are a big
part of the solution how they get

01:10:37.960 --> 01:10:43.870
developed and funded and promulgated is
another question but you're raising all

01:10:43.870 --> 01:10:49.719
the real problems I mean you saw my Yale
roundtable in 2009 I think a reasonable

01:10:49.719 --> 01:10:53.320
question would be if so what's changed
since you did that in 2009 there's been

01:10:53.320 --> 01:10:58.030
some changes but not seven years worth
of changes right that you would expect

01:10:58.030 --> 01:11:03.730
it's very slow so these are real
problems but I don't think that would

01:11:03.730 --> 01:11:08.140
stop us from sort of chipping away and
there is some evidence that sharing code

01:11:08.140 --> 01:11:11.739
sharing data and so and increases
citation rates for articles now that may

01:11:11.739 --> 01:11:16.000
be confounded because in my opinion I
see very good scholars releasing data

01:11:16.000 --> 01:11:17.920
releasing Co
so maybe there's a confounding factor

01:11:17.920 --> 01:11:22.900
there however it sort of starts to set
that that example and I certainly have

01:11:22.900 --> 01:11:26.619
more confidence in work where I can get
a hold of code and then when I can't

01:11:26.619 --> 01:11:30.159
when it's computational so I think
there's a natural advantage to doing it

01:11:30.159 --> 01:11:36.040
as well as the sort of cultural
advantages but ya know I if I thought I

01:11:36.040 --> 01:11:43.300
could sit there and change every Nobel
you know want to be mind about how they

01:11:43.300 --> 01:12:14.770
need to do their research I mean that's
yeah yeah no that's not naive it you're

01:12:14.770 --> 01:12:19.599
starting to bring in an interesting link
so I haven't talked about open access at

01:12:19.599 --> 01:12:22.630
all and you can see there's sort of
philosophical relationships between the

01:12:22.630 --> 01:12:25.690
issues and one of the reasons is because
there's lots of discussion around open

01:12:25.690 --> 01:12:29.349
access not as much around these issues
so I tend to sort of focus on these

01:12:29.349 --> 01:12:34.330
issues but some of the barriers to open
access are also barriers in the sort of

01:12:34.330 --> 01:12:37.659
world that you just described because we
start now interacting with publishers

01:12:37.659 --> 01:12:41.889
with the established journals holding
impact factor and holding sort of a more

01:12:41.889 --> 01:12:46.739
conservative walk over the way we do it
more innovative journals may perhaps be

01:12:46.739 --> 01:12:50.320
newer journals and then it's harder to
attract because they don't have the

01:12:50.320 --> 01:12:53.619
impact factor and they're doing things
like certifying code for example so

01:12:53.619 --> 01:12:58.389
those are open questions and again I
don't have easy solutions for those

01:12:58.389 --> 01:13:02.619
however I do know that we make you know
forward progress little steps by little

01:13:02.619 --> 01:13:06.580
stuff and so there are there are
journals like biostatistics for example

01:13:06.580 --> 01:13:09.639
and there's some other journals that
will run code and they'll put a kite

01:13:09.639 --> 01:13:13.630
mark on the big R or something on your
article when you're published in their

01:13:13.630 --> 01:13:19.690
journal and so that's still ongoing open
experiment to the impact but it starts

01:13:19.690 --> 01:13:22.659
to at least the community can take
advantage of these things and they are

01:13:22.659 --> 01:13:27.310
voluntarily taking advantage of it to
try and you know display the integrity

01:13:27.310 --> 01:13:30.389
of the work that they're publishing

01:13:33.000 --> 01:13:40.350
other questions yeah

01:14:14.540 --> 01:14:20.010
not that I know of no not that I know
however I've heard that before so in

01:14:20.010 --> 01:14:24.120
research compendium run my code where
we're gathering code that's associated

01:14:24.120 --> 01:14:28.110
with publications a couple of people
have said you don't will be really

01:14:28.110 --> 01:14:31.620
useful like you got the code in our give
me the state of version of the code or

01:14:31.620 --> 01:14:35.700
vice versa
or something it's really hard they're

01:14:35.700 --> 01:14:39.870
really hard for all so I don't I don't
know how to solve that in an easy way I

01:14:39.870 --> 01:14:44.280
think that is a burden that it's going
to be really difficult to push back to

01:14:44.280 --> 01:14:47.850
reviewers but that's really interesting
and and where I thought you were going

01:14:47.850 --> 01:14:53.610
to go with your comment was if I have a
mathematical sort of a pseudo code for

01:14:53.610 --> 01:14:59.100
the the code itself do I need to
actually then release the code right

01:14:59.100 --> 01:15:03.660
like if I can actually lay out the
structure yeah maybe that's enough and I

01:15:03.660 --> 01:15:06.840
think I don't think it's enough well
first of all the code sitting there so

01:15:06.840 --> 01:15:11.310
why wouldn't you release it right the
pseudo code is very helpful however hat

01:15:11.310 --> 01:15:15.060
I think there are crucial details just
even in that implementation like how

01:15:15.060 --> 01:15:18.270
you've actually you know implemented
that on the machine like what have you

01:15:18.270 --> 01:15:22.290
done in terms of digitization and you
know what type of storage of use for the

01:15:22.290 --> 01:15:25.620
numbers is that creating rounding issues
or whatever it is maybe there's just a

01:15:25.620 --> 01:15:30.180
plain old bug in there and so I think
those are both helpful but sometimes I

01:15:30.180 --> 01:15:33.330
do get the question list if I have sort
of well thought through pseudo code

01:15:33.330 --> 01:15:36.900
maybe that's enough but I think that
extra step of putting it on the machine

01:15:36.900 --> 01:15:41.360
is non-trivial and lots of mistakes can
introduce there

01:15:47.540 --> 01:15:50.540
yeah

01:16:02.800 --> 01:16:13.310
yeah yeah that's really interesting so
so again you're you're bringing in these

01:16:13.310 --> 01:16:17.150
sort of existing structures like that we
have to interact with an existing

01:16:17.150 --> 01:16:20.750
journal system that's been there a long
time that it still holds a lot of the

01:16:20.750 --> 01:16:27.320
keys to think like promotion and so I
think I think you're right that there

01:16:27.320 --> 01:16:33.590
that that type of social pressure can
come about I think it's I think it so I

01:16:33.590 --> 01:16:36.950
you know I just don't have easy answers
on you know how to interact with these

01:16:36.950 --> 01:16:42.760
these establish systems but but yeah we
do have to do it

01:17:05.679 --> 01:17:07.739
you