WEBVTT

1
00:00:10.140 --> 00:00:12.750 A:middle L:90%
So thank you all for coming, we very much appreciate it.

2
00:00:13.140 --> 00:00:15.189 A:middle L:90%
I'm Tyler Walters, I'm the dean of your university

3
00:00:15.189 --> 00:00:17.809 A:middle L:90%
libraries, and we're here to kick off Open Access

4
00:00:17.809 --> 00:00:21.289 A:middle L:90%
Week this week in October. I've got several announcements

5
00:00:21.289 --> 00:00:23.390 A:middle L:90%
We want to mention and at the very top of it

6
00:00:23.390 --> 00:00:25.910 A:middle L:90%
we want to recognize our sponsors for this opening keynote

7
00:00:25.920 --> 00:00:29.010 A:middle L:90%
address this evening. And those folks would be Department

8
00:00:29.010 --> 00:00:31.859 A:middle L:90%
of Psychology, College of Science and University Libraries.

9
00:00:31.870 --> 00:00:34.109 A:middle L:90%
There's also a committee that's put together all of

10
00:00:34.109 --> 00:00:35.750 A:middle L:90%
the events on the schedule, that is at the back of

11
00:00:35.750 --> 00:00:37.850 A:middle L:90%
the room, this week for Open Access Week. So

12
00:00:37.850 --> 00:00:40.509 A:middle L:90%
I'd just like to quickly mention those folks who worked on

13
00:00:40.509 --> 00:00:44.280 A:middle L:90%
that. They are Nathan Hall, Gail McMillan, Ginny

14
00:00:44.280 --> 00:00:48.789 A:middle L:90%
Pannabecker, Patrick Tomlin, Anita Walz, Kimberli

15
00:00:48.789 --> 00:00:50.740 A:middle L:90%
Weeks and Philip Young. So maybe you guys could

16
00:00:50.750 --> 00:00:54.060 A:middle L:90%
raise your hands or something. Well, thank you

17
00:00:55.640 --> 00:01:00.109 A:middle L:90%
Okay. So you should know tonight's

18
00:01:00.109 --> 00:01:03.109 A:middle L:90%
keynote is one of eight events that's happening this week

19
00:01:03.109 --> 00:01:04.620 A:middle L:90%
across campus. Again, there's more information on the table

20
00:01:04.620 --> 00:01:07.299 A:middle L:90%
behind you to find these things out. There's two

21
00:01:07.310 --> 00:01:10.510 A:middle L:90%
events I really would like to highlight, in addition

22
00:01:10.510 --> 00:01:14.129 A:middle L:90%
to the keynote address tonight. Tomorrow evening there is

23
00:01:14.129 --> 00:01:15.799 A:middle L:90%
a faculty and graduate student panel that will discuss open

24
00:01:15.799 --> 00:01:19.780 A:middle L:90%
access experiences and issues. That's gonna get started at

25
00:01:19.790 --> 00:01:22.140 A:middle L:90%
5:30 PM in Newman Library in the Multipurpose room. That's

26
00:01:22.140 --> 00:01:26.180 A:middle L:90%
on the first floor behind the cafe. And Thursday at

27
00:01:26.189 --> 00:01:27.829 A:middle L:90%
three o'clock in the afternoon, the Center for Open

28
00:01:27.829 --> 00:01:30.569 A:middle L:90%
Science, which our keynote speaker, Brian Nosek,

29
00:01:30.569 --> 00:01:33.689 A:middle L:90%
co directs, they're going to be leading a

30
00:01:33.689 --> 00:01:36.849 A:middle L:90%
workshop on reproducible research practices, which will also be

31
00:01:36.859 --> 00:01:40.859 A:middle L:90%
in the library's Multipurpose room. On the other side of

32
00:01:40.859 --> 00:01:42.019 A:middle L:90%
the hand out that that you have available today,

33
00:01:42.019 --> 00:01:45.420 A:middle L:90%
there's information about some of the open

34
00:01:45.420 --> 00:01:48.269 A:middle L:90%
related and open supporting services that we have in the library.

35
00:01:48.269 --> 00:01:51.629 A:middle L:90%
We do things like manage and maintain the university's

36
00:01:51.640 --> 00:01:57.920 A:middle L:90%
repository of scholarship, VTechWorks. We also provide an open access

37
00:01:57.920 --> 00:02:00.640 A:middle L:90%
fund to help fund all of you authors who want

38
00:02:00.640 --> 00:02:02.340 A:middle L:90%
to get your works published in an open access journal.

39
00:02:02.670 --> 00:02:05.959 A:middle L:90%
We also offer our own publishing services. There

40
00:02:05.959 --> 00:02:07.789 A:middle L:90%
are several journals that the library and university have supported

41
00:02:07.800 --> 00:02:12.189 A:middle L:90%
for many years. We also have more recently gotten

42
00:02:12.189 --> 00:02:14.930 A:middle L:90%
involved as well, with data consulting, helping researchers

43
00:02:14.930 --> 00:02:16.300 A:middle L:90%
to produce and share their data and make that open

44
00:02:16.300 --> 00:02:17.650 A:middle L:90%
and sharable as well. So we do

45
00:02:17.650 --> 00:02:23.669 A:middle L:90%
an awful lot around this this framework of research

46
00:02:23.669 --> 00:02:25.000 A:middle L:90%
 data management. Also on the back table is a sign

47
00:02:25.000 --> 00:02:27.680 A:middle L:90%
up sheet. We'd like to just know who came.

48
00:02:27.680 --> 00:02:30.479 A:middle L:90%
We really appreciate your attendance. There's also a form

49
00:02:30.479 --> 00:02:31.900 A:middle L:90%
in the back to sign up for an NLI credit,

50
00:02:31.900 --> 00:02:35.139 A:middle L:90%
if you would like to receive that.

51
00:02:35.150 --> 00:02:37.909 A:middle L:90%
There's also sign up list for the Open at VT

52
00:02:37.909 --> 00:02:40.210 A:middle L:90%
listserv if you're interested, and evaluation forms,

53
00:02:40.409 --> 00:02:43.210 A:middle L:90%
of course, we want you to hear your feedback

54
00:02:43.210 --> 00:02:47.629 A:middle L:90%
from today and throughout. Also to inaugurate

55
00:02:47.629 --> 00:02:52.509 A:middle L:90%
the week we have awarded two scholarships during Open

56
00:02:52.520 --> 00:02:54.659 A:middle L:90%
Access Week for the OpenCon 2014 Conference.

57
00:02:55.340 --> 00:02:59.039 A:middle L:90%
This is a sponsorship between University Libraries and the Graduate

58
00:02:59.039 --> 00:03:00.900 A:middle L:90%
School. So OpenCon is an international conference for

59
00:03:00.900 --> 00:03:05.069 A:middle L:90%
students and early career researchers on open access, open

60
00:03:05.069 --> 00:03:07.569 A:middle L:90%
data and open educational resources. It's going to be

61
00:03:07.580 --> 00:03:09.689 A:middle L:90%
held in Washington, D. C. November 15th

62
00:03:09.699 --> 00:03:13.550 A:middle L:90%
to the 17th. We did receive, this the first

63
00:03:13.550 --> 00:03:15.469 A:middle L:90%
time we've done this. We received a very strong

64
00:03:15.469 --> 00:03:17.240 A:middle L:90%
pool of candidates. And the winners, based

65
00:03:17.240 --> 00:03:21.599 A:middle L:90%
on their essays, are Jesse Gunter, masters student in

66
00:03:21.599 --> 00:03:23.509 A:middle L:90%
Public Health, and Mohammed Seyam, a doctoral student

67
00:03:23.520 --> 00:03:27.360 A:middle L:90%
in computer science. Maybe if Jesse and Mohammed are here,

68
00:03:27.370 --> 00:03:28.560 A:middle L:90%
if you could stand for just a minute.

69
00:03:28.569 --> 00:03:32.479 A:middle L:90%
We'll give you a hand, if you're here.

70
00:03:32.620 --> 00:03:38.840 A:middle L:90%
OK, they're late. Or bashful.

71
00:03:38.840 --> 00:03:40.199 A:middle L:90%
With that, we will turn it over soon

72
00:03:40.199 --> 00:03:43.599 A:middle L:90%
to our keynote speaker, who I will introduce now,

73
00:03:43.599 --> 00:03:46.659 A:middle L:90%
is Professor Brian Nosek from the University of Virginia.

74
00:03:46.039 --> 00:03:50.280 A:middle L:90%
Brian received his PhD at Yale University

75
00:03:50.280 --> 00:03:52.639 A:middle L:90%
in 2002. As I said, Professor Nosek

76
00:03:52.639 --> 00:03:53.569 A:middle L:90%
is in the Department of Psychology at the University of

77
00:03:53.569 --> 00:03:57.800 A:middle L:90%
Virginia. He received an Early Career award from the International

78
00:03:57.800 --> 00:04:00.430 A:middle L:90%
Social Cognition Network and the Society for the Psychological Study

79
00:04:00.439 --> 00:04:03.669 A:middle L:90%
of Social Issues. Early on, he co-founded

80
00:04:03.949 --> 00:04:06.569 A:middle L:90%
Project Implicit, which is an internet based, multi

81
00:04:06.569 --> 00:04:11.629 A:middle L:90%
university collaboration of research and education about thoughts and feelings

82
00:04:11.629 --> 00:04:15.219 A:middle L:90%
that exist outside of our awareness or control. Recently

83
00:04:15.219 --> 00:04:17.500 A:middle L:90%
Nosek co-founded and directs The Center for Open

84
00:04:17.500 --> 00:04:21.050 A:middle L:90%
Science, which aims to increase openness, integrity and

85
00:04:21.050 --> 00:04:27.209 A:middle L:90%
reproducibility of scientific research. COS is a nonprofit technology startup

86
00:04:27.220 --> 00:04:30.730 A:middle L:90%
with three primary activities. They are building and maintaining

87
00:04:30.730 --> 00:04:32.839 A:middle L:90%
the open science framework that supports the research workflow and

88
00:04:32.839 --> 00:04:38.040 A:middle L:90%
enables transparency. And the research process includes archiving and

89
00:04:38.040 --> 00:04:42.829 A:middle L:90%
pre registration also community building and shifting incentives such as

90
00:04:42.829 --> 00:04:46.579 A:middle L:90%
badges for articles to acknowledge open practices and conducting meta

91
00:04:46.579 --> 00:04:48.860 A:middle L:90%
science, such as estimating the reproducibility of scientific research

92
00:04:48.860 --> 00:04:54.180 A:middle L:90%
by conducting large scale crowdsourced replication.

93
00:04:54.189 --> 00:04:57.529 A:middle L:90%
Now we will turn it over to Professor Nosek whose topic

94
00:04:57.540 --> 00:05:00.000 A:middle L:90%
is improving the openness and reproducibility of research. So

95
00:05:00.009 --> 00:05:09.839 A:middle L:90%
please welcome Brian Nosek. 

96
00:05:09.839 --> 00:05:12.310 A:middle L:90%
Thanks very much for coming and for having me here.

97
00:05:12.310 --> 00:05:14.149 A:middle L:90%
I'm very glad to be able to speak here

98
00:05:14.149 --> 00:05:17.850 A:middle L:90%
today. My area of research expertise in

99
00:05:17.850 --> 00:05:21.569 A:middle L:90%
the psychology department is the gap between values and practices,

100
00:05:21.579 --> 00:05:24.850 A:middle L:90%
what we think we should do, what we're trying

101
00:05:24.850 --> 00:05:26.629 A:middle L:90%
to do, what we want to do, versus what

102
00:05:26.629 --> 00:05:30.129 A:middle L:90%
we actually do. And most of that work is

103
00:05:30.129 --> 00:05:32.079 A:middle L:90%
on implicit biases and how we're influenced in our social

104
00:05:32.079 --> 00:05:35.160 A:middle L:90%
judgment and everyday behavior. But the work that I

105
00:05:35.160 --> 00:05:38.519 A:middle L:90%
want to talk about today is an application of that

106
00:05:38.529 --> 00:05:42.360 A:middle L:90%
general interest on the gap between scientific values and scientific

107
00:05:42.360 --> 00:05:45.699 A:middle L:90%
practices. What science aims to be as a discipline

108
00:05:45.699 --> 00:05:48.180 A:middle L:90%
of trying to accumulate knowledge and then how it actually

109
00:05:48.180 --> 00:05:51.920 A:middle L:90%
works in the everyday life of a researcher and how

110
00:05:51.920 --> 00:05:55.970 A:middle L:90%
we might try to improve the alignment between the goals,

111
00:05:55.970 --> 00:05:58.029 A:middle L:90%
the ideals of what science is supposed to be,

112
00:05:58.029 --> 00:06:00.019 A:middle L:90%
and then what the reality is on the ground

113
00:06:00.139 --> 00:06:02.920 A:middle L:90%
and what scientists are rewarded for in doing their research

114
00:06:02.930 --> 00:06:06.939 A:middle L:90%
today. So the theme, given that it's open

115
00:06:06.939 --> 00:06:11.730 A:middle L:90%
access week, is going to be suggesting

116
00:06:11.730 --> 00:06:15.350 A:middle L:90%
that open is more than just access. We tend

117
00:06:15.350 --> 00:06:17.689 A:middle L:90%
to think of open access in terms of the final

118
00:06:17.689 --> 00:06:23.060 A:middle L:90%
publication of the workflow that goes into producing a body

119
00:06:23.060 --> 00:06:25.430 A:middle L:90%
of knowledge, a piece of claim, a piece

120
00:06:25.430 --> 00:06:29.180 A:middle L:90%
of evidence. But really, the relevance of openness

121
00:06:29.180 --> 00:06:31.779 A:middle L:90%
for scientific research is across the entire research lifecycle,

122
00:06:31.939 --> 00:06:35.290 A:middle L:90%
not just in who can get access to the report

123
00:06:35.300 --> 00:06:39.529 A:middle L:90%
itself. That is the outcome. So what I

124
00:06:39.529 --> 00:06:42.670 A:middle L:90%
will do is sort of give a general

125
00:06:42.680 --> 00:06:45.540 A:middle L:90%
introduction to some of the activities that we're pursuing

126
00:06:45.550 --> 00:06:47.399 A:middle L:90%
at the Center for Open Science and then hope that

127
00:06:47.410 --> 00:06:50.610 A:middle L:90%
you will have some questions, critiques,

128
00:06:50.610 --> 00:06:53.600 A:middle L:90%
ideas or ways in which the things that we

129
00:06:53.600 --> 00:06:56.699 A:middle L:90%
are doing can be more relevant to you

130
00:06:56.699 --> 00:06:59.240 A:middle L:90%
and the work that you do. We are

131
00:06:59.240 --> 00:07:01.959 A:middle L:90%
a service, a non profit organization. So we are

132
00:07:01.959 --> 00:07:05.990 A:middle L:90%
here to serve the good of research getting done better.

133
00:07:06.000 --> 00:07:09.480 A:middle L:90%
So we hope that we can find ways to

134
00:07:09.490 --> 00:07:12.899 A:middle L:90%
find something of interest to you.

135
00:07:12.910 --> 00:07:15.689 A:middle L:90%
But let me begin by articulating something about the norms

136
00:07:15.689 --> 00:07:18.449 A:middle L:90%
of science. Robert Merton in the 1950s,

137
00:07:18.449 --> 00:07:23.240 A:middle L:90%
the sociologist of science, identified what he perceived as the

138
00:07:23.240 --> 00:07:25.089 A:middle L:90%
norms of science, and there hasn't been a lot

139
00:07:25.089 --> 00:07:28.860 A:middle L:90%
of challenge to them, mostly elaboration of those norms

140
00:07:29.040 --> 00:07:30.129 A:middle L:90%
of what it is that scientists do, how it

141
00:07:30.129 --> 00:07:32.930 A:middle L:90%
is. They operate in trying to build a body

142
00:07:32.930 --> 00:07:38.040 A:middle L:90%
of knowledge. And so one of those is commonality,

143
00:07:38.040 --> 00:07:41.660 A:middle L:90%
the open sharing of information. Scientists make claims,

144
00:07:42.040 --> 00:07:44.370 A:middle L:90%
and then they provide a basis. What is the

145
00:07:44.370 --> 00:07:46.509 A:middle L:90%
evidence for making that claim? What are the methodologies

146
00:07:46.509 --> 00:07:49.100 A:middle L:90%
that were done? Here's the methods. What data

147
00:07:49.100 --> 00:07:51.029 A:middle L:90%
were used or accumulated for that? Here are the

148
00:07:51.029 --> 00:07:54.629 A:middle L:90%
data. What analysis was performed? How was that

149
00:07:54.629 --> 00:07:57.519 A:middle L:90%
interpreted? All of that is laid bare so that

150
00:07:57.660 --> 00:08:01.680 A:middle L:90%
others can evaluate the basis of the claim on the

151
00:08:01.680 --> 00:08:05.029 A:middle L:90%
evidence rather than on the authority of the individual making

152
00:08:05.029 --> 00:08:07.350 A:middle L:90%
the claim. Whereas the counter norm is secrecy,

153
00:08:07.350 --> 00:08:07.750 A:middle L:90%
you don't get to see how it is. I

154
00:08:07.750 --> 00:08:09.589 A:middle L:90%
made that claim. It's just, here is the claim,

155
00:08:09.879 --> 00:08:13.740 A:middle L:90%
and believe it or don't. Another norm that he identified

156
00:08:13.740 --> 00:08:16.560 A:middle L:90%
is universalism, that you evaluate the research based on its

157
00:08:16.560 --> 00:08:18.699 A:middle L:90%
own merit. Uh, as I said rather than

158
00:08:18.699 --> 00:08:22.879 A:middle L:90%
particular is, um, evaluating by the reputation of

159
00:08:22.879 --> 00:08:26.889 A:middle L:90%
the individual claiming it is the counter-norm. A third that

160
00:08:26.889 --> 00:08:30.079 A:middle L:90%
he identified is disinterestedness. Scientists are motivated by

161
00:08:30.079 --> 00:08:31.850 A:middle L:90%
knowledge and discovery trying to figure things out as they

162
00:08:31.850 --> 00:08:35.399 A:middle L:90%
are, whereas the counter norm being self

163
00:08:35.399 --> 00:08:37.639 A:middle L:90%
interestedness. I'm just trying to get ahead of

164
00:08:37.639 --> 00:08:39.850 A:middle L:90%
the person down the hall and try to get the

165
00:08:39.850 --> 00:08:46.700 A:middle L:90%
most awards and rewards in scientific process. Fourth is

166
00:08:46.700 --> 00:08:50.250 A:middle L:90%
organized skepticism. A scientist considers all the evidence,

167
00:08:50.250 --> 00:08:52.159 A:middle L:90%
including that that is, against one's prior work in

168
00:08:52.159 --> 00:08:56.259 A:middle L:90%
order to try to reveal the truth about nature,

169
00:08:56.639 --> 00:09:00.419 A:middle L:90%
whereas the counter norm is organized dogmatism, investing in one's

170
00:09:00.419 --> 00:09:03.230 A:middle L:90%
own career, defend one's findings against attack from others

171
00:09:03.240 --> 00:09:07.220 A:middle L:90%
and try to promote one's own view of the world,

172
00:09:07.230 --> 00:09:09.500 A:middle L:90%
whatever the evidence.

173
00:09:09.500 --> 00:09:13.399 A:middle L:90%
And then the last one that wasn't identified by Merton per se,

174
00:09:13.399 --> 00:09:15.120 A:middle L:90%
but it's a very popular one in the

175
00:09:15.120 --> 00:09:18.149 A:middle L:90%
field, is an emphasis on quality of work rather than

176
00:09:18.149 --> 00:09:20.100 A:middle L:90%
on the quantity of producing work. So these,

177
00:09:20.100 --> 00:09:22.870 A:middle L:90%
and then there's other norms have been suggested. But

178
00:09:22.870 --> 00:09:26.860 A:middle L:90%
these are five of the very popular norms of what

179
00:09:26.870 --> 00:09:31.090 A:middle L:90%
makes, uh, scientific practice unique as a body

180
00:09:31.090 --> 00:09:33.059 A:middle L:90%
of knowledge. Many of these apply to scholarship more

181
00:09:33.059 --> 00:09:35.720 A:middle L:90%
generally, and so they may apply more broadly than

182
00:09:35.720 --> 00:09:41.519 A:middle L:90%
just science versus the counter norms. So these may

183
00:09:41.519 --> 00:09:43.950 A:middle L:90%
be easy to identify, but one open question is

184
00:09:43.950 --> 00:09:46.179 A:middle L:90%
to what extent do researchers believe in and try to

185
00:09:46.179 --> 00:09:50.659 A:middle L:90%
live by these, uh and so a team led

186
00:09:50.659 --> 00:09:52.960 A:middle L:90%
by, uh, Anderson, Anderson, Martin and

187
00:09:52.960 --> 00:09:56.350 A:middle L:90%
degree, uh, did a survey of 3500 or

188
00:09:56.350 --> 00:10:00.799 A:middle L:90%
so early and mid career researchers, Uh, and

189
00:10:00.799 --> 00:10:03.720 A:middle L:90%
they identified early in mid career as early career being

190
00:10:03.730 --> 00:10:07.690 A:middle L:90%
, uh, people on postdocs. Uh, all

191
00:10:07.690 --> 00:10:09.129 A:middle L:90%
of these were NIH funded, So nrs a equivalent

192
00:10:09.129 --> 00:10:13.840 A:middle L:90%
types postdocs, uh, in mid career were faculty

193
00:10:13.840 --> 00:10:16.950 A:middle L:90%
had just received their first R 01 So average age

194
00:10:16.950 --> 00:10:20.159 A:middle L:90%
about 40 or so. So there's about 1500 or

195
00:10:20.159 --> 00:10:22.159 A:middle L:90%
a few more in each of those categories. And

196
00:10:22.159 --> 00:10:24.740 A:middle L:90%
what they asked them was, Which of these do

197
00:10:24.740 --> 00:10:28.259 A:middle L:90%
you subscribe to more? The norms that appeared

198
00:10:28.259 --> 00:10:28.960 A:middle L:90%
on the left or the counter norms appeared on the

199
00:10:28.960 --> 00:10:31.389 A:middle L:90%
right and what I'm showing you here is a cumulative

200
00:10:31.389 --> 00:10:35.179 A:middle L:90%
plot. 100% of the respondents where the gray bar

201
00:10:35.190 --> 00:10:39.070 A:middle L:90%
indicates the number of people who valued the norms over

202
00:10:39.070 --> 00:10:41.980 A:middle L:90%
the counter norms. Whereas the black bar were people

203
00:10:41.980 --> 00:10:45.480 A:middle L:90%
who said I value the counter norms over the norms

204
00:10:45.490 --> 00:10:46.799 A:middle L:90%
and how I do science and the gray hatches is

205
00:10:46.809 --> 00:10:50.019 A:middle L:90%
equivalent. So you see both men and early career

206
00:10:50.029 --> 00:10:52.250 A:middle L:90%
about 90% or so. If people are saying I

207
00:10:52.250 --> 00:10:56.480 A:middle L:90%
value those norms of science more than the counter norms

208
00:10:56.490 --> 00:10:58.690 A:middle L:90%
and what I think science should be, how I

209
00:10:58.700 --> 00:11:01.730 A:middle L:90%
how I think it ought to be. So I

210
00:11:01.730 --> 00:11:03.600 A:middle L:90%
said, Okay, that's fine. Don't tell me

211
00:11:03.600 --> 00:11:03.860 A:middle L:90%
what you think it should be. Tell me what

212
00:11:03.860 --> 00:11:07.559 A:middle L:90%
you do. Uh, and it moves this way

213
00:11:07.039 --> 00:11:09.870 A:middle L:90%
. So still, you see, people are by

214
00:11:09.870 --> 00:11:13.919 A:middle L:90%
and large more than 60% in both categories are saying

215
00:11:13.919 --> 00:11:16.480 A:middle L:90%
that they practice the norms of science over the counter

216
00:11:16.480 --> 00:11:20.159 A:middle L:90%
norms. But more people are acknowledging that those counter

217
00:11:20.159 --> 00:11:22.559 A:middle L:90%
norms also have weight in their everyday practice that they

218
00:11:22.559 --> 00:11:26.750 A:middle L:90%
do feel competitive, that they do sometimes hold back

219
00:11:26.750 --> 00:11:30.570 A:middle L:90%
information that they're not always acting in a disinterested fashion

220
00:11:30.570 --> 00:11:33.210 A:middle L:90%
, but sometimes feel relatively dogmatic or defensive, at

221
00:11:33.210 --> 00:11:37.659 A:middle L:90%
least about the findings that they have. And they

222
00:11:37.659 --> 00:11:39.080 A:middle L:90%
said, Okay, that's fine. Don't tell me

223
00:11:39.080 --> 00:11:39.899 A:middle L:90%
what you do tell me what other people in your

224
00:11:39.899 --> 00:11:46.350 A:middle L:90%
field do, and that's what occurred. So there

225
00:11:46.350 --> 00:11:50.720 A:middle L:90%
is enormous cynicism about everybody else, not practicing those

226
00:11:50.720 --> 00:11:54.460 A:middle L:90%
norms that I hold valuable and mostly try to live

227
00:11:54.460 --> 00:11:56.320 A:middle L:90%
up to, even though I might not be able

228
00:11:56.320 --> 00:11:58.850 A:middle L:90%
to live up to them as much as I idealize.

229
00:12:00.440 --> 00:12:01.480 A:middle L:90%
And if we're thinking about this in terms of

230
00:12:01.480 --> 00:12:05.210 A:middle L:90%
a behavioral issue, this is a big challenge,

231
00:12:05.220 --> 00:12:07.759 A:middle L:90%
right? If people perceive the norms of science to

232
00:12:07.759 --> 00:12:11.320 A:middle L:90%
be endorsed by oneself, but not at all practiced

233
00:12:11.330 --> 00:12:15.370 A:middle L:90%
by others, and it's a very difficult situation for

234
00:12:15.370 --> 00:12:18.889 A:middle L:90%
behavior change. People see it as a competitive landscape

235
00:12:18.889 --> 00:12:20.870 A:middle L:90%
where people are just holding information for themselves, doing

236
00:12:20.870 --> 00:12:24.500 A:middle L:90%
whatever it takes to get ahead. While I'm sitting

237
00:12:24.500 --> 00:12:26.750 A:middle L:90%
here just trying to figure stuff out, I'm motivated

238
00:12:26.750 --> 00:12:30.620 A:middle L:90%
by the curiosity of trying to understand things, to

239
00:12:30.620 --> 00:12:33.129 A:middle L:90%
discover stuff and just trying to survive in this world

240
00:12:33.129 --> 00:12:35.830 A:middle L:90%
around me, where everyone else is in it for

241
00:12:35.830 --> 00:12:37.850 A:middle L:90%
themselves, and most people are in that situation.

242
00:12:37.850 --> 00:12:41.539 A:middle L:90%
According to these numbers, most people perceive that they

243
00:12:41.539 --> 00:12:43.659 A:middle L:90%
are motivated by the ideals of science, but it's

244
00:12:43.659 --> 00:12:46.080 A:middle L:90%
not practiced out there, and as a consequence they

245
00:12:46.080 --> 00:12:48.490 A:middle L:90%
may lose out. They may not be able to

246
00:12:48.490 --> 00:12:54.009 A:middle L:90%
succeed in maintaining a career in science because maintaining a

247
00:12:54.009 --> 00:12:58.110 A:middle L:90%
career in science besides just pursuing knowledge requires doing things

248
00:12:58.110 --> 00:13:00.750 A:middle L:90%
that can get you a job, can help you keep

249
00:13:00.750 --> 00:13:03.870 A:middle L:90%
that job, and can help you advance in that career.

250
00:13:03.879 --> 00:13:05.960 A:middle L:90%
So there are pragmatic realities that every researcher has

251
00:13:05.960 --> 00:13:09.190 A:middle L:90%
to confront with. What do I need to do

252
00:13:09.190 --> 00:13:11.389 A:middle L:90%
in order to retain and be able to do this

253
00:13:11.389 --> 00:13:16.379 A:middle L:90%
work while simultaneously having ideals and values of how I

254
00:13:16.379 --> 00:13:18.389 A:middle L:90%
want to do the work? And if those are

255
00:13:18.389 --> 00:13:22.120 A:middle L:90%
not aligned, then it creates difficulties in making individual

256
00:13:22.120 --> 00:13:22.980 A:middle L:90%
decisions. What is it that I'm going to do

257
00:13:22.980 --> 00:13:26.659 A:middle L:90%
today in order to practice my science while trying to

258
00:13:26.659 --> 00:13:30.700 A:middle L:90%
still make a contribution? So there are many different

259
00:13:30.700 --> 00:13:33.470 A:middle L:90%
types of things that we can unpack about what those

260
00:13:33.470 --> 00:13:35.769 A:middle L:90%
decision points are. What I want to start is

261
00:13:35.769 --> 00:13:39.909 A:middle L:90%
just on a general introduction to the center and what

262
00:13:39.909 --> 00:13:43.419 A:middle L:90%
our overall goals are, and then take you through

263
00:13:43.419 --> 00:13:46.759 A:middle L:90%
a few different ways that we are trying to approach

264
00:13:46.769 --> 00:13:50.950 A:middle L:90%
this gap that people confront in what is what they'd

265
00:13:50.950 --> 00:13:52.340 A:middle L:90%
like to do and then what they are incentivized to

266
00:13:52.340 --> 00:13:56.899 A:middle L:90%
do. So the center just opened in the beginning

267
00:13:56.899 --> 00:14:00.149 A:middle L:90%
of last year. Uh, and it is,

268
00:14:00.159 --> 00:14:01.669 A:middle L:90%
uh, independent non profit. It's not part of

269
00:14:01.669 --> 00:14:05.590 A:middle L:90%
the University of Virginia, I'm the main link between it

270
00:14:05.600 --> 00:14:07.330 A:middle L:90%
and the university. 

271
00:14:07.340 --> 00:14:11.340 A:middle L:90%
It is tasked, as with mission, to improve

272
00:14:11.340 --> 00:14:13.750 A:middle L:90%
openness, integrity, reproducibility of scientific research. And

273
00:14:13.750 --> 00:14:16.059 A:middle L:90%
we have a staff of about 40 now. We

274
00:14:16.059 --> 00:14:18.590 A:middle L:90%
got a lot of core funding initially, so it's

275
00:14:18.590 --> 00:14:22.879 A:middle L:90%
about $12 million of base funding. But it's by

276
00:14:22.879 --> 00:14:26.720 A:middle L:90%
and large a technology company where most of our activities

277
00:14:26.720 --> 00:14:31.000 A:middle L:90%
are building infrastructure tools to help researchers do their research

278
00:14:31.009 --> 00:14:33.179 A:middle L:90%
in the way that they do it now, but make

279
00:14:33.179 --> 00:14:35.809 A:middle L:90%
it easy to move to things that are more aligned

280
00:14:35.820 --> 00:14:39.909 A:middle L:90%
with individual values. And then we also have efforts

281
00:14:39.909 --> 00:14:41.769 A:middle L:90%
in community and trying to look at the various parts

282
00:14:41.769 --> 00:14:46.309 A:middle L:90%
of the scientific ecosystem for how they incentivize behaviors that

283
00:14:46.309 --> 00:14:48.669 A:middle L:90%
might not be aligned with how people want to try

284
00:14:48.669 --> 00:14:52.700 A:middle L:90%
to live as a scientist. And then we do

285
00:14:52.700 --> 00:14:56.529 A:middle L:90%
meta science research. So actually looking at the scientific

286
00:14:56.529 --> 00:15:00.019 A:middle L:90%
process by, for example, doing large scale replications

287
00:15:00.019 --> 00:15:03.379 A:middle L:90%
of a variety of projects to estimate reproducibility and then

288
00:15:03.379 --> 00:15:05.289 A:middle L:90%
look for why is it that we are able to

289
00:15:05.289 --> 00:15:07.899 A:middle L:90%
reproduce some findings and not others. I won't get

290
00:15:07.899 --> 00:15:11.340 A:middle L:90%
into elaborate for today. But I'm happy

291
00:15:11.340 --> 00:15:15.840 A:middle L:90%
to talk about those afterwards if you like. But

292
00:15:15.840 --> 00:15:18.759 A:middle L:90%
let me give you a very general perspective on the

293
00:15:18.759 --> 00:15:22.259 A:middle L:90%
overall strategy for this gap between values and practices.

294
00:15:22.269 --> 00:15:26.299 A:middle L:90%
First thing that we aim to do is develop technologies

295
00:15:26.299 --> 00:15:30.070 A:middle L:90%
to enable people to change lot of the barriers that

296
00:15:30.070 --> 00:15:33.009 A:middle L:90%
we confront our practical barriers. I might like to

297
00:15:33.009 --> 00:15:35.789 A:middle L:90%
be more open about my research process, but what

298
00:15:35.789 --> 00:15:37.360 A:middle L:90%
it requires me to do is a lot more work,

299
00:15:37.620 --> 00:15:39.269 A:middle L:90%
and I don't have time for more work.

300
00:15:39.389 --> 00:15:41.830 A:middle L:90%
I'm not incentivized to do more work, and I'm

301
00:15:41.830 --> 00:15:46.429 A:middle L:90%
really busy as it is. So how is it

302
00:15:46.429 --> 00:15:48.690 A:middle L:90%
that I'm going to add on extra work to make

303
00:15:48.690 --> 00:15:50.250 A:middle L:90%
my data more available to others to make my materials

304
00:15:50.250 --> 00:15:52.679 A:middle L:90%
more available to others? So we're trying to build

305
00:15:52.679 --> 00:15:54.950 A:middle L:90%
tools that make it a lot easier for people to

306
00:15:54.950 --> 00:15:58.220 A:middle L:90%
do that so that it's just a matter of wanting

307
00:15:58.220 --> 00:16:00.690 A:middle L:90%
to, rather than having the technical or practical feasibility

308
00:16:00.700 --> 00:16:04.980 A:middle L:90%
to do it. Second is, we have efforts

309
00:16:04.980 --> 00:16:08.230 A:middle L:90%
to develop training on how to enact those changes,

310
00:16:08.240 --> 00:16:11.929 A:middle L:90%
a lot of these practical matters of making a research

311
00:16:11.929 --> 00:16:15.049 A:middle L:90%
process reproducible are wanting. How is it that we

312
00:16:15.049 --> 00:16:18.629 A:middle L:90%
can actually get that done? And so part of

313
00:16:18.629 --> 00:16:22.039 A:middle L:90%
the team is tasked with providing training on the tools

314
00:16:22.039 --> 00:16:25.950 A:middle L:90%
that can be used to make reproducible practices easier.

315
00:16:26.539 --> 00:16:29.330 A:middle L:90%
Uh, and then finally, the key part is

316
00:16:29.340 --> 00:16:33.750 A:middle L:90%
to address the incentives to embrace those changes. I

317
00:16:33.750 --> 00:16:36.090 A:middle L:90%
can have all of the desire in the world to

318
00:16:36.100 --> 00:16:38.940 A:middle L:90%
behave toward the ideals. But if I'm not reinforced

319
00:16:38.960 --> 00:16:41.000 A:middle L:90%
in my everyday practice for doing so, it's going

320
00:16:41.000 --> 00:16:45.759 A:middle L:90%
to be very difficult to actually change my behavior because

321
00:16:45.759 --> 00:16:48.139 A:middle L:90%
I have to overcome what our actual barriers, especially

322
00:16:48.340 --> 00:16:52.690 A:middle L:90%
the graduate student collaborators with me. They are in

323
00:16:52.690 --> 00:16:56.960 A:middle L:90%
a very difficult situation of needing a job and not

324
00:16:56.960 --> 00:16:59.820 A:middle L:90%
having one that is secure. Mine is relatively secure,

325
00:17:00.370 --> 00:17:03.019 A:middle L:90%
and so they have to pay very close attention

326
00:17:03.029 --> 00:17:04.720 A:middle L:90%
to the various incentives and how they use their time

327
00:17:04.880 --> 00:17:07.640 A:middle L:90%
in a very competitive marketplace. And so if the

328
00:17:07.640 --> 00:17:11.970 A:middle L:90%
incentives don't align with their own interests in order to

329
00:17:11.970 --> 00:17:15.180 A:middle L:90%
gain and survive in the career, then they can

330
00:17:15.180 --> 00:17:18.630 A:middle L:90%
practice the ideals and then find another job after graduate

331
00:17:18.630 --> 00:17:21.160 A:middle L:90%
school rather than meeting a career that they would like

332
00:17:21.160 --> 00:17:25.890 A:middle L:90%
to have. Okay, so the basics of the

333
00:17:25.900 --> 00:17:27.220 A:middle L:90%
technology approach and I'll start with giving a sort of

334
00:17:27.220 --> 00:17:30.000 A:middle L:90%
a general view of what we're trying to address

335
00:17:30.009 --> 00:17:33.339 A:middle L:90%
with the technology and then come back to some of

336
00:17:33.339 --> 00:17:37.509 A:middle L:90%
the issues and incentive challenges and why those exist toward

337
00:17:37.509 --> 00:17:41.289 A:middle L:90%
the end. But the basic goal of the open

338
00:17:41.289 --> 00:17:45.190 A:middle L:90%
science framework work, which is our core infrastructure,

339
00:17:45.200 --> 00:17:48.799 A:middle L:90%
is to help support the entire research lifecycle, 

340
00:17:48.809 --> 00:17:51.450 A:middle L:90%
from the conception of ideas to how we discover

341
00:17:51.450 --> 00:17:55.289 A:middle L:90%
what's known about them through designing studies, collecting data,

342
00:17:55.299 --> 00:17:59.430 A:middle L:90%
this cycle of scholarship, which is the regular

343
00:17:59.430 --> 00:18:00.819 A:middle L:90%
part of what everybody does in order to try to

344
00:18:00.819 --> 00:18:04.910 A:middle L:90%
figure stuff out. Many of the practices and efforts

345
00:18:04.920 --> 00:18:07.740 A:middle L:90%
in trying to open the workflow address just a slice

346
00:18:07.750 --> 00:18:11.710 A:middle L:90%
of that workflow, and as a consequence,

347
00:18:11.720 --> 00:18:12.549 A:middle L:90%
there are a lot of separate slices that are

348
00:18:12.549 --> 00:18:15.829 A:middle L:90%
very hard to work with. And as you know,

349
00:18:15.829 --> 00:18:18.130 A:middle L:90%
picking up new technologies and moving from one to

350
00:18:18.130 --> 00:18:19.720 A:middle L:90%
another can have a lot of transaction costs. And

351
00:18:19.720 --> 00:18:22.549 A:middle L:90%
so we try to minimize those costs by putting the

352
00:18:22.549 --> 00:18:26.299 A:middle L:90%
various services that people use together, and I'll say

353
00:18:26.299 --> 00:18:29.609 A:middle L:90%
more of that coming up. That's abstract. So

354
00:18:29.609 --> 00:18:33.400 A:middle L:90%
the tool that is the sort of the core infrastructure

355
00:18:33.400 --> 00:18:36.859 A:middle L:90%
for the, uh, for the technology option is

356
00:18:36.859 --> 00:18:38.019 A:middle L:90%
the open science framework. It's a Web application.

357
00:18:38.019 --> 00:18:40.940 A:middle L:90%
You can go right now and sign up. If

358
00:18:40.940 --> 00:18:41.809 A:middle L:90%
you have Internet access, you can go right now

359
00:18:41.819 --> 00:18:45.779 A:middle L:90%
and sign up. But you can use it,

360
00:18:45.789 --> 00:18:48.390 A:middle L:90%
uh or at least try it out and see if

361
00:18:48.390 --> 00:18:51.599 A:middle L:90%
it provides any value for the research that you do

362
00:18:51.609 --> 00:18:52.950 A:middle L:90%
. And so what I want to do is give

363
00:18:52.950 --> 00:18:56.779 A:middle L:90%
you sort of a basic overview without doing demos.

364
00:18:56.779 --> 00:18:59.519 A:middle L:90%
We don't have time for that of what its services

365
00:18:59.519 --> 00:19:00.769 A:middle L:90%
are and the problems that it aims to solve.

366
00:19:02.839 --> 00:19:04.609 A:middle L:90%
Uh, and it started with a problem that we

367
00:19:04.609 --> 00:19:07.799 A:middle L:90%
confront in our lab all of the time. And

368
00:19:07.799 --> 00:19:10.579 A:middle L:90%
I know is confronted in many other labs all of

369
00:19:10.579 --> 00:19:12.500 A:middle L:90%
the time to, uh and that I can sort

370
00:19:12.500 --> 00:19:15.980 A:middle L:90%
of summarize some sort of an amalgam sketch of an

371
00:19:15.980 --> 00:19:18.029 A:middle L:90%
experience that happens regularly in my lab. So a

372
00:19:18.029 --> 00:19:22.210 A:middle L:90%
loop is a third year grad student in my lab

373
00:19:22.210 --> 00:19:22.940 A:middle L:90%
, and we have weekly meetings with all the grad

374
00:19:22.940 --> 00:19:25.390 A:middle L:90%
students, and so he comes in. We have

375
00:19:25.390 --> 00:19:29.009 A:middle L:90%
a weekly discussion about the research that he's doing and

376
00:19:29.009 --> 00:19:30.670 A:middle L:90%
we get to some new idea. We're sort of

377
00:19:30.670 --> 00:19:32.829 A:middle L:90%
talking about it and getting excited. And that sort

378
00:19:32.829 --> 00:19:34.019 A:middle L:90%
of realized Oh my gosh, this is very similar

379
00:19:34.019 --> 00:19:37.910 A:middle L:90%
to this study that Nicole and I did a couple

380
00:19:37.910 --> 00:19:41.599 A:middle L:90%
of years ago that got dropped because she graduated and

381
00:19:41.599 --> 00:19:44.190 A:middle L:90%
went on and did other things. So maybe we

382
00:19:44.190 --> 00:19:45.289 A:middle L:90%
should go back to that study. Start with that

383
00:19:45.289 --> 00:19:48.319 A:middle L:90%
. As a pilot, he can use the methods

384
00:19:48.319 --> 00:19:51.359 A:middle L:90%
. He's got some datas initial project and then make

385
00:19:51.359 --> 00:19:55.420 A:middle L:90%
very rapid progress on the problem. And then so

386
00:19:55.420 --> 00:19:56.029 A:middle L:90%
he says, Oh, great, this solves some

387
00:19:56.029 --> 00:19:57.490 A:middle L:90%
problems for me and I say, OK, so

388
00:19:57.490 --> 00:20:00.359 A:middle L:90%
let me go get that study So I go to

389
00:20:00.369 --> 00:20:03.019 A:middle L:90%
turn to my computer and I can't find it.

390
00:20:03.210 --> 00:20:03.339 A:middle L:90%
I say, Well, okay, well, I

391
00:20:03.339 --> 00:20:06.210 A:middle L:90%
didn't do the project. Nicole did the project.

392
00:20:06.210 --> 00:20:08.009 A:middle L:90%
So let me email Nicole and she's in industry.

393
00:20:08.009 --> 00:20:10.589 A:middle L:90%
So I email Nicole, but she's an industry.

394
00:20:10.589 --> 00:20:11.650 A:middle L:90%
So two weeks later she writes back and she says

395
00:20:11.660 --> 00:20:15.170 A:middle L:90%
, What project are you talking about? And so

396
00:20:15.170 --> 00:20:17.130 A:middle L:90%
we have some back and forth trying to clarify the

397
00:20:17.130 --> 00:20:18.309 A:middle L:90%
project because there were lots of different ones and Finally

398
00:20:18.309 --> 00:20:19.750 A:middle L:90%
, she says, Oh, yeah, I remember

399
00:20:19.750 --> 00:20:22.029 A:middle L:90%
that one. Well, I don't have it on

400
00:20:22.029 --> 00:20:23.940 A:middle L:90%
the machine that I use at work, So I'm

401
00:20:23.940 --> 00:20:26.779 A:middle L:90%
going to have to go find the machine, the

402
00:20:26.779 --> 00:20:29.170 A:middle L:90%
disks where that stuff might have been. So it'll

403
00:20:29.170 --> 00:20:30.690 A:middle L:90%
be a while. So a couple of weeks go

404
00:20:30.690 --> 00:20:33.039 A:middle L:90%
by and she says, Oh, well, I

405
00:20:33.049 --> 00:20:33.849 A:middle L:90%
found it. Well, I sort of found it

406
00:20:33.859 --> 00:20:37.640 A:middle L:90%
. I found four different versions that each remember how

407
00:20:37.640 --> 00:20:38.630 A:middle L:90%
we're changing the materials. I don't remember which one

408
00:20:38.630 --> 00:20:41.750 A:middle L:90%
we did of the methods, and then I have

409
00:20:41.750 --> 00:20:42.309 A:middle L:90%
some data here. I'm not sure if it's all

410
00:20:42.309 --> 00:20:45.240 A:middle L:90%
the data, but here it is and then sends

411
00:20:45.240 --> 00:20:47.920 A:middle L:90%
it all to me. And then I pass it

412
00:20:47.920 --> 00:20:48.559 A:middle L:90%
off a nuke and he looks at it. He

413
00:20:48.559 --> 00:20:49.690 A:middle L:90%
says. I think I'm going to work on a

414
00:20:49.690 --> 00:20:56.609 A:middle L:90%
different problem. Uh, so the problem, obviously

415
00:20:56.619 --> 00:20:59.369 A:middle L:90%
, is that we're not able to keep track of

416
00:20:59.369 --> 00:21:02.490 A:middle L:90%
our own stuff for our own use. It's not

417
00:21:02.490 --> 00:21:03.470 A:middle L:90%
a matter of sharing with other people. It's a

418
00:21:03.470 --> 00:21:07.460 A:middle L:90%
matter of using it for ourselves. Of the projects

419
00:21:07.460 --> 00:21:10.579 A:middle L:90%
that we have done get lost and we're pretty organized

420
00:21:10.579 --> 00:21:15.140 A:middle L:90%
lab. But nonetheless, material gets vanishes because it's

421
00:21:15.140 --> 00:21:17.630 A:middle L:90%
also a distributed lab. And this is the true

422
00:21:17.630 --> 00:21:19.450 A:middle L:90%
for many labs is that there are many different actors

423
00:21:19.650 --> 00:21:22.630 A:middle L:90%
there in that group and out of that group at

424
00:21:22.630 --> 00:21:25.509 A:middle L:90%
different points in time, their collaborators at multiple sites

425
00:21:26.089 --> 00:21:27.930 A:middle L:90%
and the materials, the content, the data of

426
00:21:27.940 --> 00:21:32.730 A:middle L:90%
each project is distributed. Everybody has their own ad

427
00:21:32.730 --> 00:21:36.269 A:middle L:90%
hoc organization system. Uh, and if one machine

428
00:21:36.269 --> 00:21:37.579 A:middle L:90%
goes, many things go with it. Unless that

429
00:21:37.579 --> 00:21:41.309 A:middle L:90%
person happened to have it on Dropbox or some other

430
00:21:41.309 --> 00:21:42.960 A:middle L:90%
thing to back it up, Uh, and then

431
00:21:42.960 --> 00:21:45.690 A:middle L:90%
a lot of times, that's all I meant to

432
00:21:45.700 --> 00:21:47.869 A:middle L:90%
get Dropbox set up for that or whatever it was

433
00:21:47.869 --> 00:21:49.519 A:middle L:90%
. And it's gone. So these are problems that

434
00:21:49.519 --> 00:21:52.720 A:middle L:90%
shouldn't happen in modern labs, but they do all

435
00:21:52.720 --> 00:21:56.240 A:middle L:90%
of the time. They shouldn't happen because there's plenty

436
00:21:56.240 --> 00:22:00.180 A:middle L:90%
of technology available to avoid this kind of loss and

437
00:22:00.190 --> 00:22:03.319 A:middle L:90%
all of those transaction costs of trying to define stuff

438
00:22:03.319 --> 00:22:06.329 A:middle L:90%
again once it has been used. So that's where

439
00:22:06.329 --> 00:22:08.200 A:middle L:90%
we start with. The open science framework is as

440
00:22:08.200 --> 00:22:12.470 A:middle L:90%
a means of project management to help facilitate collaboration,

441
00:22:12.509 --> 00:22:17.759 A:middle L:90%
documentation and archiving of research as the research happens.

442
00:22:17.799 --> 00:22:19.720 A:middle L:90%
So the whole goal of OSF is to integrate with

443
00:22:19.720 --> 00:22:23.079 A:middle L:90%
the daily workflow of the researcher to help manage the

444
00:22:23.079 --> 00:22:26.410 A:middle L:90%
materials, the data and the files that support that

445
00:22:26.410 --> 00:22:30.529 A:middle L:90%
research so that they never get lost for one's own

446
00:22:30.529 --> 00:22:33.839 A:middle L:90%
use. So this is just a sample interface of

447
00:22:33.839 --> 00:22:37.420 A:middle L:90%
one project set up in the framework. So when

448
00:22:37.420 --> 00:22:38.660 A:middle L:90%
you log in, you get your own dashboard and

449
00:22:38.660 --> 00:22:41.990 A:middle L:90%
then you can create projects, and then you can

450
00:22:41.990 --> 00:22:45.039 A:middle L:90%
join other contributors to that project. So, Cal

451
00:22:45.039 --> 00:22:48.779 A:middle L:90%
, this is Calvin's project and has many collaborators on

452
00:22:48.779 --> 00:22:51.930 A:middle L:90%
this. And then in the files screen, you

453
00:22:51.930 --> 00:22:55.170 A:middle L:90%
just drag files over and get added to the system

454
00:22:55.440 --> 00:22:57.299 A:middle L:90%
. You can organize by different components of the research

455
00:22:57.299 --> 00:23:00.690 A:middle L:90%
process, so there might be a data component.

456
00:23:00.700 --> 00:23:03.240 A:middle L:90%
And maybe only a subset of the team has access

457
00:23:03.240 --> 00:23:06.809 A:middle L:90%
to the data. There might be demonstrations or materials

458
00:23:06.960 --> 00:23:10.599 A:middle L:90%
to organize the work and the different contributors to each

459
00:23:10.599 --> 00:23:11.910 A:middle L:90%
part of that work. There's a wiki to maintain

460
00:23:11.910 --> 00:23:15.400 A:middle L:90%
information, but all of it is for so that

461
00:23:15.410 --> 00:23:18.420 A:middle L:90%
all of those contributors, uh, no matter who

462
00:23:18.430 --> 00:23:21.859 A:middle L:90%
needs access to it, can find access to the

463
00:23:21.859 --> 00:23:23.549 A:middle L:90%
materials, and they're always in the same place.

464
00:23:23.819 --> 00:23:26.589 A:middle L:90%
And if there is, for example, a manuscript

465
00:23:26.589 --> 00:23:29.539 A:middle L:90%
ng written, but you know where the latest version

466
00:23:29.539 --> 00:23:30.359 A:middle L:90%
of the manuscript is. It's always there. It's

467
00:23:30.359 --> 00:23:33.369 A:middle L:90%
not with person A or maybe it was a person

468
00:23:33.369 --> 00:23:33.730 A:middle L:90%
be. Or maybe it was a person. See

469
00:23:33.730 --> 00:23:37.269 A:middle L:90%
, that had the last version. So that's the

470
00:23:37.269 --> 00:23:41.130 A:middle L:90%
basic goal is to help create a system that is

471
00:23:41.130 --> 00:23:42.039 A:middle L:90%
in the cloud that you will know that I can

472
00:23:42.049 --> 00:23:45.400 A:middle L:90%
always find my stuff when I need it. So

473
00:23:45.400 --> 00:23:47.460 A:middle L:90%
the ideal case when a new came in my office

474
00:23:47.460 --> 00:23:48.109 A:middle L:90%
was would have been. I just go to my

475
00:23:48.109 --> 00:23:52.130 A:middle L:90%
OSF account search for the project and then add him

476
00:23:52.130 --> 00:23:55.390 A:middle L:90%
as a project member, and then he has access

477
00:23:55.390 --> 00:24:00.609 A:middle L:90%
to all of the materials right there. The OSF

478
00:24:00.619 --> 00:24:02.579 A:middle L:90%
also has a number of other features to sort of

479
00:24:02.579 --> 00:24:03.859 A:middle L:90%
help manage that, and one of those is version

480
00:24:03.859 --> 00:24:07.190 A:middle L:90%
control. Version control is a standard and software,

481
00:24:07.190 --> 00:24:10.960 A:middle L:90%
but not a standard in science, and the basic

482
00:24:10.960 --> 00:24:12.890 A:middle L:90%
idea is simple. Our stuff changes over time,

483
00:24:12.940 --> 00:24:15.289 A:middle L:90%
and we should be able to keep track of that

484
00:24:15.349 --> 00:24:19.000 A:middle L:90%
and see previous versions as it changes. You might

485
00:24:19.000 --> 00:24:22.400 A:middle L:90%
do this and formally in manuscript writing, when you're

486
00:24:22.400 --> 00:24:23.369 A:middle L:90%
about to make a big change to a file,

487
00:24:23.369 --> 00:24:26.509 A:middle L:90%
you might upend the date to the name of the

488
00:24:26.509 --> 00:24:29.210 A:middle L:90%
file and then save as and then you work with

489
00:24:29.210 --> 00:24:30.299 A:middle L:90%
the latest version of the file and then in your

490
00:24:30.309 --> 00:24:33.789 A:middle L:90%
folder with the manuscript, you have, like seven

491
00:24:33.789 --> 00:24:36.609 A:middle L:90%
different versions with different dates in it. That is

492
00:24:36.609 --> 00:24:38.910 A:middle L:90%
a form of version control, but there's a lot

493
00:24:38.910 --> 00:24:41.460 A:middle L:90%
of software that helps back that. That makes it

494
00:24:41.460 --> 00:24:45.930 A:middle L:90%
very simple. So what happens whenever you upload a

495
00:24:45.930 --> 00:24:48.640 A:middle L:90%
new version of a file in OSF is it automatically

496
00:24:48.640 --> 00:24:52.269 A:middle L:90%
updates the version number, but you still have access

497
00:24:52.269 --> 00:24:53.130 A:middle L:90%
to all prior versions, and you can look at

498
00:24:53.130 --> 00:24:56.220 A:middle L:90%
the changes that you make from version to version of

499
00:24:56.220 --> 00:25:00.269 A:middle L:90%
different projects. The next thing that it does,

500
00:25:00.269 --> 00:25:03.309 A:middle L:90%
which is really an important element for this, this

501
00:25:03.319 --> 00:25:07.049 A:middle L:90%
efficiency process is emerging the private and public workflows.

502
00:25:07.539 --> 00:25:11.500 A:middle L:90%
When most people ask for doing something open, making

503
00:25:11.500 --> 00:25:15.130 A:middle L:90%
one's data available, it's done after the article.

504
00:25:15.289 --> 00:25:18.690 A:middle L:90%
That's the basis of that data, or the report

505
00:25:18.690 --> 00:25:21.720 A:middle L:90%
of that data has already been accepted, and once

506
00:25:21.720 --> 00:25:23.650 A:middle L:90%
article has been accepted, I want nothing to do

507
00:25:23.660 --> 00:25:26.319 A:middle L:90%
with that research project anymore, right? Most people

508
00:25:26.319 --> 00:25:29.559 A:middle L:90%
think that's where I'm done. I finish the project

509
00:25:29.559 --> 00:25:30.829 A:middle L:90%
. Why am I now going to go back and

510
00:25:30.829 --> 00:25:32.960 A:middle L:90%
do all of this extra work to make the data

511
00:25:32.960 --> 00:25:34.339 A:middle L:90%
available or a code book available. Well, the

512
00:25:34.339 --> 00:25:37.099 A:middle L:90%
goal with OSF is to have all of that curation

513
00:25:37.099 --> 00:25:40.549 A:middle L:90%
process happen. As you're doing the project, we

514
00:25:40.549 --> 00:25:42.099 A:middle L:90%
manage our data as we're doing our project. We're

515
00:25:42.099 --> 00:25:45.170 A:middle L:90%
just not preparing it for public release and some other

516
00:25:45.170 --> 00:25:49.410 A:middle L:90%
website. So, uh, what I described before

517
00:25:49.410 --> 00:25:53.109 A:middle L:90%
is project management is done entirely privately, so research

518
00:25:53.109 --> 00:25:56.779 A:middle L:90%
teams work together on their projects in a private workspace

519
00:25:56.779 --> 00:26:00.349 A:middle L:90%
that's done online. But at the top of every

520
00:26:00.349 --> 00:26:03.599 A:middle L:90%
component of every project is this button that says you

521
00:26:03.599 --> 00:26:06.730 A:middle L:90%
want to make it public. And if you click

522
00:26:06.730 --> 00:26:08.109 A:middle L:90%
that button, this thing pops up and says,

523
00:26:08.109 --> 00:26:10.430 A:middle L:90%
Oh, you're really sure you want to make that

524
00:26:10.430 --> 00:26:11.549 A:middle L:90%
public? And if you click, yes, then

525
00:26:11.549 --> 00:26:15.930 A:middle L:90%
it's public and anyone can access that. So it's

526
00:26:15.930 --> 00:26:18.549 A:middle L:90%
not now an extra process, more work to make

527
00:26:18.549 --> 00:26:22.150 A:middle L:90%
your things public. It's just a matter of desiring

528
00:26:22.150 --> 00:26:25.319 A:middle L:90%
to and moving that mouse and clicking, which isn't

529
00:26:25.329 --> 00:26:30.000 A:middle L:90%
a big barrier in terms of technology, and more

530
00:26:30.009 --> 00:26:33.480 A:middle L:90%
important part is that you have full access control over

531
00:26:33.480 --> 00:26:36.500 A:middle L:90%
the various parts of your workflow. So for example

532
00:26:36.500 --> 00:26:37.799 A:middle L:90%
, there are many datasets that we have that we're

533
00:26:37.799 --> 00:26:41.019 A:middle L:90%
using for multiple purposes their massive data sets, and

534
00:26:41.019 --> 00:26:44.329 A:middle L:90%
we might not be prepared to release the entire data

535
00:26:44.329 --> 00:26:45.599 A:middle L:90%
set on the publication of the first article, but

536
00:26:45.609 --> 00:26:48.700 A:middle L:90%
just the part of it that is relevant to reproducing

537
00:26:48.700 --> 00:26:52.329 A:middle L:90%
those results. Or we might be ready to release

538
00:26:52.329 --> 00:26:56.240 A:middle L:90%
the materials and not other parts of it. Or

539
00:26:56.240 --> 00:26:59.569 A:middle L:90%
you can organize your research so that each of those

540
00:26:59.569 --> 00:27:03.210 A:middle L:90%
pieces have separate access control, and you just release

541
00:27:03.210 --> 00:27:04.339 A:middle L:90%
the pieces that you want to make public and the

542
00:27:04.339 --> 00:27:07.779 A:middle L:90%
other parts stay private for the team. And yet

543
00:27:07.779 --> 00:27:10.490 A:middle L:90%
they're still all integrated into a single workspace, so

544
00:27:10.490 --> 00:27:11.900 A:middle L:90%
you can continue to work with both the private and

545
00:27:11.900 --> 00:27:15.460 A:middle L:90%
the public materials at the same time. And the

546
00:27:15.460 --> 00:27:18.269 A:middle L:90%
outsiders only have access to those that you gave them

547
00:27:18.269 --> 00:27:22.549 A:middle L:90%
access to. Okay, in addition to those we

548
00:27:22.549 --> 00:27:25.890 A:middle L:90%
have to already start thinking about. Incentives will get

549
00:27:25.890 --> 00:27:29.630 A:middle L:90%
back to incentives even more deeply soon. But there

550
00:27:29.630 --> 00:27:30.599 A:middle L:90%
is a number of things that one can do to

551
00:27:30.599 --> 00:27:36.150 A:middle L:90%
make openness actual thing that's attractive rather than inhibition,

552
00:27:36.150 --> 00:27:38.630 A:middle L:90%
a threat for people. One are simple things like

553
00:27:38.630 --> 00:27:41.289 A:middle L:90%
counting the number of downloads that people have of the

554
00:27:41.289 --> 00:27:45.650 A:middle L:90%
various projects tracking how many people visit the page.

555
00:27:45.650 --> 00:27:47.519 A:middle L:90%
It feels kind of good. If someone comes to

556
00:27:47.519 --> 00:27:48.470 A:middle L:90%
the website and actually looks at the things that you're

557
00:27:48.470 --> 00:27:51.960 A:middle L:90%
working on, you can see it has impact.

558
00:27:52.440 --> 00:27:53.720 A:middle L:90%
And then there are a number of things that are

559
00:27:53.720 --> 00:27:59.059 A:middle L:90%
integrated into the OSF of functional links. So links

560
00:27:59.069 --> 00:28:02.579 A:middle L:90%
to your project from other projects because they're either citing

561
00:28:02.579 --> 00:28:04.769 A:middle L:90%
it or they're using it in some way templates.

562
00:28:04.779 --> 00:28:07.980 A:middle L:90%
If someone like your project design, they can template

563
00:28:07.980 --> 00:28:11.759 A:middle L:90%
it if it's public and use that design of the

564
00:28:11.759 --> 00:28:17.160 A:middle L:90%
project as the starting point for their project or forks

565
00:28:17.160 --> 00:28:18.089 A:middle L:90%
. If you develop a measure or a tool or

566
00:28:18.089 --> 00:28:22.599 A:middle L:90%
a method or a bit of code for analysis that

567
00:28:22.599 --> 00:28:25.410 A:middle L:90%
someone wants to extend in some way, they can

568
00:28:25.410 --> 00:28:27.029 A:middle L:90%
fork it into their workspace, which essentially means make

569
00:28:27.029 --> 00:28:30.039 A:middle L:90%
a copy but retain a link and then make whatever

570
00:28:30.039 --> 00:28:33.609 A:middle L:90%
changes they want. And then, as people make

571
00:28:33.609 --> 00:28:37.009 A:middle L:90%
continued changes, that network of links always points back

572
00:28:37.009 --> 00:28:41.769 A:middle L:90%
to your original contribution. So if 500 people an

573
00:28:41.769 --> 00:28:45.089 A:middle L:90%
extension keep building on that particular tool, you'll always

574
00:28:45.089 --> 00:28:47.549 A:middle L:90%
be able to link back to what what was its

575
00:28:47.549 --> 00:28:49.509 A:middle L:90%
origin, and that is much more informative. As

576
00:28:49.509 --> 00:28:52.750 A:middle L:90%
a citation, it's really a functional citation, as

577
00:28:52.750 --> 00:28:56.210 A:middle L:90%
opposed to just a citation in an article which has

578
00:28:56.220 --> 00:28:59.569 A:middle L:90%
ambiguous meaning of how it was used. So there

579
00:28:59.569 --> 00:29:00.980 A:middle L:90%
are a number of these kinds of things integrated in

580
00:29:00.980 --> 00:29:07.180 A:middle L:90%
that are means of alternative metrics for counting contributions to

581
00:29:07.359 --> 00:29:10.960 A:middle L:90%
scientific research that are different than the ordinary metrics of

582
00:29:10.960 --> 00:29:12.950 A:middle L:90%
publication and what outlet and how many people cited it

583
00:29:15.539 --> 00:29:18.569 A:middle L:90%
. Another feature that is an important one for provenance

584
00:29:18.569 --> 00:29:23.079 A:middle L:90%
and preservation is persistent, suitable identifiers. So every

585
00:29:23.079 --> 00:29:27.660 A:middle L:90%
project in every component has a persistent identifier associated with

586
00:29:27.660 --> 00:29:32.099 A:middle L:90%
it, whether it's private or public. And it

587
00:29:32.099 --> 00:29:36.509 A:middle L:90%
automatically generates a citation in the page for you to

588
00:29:36.509 --> 00:29:38.279 A:middle L:90%
be able to cite that information. Because citation is

589
00:29:38.279 --> 00:29:42.640 A:middle L:90%
still one of the means of demonstrating impact in our

590
00:29:42.640 --> 00:29:45.980 A:middle L:90%
discipline, people still care about having acknowledgement of their

591
00:29:45.980 --> 00:29:49.460 A:middle L:90%
work, and then evaluation often is related to that

592
00:29:49.460 --> 00:29:52.210 A:middle L:90%
acknowledgement, and so we aim to make that as

593
00:29:52.210 --> 00:29:56.039 A:middle L:90%
easy as possible to document and track over time by

594
00:29:56.039 --> 00:30:00.930 A:middle L:90%
making every part of the process suitable. This also

595
00:30:00.940 --> 00:30:03.680 A:middle L:90%
helps to introduce new contributor models to science. Right

596
00:30:03.680 --> 00:30:06.730 A:middle L:90%
now, the main way in which science has done

597
00:30:06.730 --> 00:30:08.250 A:middle L:90%
is vertically integrated, right? You have to have

598
00:30:08.250 --> 00:30:11.039 A:middle L:90%
the idea. You have to do the design.

599
00:30:11.039 --> 00:30:11.339 A:middle L:90%
You have to collect the data. You have to

600
00:30:11.349 --> 00:30:14.269 A:middle L:90%
do the analysis. You have to write the report

601
00:30:14.640 --> 00:30:17.849 A:middle L:90%
. You do that yourself within your encapsulated collaborative team

602
00:30:18.539 --> 00:30:21.849 A:middle L:90%
. But crowdsourced science diversifies that by saying people have

603
00:30:21.849 --> 00:30:25.180 A:middle L:90%
lots of different strengths and can contribute in small ways

604
00:30:25.200 --> 00:30:27.569 A:middle L:90%
to larger projects. So I am great at analyzing

605
00:30:27.569 --> 00:30:30.599 A:middle L:90%
data. That's all I want to do. So

606
00:30:30.599 --> 00:30:33.019 A:middle L:90%
great. I take 50 projects and I help with

607
00:30:33.019 --> 00:30:33.930 A:middle L:90%
the data analysis. And I do know writing because

608
00:30:33.930 --> 00:30:37.680 A:middle L:90%
I'm a terrible writer and this is a means of

609
00:30:37.680 --> 00:30:40.910 A:middle L:90%
getting some acknowledgement for that. The components of the

610
00:30:40.910 --> 00:30:44.420 A:middle L:90%
research project allow me to be identified for writing great

611
00:30:44.420 --> 00:30:47.500 A:middle L:90%
code and doing great analysis. And then I can

612
00:30:47.500 --> 00:30:49.809 A:middle L:90%
be cited for that code, in particular acknowledged for

613
00:30:49.809 --> 00:30:53.440 A:middle L:90%
that contribution, rather than only through the singular process

614
00:30:53.450 --> 00:31:00.710 A:middle L:90%
of an entire report. Another factor that the OSF

615
00:31:00.710 --> 00:31:03.500 A:middle L:90%
makes easy is to register the project as a project

616
00:31:03.509 --> 00:31:07.170 A:middle L:90%
and freeze it at a particular point in time.

617
00:31:07.640 --> 00:31:10.240 A:middle L:90%
So I have this original project that I'm working on

618
00:31:10.250 --> 00:31:12.089 A:middle L:90%
that I decided that This is a particular important time

619
00:31:12.099 --> 00:31:15.769 A:middle L:90%
, uh, in the life cycle of this project

620
00:31:15.140 --> 00:31:18.690 A:middle L:90%
. So I want to freeze the project so that

621
00:31:18.690 --> 00:31:22.029 A:middle L:90%
I can point back to this moment in time what

622
00:31:22.029 --> 00:31:25.619 A:middle L:90%
the project looked like them and then I can continue

623
00:31:25.619 --> 00:31:26.839 A:middle L:90%
to work on that project. But I have this

624
00:31:26.839 --> 00:31:29.930 A:middle L:90%
new version once I register it, that is a

625
00:31:29.930 --> 00:31:33.829 A:middle L:90%
frozen version with its own ID identifier that will continue

626
00:31:33.829 --> 00:31:36.869 A:middle L:90%
to persist and always be linked to the project.

627
00:31:36.869 --> 00:31:38.910 A:middle L:90%
That continues to change over time. And there are

628
00:31:38.910 --> 00:31:42.380 A:middle L:90%
many occasions or locations in the time life cycle of

629
00:31:42.380 --> 00:31:45.039 A:middle L:90%
a project that you might want to freeze it and

630
00:31:45.039 --> 00:31:48.289 A:middle L:90%
create a registration. The most prominent one is right

631
00:31:48.299 --> 00:31:52.279 A:middle L:90%
before onset of data collection or data analysis, where

632
00:31:52.279 --> 00:31:56.940 A:middle L:90%
if you have a very strong our priority hypothesis and

633
00:31:56.940 --> 00:31:59.339 A:middle L:90%
want to test it, rather than do an exploratory

634
00:31:59.349 --> 00:32:01.650 A:middle L:90%
hypothesis generating kind of analysis, then you might want

635
00:32:01.650 --> 00:32:05.029 A:middle L:90%
to put constraints on yourself in order to have a

636
00:32:05.029 --> 00:32:08.240 A:middle L:90%
strong confirmatory test and write down your hypothesis and analysis

637
00:32:08.240 --> 00:32:10.049 A:middle L:90%
plan. What is it that we're aiming to do

638
00:32:10.049 --> 00:32:12.299 A:middle L:90%
? What we think we're going to find and how

639
00:32:12.299 --> 00:32:14.569 A:middle L:90%
are we going to analyze those data and if you

640
00:32:14.569 --> 00:32:16.220 A:middle L:90%
register that in advance, you have a certification.

641
00:32:16.339 --> 00:32:20.640 A:middle L:90%
In effect of this is in fact, a confirmatory

642
00:32:20.640 --> 00:32:23.900 A:middle L:90%
test rather than the result of exploratory analysis. Exploratory

643
00:32:23.900 --> 00:32:27.230 A:middle L:90%
analysis is very important. It's how we discover from

644
00:32:27.230 --> 00:32:29.839 A:middle L:90%
science. But there are many occasions where we want

645
00:32:29.839 --> 00:32:32.799 A:middle L:90%
to take a strong stance on hypothesis testing rather than

646
00:32:32.799 --> 00:32:37.269 A:middle L:90%
hypothesis generated and registration preregistration as a means of doing

647
00:32:37.269 --> 00:32:39.170 A:middle L:90%
that. So you create a frozen version. Say

648
00:32:39.170 --> 00:32:42.069 A:middle L:90%
this is what the project was. You can always

649
00:32:42.069 --> 00:32:43.809 A:middle L:90%
point back to that and say, This is This

650
00:32:43.809 --> 00:32:45.940 A:middle L:90%
is what it was. This is my analysis plan

651
00:32:45.950 --> 00:32:47.740 A:middle L:90%
. Here's analyze the data according to my analysis plan

652
00:32:47.750 --> 00:32:52.640 A:middle L:90%
. And here are the results. Okay, the

653
00:32:52.650 --> 00:32:54.609 A:middle L:90%
last feature component that I want to describe with the

654
00:32:54.609 --> 00:32:58.900 A:middle L:90%
technology and then start to move more towards the direction

655
00:32:58.900 --> 00:33:01.710 A:middle L:90%
that this goes. And the incentives work is connecting

656
00:33:01.710 --> 00:33:06.809 A:middle L:90%
services that researchers use. So the goal of the

657
00:33:06.809 --> 00:33:09.819 A:middle L:90%
framework is to be a connector of everything that researchers

658
00:33:09.819 --> 00:33:13.440 A:middle L:90%
use to do their research. Because one of the

659
00:33:13.440 --> 00:33:16.990 A:middle L:90%
main places of losing information about reproducibility is the loss

660
00:33:17.000 --> 00:33:21.789 A:middle L:90%
of knowledge. As people transition between their various technologies

661
00:33:21.789 --> 00:33:24.230 A:middle L:90%
they use to actually do the research, and you

662
00:33:24.240 --> 00:33:28.670 A:middle L:90%
can go into your data generation tool and then out

663
00:33:28.670 --> 00:33:30.069 A:middle L:90%
of that and into your data analysis tool now that

664
00:33:30.069 --> 00:33:34.519 A:middle L:90%
into your figure generating tools. And all of those

665
00:33:34.519 --> 00:33:37.450 A:middle L:90%
transitions have costs to just the work to get from

666
00:33:37.450 --> 00:33:40.880 A:middle L:90%
one to another. But also there's information and process

667
00:33:40.880 --> 00:33:44.710 A:middle L:90%
lost. So if we can connect those services together

668
00:33:44.720 --> 00:33:46.609 A:middle L:90%
into a single workflow, then it's easy to log

669
00:33:46.609 --> 00:33:50.109 A:middle L:90%
and track what it is that happened across those.

670
00:33:50.460 --> 00:33:52.019 A:middle L:90%
And then we can help those different services talk to

671
00:33:52.019 --> 00:33:54.890 A:middle L:90%
each other so that you can be much more efficient

672
00:33:54.900 --> 00:33:58.309 A:middle L:90%
in your work. So what I'm showing you here

673
00:33:58.309 --> 00:34:00.460 A:middle L:90%
are some of the add ons that we have connected

674
00:34:00.460 --> 00:34:01.900 A:middle L:90%
to the open science framework. So if you are

675
00:34:01.900 --> 00:34:05.890 A:middle L:90%
a git hub user or use Amazon simple storage service

676
00:34:05.890 --> 00:34:07.970 A:middle L:90%
, this is if you have lots of lots and

677
00:34:07.970 --> 00:34:08.969 A:middle L:90%
lots of data, you need a very, very

678
00:34:08.969 --> 00:34:12.579 A:middle L:90%
cheap place to store it. Amazon S three is

679
00:34:12.579 --> 00:34:15.320 A:middle L:90%
a good service for that. Almost. It's very

680
00:34:15.320 --> 00:34:16.829 A:middle L:90%
rarely used by scientists. Now it's almost only used

681
00:34:16.829 --> 00:34:22.230 A:middle L:90%
by software developers because it has some some overhead and

682
00:34:22.230 --> 00:34:24.059 A:middle L:90%
learning how to access it. But we created a

683
00:34:24.059 --> 00:34:27.400 A:middle L:90%
simplified interface so it's very easy to make. Use

684
00:34:27.400 --> 00:34:31.349 A:middle L:90%
of fig share is another popular place for storing materials

685
00:34:31.360 --> 00:34:35.219 A:middle L:90%
and data from research and dropbox. A lot of

686
00:34:35.219 --> 00:34:37.139 A:middle L:90%
people use so all of those are connected to the

687
00:34:37.139 --> 00:34:39.610 A:middle L:90%
open science framework so that if you have a project

688
00:34:39.619 --> 00:34:43.460 A:middle L:90%
you can connect your Dropbox folder that has stuff relevant

689
00:34:43.460 --> 00:34:45.820 A:middle L:90%
to it. You're get Hub, Repo or any

690
00:34:45.820 --> 00:34:49.090 A:middle L:90%
other tools to that project. And then any other

691
00:34:49.090 --> 00:34:51.929 A:middle L:90%
collaborator on that project has access to those as well

692
00:34:51.940 --> 00:34:53.960 A:middle L:90%
, even if they don't use those other services and

693
00:34:53.960 --> 00:34:57.289 A:middle L:90%
they all exist in a single file tree. So

694
00:34:57.289 --> 00:34:59.559 A:middle L:90%
it's like you're working with all of these different systems

695
00:34:59.559 --> 00:35:02.079 A:middle L:90%
at the same time, and they can communicate with

696
00:35:02.079 --> 00:35:05.539 A:middle L:90%
each other if they have things to communicate with.

697
00:35:06.639 --> 00:35:08.809 A:middle L:90%
So this is going back to this overall goal is

698
00:35:08.809 --> 00:35:12.619 A:middle L:90%
a means of trying to connect those services across the

699
00:35:12.619 --> 00:35:15.159 A:middle L:90%
workflow with the idea that if we can connect services

700
00:35:15.159 --> 00:35:17.019 A:middle L:90%
for all the different things that you do in your

701
00:35:17.019 --> 00:35:21.090 A:middle L:90%
research, then you have one place that you can

702
00:35:21.090 --> 00:35:22.619 A:middle L:90%
go and you can select from the different kinds of

703
00:35:22.619 --> 00:35:25.489 A:middle L:90%
things that you use, uh and then and then

704
00:35:25.489 --> 00:35:28.789 A:middle L:90%
take advantage of the fact that these things are all

705
00:35:28.789 --> 00:35:31.059 A:middle L:90%
connected. So these are a variety of the different

706
00:35:31.070 --> 00:35:36.170 A:middle L:90%
types of services that researchers might use, particularly the

707
00:35:36.170 --> 00:35:38.730 A:middle L:90%
social behavioral sciences Is the emphasis here. Uh,

708
00:35:38.739 --> 00:35:43.559 A:middle L:90%
and the OSF is connecting through a peer relationships application

709
00:35:43.559 --> 00:35:45.670 A:middle L:90%
. Programming interfaces? Uh huh. Uh huh.

710
00:35:46.239 --> 00:35:51.170 A:middle L:90%
Information exchange between these different services. So you can

711
00:35:51.170 --> 00:35:54.010 A:middle L:90%
imagine, for example, if your publisher is connected

712
00:35:54.010 --> 00:35:58.260 A:middle L:90%
to your data analytics tools connected to your data repository

713
00:35:58.739 --> 00:36:01.119 A:middle L:90%
, then what we could do is make every influence

714
00:36:01.119 --> 00:36:05.599 A:middle L:90%
test in your manuscript a link, the f value

715
00:36:05.599 --> 00:36:07.019 A:middle L:90%
, the T test, whatever it is, if

716
00:36:07.019 --> 00:36:10.760 A:middle L:90%
the person clicks the link, then up comes the

717
00:36:10.769 --> 00:36:15.260 A:middle L:90%
code analysis code that generated that test. And up

718
00:36:15.260 --> 00:36:17.170 A:middle L:90%
comes the data that the analysis code was applied to

719
00:36:17.179 --> 00:36:20.420 A:middle L:90%
in order to generate that test. And then you

720
00:36:20.420 --> 00:36:22.710 A:middle L:90%
can reproduce the analysis immediately. And then you can

721
00:36:22.710 --> 00:36:23.050 A:middle L:90%
also say, Well, I would have added a

722
00:36:23.050 --> 00:36:24.679 A:middle L:90%
cove area there. Let's see what happens if I

723
00:36:24.679 --> 00:36:28.519 A:middle L:90%
had a cove area and then you can be interacting

724
00:36:28.630 --> 00:36:30.059 A:middle L:90%
precisely with their data with how it is. They

725
00:36:30.059 --> 00:36:34.710 A:middle L:90%
did their research in real time rather than writing to

726
00:36:34.710 --> 00:36:37.559 A:middle L:90%
the person. If you even bother saying I wonder

727
00:36:37.559 --> 00:36:38.420 A:middle L:90%
what would happen often. Just read the paper.

728
00:36:38.420 --> 00:36:40.010 A:middle L:90%
We say, Oh, they should have done that

729
00:36:40.019 --> 00:36:42.940 A:middle L:90%
and that's the end of it. But you could

730
00:36:42.949 --> 00:36:46.130 A:middle L:90%
just immediately do that and learn a lot more and

731
00:36:46.130 --> 00:36:51.280 A:middle L:90%
be more interactive with the research process itself. Okay

732
00:36:51.280 --> 00:36:52.550 A:middle L:90%
, so that's where we're heading. We have about

733
00:36:52.550 --> 00:36:53.480 A:middle L:90%
five of these connected now, and there's about a

734
00:36:53.480 --> 00:36:57.579 A:middle L:90%
dozen of them in development, and many more will

735
00:36:57.590 --> 00:36:59.780 A:middle L:90%
be in development. We hope, in the very

736
00:36:59.780 --> 00:37:04.579 A:middle L:90%
near future for connecting the services. I want to

737
00:37:04.590 --> 00:37:09.000 A:middle L:90%
mention another part of the technology projects because it's because

738
00:37:09.000 --> 00:37:14.280 A:middle L:90%
it's sort of leading into a larger direction here for

739
00:37:14.280 --> 00:37:16.400 A:middle L:90%
the kinds of things that OSF can do, but

740
00:37:16.400 --> 00:37:20.000 A:middle L:90%
also the partnerships that can be formed. And this

741
00:37:20.000 --> 00:37:22.429 A:middle L:90%
is a collaboration with share, which Tyler Walters was

742
00:37:22.429 --> 00:37:25.739 A:middle L:90%
just. I think the term here is installed,

743
00:37:27.130 --> 00:37:31.070 A:middle L:90%
installed as she served the installation process of the president

744
00:37:31.070 --> 00:37:34.789 A:middle L:90%
. I was expecting to see him in pose for

745
00:37:34.789 --> 00:37:37.250 A:middle L:90%
him or something, uh, from this weekend.

746
00:37:37.929 --> 00:37:44.639 A:middle L:90%
But the share team is trying to facilitate information exchange

747
00:37:44.639 --> 00:37:47.980 A:middle L:90%
about research objects, and we are partnering on the

748
00:37:47.980 --> 00:37:53.699 A:middle L:90%
technology development of that And the starting project is a

749
00:37:53.699 --> 00:37:59.000 A:middle L:90%
notification system. There are many different providers, different

750
00:37:59.000 --> 00:38:01.909 A:middle L:90%
services, different groups that are generating information, events

751
00:38:01.909 --> 00:38:05.320 A:middle L:90%
that happen about research. Data set is available.

752
00:38:05.320 --> 00:38:07.440 A:middle L:90%
The publication is available. These materials have been made

753
00:38:07.440 --> 00:38:10.409 A:middle L:90%
available, and all of them come with different kinds

754
00:38:10.420 --> 00:38:14.179 A:middle L:90%
of metadata because none of them have been working in

755
00:38:14.179 --> 00:38:15.940 A:middle L:90%
concert. They each are working on their own because

756
00:38:15.940 --> 00:38:17.250 A:middle L:90%
they have their own problems to solve. And so

757
00:38:17.250 --> 00:38:20.639 A:middle L:90%
what share aims to do is take all of these

758
00:38:20.639 --> 00:38:22.800 A:middle L:90%
different kinds of metadata about these different research events that

759
00:38:22.800 --> 00:38:28.360 A:middle L:90%
are happening around the world and align them and normalize

760
00:38:28.360 --> 00:38:31.909 A:middle L:90%
them into a single schema that makes that whole stream

761
00:38:31.909 --> 00:38:36.280 A:middle L:90%
of everything that's happening in research available to anybody that

762
00:38:36.280 --> 00:38:38.110 A:middle L:90%
wants to consume it. So there are a number

763
00:38:38.110 --> 00:38:40.050 A:middle L:90%
of different. There's, like 30 or so providers

764
00:38:40.050 --> 00:38:43.869 A:middle L:90%
so far linked to this, Uh, I think

765
00:38:43.869 --> 00:38:46.440 A:middle L:90%
it's Virginia Tech. Uh, repository is one of

766
00:38:46.440 --> 00:38:50.170 A:middle L:90%
them. V Tech works. Yeah, is already

767
00:38:50.170 --> 00:38:53.110 A:middle L:90%
linked, But many different providers, publishers, data

768
00:38:53.110 --> 00:38:58.820 A:middle L:90%
providers, institutions, whatever are generating metadata about events

769
00:38:58.820 --> 00:39:02.489 A:middle L:90%
that occur in their research zone, and those are

770
00:39:02.489 --> 00:39:07.280 A:middle L:90%
getting normalized into the framework and share and then push

771
00:39:07.280 --> 00:39:12.039 A:middle L:90%
through as a notification system. And this API anybody

772
00:39:12.039 --> 00:39:15.070 A:middle L:90%
can connect to so that they can provide ways to

773
00:39:15.070 --> 00:39:19.090 A:middle L:90%
surface that information. And so there's RSS feeds or

774
00:39:19.090 --> 00:39:21.539 A:middle L:90%
resource sink or pub sub hubbub, which is the

775
00:39:21.539 --> 00:39:24.110 A:middle L:90%
best name ever of anything. Uh and, um

776
00:39:24.119 --> 00:39:29.070 A:middle L:90%
, we have to consumers already through OSF for that

777
00:39:29.070 --> 00:39:32.150 A:middle L:90%
content coming through. So the basic idea is everything

778
00:39:32.150 --> 00:39:37.150 A:middle L:90%
that ever happens in research should be accessible for somebody

779
00:39:37.619 --> 00:39:38.289 A:middle L:90%
. And so if we can find a way to

780
00:39:38.289 --> 00:39:42.719 A:middle L:90%
put them all through a single stream, then anybody

781
00:39:42.719 --> 00:39:45.010 A:middle L:90%
can read that stream and learn about whatever it is

782
00:39:45.010 --> 00:39:47.010 A:middle L:90%
that's happening wherever it's happening. And so let me

783
00:39:47.010 --> 00:39:50.909 A:middle L:90%
give you two illustrations of how we're using this already

784
00:39:50.920 --> 00:39:54.650 A:middle L:90%
with OSF Search and OSF metadata so you can use

785
00:39:54.650 --> 00:39:59.170 A:middle L:90%
the open science framework. It's consuming the information from

786
00:39:59.170 --> 00:40:01.369 A:middle L:90%
that happy and do searches on it so that you

787
00:40:01.369 --> 00:40:06.510 A:middle L:90%
can discover from whichever provider is coming. What information

788
00:40:06.590 --> 00:40:08.139 A:middle L:90%
is available to that search term, right? So

789
00:40:08.139 --> 00:40:10.320 A:middle L:90%
search for science and these sorts of things come up

790
00:40:12.219 --> 00:40:15.639 A:middle L:90%
, and then it's not just because the OSF has

791
00:40:15.639 --> 00:40:17.380 A:middle L:90%
all of these other services. It's not just making

792
00:40:17.380 --> 00:40:21.960 A:middle L:90%
available what information is out there but also providing an

793
00:40:21.960 --> 00:40:23.619 A:middle L:90%
opportunity to curate it. So there's a curate,

794
00:40:23.619 --> 00:40:27.460 A:middle L:90%
this button if you want to curate that. For

795
00:40:27.460 --> 00:40:30.889 A:middle L:90%
example, if you are this researcher and you say

796
00:40:30.900 --> 00:40:32.900 A:middle L:90%
, Oh, I have other materials that are relevant

797
00:40:32.929 --> 00:40:36.539 A:middle L:90%
to that paper that just came through. I want

798
00:40:36.539 --> 00:40:39.079 A:middle L:90%
to add my other materials or my data or I

799
00:40:39.090 --> 00:40:42.989 A:middle L:90%
already put it into another service and I want to

800
00:40:42.989 --> 00:40:45.590 A:middle L:90%
connect those things together. Then you could collect the

801
00:40:45.590 --> 00:40:49.340 A:middle L:90%
curate this button and curate your content, or someone

802
00:40:49.340 --> 00:40:51.139 A:middle L:90%
else might be able to help you curate it.

803
00:40:51.510 --> 00:40:52.949 A:middle L:90%
And this is an example of a curated project.

804
00:40:53.039 --> 00:40:57.789 A:middle L:90%
So this is a project, and there's different notification

805
00:40:57.789 --> 00:41:00.260 A:middle L:90%
events happening. The article was published in clause.

806
00:41:00.480 --> 00:41:02.010 A:middle L:90%
The data is stored in data one, and so

807
00:41:02.010 --> 00:41:06.739 A:middle L:90%
those different notifications were merged as separate components in OSF

808
00:41:07.110 --> 00:41:08.780 A:middle L:90%
, and now all of that information is available together

809
00:41:09.510 --> 00:41:12.969 A:middle L:90%
. So for us, there's two different views happening

810
00:41:12.969 --> 00:41:15.739 A:middle L:90%
on what OSF is doing. One is this bottom

811
00:41:15.739 --> 00:41:17.670 A:middle L:90%
up way of giving researchers access to be able to

812
00:41:17.670 --> 00:41:21.760 A:middle L:90%
use the OSF to help them manage their own workflow

813
00:41:21.760 --> 00:41:23.539 A:middle L:90%
, to manage their materials and data privately and publicly

814
00:41:23.539 --> 00:41:25.980 A:middle L:90%
. If they want to make things available and then

815
00:41:25.980 --> 00:41:30.130 A:middle L:90%
this data stream that's coming through and blasting everything that's

816
00:41:30.130 --> 00:41:34.239 A:middle L:90%
happening in research that's making those materials available in OSF

817
00:41:34.610 --> 00:41:37.969 A:middle L:90%
and giving the opportunity to link it to researchers doing

818
00:41:37.969 --> 00:41:39.349 A:middle L:90%
their work flow as they do it. So if

819
00:41:39.349 --> 00:41:42.420 A:middle L:90%
they want to make things more available now, they

820
00:41:42.420 --> 00:41:45.360 A:middle L:90%
can integrate that much more easily with other content and

821
00:41:45.360 --> 00:41:49.019 A:middle L:90%
materials. So that then becomes a registry in a

822
00:41:49.019 --> 00:41:51.880 A:middle L:90%
large part is there's many different ways to access that

823
00:41:51.880 --> 00:41:54.260 A:middle L:90%
content across a variety of president. But anyone else

824
00:41:54.260 --> 00:41:57.000 A:middle L:90%
can provide their own interface to this, too.

825
00:41:57.000 --> 00:41:59.150 A:middle L:90%
It doesn't have to be OSF because it's an open

826
00:41:59.159 --> 00:42:02.199 A:middle L:90%
API. Okay, so I want to stop with

827
00:42:02.199 --> 00:42:06.380 A:middle L:90%
the technology part and just mention that all of those

828
00:42:06.380 --> 00:42:08.920 A:middle L:90%
things are, uh, the Center for Open Science

829
00:42:08.920 --> 00:42:12.920 A:middle L:90%
doesn't develop any intellectual property for itself. So everything

830
00:42:12.920 --> 00:42:16.010 A:middle L:90%
is open source, freely available technology, and we

831
00:42:16.010 --> 00:42:19.840 A:middle L:90%
have no monetization scheme at all. So everything is

832
00:42:19.840 --> 00:42:22.659 A:middle L:90%
free, and the goal is to build services that

833
00:42:22.659 --> 00:42:25.369 A:middle L:90%
can be used and that can be community driven,

834
00:42:25.380 --> 00:42:29.030 A:middle L:90%
that many people can contribute to those services and the

835
00:42:29.030 --> 00:42:31.190 A:middle L:90%
development and maintenance and sustaining of those. If you

836
00:42:31.190 --> 00:42:32.840 A:middle L:90%
want to talk about, how can you do something

837
00:42:32.840 --> 00:42:36.619 A:middle L:90%
that's only free. We can talk about the sustainability

838
00:42:36.619 --> 00:42:40.010 A:middle L:90%
parts later, but I'm happy to talk. Uh

839
00:42:40.500 --> 00:42:43.969 A:middle L:90%
, okay, so that's it for some of the

840
00:42:43.969 --> 00:42:45.659 A:middle L:90%
technology. Now, I want to talk about two

841
00:42:45.659 --> 00:42:47.719 A:middle L:90%
other pieces in the remaining time that we have one

842
00:42:47.719 --> 00:42:52.280 A:middle L:90%
very briefly, and that is training services. So

843
00:42:52.280 --> 00:42:53.739 A:middle L:90%
part of the challenge with a lot of the reproducibility

844
00:42:53.739 --> 00:42:57.760 A:middle L:90%
practices is that people aren't getting access to the kinds

845
00:42:57.760 --> 00:43:00.239 A:middle L:90%
of training for the tools that could be very useful

846
00:43:00.449 --> 00:43:04.010 A:middle L:90%
to improve their efficiency and quality of research. So

847
00:43:04.599 --> 00:43:07.019 A:middle L:90%
we have a consulting, uh, email address right

848
00:43:07.019 --> 00:43:08.159 A:middle L:90%
here. Stats Consulting at C u s dot io

849
00:43:08.159 --> 00:43:12.159 A:middle L:90%
There's a website. See us dot io stat underscore

850
00:43:12.159 --> 00:43:15.929 A:middle L:90%
consulting. Uh, and you can arrange. You

851
00:43:15.929 --> 00:43:17.840 A:middle L:90%
can send email questions you can arrange an online hangout

852
00:43:17.849 --> 00:43:20.920 A:middle L:90%
you can do a lot of you can arrange an

853
00:43:20.920 --> 00:43:22.269 A:middle L:90%
in person training. We have one here on Thursday

854
00:43:22.510 --> 00:43:27.469 A:middle L:90%
for different parts of reproducible practices, either using tools

855
00:43:27.469 --> 00:43:30.920 A:middle L:90%
and computation, doing different kinds of analysis or helping

856
00:43:30.920 --> 00:43:34.039 A:middle L:90%
manage workflow just to point out Tyler mentioned at the

857
00:43:34.039 --> 00:43:36.320 A:middle L:90%
beginning. But just to point out the workshop that

858
00:43:36.320 --> 00:43:37.760 A:middle L:90%
will be here this week, it's on the 23rd

859
00:43:37.769 --> 00:43:42.789 A:middle L:90%
, three o'clock in the library multipurpose room and Courtney

860
00:43:42.789 --> 00:43:45.460 A:middle L:90%
Soderbergh, who is on the cause. Staff will

861
00:43:45.460 --> 00:43:47.309 A:middle L:90%
be running that so if that's of interest and also

862
00:43:47.309 --> 00:43:51.829 A:middle L:90%
be giving training on OSF use and some practical things

863
00:43:51.829 --> 00:43:54.320 A:middle L:90%
as well. So that's all I'll say about training

864
00:43:55.000 --> 00:43:58.440 A:middle L:90%
. So I want to return back to the theme

865
00:43:58.449 --> 00:44:01.840 A:middle L:90%
of openness more than access. And the point that

866
00:44:01.840 --> 00:44:04.989 A:middle L:90%
I've been making so far on that is that there

867
00:44:04.989 --> 00:44:07.780 A:middle L:90%
is an entire workflow that can be made available.

868
00:44:07.840 --> 00:44:12.019 A:middle L:90%
And our values suggest that that's part of what can

869
00:44:12.019 --> 00:44:15.750 A:middle L:90%
help with scientific research in the pursuit of knowledge is

870
00:44:15.750 --> 00:44:17.010 A:middle L:90%
knowing the basis of claims. If I want to

871
00:44:17.010 --> 00:44:20.610 A:middle L:90%
dig deeper in principle, I can dig deeper.

872
00:44:20.789 --> 00:44:22.099 A:middle L:90%
I don't have to, but I can't on two

873
00:44:22.099 --> 00:44:23.780 A:middle L:90%
different things. And if you want to find out

874
00:44:23.780 --> 00:44:25.849 A:middle L:90%
how I've arranged something, how I did something,

875
00:44:25.849 --> 00:44:30.030 A:middle L:90%
you can find that out. Um, and there's

876
00:44:30.039 --> 00:44:31.960 A:middle L:90%
a lot of reasons to be caring about that beyond

877
00:44:31.960 --> 00:44:35.409 A:middle L:90%
just the ideal of wouldn't that be a good thing

878
00:44:35.789 --> 00:44:37.989 A:middle L:90%
? And this has been making a lot of news

879
00:44:37.989 --> 00:44:40.429 A:middle L:90%
headlines in the last five or six years about concerns

880
00:44:40.440 --> 00:44:44.699 A:middle L:90%
in Reproducibility that there may actually be a problem to

881
00:44:44.699 --> 00:44:46.219 A:middle L:90%
address rather than it just being a good thing to

882
00:44:46.219 --> 00:44:50.510 A:middle L:90%
do in principle. Here are headlines that came out

883
00:44:50.510 --> 00:44:52.300 A:middle L:90%
in 2011 and nature, believe it or not,

884
00:44:52.300 --> 00:44:54.380 A:middle L:90%
how much can we rely on published data on potential

885
00:44:54.380 --> 00:44:59.719 A:middle L:90%
drug targets and raise standards for pre clinical cancer research

886
00:45:00.090 --> 00:45:02.440 A:middle L:90%
? These two reports were ones from Bayer and Amgen

887
00:45:02.670 --> 00:45:07.559 A:middle L:90%
Industrial Laboratories that we're trying to replicate. A couple

888
00:45:07.570 --> 00:45:10.710 A:middle L:90%
a few dozen, I guess, studies each in

889
00:45:10.710 --> 00:45:15.489 A:middle L:90%
biomedicine, mostly in oncology research, trying to take

890
00:45:15.500 --> 00:45:17.480 A:middle L:90%
what was found in basic labs, replicate those results

891
00:45:17.480 --> 00:45:21.260 A:middle L:90%
and then extend them in order to develop new therapies

892
00:45:21.260 --> 00:45:24.289 A:middle L:90%
, pharmaceuticals or whatever. And they reported very poor

893
00:45:24.300 --> 00:45:28.630 A:middle L:90%
reproducibility rates and their efforts to take these basic labs

894
00:45:28.630 --> 00:45:30.960 A:middle L:90%
. One of them was able to produce 11% of

895
00:45:30.960 --> 00:45:32.780 A:middle L:90%
the original findings. The other 1 25% of those

896
00:45:32.780 --> 00:45:37.210 A:middle L:90%
original findings of published studies that have been having a

897
00:45:37.210 --> 00:45:40.210 A:middle L:90%
high impact in the research literature and their argument was

898
00:45:40.210 --> 00:45:44.389 A:middle L:90%
that they didn't screw up. It's the field that

899
00:45:44.389 --> 00:45:46.570 A:middle L:90%
screwed up, and things are happening that are making

900
00:45:46.570 --> 00:45:50.059 A:middle L:90%
it so that published results are not as reliable as

901
00:45:50.059 --> 00:45:52.380 A:middle L:90%
they ought to be. There's many things to debate

902
00:45:52.380 --> 00:45:54.449 A:middle L:90%
about of where the source of the challenges are whether

903
00:45:54.449 --> 00:45:58.460 A:middle L:90%
they identified those percentages correctly. They didn't have a

904
00:45:58.460 --> 00:46:02.340 A:middle L:90%
random sample, for example, of of research projects

905
00:46:02.340 --> 00:46:04.690 A:middle L:90%
, and it's not. They didn't make any of

906
00:46:04.690 --> 00:46:07.130 A:middle L:90%
their data available or any of what studies they were

907
00:46:07.130 --> 00:46:07.590 A:middle L:90%
, so no one could critique how it is.

908
00:46:07.590 --> 00:46:10.400 A:middle L:90%
They actually did it, but nonetheless, they got

909
00:46:10.400 --> 00:46:16.070 A:middle L:90%
plenty of attention for the challenges of Reproducibility. And

910
00:46:16.070 --> 00:46:17.920 A:middle L:90%
at the same time, there's been a number of

911
00:46:17.920 --> 00:46:22.559 A:middle L:90%
researchers and meta science demonstrating that there are really curious

912
00:46:22.570 --> 00:46:24.349 A:middle L:90%
issues in how it is the scientific. The published

913
00:46:24.349 --> 00:46:29.599 A:middle L:90%
literature corresponds to what might actually be happening in research

914
00:46:30.079 --> 00:46:32.409 A:middle L:90%
. This is one figure from Daniel Finn Ellie's work

915
00:46:32.420 --> 00:46:36.710 A:middle L:90%
on looking at the rates of positive findings across a

916
00:46:36.710 --> 00:46:39.260 A:middle L:90%
variety of different disciplines in science. So positive finding

917
00:46:39.260 --> 00:46:43.929 A:middle L:90%
meaning confirming the hypothesis found a difference in the treatments

918
00:46:43.940 --> 00:46:45.880 A:middle L:90%
between different conditions are found. An association that was

919
00:46:45.880 --> 00:46:49.849 A:middle L:90%
hypothesized. What he found was that you know,

920
00:46:49.849 --> 00:46:52.000 A:middle L:90%
from physics on through psychology, psychology of the winners

921
00:46:52.380 --> 00:46:58.500 A:middle L:90%
get 85% or greater confirmation of the hypothesis in the

922
00:46:58.980 --> 00:47:04.099 A:middle L:90%
in the reported results. That's a remarkable success rate

923
00:47:04.119 --> 00:47:07.199 A:middle L:90%
of showing positive results in literatures, and it's even

924
00:47:07.199 --> 00:47:12.579 A:middle L:90%
more remarkable when it's contrasted with the power of research

925
00:47:12.579 --> 00:47:15.690 A:middle L:90%
designs. So there are many projects showing this.

926
00:47:15.690 --> 00:47:17.320 A:middle L:90%
This is just one example that we did last year

927
00:47:17.920 --> 00:47:22.179 A:middle L:90%
investigating the power of research designs. And so in

928
00:47:22.179 --> 00:47:23.849 A:middle L:90%
this case, we did a meta analysis of meta

929
00:47:23.849 --> 00:47:28.739 A:middle L:90%
analyses, looking at a variety of different areas in

930
00:47:28.739 --> 00:47:30.789 A:middle L:90%
neuroscience, the overall power to detect defects that they

931
00:47:30.789 --> 00:47:35.519 A:middle L:90%
were investigating and what I'm showing you here on this

932
00:47:35.519 --> 00:47:39.010 A:middle L:90%
history Graham plot is the median power of the various

933
00:47:39.010 --> 00:47:42.690 A:middle L:90%
meta analyses. So many of them had extremely low

934
00:47:42.690 --> 00:47:45.750 A:middle L:90%
power. And then there's a small minority that had

935
00:47:45.760 --> 00:47:49.920 A:middle L:90%
very, very high power. But the overall medium

936
00:47:49.920 --> 00:47:52.329 A:middle L:90%
was something like 30%. And I remember that,

937
00:47:52.329 --> 00:47:54.409 A:middle L:90%
right? Maybe it's some 20 something percent power.

938
00:47:54.780 --> 00:47:57.860 A:middle L:90%
If you're not familiar with the concept of power,

939
00:47:58.030 --> 00:48:04.090 A:middle L:90%
20% power means that assuming everything being investigated is true

940
00:48:05.070 --> 00:48:07.030 A:middle L:90%
. The positive result rate that we would expect is

941
00:48:07.030 --> 00:48:12.579 A:middle L:90%
20%. That's how strong, uh, sample how

942
00:48:12.579 --> 00:48:14.900 A:middle L:90%
large the samples, How the size of the effects

943
00:48:14.900 --> 00:48:17.570 A:middle L:90%
being investigated. We'd expect to 20% positive result result

944
00:48:17.570 --> 00:48:20.900 A:middle L:90%
rate, even if everything were studying is true.

945
00:48:21.570 --> 00:48:25.619 A:middle L:90%
Whereas neuroscience back on this page has 85% positive result

946
00:48:25.619 --> 00:48:30.000 A:middle L:90%
rate. So if the power is only about 20%

947
00:48:30.010 --> 00:48:34.110 A:middle L:90%
assuming these are comparable and yet 85% of the results

948
00:48:34.110 --> 00:48:37.579 A:middle L:90%
are positive, it's not possible to get that lucky

949
00:48:37.969 --> 00:48:39.829 A:middle L:90%
. So it suggests that there's something else happening in

950
00:48:39.829 --> 00:48:44.670 A:middle L:90%
the research literature. Negative results are systematically getting excluded

951
00:48:44.670 --> 00:48:47.789 A:middle L:90%
, perhaps, or results that are partially positive or

952
00:48:47.789 --> 00:48:52.900 A:middle L:90%
getting amplified and turned into positive results. Beyond their

953
00:48:52.900 --> 00:48:55.829 A:middle L:90%
evidence. There are a couple other ways to illustrate

954
00:48:55.840 --> 00:48:58.650 A:middle L:90%
some of these challenges. I'll just give a brief

955
00:48:58.659 --> 00:49:01.809 A:middle L:90%
example with this. The basic idea is that when

956
00:49:01.809 --> 00:49:06.440 A:middle L:90%
we read a report of some analysis reporting a phenomenon

957
00:49:06.869 --> 00:49:07.239 A:middle L:90%
, we assume, well, that's what the data

958
00:49:07.239 --> 00:49:10.349 A:middle L:90%
show they just did. The analysis. There's the

959
00:49:10.349 --> 00:49:13.690 A:middle L:90%
answer, and I might have analyzed another way.

960
00:49:13.690 --> 00:49:16.460 A:middle L:90%
But that's that's the answer we were interested in looking

961
00:49:16.460 --> 00:49:21.010 A:middle L:90%
at variation in replica bility. And so, uh

962
00:49:21.019 --> 00:49:22.710 A:middle L:90%
, what I'm showing you here are 28 different teams

963
00:49:22.949 --> 00:49:30.519 A:middle L:90%
that investigative research and these are This is an odd

964
00:49:30.530 --> 00:49:32.789 A:middle L:90%
ratio on the X axis. A one means no

965
00:49:32.789 --> 00:49:36.320 A:middle L:90%
effect at all. There was no difference between these

966
00:49:36.320 --> 00:49:38.400 A:middle L:90%
treatment conditions or no association, uh, detected,

967
00:49:38.869 --> 00:49:43.539 A:middle L:90%
and then the dot is the effect size estimate that

968
00:49:43.539 --> 00:49:45.949 A:middle L:90%
the team got. And then this is the 95%

969
00:49:45.949 --> 00:49:50.590 A:middle L:90%
confidence interval the lines. And so an ordinary use

970
00:49:50.590 --> 00:49:53.579 A:middle L:90%
you would consider any confidence interval that doesn't overlap the

971
00:49:53.579 --> 00:49:57.980 A:middle L:90%
dotted line to be a significant result positive result and

972
00:49:57.980 --> 00:50:00.639 A:middle L:90%
about two thirds of these positive results about one third

973
00:50:00.639 --> 00:50:02.510 A:middle L:90%
or not positive. It doesn't really matter what the

974
00:50:02.510 --> 00:50:06.130 A:middle L:90%
research question is. But what we were investigating here

975
00:50:06.130 --> 00:50:09.159 A:middle L:90%
was the likelihood of players with darker skin tone getting

976
00:50:09.159 --> 00:50:12.889 A:middle L:90%
red cards in soccer than players with lighter skin tone

977
00:50:13.659 --> 00:50:16.019 A:middle L:90%
. And so 28 different teams all investigated that same

978
00:50:16.030 --> 00:50:20.170 A:middle L:90%
hypothesis, and about two thirds of them found positive

979
00:50:20.170 --> 00:50:22.869 A:middle L:90%
evidence. Now, the twist for this is that

980
00:50:22.869 --> 00:50:25.190 A:middle L:90%
they all use the same data set. Uh huh

981
00:50:25.659 --> 00:50:29.760 A:middle L:90%
. So we gave them the research question and the

982
00:50:29.760 --> 00:50:30.659 A:middle L:90%
data and said to analyze it, how you want

983
00:50:30.659 --> 00:50:34.880 A:middle L:90%
to analyze it and just send us the result?

984
00:50:35.760 --> 00:50:38.269 A:middle L:90%
Uh, the additional factors that make this more compelling

985
00:50:38.280 --> 00:50:39.940 A:middle L:90%
. Well, so and then. So what you

986
00:50:39.940 --> 00:50:43.090 A:middle L:90%
see is a great deal of variability, even though

987
00:50:43.090 --> 00:50:45.269 A:middle L:90%
they're using the same data set the same question.

988
00:50:45.860 --> 00:50:47.989 A:middle L:90%
What we also did is the teams that are involved

989
00:50:49.000 --> 00:50:52.840 A:middle L:90%
are very quantitatively sophisticated many of them have is their

990
00:50:52.849 --> 00:50:55.409 A:middle L:90%
p or the lead of it. Our instructors of

991
00:50:55.409 --> 00:50:59.230 A:middle L:90%
statistics, in whatever discipline that they're in, have

992
00:50:59.230 --> 00:51:01.400 A:middle L:90%
a great deal of experience have published in methodology journals

993
00:51:01.409 --> 00:51:05.349 A:middle L:90%
. One of them used the technique that he invented

994
00:51:05.579 --> 00:51:08.780 A:middle L:90%
for doing the data analysis. Um uh, so

995
00:51:08.780 --> 00:51:12.150 A:middle L:90%
they all had a good deal of experience. They

996
00:51:12.150 --> 00:51:15.050 A:middle L:90%
weren't just random, uh, analysts. And also

997
00:51:15.050 --> 00:51:17.400 A:middle L:90%
we had a peer review phase. So everyone did

998
00:51:17.400 --> 00:51:21.429 A:middle L:90%
. An initial analysis sent us the initial analysis with

999
00:51:21.429 --> 00:51:24.780 A:middle L:90%
what transformations. What exclusions? What model they used

1000
00:51:25.159 --> 00:51:29.090 A:middle L:90%
. And we took all that information, removed every

1001
00:51:29.099 --> 00:51:31.239 A:middle L:90%
bit of information about what actual results were observed and

1002
00:51:31.239 --> 00:51:35.980 A:middle L:90%
then made those analysis strategies available to the rest of

1003
00:51:35.980 --> 00:51:37.099 A:middle L:90%
the teams. And everybody had to be a peer

1004
00:51:37.099 --> 00:51:39.909 A:middle L:90%
reviewer for at least three others. And so they

1005
00:51:39.909 --> 00:51:43.780 A:middle L:90%
got a lot of feedback and information exchange about the

1006
00:51:43.780 --> 00:51:46.239 A:middle L:90%
strengths and limitations of their various strategies. And then

1007
00:51:46.239 --> 00:51:49.539 A:middle L:90%
they went back and did their results of the analysis

1008
00:51:49.539 --> 00:51:51.570 A:middle L:90%
again with that feedback. And then these are the

1009
00:51:51.570 --> 00:51:53.469 A:middle L:90%
final things. So if anything were expected a great

1010
00:51:53.469 --> 00:51:58.070 A:middle L:90%
deal of convergence in the analysis approaches, given that

1011
00:51:58.070 --> 00:52:00.400 A:middle L:90%
they all were talking to each other about their analysis

1012
00:52:00.400 --> 00:52:02.000 A:middle L:90%
approaches. And nonetheless, we see a great deal

1013
00:52:02.000 --> 00:52:06.960 A:middle L:90%
of variability and what most papers do. Any single

1014
00:52:06.960 --> 00:52:09.940 A:middle L:90%
paper that you read does one of these. No

1015
00:52:09.940 --> 00:52:12.599 A:middle L:90%
one does 28 of them, because that'd be a

1016
00:52:12.599 --> 00:52:14.920 A:middle L:90%
silly thing to do. But we went ahead and

1017
00:52:14.920 --> 00:52:17.340 A:middle L:90%
tried it, and we're very surprised at how much

1018
00:52:17.340 --> 00:52:22.559 A:middle L:90%
variability we actually observed across the different data. Now

1019
00:52:22.050 --> 00:52:28.380 A:middle L:90%
, analysts suggesting again that the potential value of opening

1020
00:52:28.420 --> 00:52:31.210 A:middle L:90%
materials and making data available gives an opportunity to see

1021
00:52:31.210 --> 00:52:34.880 A:middle L:90%
how the different assumptions that went into the initial test

1022
00:52:35.139 --> 00:52:37.369 A:middle L:90%
might actually be less certain. There is more variability

1023
00:52:37.369 --> 00:52:43.210 A:middle L:90%
to be observed in those potential outcomes for time.

1024
00:52:43.210 --> 00:52:45.090 A:middle L:90%
I think I'm going to skip selective reporting, but

1025
00:52:45.090 --> 00:52:47.219 A:middle L:90%
there is a great paper just came out last month

1026
00:52:47.219 --> 00:52:51.650 A:middle L:90%
in science. I won't explain the details, but

1027
00:52:51.650 --> 00:52:54.480 A:middle L:90%
shows that people that get strong results are very likely

1028
00:52:54.480 --> 00:52:57.760 A:middle L:90%
to write them up to publish. People who get

1029
00:52:57.760 --> 00:53:00.659 A:middle L:90%
negative results are very unlikely to write them up to

1030
00:53:00.659 --> 00:53:02.400 A:middle L:90%
publish, uh, probably presuming that they would never

1031
00:53:02.400 --> 00:53:06.280 A:middle L:90%
make it through the review cycle anyway. And this

1032
00:53:06.280 --> 00:53:09.349 A:middle L:90%
is not a case where the results are Well,

1033
00:53:09.349 --> 00:53:12.789 A:middle L:90%
that was just the first test of some outlandish idea

1034
00:53:12.789 --> 00:53:13.889 A:middle L:90%
, so I didn't get it. So it's not

1035
00:53:13.889 --> 00:53:15.840 A:middle L:90%
worth sharing with the world. This is from tests

1036
00:53:15.840 --> 00:53:20.210 A:middle L:90%
, which is a time series experiments in social sciences

1037
00:53:20.210 --> 00:53:22.210 A:middle L:90%
, which is a peer reviewed process, a very

1038
00:53:22.219 --> 00:53:24.909 A:middle L:90%
high precious sample. And so people usually do this

1039
00:53:24.909 --> 00:53:28.070 A:middle L:90%
as like the fifth study in a line of studies

1040
00:53:28.070 --> 00:53:30.000 A:middle L:90%
in order to get a representative sample. And when

1041
00:53:30.000 --> 00:53:32.039 A:middle L:90%
they're getting negative results, they're just dumping them,

1042
00:53:32.210 --> 00:53:36.250 A:middle L:90%
even though it's a very high effort process. So

1043
00:53:36.250 --> 00:53:39.599 A:middle L:90%
that's a very interesting paper I recommend. The curious

1044
00:53:39.599 --> 00:53:43.000 A:middle L:90%
thing about all of these is that the problems that

1045
00:53:43.000 --> 00:53:45.739 A:middle L:90%
have been identified in this concern about reproducibility writ large

1046
00:53:45.750 --> 00:53:50.070 A:middle L:90%
are things like low power over abundance of positive results

1047
00:53:50.070 --> 00:53:52.510 A:middle L:90%
, ignoring negative, no results, pursuing question where

1048
00:53:52.510 --> 00:53:55.960 A:middle L:90%
research practices, pursuing many different analytic strategies and just

1049
00:53:55.960 --> 00:53:59.639 A:middle L:90%
reporting the one that looks the best. Uh,

1050
00:53:59.650 --> 00:54:02.019 A:middle L:90%
not actually doing replication, because there's no incentive for

1051
00:54:02.019 --> 00:54:05.610 A:middle L:90%
it. Uh, and limitations of Knowles hypothesis,

1052
00:54:05.610 --> 00:54:07.889 A:middle L:90%
significance. Testing. All of those problems are the

1053
00:54:07.900 --> 00:54:13.130 A:middle L:90%
focus of methodologies Attention now, and they're all the

1054
00:54:13.130 --> 00:54:15.139 A:middle L:90%
same ones that were identified in the 19 fifties 19

1055
00:54:15.139 --> 00:54:21.630 A:middle L:90%
seventies by methodologies methodologies. Then simultaneously, the solutions

1056
00:54:21.630 --> 00:54:24.110 A:middle L:90%
that are being suggested now focus on transparency, making

1057
00:54:24.110 --> 00:54:27.989 A:middle L:90%
the process available. Being very clear between what's a

1058
00:54:27.989 --> 00:54:31.739 A:middle L:90%
confirmatory test hypothesis. Testing versus exploratory analysis hypothesis.

1059
00:54:31.739 --> 00:54:36.119 A:middle L:90%
Generating you can't hypothesis, generate and test on the

1060
00:54:36.119 --> 00:54:39.659 A:middle L:90%
same data and doing replication. Those are the methodologies

1061
00:54:39.659 --> 00:54:43.019 A:middle L:90%
claims of what we should do. And they were

1062
00:54:43.019 --> 00:54:44.949 A:middle L:90%
the methodologies claims of what we should do in the

1063
00:54:44.949 --> 00:54:49.280 A:middle L:90%
19 fifties. 1970 So none of this is actually

1064
00:54:49.280 --> 00:54:52.559 A:middle L:90%
new. It's just louder, Uh, and what's

1065
00:54:52.559 --> 00:54:55.750 A:middle L:90%
surprising is with that amount of time having passed,

1066
00:54:55.750 --> 00:54:59.760 A:middle L:90%
why isn't it that things have changed at all?

1067
00:55:00.340 --> 00:55:01.960 A:middle L:90%
And our contention is that it's not that the problems

1068
00:55:01.960 --> 00:55:05.980 A:middle L:90%
are misidentified or the solutions are misidentified? It's that

1069
00:55:05.980 --> 00:55:09.429 A:middle L:90%
there is no not enough attention to implementation of how

1070
00:55:09.429 --> 00:55:13.510 A:middle L:90%
to actually get those solutions into the daily workflow of

1071
00:55:13.510 --> 00:55:15.880 A:middle L:90%
researchers because it doesn't address what we think is the

1072
00:55:15.880 --> 00:55:19.420 A:middle L:90%
key problem, which is. The incentives for my

1073
00:55:19.420 --> 00:55:22.250 A:middle L:90%
success are focused on me getting it published, not

1074
00:55:22.250 --> 00:55:23.400 A:middle L:90%
on me getting it right. If I get it

1075
00:55:23.400 --> 00:55:25.869 A:middle L:90%
published, I get the rewards I'm seeking. I

1076
00:55:25.869 --> 00:55:29.139 A:middle L:90%
get my job, I get to keep my job

1077
00:55:29.150 --> 00:55:31.599 A:middle L:90%
and I get rewards and career aspirations. If it

1078
00:55:31.599 --> 00:55:34.760 A:middle L:90%
turns out not to be right. Well, probably

1079
00:55:34.769 --> 00:55:37.059 A:middle L:90%
it's harder to know because no one ever bothers to

1080
00:55:37.059 --> 00:55:39.559 A:middle L:90%
replicate most of the work. Uh, and that's

1081
00:55:39.559 --> 00:55:43.420 A:middle L:90%
sort of the long tail of yes stuff sometimes doesn't

1082
00:55:43.420 --> 00:55:45.559 A:middle L:90%
work out in the end, but really, the

1083
00:55:45.559 --> 00:55:49.159 A:middle L:90%
immediate incentive is on publication. And so if we're

1084
00:55:49.159 --> 00:55:51.539 A:middle L:90%
going to try to address this challenge, we need

1085
00:55:51.539 --> 00:55:53.039 A:middle L:90%
to have attention to what is the things that are

1086
00:55:53.039 --> 00:55:57.019 A:middle L:90%
driving my behavior as a scientist, even if I

1087
00:55:57.019 --> 00:55:59.650 A:middle L:90%
possess those ideals that I want to do things in

1088
00:55:59.650 --> 00:56:01.340 A:middle L:90%
an open, transparent, reproducible manner, etcetera.

1089
00:56:02.630 --> 00:56:06.679 A:middle L:90%
So there are many challenges that get in the way

1090
00:56:06.679 --> 00:56:08.150 A:middle L:90%
of this and that. But most of them are

1091
00:56:08.150 --> 00:56:10.980 A:middle L:90%
behavioral ones. So I talked already about the perceived

1092
00:56:10.980 --> 00:56:15.510 A:middle L:90%
norms that we think I want to do it that

1093
00:56:15.510 --> 00:56:16.460 A:middle L:90%
way. I mostly do it that way, and

1094
00:56:16.460 --> 00:56:20.699 A:middle L:90%
everybody else doesn't do it that way. Behavior change

1095
00:56:20.699 --> 00:56:23.630 A:middle L:90%
is very difficult in that kind of circumstance. We

1096
00:56:23.630 --> 00:56:28.380 A:middle L:90%
also possess these fantastic frontal lobes that if we have

1097
00:56:28.389 --> 00:56:30.369 A:middle L:90%
flexibility in our analysis, if we have flexibility and

1098
00:56:30.369 --> 00:56:32.469 A:middle L:90%
what it is we report that's coming out from the

1099
00:56:32.469 --> 00:56:36.570 A:middle L:90%
lab, then I'll find really good reasons to think

1100
00:56:36.570 --> 00:56:38.670 A:middle L:90%
that the analysis that looks best is actually the best

1101
00:56:38.670 --> 00:56:42.949 A:middle L:90%
analysis to do. And the studies that worked are

1102
00:56:42.949 --> 00:56:45.019 A:middle L:90%
actually the ones that I did right and those other

1103
00:56:45.019 --> 00:56:47.030 A:middle L:90%
ones that didn't work that showed negative results. I

1104
00:56:47.030 --> 00:56:50.199 A:middle L:90%
totally screwed those up. I shouldn't report them to

1105
00:56:50.199 --> 00:56:52.159 A:middle L:90%
anybody because those were terrible studies. We should have

1106
00:56:52.159 --> 00:56:53.820 A:middle L:90%
never done it in the first place. And so

1107
00:56:53.820 --> 00:56:57.659 A:middle L:90%
we're doing this not necessarily by intention, but because

1108
00:56:57.659 --> 00:57:00.590 A:middle L:90%
we have skin in the game. It matters for

1109
00:57:00.590 --> 00:57:04.340 A:middle L:90%
me to get positive results that are novel and that

1110
00:57:04.340 --> 00:57:06.800 A:middle L:90%
are a clean story, because that's what I'm incentivized

1111
00:57:06.800 --> 00:57:08.730 A:middle L:90%
to get. The best journals want a beautiful,

1112
00:57:08.730 --> 00:57:13.269 A:middle L:90%
positive novel, Clean story. And so the incentives

1113
00:57:13.269 --> 00:57:15.880 A:middle L:90%
drive that. Even though what's happening in my lab

1114
00:57:15.880 --> 00:57:19.130 A:middle L:90%
every day is a mess, we're at the boundaries

1115
00:57:19.130 --> 00:57:21.059 A:middle L:90%
of knowledge. We're pushing out into things we don't

1116
00:57:21.059 --> 00:57:23.269 A:middle L:90%
understand or mostly have ideas that turn out. Nope

1117
00:57:23.269 --> 00:57:24.519 A:middle L:90%
, not at all like that. It looks like

1118
00:57:24.519 --> 00:57:27.250 A:middle L:90%
this. Okay, let's turn that way and try

1119
00:57:27.250 --> 00:57:30.530 A:middle L:90%
that, and we're fumbling around figuring stuff up,

1120
00:57:30.539 --> 00:57:32.929 A:middle L:90%
out, and that's kind of what's great about doing

1121
00:57:32.929 --> 00:57:36.050 A:middle L:90%
it, because we don't understand it. We are

1122
00:57:36.050 --> 00:57:37.980 A:middle L:90%
discovering new things, and then we're getting to a

1123
00:57:37.980 --> 00:57:38.530 A:middle L:90%
place where we think we have some confidence, and

1124
00:57:38.530 --> 00:57:40.750 A:middle L:90%
then we give ourselves a tough test, and sometimes

1125
00:57:42.019 --> 00:57:45.389 A:middle L:90%
it works out. Uh, but we're not incentivized

1126
00:57:45.389 --> 00:57:47.570 A:middle L:90%
to report it that way. And so we don't

1127
00:57:47.579 --> 00:57:51.539 A:middle L:90%
as much. And we do have, uh,

1128
00:57:51.550 --> 00:57:53.820 A:middle L:90%
motivations that might lead us to report it to be

1129
00:57:53.820 --> 00:57:58.510 A:middle L:90%
more beautiful than it actually is. And also,

1130
00:57:58.510 --> 00:58:00.289 A:middle L:90%
we operate with minimal accountability. You don't know what

1131
00:58:00.289 --> 00:58:04.070 A:middle L:90%
happens in my lab. You only see the report

1132
00:58:04.070 --> 00:58:07.210 A:middle L:90%
at the end. And we know very mature literature

1133
00:58:07.210 --> 00:58:09.400 A:middle L:90%
on accountability shows that when you're not being observed,

1134
00:58:09.400 --> 00:58:13.909 A:middle L:90%
you don't take the perspective of an outsider and you

1135
00:58:13.909 --> 00:58:16.199 A:middle L:90%
might make decisions that you wouldn't make. If you

1136
00:58:16.199 --> 00:58:19.679 A:middle L:90%
thought that those decisions can be observed, you might

1137
00:58:19.679 --> 00:58:21.579 A:middle L:90%
even recognize that you're doing it. Which is the

1138
00:58:21.590 --> 00:58:23.679 A:middle L:90%
most pernicious part of that challenge. And then the

1139
00:58:23.679 --> 00:58:27.039 A:middle L:90%
fourth challenges that I'm busy and so are you.

1140
00:58:27.719 --> 00:58:30.210 A:middle L:90%
We have a lot of stuff to do. So

1141
00:58:30.210 --> 00:58:32.400 A:middle L:90%
all of this great aspirational things to be more reproducible

1142
00:58:32.400 --> 00:58:36.309 A:middle L:90%
, to do replications, to be open our things

1143
00:58:36.309 --> 00:58:37.599 A:middle L:90%
, to say, that's more work that's going to

1144
00:58:37.599 --> 00:58:39.440 A:middle L:90%
make it less likely for me to get a job

1145
00:58:39.820 --> 00:58:43.639 A:middle L:90%
. And so, unless we can actually develop technologies

1146
00:58:43.639 --> 00:58:46.559 A:middle L:90%
and tools and incentives that integrate well into that daily

1147
00:58:46.559 --> 00:58:50.030 A:middle L:90%
workflow. We're not likely to actually get any of

1148
00:58:50.030 --> 00:58:53.219 A:middle L:90%
these things done, so I'll just mention to incentive

1149
00:58:53.219 --> 00:58:55.389 A:middle L:90%
approach is very briefly and then stop so that we

1150
00:58:55.389 --> 00:58:58.659 A:middle L:90%
can have a little bit of discussion before we have

1151
00:58:58.659 --> 00:59:00.510 A:middle L:90%
to go. And one of those is a signaling

1152
00:59:00.510 --> 00:59:05.880 A:middle L:90%
approach is because these are valued because openness is valued

1153
00:59:05.880 --> 00:59:09.300 A:middle L:90%
because in some cases, making very clear confirmatory tests

1154
00:59:09.300 --> 00:59:12.309 A:middle L:90%
is something that is valued. Then if you can

1155
00:59:12.309 --> 00:59:15.610 A:middle L:90%
signal when it's happening, then it's a lot easier

1156
00:59:15.619 --> 00:59:17.699 A:middle L:90%
for those norms that people hold in mind to turn

1157
00:59:17.699 --> 00:59:22.780 A:middle L:90%
into daily practices. Uh, and so there's a

1158
00:59:22.789 --> 00:59:24.670 A:middle L:90%
easy way to signal is badges. Uh, so

1159
00:59:24.670 --> 00:59:28.119 A:middle L:90%
we have a open committee. If you want to

1160
00:59:28.119 --> 00:59:29.590 A:middle L:90%
join the committee, you're welcome to do so.

1161
00:59:29.590 --> 00:59:31.840 A:middle L:90%
An open committee that's working has worked on specifications for

1162
00:59:31.840 --> 00:59:35.599 A:middle L:90%
open data, open materials and pre registration. What

1163
00:59:35.599 --> 00:59:37.760 A:middle L:90%
does it take to earn that bad? And journals

1164
00:59:37.769 --> 00:59:43.019 A:middle L:90%
or any other badge issuer can adopt those specifications and

1165
00:59:43.030 --> 00:59:46.550 A:middle L:90%
issue badges for research that's done so. An article

1166
00:59:46.550 --> 00:59:49.960 A:middle L:90%
, for example, psych science has adopted these badges

1167
00:59:49.960 --> 00:59:52.920 A:middle L:90%
. It's a leading empirical Journal and Psychology. And

1168
00:59:52.920 --> 00:59:54.650 A:middle L:90%
if you get trips this week in psychological science,

1169
00:59:54.650 --> 00:59:58.119 A:middle L:90%
it's an email. Each week you might see these

1170
00:59:58.119 --> 01:00:00.579 A:middle L:90%
little badges there for those articles that have met those

1171
01:00:00.579 --> 01:00:04.820 A:middle L:90%
criteria and applied to receive the badge. And so

1172
01:00:04.820 --> 01:00:07.449 A:middle L:90%
the badge badges show that this person has open data

1173
01:00:07.449 --> 01:00:08.630 A:middle L:90%
open materials, and in the article, it provides

1174
01:00:08.630 --> 01:00:12.610 A:middle L:90%
a link to those data materials so you can go

1175
01:00:12.610 --> 01:00:15.239 A:middle L:90%
right to the basis of the figures in the paper

1176
01:00:16.110 --> 01:00:21.550 A:middle L:90%
. Uh, the badges system is also now we

1177
01:00:21.550 --> 01:00:23.570 A:middle L:90%
have a baking process. So the badge itself has

1178
01:00:23.570 --> 01:00:25.949 A:middle L:90%
digital information about who the issuer is, what it's

1179
01:00:25.949 --> 01:00:29.030 A:middle L:90%
issued for the d. O I for what it's

1180
01:00:29.030 --> 01:00:31.239 A:middle L:90%
linked to and can become a link itself rather than

1181
01:00:31.239 --> 01:00:37.070 A:middle L:90%
just an image for connecting you to the materials and

1182
01:00:37.070 --> 01:00:39.579 A:middle L:90%
the data or the preregistration. So that just does

1183
01:00:39.590 --> 01:00:43.260 A:middle L:90%
a very small thing, which is make visible things

1184
01:00:43.260 --> 01:00:45.730 A:middle L:90%
that people are already doing but aren't getting any reward

1185
01:00:45.730 --> 01:00:47.739 A:middle L:90%
for now. So at least gives them some acknowledgement

1186
01:00:49.210 --> 01:00:50.900 A:middle L:90%
. But it does a bigger thing in signaling,

1187
01:00:50.900 --> 01:00:52.630 A:middle L:90%
which is it, says psych. Science cares about

1188
01:00:52.630 --> 01:00:55.739 A:middle L:90%
this, and these people care about it, and

1189
01:00:55.739 --> 01:00:58.860 A:middle L:90%
they were able to do it, and if they

1190
01:00:58.860 --> 01:01:00.329 A:middle L:90%
were able to do it, that must mean that

1191
01:01:00.329 --> 01:01:01.119 A:middle L:90%
it isn't such a barrier to do that. I

1192
01:01:01.119 --> 01:01:04.639 A:middle L:90%
couldn't do it, too, and there is an

1193
01:01:04.639 --> 01:01:07.230 A:middle L:90%
opportunity to then do it. And so norms can

1194
01:01:07.230 --> 01:01:10.239 A:middle L:90%
shift very rapidly if the values are already present.

1195
01:01:10.610 --> 01:01:13.769 A:middle L:90%
If everyone already agrees that those things are good to

1196
01:01:13.769 --> 01:01:15.360 A:middle L:90%
do, then signaling that it's possible to do will

1197
01:01:15.360 --> 01:01:19.960 A:middle L:90%
accelerate the change. At least that's the theory we'll

1198
01:01:19.960 --> 01:01:23.960 A:middle L:90%
see in practice. A second incentive approach to try

1199
01:01:23.960 --> 01:01:28.139 A:middle L:90%
to address some of these challenges is registered reports.

1200
01:01:28.710 --> 01:01:30.559 A:middle L:90%
So here's the cartoon version of how research gets done

1201
01:01:30.559 --> 01:01:32.289 A:middle L:90%
. You design a study, you collect and analyze

1202
01:01:32.289 --> 01:01:34.679 A:middle L:90%
the data, you write up a report and then

1203
01:01:34.679 --> 01:01:37.260 A:middle L:90%
you publish it in a journal. Of course,

1204
01:01:37.260 --> 01:01:39.300 A:middle L:90%
there is this thing peer review that happens after the

1205
01:01:39.300 --> 01:01:43.670 A:middle L:90%
report is written before it is published. But of

1206
01:01:43.670 --> 01:01:45.579 A:middle L:90%
course, in the standard model of peer review,

1207
01:01:45.630 --> 01:01:49.860 A:middle L:90%
the research is done. You've already done everything you

1208
01:01:49.860 --> 01:01:52.420 A:middle L:90%
can to make it as positive as novel and as

1209
01:01:52.420 --> 01:01:55.360 A:middle L:90%
clean as you can do. So all the incentives

1210
01:01:55.369 --> 01:01:59.800 A:middle L:90%
are geared towards making it as beautiful as possible for

1211
01:01:59.800 --> 01:02:01.730 A:middle L:90%
peer review to get it to the promised land published

1212
01:02:02.400 --> 01:02:07.780 A:middle L:90%
, Uh, and rather than having the incentives aligned

1213
01:02:07.780 --> 01:02:10.880 A:middle L:90%
to promoting accuracy and so registered reports just does one

1214
01:02:10.880 --> 01:02:15.219 A:middle L:90%
thing. It's just peer review after the design phase

1215
01:02:15.900 --> 01:02:17.000 A:middle L:90%
. So the goal with pure review in this case

1216
01:02:17.010 --> 01:02:21.360 A:middle L:90%
is that the individual researcher writes up the research question

1217
01:02:21.360 --> 01:02:23.550 A:middle L:90%
they're going to ask and the design to test that

1218
01:02:23.550 --> 01:02:27.650 A:middle L:90%
question, and they have to specify why this is

1219
01:02:27.650 --> 01:02:30.869 A:middle L:90%
important to know. They have to specify why the

1220
01:02:30.869 --> 01:02:32.369 A:middle L:90%
outcome should be known, regardless of what the outcome

1221
01:02:32.369 --> 01:02:35.800 A:middle L:90%
is, because there's a lot of research that we

1222
01:02:35.809 --> 01:02:38.090 A:middle L:90%
would look at in advance of doing it and say

1223
01:02:38.090 --> 01:02:39.679 A:middle L:90%
, If it comes out that way, it's really

1224
01:02:39.679 --> 01:02:42.710 A:middle L:90%
interesting. If it comes out that way, it's

1225
01:02:42.710 --> 01:02:45.510 A:middle L:90%
not at all interesting because it wouldn't it be amazing

1226
01:02:45.510 --> 01:02:50.210 A:middle L:90%
if kind of study and people say when nothing happens

1227
01:02:50.210 --> 01:02:51.630 A:middle L:90%
, Yeah, that would have been amazing. There

1228
01:02:51.630 --> 01:02:52.460 A:middle L:90%
was no way that was going to happen. And

1229
01:02:52.460 --> 01:02:54.059 A:middle L:90%
so if you do peer review on that kind of

1230
01:02:54.059 --> 01:02:57.070 A:middle L:90%
work in advance, then much of it won't get

1231
01:02:57.070 --> 01:02:59.449 A:middle L:90%
through the process. So in those cases it's more

1232
01:02:59.449 --> 01:03:02.670 A:middle L:90%
exploratory at first before moving to this. But for

1233
01:03:02.670 --> 01:03:05.750 A:middle L:90%
many questions, we want to know the answer,

1234
01:03:05.750 --> 01:03:07.510 A:middle L:90%
because the answer is important to know, regardless of

1235
01:03:07.510 --> 01:03:10.139 A:middle L:90%
whether it's a positive or negative result. And so

1236
01:03:10.139 --> 01:03:13.380 A:middle L:90%
you have to make the case to the peer reviewers

1237
01:03:13.650 --> 01:03:15.739 A:middle L:90%
that, in fact, your design is a good

1238
01:03:15.739 --> 01:03:17.619 A:middle L:90%
test of the question. And so all these incentives

1239
01:03:17.619 --> 01:03:21.909 A:middle L:90%
are aligned for the individual researcher to make it the

1240
01:03:21.909 --> 01:03:23.989 A:middle L:90%
best possible test that they can. And for peer

1241
01:03:23.989 --> 01:03:28.719 A:middle L:90%
reviewers, the incentives are an interesting shift, because

1242
01:03:28.780 --> 01:03:30.880 A:middle L:90%
in many of these cases where it's important to know

1243
01:03:30.880 --> 01:03:32.059 A:middle L:90%
the answer one way or another is, it's because

1244
01:03:32.059 --> 01:03:35.949 A:middle L:90%
there is a theoretical debate that there is a camp

1245
01:03:35.960 --> 01:03:38.949 A:middle L:90%
as as if they were camps that are promoting one

1246
01:03:38.949 --> 01:03:40.800 A:middle L:90%
point of view and another camp that promotes another point

1247
01:03:40.800 --> 01:03:45.250 A:middle L:90%
of view in the ordinary process. If you get

1248
01:03:45.250 --> 01:03:47.860 A:middle L:90%
a result that provides support for one camp, then

1249
01:03:47.860 --> 01:03:50.710 A:middle L:90%
they think it's a great paper, and the other

1250
01:03:50.710 --> 01:03:52.820 A:middle L:90%
camp finds lots and lots of problems with it.

1251
01:03:52.599 --> 01:03:54.989 A:middle L:90%
Here. The results are not known, so both

1252
01:03:54.989 --> 01:03:57.849 A:middle L:90%
camps have the same incentive, which is to make

1253
01:03:57.849 --> 01:03:59.929 A:middle L:90%
the test the best possible test that it can be

1254
01:04:00.289 --> 01:04:01.349 A:middle L:90%
. And so the peer review process can in fact

1255
01:04:01.480 --> 01:04:05.090 A:middle L:90%
potentially even just improve the overall design and the diagnostics

1256
01:04:05.090 --> 01:04:08.269 A:middle L:90%
city of the results because, as you may,

1257
01:04:08.269 --> 01:04:11.349 A:middle L:90%
if you've observed this yourself in practice many times where

1258
01:04:11.349 --> 01:04:15.630 A:middle L:90%
these theoretical debates, the actual published literature has to

1259
01:04:15.630 --> 01:04:18.420 A:middle L:90%
literatures that essentially side by side rather than directly addressing

1260
01:04:18.420 --> 01:04:21.630 A:middle L:90%
each other because they're using different enough methodologies that they're

1261
01:04:21.630 --> 01:04:25.550 A:middle L:90%
not actually able to address that where the core contention

1262
01:04:25.550 --> 01:04:27.900 A:middle L:90%
is between the points of view. Whereas if they're

1263
01:04:27.900 --> 01:04:30.260 A:middle L:90%
confronted with that in advance, then they have to

1264
01:04:30.260 --> 01:04:32.630 A:middle L:90%
work through those differences. So this isn't just an

1265
01:04:32.630 --> 01:04:36.550 A:middle L:90%
imagination. There are 13 journals that have adopted this

1266
01:04:38.380 --> 01:04:42.739 A:middle L:90%
in different forms that are trying it out. And

1267
01:04:42.750 --> 01:04:45.610 A:middle L:90%
we are in the May I think of this year

1268
01:04:45.190 --> 01:04:48.460 A:middle L:90%
. We released a special issue of social psychology with

1269
01:04:48.460 --> 01:04:53.230 A:middle L:90%
15 of these registered reports and and I was a

1270
01:04:53.239 --> 01:04:55.420 A:middle L:90%
guest editor for this. And it is an amazing

1271
01:04:55.420 --> 01:04:58.760 A:middle L:90%
process that's very different than our standard peer review to

1272
01:04:58.760 --> 01:05:01.389 A:middle L:90%
have people working through this research before before the results

1273
01:05:01.389 --> 01:05:04.269 A:middle L:90%
are known. So there's lots of interest across a

1274
01:05:04.269 --> 01:05:08.119 A:middle L:90%
variety of different disciplines. The life just announced a

1275
01:05:08.119 --> 01:05:13.030 A:middle L:90%
partnership with us for doing this process for reproducibility project

1276
01:05:13.039 --> 01:05:15.079 A:middle L:90%
of cancer biology, so we're doing replications of 50

1277
01:05:15.079 --> 01:05:19.309 A:middle L:90%
high impact cancer biology studies from 2010 to 2012,

1278
01:05:19.690 --> 01:05:23.179 A:middle L:90%
and all of them are going through peer review in

1279
01:05:23.179 --> 01:05:27.389 A:middle L:90%
advance instead of once the results are known. Okay

1280
01:05:27.400 --> 01:05:29.739 A:middle L:90%
, so I want to bring this to a close

1281
01:05:29.750 --> 01:05:31.550 A:middle L:90%
, since we've been going for long enough and just

1282
01:05:31.550 --> 01:05:35.699 A:middle L:90%
noting that all of these incentive challenges are ones that

1283
01:05:35.699 --> 01:05:39.239 A:middle L:90%
have many different components in them. Uh, it's

1284
01:05:39.239 --> 01:05:44.309 A:middle L:90%
not just the researchers themselves, but and the publishing

1285
01:05:44.389 --> 01:05:46.210 A:middle L:90%
elements, but it's also things like the university systems

1286
01:05:46.210 --> 01:05:48.599 A:middle L:90%
and reward. People can say All this is well

1287
01:05:48.599 --> 01:05:51.849 A:middle L:90%
and good, but my tenure committee wants to see

1288
01:05:51.860 --> 01:05:56.750 A:middle L:90%
X number of articles in X level journals, and

1289
01:05:56.750 --> 01:05:58.659 A:middle L:90%
they grade the journals. And so I have to

1290
01:05:58.889 --> 01:06:00.849 A:middle L:90%
live up to that. Societies play a role and

1291
01:06:00.849 --> 01:06:03.239 A:middle L:90%
what kinds of things that they reinforce. Funders can

1292
01:06:03.239 --> 01:06:06.760 A:middle L:90%
make requests or demands or not to suggest things.

1293
01:06:06.760 --> 01:06:11.369 A:middle L:90%
And so addressing the incentive structures is going to require

1294
01:06:11.369 --> 01:06:14.510 A:middle L:90%
all of these different groups to work interactively and communicate

1295
01:06:14.519 --> 01:06:16.340 A:middle L:90%
on how it is to shift, uh, to

1296
01:06:16.349 --> 01:06:20.199 A:middle L:90%
move towards a more open and reproducible science, and

1297
01:06:20.199 --> 01:06:24.110 A:middle L:90%
that is happening at the top. So you,

1298
01:06:24.119 --> 01:06:27.570 A:middle L:90%
if you are linked to the biomedical or behavioral sciences

1299
01:06:27.570 --> 01:06:29.869 A:middle L:90%
at all. You may have seen NIH making moves

1300
01:06:29.869 --> 01:06:30.920 A:middle L:90%
on this. They've made a few different moves on

1301
01:06:31.059 --> 01:06:34.800 A:middle L:90%
promoting more open data and changing their reporting practices.

1302
01:06:34.800 --> 01:06:36.710 A:middle L:90%
But that's just the beginning. There are many more

1303
01:06:38.079 --> 01:06:42.719 A:middle L:90%
that are in development and underway. They're also doing

1304
01:06:42.719 --> 01:06:45.230 A:middle L:90%
a lot of developing some new training programs. But

1305
01:06:45.230 --> 01:06:48.320 A:middle L:90%
this has also gotten the attention of the White House

1306
01:06:48.780 --> 01:06:51.159 A:middle L:90%
. So this is the least read journal ever The

1307
01:06:51.159 --> 01:06:54.659 A:middle L:90%
Daily Journal of the United States government, Uh,

1308
01:06:54.659 --> 01:06:58.630 A:middle L:90%
and what? Uh, their stuff pushed into that

1309
01:06:58.630 --> 01:07:00.800 A:middle L:90%
journal every day, and there was a 28 points

1310
01:07:00.800 --> 01:07:04.199 A:middle L:90%
or 29 point memo from the Office of Science and

1311
01:07:04.199 --> 01:07:08.619 A:middle L:90%
Technology Policy at the White House, where number 11

1312
01:07:08.630 --> 01:07:12.389 A:middle L:90%
, uh, was given recent evidence of the reproducibility

1313
01:07:12.389 --> 01:07:14.690 A:middle L:90%
of a surprising number of public scientific findings. How

1314
01:07:14.690 --> 01:07:16.130 A:middle L:90%
can the federal government leverage its role to help address

1315
01:07:16.130 --> 01:07:19.829 A:middle L:90%
this problem? And what they are signaling when they

1316
01:07:19.840 --> 01:07:23.619 A:middle L:90%
send something in the Federal Register is that they are

1317
01:07:23.619 --> 01:07:26.750 A:middle L:90%
getting ready to act, and this request for information

1318
01:07:26.750 --> 01:07:28.840 A:middle L:90%
is a public request. And almost no one ever

1319
01:07:28.849 --> 01:07:30.949 A:middle L:90%
responds to these because it's in a journal that no

1320
01:07:30.949 --> 01:07:33.210 A:middle L:90%
one ever reads. But it is the official way

1321
01:07:33.219 --> 01:07:35.400 A:middle L:90%
that the White House signals that it's about to do

1322
01:07:35.400 --> 01:07:40.179 A:middle L:90%
stuff. Uh, and, uh, we were

1323
01:07:40.179 --> 01:07:43.320 A:middle L:90%
chatting with folks there about this, and they said

1324
01:07:43.329 --> 01:07:45.349 A:middle L:90%
this just got released and we said, Okay,

1325
01:07:45.360 --> 01:07:47.610 A:middle L:90%
Uh, and so there's a lot of people responded

1326
01:07:47.619 --> 01:07:49.840 A:middle L:90%
to this particular issue. They had 29 issues and

1327
01:07:49.840 --> 01:07:51.849 A:middle L:90%
, like, 90% of the responses were on this

1328
01:07:51.849 --> 01:07:55.760 A:middle L:90%
one. So they got a lot of feedback on

1329
01:07:55.769 --> 01:07:58.219 A:middle L:90%
reproducibility and what to do about it. But they

1330
01:07:58.219 --> 01:08:00.030 A:middle L:90%
are going to do something about it, and they're

1331
01:08:00.030 --> 01:08:00.570 A:middle L:90%
probably going to do something about it. To the

1332
01:08:00.570 --> 01:08:03.429 A:middle L:90%
29 different federal agencies that fund all of the different

1333
01:08:03.429 --> 01:08:06.429 A:middle L:90%
areas of science and, of course, a challenge

1334
01:08:06.429 --> 01:08:10.570 A:middle L:90%
and then goes without saying that Congress is not now

1335
01:08:10.570 --> 01:08:12.659 A:middle L:90%
interested and well, I've been getting calls from,

1336
01:08:12.670 --> 01:08:19.010 A:middle L:90%
uh, the staff for the Congress's Committee Science Committee

1337
01:08:19.010 --> 01:08:25.779 A:middle L:90%
, Lamar Smith's staffers about what to do about reproducibility

1338
01:08:26.170 --> 01:08:28.739 A:middle L:90%
. And, as you can imagine with other things

1339
01:08:28.750 --> 01:08:30.390 A:middle L:90%
that the federal government has waded into in science,

1340
01:08:30.399 --> 01:08:33.319 A:middle L:90%
if they impose a solution for us, we probably

1341
01:08:33.319 --> 01:08:36.909 A:middle L:90%
won't like it. IRBs, if you have familiar

1342
01:08:36.909 --> 01:08:42.350 A:middle L:90%
with IRBs, are the result of intervention by policy

1343
01:08:42.359 --> 01:08:44.569 A:middle L:90%
, and they've been a nightmare for many years,

1344
01:08:44.569 --> 01:08:45.930 A:middle L:90%
and they're just now starting to turn. There's some

1345
01:08:45.930 --> 01:08:50.279 A:middle L:90%
great move there, Um, but the the key

1346
01:08:50.279 --> 01:08:55.090 A:middle L:90%
for us in terms of collective community action is to

1347
01:08:55.090 --> 01:08:58.710 A:middle L:90%
get in front of these problems before Congress has the

1348
01:08:58.710 --> 01:09:00.439 A:middle L:90%
motivation to act. The interest is there, and

1349
01:09:00.439 --> 01:09:03.569 A:middle L:90%
there is becoming more interest. Uh, but But

1350
01:09:03.569 --> 01:09:08.060 A:middle L:90%
the federal government won't act if it's becoming clear that

1351
01:09:08.069 --> 01:09:11.460 A:middle L:90%
the sciences or whatever community is interested in is taking

1352
01:09:11.460 --> 01:09:13.949 A:middle L:90%
care of the challenges itself. And we can come

1353
01:09:13.949 --> 01:09:15.390 A:middle L:90%
up with much better, much more effective solutions that

1354
01:09:15.399 --> 01:09:18.439 A:middle L:90%
take into account what's actually happening on the ground in

1355
01:09:18.439 --> 01:09:23.340 A:middle L:90%
the sciences. Then a bureaucratic solution is likely able

1356
01:09:23.340 --> 01:09:25.430 A:middle L:90%
to. So to the extent that we can be

1357
01:09:25.430 --> 01:09:28.489 A:middle L:90%
in front of this by actually pursuing as societies,

1358
01:09:28.489 --> 01:09:30.239 A:middle L:90%
funders, institutions and otherwise, the kinds of things

1359
01:09:30.239 --> 01:09:32.399 A:middle L:90%
that we can do to show that we're addressing these

1360
01:09:32.399 --> 01:09:35.840 A:middle L:90%
problems, then the kinds of solutions that are also

1361
01:09:35.840 --> 01:09:41.079 A:middle L:90%
pushed top down might be much more gentle of the

1362
01:09:41.079 --> 01:09:43.949 A:middle L:90%
kind that we would like. So ultimately, our

1363
01:09:43.949 --> 01:09:46.729 A:middle L:90%
goal is to address this gap, that people's goals

1364
01:09:46.739 --> 01:09:49.060 A:middle L:90%
of how they want to be are aligned with how

1365
01:09:49.060 --> 01:09:51.470 A:middle L:90%
they actually get to be and that they can see

1366
01:09:51.470 --> 01:09:54.510 A:middle L:90%
that others are that way too. And that we're

1367
01:09:54.510 --> 01:09:57.970 A:middle L:90%
all just in the business of the largest public works

1368
01:09:57.970 --> 01:10:00.250 A:middle L:90%
project in human history, which is building a body

1369
01:10:00.250 --> 01:10:03.220 A:middle L:90%
of knowledge for the betterment of humankind. So I'm

1370
01:10:03.220 --> 01:10:05.420 A:middle L:90%
going to stop there and thank you for your attention

1371
01:10:05.420 --> 01:10:13.869 A:middle L:90%
in time. Okay? Yeah. Frank. Yeah

1372
01:10:15.159 --> 01:10:17.229 A:middle L:90%
, A couple questions. If there are a couple

1373
01:10:17.229 --> 01:10:21.239 A:middle L:90%
questions I'm happy to answer, how do we handle

1374
01:10:21.239 --> 01:10:25.850 A:middle L:90%
confidentiality of the identity of participants in storing the data

1375
01:10:25.850 --> 01:10:29.159 A:middle L:90%
online? It's a very important issue. Uh,

1376
01:10:29.170 --> 01:10:30.350 A:middle L:90%
and so there's a few different things, uh,

1377
01:10:30.359 --> 01:10:33.569 A:middle L:90%
to address with that one. Is that the You

1378
01:10:33.569 --> 01:10:36.829 A:middle L:90%
can store data, uh, private data and leave

1379
01:10:36.829 --> 01:10:42.260 A:middle L:90%
it private on the on OSF. But there are

1380
01:10:42.260 --> 01:10:45.449 A:middle L:90%
many different repositories that have varying degrees of data security

1381
01:10:45.449 --> 01:10:48.300 A:middle L:90%
standards and approaches, uh, that are going to

1382
01:10:48.300 --> 01:10:51.569 A:middle L:90%
be connected to the OSF, and so the key

1383
01:10:51.569 --> 01:10:54.810 A:middle L:90%
will be matching. Whatever the security needs are for

1384
01:10:54.810 --> 01:10:57.739 A:middle L:90%
one's own data for the kinds of service that are

1385
01:10:57.739 --> 01:11:01.010 A:middle L:90%
storing it. I wouldn't store any actual identifying data

1386
01:11:01.020 --> 01:11:03.119 A:middle L:90%
in a public format at all online. That just

1387
01:11:03.119 --> 01:11:05.470 A:middle L:90%
is a nonstarter. And so what? The thing

1388
01:11:05.470 --> 01:11:09.000 A:middle L:90%
with moving towards openness is that security becomes even more

1389
01:11:09.000 --> 01:11:12.880 A:middle L:90%
important because because errors will occur, people accidentally expose

1390
01:11:12.880 --> 01:11:15.039 A:middle L:90%
information that shouldn't be so. Data verse and a

1391
01:11:15.039 --> 01:11:17.909 A:middle L:90%
couple of other groups are involved in trying to develop

1392
01:11:17.909 --> 01:11:23.460 A:middle L:90%
tools so that you can automatically or facilitate the identification

1393
01:11:23.460 --> 01:11:27.579 A:middle L:90%
of sensitive data and help screen that as a software

1394
01:11:27.579 --> 01:11:30.220 A:middle L:90%
solution, let alone being able to make sure that

1395
01:11:30.220 --> 01:11:31.159 A:middle L:90%
you store it in places where it should be stored

1396
01:11:31.229 --> 01:11:32.869 A:middle L:90%
like that. Are HIPAA compliant, for example,

1397
01:11:34.659 --> 01:11:39.770 A:middle L:90%
other questions. How what's that managed to? This

1398
01:11:39.770 --> 01:11:41.930 A:middle L:90%
is a free service. How have you managed to

1399
01:11:41.939 --> 01:11:45.789 A:middle L:90%
control costs for data stores? Last I heard,

1400
01:11:45.260 --> 01:11:49.020 A:middle L:90%
Yeah, the rest of few years wouldn't offer free

1401
01:11:49.020 --> 01:11:58.739 A:middle L:90%
services coming outside positive scalability. Yeah, so,

1402
01:11:58.750 --> 01:12:00.760 A:middle L:90%
uh, right now we're able to do it because

1403
01:12:00.770 --> 01:12:05.729 A:middle L:90%
the usage doesn't exceed our capacity. Uh, the

1404
01:12:05.739 --> 01:12:08.850 A:middle L:90%
it's a problem that we would like to have,

1405
01:12:08.859 --> 01:12:11.109 A:middle L:90%
which is Oh, man, this is getting so

1406
01:12:11.109 --> 01:12:14.239 A:middle L:90%
much use that it's exceeding our capacity to fund it

1407
01:12:14.250 --> 01:12:18.039 A:middle L:90%
because, um, many repositories are worried about this

1408
01:12:18.039 --> 01:12:20.569 A:middle L:90%
problem of sustainability. How are they going to maintain

1409
01:12:20.579 --> 01:12:24.760 A:middle L:90%
their services? The curious thing is that most repositories

1410
01:12:24.760 --> 01:12:27.699 A:middle L:90%
right now are very under used. And so the

1411
01:12:27.699 --> 01:12:30.100 A:middle L:90%
sustainability is really can we get enough users to justify

1412
01:12:30.109 --> 01:12:33.859 A:middle L:90%
continued investment in these resources? Not that the resources

1413
01:12:33.859 --> 01:12:38.390 A:middle L:90%
aren't available. So for us, the goal is

1414
01:12:38.390 --> 01:12:40.510 A:middle L:90%
to make sure that we actually can make movement on

1415
01:12:40.510 --> 01:12:42.800 A:middle L:90%
the front end of getting it to the point where

1416
01:12:42.800 --> 01:12:45.579 A:middle L:90%
we have so much demand that we need more resources

1417
01:12:45.050 --> 01:12:47.250 A:middle L:90%
. Because if the OSF works funding, it won't

1418
01:12:47.260 --> 01:12:49.560 A:middle L:90%
be a problem, because it's the kind of thing

1419
01:12:49.560 --> 01:12:51.119 A:middle L:90%
that everybody says. Oh yeah, that would be

1420
01:12:51.119 --> 01:12:53.390 A:middle L:90%
a good thing to have. It's not like it

1421
01:12:53.390 --> 01:12:55.399 A:middle L:90%
was an idea unique to us. Many people have

1422
01:12:55.399 --> 01:12:58.100 A:middle L:90%
had this idea. Uh, it's just a matter

1423
01:12:58.100 --> 01:13:00.729 A:middle L:90%
of execution. And if we can get a common

1424
01:13:00.739 --> 01:13:02.710 A:middle L:90%
platform, and for us it's connecting all of these

1425
01:13:02.710 --> 01:13:05.130 A:middle L:90%
repositories together, we don't need to host any of

1426
01:13:05.130 --> 01:13:08.090 A:middle L:90%
the data. We can just connect to all the

1427
01:13:08.090 --> 01:13:11.010 A:middle L:90%
repositories that already exist, and there are many hundreds

1428
01:13:11.020 --> 01:13:14.079 A:middle L:90%
of repositories and we were connected to five of them

1429
01:13:14.649 --> 01:13:16.340 A:middle L:90%
. Um, but those if all those repositories are

1430
01:13:16.340 --> 01:13:20.149 A:middle L:90%
spreading the load, then that helps with sustainability challenges

1431
01:13:20.149 --> 01:13:24.670 A:middle L:90%
, especially if one's have joint agreements to cover each

1432
01:13:24.670 --> 01:13:27.369 A:middle L:90%
other's data. if they go down for some reason

1433
01:13:27.850 --> 01:13:30.760 A:middle L:90%
, and then a longer term solution is torrents.

1434
01:13:30.350 --> 01:13:34.640 A:middle L:90%
So there's many open, uh, storage places,

1435
01:13:34.640 --> 01:13:38.390 A:middle L:90%
and they're all on our individual machines and so bit

1436
01:13:38.390 --> 01:13:41.159 A:middle L:90%
torrent. You may know if you, uh,

1437
01:13:41.550 --> 01:13:44.500 A:middle L:90%
are less than 30 or have Children less than 30

1438
01:13:44.500 --> 01:13:45.979 A:middle L:90%
as the place to get music and videos for free

1439
01:13:46.449 --> 01:13:48.390 A:middle L:90%
, but it can be used for good instead of

1440
01:13:48.390 --> 01:13:53.600 A:middle L:90%
evil, which is a means of connecting All of

1441
01:13:53.600 --> 01:13:56.760 A:middle L:90%
those, uh, is to spread the data around

1442
01:13:56.760 --> 01:14:00.539 A:middle L:90%
too many different sources. And just through knowing where

1443
01:14:00.539 --> 01:14:01.369 A:middle L:90%
all of the bits of data are pulled the data

1444
01:14:01.380 --> 01:14:04.079 A:middle L:90%
so you could host a gigabyte of science and I

1445
01:14:04.079 --> 01:14:05.920 A:middle L:90%
could host a gigabyte of science, and if we

1446
01:14:05.920 --> 01:14:09.989 A:middle L:90%
all are hosting bits of it on our machines than

1447
01:14:10.000 --> 01:14:12.659 A:middle L:90%
the sustainability is a lot easier because many different machines

1448
01:14:12.659 --> 01:14:14.789 A:middle L:90%
could crash. Then you could still recover all of

1449
01:14:14.800 --> 01:14:16.579 A:middle L:90%
the data. That's a much longer term solution and

1450
01:14:16.579 --> 01:14:18.489 A:middle L:90%
not quite the question you asked. But that's sort

1451
01:14:18.489 --> 01:14:23.699 A:middle L:90%
of in the realm. That addressing okay, Anything

1452
01:14:23.699 --> 01:14:27.210 A:middle L:90%
else? Yeah, please. Curious Later. Does

1453
01:14:27.210 --> 01:14:31.010 A:middle L:90%
it support industrial projects? Does it support industrial projects

1454
01:14:31.010 --> 01:14:34.859 A:middle L:90%
and their management? Yes, it's open and free

1455
01:14:34.859 --> 01:14:38.149 A:middle L:90%
for use for any, so, yeah, it

1456
01:14:38.149 --> 01:14:40.609 A:middle L:90%
can be used for any any group or any service

1457
01:14:40.609 --> 01:14:42.460 A:middle L:90%
. It is a free service. Okay, so

1458
01:14:42.470 --> 01:14:45.479 A:middle L:90%
let me sign up there to get this much of

1459
01:14:45.489 --> 01:14:48.069 A:middle L:90%
memories for you. Or is it something like that

1460
01:14:48.079 --> 01:14:50.069 A:middle L:90%
? Right now? There are no limits at all

1461
01:14:50.590 --> 01:14:54.689 A:middle L:90%
. There is an individual file size limit. But

1462
01:14:54.699 --> 01:14:56.789 A:middle L:90%
if you confronted that in what you were trying to

1463
01:14:56.789 --> 01:14:58.520 A:middle L:90%
store, then you would email us and say,

1464
01:14:58.529 --> 01:15:00.250 A:middle L:90%
Can we store this? And if it gets into

1465
01:15:00.250 --> 01:15:02.069 A:middle L:90%
the terabytes, we might say, Oh, we

1466
01:15:02.069 --> 01:15:03.840 A:middle L:90%
need to think about how we're going to do that

1467
01:15:03.850 --> 01:15:06.649 A:middle L:90%
. But outside of that, probably not a problem

1468
01:15:06.649 --> 01:15:10.100 A:middle L:90%
. Now when we get really busy, then it

1469
01:15:10.100 --> 01:15:12.949 A:middle L:90%
might become more challenging. But there are other repositories

1470
01:15:12.960 --> 01:15:14.949 A:middle L:90%
, like data verses, one of our connections.

1471
01:15:15.439 --> 01:15:16.710 A:middle L:90%
Uh, they're based at Harvard and the Institute of

1472
01:15:16.720 --> 01:15:19.880 A:middle L:90%
Quantitative Social Sciences. They are insisting every time I

1473
01:15:19.880 --> 01:15:21.279 A:middle L:90%
ask them, they say No, no, no

1474
01:15:21.289 --> 01:15:25.069 A:middle L:90%
, it's unlimited storage. Well, store whatever I

1475
01:15:25.069 --> 01:15:27.489 A:middle L:90%
say. Yeah, unlimited storage. They say unlimited

1476
01:15:27.489 --> 01:15:29.949 A:middle L:90%
storage. So I'm we're hoping to push them to

1477
01:15:29.949 --> 01:15:30.680 A:middle L:90%
their limit. They say No, no, no

1478
01:15:30.680 --> 01:15:32.770 A:middle L:90%
. That's not what we meant. You said unlimited

1479
01:15:33.239 --> 01:15:36.770 A:middle L:90%
. So we'll see you. There's another hand.

1480
01:15:40.840 --> 01:15:46.649 A:middle L:90%
Yes. Mhm. Sure. Yeah. If you

1481
01:15:46.659 --> 01:15:50.750 A:middle L:90%
were. Oh, were they? I just No

1482
01:15:50.750 --> 01:15:54.100 A:middle L:90%
. From what I know, some signs. Sometimes

1483
01:15:54.109 --> 01:15:57.859 A:middle L:90%
really bitter debate about how things should be he made

1484
01:15:58.539 --> 01:16:04.460 A:middle L:90%
. So when you publicly sports after a year Thanks

1485
01:16:06.430 --> 01:16:11.810 A:middle L:90%
. Yeah, with a words like we're finished.

1486
01:16:12.630 --> 01:16:18.569 A:middle L:90%
Yeah. So what? What arguments? That would

1487
01:16:18.569 --> 01:16:21.880 A:middle L:90%
be great to add because it was so interesting.

1488
01:16:21.890 --> 01:16:25.770 A:middle L:90%
It was done privately as a standard period. We

1489
01:16:25.770 --> 01:16:27.550 A:middle L:90%
only want to push so many boundaries. And we

1490
01:16:27.550 --> 01:16:30.170 A:middle L:90%
didn't push the open review boundary on this, but

1491
01:16:30.170 --> 01:16:32.310 A:middle L:90%
it would have been really interesting to make a lot

1492
01:16:32.310 --> 01:16:36.039 A:middle L:90%
of that discussion available because it is theoretically rich,

1493
01:16:36.050 --> 01:16:38.920 A:middle L:90%
right? Exactly. You're saying it's debates about how

1494
01:16:38.920 --> 01:16:41.829 A:middle L:90%
to even test the question, often become the way

1495
01:16:41.829 --> 01:16:45.250 A:middle L:90%
to identify where the actual rub in the theories are

1496
01:16:45.630 --> 01:16:48.510 A:middle L:90%
Rather than that being sort of the getting just Oh

1497
01:16:48.510 --> 01:16:49.909 A:middle L:90%
, it's the details of how you do it.

1498
01:16:49.920 --> 01:16:51.460 A:middle L:90%
No, no, that's really where it's at.

1499
01:16:51.930 --> 01:16:55.060 A:middle L:90%
So we didn't publish that. We just published it

1500
01:16:55.060 --> 01:16:58.319 A:middle L:90%
like regular reports. And then there are commentaries.

1501
01:16:58.329 --> 01:17:00.859 A:middle L:90%
So the original authors were all invited to write a

1502
01:17:00.859 --> 01:17:03.489 A:middle L:90%
commentary if they wish to on the report. And

1503
01:17:03.489 --> 01:17:05.569 A:middle L:90%
then they if they did write a commentary, the

1504
01:17:05.579 --> 01:17:09.229 A:middle L:90%
replication teams, they were all replications. In this

1505
01:17:09.229 --> 01:17:12.159 A:middle L:90%
case, the out there registered reports. They're all

1506
01:17:12.170 --> 01:17:14.850 A:middle L:90%
trying to redo a study that had been originally public

1507
01:17:15.430 --> 01:17:17.149 A:middle L:90%
. But then the replication teams were able to write

1508
01:17:17.149 --> 01:17:19.250 A:middle L:90%
a reply. But it wasn't quite what you're suggesting

1509
01:17:19.250 --> 01:17:25.390 A:middle L:90%
. Which would be really fun to do between the

1510
01:17:25.399 --> 01:17:28.750 A:middle L:90%
first community and the phone. Who do you think

1511
01:17:29.529 --> 01:17:33.239 A:middle L:90%
for the four coming in my opinion. Researcher.

1512
01:17:33.750 --> 01:17:39.060 A:middle L:90%
I like more or less. Mhm. Very interesting

1513
01:17:40.329 --> 01:17:43.760 A:middle L:90%
. Kind of. All right. No, you're

1514
01:17:43.760 --> 01:17:48.149 A:middle L:90%
a little bit ready. Yeah, Okay. Who

1515
01:17:48.149 --> 01:17:53.409 A:middle L:90%
do you think should the first huge. Well,

1516
01:17:53.420 --> 01:17:56.710 A:middle L:90%
they have different opportunities to make change, right?

1517
01:17:56.710 --> 01:18:00.020 A:middle L:90%
So individual researchers can do things right now, the

1518
01:18:00.029 --> 01:18:02.109 A:middle L:90%
things that individual researchers can do right now is start

1519
01:18:02.109 --> 01:18:05.350 A:middle L:90%
to integrate open and reproducible practices into their workflow.

1520
01:18:05.479 --> 01:18:10.939 A:middle L:90%
And as reviewers ask that of people that they're reviewing

1521
01:18:11.420 --> 01:18:13.699 A:middle L:90%
, uh and so those are the main mechanisms we

1522
01:18:13.699 --> 01:18:15.560 A:middle L:90%
have individually to make change. But we also have

1523
01:18:15.569 --> 01:18:18.850 A:middle L:90%
the pragmatic realities of the world around us. And

1524
01:18:18.850 --> 01:18:23.210 A:middle L:90%
if we ignore those entirely, then we won't survive

1525
01:18:23.210 --> 01:18:26.880 A:middle L:90%
long in the science. Funders can have a big

1526
01:18:26.880 --> 01:18:28.640 A:middle L:90%
lever, which is Do you want the money?

1527
01:18:29.359 --> 01:18:32.390 A:middle L:90%
And that is so change can happen very quickly.

1528
01:18:32.489 --> 01:18:36.039 A:middle L:90%
But funders, especially federal funders, have many different

1529
01:18:36.039 --> 01:18:41.039 A:middle L:90%
constituencies to deal with. Uh, and so it's

1530
01:18:41.039 --> 01:18:43.890 A:middle L:90%
much harder for change to happen, especially at the

1531
01:18:43.890 --> 01:18:46.520 A:middle L:90%
federal level. So we've observed much more, uh

1532
01:18:46.529 --> 01:18:50.750 A:middle L:90%
, adaptability to private foundations because they say, Well

1533
01:18:50.760 --> 01:18:54.170 A:middle L:90%
, we're just going to change our rules and that's

1534
01:18:54.180 --> 01:18:55.890 A:middle L:90%
so if they don't want the money, they don't

1535
01:18:55.890 --> 01:18:57.770 A:middle L:90%
have to have the money. Uh, and so

1536
01:18:57.779 --> 01:19:00.329 A:middle L:90%
many of them are now insisting on more open data

1537
01:19:00.340 --> 01:19:02.329 A:middle L:90%
. Uh, and some of them are starting to

1538
01:19:02.329 --> 01:19:06.569 A:middle L:90%
talk about insisting on preregistration, uh, whether just

1539
01:19:06.579 --> 01:19:10.020 A:middle L:90%
of what the research goals are, let alone whether

1540
01:19:10.020 --> 01:19:13.770 A:middle L:90%
it's confirmatory. Um, but I will say that

1541
01:19:13.770 --> 01:19:17.670 A:middle L:90%
the movement on funders at the federal level is gaining

1542
01:19:17.670 --> 01:19:21.079 A:middle L:90%
steam. Open data requirements. Uh, every agency

1543
01:19:21.079 --> 01:19:23.739 A:middle L:90%
with in the U. S. Every agency with

1544
01:19:23.800 --> 01:19:27.039 A:middle L:90%
$100 million budget or more for research has to have

1545
01:19:27.039 --> 01:19:29.579 A:middle L:90%
an open data plan, and those are just getting

1546
01:19:29.579 --> 01:19:30.899 A:middle L:90%
consolidated. Now. The White House is going to

1547
01:19:30.899 --> 01:19:34.020 A:middle L:90%
move further on that, and many of them will

1548
01:19:34.020 --> 01:19:38.829 A:middle L:90%
start to develop enforcement mechanisms. So the NIH for

1549
01:19:38.829 --> 01:19:40.510 A:middle L:90%
example, is talking about How is it that we

1550
01:19:40.510 --> 01:19:43.130 A:middle L:90%
can actually enforce because they have an open data policy

1551
01:19:43.140 --> 01:19:45.229 A:middle L:90%
, but no one does anything about it. Uh

1552
01:19:45.239 --> 01:19:46.229 A:middle L:90%
, and so it has no impact. Um,

1553
01:19:46.239 --> 01:19:49.829 A:middle L:90%
so it's both probably have to happen, But my

1554
01:19:49.829 --> 01:19:53.270 A:middle L:90%
view, the long solution is the bottom up one

1555
01:19:53.279 --> 01:19:55.510 A:middle L:90%
. It's like I want to integrate my entire workflow

1556
01:19:55.510 --> 01:19:58.939 A:middle L:90%
, not just the stuff that gets public who are

1557
01:19:58.939 --> 01:20:00.619 A:middle L:90%
your competitors. So if I wanted to pre register

1558
01:20:01.210 --> 01:20:05.460 A:middle L:90%
study the only one? No. There are other

1559
01:20:05.470 --> 01:20:09.689 A:middle L:90%
ways to preregister, depending on sub fields. We

1560
01:20:09.689 --> 01:20:12.789 A:middle L:90%
have an interesting position in that we have no monetization

1561
01:20:12.789 --> 01:20:15.149 A:middle L:90%
and no intellectual property, and everything we make is

1562
01:20:15.149 --> 01:20:17.539 A:middle L:90%
freely available. So functionally we have no competitors because

1563
01:20:17.539 --> 01:20:19.390 A:middle L:90%
if somebody else is doing it, then we'll just

1564
01:20:19.390 --> 01:20:20.909 A:middle L:90%
connect to them. That means we don't have to

1565
01:20:20.909 --> 01:20:24.670 A:middle L:90%
build it. Um, so that's really good for

1566
01:20:24.670 --> 01:20:27.220 A:middle L:90%
us. So there are a few other places to

1567
01:20:27.220 --> 01:20:29.569 A:middle L:90%
pre register. There is another sense in which there

1568
01:20:29.569 --> 01:20:32.789 A:middle L:90%
is a competitive landscape to which is there are commercial

1569
01:20:32.789 --> 01:20:35.539 A:middle L:90%
entities. Like if Google decided it wants to do

1570
01:20:35.539 --> 01:20:38.840 A:middle L:90%
this, we'd be gone. Uh, they would

1571
01:20:38.840 --> 01:20:42.489 A:middle L:90%
just do it. Uh, and the risk of

1572
01:20:42.489 --> 01:20:44.949 A:middle L:90%
course, is that the system is closed. Uh

1573
01:20:44.949 --> 01:20:46.510 A:middle L:90%
, and but Google has so much power that everyone

1574
01:20:46.510 --> 01:20:48.289 A:middle L:90%
would start using it because Google is really good at

1575
01:20:48.289 --> 01:20:49.909 A:middle L:90%
doing things, and we'll do it that way.

1576
01:20:49.920 --> 01:20:53.939 A:middle L:90%
Uh, and even though a free, completely open

1577
01:20:53.939 --> 01:20:58.020 A:middle L:90%
system that's ideal idealized, at least not ideal necessarily

1578
01:20:58.029 --> 01:21:00.560 A:middle L:90%
, uh, would wouldn't be able to survive that

1579
01:21:00.569 --> 01:21:03.119 A:middle L:90%
. But there are few organizations for registering. Uh

1580
01:21:03.210 --> 01:21:04.989 A:middle L:90%
, if you do clinical trials research, you have

1581
01:21:04.989 --> 01:21:09.000 A:middle L:90%
to register by law, clinical trials, Gov.

1582
01:21:09.000 --> 01:21:11.069 A:middle L:90%
And a few others. You can register at clinical

1583
01:21:11.069 --> 01:21:14.079 A:middle L:90%
trials, Gov. For other kinds of research.

1584
01:21:14.090 --> 01:21:15.899 A:middle L:90%
But it's still a limited range, and what they

1585
01:21:15.899 --> 01:21:20.199 A:middle L:90%
support economics now has a couple of registries, the

1586
01:21:20.210 --> 01:21:24.500 A:middle L:90%
to sponsors, one in particular. And then there

1587
01:21:24.500 --> 01:21:27.539 A:middle L:90%
are a few more fledgling ones that are very small

1588
01:21:27.909 --> 01:21:30.210 A:middle L:90%
. Ours is a generalist solution, and so we

1589
01:21:30.210 --> 01:21:33.560 A:middle L:90%
are building connections to clinical trials, to the registry

1590
01:21:33.560 --> 01:21:36.560 A:middle L:90%
and to whatever other registries come up, so that

1591
01:21:36.569 --> 01:21:40.289 A:middle L:90%
if you are an economist and you want to be

1592
01:21:40.289 --> 01:21:43.720 A:middle L:90%
able to submit to the registry, you can still

1593
01:21:43.720 --> 01:21:45.439 A:middle L:90%
register in hours and push it into their registry,

1594
01:21:45.810 --> 01:21:48.550 A:middle L:90%
and then you can select from a variety of registries

1595
01:21:48.550 --> 01:21:50.739 A:middle L:90%
if you want to populate them there, it's the

1596
01:21:50.739 --> 01:21:55.300 A:middle L:90%
same with data and other things. Yeah. So

1597
01:21:55.300 --> 01:21:59.939 A:middle L:90%
what? Questions of the registration of the experimental design

1598
01:22:00.100 --> 01:22:03.100 A:middle L:90%
One. How much does that typically cost? That's

1599
01:22:03.100 --> 01:22:09.489 A:middle L:90%
just do that or for individual submitting them. And

1600
01:22:09.489 --> 01:22:13.119 A:middle L:90%
then also mhm. What are the risks for?

1601
01:22:14.100 --> 01:22:15.970 A:middle L:90%
Have you seen this playoff for me? A lot

1602
01:22:15.970 --> 01:22:18.390 A:middle L:90%
of time to I don't want competitors. This is

1603
01:22:18.390 --> 01:22:23.550 A:middle L:90%
your doing too soon process. And that really putting

1604
01:22:23.560 --> 01:22:27.199 A:middle L:90%
your this is that you're asking for people who are

1605
01:22:27.199 --> 01:22:31.079 A:middle L:90%
in a similar field work their concern there. And

1606
01:22:31.079 --> 01:22:34.329 A:middle L:90%
then also what the so many questions, So many

1607
01:22:34.329 --> 01:22:36.439 A:middle L:90%
questions. So let me start with the second one

1608
01:22:36.439 --> 01:22:39.750 A:middle L:90%
because I've already forgotten the first one. Uh and

1609
01:22:39.750 --> 01:22:42.399 A:middle L:90%
then you can repeat the first one. Uh,

1610
01:22:42.409 --> 01:22:45.109 A:middle L:90%
so the challenge of I don't want to make my

1611
01:22:45.109 --> 01:22:46.739 A:middle L:90%
stuff available because I'll get scooped is one that a

1612
01:22:46.739 --> 01:22:49.119 A:middle L:90%
lot of people have trepidation of of going open,

1613
01:22:49.800 --> 01:22:53.479 A:middle L:90%
and I think it's a totally reasonable concern and then

1614
01:22:53.479 --> 01:22:56.010 A:middle L:90%
not actually a concern in practice. Uh, and

1615
01:22:56.010 --> 01:22:58.479 A:middle L:90%
this, you know, an example of this is

1616
01:22:58.489 --> 01:23:00.810 A:middle L:90%
archive a, r x, ivy dot org,

1617
01:23:00.819 --> 01:23:02.359 A:middle L:90%
which is the physics and now spanned out to a

1618
01:23:02.359 --> 01:23:04.590 A:middle L:90%
couple of other journals. It's a manuscript sharing service

1619
01:23:04.619 --> 01:23:09.180 A:middle L:90%
. Most all physicists use, uh, archive to

1620
01:23:09.189 --> 01:23:12.279 A:middle L:90%
share their manuscripts as soon as they finished writing them

1621
01:23:12.430 --> 01:23:15.729 A:middle L:90%
and then make them available early on. Everybody in

1622
01:23:15.739 --> 01:23:16.720 A:middle L:90%
physics said, there's no way I'm gonna use that

1623
01:23:16.729 --> 01:23:20.010 A:middle L:90%
because people will scoop my idea and do a real

1624
01:23:20.010 --> 01:23:21.350 A:middle L:90%
quick study. Do a crap. You get it

1625
01:23:21.350 --> 01:23:25.869 A:middle L:90%
published before my careful study gets through the publication pipeline

1626
01:23:26.170 --> 01:23:28.869 A:middle L:90%
and then I'll have been scooped. But then they

1627
01:23:28.880 --> 01:23:30.899 A:middle L:90%
rapidly realized that posted an archive. Actually, his

1628
01:23:30.899 --> 01:23:34.239 A:middle L:90%
first establishes precedents. Oh, they're the ones that

1629
01:23:34.239 --> 01:23:36.170 A:middle L:90%
posted it there. So the first one to make

1630
01:23:36.170 --> 01:23:39.260 A:middle L:90%
it available, posted. I think the same will

1631
01:23:39.260 --> 01:23:41.720 A:middle L:90%
occur across the entire life cycle, right? As

1632
01:23:41.720 --> 01:23:43.760 A:middle L:90%
soon as you make your stuff available, you've marked

1633
01:23:43.770 --> 01:23:45.810 A:middle L:90%
. I'm working on this and really it turns into

1634
01:23:45.810 --> 01:23:48.630 A:middle L:90%
execution. Ideas are cheap, so I'm going to

1635
01:23:48.630 --> 01:23:50.699 A:middle L:90%
put every single idea I've ever had online, and

1636
01:23:50.699 --> 01:23:53.600 A:middle L:90%
then I'll get the victory. For all of those

1637
01:23:53.600 --> 01:23:55.829 A:middle L:90%
ideas. Lots of other people have had those same

1638
01:23:55.829 --> 01:23:58.720 A:middle L:90%
ideas. It's really on actually getting the work work

1639
01:23:58.720 --> 01:24:00.819 A:middle L:90%
done, So that's one way to answer it.

1640
01:24:00.829 --> 01:24:01.680 A:middle L:90%
But a second way to answer, which is No

1641
01:24:01.680 --> 01:24:04.090 A:middle L:90%
, you're right. So what you can do is

1642
01:24:04.100 --> 01:24:10.510 A:middle L:90%
on OSF is registered to keep it private and simultaneously

1643
01:24:10.510 --> 01:24:12.439 A:middle L:90%
with the with the peer review process. It doesn't

1644
01:24:12.439 --> 01:24:14.939 A:middle L:90%
completely solve it because they have to. Other people

1645
01:24:14.939 --> 01:24:16.520 A:middle L:90%
have to read it, but we actually have implemented

1646
01:24:16.520 --> 01:24:18.619 A:middle L:90%
. So you can have double blind review even of

1647
01:24:18.619 --> 01:24:21.859 A:middle L:90%
registrations or of studies while making your data and materials

1648
01:24:21.859 --> 01:24:25.720 A:middle L:90%
available so you can send a view only link and

1649
01:24:25.720 --> 01:24:29.439 A:middle L:90%
anonymized it of your OSF project for peer review,

1650
01:24:29.449 --> 01:24:33.729 A:middle L:90%
whether pre or post and as blind as your manuscript

1651
01:24:33.729 --> 01:24:36.560 A:middle L:90%
is blind, uh, so that you can protect

1652
01:24:36.560 --> 01:24:39.739 A:middle L:90%
at least identity. But you do run the risk

1653
01:24:39.739 --> 01:24:41.869 A:middle L:90%
of you've now made your material available so someone else

1654
01:24:41.869 --> 01:24:43.359 A:middle L:90%
could see it and say, Oh, I'm going

1655
01:24:43.359 --> 01:24:45.819 A:middle L:90%
to do that now. The reality is, most

1656
01:24:45.819 --> 01:24:46.579 A:middle L:90%
people are already busy enough with the things are doing

1657
01:24:46.579 --> 01:24:49.829 A:middle L:90%
themselves to go stealing other ideas, uh, that

1658
01:24:49.829 --> 01:24:53.329 A:middle L:90%
are already written up. And so the person is

1659
01:24:53.329 --> 01:24:54.970 A:middle L:90%
presumably already going to do them. So I have

1660
01:24:54.970 --> 01:24:57.119 A:middle L:90%
to work really fast to steal your idea rather than

1661
01:24:57.119 --> 01:25:00.220 A:middle L:90%
just keep working on working on. But it's a

1662
01:25:00.220 --> 01:25:01.409 A:middle L:90%
concern that won't go away immediately. People have.

1663
01:25:01.989 --> 01:25:06.199 A:middle L:90%
So what was the first question? Cost anything?

1664
01:25:06.590 --> 01:25:10.420 A:middle L:90%
I remembered it before you said so. No,

1665
01:25:10.420 --> 01:25:12.810 A:middle L:90%
it doesn't cost anything. So it's just a peer

1666
01:25:12.810 --> 01:25:15.029 A:middle L:90%
review process in advance. It doesn't get published.

1667
01:25:15.029 --> 01:25:16.289 A:middle L:90%
And so all of all the ones that are doing

1668
01:25:16.289 --> 01:25:19.060 A:middle L:90%
it so far there's There's no charge for the register

1669
01:25:21.390 --> 01:25:24.760 A:middle L:90%
. Yeah, sort of make use of this system

1670
01:25:24.760 --> 01:25:28.399 A:middle L:90%
all the way through the research process and to get

1671
01:25:28.399 --> 01:25:30.210 A:middle L:90%
all the data together. And they decided to submit

1672
01:25:30.220 --> 01:25:32.220 A:middle L:90%
to a journal. That's not open access, are

1673
01:25:32.220 --> 01:25:34.239 A:middle L:90%
you? Is there any concern with any kind of

1674
01:25:35.189 --> 01:25:41.939 A:middle L:90%
function of having data Open access is published, whose

1675
01:25:41.939 --> 01:25:45.869 A:middle L:90%
journals like, which is most every journal? Uh

1676
01:25:45.880 --> 01:25:47.239 A:middle L:90%
, no, there aren't any legal ramifications because even

1677
01:25:47.239 --> 01:25:50.840 A:middle L:90%
some of those journals do have open data encouragement or

1678
01:25:50.840 --> 01:25:55.500 A:middle L:90%
requirement, and more of them are looking to do

1679
01:25:55.500 --> 01:25:58.189 A:middle L:90%
that. Of course you know, making it all

1680
01:25:58.189 --> 01:26:01.250 A:middle L:90%
open is an ideal, but that isn't the present

1681
01:26:01.250 --> 01:26:06.250 A:middle L:90%
reality. And so we have lots of developing partnerships

1682
01:26:06.250 --> 01:26:11.090 A:middle L:90%
with commercial publishers that have mostly closed access work to

1683
01:26:11.090 --> 01:26:13.210 A:middle L:90%
open as much of the process as as possible,

1684
01:26:13.579 --> 01:26:15.140 A:middle L:90%
and they're not going to give up subscriptions easily.

1685
01:26:15.340 --> 01:26:17.850 A:middle L:90%
So that part may not be able to convince them

1686
01:26:17.850 --> 01:26:19.750 A:middle L:90%
to change, but we may not have to convince

1687
01:26:19.750 --> 01:26:23.430 A:middle L:90%
them. The reality may overwhelm that. The world

1688
01:26:23.430 --> 01:26:25.399 A:middle L:90%
is moving open and they just have to shift there

1689
01:26:25.779 --> 01:26:29.680 A:middle L:90%
funding model in order to accommodate but making all of

1690
01:26:29.680 --> 01:26:32.369 A:middle L:90%
the data and methods open. And then the manuscript

1691
01:26:32.369 --> 01:26:34.949 A:middle L:90%
itself, functionally, is going to be making it

1692
01:26:34.949 --> 01:26:40.960 A:middle L:90%
all open. So it moves very quickly. Is

1693
01:26:40.960 --> 01:26:49.609 A:middle L:90%
that decrease? Someone's motivations subscribed those terms. Uh

1694
01:26:49.619 --> 01:26:53.340 A:middle L:90%
, so the problem of subscription is not going to

1695
01:26:53.340 --> 01:26:57.869 A:middle L:90%
be solved easily. Uh, unless scientists act collectively

1696
01:26:57.880 --> 01:27:00.880 A:middle L:90%
to solve it. Because right now what happens is

1697
01:27:00.079 --> 01:27:03.079 A:middle L:90%
I pay zero attention to how much any journal costs

1698
01:27:03.079 --> 01:27:08.449 A:middle L:90%
my university to subscribe to. And I demand that

1699
01:27:08.449 --> 01:27:11.609 A:middle L:90%
they give me access to all of these journals that

1700
01:27:11.609 --> 01:27:14.159 A:middle L:90%
I need to do my research. Uh, and

1701
01:27:14.159 --> 01:27:15.340 A:middle L:90%
so the cost is completely opaque to me. It's

1702
01:27:15.350 --> 01:27:17.960 A:middle L:90%
It's totally irrelevant to me. Uh, although it

1703
01:27:17.960 --> 01:27:19.800 A:middle L:90%
is highly relevant to me because it has a cost

1704
01:27:19.800 --> 01:27:23.430 A:middle L:90%
to the university. It's a big burden. Um

1705
01:27:23.439 --> 01:27:26.890 A:middle L:90%
, so if if the scientists who are at both

1706
01:27:26.890 --> 01:27:28.829 A:middle L:90%
ends of it are giving away their scholarship and then

1707
01:27:28.829 --> 01:27:32.619 A:middle L:90%
demanding their university by a back, aren't aware of

1708
01:27:32.619 --> 01:27:34.949 A:middle L:90%
that. And then don't actually connect the dots in

1709
01:27:34.949 --> 01:27:36.329 A:middle L:90%
order to say, You know what? This is

1710
01:27:36.329 --> 01:27:40.949 A:middle L:90%
ridiculous. What we should do is all just decide

1711
01:27:40.960 --> 01:27:42.979 A:middle L:90%
. I've been told this is anti trust. So

1712
01:27:42.989 --> 01:27:45.060 A:middle L:90%
maybe breaking the law by suggesting it. But what

1713
01:27:45.060 --> 01:27:48.659 A:middle L:90%
we should do is decide that 200 top universities in

1714
01:27:48.659 --> 01:27:53.920 A:middle L:90%
USA on January 1st 2017. We are cancelling all

1715
01:27:53.920 --> 01:27:58.899 A:middle L:90%
of our subscriptions. So everybody figured out we're doing

1716
01:27:58.899 --> 01:28:00.579 A:middle L:90%
on that day. We're canceling all of our descriptions

1717
01:28:00.579 --> 01:28:02.619 A:middle L:90%
. We can't afford it. And then the scientists

1718
01:28:02.619 --> 01:28:04.520 A:middle L:90%
and societies would say, Okay, well, we're

1719
01:28:04.520 --> 01:28:06.600 A:middle L:90%
going to do. There are options. There's already

1720
01:28:06.600 --> 01:28:10.789 A:middle L:90%
infrastructure to support it. And we just we just

1721
01:28:10.789 --> 01:28:13.189 A:middle L:90%
pick up the editorial boards and then move them over

1722
01:28:13.189 --> 01:28:14.880 A:middle L:90%
here, give it a new name. It's the

1723
01:28:14.880 --> 01:28:16.609 A:middle L:90%
same journal. We're doing all that work anyway.

1724
01:28:16.619 --> 01:28:18.319 A:middle L:90%
So that would be the way to do it.

1725
01:28:18.319 --> 01:28:21.850 A:middle L:90%
Is the coordinate solve the coordination problem? Going to

1726
01:28:21.850 --> 01:28:27.000 A:middle L:90%
be hard to do outside of we can do it

1727
01:28:27.869 --> 01:28:28.619 A:middle L:90%
. Oh, no. I have to break the

1728
01:28:28.619 --> 01:28:31.350 A:middle L:90%
law. I don't know. Yeah. Fergie,

1729
01:28:31.350 --> 01:28:36.449 A:middle L:90%
check. There's a certain amount of certain torch to

1730
01:28:36.460 --> 01:28:45.029 A:middle L:90%
increase. Yeah, Parliament as well So how how

1731
01:28:45.029 --> 01:28:50.689 A:middle L:90%
do you speak? Mhm potential. Okay, science

1732
01:28:50.689 --> 01:28:57.100 A:middle L:90%
, since special funds for research. Yeah, Okay

1733
01:28:59.369 --> 01:29:01.659 A:middle L:90%
. Uh, at least from the experiences that I've

1734
01:29:01.659 --> 01:29:03.979 A:middle L:90%
had in this so far, which are just anecdotal

1735
01:29:03.989 --> 01:29:06.500 A:middle L:90%
, uh, is that it can accelerate the movement

1736
01:29:06.510 --> 01:29:10.779 A:middle L:90%
for open science because private funders tend to be more

1737
01:29:10.779 --> 01:29:13.869 A:middle L:90%
facile, uh, and able to make decisions on

1738
01:29:13.880 --> 01:29:16.649 A:middle L:90%
getting, uh, prioritizing return on their investment,

1739
01:29:16.659 --> 01:29:19.539 A:middle L:90%
Uh, in a way that federal government may want

1740
01:29:19.539 --> 01:29:23.460 A:middle L:90%
to do but has a much harder challenge and actually

1741
01:29:23.460 --> 01:29:27.640 A:middle L:90%
changing policy. So we've had conversations with a number

1742
01:29:27.640 --> 01:29:30.039 A:middle L:90%
of funders Just said this is going to be our

1743
01:29:30.039 --> 01:29:30.630 A:middle L:90%
new policy, and they go and do it like

1744
01:29:30.640 --> 01:29:32.409 A:middle L:90%
, welcome. Trust in the U. K.

1745
01:29:32.409 --> 01:29:35.029 A:middle L:90%
Has changed a lot of policies on open data,

1746
01:29:35.029 --> 01:29:38.869 A:middle L:90%
right. They now mandate that everything that they fund

1747
01:29:38.869 --> 01:29:40.979 A:middle L:90%
be published open access, uh, and they'll pay

1748
01:29:40.979 --> 01:29:43.930 A:middle L:90%
for it. Um, if it's not published in

1749
01:29:43.939 --> 01:29:45.930 A:middle L:90%
the Life, which is the free open access journal

1750
01:29:45.930 --> 01:29:48.520 A:middle L:90%
, which they are underwriting the costs for So private

1751
01:29:48.520 --> 01:29:53.949 A:middle L:90%
funders can move this very quickly where the federal government

1752
01:29:53.949 --> 01:29:58.300 A:middle L:90%
can't as easily. So I guess I I spoke

1753
01:29:58.300 --> 01:30:02.079 A:middle L:90%
to generally, I guess, like slowly never.

1754
01:30:02.460 --> 01:30:06.579 A:middle L:90%
But I guess I'm very multi pharmaceutical firms. Okay

1755
01:30:06.590 --> 01:30:10.810 A:middle L:90%
. Okay. Yes, yes, that's different.

1756
01:30:10.869 --> 01:30:15.119 A:middle L:90%
So there's, um And then it also bridges into

1757
01:30:15.130 --> 01:30:17.649 A:middle L:90%
other kinds of intellectual property concerns where patenting is a

1758
01:30:17.649 --> 01:30:19.810 A:middle L:90%
prairie. Right? So, in engineering, uh

1759
01:30:19.819 --> 01:30:23.689 A:middle L:90%
, there is a lot more interest and divide and

1760
01:30:23.699 --> 01:30:26.939 A:middle L:90%
individual labs and across labs, and whether we're building

1761
01:30:26.939 --> 01:30:30.880 A:middle L:90%
for patent purposes, commercialization or whether we're building for

1762
01:30:30.960 --> 01:30:35.220 A:middle L:90%
knowledge for knowledge's sake and those what the open science

1763
01:30:35.220 --> 01:30:38.770 A:middle L:90%
movement will do is just make very clear which is

1764
01:30:38.770 --> 01:30:43.159 A:middle L:90%
which it's. There's lots of good reasons to its

1765
01:30:43.159 --> 01:30:46.609 A:middle L:90%
promote commercialization to have companies and organizations investing in research

1766
01:30:46.619 --> 01:30:49.149 A:middle L:90%
in order to build new products and tools and technologies

1767
01:30:49.149 --> 01:30:53.180 A:middle L:90%
to sell. Um, but it should be very

1768
01:30:53.180 --> 01:30:57.560 A:middle L:90%
clear which purposes for and the private purpose it will

1769
01:30:57.560 --> 01:31:00.119 A:middle L:90%
be very clearly stated. This is private purpose.

1770
01:31:00.119 --> 01:31:03.100 A:middle L:90%
Were actually pursuing knowledge for our game that we hope

1771
01:31:03.100 --> 01:31:04.569 A:middle L:90%
eventually will be your game, too, because you'll

1772
01:31:04.569 --> 01:31:06.909 A:middle L:90%
buy our stuff and like it. But really,

1773
01:31:06.909 --> 01:31:15.180 A:middle L:90%
it's ours first rather than academic science. Yes.

1774
01:31:15.060 --> 01:31:19.810 A:middle L:90%
Are you really as it value? Remove the peer

1775
01:31:19.810 --> 01:31:24.380 A:middle L:90%
review process from later stages, or you're just adding

1776
01:31:25.060 --> 01:31:29.310 A:middle L:90%
to the earliest. Yeah, So there's a number

1777
01:31:29.310 --> 01:31:30.380 A:middle L:90%
of different ways. You know, my I wrote

1778
01:31:30.390 --> 01:31:34.239 A:middle L:90%
one paper called Opening Scientific Communication, which sort of

1779
01:31:34.250 --> 01:31:38.819 A:middle L:90%
outlines sort of that are, you know, imagined

1780
01:31:38.819 --> 01:31:41.109 A:middle L:90%
utopia of how this process would work, and it

1781
01:31:41.119 --> 01:31:44.020 A:middle L:90%
really just opens the process so that peer review is

1782
01:31:44.020 --> 01:31:45.680 A:middle L:90%
continuous, which is what it is in reality,

1783
01:31:45.760 --> 01:31:47.930 A:middle L:90%
right? Because we read stuff in our lab that

1784
01:31:47.930 --> 01:31:50.000 A:middle L:90%
was published 20 years ago, and we decide whether

1785
01:31:50.000 --> 01:31:51.649 A:middle L:90%
we take it seriously or not what we want to

1786
01:31:51.649 --> 01:31:55.069 A:middle L:90%
do with it. We are acting as peer reviewers

1787
01:31:55.069 --> 01:31:57.069 A:middle L:90%
in the application of our own research to do new

1788
01:31:57.069 --> 01:32:00.380 A:middle L:90%
things with it. None of that is accessible to

1789
01:32:00.380 --> 01:32:01.909 A:middle L:90%
anybody else except for us. So if we can

1790
01:32:01.909 --> 01:32:05.409 A:middle L:90%
move that process of what is the natural evaluation of

1791
01:32:05.409 --> 01:32:11.250 A:middle L:90%
research for research purposes to a public discussion, then

1792
01:32:11.260 --> 01:32:14.029 A:middle L:90%
I think we will meet better and more efficiently that

1793
01:32:14.039 --> 01:32:16.460 A:middle L:90%
ongoing peer review some things that are taken very seriously

1794
01:32:16.460 --> 01:32:19.069 A:middle L:90%
at first. Over time, I'm not taken seriously

1795
01:32:19.069 --> 01:32:21.479 A:middle L:90%
at all, and vice versa. Right plate tectonics

1796
01:32:21.479 --> 01:32:25.489 A:middle L:90%
theory. Everybody said this is absurd and then Nope

1797
01:32:25.500 --> 01:32:28.279 A:middle L:90%
, that's how it works. So there's lots of

1798
01:32:28.279 --> 01:32:30.260 A:middle L:90%
things that happen in evaluation and change over time of

1799
01:32:30.260 --> 01:32:32.880 A:middle L:90%
the same stuff, and that should be part of

1800
01:32:35.350 --> 01:32:39.899 A:middle L:90%
Yeah, um, is what you just said Something

1801
01:32:39.899 --> 01:32:44.439 A:middle L:90%
about You have a paper online. It's 20 years

1802
01:32:44.449 --> 01:32:45.789 A:middle L:90%
, 20 years old, and you want to see

1803
01:32:45.789 --> 01:32:48.189 A:middle L:90%
if it's use formula and you can then see the

1804
01:32:48.189 --> 01:32:55.760 A:middle L:90%
comment. Okay? Yeah. Yes, here with

1805
01:32:56.850 --> 01:33:00.979 A:middle L:90%
years. So this is like, it's open evaluation

1806
01:33:00.979 --> 01:33:03.229 A:middle L:90%
. That's ongoing evaluation, right? You can imagine

1807
01:33:03.229 --> 01:33:09.369 A:middle L:90%
an entire enriched service. Uh, that is just

1808
01:33:09.380 --> 01:33:13.270 A:middle L:90%
commenting and discussion on articles on claims. Uh,

1809
01:33:13.279 --> 01:33:15.470 A:middle L:90%
that also shows its evolution over time. Right?

1810
01:33:15.470 --> 01:33:19.210 A:middle L:90%
So representing the peer review process on a timeline,

1811
01:33:19.220 --> 01:33:21.109 A:middle L:90%
right? Make it simple, right? Make it

1812
01:33:21.109 --> 01:33:24.600 A:middle L:90%
yelp ratings. Everybody gives a star rating, uh

1813
01:33:24.609 --> 01:33:27.680 A:middle L:90%
, to each article. And then you look over

1814
01:33:27.680 --> 01:33:30.329 A:middle L:90%
time. How many stars does it get? You

1815
01:33:30.329 --> 01:33:31.539 A:middle L:90%
know, is it five stars in 2000 and seven

1816
01:33:31.539 --> 01:33:32.979 A:middle L:90%
, but then down to a three star and then

1817
01:33:32.979 --> 01:33:38.649 A:middle L:90%
back up? That's not how it's going. Uh

1818
01:33:38.659 --> 01:33:41.779 A:middle L:90%
, so we have a commenting engine. That's a

1819
01:33:41.789 --> 01:33:45.699 A:middle L:90%
very early commenting engine in OSF, uh, for

1820
01:33:45.699 --> 01:33:50.380 A:middle L:90%
OSF projects. Uh, so but we're ingesting now

1821
01:33:50.390 --> 01:33:54.539 A:middle L:90%
everything through a notification service. So l s f

1822
01:33:54.539 --> 01:33:57.979 A:middle L:90%
should be able to ingest all of science, uh

1823
01:33:57.979 --> 01:34:00.569 A:middle L:90%
, and then represent that for open evaluation of it

1824
01:34:00.949 --> 01:34:03.609 A:middle L:90%
. Right. Pub Med Central also has a commenting

1825
01:34:03.609 --> 01:34:05.250 A:middle L:90%
in so many other places. It doesn't need to

1826
01:34:05.250 --> 01:34:08.029 A:middle L:90%
be us. Uh, we'll link to all of

1827
01:34:08.029 --> 01:34:10.670 A:middle L:90%
those and try to integrate them together so that wherever

1828
01:34:10.670 --> 01:34:13.010 A:middle L:90%
you do your commenting, uh, there can be

1829
01:34:13.020 --> 01:34:15.229 A:middle L:90%
a place in which that all gets consolidated or at

1830
01:34:15.229 --> 01:34:23.229 A:middle L:90%
least connected. I've exhausted you. Okay. Thank

1831
01:34:23.229 --> 01:34:28.680 A:middle L:90%
you for your time. Appreciate a few stickers if

1832
01:34:28.680 --> 01:34:30.369 A:middle L:90%
you want stickers.