So thank you all for coming, we very much appreciate it.
I'm Tyler Walters, I'm the dean of your university
libraries, and we're here to kick off Open Access
Week this week in October. I've got several announcements
We want to mention and at the very top of it
we want to recognize our sponsors for this opening keynote
address this evening. And those folks would be Department
of Psychology, College of Science and University Libraries.
There's also a committee that's put together all of
the events on the schedule, that is at the back of
the room, this week for Open Access Week. So
I'd just like to quickly mention those folks who worked on
that. They are Nathan Hall, Gail McMillan, Ginny
Pannabecker, Patrick Tomlin, Anita Walz, Kimberli
Weeks and Philip Young. So maybe you guys could
raise your hands or something. Well, thank you
Okay. So you should know tonight's
keynote is one of eight events that's happening this week
across campus. Again, there's more information on the table
behind you to find these things out. There's two
events I really would like to highlight, in addition
to the keynote address tonight. Tomorrow evening there is
a faculty and graduate student panel that will discuss open
access experiences and issues. That's gonna get started at
5:30 PM in Newman Library in the Multipurpose room. That's
on the first floor behind the cafe. And Thursday at
three o'clock in the afternoon, the Center for Open
Science, which our keynote speaker, Brian Nosek,
co directs, they're going to be leading a
workshop on reproducible research practices, which will also be
in the library's Multipurpose room. On the other side of
the hand out that that you have available today,
there's information about some of the open
related and open supporting services that we have in the library.
We do things like manage and maintain the university's
repository of scholarship, VTechWorks. We also provide an open access
fund to help fund all of you authors who want
to get your works published in an open access journal.
We also offer our own publishing services. There
are several journals that the library and university have supported
for many years. We also have more recently gotten
involved as well, with data consulting, helping researchers
to produce and share their data and make that open
and sharable as well. So we do
an awful lot around this this framework of research
data management. Also on the back table is a sign
up sheet. We'd like to just know who came.
We really appreciate your attendance. There's also a form
in the back to sign up for an NLI credit,
if you would like to receive that.
There's also sign up list for the Open at VT
listserv if you're interested, and evaluation forms,
of course, we want you to hear your feedback
from today and throughout. Also to inaugurate
the week we have awarded two scholarships during Open
Access Week for the OpenCon 2014 Conference.
This is a sponsorship between University Libraries and the Graduate
School. So OpenCon is an international conference for
students and early career researchers on open access, open
data and open educational resources. It's going to be
held in Washington, D. C. November 15th
to the 17th. We did receive, this the first
time we've done this. We received a very strong
pool of candidates. And the winners, based
on their essays, are Jesse Gunter, masters student in
Public Health, and Mohammed Seyam, a doctoral student
in computer science. Maybe if Jesse and Mohammed are here,
if you could stand for just a minute.
We'll give you a hand, if you're here.
OK, they're late. Or bashful.
With that, we will turn it over soon
to our keynote speaker, who I will introduce now,
is Professor Brian Nosek from the University of Virginia.
Brian received his PhD at Yale University
in 2002. As I said, Professor Nosek
is in the Department of Psychology at the University of
Virginia. He received an Early Career award from the International
Social Cognition Network and the Society for the Psychological Study
of Social Issues. Early on, he co-founded
Project Implicit, which is an internet based, multi
university collaboration of research and education about thoughts and feelings
that exist outside of our awareness or control. Recently
Nosek co-founded and directs The Center for Open
Science, which aims to increase openness, integrity and
reproducibility of scientific research. COS is a nonprofit technology startup
with three primary activities. They are building and maintaining
the open science framework that supports the research workflow and
enables transparency. And the research process includes archiving and
pre registration also community building and shifting incentives such as
badges for articles to acknowledge open practices and conducting meta
science, such as estimating the reproducibility of scientific research
by conducting large scale crowdsourced replication.
Now we will turn it over to Professor Nosek whose topic
is improving the openness and reproducibility of research. So
please welcome Brian Nosek.
Thanks very much for coming and for having me here.
I'm very glad to be able to speak here
today. My area of research expertise in
the psychology department is the gap between values and practices,
what we think we should do, what we're trying
to do, what we want to do, versus what
we actually do. And most of that work is
on implicit biases and how we're influenced in our social
judgment and everyday behavior. But the work that I
want to talk about today is an application of that
general interest on the gap between scientific values and scientific
practices. What science aims to be as a discipline
of trying to accumulate knowledge and then how it actually
works in the everyday life of a researcher and how
we might try to improve the alignment between the goals,
the ideals of what science is supposed to be,
and then what the reality is on the ground
and what scientists are rewarded for in doing their research
today. So the theme, given that it's open
access week, is going to be suggesting
that open is more than just access. We tend
to think of open access in terms of the final
publication of the workflow that goes into producing a body
of knowledge, a piece of claim, a piece
of evidence. But really, the relevance of openness
for scientific research is across the entire research lifecycle,
not just in who can get access to the report
itself. That is the outcome. So what I
will do is sort of give a general
introduction to some of the activities that we're pursuing
at the Center for Open Science and then hope that
you will have some questions, critiques,
ideas or ways in which the things that we
are doing can be more relevant to you
and the work that you do. We are
a service, a non profit organization. So we are
here to serve the good of research getting done better.
So we hope that we can find ways to
find something of interest to you.
But let me begin by articulating something about the norms
of science. Robert Merton in the 1950s,
the sociologist of science, identified what he perceived as the
norms of science, and there hasn't been a lot
of challenge to them, mostly elaboration of those norms
of what it is that scientists do, how it
is. They operate in trying to build a body
of knowledge. And so one of those is commonality,
the open sharing of information. Scientists make claims,
and then they provide a basis. What is the
evidence for making that claim? What are the methodologies
that were done? Here's the methods. What data
were used or accumulated for that? Here are the
data. What analysis was performed? How was that
interpreted? All of that is laid bare so that
others can evaluate the basis of the claim on the
evidence rather than on the authority of the individual making
the claim. Whereas the counter norm is secrecy,
you don't get to see how it is. I
made that claim. It's just, here is the claim,
and believe it or don't. Another norm that he identified
is universalism, that you evaluate the research based on its
own merit. Uh, as I said rather than
particular is, um, evaluating by the reputation of
the individual claiming it is the counter-norm. A third that
he identified is disinterestedness. Scientists are motivated by
knowledge and discovery trying to figure things out as they
are, whereas the counter norm being self
interestedness. I'm just trying to get ahead of
the person down the hall and try to get the
most awards and rewards in scientific process. Fourth is
organized skepticism. A scientist considers all the evidence,
including that that is, against one's prior work in
order to try to reveal the truth about nature,
whereas the counter norm is organized dogmatism, investing in one's
own career, defend one's findings against attack from others
and try to promote one's own view of the world,
whatever the evidence.
And then the last one that wasn't identified by Merton per se,
but it's a very popular one in the
field, is an emphasis on quality of work rather than
on the quantity of producing work. So these,
and then there's other norms have been suggested. But
these are five of the very popular norms of what
makes, uh, scientific practice unique as a body
of knowledge. Many of these apply to scholarship more
generally, and so they may apply more broadly than
just science versus the counter norms. So these may
be easy to identify, but one open question is
to what extent do researchers believe in and try to
live by these, uh and so a team led
by, uh, Anderson, Anderson, Martin and
degree, uh, did a survey of 3500 or
so early and mid career researchers, Uh, and
they identified early in mid career as early career being
, uh, people on postdocs. Uh, all
of these were NIH funded, So nrs a equivalent
types postdocs, uh, in mid career were faculty
had just received their first R 01 So average age
about 40 or so. So there's about 1500 or
a few more in each of those categories. And
what they asked them was, Which of these do
you subscribe to more? The norms that appeared
on the left or the counter norms appeared on the
right and what I'm showing you here is a cumulative
plot. 100% of the respondents where the gray bar
indicates the number of people who valued the norms over
the counter norms. Whereas the black bar were people
who said I value the counter norms over the norms
and how I do science and the gray hatches is
equivalent. So you see both men and early career
about 90% or so. If people are saying I
value those norms of science more than the counter norms
and what I think science should be, how I
how I think it ought to be. So I
said, Okay, that's fine. Don't tell me
what you think it should be. Tell me what
you do. Uh, and it moves this way
. So still, you see, people are by
and large more than 60% in both categories are saying
that they practice the norms of science over the counter
norms. But more people are acknowledging that those counter
norms also have weight in their everyday practice that they
do feel competitive, that they do sometimes hold back
information that they're not always acting in a disinterested fashion
, but sometimes feel relatively dogmatic or defensive, at
least about the findings that they have. And they
said, Okay, that's fine. Don't tell me
what you do tell me what other people in your
field do, and that's what occurred. So there
is enormous cynicism about everybody else, not practicing those
norms that I hold valuable and mostly try to live
up to, even though I might not be able
to live up to them as much as I idealize.
And if we're thinking about this in terms of
a behavioral issue, this is a big challenge,
right? If people perceive the norms of science to
be endorsed by oneself, but not at all practiced
by others, and it's a very difficult situation for
behavior change. People see it as a competitive landscape
where people are just holding information for themselves, doing
whatever it takes to get ahead. While I'm sitting
here just trying to figure stuff out, I'm motivated
by the curiosity of trying to understand things, to
discover stuff and just trying to survive in this world
around me, where everyone else is in it for
themselves, and most people are in that situation.
According to these numbers, most people perceive that they
are motivated by the ideals of science, but it's
not practiced out there, and as a consequence they
may lose out. They may not be able to
succeed in maintaining a career in science because maintaining a
career in science besides just pursuing knowledge requires doing things
that can get you a job, can help you keep
that job, and can help you advance in that career.
So there are pragmatic realities that every researcher has
to confront with. What do I need to do
in order to retain and be able to do this
work while simultaneously having ideals and values of how I
want to do the work? And if those are
not aligned, then it creates difficulties in making individual
decisions. What is it that I'm going to do
today in order to practice my science while trying to
still make a contribution? So there are many different
types of things that we can unpack about what those
decision points are. What I want to start is
just on a general introduction to the center and what
our overall goals are, and then take you through
a few different ways that we are trying to approach
this gap that people confront in what is what they'd
like to do and then what they are incentivized to
do. So the center just opened in the beginning
of last year. Uh, and it is,
uh, independent non profit. It's not part of
the University of Virginia, I'm the main link between it
and the university.
It is tasked, as with mission, to improve
openness, integrity, reproducibility of scientific research. And
we have a staff of about 40 now. We
got a lot of core funding initially, so it's
about $12 million of base funding. But it's by
and large a technology company where most of our activities
are building infrastructure tools to help researchers do their research
in the way that they do it now, but make
it easy to move to things that are more aligned
with individual values. And then we also have efforts
in community and trying to look at the various parts
of the scientific ecosystem for how they incentivize behaviors that
might not be aligned with how people want to try
to live as a scientist. And then we do
meta science research. So actually looking at the scientific
process by, for example, doing large scale replications
of a variety of projects to estimate reproducibility and then
look for why is it that we are able to
reproduce some findings and not others. I won't get
into elaborate for today. But I'm happy
to talk about those afterwards if you like. But
let me give you a very general perspective on the
overall strategy for this gap between values and practices.
First thing that we aim to do is develop technologies
to enable people to change lot of the barriers that
we confront our practical barriers. I might like to
be more open about my research process, but what
it requires me to do is a lot more work,
and I don't have time for more work.
I'm not incentivized to do more work, and I'm
really busy as it is. So how is it
that I'm going to add on extra work to make
my data more available to others to make my materials
more available to others? So we're trying to build
tools that make it a lot easier for people to
do that so that it's just a matter of wanting
to, rather than having the technical or practical feasibility
to do it. Second is, we have efforts
to develop training on how to enact those changes,
a lot of these practical matters of making a research
process reproducible are wanting. How is it that we
can actually get that done? And so part of
the team is tasked with providing training on the tools
that can be used to make reproducible practices easier.
Uh, and then finally, the key part is
to address the incentives to embrace those changes. I
can have all of the desire in the world to
behave toward the ideals. But if I'm not reinforced
in my everyday practice for doing so, it's going
to be very difficult to actually change my behavior because
I have to overcome what our actual barriers, especially
the graduate student collaborators with me. They are in
a very difficult situation of needing a job and not
having one that is secure. Mine is relatively secure,
and so they have to pay very close attention
to the various incentives and how they use their time
in a very competitive marketplace. And so if the
incentives don't align with their own interests in order to
gain and survive in the career, then they can
practice the ideals and then find another job after graduate
school rather than meeting a career that they would like
to have. Okay, so the basics of the
technology approach and I'll start with giving a sort of
a general view of what we're trying to address
with the technology and then come back to some of
the issues and incentive challenges and why those exist toward
the end. But the basic goal of the open
science framework work, which is our core infrastructure,
is to help support the entire research lifecycle,
from the conception of ideas to how we discover
what's known about them through designing studies, collecting data,
this cycle of scholarship, which is the regular
part of what everybody does in order to try to
figure stuff out. Many of the practices and efforts
in trying to open the workflow address just a slice
of that workflow, and as a consequence,
there are a lot of separate slices that are
very hard to work with. And as you know,
picking up new technologies and moving from one to
another can have a lot of transaction costs. And
so we try to minimize those costs by putting the
various services that people use together, and I'll say
more of that coming up. That's abstract. So
the tool that is the sort of the core infrastructure
for the, uh, for the technology option is
the open science framework. It's a Web application.
You can go right now and sign up. If
you have Internet access, you can go right now
and sign up. But you can use it,
uh or at least try it out and see if
it provides any value for the research that you do
. And so what I want to do is give
you sort of a basic overview without doing demos.
We don't have time for that of what its services
are and the problems that it aims to solve.
Uh, and it started with a problem that we
confront in our lab all of the time. And
I know is confronted in many other labs all of
the time to, uh and that I can sort
of summarize some sort of an amalgam sketch of an
experience that happens regularly in my lab. So a
loop is a third year grad student in my lab
, and we have weekly meetings with all the grad
students, and so he comes in. We have
a weekly discussion about the research that he's doing and
we get to some new idea. We're sort of
talking about it and getting excited. And that sort
of realized Oh my gosh, this is very similar
to this study that Nicole and I did a couple
of years ago that got dropped because she graduated and
went on and did other things. So maybe we
should go back to that study. Start with that
. As a pilot, he can use the methods
. He's got some datas initial project and then make
very rapid progress on the problem. And then so
he says, Oh, great, this solves some
problems for me and I say, OK, so
let me go get that study So I go to
turn to my computer and I can't find it.
I say, Well, okay, well, I
didn't do the project. Nicole did the project.
So let me email Nicole and she's in industry.
So I email Nicole, but she's an industry.
So two weeks later she writes back and she says
, What project are you talking about? And so
we have some back and forth trying to clarify the
project because there were lots of different ones and Finally
, she says, Oh, yeah, I remember
that one. Well, I don't have it on
the machine that I use at work, So I'm
going to have to go find the machine, the
disks where that stuff might have been. So it'll
be a while. So a couple of weeks go
by and she says, Oh, well, I
found it. Well, I sort of found it
. I found four different versions that each remember how
we're changing the materials. I don't remember which one
we did of the methods, and then I have
some data here. I'm not sure if it's all
the data, but here it is and then sends
it all to me. And then I pass it
off a nuke and he looks at it. He
says. I think I'm going to work on a
different problem. Uh, so the problem, obviously
, is that we're not able to keep track of
our own stuff for our own use. It's not
a matter of sharing with other people. It's a
matter of using it for ourselves. Of the projects
that we have done get lost and we're pretty organized
lab. But nonetheless, material gets vanishes because it's
also a distributed lab. And this is the true
for many labs is that there are many different actors
there in that group and out of that group at
different points in time, their collaborators at multiple sites
and the materials, the content, the data of
each project is distributed. Everybody has their own ad
hoc organization system. Uh, and if one machine
goes, many things go with it. Unless that
person happened to have it on Dropbox or some other
thing to back it up, Uh, and then
a lot of times, that's all I meant to
get Dropbox set up for that or whatever it was
. And it's gone. So these are problems that
shouldn't happen in modern labs, but they do all
of the time. They shouldn't happen because there's plenty
of technology available to avoid this kind of loss and
all of those transaction costs of trying to define stuff
again once it has been used. So that's where
we start with. The open science framework is as
a means of project management to help facilitate collaboration,
documentation and archiving of research as the research happens.
So the whole goal of OSF is to integrate with
the daily workflow of the researcher to help manage the
materials, the data and the files that support that
research so that they never get lost for one's own
use. So this is just a sample interface of
one project set up in the framework. So when
you log in, you get your own dashboard and
then you can create projects, and then you can
join other contributors to that project. So, Cal
, this is Calvin's project and has many collaborators on
this. And then in the files screen, you
just drag files over and get added to the system
. You can organize by different components of the research
process, so there might be a data component.
And maybe only a subset of the team has access
to the data. There might be demonstrations or materials
to organize the work and the different contributors to each
part of that work. There's a wiki to maintain
information, but all of it is for so that
all of those contributors, uh, no matter who
needs access to it, can find access to the
materials, and they're always in the same place.
And if there is, for example, a manuscript
ng written, but you know where the latest version
of the manuscript is. It's always there. It's
not with person A or maybe it was a person
be. Or maybe it was a person. See
, that had the last version. So that's the
basic goal is to help create a system that is
in the cloud that you will know that I can
always find my stuff when I need it. So
the ideal case when a new came in my office
was would have been. I just go to my
OSF account search for the project and then add him
as a project member, and then he has access
to all of the materials right there. The OSF
also has a number of other features to sort of
help manage that, and one of those is version
control. Version control is a standard and software,
but not a standard in science, and the basic
idea is simple. Our stuff changes over time,
and we should be able to keep track of that
and see previous versions as it changes. You might
do this and formally in manuscript writing, when you're
about to make a big change to a file,
you might upend the date to the name of the
file and then save as and then you work with
the latest version of the file and then in your
folder with the manuscript, you have, like seven
different versions with different dates in it. That is
a form of version control, but there's a lot
of software that helps back that. That makes it
very simple. So what happens whenever you upload a
new version of a file in OSF is it automatically
updates the version number, but you still have access
to all prior versions, and you can look at
the changes that you make from version to version of
different projects. The next thing that it does,
which is really an important element for this, this
efficiency process is emerging the private and public workflows.
When most people ask for doing something open, making
one's data available, it's done after the article.
That's the basis of that data, or the report
of that data has already been accepted, and once
article has been accepted, I want nothing to do
with that research project anymore, right? Most people
think that's where I'm done. I finish the project
. Why am I now going to go back and
do all of this extra work to make the data
available or a code book available. Well, the
goal with OSF is to have all of that curation
process happen. As you're doing the project, we
manage our data as we're doing our project. We're
just not preparing it for public release and some other
website. So, uh, what I described before
is project management is done entirely privately, so research
teams work together on their projects in a private workspace
that's done online. But at the top of every
component of every project is this button that says you
want to make it public. And if you click
that button, this thing pops up and says,
Oh, you're really sure you want to make that
public? And if you click, yes, then
it's public and anyone can access that. So it's
not now an extra process, more work to make
your things public. It's just a matter of desiring
to and moving that mouse and clicking, which isn't
a big barrier in terms of technology, and more
important part is that you have full access control over
the various parts of your workflow. So for example
, there are many datasets that we have that we're
using for multiple purposes their massive data sets, and
we might not be prepared to release the entire data
set on the publication of the first article, but
just the part of it that is relevant to reproducing
those results. Or we might be ready to release
the materials and not other parts of it. Or
you can organize your research so that each of those
pieces have separate access control, and you just release
the pieces that you want to make public and the
other parts stay private for the team. And yet
they're still all integrated into a single workspace, so
you can continue to work with both the private and
the public materials at the same time. And the
outsiders only have access to those that you gave them
access to. Okay, in addition to those we
have to already start thinking about. Incentives will get
back to incentives even more deeply soon. But there
is a number of things that one can do to
make openness actual thing that's attractive rather than inhibition,
a threat for people. One are simple things like
counting the number of downloads that people have of the
various projects tracking how many people visit the page.
It feels kind of good. If someone comes to
the website and actually looks at the things that you're
working on, you can see it has impact.
And then there are a number of things that are
integrated into the OSF of functional links. So links
to your project from other projects because they're either citing
it or they're using it in some way templates.
If someone like your project design, they can template
it if it's public and use that design of the
project as the starting point for their project or forks
. If you develop a measure or a tool or
a method or a bit of code for analysis that
someone wants to extend in some way, they can
fork it into their workspace, which essentially means make
a copy but retain a link and then make whatever
changes they want. And then, as people make
continued changes, that network of links always points back
to your original contribution. So if 500 people an
extension keep building on that particular tool, you'll always
be able to link back to what what was its
origin, and that is much more informative. As
a citation, it's really a functional citation, as
opposed to just a citation in an article which has
ambiguous meaning of how it was used. So there
are a number of these kinds of things integrated in
that are means of alternative metrics for counting contributions to
scientific research that are different than the ordinary metrics of
publication and what outlet and how many people cited it
. Another feature that is an important one for provenance
and preservation is persistent, suitable identifiers. So every
project in every component has a persistent identifier associated with
it, whether it's private or public. And it
automatically generates a citation in the page for you to
be able to cite that information. Because citation is
still one of the means of demonstrating impact in our
discipline, people still care about having acknowledgement of their
work, and then evaluation often is related to that
acknowledgement, and so we aim to make that as
easy as possible to document and track over time by
making every part of the process suitable. This also
helps to introduce new contributor models to science. Right
now, the main way in which science has done
is vertically integrated, right? You have to have
the idea. You have to do the design.
You have to collect the data. You have to
do the analysis. You have to write the report
. You do that yourself within your encapsulated collaborative team
. But crowdsourced science diversifies that by saying people have
lots of different strengths and can contribute in small ways
to larger projects. So I am great at analyzing
data. That's all I want to do. So
great. I take 50 projects and I help with
the data analysis. And I do know writing because
I'm a terrible writer and this is a means of
getting some acknowledgement for that. The components of the
research project allow me to be identified for writing great
code and doing great analysis. And then I can
be cited for that code, in particular acknowledged for
that contribution, rather than only through the singular process
of an entire report. Another factor that the OSF
makes easy is to register the project as a project
and freeze it at a particular point in time.
So I have this original project that I'm working on
that I decided that This is a particular important time
, uh, in the life cycle of this project
. So I want to freeze the project so that
I can point back to this moment in time what
the project looked like them and then I can continue
to work on that project. But I have this
new version once I register it, that is a
frozen version with its own ID identifier that will continue
to persist and always be linked to the project.
That continues to change over time. And there are
many occasions or locations in the time life cycle of
a project that you might want to freeze it and
create a registration. The most prominent one is right
before onset of data collection or data analysis, where
if you have a very strong our priority hypothesis and
want to test it, rather than do an exploratory
hypothesis generating kind of analysis, then you might want
to put constraints on yourself in order to have a
strong confirmatory test and write down your hypothesis and analysis
plan. What is it that we're aiming to do
? What we think we're going to find and how
are we going to analyze those data and if you
register that in advance, you have a certification.
In effect of this is in fact, a confirmatory
test rather than the result of exploratory analysis. Exploratory
analysis is very important. It's how we discover from
science. But there are many occasions where we want
to take a strong stance on hypothesis testing rather than
hypothesis generated and registration preregistration as a means of doing
that. So you create a frozen version. Say
this is what the project was. You can always
point back to that and say, This is This
is what it was. This is my analysis plan
. Here's analyze the data according to my analysis plan
. And here are the results. Okay, the
last feature component that I want to describe with the
technology and then start to move more towards the direction
that this goes. And the incentives work is connecting
services that researchers use. So the goal of the
framework is to be a connector of everything that researchers
use to do their research. Because one of the
main places of losing information about reproducibility is the loss
of knowledge. As people transition between their various technologies
they use to actually do the research, and you
can go into your data generation tool and then out
of that and into your data analysis tool now that
into your figure generating tools. And all of those
transitions have costs to just the work to get from
one to another. But also there's information and process
lost. So if we can connect those services together
into a single workflow, then it's easy to log
and track what it is that happened across those.
And then we can help those different services talk to
each other so that you can be much more efficient
in your work. So what I'm showing you here
are some of the add ons that we have connected
to the open science framework. So if you are
a git hub user or use Amazon simple storage service
, this is if you have lots of lots and
lots of data, you need a very, very
cheap place to store it. Amazon S three is
a good service for that. Almost. It's very
rarely used by scientists. Now it's almost only used
by software developers because it has some some overhead and
learning how to access it. But we created a
simplified interface so it's very easy to make. Use
of fig share is another popular place for storing materials
and data from research and dropbox. A lot of
people use so all of those are connected to the
open science framework so that if you have a project
you can connect your Dropbox folder that has stuff relevant
to it. You're get Hub, Repo or any
other tools to that project. And then any other
collaborator on that project has access to those as well
, even if they don't use those other services and
they all exist in a single file tree. So
it's like you're working with all of these different systems
at the same time, and they can communicate with
each other if they have things to communicate with.
So this is going back to this overall goal is
a means of trying to connect those services across the
workflow with the idea that if we can connect services
for all the different things that you do in your
research, then you have one place that you can
go and you can select from the different kinds of
things that you use, uh and then and then
take advantage of the fact that these things are all
connected. So these are a variety of the different
types of services that researchers might use, particularly the
social behavioral sciences Is the emphasis here. Uh,
and the OSF is connecting through a peer relationships application
. Programming interfaces? Uh huh. Uh huh.
Information exchange between these different services. So you can
imagine, for example, if your publisher is connected
to your data analytics tools connected to your data repository
, then what we could do is make every influence
test in your manuscript a link, the f value
, the T test, whatever it is, if
the person clicks the link, then up comes the
code analysis code that generated that test. And up
comes the data that the analysis code was applied to
in order to generate that test. And then you
can reproduce the analysis immediately. And then you can
also say, Well, I would have added a
cove area there. Let's see what happens if I
had a cove area and then you can be interacting
precisely with their data with how it is. They
did their research in real time rather than writing to
the person. If you even bother saying I wonder
what would happen often. Just read the paper.
We say, Oh, they should have done that
and that's the end of it. But you could
just immediately do that and learn a lot more and
be more interactive with the research process itself. Okay
, so that's where we're heading. We have about
five of these connected now, and there's about a
dozen of them in development, and many more will
be in development. We hope, in the very
near future for connecting the services. I want to
mention another part of the technology projects because it's because
it's sort of leading into a larger direction here for
the kinds of things that OSF can do, but
also the partnerships that can be formed. And this
is a collaboration with share, which Tyler Walters was
just. I think the term here is installed,
installed as she served the installation process of the president
. I was expecting to see him in pose for
him or something, uh, from this weekend.
But the share team is trying to facilitate information exchange
about research objects, and we are partnering on the
technology development of that And the starting project is a
notification system. There are many different providers, different
services, different groups that are generating information, events
that happen about research. Data set is available.
The publication is available. These materials have been made
available, and all of them come with different kinds
of metadata because none of them have been working in
concert. They each are working on their own because
they have their own problems to solve. And so
what share aims to do is take all of these
different kinds of metadata about these different research events that
are happening around the world and align them and normalize
them into a single schema that makes that whole stream
of everything that's happening in research available to anybody that
wants to consume it. So there are a number
of different. There's, like 30 or so providers
so far linked to this, Uh, I think
it's Virginia Tech. Uh, repository is one of
them. V Tech works. Yeah, is already
linked, But many different providers, publishers, data
providers, institutions, whatever are generating metadata about events
that occur in their research zone, and those are
getting normalized into the framework and share and then push
through as a notification system. And this API anybody
can connect to so that they can provide ways to
surface that information. And so there's RSS feeds or
resource sink or pub sub hubbub, which is the
best name ever of anything. Uh and, um
, we have to consumers already through OSF for that
content coming through. So the basic idea is everything
that ever happens in research should be accessible for somebody
. And so if we can find a way to
put them all through a single stream, then anybody
can read that stream and learn about whatever it is
that's happening wherever it's happening. And so let me
give you two illustrations of how we're using this already
with OSF Search and OSF metadata so you can use
the open science framework. It's consuming the information from
that happy and do searches on it so that you
can discover from whichever provider is coming. What information
is available to that search term, right? So
search for science and these sorts of things come up
, and then it's not just because the OSF has
all of these other services. It's not just making
available what information is out there but also providing an
opportunity to curate it. So there's a curate,
this button if you want to curate that. For
example, if you are this researcher and you say
, Oh, I have other materials that are relevant
to that paper that just came through. I want
to add my other materials or my data or I
already put it into another service and I want to
connect those things together. Then you could collect the
curate this button and curate your content, or someone
else might be able to help you curate it.
And this is an example of a curated project.
So this is a project, and there's different notification
events happening. The article was published in clause.
The data is stored in data one, and so
those different notifications were merged as separate components in OSF
, and now all of that information is available together
. So for us, there's two different views happening
on what OSF is doing. One is this bottom
up way of giving researchers access to be able to
use the OSF to help them manage their own workflow
, to manage their materials and data privately and publicly
. If they want to make things available and then
this data stream that's coming through and blasting everything that's
happening in research that's making those materials available in OSF
and giving the opportunity to link it to researchers doing
their work flow as they do it. So if
they want to make things more available now, they
can integrate that much more easily with other content and
materials. So that then becomes a registry in a
large part is there's many different ways to access that
content across a variety of president. But anyone else
can provide their own interface to this, too.
It doesn't have to be OSF because it's an open
API. Okay, so I want to stop with
the technology part and just mention that all of those
things are, uh, the Center for Open Science
doesn't develop any intellectual property for itself. So everything
is open source, freely available technology, and we
have no monetization scheme at all. So everything is
free, and the goal is to build services that
can be used and that can be community driven,
that many people can contribute to those services and the
development and maintenance and sustaining of those. If you
want to talk about, how can you do something
that's only free. We can talk about the sustainability
parts later, but I'm happy to talk. Uh
, okay, so that's it for some of the
technology. Now, I want to talk about two
other pieces in the remaining time that we have one
very briefly, and that is training services. So
part of the challenge with a lot of the reproducibility
practices is that people aren't getting access to the kinds
of training for the tools that could be very useful
to improve their efficiency and quality of research. So
we have a consulting, uh, email address right
here. Stats Consulting at C u s dot io
There's a website. See us dot io stat underscore
consulting. Uh, and you can arrange. You
can send email questions you can arrange an online hangout
you can do a lot of you can arrange an
in person training. We have one here on Thursday
for different parts of reproducible practices, either using tools
and computation, doing different kinds of analysis or helping
manage workflow just to point out Tyler mentioned at the
beginning. But just to point out the workshop that
will be here this week, it's on the 23rd
, three o'clock in the library multipurpose room and Courtney
Soderbergh, who is on the cause. Staff will
be running that so if that's of interest and also
be giving training on OSF use and some practical things
as well. So that's all I'll say about training
. So I want to return back to the theme
of openness more than access. And the point that
I've been making so far on that is that there
is an entire workflow that can be made available.
And our values suggest that that's part of what can
help with scientific research in the pursuit of knowledge is
knowing the basis of claims. If I want to
dig deeper in principle, I can dig deeper.
I don't have to, but I can't on two
different things. And if you want to find out
how I've arranged something, how I did something,
you can find that out. Um, and there's
a lot of reasons to be caring about that beyond
just the ideal of wouldn't that be a good thing
? And this has been making a lot of news
headlines in the last five or six years about concerns
in Reproducibility that there may actually be a problem to
address rather than it just being a good thing to
do in principle. Here are headlines that came out
in 2011 and nature, believe it or not,
how much can we rely on published data on potential
drug targets and raise standards for pre clinical cancer research
? These two reports were ones from Bayer and Amgen
Industrial Laboratories that we're trying to replicate. A couple
a few dozen, I guess, studies each in
biomedicine, mostly in oncology research, trying to take
what was found in basic labs, replicate those results
and then extend them in order to develop new therapies
, pharmaceuticals or whatever. And they reported very poor
reproducibility rates and their efforts to take these basic labs
. One of them was able to produce 11% of
the original findings. The other 1 25% of those
original findings of published studies that have been having a
high impact in the research literature and their argument was
that they didn't screw up. It's the field that
screwed up, and things are happening that are making
it so that published results are not as reliable as
they ought to be. There's many things to debate
about of where the source of the challenges are whether
they identified those percentages correctly. They didn't have a
random sample, for example, of of research projects
, and it's not. They didn't make any of
their data available or any of what studies they were
, so no one could critique how it is.
They actually did it, but nonetheless, they got
plenty of attention for the challenges of Reproducibility. And
at the same time, there's been a number of
researchers and meta science demonstrating that there are really curious
issues in how it is the scientific. The published
literature corresponds to what might actually be happening in research
. This is one figure from Daniel Finn Ellie's work
on looking at the rates of positive findings across a
variety of different disciplines in science. So positive finding
meaning confirming the hypothesis found a difference in the treatments
between different conditions are found. An association that was
hypothesized. What he found was that you know,
from physics on through psychology, psychology of the winners
get 85% or greater confirmation of the hypothesis in the
in the reported results. That's a remarkable success rate
of showing positive results in literatures, and it's even
more remarkable when it's contrasted with the power of research
designs. So there are many projects showing this.
This is just one example that we did last year
investigating the power of research designs. And so in
this case, we did a meta analysis of meta
analyses, looking at a variety of different areas in
neuroscience, the overall power to detect defects that they
were investigating and what I'm showing you here on this
history Graham plot is the median power of the various
meta analyses. So many of them had extremely low
power. And then there's a small minority that had
very, very high power. But the overall medium
was something like 30%. And I remember that,
right? Maybe it's some 20 something percent power.
If you're not familiar with the concept of power,
20% power means that assuming everything being investigated is true
. The positive result rate that we would expect is
20%. That's how strong, uh, sample how
large the samples, How the size of the effects
being investigated. We'd expect to 20% positive result result
rate, even if everything were studying is true.
Whereas neuroscience back on this page has 85% positive result
rate. So if the power is only about 20%
assuming these are comparable and yet 85% of the results
are positive, it's not possible to get that lucky
. So it suggests that there's something else happening in
the research literature. Negative results are systematically getting excluded
, perhaps, or results that are partially positive or
getting amplified and turned into positive results. Beyond their
evidence. There are a couple other ways to illustrate
some of these challenges. I'll just give a brief
example with this. The basic idea is that when
we read a report of some analysis reporting a phenomenon
, we assume, well, that's what the data
show they just did. The analysis. There's the
answer, and I might have analyzed another way.
But that's that's the answer we were interested in looking
at variation in replica bility. And so, uh
, what I'm showing you here are 28 different teams
that investigative research and these are This is an odd
ratio on the X axis. A one means no
effect at all. There was no difference between these
treatment conditions or no association, uh, detected,
and then the dot is the effect size estimate that
the team got. And then this is the 95%
confidence interval the lines. And so an ordinary use
you would consider any confidence interval that doesn't overlap the
dotted line to be a significant result positive result and
about two thirds of these positive results about one third
or not positive. It doesn't really matter what the
research question is. But what we were investigating here
was the likelihood of players with darker skin tone getting
red cards in soccer than players with lighter skin tone
. And so 28 different teams all investigated that same
hypothesis, and about two thirds of them found positive
evidence. Now, the twist for this is that
they all use the same data set. Uh huh
. So we gave them the research question and the
data and said to analyze it, how you want
to analyze it and just send us the result?
Uh, the additional factors that make this more compelling
. Well, so and then. So what you
see is a great deal of variability, even though
they're using the same data set the same question.
What we also did is the teams that are involved
are very quantitatively sophisticated many of them have is their
p or the lead of it. Our instructors of
statistics, in whatever discipline that they're in, have
a great deal of experience have published in methodology journals
. One of them used the technique that he invented
for doing the data analysis. Um uh, so
they all had a good deal of experience. They
weren't just random, uh, analysts. And also
we had a peer review phase. So everyone did
. An initial analysis sent us the initial analysis with
what transformations. What exclusions? What model they used
. And we took all that information, removed every
bit of information about what actual results were observed and
then made those analysis strategies available to the rest of
the teams. And everybody had to be a peer
reviewer for at least three others. And so they
got a lot of feedback and information exchange about the
strengths and limitations of their various strategies. And then
they went back and did their results of the analysis
again with that feedback. And then these are the
final things. So if anything were expected a great
deal of convergence in the analysis approaches, given that
they all were talking to each other about their analysis
approaches. And nonetheless, we see a great deal
of variability and what most papers do. Any single
paper that you read does one of these. No
one does 28 of them, because that'd be a
silly thing to do. But we went ahead and
tried it, and we're very surprised at how much
variability we actually observed across the different data. Now
, analysts suggesting again that the potential value of opening
materials and making data available gives an opportunity to see
how the different assumptions that went into the initial test
might actually be less certain. There is more variability
to be observed in those potential outcomes for time.
I think I'm going to skip selective reporting, but
there is a great paper just came out last month
in science. I won't explain the details, but
shows that people that get strong results are very likely
to write them up to publish. People who get
negative results are very unlikely to write them up to
publish, uh, probably presuming that they would never
make it through the review cycle anyway. And this
is not a case where the results are Well,
that was just the first test of some outlandish idea
, so I didn't get it. So it's not
worth sharing with the world. This is from tests
, which is a time series experiments in social sciences
, which is a peer reviewed process, a very
high precious sample. And so people usually do this
as like the fifth study in a line of studies
in order to get a representative sample. And when
they're getting negative results, they're just dumping them,
even though it's a very high effort process. So
that's a very interesting paper I recommend. The curious
thing about all of these is that the problems that
have been identified in this concern about reproducibility writ large
are things like low power over abundance of positive results
, ignoring negative, no results, pursuing question where
research practices, pursuing many different analytic strategies and just
reporting the one that looks the best. Uh,
not actually doing replication, because there's no incentive for
it. Uh, and limitations of Knowles hypothesis,
significance. Testing. All of those problems are the
focus of methodologies Attention now, and they're all the
same ones that were identified in the 19 fifties 19
seventies by methodologies methodologies. Then simultaneously, the solutions
that are being suggested now focus on transparency, making
the process available. Being very clear between what's a
confirmatory test hypothesis. Testing versus exploratory analysis hypothesis.
Generating you can't hypothesis, generate and test on the
same data and doing replication. Those are the methodologies
claims of what we should do. And they were
the methodologies claims of what we should do in the
19 fifties. 1970 So none of this is actually
new. It's just louder, Uh, and what's
surprising is with that amount of time having passed,
why isn't it that things have changed at all?
And our contention is that it's not that the problems
are misidentified or the solutions are misidentified? It's that
there is no not enough attention to implementation of how
to actually get those solutions into the daily workflow of
researchers because it doesn't address what we think is the
key problem, which is. The incentives for my
success are focused on me getting it published, not
on me getting it right. If I get it
published, I get the rewards I'm seeking. I
get my job, I get to keep my job
and I get rewards and career aspirations. If it
turns out not to be right. Well, probably
it's harder to know because no one ever bothers to
replicate most of the work. Uh, and that's
sort of the long tail of yes stuff sometimes doesn't
work out in the end, but really, the
immediate incentive is on publication. And so if we're
going to try to address this challenge, we need
to have attention to what is the things that are
driving my behavior as a scientist, even if I
possess those ideals that I want to do things in
an open, transparent, reproducible manner, etcetera.
So there are many challenges that get in the way
of this and that. But most of them are
behavioral ones. So I talked already about the perceived
norms that we think I want to do it that
way. I mostly do it that way, and
everybody else doesn't do it that way. Behavior change
is very difficult in that kind of circumstance. We
also possess these fantastic frontal lobes that if we have
flexibility in our analysis, if we have flexibility and
what it is we report that's coming out from the
lab, then I'll find really good reasons to think
that the analysis that looks best is actually the best
analysis to do. And the studies that worked are
actually the ones that I did right and those other
ones that didn't work that showed negative results. I
totally screwed those up. I shouldn't report them to
anybody because those were terrible studies. We should have
never done it in the first place. And so
we're doing this not necessarily by intention, but because
we have skin in the game. It matters for
me to get positive results that are novel and that
are a clean story, because that's what I'm incentivized
to get. The best journals want a beautiful,
positive novel, Clean story. And so the incentives
drive that. Even though what's happening in my lab
every day is a mess, we're at the boundaries
of knowledge. We're pushing out into things we don't
understand or mostly have ideas that turn out. Nope
, not at all like that. It looks like
this. Okay, let's turn that way and try
that, and we're fumbling around figuring stuff up,
out, and that's kind of what's great about doing
it, because we don't understand it. We are
discovering new things, and then we're getting to a
place where we think we have some confidence, and
then we give ourselves a tough test, and sometimes
it works out. Uh, but we're not incentivized
to report it that way. And so we don't
as much. And we do have, uh,
motivations that might lead us to report it to be
more beautiful than it actually is. And also,
we operate with minimal accountability. You don't know what
happens in my lab. You only see the report
at the end. And we know very mature literature
on accountability shows that when you're not being observed,
you don't take the perspective of an outsider and you
might make decisions that you wouldn't make. If you
thought that those decisions can be observed, you might
even recognize that you're doing it. Which is the
most pernicious part of that challenge. And then the
fourth challenges that I'm busy and so are you.
We have a lot of stuff to do. So
all of this great aspirational things to be more reproducible
, to do replications, to be open our things
, to say, that's more work that's going to
make it less likely for me to get a job
. And so, unless we can actually develop technologies
and tools and incentives that integrate well into that daily
workflow. We're not likely to actually get any of
these things done, so I'll just mention to incentive
approach is very briefly and then stop so that we
can have a little bit of discussion before we have
to go. And one of those is a signaling
approach is because these are valued because openness is valued
because in some cases, making very clear confirmatory tests
is something that is valued. Then if you can
signal when it's happening, then it's a lot easier
for those norms that people hold in mind to turn
into daily practices. Uh, and so there's a
easy way to signal is badges. Uh, so
we have a open committee. If you want to
join the committee, you're welcome to do so.
An open committee that's working has worked on specifications for
open data, open materials and pre registration. What
does it take to earn that bad? And journals
or any other badge issuer can adopt those specifications and
issue badges for research that's done so. An article
, for example, psych science has adopted these badges
. It's a leading empirical Journal and Psychology. And
if you get trips this week in psychological science,
it's an email. Each week you might see these
little badges there for those articles that have met those
criteria and applied to receive the badge. And so
the badge badges show that this person has open data
open materials, and in the article, it provides
a link to those data materials so you can go
right to the basis of the figures in the paper
. Uh, the badges system is also now we
have a baking process. So the badge itself has
digital information about who the issuer is, what it's
issued for the d. O I for what it's
linked to and can become a link itself rather than
just an image for connecting you to the materials and
the data or the preregistration. So that just does
a very small thing, which is make visible things
that people are already doing but aren't getting any reward
for now. So at least gives them some acknowledgement
. But it does a bigger thing in signaling,
which is it, says psych. Science cares about
this, and these people care about it, and
they were able to do it, and if they
were able to do it, that must mean that
it isn't such a barrier to do that. I
couldn't do it, too, and there is an
opportunity to then do it. And so norms can
shift very rapidly if the values are already present.
If everyone already agrees that those things are good to
do, then signaling that it's possible to do will
accelerate the change. At least that's the theory we'll
see in practice. A second incentive approach to try
to address some of these challenges is registered reports.
So here's the cartoon version of how research gets done
. You design a study, you collect and analyze
the data, you write up a report and then
you publish it in a journal. Of course,
there is this thing peer review that happens after the
report is written before it is published. But of
course, in the standard model of peer review,
the research is done. You've already done everything you
can to make it as positive as novel and as
clean as you can do. So all the incentives
are geared towards making it as beautiful as possible for
peer review to get it to the promised land published
, Uh, and rather than having the incentives aligned
to promoting accuracy and so registered reports just does one
thing. It's just peer review after the design phase
. So the goal with pure review in this case
is that the individual researcher writes up the research question
they're going to ask and the design to test that
question, and they have to specify why this is
important to know. They have to specify why the
outcome should be known, regardless of what the outcome
is, because there's a lot of research that we
would look at in advance of doing it and say
, If it comes out that way, it's really
interesting. If it comes out that way, it's
not at all interesting because it wouldn't it be amazing
if kind of study and people say when nothing happens
, Yeah, that would have been amazing. There
was no way that was going to happen. And
so if you do peer review on that kind of
work in advance, then much of it won't get
through the process. So in those cases it's more
exploratory at first before moving to this. But for
many questions, we want to know the answer,
because the answer is important to know, regardless of
whether it's a positive or negative result. And so
you have to make the case to the peer reviewers
that, in fact, your design is a good
test of the question. And so all these incentives
are aligned for the individual researcher to make it the
best possible test that they can. And for peer
reviewers, the incentives are an interesting shift, because
in many of these cases where it's important to know
the answer one way or another is, it's because
there is a theoretical debate that there is a camp
as as if they were camps that are promoting one
point of view and another camp that promotes another point
of view in the ordinary process. If you get
a result that provides support for one camp, then
they think it's a great paper, and the other
camp finds lots and lots of problems with it.
Here. The results are not known, so both
camps have the same incentive, which is to make
the test the best possible test that it can be
. And so the peer review process can in fact
potentially even just improve the overall design and the diagnostics
city of the results because, as you may,
if you've observed this yourself in practice many times where
these theoretical debates, the actual published literature has to
literatures that essentially side by side rather than directly addressing
each other because they're using different enough methodologies that they're
not actually able to address that where the core contention
is between the points of view. Whereas if they're
confronted with that in advance, then they have to
work through those differences. So this isn't just an
imagination. There are 13 journals that have adopted this
in different forms that are trying it out. And
we are in the May I think of this year
. We released a special issue of social psychology with
15 of these registered reports and and I was a
guest editor for this. And it is an amazing
process that's very different than our standard peer review to
have people working through this research before before the results
are known. So there's lots of interest across a
variety of different disciplines. The life just announced a
partnership with us for doing this process for reproducibility project
of cancer biology, so we're doing replications of 50
high impact cancer biology studies from 2010 to 2012,
and all of them are going through peer review in
advance instead of once the results are known. Okay
, so I want to bring this to a close
, since we've been going for long enough and just
noting that all of these incentive challenges are ones that
have many different components in them. Uh, it's
not just the researchers themselves, but and the publishing
elements, but it's also things like the university systems
and reward. People can say All this is well
and good, but my tenure committee wants to see
X number of articles in X level journals, and
they grade the journals. And so I have to
live up to that. Societies play a role and
what kinds of things that they reinforce. Funders can
make requests or demands or not to suggest things.
And so addressing the incentive structures is going to require
all of these different groups to work interactively and communicate
on how it is to shift, uh, to
move towards a more open and reproducible science, and
that is happening at the top. So you,
if you are linked to the biomedical or behavioral sciences
at all. You may have seen NIH making moves
on this. They've made a few different moves on
promoting more open data and changing their reporting practices.
But that's just the beginning. There are many more
that are in development and underway. They're also doing
a lot of developing some new training programs. But
this has also gotten the attention of the White House
. So this is the least read journal ever The
Daily Journal of the United States government, Uh,
and what? Uh, their stuff pushed into that
journal every day, and there was a 28 points
or 29 point memo from the Office of Science and
Technology Policy at the White House, where number 11
, uh, was given recent evidence of the reproducibility
of a surprising number of public scientific findings. How
can the federal government leverage its role to help address
this problem? And what they are signaling when they
send something in the Federal Register is that they are
getting ready to act, and this request for information
is a public request. And almost no one ever
responds to these because it's in a journal that no
one ever reads. But it is the official way
that the White House signals that it's about to do
stuff. Uh, and, uh, we were
chatting with folks there about this, and they said
this just got released and we said, Okay,
Uh, and so there's a lot of people responded
to this particular issue. They had 29 issues and
, like, 90% of the responses were on this
one. So they got a lot of feedback on
reproducibility and what to do about it. But they
are going to do something about it, and they're
probably going to do something about it. To the
29 different federal agencies that fund all of the different
areas of science and, of course, a challenge
and then goes without saying that Congress is not now
interested and well, I've been getting calls from,
uh, the staff for the Congress's Committee Science Committee
, Lamar Smith's staffers about what to do about reproducibility
. And, as you can imagine with other things
that the federal government has waded into in science,
if they impose a solution for us, we probably
won't like it. IRBs, if you have familiar
with IRBs, are the result of intervention by policy
, and they've been a nightmare for many years,
and they're just now starting to turn. There's some
great move there, Um, but the the key
for us in terms of collective community action is to
get in front of these problems before Congress has the
motivation to act. The interest is there, and
there is becoming more interest. Uh, but But
the federal government won't act if it's becoming clear that
the sciences or whatever community is interested in is taking
care of the challenges itself. And we can come
up with much better, much more effective solutions that
take into account what's actually happening on the ground in
the sciences. Then a bureaucratic solution is likely able
to. So to the extent that we can be
in front of this by actually pursuing as societies,
funders, institutions and otherwise, the kinds of things
that we can do to show that we're addressing these
problems, then the kinds of solutions that are also
pushed top down might be much more gentle of the
kind that we would like. So ultimately, our
goal is to address this gap, that people's goals
of how they want to be are aligned with how
they actually get to be and that they can see
that others are that way too. And that we're
all just in the business of the largest public works
project in human history, which is building a body
of knowledge for the betterment of humankind. So I'm
going to stop there and thank you for your attention
in time. Okay? Yeah. Frank. Yeah
, A couple questions. If there are a couple
questions I'm happy to answer, how do we handle
confidentiality of the identity of participants in storing the data
online? It's a very important issue. Uh,
and so there's a few different things, uh,
to address with that one. Is that the You
can store data, uh, private data and leave
it private on the on OSF. But there are
many different repositories that have varying degrees of data security
standards and approaches, uh, that are going to
be connected to the OSF, and so the key
will be matching. Whatever the security needs are for
one's own data for the kinds of service that are
storing it. I wouldn't store any actual identifying data
in a public format at all online. That just
is a nonstarter. And so what? The thing
with moving towards openness is that security becomes even more
important because because errors will occur, people accidentally expose
information that shouldn't be so. Data verse and a
couple of other groups are involved in trying to develop
tools so that you can automatically or facilitate the identification
of sensitive data and help screen that as a software
solution, let alone being able to make sure that
you store it in places where it should be stored
like that. Are HIPAA compliant, for example,
other questions. How what's that managed to? This
is a free service. How have you managed to
control costs for data stores? Last I heard,
Yeah, the rest of few years wouldn't offer free
services coming outside positive scalability. Yeah, so,
uh, right now we're able to do it because
the usage doesn't exceed our capacity. Uh, the
it's a problem that we would like to have,
which is Oh, man, this is getting so
much use that it's exceeding our capacity to fund it
because, um, many repositories are worried about this
problem of sustainability. How are they going to maintain
their services? The curious thing is that most repositories
right now are very under used. And so the
sustainability is really can we get enough users to justify
continued investment in these resources? Not that the resources
aren't available. So for us, the goal is
to make sure that we actually can make movement on
the front end of getting it to the point where
we have so much demand that we need more resources
. Because if the OSF works funding, it won't
be a problem, because it's the kind of thing
that everybody says. Oh yeah, that would be
a good thing to have. It's not like it
was an idea unique to us. Many people have
had this idea. Uh, it's just a matter
of execution. And if we can get a common
platform, and for us it's connecting all of these
repositories together, we don't need to host any of
the data. We can just connect to all the
repositories that already exist, and there are many hundreds
of repositories and we were connected to five of them
. Um, but those if all those repositories are
spreading the load, then that helps with sustainability challenges
, especially if one's have joint agreements to cover each
other's data. if they go down for some reason
, and then a longer term solution is torrents.
So there's many open, uh, storage places,
and they're all on our individual machines and so bit
torrent. You may know if you, uh,
are less than 30 or have Children less than 30
as the place to get music and videos for free
, but it can be used for good instead of
evil, which is a means of connecting All of
those, uh, is to spread the data around
too many different sources. And just through knowing where
all of the bits of data are pulled the data
so you could host a gigabyte of science and I
could host a gigabyte of science, and if we
all are hosting bits of it on our machines than
the sustainability is a lot easier because many different machines
could crash. Then you could still recover all of
the data. That's a much longer term solution and
not quite the question you asked. But that's sort
of in the realm. That addressing okay, Anything
else? Yeah, please. Curious Later. Does
it support industrial projects? Does it support industrial projects
and their management? Yes, it's open and free
for use for any, so, yeah, it
can be used for any any group or any service
. It is a free service. Okay, so
let me sign up there to get this much of
memories for you. Or is it something like that
? Right now? There are no limits at all
. There is an individual file size limit. But
if you confronted that in what you were trying to
store, then you would email us and say,
Can we store this? And if it gets into
the terabytes, we might say, Oh, we
need to think about how we're going to do that
. But outside of that, probably not a problem
. Now when we get really busy, then it
might become more challenging. But there are other repositories
, like data verses, one of our connections.
Uh, they're based at Harvard and the Institute of
Quantitative Social Sciences. They are insisting every time I
ask them, they say No, no, no
, it's unlimited storage. Well, store whatever I
say. Yeah, unlimited storage. They say unlimited
storage. So I'm we're hoping to push them to
their limit. They say No, no, no
. That's not what we meant. You said unlimited
. So we'll see you. There's another hand.
Yes. Mhm. Sure. Yeah. If you
were. Oh, were they? I just No
. From what I know, some signs. Sometimes
really bitter debate about how things should be he made
. So when you publicly sports after a year Thanks
. Yeah, with a words like we're finished.
Yeah. So what? What arguments? That would
be great to add because it was so interesting.
It was done privately as a standard period. We
only want to push so many boundaries. And we
didn't push the open review boundary on this, but
it would have been really interesting to make a lot
of that discussion available because it is theoretically rich,
right? Exactly. You're saying it's debates about how
to even test the question, often become the way
to identify where the actual rub in the theories are
Rather than that being sort of the getting just Oh
, it's the details of how you do it.
No, no, that's really where it's at.
So we didn't publish that. We just published it
like regular reports. And then there are commentaries.
So the original authors were all invited to write a
commentary if they wish to on the report. And
then they if they did write a commentary, the
replication teams, they were all replications. In this
case, the out there registered reports. They're all
trying to redo a study that had been originally public
. But then the replication teams were able to write
a reply. But it wasn't quite what you're suggesting
. Which would be really fun to do between the
first community and the phone. Who do you think
for the four coming in my opinion. Researcher.
I like more or less. Mhm. Very interesting
. Kind of. All right. No, you're
a little bit ready. Yeah, Okay. Who
do you think should the first huge. Well,
they have different opportunities to make change, right?
So individual researchers can do things right now, the
things that individual researchers can do right now is start
to integrate open and reproducible practices into their workflow.
And as reviewers ask that of people that they're reviewing
, uh and so those are the main mechanisms we
have individually to make change. But we also have
the pragmatic realities of the world around us. And
if we ignore those entirely, then we won't survive
long in the science. Funders can have a big
lever, which is Do you want the money?
And that is so change can happen very quickly.
But funders, especially federal funders, have many different
constituencies to deal with. Uh, and so it's
much harder for change to happen, especially at the
federal level. So we've observed much more, uh
, adaptability to private foundations because they say, Well
, we're just going to change our rules and that's
so if they don't want the money, they don't
have to have the money. Uh, and so
many of them are now insisting on more open data
. Uh, and some of them are starting to
talk about insisting on preregistration, uh, whether just
of what the research goals are, let alone whether
it's confirmatory. Um, but I will say that
the movement on funders at the federal level is gaining
steam. Open data requirements. Uh, every agency
with in the U. S. Every agency with
$100 million budget or more for research has to have
an open data plan, and those are just getting
consolidated. Now. The White House is going to
move further on that, and many of them will
start to develop enforcement mechanisms. So the NIH for
example, is talking about How is it that we
can actually enforce because they have an open data policy
, but no one does anything about it. Uh
, and so it has no impact. Um,
so it's both probably have to happen, But my
view, the long solution is the bottom up one
. It's like I want to integrate my entire workflow
, not just the stuff that gets public who are
your competitors. So if I wanted to pre register
study the only one? No. There are other
ways to preregister, depending on sub fields. We
have an interesting position in that we have no monetization
and no intellectual property, and everything we make is
freely available. So functionally we have no competitors because
if somebody else is doing it, then we'll just
connect to them. That means we don't have to
build it. Um, so that's really good for
us. So there are a few other places to
pre register. There is another sense in which there
is a competitive landscape to which is there are commercial
entities. Like if Google decided it wants to do
this, we'd be gone. Uh, they would
just do it. Uh, and the risk of
course, is that the system is closed. Uh
, and but Google has so much power that everyone
would start using it because Google is really good at
doing things, and we'll do it that way.
Uh, and even though a free, completely open
system that's ideal idealized, at least not ideal necessarily
, uh, would wouldn't be able to survive that
. But there are few organizations for registering. Uh
, if you do clinical trials research, you have
to register by law, clinical trials, Gov.
And a few others. You can register at clinical
trials, Gov. For other kinds of research.
But it's still a limited range, and what they
support economics now has a couple of registries, the
to sponsors, one in particular. And then there
are a few more fledgling ones that are very small
. Ours is a generalist solution, and so we
are building connections to clinical trials, to the registry
and to whatever other registries come up, so that
if you are an economist and you want to be
able to submit to the registry, you can still
register in hours and push it into their registry,
and then you can select from a variety of registries
if you want to populate them there, it's the
same with data and other things. Yeah. So
what? Questions of the registration of the experimental design
One. How much does that typically cost? That's
just do that or for individual submitting them. And
then also mhm. What are the risks for?
Have you seen this playoff for me? A lot
of time to I don't want competitors. This is
your doing too soon process. And that really putting
your this is that you're asking for people who are
in a similar field work their concern there. And
then also what the so many questions, So many
questions. So let me start with the second one
because I've already forgotten the first one. Uh and
then you can repeat the first one. Uh,
so the challenge of I don't want to make my
stuff available because I'll get scooped is one that a
lot of people have trepidation of of going open,
and I think it's a totally reasonable concern and then
not actually a concern in practice. Uh, and
this, you know, an example of this is
archive a, r x, ivy dot org,
which is the physics and now spanned out to a
couple of other journals. It's a manuscript sharing service
. Most all physicists use, uh, archive to
share their manuscripts as soon as they finished writing them
and then make them available early on. Everybody in
physics said, there's no way I'm gonna use that
because people will scoop my idea and do a real
quick study. Do a crap. You get it
published before my careful study gets through the publication pipeline
and then I'll have been scooped. But then they
rapidly realized that posted an archive. Actually, his
first establishes precedents. Oh, they're the ones that
posted it there. So the first one to make
it available, posted. I think the same will
occur across the entire life cycle, right? As
soon as you make your stuff available, you've marked
. I'm working on this and really it turns into
execution. Ideas are cheap, so I'm going to
put every single idea I've ever had online, and
then I'll get the victory. For all of those
ideas. Lots of other people have had those same
ideas. It's really on actually getting the work work
done, So that's one way to answer it.
But a second way to answer, which is No
, you're right. So what you can do is
on OSF is registered to keep it private and simultaneously
with the with the peer review process. It doesn't
completely solve it because they have to. Other people
have to read it, but we actually have implemented
. So you can have double blind review even of
registrations or of studies while making your data and materials
available so you can send a view only link and
anonymized it of your OSF project for peer review,
whether pre or post and as blind as your manuscript
is blind, uh, so that you can protect
at least identity. But you do run the risk
of you've now made your material available so someone else
could see it and say, Oh, I'm going
to do that now. The reality is, most
people are already busy enough with the things are doing
themselves to go stealing other ideas, uh, that
are already written up. And so the person is
presumably already going to do them. So I have
to work really fast to steal your idea rather than
just keep working on working on. But it's a
concern that won't go away immediately. People have.
So what was the first question? Cost anything?
I remembered it before you said so. No,
it doesn't cost anything. So it's just a peer
review process in advance. It doesn't get published.
And so all of all the ones that are doing
it so far there's There's no charge for the register
. Yeah, sort of make use of this system
all the way through the research process and to get
all the data together. And they decided to submit
to a journal. That's not open access, are
you? Is there any concern with any kind of
function of having data Open access is published, whose
journals like, which is most every journal? Uh
, no, there aren't any legal ramifications because even
some of those journals do have open data encouragement or
requirement, and more of them are looking to do
that. Of course you know, making it all
open is an ideal, but that isn't the present
reality. And so we have lots of developing partnerships
with commercial publishers that have mostly closed access work to
open as much of the process as as possible,
and they're not going to give up subscriptions easily.
So that part may not be able to convince them
to change, but we may not have to convince
them. The reality may overwhelm that. The world
is moving open and they just have to shift there
funding model in order to accommodate but making all of
the data and methods open. And then the manuscript
itself, functionally, is going to be making it
all open. So it moves very quickly. Is
that decrease? Someone's motivations subscribed those terms. Uh
, so the problem of subscription is not going to
be solved easily. Uh, unless scientists act collectively
to solve it. Because right now what happens is
I pay zero attention to how much any journal costs
my university to subscribe to. And I demand that
they give me access to all of these journals that
I need to do my research. Uh, and
so the cost is completely opaque to me. It's
It's totally irrelevant to me. Uh, although it
is highly relevant to me because it has a cost
to the university. It's a big burden. Um
, so if if the scientists who are at both
ends of it are giving away their scholarship and then
demanding their university by a back, aren't aware of
that. And then don't actually connect the dots in
order to say, You know what? This is
ridiculous. What we should do is all just decide
. I've been told this is anti trust. So
maybe breaking the law by suggesting it. But what
we should do is decide that 200 top universities in
USA on January 1st 2017. We are cancelling all
of our subscriptions. So everybody figured out we're doing
on that day. We're canceling all of our descriptions
. We can't afford it. And then the scientists
and societies would say, Okay, well, we're
going to do. There are options. There's already
infrastructure to support it. And we just we just
pick up the editorial boards and then move them over
here, give it a new name. It's the
same journal. We're doing all that work anyway.
So that would be the way to do it.
Is the coordinate solve the coordination problem? Going to
be hard to do outside of we can do it
. Oh, no. I have to break the
law. I don't know. Yeah. Fergie,
check. There's a certain amount of certain torch to
increase. Yeah, Parliament as well So how how
do you speak? Mhm potential. Okay, science
, since special funds for research. Yeah, Okay
. Uh, at least from the experiences that I've
had in this so far, which are just anecdotal
, uh, is that it can accelerate the movement
for open science because private funders tend to be more
facile, uh, and able to make decisions on
getting, uh, prioritizing return on their investment,
Uh, in a way that federal government may want
to do but has a much harder challenge and actually
changing policy. So we've had conversations with a number
of funders Just said this is going to be our
new policy, and they go and do it like
, welcome. Trust in the U. K.
Has changed a lot of policies on open data,
right. They now mandate that everything that they fund
be published open access, uh, and they'll pay
for it. Um, if it's not published in
the Life, which is the free open access journal
, which they are underwriting the costs for So private
funders can move this very quickly where the federal government
can't as easily. So I guess I I spoke
to generally, I guess, like slowly never.
But I guess I'm very multi pharmaceutical firms. Okay
. Okay. Yes, yes, that's different.
So there's, um And then it also bridges into
other kinds of intellectual property concerns where patenting is a
prairie. Right? So, in engineering, uh
, there is a lot more interest and divide and
individual labs and across labs, and whether we're building
for patent purposes, commercialization or whether we're building for
knowledge for knowledge's sake and those what the open science
movement will do is just make very clear which is
which it's. There's lots of good reasons to its
promote commercialization to have companies and organizations investing in research
in order to build new products and tools and technologies
to sell. Um, but it should be very
clear which purposes for and the private purpose it will
be very clearly stated. This is private purpose.
Were actually pursuing knowledge for our game that we hope
eventually will be your game, too, because you'll
buy our stuff and like it. But really,
it's ours first rather than academic science. Yes.
Are you really as it value? Remove the peer
review process from later stages, or you're just adding
to the earliest. Yeah, So there's a number
of different ways. You know, my I wrote
one paper called Opening Scientific Communication, which sort of
outlines sort of that are, you know, imagined
utopia of how this process would work, and it
really just opens the process so that peer review is
continuous, which is what it is in reality,
right? Because we read stuff in our lab that
was published 20 years ago, and we decide whether
we take it seriously or not what we want to
do with it. We are acting as peer reviewers
in the application of our own research to do new
things with it. None of that is accessible to
anybody else except for us. So if we can
move that process of what is the natural evaluation of
research for research purposes to a public discussion, then
I think we will meet better and more efficiently that
ongoing peer review some things that are taken very seriously
at first. Over time, I'm not taken seriously
at all, and vice versa. Right plate tectonics
theory. Everybody said this is absurd and then Nope
, that's how it works. So there's lots of
things that happen in evaluation and change over time of
the same stuff, and that should be part of
Yeah, um, is what you just said Something
about You have a paper online. It's 20 years
, 20 years old, and you want to see
if it's use formula and you can then see the
comment. Okay? Yeah. Yes, here with
years. So this is like, it's open evaluation
. That's ongoing evaluation, right? You can imagine
an entire enriched service. Uh, that is just
commenting and discussion on articles on claims. Uh,
that also shows its evolution over time. Right?
So representing the peer review process on a timeline,
right? Make it simple, right? Make it
yelp ratings. Everybody gives a star rating, uh
, to each article. And then you look over
time. How many stars does it get? You
know, is it five stars in 2000 and seven
, but then down to a three star and then
back up? That's not how it's going. Uh
, so we have a commenting engine. That's a
very early commenting engine in OSF, uh, for
OSF projects. Uh, so but we're ingesting now
everything through a notification service. So l s f
should be able to ingest all of science, uh
, and then represent that for open evaluation of it
. Right. Pub Med Central also has a commenting
in so many other places. It doesn't need to
be us. Uh, we'll link to all of
those and try to integrate them together so that wherever
you do your commenting, uh, there can be
a place in which that all gets consolidated or at
least connected. I've exhausted you. Okay. Thank
you for your time. Appreciate a few stickers if
you want stickers.