[Music]
[Jeff Spies] I am excited to be here. I'm a big fan of
the library here, I know of a lot of the folks.
It really is a great place. I'm going to put a little different
slant on my talk than I usually do, so I hope that goes well.
It might be a little darker than usual, we'll see.
So you heard about my background. This is the way that I think
probably best describes what I do.
So when I think about why I got into science, why I got into research,
technology, the stats, it's because I want to help people.
I grew up with this idea of wanting to
be a doctor. My dad's a doctor, my sister is now a
doctor. My mom likes to joke that she went
through medical school twice. I have this affinity of just wanting
to help people -- that's I think
probably it's probably a common goal from a lot
of scholars, a lot of people in this room. I want to
find ways that the work I do maybe not directly like in the like real
doctors do
reduce suffering save lives
and so i want to just put a question out there
you know if i'm doing this science this basic science creating this technology
which should help other people do their science do their scholarship
how many lives can i help make better how many lives could i save
and we need to think about this cumulatively
so together what is the the effect maybe i can't save anybody but i bet
with a few people in the room we definitely could maybe i can't save
10 people or 100 people a thousand people
but with people in the room i bet our numbers are in that range
hopefully i think i don't know exponentially i hope i make an
exponential impact again the software i write the technology i develop the
methods i develop i hope that helps other people help
other people i hope the students i train go on and
make an impact that small world network should
see itself out in this exponential distribution
the scaling model and i want to think about this longitudinally
over time it's not just one paper that i write it's not just one data set i put
together it's not just one piece of software right
it's my my my careers effort
okay so go back to the usual presentation
Center for Open Science is a nonprofit in Charlottesville, Virginia.
Our mission is to increase openness, integrity, and reproducibility of scholarship.
For me this is in service of research efficiency quality innovation inclusion
and diversity and these latter attributes really serve
efficiency openness is not in the service of
openness diversity is not in the service of diversity in and of itself it's to
make research more efficient it's to make it higher quality it's to
make it more innovative
everything we do at the center is free and open
if you hear about anything any of our services during the talk
you can go online and find out how to access them
and we're lucky to be able to have this type of a model
to what we do
because we're lucky to have generous funders and we have very generous
funders
so the problem that we're really dealing with is this gap between scholarly
values and scholarly practices
we believe in openness i don't think there's any scholar on campus
that doesn't believe in the concept of openness of transparency of inclusivity
of having their work made available to others having knowledge be public and be
accumulated and added upon these are definitional attributes of
science of research of scholarship
but sometimes gap forms between the values that i endorse and
believe in in my practices what i do on a
day-to-day basis
and a lot of this has to do with the incentives the incentives for
individual success are focused on getting published
not on being accurate not on getting it right
this is a problem because then what i do in my work has this bias to it
it's not because i'm i'm evil it's not because i'm bad
i'm not doing this intentionally mostly but my work will be skewed towards what
is publishable because i need to be published in order
to maintain a career to get grants to get tenure to get have
students do that work to make that impact that i
originally wanted to make i had a cold last week so you're gonna probably
hear me cough and and whatnot during this
i'll try not to um so i do these things my this this gap is
there because i'm human
we're easily influenced by these biases most of these things i don't even
recognize that's an important thing to note
and if it's not if it's not focused on getting it right it's also not focused
on rigor reproducibility replicability reuse
and extension again these definitional attributes of science of research of
scholarship
so what am i biased towards as i conduct my work
well i'm biased towards producing novel results positive results
significant results clean results very clean narratives confirmatory
narratives because that's what i know will get
published
i'm going to try to spin negative results
to be positive i'm going to dig and dig and dig until they're not negative
anymore or i'm probably not going to mention them
if the story is not clean i'm going to clean it up
and i'm probably not even going to do replications or if i do i'm not going to
call them replications i'll call them a pilot study
and it probably won't get published they'll probably sit in the file drawer
i don't think i'm alone
anderson and her colleagues asked scholars what they endorsed
and so these these these were norms or counter norms
so things like being open or closed evaluating research on its merit
or by reputation of this of the researcher
motivated by knowledge and discovery or competition
considering all evidence even if it works against one's career
versus just promoting one's own findings quality versus quantity
and so what she found is that overwhelmingly
people endorse the values that's the gray bar the light gray bar
the hashed bar is how many of how many people endorsed an equal number
of the values the the norms and the counter norms
so not many and the black bar is how many people endorsed more counter
norms than norms so again i think most people
believe in these values they're probably not here if they don't
they'd be making a lot more money elsewhere i'm guessing
than in academia
but what do they do in practice well that gray bar gets a little smaller
the hash bar gets a little wider and you see more
more in that black region
in practice we fudge a little bit
we cut corners we sell exploratory uh research as confirmatory
and then the kicker what do your peers do
or your peers lie cheat and steal and these this is your competition and
so again we're already biased
by publication by trying to promote our careers by trying to get ahead
to make an impact
and then we feel like all of our peers are doing all of this negative stuff
and this impacts scholarship this impacts what we know about the world
so one of the projects that the center conducted was the reproducibility
project in psychology
we replicated 100 studies from three journals
we found that in the in the original studies
97 percent of those had significant p-values
p-values less than 0.05 positive results
that thing that publishers want
in the replications the distribution looked very different
this doesn't say much necessarily about reproducibility or technically
replicability we don't know why these things didn't
replicate
but they didn't it's probably in part due to this publication
bias this cutting of corners this p hacking i
want that p of less than 0.05 and i i hack at it
until i get there
we're doing that study by the way in cancer biology
and some of those findings are starting to to
come out
so open is the solution that's a claim i'm going to make i'm going to try to
support that but that that is a claim and i think
we're making progress as a as a as a culture we're taking steps
towards openness here's one example this very small
example badges these silly little images
that get attached to journal publications if someone for example
makes their data open
so you see these badges they're just on the website they're on the article
they're not even linked to anything there's a picture
of this badge
and we looked at what was the result of this after adoption by a journal
compared to other journals what did that badge do
what we saw in psychological science was that after adoption we had a 10-fold
increase in data sharing compared to these other journals
we increased data sharing by 10-fold with a little badge
so what this is i think it's probably a little bit of gamification
you have a badge now i need a badge i don't want my
journal art to be there without a badge on it
but it's also a way to signal what our what our values are
i i believe in openness and now i have that on my paper
and maybe i'll i'll want that on my next paper and other people can see that now
so now there's accountability that develops across this
there's a culture that can develop with this
okay so here's where i reveal
something about myself of late
i've been feeling a little impatient i've been feeling urgency
i think we're moving a little too slow and this has been building up for a
while talking stakeholders who don't know how to move forward who
don't feel like they can do anything about this
and just i mean the state of the world is one where where you know there's
we know there's big problems we know we have problems to deal with
we know we have problems to deal with in science and scholarship those are the
things we work on
but could it be more efficient
so i a couple weeks ago i was invited to be on a panel at the gates foundation
grand challenges meeting and it's an interesting meeting because
along with researchers they bring in practitioners and policy makers and
entrepreneurs from developing countries so it's a really
interesting mix of people and when the gates foundation talks
about grand challenges they talk about grand challenges i mean big problems
uh if you want to take a look at this later this is i did not draw that by the
way that's their website uh uh
warren buffett who donated the largest gift in phil
philanthropic history to the gates foundation
asked you know what what do you do with my money what what what is the return on
my investment and so in their annual letter they wrote
a letter back to him and here's just one piece of that they
said the magic number is zero uh this is the number we're striving
towards every day at the foundation zero malaria zero tb zero hiv zero
malnutrition zero preventable deaths zero difference between the
health of a poor kid and every other kid that's that that is a big challenge that
is an amazing statement to put out there in terms of what their values are and
what they plan to to attempt to do and they say it's not
out of this world to do that in 1988 when the world health assembly
decided to eradicate polio there were over 300 000 cases hundred
thousand deaths per year
now with the support of folks like the gates foundation
they're down to thirty seven not thirty seven thousand
thirty seven cases of polio per year and of course the gates foundation wants
that to be zero we can make change this stuff is working
so i was i was inspired by this meeting
and i started meeting people and a lot of people
dealing with problems like this on a day to day basis
i met a guy who
is developing a non-invasive way to detect cervical cancer
in india
they suffer disproportionately from from this cancer
than in other nations any poor nation suffers disproportionately it's it's
entirely preventable with proper screening screening that
exists uh uh um it's not as easy as it is it
could be as is not as cheap as it could be etc so
he's developed this device to make this easy
to make this cheap to make this affordable and accessible
but they're 74 thousand women dying annually in india
from cervical cancer that could be prevented
one in 53 women will get cervical cancer in india
in other countries it's one in 100.
i'm another guy who's working in india to
put just very simple primary care offices in neighborhoods and then cut
the price of those primary care services in half
simple services checkups um uh sort of the
initial visits when you start feeling sick
because in india
44 of the population
is over 50 kilometers away
from well-resourced hospitals
people could have to drive for hours upon having
symptoms of a heart attack
or if you if you have what you think could be a cold but
really could be something else you're probably just going to think it's a cold
and move on
and so he thinks that by by building these primary care facilities
you can get preventative medicine up to a point where
you don't need those well-resourced hospitals as much
i talked to a another person working in policy
evidence-based policy
and he said he had been trying to convince a west african government
that ebola was not a central african disease it was not a central african
problem it could come to west africa
2014 the 2016 we saw that
and when you told me this i remember a conversation that i had
overheard at lunch the day before i sat down on a table and they were in
in a conversation they were talking about migration patterns across
africa and they mentioned that the work was being published it was in the
process of being published i don't know if it was in press or or
under review but it wasn't available
the the person that the gentleman who was talking
i told the woman he was talking to that he would let her know when it was
when when it was available
sort of all of this all of my experience in this space of
talking to journals and and institutions and
funders and whatnot it sort of clicked that this is
this is a this is a this is a big problem
people are dying people died what if he needed that to make the case what if
what what if that wasn't available when he
needed it to make the case that these western african governments
need to prepare
these are real lives this is real suffering on the line
and i want my work to be rigorous reproducible replicable reusable and
extendable because of this i started with that statement that i
wanted to try to help things like this but it's not
even though we are making progress i want this the slope of these
lines to be better i want deaths to to decrease faster i
want vaccines to increase faster these are all the images that the gates
wrote in their letter to warren buffett
so how many lives can i help make better how many lives can i save
there seemed to be another question that could go along with this
how many lives am i all right with not saving
because i'm okay with this perhaps two year publication process
i've talked to several funders this started sort of my frustration my
mother's this impatience who knew about good work that had been
in the peer review process for years that other people were waiting for that
would change the space and even the funder couldn't convince
them to make it available sooner and so when we
asked this question with that same approach
well maybe that one paper maybe i shouldn't blame myself that i'm
that people are dying because i i i want to publish in a
in a certain journal but as an ecosystem how many lives are we all right
not saving how much suffering how much how many people are we all right not
helping
how much is that publication worth
so we know the price of the cost of a publication the library especially knows
the cost of a publication they pay for it you can run into a
paywall right now and see what the the rental price for publication is
here's a survey done of chinese universities
on average a professor in china makes 8 600
a new professor makes about 3 100 u.s dollars
the equivalent this is a survey of 100 universities
for example the bonus they get to publish in a nature or science say top
impact journal is 43 000
that's the average some institutions paid as much as 165
000
we have a skewed incentive structure i don't know if this is the system that
i would invent if i was starting from scratch
i think i would be a bit embarrassed to tell that person
who is trying to convince the government that ebola could be in our at our front
steps that hey i i need to get this i need to
get this publication it's good for my cv it'll help me help
other people in the long run i don't know if that's a case i can make
it's not just china these cash bonuses exist in the u.s
i've heard of cash bonuses in you know in the tens of thousands of dollars
uh uh there's a a little science or nature commentary
uh a survey done quite a few universities were in the
thousands of dollars they're usually management
schools business schools but this does happen
but beyond that my professional success is derived from my the number of
publications and where i publish i'm not picking on the uc healthcare
system because this happens everywhere but this is one
quantitative approach to this where they really looked at this
for every 10 publications from physicians in the uc healthcare system
there was an increase 2.4 percent increase in salary
controlling for specialty institution rank and share
this is your salary this is this is how much publications are worth
so then the dark question that i think perhaps we should ask how many lives is
a science or nature publication worth how many lives is 10 publications worth
it's a darker question i think they are a little too dark
uh uh but it's a dark question but i think one that that falls from this
logically
malaria accounts for 429 000 deaths worldwide 92 percent
in africa if i could decrease that by one percent if we could
decrease that by one percent we'd say four thousand lives
point one percent four hundred lives point o one percent forty two lives .001
percent four lives let's say we're down in the 0.0001 range
you know uh very small i can't do much myself
then together where would we be if the system was a little more efficient
a little higher quality a little more rigorous a little more replicable a
little more reproducible
and the sad question here is that how many lives is that worth how many
african lives is worth that's a horrible question to ask and of
course they work the same to you they're worth
the same i'm sure but as a culture i think we know
they're probably not worth the same and this is this bias that has crept in
that actually exists that when we look at it this way
is something to think about so i want my work to be rigorous reproducible replica
reusable extendable because of this but it's not
and so what do i do well i can't do a whole lot and that's
i'll i'll tell you that now
and this is a tough problem and i don't know if the weight is on us individually
i don't know if we can carry this weight i had this sort of
this ball in my throat at this gates meeting just
like what are we doing why aren't we moving faster
we know some of these things are problematic why aren't we just changing
them and then thinking about well the comment
is typically institutions change slowly by definition
they're meant to do to do that i don't know again if that's
a that's something i could tell someone dealing with this day to day
but i don't have to deal with it to stay today i was just at a at a
fancy hotel in washington dc uh getting fed good meals
they let us into some of the smithsonians at night uh while we had
dinner uh after hours were able to walk around
the museums uh i don't deal with this day today i
just i just went to this event i have a lot of privilege i don't really
understand what this is like
and i understand that i'm part of that problem
and as soon as i got home i got home to about
being gone for seven days i got home to email
and that that that that knot in my throat started to dissipate a little bit
i didn't feel it as much anymore
i had to deal with all these emails i was busy uh
i got a cold like i said this weekend uh and was was in pretty bad shape on on
friday i and i i did a little bit of work
but uh i also binged watched uh forged in fire has anybody seen that
no oh wow you need to watch the show it is really it is if
you don't think a show about bladesmithing
could be just outstanding you will be surprised
but this is that psychology at work this is what happens maybe and it may happen
with those people working day to day this may be a protective mechanism
but i think we need to go back to this on occasion and think about that
question as we really try to inspire our work
others work and think about how important are these changes
how is important that we do change this culture this slow moving culture
we don't always have that luxury to do it directly i'm not a funder i can't
mandate change
i'm not an institution at a university where i can change a 10-year process
so i'll give you a few
ideas a few directions that we're moving things that we think we can do that will
result in change and again if we think about our
cumulative effort if a lot of us do this from this
bottom-up approach we may be able to actually make bigger
change than we would we would expect
so this is the research workflow this is what i have to do to get those jobs to
get those grants to get those cash balances evidently
to get around this life cycle as as often
as possible and as quickly as possible being as as high as as high of impact of
journal as possible
what we typically do when we talk about solutions like openness
like open access policies for example they're appended to the workflow you're
telling me that after i do this after i do all my
work and publish then make something open
to me who's inundated by email who has a paper to get out the door
who has developers to to talk to about projects
who has grants to write presentations to prep i just
i don't have the time because all i need to do is publish
i'll do it after i publish my next paper maybe
we know how that goes we know how easy it is to slip into that
that train of thought this is the same with reproducibility the same with
curation same with preservation the same with
reporting the same with the the integration of new
tools
hey you can you can use r you don't know how to program we'll learn how to
program and use r don't know the command line we'll learn
that too and then use r it takes a big investment if i'm already
successful getting around this life cycle
because the researchers that are in academia probably have a
degree of success they already know how to do this
they have a workflow that works and this goes for people as well
in the libraries for example i think we append our services
to the like to the life cycle come to come to my office for an hour
i'll tell you how to write a data dictionary you go and do that
come back we'll review it you go back do some back and forth on it
and then you'll have your data ready to to put in the institutional repository
again to me in my experience as a student working on my dissertation
that seemed like a lot of work i had to walk to the library first of all
i had to do this office hour that i really didn't understand even
why data dictionary was important or what it was i don't have expertise in
that necessarily i don't care about sharing or curation
from at least my professional success point of view so we have to
think about a different paradigm and that paradigm is research workflow
integration not appending to the process but
integrating with the process so the osf the open science framework is
a service i'll show you and the idea is that we want to close
these gaps we're going to fill these gaps when we bring them closer together
make the barriers of getting from one vertical to another as smooth and
seamless as possible
so you can go online today and register it's free
completely free it's all open source it's available at osf.io it's a content
management collaboration management project management workflow management
platform for organizing sharing work you can use
it privately with a lab it just gets you out of email
uh has value in up in its in in that way in
and of itself but if you decide you want to make
certain things open you can we don't mandate it but we make it easy
to do to do that and the principles here are meeting
users where they are i need to respect that
you're busy you have you have incentives at play
you have things you care about you have expertise
and i might need to come to you
if i if i want to cause change we need to respect those incentives they
are strong forces a lot of times again we don't
know the influence of them
but they do they do change the way we do our work
we respect current workflow respect that people are already successful
in their in their own workflows they know how to get publications why would
they change that
but this is where researchers are finding their day-to-day success
if i can make a case for any of these things being valuable
knowing that i can't immediately change the incentive structures
it's probably here
if i can show them that they could be more successful
and so we could demonstrate that this makes research more efficient
of higher quality more accessible saves more lives we can make a value
statement about it
better we could demonstrate that researchers will get published
more often align with their current incentives
even better we can make it easy
and the best make it automatic researchers don't even have to care
about it they don't have to learn how to do it they don't have to care about it
it just happens automatically and they get that value and that over
time is they start seeing the value and what that
implementation that feature that service that person was
then perhaps they'll they'll integrate it more completely into their workflow
they'll take the time to really learn r and move all of their scripts over
for example or they'll contact the library before
they start collecting data to get a plan in place
so it could look like this preservation is a good example
if people are working with files in the system already
and they're adding metadata as they collaborate telling people what's what
why couldn't preservation happen automatically and maybe it's not the
type of preservation we talk about in the library
where it has this rich metadata this linked data
but if it just enters into a preservation
environment we can ask researchers for that later
we can try to do that work ourselves we can try to do that work automatically
machines are getting pretty smart google demonstrates that you can you can
do a lot with unstructured data these things are coming down the
pipeline but if not well maybe that's what maybe
this is an occasion to to uh uh say hey i saw your i saw your
work online i saw it's being sent to our system it
seems like it would be really really important for other people to have
access to in fact already 20 people downloaded it you know
what we found is that uh by adding a little bit of
metadata this work becomes much more discoverable
more people would cite it
again we're aligning with their incentives we're showing them value
and integrating in their workflow so maybe you start by just asking a few
questions i don't understand what this column label is
if you just send me a quick note of what it is i'll add it to the data dictionary
for you and then we'll perhaps talk later i
think there's an incremental approach here that can be used
version control uh in the usf this happens automatically
if you store a file with us it's automatically versioned you don't have
to think about it you don't know git or github it just happens
i you should learn git and github those are good tools to use
much better version control systems than we offer but this does the job
you get an immutable copy of that data you can go back and look at other
versions you can make sure that that it has not changed by looking at hashes and
these other other things that we in the information
space sort of care about but if you don't it's an easy backup
right now i can't tell you i almost had to pull my
slides off there this morning when i thought i over
wrote them it happens all the time and i just go back and go to the last version
and download it and there it is and i know how to use git i'm a computer
scientist i know i know these systems pretty well
but they're not easy it takes a lot of time
okay here's another example don't take this one too seriously
but what if there was a interface that perhaps popped up
when you did something and doesn't have to be clippy necessarily
but it could say for example it looks like you've uploaded a data file
maybe it detected that because it's a square file a rectangular file
a csv file perhaps we it looks like you've uploaded this file
with social security numbers in id column
how would you like to proceed do you identify warn me before sharing this
cancel okay so social security numbers are easy to
detect with regular expressions at this computers can do this very
easily and all of a sudden now we have a research workflow integrated approach
to bringing in this compliance concept to the researcher that could say
delete the column it could say add anonymous identifiers it could say
contact a librarian but now we're at the place that it matters
they're uploading the data this is the time to make that change
not at the end when i'm making it public and forgotten about what i've done
forgotten that we even use social security numbers where the grad student
left before they were able to actually go back and de-identify everything
now's the time to do it
here's another real world example that meeting users where they are
if you use drop drop box right now use google drive
and and these other very nice products fine
use them we'll have you in the same system getting those same
uh hopefully getting that same value from the system
even with these other tools and right now we integrate with uh
quite a few a few of these types of tools mostly in storage
some and analytics some preservation some in uh
citation management we're adding more so again meeting them where they are
respecting that they already have a workflow
i can't just tell them use this new system i want to make this as easy as
possible for people and the benefit of this is that it
respects another
concept that we forget and that's research is hard
and it's this context it's all of this stuff before the paper's published that
actually matters that's what tells me what was
done that's what tells me where there
potentially could be these biases that would that's what tells me where
some might have made a mistake very benign mistake in a data clean
script
so if we take this workflow approach not only do we get that access
to the researcher we get access to the actual scholarship
and so if we seek to facilitate these these things
these definitional attributes of science of research of scholarship
we need to move beyond descriptions of outcomes
just those reports to descriptions of process
or better sharing of the actual process
and we can keep having these revolutions this open access revolution the open
data revolution next we'll be open materials and open data cleaning scripts
and open software and open protocols et cetera et cetera we can cut
to the chase we can say it's all important otherwise
we wouldn't do it we know researchers would not be doing this it
wasn't important if it didn't help them achieve that
outcome they wouldn't be doing that so we can say it's all important we
respect all of it
all of these pieces make up the actual work it's not just that that
top circle there that is not the scholarship it is the it is that
life cycle is that workflow that is the scholarship
and there's another benefit this open workflow approach makes it easy to
collaborate integrate expertise and give credit
so there's all this value in the library for example
that's just not part of the research process right now but that should be
the researchers don't have that expertise but the library does
we should respect that just as much as we respect
you know that final report people should be able to have careers as software
developers as statisticians in research and be just
as respected as someone who puts out papers every year
same goes for data collectors why aren't there people who just specializing
being really good at collecting data
it's how businesses would do this it's a scalable way of doing things you have
expertise you you use that expertise instead right now we
have to be jacks of all trades you have to do it all if you want to be
a successful researcher that's not realistic that's especially
not realistic with the big problems that we have to solve
these big problems require big science big research
and i know big science is an inflated term
it's an overloaded term there's a reason why we call that big
science i think we need to take this term back
or at least come up with a new term that's similar to this
to solve those big problems we needed a lot of people
unfortunately at the time
so we needed a lot of resources for those people and then we needed a lot of
management of the people then managing the resources this created
these vertical structures and these big labs and that's the big science we think
about in a somewhat negative way today
we need to think big and i think we need to think big like big data
my favorite definition of big data of big in big data
is that because of its size it requires new tools methods and paradigms
that's it if you have a gigabyte file and you're having a hard time with it
that is big data if i have a super computing cluster uh
or a computing cluster or a supercomputer
that i have access to and i have a petabyte of data and that's
troublesome for me that's big data but either way
it's important that we have to come up with new ways of doing things because of
the size because of the scale so we can call it
team science or team research network science open science urgent science
whatever we want but this is very important especially
when we think back to how we have to solve these big
problems we need we need as many people on these problems as possible
we need that diversity we don't need diversity for diversity's sake we need
that diversity because we'll make that scholarship better it will make
that research more efficient
and so we need a place for these people to be have access to to discover and
then to get involved with and again this open workflow approach
just just provides that
okay so i talked a lot about journals and you know i i thinking back to that
that frustration uh i i certainly am frustrated in
thinking that you know a lot of this uh is in support
of a pretty exploitive business lock-in
vendor lock-in based business model that is increasingly uh charging
increasing prices to libraries to the point that we can't really even afford
them anymore that not only locks us into these
subscriptions but locks people out of of access to them
that keeps this what should be public knowledge away from
most of the world even in the us there are many schools that just can't
afford these things
even big research institutions have made statements that they can't afford these
increasing costs
so there's a better way to do to do publications
we need to move away from these vendor lock-in models
but there's an easy thing that we can do today we can support this concept of
pre-prints before you submit your peer review
article or your article for peer review deposit a preprint archive is the best
example of this bioarchive ssr and there's other preprint servers
we at osf have one uh it looks like a pre-print server
you can store your reports there
it's built on top of the open science framework you get all of the values of
that system but if you just want to start by by drag
and dropping a pdf onto the onto the uh screen and adding a little
bit of information you can and now we've at least achieved
that rapid dissemination it's not getting held up in peer review
or in long publication process or going to you
know a high impact journal and then going down the ladder
uh reject reject reject reject reject until i finally hit one that
takes the paper at least it's out there at least
it's it's it's fair it's findable it's accessible
we need to move beyond pdfs if we want to be interoperable
at least it's there
and we're supporting communities that want to want to promote these
so we're building the infrastructure but there's people that are passionate about
this that feel this urgency that want to see
this rapid dissemination of knowledge and they know how to talk to their
communities so this is how we're scaling our services
at social archive philip cohen knows knows how to speak to sociologists he
knows them he knows what what challenges they face
he knows how to work with them and so he is the perfect person to work in that
community to talk about preprints in sociology and
so we we set up so sharkkai for philip and we've done this over and over again
and we continue doing this this list is growing and growing
and this is the scale that we can try to achieve by partnering with
people in these disciplines with experts outside of
our domains at least then just some other general comments we
can support open access open output and open workflow policies
i've heard of open access policies i've heard of one open output policy
which is sort of neat a different way of thinking about
extending it to more than just the the document the report but i think we
need to get to open workflow policies another important piece of all this is
you know we are locked in and we just keep making it worse and worse and worse
we have to move away from from signing these copyright assignment agreements
we we cannot continue signing these exclusive licensing agreements
if we value access if we value people in developing countries having
this knowledge to actually change things themselves
they're already working hard to do it if we want to give them the information to
make that more efficient to change the the the the slope of those
curves we need we can't get into these these
lock-in based contracts
and we need to state what our values are we need to state them explicitly
we need to make those statements public and transparent
and then we need to facilitate and resource practices that align with those
values sometimes it's easier said than done
sometimes it does feel like things are moving slow but progress is happening
and i think it can happen it has to happen in this multi-faceted
way this multi-stakeholder way funders societies journals institutions
researchers
so again i apologize if this content was a little dark
this question was a little dark i'll leave you with one more suggestion
and that is to at your earliest convenience
watch fortune fire it really is a pretty good
show thank you
we have time for questions
so when we think about open workflows we're thinking about how to apply open
workflow to science
when you think about are there existing like bench processes
that you might apply across sciences and are there standard processes that
you might incorporate within an open workflow you can say well we use
this process with this process with this process but then you get into
always laying back on like a standard
uh and there's elements that show up in other discussions around these topics
about you know the role of exploration and creativity in in this
and i think that that has to be respected that has to be maintained in
fact most of research is that creative exploratory
process we just pitch it as confirmatory that's just
what publishers want so that's what we give them
so what do we do first of all so can we can we develop those standards i
think yes some disciplines are more than others the workflow is very
very custom very personalized if i have one piece of hardware that you
don't have or one piece of software that you don't use our workflows seem
like they're worlds apart and part of the issue is that all of these things
need to work together that's why trying to close those gaps is
very important it's why this idea of connecting the
workflow abstracting api so that things talk to
each other in an interoperable way is so important
because once they can start working together we could envision a point where
you know you click a button and things just start going
the data just starts getting collected it starts getting processed in a certain
way it starts getting analyzed all the
provenance is tracked all these things happen in that standard way
and then we can really very efficiently look at things like reproducibility
replicability more than that it allows us to to
extend the work in a much easier manner so people are often surprised that the
pharmaceutical industry actually is quite supportive of
replication efforts they spend a lot of money
developing out drugs and therapies
based on on more basic science that just doesn't replicate and they'll spend
millions and millions of dollars trying and trying and it doesn't replicate
but if they were able to very quickly bring that in
get whatever they need to get as the output or just show that
it's valid it would save a lot of money early on
if i'm trying to extend your work probably one of the core values of
science again is standing on the shoulders of giants and it was a
misnomer uh but that's a you know it's a it's a
good concept uh i if i'm to extend your work
i need to know how it works i need to make sure it works i often do do some
pilot of it just make sure it works in some way
and so why not formalize that so that's but that's never ever to say
that the exploration isn't important it's just let's formalize this piece of
it let's state clearly what it is this is
this is replication or this is a confirmatory
study and then this is exploratory and then
both are respected
as as essential to the process instead of well this one's what's
respected but we all sort of know what's really
this that that doesn't help you know the the accuracy of our corpus
of knowledge
don't we have very reproducible workflows
you're looking forward to exploration but standing on
all of the things like units and measures and astm
standards that we don't even realize are there because
we just use them all the time so there is a vast amount that is standardized
and we just take it for granted but if we didn't have this we could have a
problem yeah we we do have uh some great uh
organizations like like likeness that they're doing these things
uh um there are communities that really are promoting the formalization of this
stuff i think one of the problems is that it
often happens the use of those things happens in a
siloed way and i think that extends from our
culture a very siloed culture but i might have a workflow for one piece of
the the the process but then what happens after
that and that's where that's where there's
now room for for me to sort of insert my biases where i now have some
some some ways to change things uh and so i
think it's we need to continue this process thinking about it holistically
as a workflow well now if i'm developing a standard
for this let's make sure i i take care of how this meets you know
what is coming into it and going out of it
let's take care of the inputs and outputs as well
that that sort of thinking might help those that interface design might help
in thinking and pushing this this goal of making
this more seamless more uh easy and automatic
what are the journals that are currently kind of the gauntlet
that researchers have to go through but what does this do
to that relationship
so if we agree that that these this lock-in closed access
style of business model is not aligned with our values
then i think we you know if we can if we can uh really
think about that urgency and and and and try to make movements and as as
stakeholders i think that we need to to work with
them to either try to change or get out of the way
uh there's there's other models of publication that that
can be tested uh and this is this is one area where
um uh you know we shouldn't just do this willy-nilly
let's let's take a scholarly approach to this let's let's
study this you know using our scholarly methods
uh let's let's try this let's look at the peer review process let's see if we
can make it more efficient see if we can uh remove some of the biases the concept
of registered reports is a good one where you move peer review from at the
end to the beginning so you're reviewed for your your methods
your idea and your methods not on the outcome
and then as long as you collect the data in the way you said you're going to and
analyzing the way you said you're going to
you're published so it removes all this this
this bias of me trying to get outcomes that align with what gets
me that publication plus it gives me a good uh valuable
insight before i actually start collecting the data so we
need to try stuff like this and see if it works
right now we can't do any of that we can't even study the current system
because everything is so closed and locked in
uh and so i think it's if we if we think that that you know we could be doing
better we could be moving faster we could be
uh doing higher quality work more reproducible work more replicable work
bringing in more people more diversity i think we need to start uh
you know making some choices and those choices are going to be is this the
model that we want to maintain or do we want to
push for something new something better
but we should work with journals i mean we should try to work with them
some are are trying to make progress looking at some of these things there
are some progressive groups but this this
vendor lock-in business model is is is pretty prevalent
uh we need to move past that pretty quickly
no you touched on what i was
thinking about there's some other things that i still need
i'll deal with qualitative data and researchers in that domain
yeah and how we avoid imperialism from this model or what we perceive
imperialism to what they do how they perceive what they do
yeah and and i can picture that hypothetically a provost might
embrace the open size framework and it doesn't fit with a large
part of what the faculty and graduate students in the university
perceive their doing and they perceive they're stuck and marginalized
yeah we shouldn't have called it the open science framework
we shouldn't have used the word open or science most likely
you don't have to be open to use it uh and it's not just for for science
uh it's actually a pretty flexible and robust system for many types of
workflows but as we think about these new
paradigms and these new models new tools i think we just need to be clear about
what they're for this is for confirmatory research this
is for exploratory research this is for qualitative work this is for
quantitative work this is for mixed methods work
none is better than the other necessarily
but we need to state clearly what they are and respect the fact that
they all have have a pardon in the process
uh uh john tukey the father of exploratory data analysis
said that we have to wear multiple hats we have to be
sometimes the the you know the investigator the detective
we have to be the judge but to be the jury we have to play these multiple
roles and they're all important but just saying one is important but
doing the others and not stating it it creates statistical biases the
literature is biased at a quantitative level because of this
uh it's it's it's somewhat subtle somewhat not intuitive
um but we we need to really start saying what is what
it does make a difference for how how we interpret what what
comes out of that research but you make a good point i think this is this is one
of these roles
for perhaps the library of uh you know really facilitation
education support
communication how do we communicate these things effectively
this is a scholarly communication problem i think the libraries
are well positioned to start helping with some of those things
and we don't have to buy into it wholesale i mean we don't just
jump in and do it all at once we can make incremental steps
i just think it's important that we start i think it's it's urgent that we
start
thank you very much
you