the director of publishing strategy at University Libraries. Peter
Potter, who is going to moderate tonight's discussion and
introduce our panelists. Peter. Um Thanks everybody.
Uh again for joining us. I'm just going to
get very brief introductions. Um you all know many
of these people, but I just want to quick
intro. So before we get started and we have
someone you know, Daniel Chen on our panel um
and engage the candidate genetics, bioinformatics and computational biology
at Virginia Tech. Um He has current interests are
in understanding changing attitudes within social networks and performing analytics
for precision and when he's not doing all of these
things, he's a volunteer instructors for software and data
and Dean DePauw needs no introduction to this group,
but I'll just say that she is vice president and
Dean for graduate education. In fact, she also
holds academic appointments and the department of Sociology and Human
nutrition Food Society. She is nationally recognized for her
tireless commitment to improving graduate education as indicated by this
very class. And most recently she was honored by
the Council of Graduate School reversed Debra Stewart Award for
Outstanding leadership and Graduate education. Um Sally Morton is
here a second to my right. Dean Morton has
been dean at the College of Science for nearly a
year. She had a number of positions before coming
to Virginia Tech, most recently at the University of
Pittsburgh, or she was professor and chair of the
department of biostatistics in the graduate school help. She
has more than 30 years of experience and academic and
research settings focusing on patient centered comparative effectiveness and evidence
synthesis. Dean Morton remains active in the field of
biostatistics. Most recently co editing a book entitled methods
and comparative effectiveness research published by CRC Press just a
few months ago. David Radcliffe, to her right
, is professor of English at Virginia Tech. In
addition to his own particular interest in Spencer in the
romantic poets. He's been working on digital projects for
well over 25 years now. Mostly databases covering a
range of topics including watermarks and paper, medieval poetry
. Lord Byron and T. S Eliot. He's
currently working on historical documents containing demographic information that can
be extracted to shed light on social, political and
literary networks and early 19th century England and the big
in the bigger picture, Professor Radcliffe is interested in
building a digital infrastructure for scholarly information in the humanities
. Okay, next to uh Karen DePaul is Laura
Sands, who is professor of human development and a
member of the faculty in the Center for Gerontology at
Virginia Tech. Her research focuses on improving the efficiency
and effectiveness of care pathways for vulnerable older adults.
Dr. Sands came to tech from Purdue University where she
was Katherine Burke, professor of nursing and director of
research in the school of nursing. Just recently she
was named editor in chief of the journal Innovation and
aging, which is published by Oxford University Press.
It is a gerontology gerontological Society of America's first open
access journal. So not only is she starting a
new journal, she's starting one with an entirely new
business model. So that's a huge undertaking and I'm
sure she can shed some light on that tonight,
Jonathan Petters to my right is a data management consultant
here at Virginia Tech in the University of Libraries by
training. He is an atmospheric scientist with several years
of research programming and educational experience before coming to tech
last august he was a data management consultant at Johns
Hopkins in the libraries. Mhm. And before that
he was a fellow at the U. S.
Department of Energy where he investigated approaches to digital data
management and analysis techniques and scientific research. In short
, he's someone that all of you should know if
you ever need data management consulting. And it's an
area of real strength here in the libraries. So
Jonathan will start us off with a brief presentation,
gives some background on the key themes will be discussing
and after that we will open it up for discussion
. All right, Greg I just stand right here
. Does that sound good? All right. So
this is the topic for today? Open research,
open data transparency, sharing and reproducibility and scholarship.
I think I have seven slides. So this will
be short and be sure as possible and get your
hearing. Whatever everyone else here has to say so
in the beginning Or okay. Not in the beginning
but in the 17th century for many, many centuries
. The way we've been doing scholarly discourse journals and
books. This is the 1st 1st journal in England
philosophical transactions and Principia from Newton 21st Century The Future
? How does much scholarly discourse occur? Journals and
books? Step but not everything right. We now
have this thing that was invented a seventies, eighties
and nineties World Wide Web which allows us to share
a lot more things than just text. Clearly,
the main reason that was invented was for cat pictures
and puppies. But it does allow us to share
lots of other things we can share as opposed to
just text. We could share video, we could
share audio, we could share images, we can
share all kinds of different formats for research. And
this has been a disruptive influence since, since it
came into came into being. Uh huh. Not
only do we have a lot more data that's shareable
. There's a lot more of it. So,
we have all the all of us are now carrying
around super computers in our pockets. All kinds of
sensors and these things can grab all kinds of information
and can be used for all kinds of analysis.
We have a lot of data that's now available online
. So this is a database that David Radcliffe makes
available about Lord Byron and database of criticisms. Letters
and and other information about the virus. So hold
on to more data we can use. So how
are we going to make how we gonna make good
use of this information for for research purposes? And
uh we already mentioned the concept of software carpentry.
If you guys haven't heard of data and software carpentry
, they're fabulous. Uh fabulous things to help researchers
learn how to do good software and coding practices.
We're actually going to be bringing this online as well
in the library soon. It's coming on. All
right. So we have all we have all this
data now available. We can share all these other
research projects and things have become more computational intensive,
data intensive, digitally intensive. And that is that
has occurred in general. We've gotten to this kind
of position where you can almost see it in some
cases, a narrative that we read in a publication
in journals, more like this advertising, it's a
state it's a state of view. It gives you
some idea of what's going on, but it's not
the whole picture. So farina, we read a
journal article in some academic journal, we can understand
something about the conclusions and somehow got their literature background
. But not everything. There's other things that are
being done that we don't have access to the workflow
of the data and some maybe in some cases the
code. So if you could share all these things
, including the data, we may be in a
better position to understand more about what's going on the
research. So we can be in a position to
enhance the reproducibility or rec rec applicability of the research
. No more about what's going on. This has
become an issue in some fields, Uh biomedical research
. There have been several studies where people have tried
to reproduce the results from other papers and they've had
trouble doing so. Not necessarily saying that science is
bad, but there have been issues. What does
this mean if we can't recreate what's already been done
? Says this group at engines as a pharmaceutical company
that did this kind of study in 2011 was unable
to reproduce a lot of results, ironically, we're
talking about open data here. They have not released
the data from their from their reproducibility things. It's
kind of fun. So, but this has been
, has been a concern in some field. We
also have this issue of publication bias. Journalists typically
publish only novel ideas, hypothesis confirming things. So
there's a lot of research that maybe doesn't get into
journals that maybe abuse to other folks. One other
thing we can do if we share a bunch of
other research outputs as opposed to just the publications is
we can facilitate new research. So somebody could use
my research for meteorology. Maybe they can use it
in an adjacent discipline. If I made that data
and code available. Maybe somebody could use it in
ways I hadn't even thought or an entirely different distance
. No way to know the stuff is discoverable and
reusable. Maybe something can be done disparities. That
sounds like a wonderful world that will share everything and
new research will happen and everything will be great.
But there are plenty of barriers to this. Uh
Certainly there are technical barriers and setting up the infrastructure
to to share this stuff and usable formats uh with
the appropriate metadata, with the appropriate descriptions so people
can find it, discover it, make use of
it. But I actually I think that these are
these are more surmountable issues. We have other other
very thorny issues, legal issues, how to re
share this data without breaking the law without violating ethics
. Uh One example, I'll have, we're talking
about some folks here from biomedical research here, sharing
human subjects data is very hard. It's got a
lot of issues can be done, but there are
there are certain issues to overcome. We don't want
to violate the confidentiality of human subjects. That's a
bad thing. So how can we do this?
A couple of simple, a couple of ways I
can describe. Not simple ways is d identifying the
data. The human subjects say that like removing the
identifiers such as that data can be shared more widely
, potentially mitigating the risk of confidentiality issues. But
removing them is very difficult. It's not impossible.
We can also use data agreement data use agreements to
share this stuff in a legal framework from what maybe
perhaps one institution to another. Uh This can be
an onerous process. If you're going to have one
RB talk to another I. R. B.
Institutional review board already. So these are not easy
things to do very tight. They take time and
effort. There are cultural area just showing that academic
practices haven't changed, how we're sharing scholarly discourse hasn't
changed very much in the last couple centuries. So
academic practices don't always change very fast. There is
a concept of big science versus long tail research.
Big science. Think of astronomers who are all working
with satellite data that's coming from satellites, satellites are
incredibly expensive. Not every research group can buy a
satellite and put it up in the space. So
the communities get together and agree we're all going to
use our, use this data from the satellite and
have standards and protocols for doing so. You can
contrast that with what I call a long tail research
, which is like a researcher working in their individual
group, maybe in one office, in place like
this Virginia tech where the data more likely is being
kept within the group and it's not really going out
anywhere unless somebody else asked for it. This trust
built between between one researcher and another. It's less
likely to be to be shared more widely. And
there are disciplinary differences. So there's some there's some
places where in some disciplines where data sharing and trying
to make data available and we are all going to
use the same kind of datasets is something that happens
more frequently. So astronomy as an example, they've
been doing this more more widely, but there's other
ones or something. People have people I like to
call it data mining. Your data is mine.
My days. Mm. Okay. So how can
we how can we incentivize change? How can we
make a different world where we're sharing things? Well
, one way is to to elevate data code and
other research products up to the level of publications.
So we can cite data sets. There's a group
data site that helps link data publication. Uh,
so when we can cite the data sets, if
somebody uses my data in and work beside the data
as opposed to signing us in the publication that's Associates
. So, so setting up a system of credit
or researchers can get credit for doing this. Funders
are doing projects to try and get the data out
from underneath server servers and basements in places like this
. The NIH has this commons credit pilot model where
they're getting researchers to actually do their research on cloud
storage on cloud services like amazon. So the data
is one step closer to being more widely available.
Mm can have contests and has also funded a contest
on open data to get researchers to make the data
more widely accessible. Here comes another issue, though
. If you read the little byline down the bottom
hundreds of researchers picked the clinical trial data from a
major blood pressure study. To the dismay of some
who collected the information. So the researchers who collected
this data or a little nonplussed that they weren't able
to mind the data for some publications before it was
made available to the world. This is a concern
of scooping. If you heard the concept scooping and
for early career researchers like yourselves might be something to
be concerned about trying to establish your research, your
research career. Do you want to make everything available
as you're doing your research? Right? Maybe,
maybe not. I like the burger. I always
like to bring up max Planck. I don't have
exact quote manager of the exactly, but I will
use the corollary to it. He likes to say
that science advances one funeral or retirement at the time
. Uh, he is and I always like to
think academic practices tend to change very slowly. So
as older folks retiring out, younger folks have an
ability opportunity to change the way things are being done
uh in general, not saying everybody is not present
company. Finally, just a couple of the players
who are involved in this are a couple of non
governmental organizations want to mention who are really advocating for
science transparency and reproducible in science. Research Date Alliance
is an international organization that's their tagline research data sharing
without barriers. It's really cool organization. Were involved
in a little bit in the libraries. The Center
for Open Science, just up the street in Charlottesville
uh is all for transparency and openness and scholarship.
Have a wonderful program platform called the Open Science Framework
, which I recommend jail. Take a look at
sparked scholarly publishing and research coalition is a really nice
advocator with NBC for this kind of, this kind
of thing. Mhm. Then we can also talk
about two other big things. Funders funders have data
sharing requirements and data management requirements that they are slowly
rolling in and expecting researchers who they fund to do
. And clearly that's a big lever. Uh they
control the money, they can get people to change
a little bit of what they're doing. Actually worked
this market always at Hopkins. I worked with Smart
to do this. Make this resource group of all
the U. S. Federal data sharing requirements open
a nice little snazzy interface. I'll look up to
be able to look at for the right next organization
. And publishers of course can tell us, can
tell researchers, hey, if you are going to
published in this journal, Where is the data?
Underline us? And they're starting to do this in
2016. The end the end of the year,
nature started doing this data availability statement, whereas the
day they're going to be that underlies this publicly.
So clearly this is another uh lever publications are still
the currency I'd like to mention the libraries are in
position. We're here to try and help with some
of these thorny issues. We have a repository for
sharing data uh generated here at Virginia Tech. And
as Peter mentioned here to help researchers think through these
historical problems about how to share data and make it
around. So with that I'm in the same done
. I asked the question already, would you welcome
back if you would please? How far back?
No, no. Right there, just please.
Um because I'd like to have you talk a little
bit more about the Center for Open Science. I'm
not as familiar with the other ones, but I
know they have uh taken um a completely different.
I mean a different approach to where the data are
available at various stages. And the process is more
open rather than and I think it's a good example
because we think of doing our our research and then
having the data and open science has done something different
. What would you specifically about their platforms? So
the open science framework, they advocate for reliable things
, but specifically about this. So this is a
platform that's online where researchers of all kinds. And
we actually have, we start just a relationship with
them through Virginia Virginia Tech as well. You can
set up accounts on the Open Science Framework and create
what they call projects. And you can you can
actually store all of the information assisting with the project
online. So you're Associated Publications, your project plan
, your data, your code. And this is
a platform that allows for this to be put up
in a nice and structured way, structured way that's
appropriate for research and then you can give different people
access to it. So you can give your advisor
access to the same the same project and they could
collaborate with you, they could modify the files and
on the platform where you give access to other collaborators
and they have it in such a way where it's
once you're done with the project or as you're working
on the project, you could actually make this publicly
accessible the whole time. Or you can make it
accessible to certain certain segments of a certain people as
you're working on your research. And the same thing
with that comes credit model, I mentioned this when
you're doing this, the project is one step closer
to being made publicly available at the right time.
Whatever that is, you could be at the,
as soon as you start the project that I'm gonna
do everything openly or okay. I'm done with this
project. But now everything's nicely compacted and organized and
it's more clay clear closer to a position where I
could share it via data repository and make it publicly
accessible. So yes, yes. It's a process
orientation as opposed to just product of data, which
is, which is certainly a big problem. And
a lot of times when people say they want to
share data, I want to share my data.
They don't think about it until the end of the
project. You want to think about the very beginning
of the project, as we do at the end
. You gonna be like, boy, this is
gonna be really hard to get together new project and
you got your project. This is one of the
classes. Okay enough for me. Well, so
we did have so a suggestion was we had time
for discussion of course, but uh there was an
opening question that we share with you guys. Um
and this is the question and certainly you guys can
of course, I'm sure to say what I suppose
you'd like to say. But the question we shared
with the with the panelists here to let all of
them talk as opposed to meet. So one of
the main arguments in favor of open research and open
data is that the collective benefits of the research system
outweigh the drawbacks. And such benefits are the ones
I already mentioned, helping to address the problems of
reproducibility, opening up new research pathways. Do we
agree with this argument or not? And if we
do, what are the principal barriers, we might
see him moving towards the system? I'm gonna sit
down. Great. Thanks general. So, um
, take a minute, a few seconds to digest
that and I will open it up to the panel
whoever would like to start. Sure, sure.
Help you to start. So as you heard,
I'm a statistician and I've worked in healthcare my entire
career in addition to working on patient centered comparative effectiveness
research. I'm also an expert in systematic reviews.
And when you talk about publication bias, I deal
with that a lot. The idea that particularly non
significant results are either not published or they lag in
publication time and this has been shown. So I
agree with this. I think that the benefits do
outweigh the drawbacks. And in fact in my world
, the international consortium of Medical Journal editors have already
put out a statement about the sharing of clinical trial
data. There was a national academies panel that produced
a report in 2015 titled sharing clinical trial data maximizing
benefits, minimizing risks. And they actually lay out
a timeline for the sharing of those data. So
if you had a clinical trial within six months,
those data would be shared. Um I do want
to bring out a couple of issues. And the
first is that I work in patient centered outcomes research
. And when you talk to patients who have shared
their data with you, they are concerned about confidentiality
but they also want their data shared. That's why
they've talked to you about their lives and they've shared
their data. They want their data to have an
impact on patients. So I believe we have an
ethical responsibility to do that in an ethical way.
That's kind of the other side of the coin.
Most people think patients are nervous about their data being
shared Actually when you talk to them they want it
shared and they ask you to take that seriously.
So that's an interesting point that comes from that world
. The issue of credit and you are early career
researchers yourselves. I'm a dean. Um and I
think a lot about PMT promotion and tenure and what
are the criteria by which we evaluate scholarship in a
setting like Virginia Tech. And I think we're going
to have to be open to the idea that if
you produce a data set that that in itself is
a product not just the publications that come out of
it. I don't think we're doing that yet.
But I think here at Virginia to Apple as well
as other institutions. We're going to have to think
very flexibly on that point. Another point when you
see something like mining the hard data, I think
that was probably the Framingham dataset perhaps. But as
a statistician, the old adage torture the data and
it will confess. Right? So you know I
mean if we just run a million tests so there
is an issue of secondary analyses and the quality of
those and the assessment of that quality. So that's
kind of a thorning point. I think there has
to be a funding model that allows the storage of
these data and it isn't just the data set.
It's not the analysis data set is not even the
raw data. It's how you got from the raw
data to the analysis data set. It's how the
variables were defined, who dropped out of the raw
data set. Target sample, how'd you get from
there? It's that metadata and it's a very complex
process. It's not uni dimensional and it's hard to
think about how to how to how to present that
metadata and the steps that were taken. It's not
an easy thing to do. It's not just the
code. And I think we have to think hard
about that. Do we, do we have an
issue with that with just where is it all gonna
be, where you're gonna be stored? Who's going
to pay for it? So when you think about
funding agencies and I do a lot of work for
pick Corey, the patient centered actions research institute,
there are asking people were going to give you money
and you must share your data. And the question
is who pays for that? Um where is it
stored? It's nice to see the Virginia Tech and
other institutions. But I mean, how many is
it? Seven years? Like your taxes? I
mean people haven't thought so. I think the devil's
in the details, so all and others want to
speak. But I personally passionately feel this is an
ethical responsibility. And at least in my world it's
where the world's going. So we have to think
about how to do it. Thank you. David
. Open to access what's not to like. Although
in fact when when pokes around, I hope we'll
get into some pros and cons, it turns out
to be a very complicated problem where things that benefit
a lot of people a little are going to harm
a few people a whole lot. And the and
the conflicts of the complexities of the legal matter and
the matter of dealing with media and moving from paper
to digital, the institutional matters and metadata issues.
Uh This is an enormously enormously complicated subject with many
different angles and threads. I thought the introduction was
really splendid. Is sort of synthesizing some of the
complexities that we get into. Well, let me
tell my story. Uh huh. Open access changed
my life. I used to be an 18th century
scholar and my studies sort of do genetic histories of
poetry, how things get transmitted over the generations,
bit by bit through lots of minor minor poets and
so on and so forth. I used to spend
all my time working on this huge microphone collection up
in Charlottesville Which had 25,000 I think 18th century books
which had all been filmed and murder up there.
And I run the film and make the Xeroxes and
endless horizons of poetry. Nobody ever looked at before
. But then at last it went digital and the
University of Virginia persons the dataset put the microfilm into
storage and I no longer had access to it because
I couldn't get into it through the University of Virginia
. And after five years of quelling and gnashing of
teeth, I became a 19th century scholar Because Google
made 19th century books available, whereas the 18th century
books were all under, behind the price wall.
Uh Virginia check wasn't gonna leap did eventually. Uh
, it was too late for me. You know
, just access to data is really, really important
. Uh, when you're, when you're shaping a
career, you always have to sort of think about
what's available now and what will be available at five
years or 10 years or 20 years really. And
, and to who, whether you're a consumer of
data or a producer of data, these are,
these are important issues. Uh, we talked about
incentive issues. This is a big complicated one and
it's probably different from me being in the humanities and
it will be for many of you working in the
, in the sciences. But I've always had a
dual career. There was paper paper scholarship which I
published for 10 year promotion and raises. And then
there was my real career which is working on digital
material, which was never anything you could submit with
your annual report because it wasn't peer reviewed and there
really wasn't a basis for for for peer reviewing it
right. It takes 10 years to do a database
, lots of multidisciplinary considerations and nervous subject and all
the rest of it Can it for zip for my
tenure and promotion. But if I were to give
a 10 minute presentation at the modern language association,
talking about what databases are good for that would count
. You see. And it's sort of crazy.
But this is the world that that you live in
and you have to, you know, sort of
be attentive to both sides. You want to be
aware of the pragmatic issues, but you might also
for one reason or rather decide that you're going to
do an end run around it and pursue the scholarship
. You really care about whether or not the reward
system is necessarily going to make it worth to you
, what it might be might be worth to others
. Right now. We're in the humanities, we're
trying very, very hard to hire young people with
digital chops and they come and they suddenly thrust into
a promotion and tender system that doesn't value digital scholarship
. Uh, this leads to scary results. Uh
, we hire somebody because they're really quick with databases
and they're expected to publish scholarly books and articles.
You see the problem. Uh, but it's a
dynamic problem and it changed, it's changing from year
to year to year. So I think it would
sort of behoove us all to have conversations like this
and for you to talk to other graduate students and
other faculty and just get a sense of, you
know, how these issues are playing out in your
feeling where we are now, but also Which were
young, where we might be 10 or 20 years
out and that will probably be in a very different
place. It's great. I'm Daniel. Okay,
so that's first discussing stories. So I got my
Masters in Public Health and epidemiology. And so for
those of you who don't know what epidemiology is,
it's if you've ever seen the movie contagion, that's
one analogy I like to make. The other analogy
I like to make is were the people that tell
you eggs are good for you and then the day
after tell you eggs are bad for you and then
it's good for you. But no, it's the
egg yolk, that's bad for you enough. So
that's so that's my field in a nutshell. And
so along the lines of open research and open data
. Um since I am instructor for software carpentry and
data carpentry, um I'm very focused on the tools
that are used in just research in general. Does
everybody here know the term software carpentry I guess I
can. So software carpentry uh This is recorded.
So I really hope I'm not like making stuff up
. But uh so um back in the 90s it
was something started by a guy named um Greg Wilson
. So he was a computer scientist who essentially have
the problem of his job. He was hired to
parallel eyes programs and that's what he was hired for
. And he realized that as he was hired,
he had to work with research scientists who didn't really
, who weren't really trained as computer scientists. So
they didn't really learn how to program other than um
you know, you fumble as you go, which
a lot of grad students go through. And he
tells the story about um he was trying to paralyze
a piece of software written by research scientists and he
was so happy that he saw a function. That
that's something that's never happened before. And sooner or
later he realizes that the functions are really only the
bits of code that the researcher didn't want run.
Right? So for those who are in computer science
, that's really just like a massive way to comment
our code is how the research scientist was doing there
. Mhm. And so he tried he realized that
that was a systematic problem in in research ordering research
software where you you tell all these research scientists that
like, hey, you need to do this and
you have, the only way you can do this
is with computers, but your training isn't in computers
, it's you know in uh huh in english or
something. And so he started putting together um lesson
plans around how to do computing, right? So
the goal is really yeah to teach basic skills for
research computing I guess it's written over there, so
I'm sort of here um And it's for you know
mainly for grad students and their faculty to come in
through the essentially like a two day workshop where we
sort of set everything right? Like you've probably learned
a whole bunch of um Bad habits along the way
. Like I taught one like in 2015 I guess
. Thanks. Um And you know we're trying to
just show you like hey they're a bunch of concepts
that you've probably heard of, but let's just like
formalise everything so you can go on your road down
doing best practices. Um So that's that's what I
do. Um I. P. For software and
data, carpentry is essentially the same thing, but
more data focus instead of software focused a lot of
the material overlap and it's publicly available. It's under
a creative commons license so you can just go to
it. What's really nice about it is also there
are hundreds of instructors and there's hundreds of workshops being
taught every year. So we get feedback from oh
this didn't work and this didn't work this work and
this didn't work. So the boss cemetery is constantly
being refined and tweet and I maintain one of the
our lesson plans. So if you want to learn
are like that's I maintain it and I get a
bunch of things like hey there's a mistake here and
it's like thanks for fixing it. It's a good
thing you're doing it because your, your life teaching
and like a students like this didn't work. So
, um, so that's all, um, it's
all done publicly and it's like a giant collaboration with
like hundreds of other people, so that's what suffering
data, carpentry are. So let me ask them
. So it sounds like you're a proponent of open
research, but you're at a point in your career
where you are going to face many of these challenges
we're talking about. So how do you feel about
in terms of your thinking about your own? So
I guess I'm so as a grad student, I
guess I've been pretty lucky that I haven't had too
much pushback from my advisor per se. Um,
my general school of Thought is just do everything on
some, some open forum and um, it's in
that way, it's almost hard, like my ration
, this hasn't happened to me yet, but my
rationale is it's almost very difficult to get scooped if
that happens because you have already published everything out there
. So if you do get scooped, you can
say, hey, like I wrote about this,
you know, at the very least like you would
either get credit to what you've written or you just
made a friend or a collaborator with your research.
So that's my general school of thought as to going
down this route for the other thing. That's nice
with if you follow like an open data mindset um
is especially with reproducibility. Um that's, you know
, reproducible research is one thing, but it's also
as a grad student, how many of you guys
have lost work because you something bad happened to a
computer. So, but how many of you learn
from that and then set up a reproducible workflow so
that if that happens again, which I pretty much
have to wipe my computer every two months for whatever
reason, like I can just rerun whatever code I
have and I get all my results back like so
you know, there's there's open research and open data
, but like again, like I like to I
address everything or I looked through everything from tools perspective
and so um it's more than just, you know
, reproducible research as the finished product. But as
you're working, you know, it's very good to
have, you know, delete everything and make sure
everything runs again. If you're if you're running a
lab that's really important because grad students will flow in
and out of your lab and you would really want
the next person who's getting trained to be able to
rerun the same piece of analysis that was already there
, which is also a giant message. You've inherited
code from somebody and have no idea what it's like
. Mhm. Okay. Okay laura contrarian. Sam
I actually have always used secondary, I started with
my sister probably before you were born using secondary data
of people who were interviewed when they were born.
Came back when they were 18 years old, came
back when they were 42 years old and came back
when they were 64 years Mrs effort. And I
use those data to show that how one manages their
health conditions early in adulthood affects their cognitive functioning later
in adulthood to the exclude, I did not study
those. Actually, we're performing not as well with
untreated, hyper compared to those who that was my
dissertation. And ever since then I'm always here other
people. But I'm gonna be contrary because We need
it we need another 50 points And I had the
opportunity contrary. Um because I work in a research
society that's 6000 people strong and international research society and
it's made up primarily of four disciplines. Gerontologists are
people who studied the aging process primarily in humans but
well. and so gerontologist, we have four very
discipline specific journals. Biological sciences, Medical sciences,
Social sciences, psychological sciences. And then we have
a social worker and it's all very relevant. Heard
over and over again. There's power of discipline specific
death for increasing knowledge within a discipline. Not only
knowledge, theoretical knowledge but also content knowledge and also
the method, logic, knowledge that specific is all
very important. But we're rejecting in our society,
rejecting a about 80 of some and they're not bad
submissions. They go up to other journals, get
published and get high citation rates. And so why
would we create a new interdisciplinary journal? Because it's
about time And yeah, because we want to start
capturing some of the public that were we're reject.
So that means I get to sit at this table
with all these people from all kinds of this.
And what happens when we get together is I get
to learn about things that I don't do. And
I want to tell you a story about a colleague
of mine who is in charge of that social the
journal that's more about social work sort of hands on
with regard always. So she began collecting data and
2006 um of older New Jersey residents. And she
had I spent a lot of time conceptualizing what kind
of a to she wanted to collect because she was
really interested in the process of healthy aging. And
so the question is, what is healthy aging?
So that that's really conceptually a very difficult question to
answer, and I don't think we have an answer
right? But so she has to figure that out
. So she she's conceptualizing, I really want to
figure out what a healthy ageing is, but I
don't really know what data collected. So I'm gonna
rely on theory, I'm gonna rely on prior research
and I'm gonna look up very conceptual model, I
can never find and I will figure out how to
measure healthy aging. So the conceptual homework was tremendous
. And so then she says, okay, I
think I know where I'm going through all this spaghetti
and meatball diagrams associated with healthy aging meaning, you
know, this goes to their this close to their
their supposed to there. And then she says,
okay, now that I know what I want to
measure, I gotta figure out how to measure it
. And so then she had to go through this
whole process of figuring out what measures to uh to
give her respondents. So for example, we're questioning
your best measures how well we can physically what questionnaire
best measures how depressed we are. And so she
spent all this time figuring out those measures with the
highest face contents, constructive criterion validity. Is it
all critical to the internal validity of your of your
stuff? Right? So she had to come up
with the best measures possible. She did that.
And then she went and she spent all this time
trying to determine from whom to collect data because this
is all about healthy aging. So you begin at
birth, you know? And Well that's true.
She was already my age, so that wasn't work
, right? And so she decided to start collecting
data from 50 onwards and she had some inclusion exclusion
, I get all this took time. And then
the biggest bugaboo was how does she collect this data
in the context of the funding that she had at
that time and the time period that so she has
invested a huge amount of ever into collecting these things
, right? And to tell her at the end
of her NIH funding that she had to discuss these
data immediately six months afterwards is rather unfortunate because she
had so much that she had learned in the context
of gathering the data and not only in terms of
conceptualizing the project and measures and everything, but also
in the interviews because in the interviews, she's listening
to the people and she's listening to what they have
to say. And she has developed hypotheses on top
of hypotheses that have a hypothesis and then suddenly as
a you got to get this leg up, She
was nonplussed when she was when she was confronted with
these kind of guys. She was really upset me
. I'm like 80 data data, I can't get
enough, you know, and I'll take anybody staying
and I'll figure out something to do with it.
I'm selling, look, I'm a data monitor,
I wouldn't, but you know, I really had
incredible appreciation for the incredible human capacity in the energy
and the finances associated with collecting those videos. She
in fact, is very generous. And so my
graduate students are using her data to ask questions that
she was not ready or did not have the resources
to ask. So it's not as if she is
not sharing the data but to have and put it
out on on a website and for her to create
the data dictionaries and explain the conceptual framework behind it
and etcetera etcetera etcetera would take a huge amount of
our times and then would reduce her productivity as a
scientific researcher because you'll be so busy in the sort
of the tangibles of putting up this data set that
you couldn't write it. So that's my contrarian speech
for the evening. And I want you to appreciate
that. I have always yes other people and I
spent a lot of time when I allies data understanding
those data, we're trying to analyze it. I'll
give you a very broad example and I'm going to
go back to your example because you said you said
we should put data and the best way to do
it is is getting to buy it right? It's
one way, yeah, but there's no perfect place
. We'll do you identify it. Um and and
that's a great thing to do is be identified data
. But recently I had the privilege of being able
to analyze data collected from all of new Jerusalem is
in all of new york and it was Medicare,
Medicaid. Do we cut and humid data In 2011
, 12 and 13. There was an event that
occurred in 2012 over 29 in New Jersey. In
new york. And social reasons Hurricane Sandy, no
relation to me. Yes. Yeah. So Hurricane
Sandy comes in and just basically demolishes the coastline.
And so we're using these data, real data from
real people, understand what happens to order adults in
the context of heard. Yes. And the frustrating
thing for me is I don't know about you except
that I was like grueling. I heard that I
had a Dui fema, had Medicare and Medicaid data
from school residents of New Jersey, new york,
Right? Unbelievable. What happened is the Federal funding
Agency gave the contract mm from out in this California
where I'm proud. But it was a Red University
gold university. And so they get answered giving a
they literally like a better. So in the case
so this this just entity takes a data merges it
. Great. There's a lot of work so appreciative
. But then they're so worried about the identification that
they went overboard and they started bending the data men
or putting things into groups or categories. And in
this case it was time categories and everything was put
in monthly a lot. But the darndest thing that
hurricane did not begin on did not occur in the
beginning. Yeah, a problem. And then they
started bidding, you know, you know, sort
of services like durable medical equipment, well durable medical
equipment and includes everything from walkers and canes. Two
uh, nebulizer is needed for people with serious lung
disease to be able to breathe. It's these nebulizer
usually require electricity. There's some battery power um to
heart monitors. To CPAP machines would you typically use
for people would sleep apnea, trouble breathing. And
so they didn't know what they were doing. And
so this d identification processes can't be without understanding data
. Right? And so, you know, I'm
a fan of being able to use these data,
but I'm also a fan of understanding the data which
one is using and not only understanding data. So
what does each data element mean conceptually, really?
What does it mean? And so much of my
work in the athletic phase of our projects is to
get an estimate or a statistic. Uh huh.
And then I use that number and I go back
to anything in the little or on CMS websites or
the cbc website to check that my estimate is reasonable
. That it's like what we would expect when it's
taken back. If you if you like our nerd
like me, you would go to CMS and you
will be able to look at him at the United
States. It's really they really have and each state
has a slightly different color and you click on the
state and then you do this table and you can
say heart disease and it'll give you the estimate for
that state. So you never should use secondary data
without being able to personally validate the finding that you
have is what you would expect either by published research
or by estimate published by the government. So,
you know, I'm a little bit of a country
and I think it's very, very important to share
data. I've made my research career I don't know
how many Let's say 130 and its secondary didn't I
didn't collect any of that data time of lazy that
way. But I do spend a lot of time
making sure the way I analyze the data is correct
. And there's a lot of not only external lawyers
, people internal validity in terms of terms of the
assessments I might and I would say that in fact
most researchers are like that and I know I just
tried to article data Davison. Oh researchers are reporting
false data. That's a very small minority. Most
researchers have high integrity as you wouldn't see it.
Okay, so be careful about falling into that crap
that without sport all days in open that people.
Sorry. So you're if I'm hearing you correctly then
you're saying you benefited from open data but but so
you're not saying you're against open data, but you're
saying that what we need to have, we need
to restrictions for we need to respect the people who
collected that data and create um a broad swath rules
that you must really sure data within six months at
the end of funding I think is not really for
those who spent so much time to collect, stayed
up. And I've collected data from clinical trials.
I've analyzed those data retrospectively, I analyzed data from
prospective press we're talking about but nonetheless, I think
that we have to be a little bit considerate to
people who have taken all that effort to collect those
data. Because fundamentally the phenomenon that we studied training
courses, what if your colleagues were given more credit
for that data collection? That's so interesting that you
said that because um because the Oxford University Press publishers
are journal. And so we all get together all
the editors in the US get together one weekend and
we go through all these sessions about how do we
do this ethically and that ethically. And you know
, it's just like, you know, it's like
come to truth lonely and for all of us.
And so we're in this ethics session. I said
, yeah, I've talked to my research um what
I have a graduate student who's using the data from
that woman. I said, you know, not
really encouraged him to include her on the publication.
She is that's on that. Unless she told them
what the research question is, unless she drove every
hypothesis less and less and less and less and stayed
all these criteria. Students should not be on the
publication just because it was her data. And I
said, but you don't understand that the conceptualization of
these data came from her and were not for her
he wouldn't have these data and she wouldn't be able
to help him understand the nuances of why she chose
this instrument versus that instrument. Um but but this
ou p ethicists can't be on the paper unless she's
done X percent. So what is X percent of
the work when you're the one who collected the data
? So there is a project I know about called
Project Credit which is trying to So we are all
credit and academic world these publications and text right?
But there are all these other ways to work and
help with research writing code, good code to analyze
stuff, create datasets like the one who just Professor
Sanchez mentioned all these other ways to contribute to research
but these people tend to get left out of the
scientific merit. You know if you're a research software
engineer, where is your career track in academia?
You just really want to write good code or create
datasets like this. So people are working on new
models of contributor ship to try and try and give
some of these folks some of the credit that they
deserve within academia. Of course it's one thing to
set up a mechanism just like data citations. It's
quite another for it to be accepted by the candidate
. Okay so um I want to give me a
chance to answer the question then. Think about questions
that you all want to ask as a follow up
to this and I'll be brief. Um but I
want to go back to that question. Um I
am in favor of open research and open data.
Mm Because I think it has collected benefits as it
says. But I want to add to those benefits
because for me it's not just I think it helps
with reproducing the results and new pathways. But I
think I would argue for two things. One is
the interdisciplinary t when one is looking at the data
um and the different perspectives. Because how a social
scientist might look at some data could it might be
different than a bio mechanisms. And I think bringing
those kind of perspective. So I agree that there
is collective, I think there's an individual benefits as
well as the collective ones. And I would just
add to that. Um what are the principal barriers
? What we've heard some of them this evening.
But what I want to put up on the table
is the cultural context, not of the data but
of higher education because we tend to be isolated in
our disciplines, hold our data close for our own
good and that is the system that reinforces that sally
you brought up the tenure and promotion. Um and
uh and I think that we can do this and
we can still get tenure and promoted um and get
the questions out there. It's just our culture,
as in academia, that's to me the biggest barrier
that we face. David. I think you're example
. So I'm that's what I would have. Do
you want to add something to it? I know
in the humanities, one of the things that we
struggle with in terms of institutional practices is information versus
interpretation. We highly value interpretation. That's what you
do in books and articles. Information, It's just
being counted. You know, we we don't care
for that And the reward system is all based on
interpreting things. Of course here we are in 2017
, right? Where the world is driven by information
, we not only don't have the information, we
have no incentives for humanity scholars to go out and
gather the information and make the database. Of course
, the federal government isn't helping either. And so
it's a cultural problem. It really is. I
mean, everybody has a brain knows that this is
the information age is what would you be thinking about
? Try hiring a graduate student in history or in
english or philosophy with digital chops. They don't have
them because their dissertation directors don't have them. And
if the dissertation director doesn't have them usually doesn't value
them. It where are they going to learn it
? Um, and and and and so it's gonna
be a complicated change is gonna come. But and
that's why we have to have a conversation with preparing
the future faculty with these kind of courses so that
people can make you all can make your own decisions
but understand it and abroad in a broader context.
Mm Great so franklin please. And particularly raising any
of the any of the issues here. If you
dealt with this kind of um issue yourself or you're
you're concerned about how you're going to deal with data
in your own research. Um I'm not so clear
about the banana of the open data. My son
institute, they collect data and the data neglected.
Um Does that come for open data? Yeah but
I would say no it's not. I think I
would always say their their degrees of openness and and
different facets to being open. So free data that
is available online is accessible by a wide number of
people but there are a lot of people who don't
have access to to do the online environment. So
it's not open to that if the data costs money
. So that's one barrier that might be. But
if you are the data open to um are they
open accessible to somebody who's got that language? So
we start a lot of things in english and in
academia, not everybody speaks english. So there are
several different facets of openness I would say some in
general something that costs money. Data that you have
to access. The we are paying for something a
barry we'd like to see removed but we must acknowledge
this stuff doesn't isn't isn't free servers documenting data structuring
its reusable. This stuff is all it all it
takes the cost. Like the Professor Sanchez just saying
enormous amounts of effort involved in getting this data sets
. So I would not call that open data if
it was for for money. But there are many
degrees, I think it's free of all barriers.
There's some sort of data that's free of all barriers
and the state that's entirely restricted clubs. And there's
a spectrum is that we have to be how many
of you have published or your faculty have published an
open access raise them higher. So I just wanted
to. Well, I mean it's happening. Yeah
. And we're their requirements for any of you published
open access. For the requirements around making data methods
or anything available as well. For the requirements.
Well, over access. I mean over access,
Colonel. Yes. Yes. So about a third
of you raised your hand sarah. You had to
share the code and that was the journal made that
requirement. Um, I'm interested in laura your perspective
from the journal, starting a new journal and trying
to figure out how do you develop a policy around
this? So open access is a new model for
us and that the biggest problem that I'm having right
now with an open access journal from a very traditional
research society. First of all, we spent gerontology
And then and then secondly our society started about 70
years ago and we have these discipline specific journals and
they're pretty set in their ways and they're all print
journals, but then they're always complaining about how they
don't have enough pages for issues, they can't publish
enough. And so that's all the problems. So
we go to this open access model and some are
yeah, stupefied that they would that, you know
, there are author publication fees for publishing in the
journal and it's a huge barrier, a social barrier
, a conceptual barrier for some the individuals. And
yet one of the things that they're not appreciating is
the freedom it gives one and the freedom it gives
one, not only in terms of being able to
post their published paper in any medium, they care
too, um you know, on their on their
personal web page wherever, so that it has much
more availability online. But the freedom for it to
be much more easily translated by people outside of your
specific area of interest, because one of things you
have to understand is you're not you're not doing research
so that you can self site among a small circle
of scientists, right? You're doing research because you're
trying to change some condition and you want to be
able to make your results available to people outside of
your circle of scientists. Right? So so that's
another advantage of open access. But one has to
do that in the context of changing very um stayed
research society into accepting this new model. And so
my big concession was as the pages right now if
you download the pdf, it's gonna look just like
a print journal where you're downloading Wikipedia. And it's
so that they have that often. Oh this is
really legitimate. This looks like exactly the other journals
. But my goal in 10 years is to be
able to get that code up and online. And
people to be able to run um tiny bits of
code using data that the authors are posting. For
example, I'm like I don't even read I never
read the beginning and end of an article. I
only read the well first I read the tables and
then I'll look at the methods to see and make
sure I understand the tables, right? And don't
even read the results because that's just what's on the
table for the figures. Right? So the bottom
line is I would love to be able to click
on cells within the table and change you know,
parameters. And you can do that in a d
identified environment and say, what would this be like
for Just males under 18? You know, what
would my estimate B get my market house? That's
pretty much that's pretty powerful. Yeah. But it's
not gonna happen today because I gotta make this journal
look exactly like these journals that we've had for 70
years. Right? And then next year I'm going
to do a little tweet, you know, and
I'm gonna put in like oh they can link to
outside sources, not just the references and their own
thing. And the next year I'll add another tweak
and another tweak. So that little by little I
can help my society and culture it to an open
access journal that has this power for conveying information like
no print journal can without reference reference to the 18th
and 19th century. So I was a founding editor
of a journal called Statistics and Policy, which is
open access. And one of the things we did
is pair up with a professional society, the American
Statistical Association to make the first couple years where there
weren't page charges because the society, but we had
the same struggle. And I think of getting people
to accept it as uh reputable journal and the same
thing if you print out, it looks just like
a paper journal and the same thing. It's making
me laugh, but it requires that that that authors
produce their code. But one of my concerns is
going back to this gentleman's question here about cost.
Once you ask authors to have to pay page cost
, though I realize the benefits it could um bifurcate
those of us that are more generous institutions, for
example, versus those of us who aren't. So
it has a possibility of driving a wedge there though
, in our journal, we actually have, you
can apply to have the costs reduce if you come
from certain institutions. That sounds like you're thinking about
that. So that's all the same. And so
everything about these two Germans from, You know what
, we're actually not really super separate researches, but
the same idea is going across the same society.
1st 200 accepted articles are free And our journal.
Um but and and then what we have, we're
gonna waive fees for 25%. But one of the
things that you don't know the history of why we're
not really interested in this, this really um this
this old model of libraries paying some description fees to
journals. And I was on a library committee for
well eight years at Purdue. And every committee meeting
we got together and we route the data and we
had to get rid of X number of holdings per
and I think you're going to speak better to that
than I. But they're competing curves right in terms
of funding to the libraries because of state revenues declining
and subscription piece increasing exponentially. And there's the sheer
problem that the amount of research that's being published,
there's been an explosion. And the idea of open
access journals was that you if you can shift the
system over to open access then libraries don't have to
pay for subscriptions and we can fund the system in
a different way. The problem right now is that
we're still paying for subscriptions and we're paying a lot
of money for these are processing charges and there's been
no real savings. There might have been in some
cases, some savings, but the system is kind
of in this impasse right now and you all are
coming out right at the time where you're gonna get
to enjoy the benefits of that. But it is
it is a challenge. Um So those of you
published an open access journals, did any of you
have to pay article processing charges colonisers? Are you
uh are you familiar with that concept? The article
processing charge? Because it's basically, I mean,
I know that science has always had a custom of
for page charges that isn't page charges, right?
That isn't like that in book publishing, but in
journal articles in the journal world now with um increasingly
going to open access the way it gets paid is
on the front end of the process and the authors
are expected in many cases, not all cases,
there are many open access journals that don't charge a
pcs, but in the fields where a lot of
the big disciplinary journals are, that's how the journals
are able to make money and survive. So they're
charging article processing charges. And in some cases you
can get that money paid for from a research grant
or your institution may have funding. But as we
said here, many institutions you don't, you don't
have this uh well funded a face to draw upon
. So it's a big, what do we do
here? What's the library have we have a open
access um, fund that we started about Four years
ago, I think 2013 2013 or 2014. We
started with the fund was about Philip was 15,000 20,000
, something like that. About 20 25,000 this year
it's 100,000 And that's in chunks of$2,000 3000 um
at the time, so That's how much demand there
is for it. And that's money. That's$100,000
. That came from somewhere else in the budget.
And because we didn't get extra money in our collections
budget to cover, so that's just another expense on
top of what we're already having to pay for it
. But the students are faculty Canada is that for
students or faculty or students or faculty? Anybody can
apply can apply. So if you are publishing in
the journal that is open access and there is a
requirement for and a pc you can apply for funding
from the library. Yes. So this question today
has been focused on the kind of User data inside
of academia. Was wondering if you can shed light
on how this is affecting industry, especially in biology
where I come from uh having open access journal since
early 2000. I was wondering if there's been any
shift in the kind of data usage by the industry
? Had they are responding more project as an outcome
of accessibility research. You don't want to take that
on perilously close Mr muscle. I know in pharma
there have actually been some collaborative efforts to share data
across companies. I mean you naturally think they're competitive
so they don't want to share data. But there
actually have been some successes there. So I think
people are trying to think in that way. That's
that's all I know of an in pharmaceutical companies.
It's not related to publication per se, but more
the sharing of data sets. That's I mean I
think that's uh that's good news. Okay. And
industry stands to benefit greatly from this open open data
. So they I mean, you pointed out the
irony of engine not being able to reproduce the results
of its research and yet they don't make their data
available, but they're benefiting from they stand to benefit
from I think you see a small companies startups,
biomedical startups can benefit greatly from open access publications.
Because how else are they going to get access to
this corpus of knowledge behind told terms, benefit curtain
. Yeah. What about the question of that's come
up a number of times on reproducibility or reckless right
click ability. And we were talking this afternoon about
this beforehand and there seems to be some we're not
necessarily clear that on the definitions of lows. Um
you might I think, yeah, there's an article
in science last year, late last year talking about
how we don't actually agree on what these words mean
. Mhm. Some people think for disability means what
replica replica ability means and the opposite. We talked
to, one group of researchers will say Republic ability
is this and the other people. That's the reproducibility
. So the two the two fastest I probably could
do a lot more variations, but one of them
, one of them, whichever one we want to
choose is I've got access to the data to somebody
else is giving me like open data and I run
the analysis based on that data, do a similar
analysis and see if I can come up with the
same results and same conclusions. So that could be
reproducibility or replica built and the other option would be
trying to do a whole independent study. So somebody
else has got this, got this research study over
here and I want to see if I can reproduce
replicate those results by gathering my own data and doing
the whole thing from the weekend and then you can
imagine different variants in between. But I found it
very, very funny that we can't agree on what
these words mean. Well, just the definition of
open and yeah, people don't know what that means
. Thanks. I thought maybe you could talk a
little bit more about what men analysis is as an
analytic technique and how it kind of addresses some of
those issues because really it's not about always looking at
the same day as it, but looking across trials
to the same amount. So I don't know if
people heard of the word meta analysis raise your hand
, expert experts here. So, so I I
work in the area of systematic reviews which are reviews
of prior evidence, prior studies that have been done
, published or unpublished. So that's one of the
difficulties. How do you find the unpublished studies?
Um And it's a systematic review because it's a comprehensive
and hopefully hopefully replicable process for finding those studies.
And assessing the meta analysis is the statistical inferential technique
by which we combine the results across the study.
So to take a simple example, if we had
10 randomized controlled trials and we have an odds ratio
for, say, death in each of those trials
. And those trials are actually looking at a particular
treatment which is common across them, which then goes
to, is it really common? Can we somehow
combine those odds ratios? A simple way to do
? It would be just take a simple average of
the odds ratios. That's not a great way because
some of the studies could be bigger than others and
may be more precise odds ratios. Meta analytic techniques
basically figure out how to average across the studies to
really um really simplifying. But it's a pretty interesting
technique because you're actually looking at different studies and trying
to understand not so much why they're the same but
why they're actually different. What what explains the heterogeneity
across the study? One study appears to work the
other doesn't. Why is it are the patients different
? Is intervention actually different? So I think meta
analysis some of the techniques people have been thinking about
reproducibility because in some sense reproducibility is like doing meta
analysis because you're looking at a new study and trying
to figure out how to combine it or look at
it in contrast to previous ones. So a lot
of people thinking in the meta analysis world about reproducibility
is that and does. So I sort of posed
this question. I think last semester room we had
one of these forums except I was out there.
But if we sort of overlook the open access or
paid access because you know, journalist need to make
money somehow it's just totally like a source of income
the same for you know, people who curate data
. Um I guess the question to everyone would be
um wouldn't sharing your code or open sourcing your things
be the appropriate next step because you can have two
groups to, you know researchers who either bought the
same data set, but they want to, they
read one thing in the papers, anything. Oh
that's a really nice technique. I want to reproduce
it. Um If they had the code available,
they would be able to incorporate that information and you
know that overlooks the whole like paying for things.
So I guess would just mandating uh you know,
every time you publish a paper, the analysis,
the code that went into the analysis for all the
figures and all that be submitted at the same time
. Would that be the appropriate next step? Because
this is going to be a very slow process to
open up research and open up science. Um,
I personally know that if I know that someone else
is going to be looking at my code, I'll
be pretty sure I'll be like going through that and
make sure I'm not making any more mistakes because someone
else is going to look at this. And that
is essentially my research career on the line. So
that kind of raises another question that we were talking
about today, which is, so publication is already
a slow enough process. And if you have to
get it becomes standard that you submit your code and
your data and then the readers have to review that
. Is that going to slow the process down?
And is that going to then every old world impact
that we have to deal with what people think of
that? I mean the real, I mean,
so you have the reviewers for the paper, but
I guess like the reviewers of the code would be
the grad student that's trying to run the code right
, that sort of. Um, so I don't
think it would slow down the actual paper publishing part
. It's more just like, hey, I can't
run this and then you have some offline conversation before
you like, hey, this guy just wrote print
P equals.5 at the end of the day.
So I take that even a step further. Um
, so I've worked at Corey patient centered outcomes research
institute and one of the things the institute is struggling
with is how much funding do you put into new
studies versus say the data storage or the checking for
reproducibility or funding the capabilities within the studies to do
this type of meta analytic. So it's it's a
real tension for the nation with funding being limited.
Where do we spend it? And I, so
I think that's an interesting question as well. I
don't wanna following on what Daniel said. I wouldn't
want to mention something else. We're talking about trying
to make research reproducible kind of for your own benefit
in some ways. If somebody else, somebody else
is going to look at this boy. I sure
wanted to be right. I don't want to share
something that's grossly wrong. It makes me look terrible
. Uh I like to say when I talk to
researchers, especially guidance students about why should you do
better data management? Why should you think about backing
up your data documenting and all this stuff? Well
, yes, you can make it makes it easier
to share. So that's nice. I'm for sharing
. I like that. But you're most likely collaborator
. I like to tell everybody is you in six
months, you're the most likely user of all this
stuff. So you leave, you're doing research project
, you leave it for some reason you come back
to a six months later, you're going to be
the most likely benefit of what what was what was
this line of my code for? What was this
column? Units? You're the one who's going to
benefit from this stuff and then you can you can
share with other people. But taking some time and
effort to do this will benefit you force them for
most. I think that actually we should slow down
the complication. Uh huh. Uh huh. Yeah
. Well I guess there's a question of can we
and should we? It's it's hard for me to
imagine a world where we're gonna all get together and
say, okay let's slow down publication. I just
don't. You start you start that's great. You
start anybody. I think it's a great question.
And I think that what we have to think about
is are we publishing? Because this finding is really
going to transform our way of thinking about something our
way of caring for a patient in the case of
As Sally's in my field and you're 12 or are
we publishing? Because this is pretty good. And
I think I can get published and it's another line
on my vita. And I'd like to say that
I would like to change that attitude of it being
another line on the vita. And that all our
publications should be things that really are transformative. Whether
they have to do with transforming the human condition or
you know, the ecosystem or whatever it is,
they all have to they should be good enough that
they have true meaning and they don't just get archived
forever wherever. And I was wondering what you all
think about some journalists like Evelyn. Mhm. Here
J. And others that are experimenting and post publication
to your review or trouble public comments while I comments
after publication maybe as one way to sort of have
both. Did you all hear that question about post
peer review and what do the panelists think about incorporating
different kinds of your review which could have the impact
of of speeding things up? It's kind of happening
already. I mean, you know, not not
to that extent that it is obvious in those journals
, but if your article is not cited, one
has to take a deep look at why it was
incited if your article finds itself cited by another journal
and they said but they didn't do this where they
messed this up or you know, it does happen
to a certain extent already but not to that critical
extent. But as it both as a as an
author, as a reviewer and as an editor,
I have seen people criticize other people's work because it
wasn't exactly the way they do it and that's that's
not scientific or Mhm. I had a question about
how data analytics the field and computer science and machine
learning. How does that play its part and more
? You're just a merry journey. The analysis of
your data, getting this to be done by machines
as opposed to people and all the way you think
I'm in the computer. For me. Machine readability
is the Holy Grail. I know how to write
for human beings. It's so much harder to write
for a machine in some respects. And yet,
you know the possibility of going Beyonce data mining to
be able to have machine readers reacting intelligently with data
sets and documents. You know, what a wonderful
concept that that that that would be like inventing the
wheel. Uh we're a ways from that yet,
but it's a very important conversation if I think so
, I would say in my world the men analysis
world people are already trying to automate particularly the early
stages of abstract and even retrieval of data from the
articles and doing that almost in real time as information
comes in. So it's really your field is revolutionizing
what I do. So how'd that work with abstract
? Yeah, one of the biggest problems in a
systematic review is actually reading through the articles you find
usually two orders of magnitude more than appearance. So
you might be looking at 100,000 articles to at their
titles and abstracts. Reading 10,000 and then 1000 are
appearing and we're trying to automate that. And it's
difficult because of the reason that you raise because everything
isn't um digitized. And then you have to do
text mining. So and then if you're actually pulling
the data out from the tables but people are thinking
about that to try to speed up the systematic views
. And then in real time as new information comes
in updating the meta analytic results. Death soda the
most current information for the patient and clinician. So
it's pretty exciting. Great question and a comment.
So this is an open data forum and what Professor
Radcliffe was saying about human machine readability. Um I
guess when something to think about, especially if you
end up working for a company that says oh we
want to do open blank um and or open data
and we'll release our data and your data set is
released in PDS, which is very common um for
some reason or another that I don't really understand.
Uh don't do that. Uh PdF's are actually a
terrible form of data sharing and so like, that
follows the like yeah, you were um following the
word of the law, not the spirit when you
end up doing something like that, so shame the
people who do and yourself is from from a perspective
of a publisher. I've just been it's been amazing
to me the longevity of the pdf because it was
just it was not meant to be the version of
record that it has become in many cases, but
it's just stuck around because there's nothing really better at
the moment. So it's one of those things that
just sort of keeps on going. So this is
sort of a different topic. We've been kind of
speaking about data that comes out of studies at the
end. But in my world, the clinical trials
world, there is now a requirement that uh trials
are registered prior to them being implemented. So clinical
trials Gm If you're funded by NIH, you're supposed
to register your trial and this is moving more into
observational studies as well. The reason for it in
the clinical trial setting is to make sure we know
which trials were conducted in case they were not published
. So we have a record and also some people
identify their primary endpoints, they don't sort of do
the test afterwards and say, well mortality wasn't significant
. Such as such was So I will publish on
it. It's kind of that that torture the data
thing I came up with before. But I actually
think this is this may move into other fields as
well. I don't perhaps it already has. But
keeping a record of what science is being conducted early
. So you can see the whole process through registries
at the Center for Open Science has been done this
preregistration contest. They got a million dollars and gave
it out$1000 at a time for people who would
pre register their studies no interested whether clinical or whatever
. Uh and just so that so there is a
statement that we're trying to conduct this research. So
it's also also texted publication bias this year. Right
. Somebody I did this wonderful research and it didn't
pan out the way it was. So nobody will
ever know about it. And I don't know what
you guys, I know I have a couple of
couple of discussions as a graduate student at conferences where
I told somebody, hey, I'm going to do
this. I was talking about what we're going to
go do this and talking to somebody who then said
, oh, you don't want to try that,
You don't want to do that. We tried that
like three years ago. Hmm. Here's some reasons
why you don't want to go that way. That's
the only way you find out. So, these
these ideas, hopefully we'll well assuage these problems.
I may be in the minority in here. But
regarding the discussion, I just don't see much incentive
for the individual researcher to participate in open research because
you quote benefits, helping address the problem of reproducibility
. That's actually risk um associated with the researcher because
then they're going to be questioned in the future.
Um and also opening up new research pathways. Who's
to say you don't open up the research pathway that
that researcher is pursuing at the moment. So I
guess my question is how would you incentivize open research
for the individual researcher? I'm not arguing that the
benefits to the community outweigh the drawbacks, but at
least for the individual who's putting their time and sweat
and effort. Yeah. Into this research, what
are they going to get out of it? And
how do you provide them with more than just feeling
of selflessness provide to the community? No, this
is really right now. It's a bunch of sticks
. You get funding from NIH, you have to
do it. You want to publish in these journals
, you have to do it. But that's the
very question. Is it just the ethical you should
do this because it we will benefit others. And
given that PMT is compounded by this that we've been
talking about promotion and tenure particular in the academic setting
. There aren't those incentives for you. And the
most important question. And is this sort of brings
it around to the point I'd like to close on
which is having to do with sort of as an
institution. How do we deal with this? And
is it something that has to be dealt with at
the institutional level for at the national level? But
what do we do? How realistic within an institution
do you move towards this? If we decide this
is the way to go? How does it,
how does it happen? Where does change happen?
Is it is a bottom up, top down.
So I participated in a workshop at triple A S
. It's uh does science that we talked about this
transparency, open, open this scholarship and the the
working group I worked with, talked with for an
hour or so, really was hinged on credit.
How do researchers who are doing this get credit for
doing it? Because as you said, this is
currently, it's currently like you're sticking your neck out
and what you're going to get for it. So
how do we have a credit system that gives researchers
who are going to share their data, share their
code, do things openly. How are they going
to get credit for doing this And in our academic
system? Currently they do not, you don't benefit
, you'll benefit directly. Maybe you'd benefit through collaborations
. Daniel mentioned you might get any collaborations or maybe
some other ways. But this is this is one
big concern. I think there are some sub disciplines
where credit is given, particularly for the collective.
I would just start with the differences we might learn
from those, but I wonder if if we get
into this kind of Yeah, Situation where one institution
were compared against these other institutions. And so we
say, well, until they do it, we're
not going to do it. And so we're all
sort of sitting around not dealing with the issue is
that that sort of institutional lethargy, we don't have
a system like in the UK where you've got the
, you know, research councils of the United Kingdom
where they there's a top down kind of mandate we
have, we have granting agencies that may require things
, but it's not it's not making I mean,
as you said, it's a series of sticks rather
than really a system. Is that something that will
be coming? Do we think that's eventually going to
be, it's going to be required and the institutions
are going to develop policies around this? I go
back to the Dean's point, you've got to change
the culture and developing a culture that would that is
based not only sharing on data, but sharing generally
would be useful. I want some of the best
sharers in human history has been the merchant networks working
in europe in the late middle Ages into the 20th
century's are people who knew how to how to make
a guilder. Um but the strength of the merchant
community was always in sharing and often very generously with
no expectation of an immediate return. More recently,
we've lost a lot of that with our emphasis on
efficiency and, you know, grinding everything down to
the last time to get the grant proposals. But
I think if we change our behavior is a little
bit we will discover even greater money making student teaching
opportunities where we have more of a culture of of
sharing. And maybe working through the library was sharing
data might not be a bad place to sort of
change the culture here at the University General. We're
happy to help. Yeah. Yes. Syria.
He said been saying back to today the year that
you describe the trader was in a place and their
life might be dependent on sharing. Mm Right now
in the digital we're actually I want to share because
we can overshare isn't and the communication companies just get
every data and then just buy us leaving, create
find street money. The issue. Guess that's so
where is the threshold? How can we Because also
in software, developer free and labor software is different
from college. And there's a fundamental mistake because you
take the tools and it's free and you share your
work, your nutrition passed on, there's a difference
. So if you share, if you publish your
form, it's going to be different than just like
sharing ideas on how to. And also somebody said
before, why don't attribute person that gave the deficit
? Why don't we attribute like the tool that was
designed to get this data the number of design before
. It's not that easy. And we started this
class with metaphor of an hour. Everybody was going
to blindfold and he was touching part of it.
So they were touching one part of it and we're
trying to make sense. That's what we're trying to
do here. We're making sense of different parts of
I think that's a very shrewd remark and I think
sharing is a context specific sort of thing and one
wants to think about that. But certainly code sharing
changed by thinking about a lot of things, including
what I do with my own texts. What a
wonderful example of how sharing can change things in dramatic
ways. So, um we're going to close up
here, but I want to I asked two questions
as we finish up. So having heard this,
Yeah. Going to ask you to raise your hands
if that question we started with right there. How
many people agree with that argument? I was scared
for us. How many degree you are clear about
it? I appreciate your across a little bit.
Mhm. Well, I mean, it boils down
to the collective collective benefits for the research system and
not the researcher. So if you're just considering researcher
than this, I would say this is false.
Absolutely false. But in terms of the academic,
the scientific community, I mean, yes, it's
absolutely true. So, And do we think that
is there say 10 years from now? Will we
be, will this be more of a given or
the expectations? Are we talking about this Something that's
inevitable, is it inevitable within 10 years? 20
years? What do people think? Mhm. I
see, I see a shaking head back there.
I think there's a major difference between open access publishing
research and interpretation functions and the data. I think
the data, he will pay for all those different
elements that go into a dataset, usability dataset people
will start paying a lot for that hoovering it up
as it were paying smart young. We're smart hey
academic researchers their ability to understand manipulate, find and
contribute to base its and that's where the gold is
interpretations with algorithmic machine. You get all these things
we can talk about. I hate to say it's
going to become pretty cheap. Everybody's going to have
an opinion about a person who in two days basket
. Well, I think maybe there were another direction
open the data and data people alternate is kind of
open access. Uh It has some it's always some
access to it. That access may come from his
experience, his reputation and if everyone can get access
to it, someone may not qualify to mess with
the day to mess with concern. Okay. And
that's why we're kind of creates chaos and you are
hard to trust over access. Everyone can do it
. It's like rid of blocks that are reading an
article. It's all internal. Mhm center data sets
with logic. I have one question. If all
the data is open, then we're covered for the
cost of collecting maybe you share in english. That's
a very good question. And it's there's no there
there's no federalized way of doing that. And that's
I think what we're going to see more and more
of going forward is that we have, we have
these different repositories, we have different, I mean
everything that the incentives are coming from different directions and
over time I hope would be that we developed some
sort of more routinized way of this happening and we
distribute the costs more effectively. I don't really know
how that's going to happen but I certainly hear the
conversations going about a more um uh huh you know
an infrastructure, a national federal infrastructure for supporting open
data, open publications. Well, it seems to
be the universities and distortions in universities right today.
Maybe not another two or three years for the well
placed because of because so far the accumulation and aggregation
of all the state and scrubbing all that kind of
thing that has to want is not yet being done
with respect to the kinds of I think research programs
and university engage in by the private sector. It's
going on all these things are going all the time
. But if you look at what the private sector
has done in areas where they make lot of money
fast but particularly financial areas in areas where there are
some of the things with big pharma and Sophia universities
have steeled March and by by sharing their data and
finding ways not finding ways installation, finding ways to
find the budget to start putting these repositories together and
being the political master as well as the driver of
the data itself. Universities should move now because they're
going to lose the march within to push it forward
. While we can still think you have to pay
to get your article published. Just think what it's
going to be like when you have to pay to
get runs on private. What? Mhm. All
right on that. I want to thank you all
for joining us. Please thank the panel considered the
prison is, you know.