Yeah. Mm. So thank you all for coming
today. Just a quick show of hands. How
many people's cars turn into pumpkins at four. Okay
, so that's a good number. So what I'm
gonna try and do, um, because I know
some other people have to leave early because of,
um, they have to get someplace in traffic,
may or may not be a nightmare. Um,
I'm going to try and get the majority of the
content into the first hour and then have, you
know, extended Q. And A. For the
last half an hour so that people who have to
leave early still get most of the content. Um
, and also just a quick raise of hands.
How many of you were at Brian Nosek's talk
on Monday? Okay. So for those of you
who are at the talk, there will not be
too much review. The first couple of slides set
up may sound somewhat familiar. Um, but most
of it is going to be new material. Um
, Right, so just to introduce myself, my
name is Courtney Soderberg. I'm the statistical method logical
consultant at the Center for Open Science or a nonprofit
in Charlottesville Virginia. And today I'm gonna be talking
to you about the reproducibility crisis in science and how
openness can increase the validity of published results. Um
and so as I said, just a little bit
about the center, we are a non profit startup
in Charlottesville and we are really committed to increasing openness
and transparency in science and all branches of science.
We do that through three main things. The first
is oh no, ha ha, I thought that
was a laser pointer. There we go. Um
the first one of these is infrastructure. So I'll
be talking later about the open science framework, which
is something that we built to facilitate open collaboration among
scientists. The next is meta science. So we
do science about science looking at replica bility rates in
different fields. And finally we have community efforts where
we attempt to, you know, increase different types
of behaviors, make scientists aware of various um new
tools to help their workflow. And as part of
that um is my job where I go around and
kind of talk to scientists about workflow and practices and
then actually help them implement these things in their own
research. So a lot of you in this room
come from different scientific disciplines, some of your librarian
, some of your statisticians and then a lot of
you are scientists from everything from psychology and health science
. And in general scientists, no matter what they
study, share a few common broad ideals. The
first is innovative ideas. As scientists. We want
to be studying new ideas, we want to be
developing and figuring out new connections, new phenomenon that
are out there in the real world. The second
thing, though, is we want to have reproducible
results, so we want to have faith that we
found something and that will be able to find it
again and then other people will be able to find
it again because we take this reproducibility is kind of
a hallmark of reality. So if I find something
, but I'm the only one who could ever find
it that may suggest that maybe it's not quite as
robust as we thought it was, Maybe it was
just noise. And so the last thing is that
in general sciences about accumulating knowledge, so each individual
study is a piece of evidence and over time those
pieces of evidence accumulate into some broader understanding about how
the world works. And so if these are the
scientific ideals, the vast majority of scientific disciplines have
kind of taken on this. I hate saying this
word, hypothetical deductive scientific model, which if you
remember way back when when you took research methods,
probably an undergrad. Um this is the general scientific
method circle, right? So you generate some specific
hypotheses, you design your study to test those,
collect your, collect your data, analyze the data
, to test our hypothesis, then you interpret your
data and then you publish your results or you go
on to the next experiment and then you just repeat
this over and over and over again. And so
the idea is that by following this model, what
comes out at this end point? Should this point
, it really doesn't trouble, does it? So
what comes out at this end point is something that
we should have a good amount of faith in that
it's real. So if we get a significant result
, P is less than.5, and we went
through this model, that should mean that that result
is trustworthy, and we've discovered something real about how
the world works. So the problem is that over
the last few years there's been mounting evidence to suggest
that perhaps the results of our scientific endeavor is not
quite as solid as we thought it was. So
there's mounting evidence to suggest that replica bility rates in
a broad variety of scientific fields, everything from cancer
biology to pharmacology, too. Psychology and neuroscience aren't
all that high. There's a lot of failures to
replicate, which suggests that perhaps a lot of what
is published in the literature is a false positive.
So you found a significant result, but it was
actually a false result. You were just picking up
a noise that effect isn't actually out there in the
real world. And so these low rates of replication
, these potentially high rates of false positives have suggested
that the published literature isn't quite as valid as we
thought it was, and so we're not actually accumulating
as much knowledge as we thought. And so what's
causing these potential problems? So there are a long
list of things and these are in no way kind
of a there are other problems that are not on
this list. Um, there's a whole another set
of problems that talks specifically about theory and theoretical specification
. These are more methodological problems, but there's still
an important subsection. And so the first is low
power. I'm not going to go into great detail
about that today, just because you could have,
like a whole nother to our workshop on that,
but the idea is that in general, across a
wide variety of scientific disciplines, the power of studies
that are being run is particularly low. So,
statistical power for those of you who may not know
is just the likelihood that you find a significant result
if that effect is really out there in the real
world. So if I'm testing whether there is a
average height difference between men or men and women,
the power of my study is how likely I am
defined that there is a height difference between men and
women. Um and so there's some research that's been
done that shows that power in things like neuroscience and
psychology, It's right around 31 41%. Um which
means that if I were to take 100 samples of
men and women with that power, only 31 of
those samples would come back with a height difference.
So that's a really low kind of return on investment
in terms of how many studies I would have to
do to get a significant result when that effect is
actually real in the population. And so low power
is a problem for reproducibility and replica bility because it
can lead to results not replicating, not because the
effect isn't real, but because you just don't have
enough participants, subjects mice, whatever you're studying,
um, you don't have enough statistical power to find
that effect. Um, and so these other problems
are more related to false positives. So when you
find a significant result, but nothing was actually there
in the real world. Um, and some of
these I'm going to go into in greater detail later
on. So there is questionable research practices, um
, over abundance of positive results. So there are
literally too many positive results in the literature. Um
, if we think about how underpowered a lot of
sciences are, and also just that everything we study
is probably not real, we should have a higher
proportion of non positive or no results in the literature
and actuality. Um, it depends a little bit
on the discipline, but it's usually around 90 95
of, you know, published results are positive,
which is just literally too high mathematically, um kind
of going along with that, ignoring all results.
As scientists, we generally, we pay a lot
of attention to positive results. We don't really do
anything with no results, right, They just kind
of get pushed into a file drawer, forgotten,
lonely and sad, unused. Um, another part
is a lack of replication. So, because the
scientists, we really are interested in these innovative results
, usually somebody does a study and then nobody ever
tries to replicate that study. And so it's hard
to kind of accumulate knowledge about that specific study just
because if nobody replicates it, you don't actually get
accurate estimate of how replicable that effect is, and
finally, the limits of null hypothesis testing. So
null hypothesis testing is very good for some things,
it has some problems in other areas, um but
in general it's the only in most disciplines, it's
really what is the only thing that is used for
scientific investigation, which is creating some problems? Um
So I'm going to go into kind of three specific
areas and what researchers can do to help the problems
to help deal with those problems in their own research
, kind of really practical implementations of what researchers can
do. Um But I did want to discuss a
little bit about why people engage in these behaviors just
because I think a lot of times when we start
talking about lack of replica bility and all results and
things like that, the first thing that people think
about is fraud um and they think that, you
know, there's nefarious behavior going on in any,
and this can, you know, lead to a
lot of people feeling very right. If people's results
don't replicate, that is not necessarily an indication that
they engaged in any fraudulent behavior, right? There
is fraud that happens, but that is actually a
very small proportion of these failures to replicate. Most
of it is studies that were done in very good
faith by by scientists who believe that they were doing
things right and by the way their field was actually
doing science, they were doing what their field considered
right, but some things were going on behind the
background that was kind of causing some unconscious motivation to
seep into how the science was done and kind of
alter that ideal scientific circle. Um so the first
thing is perceived norms. Um I'm a social psychologist
by trade. Um that was my PhD my background
and such psychologists talk a lot about how people tend
to engage in whatever they think the normative behavior is
. So generally, if you see other scientists working
in a particular fashion, you'll start to work in
that fashion as well because you think that is what
is normative lee done. Um so right now,
most scientists, they kind of, they don't share
their data, they don't really openly publish their data
sets of material a lot of times they don't publicly
post their code. And so as younger scientists,
when we're trained up, we kind of see how
it's done and go, okay, this is how
it's being done, that is what I will do
to and so these behaviors kind of continually go through
the generations. Another thing is motivated reasoning. So
what this means is that people have cognitive biases and
we unconsciously think in ways that conform to what we
would like to believe. Um, so this will
come back in when we talk about those questionable research
practices. But the idea is that if you are
, you know, when you design a study,
you probably have a particular hypothesis in mind, you
are motivated to find that hypothesis. You're motivated to
find that significant result. And because of that motivation
, it can start causing you to unconsciously engage in
behaviors and you don't realize you're doing it. But
you know, after the fact you may go,
okay, well why did I do that? And
you can come up with a completely justifiable reason why
you do that. But it was because you had
that hypothesis in the first place, this will become
a little bit more clear and a couple of slides
when I give a concrete example. Um, so
another thing is minimal accountability. When I'm, you
know, working in a lab, I'm the only
one who really kind of knows what's going on.
Probably my lab colleagues also, no, but anybody
buddy outside my lab, all they see is my
research paper. They don't see the whole process.
And so a lot of this motivated reasoning can happen
because only the endpoint is known. Nobody knows where
we started and nobody kind of knows how we got
to the final product. And so there's not a
lot of work flow documentation across the entire workflow.
And so that can lead to minimal accountability. The
last one or the second to last one is the
incentive structures. Right? In terms of how 10
years given out, in terms of how people get
jobs. It's all about how many papers do you
have published and what is the impact of the journals
you have them published in? And so because of
this incentive structure, it really motivates people to look
for those significant results, which can set off a
lot of this motivated reasoning. So it kind of
feeds back into the top and goes down. And
the final one is, I'm busy, right?
All scientists are very busy people, we have a
lot to do. And so any time you talk
about maybe changing aspects of the scientific workflow, the
first thing that pops into everybody's mind and I completely
get this is, oh God, that sounds like
more work, which, you know, nobody wants
to do more work, nobody has time to do
more work. Um but what I would like to
argue is that a lot of these changes are small
. They're not necessarily changing a great amount of work
, They're just moving work around and that they can
have really nice consequences for kind of the longevity of
individual scientists work and the longevity of research lines as
a whole. And so that short to long term
trade off is potentially worth wild. So what can
fix a lot of these problems? Many people have
been talking about many different things. I'm going to
specifically talk about three things in a very kind of
concrete way in terms of how they can be implemented
. The first is pre registering studies followed by increased
documentation of the entire workflow, and the final thing
is changing norms and incentive structures. So the first
two can very much be done by an individual researcher
or an individual lab. They're not really contingent on
anyone else. The last one is kind of contingent
on the greater scientific community as a whole. Um
but I think it's particularly important and things have begun
to shift in that area. Um And so I
did want to talk about it just a bit at
the end. Um but kind of, the overarching
theme of all of this is increasing openness and transparency
of research, by engaging in more open and transparent
research practices that can actually fix a lot of the
problems I was talking about on the other slides and
make the published literature a little bit more trustworthy,
a little bit more reproducible and a little bit more
valid. So, the first thing I'm going to
talk about is study preregistration. Um So, how
many of you have heard of something called clinical trials
dot gov. Okay, so a couple of people
in the room. Um so clinical trials dot gov
is a obviously it's a website run by the government
um and all phase three clinical trials that receive funding
from NIH and NSF before they start have to go
to this website and say exactly what they're gonna do
. So they say their hypotheses, they say they're
what analyses there are going to run. They say
what they're dependent variables are they basically right up the
intro methods and analysis portion of a research paper and
uploaded to the site. So clinical trials have studied
preregistration. That is what study preregistration is, which
is before a study is run ideally before the study
is run. If you work in an area where
you use large data sets that are already existent,
um it would be before you looked at the data
and started doing data analysis, you specify your research
question your hypotheses, study design materials and data and
data analysis plan. And so what this does,
this does a couple of different things. So the
first is a decrease of something called harking which is
little term that stands for hypothesizing after results are known
. Yeah, the last three letters should have been
lower case. They don't stand for anything. And
so basically what this means is because you've specified what
your hypothesis is up front. If let's say your
results turn out to be different than you expected,
you can't then after the fact go back and change
what your hypothesis was. That's what harking is.
It can be problematic because what it does is it
presents hypotheses data analytic choices as a priori when really
they were post talk when they were made after you
looked at your data. So the second thing it
does is it decreases what are known as researcher degrees
of freedom gonna go into more detail about this on
the next slide. But this is another name for
those questionable research practices and the reason it decreases these
is because it locks you in a priority to a
specific analysis plan. And the last thing it kind
of does is it decreases that file drawer problem.
So if most all results are kind of put in
a drawer and never published. We as the scientific
community don't really know they existed because of pre registration
. If that preregistration is public like it is on
clinical trials dot gov we have a record of every
single study that was run, we may not know
the results of it, but we know that hey
, it was done and it wasn't published. So
we can go and seek out those researchers and say
, hey, what did your study look like?
And so we are better able to find unpublished studies
and better able to accumulate knowledge using those studies.
And so I did want to talk in more detail
about researcher degrees of freedom because they are kind of
the particular crux of why pre analysis are. Pre
pre registration is important. So research degrees of freedom
are all data processing and analytical choices that are made
after seeing and interacting with your data. So they're
data dependent choices. And so to give you some
examples of what counts as a researcher degree of freedom
. These are things like should more data be collected
, Should some observations be excluded? What conditions should
be compared? Which variables should I control for?
Which variables should I use as my main devi what
is the statistical effective interest to test my hypothesis?
Now, what you'll notice is that these are all
very important questions, right? You cannot do a
research study without asking and answering these questions. So
the issue is not the question. The issue is
when these questions are asked and answered by the researcher
. If you think about these questions and come up
with an answer before you started, you know,
analyzing your data, everything is good, everything is
fine. However, if you start answering these questions
after you've looked at your data, all of a
sudden the answers to these questions become data dependent.
The way your data is turning out is going to
probably unconsciously change how you answer these questions and that's
going to create some problems specifically what it's going to
do. It's really going to inflate your false positive
rate and your p values aren't going to be informative
anymore. So basically when you start making data dependent
choices, the number of false positives are getting.
So the number of times you're going to show a
significant effect when that effect is not really there in
the population is going to start going up Now.
This can go up to anywhere around 61 false positive
rate. Um so if you're from a field where
you usually set your false positive rate, your p
value 2.055 61 is really high. Right? That's
better. I mean Flipping a coin would actually give
you lower odds than 61%. And so because that
false positive rate is so high, the p values
you're getting from your data analysis aren't really informative anymore
. They're not telling you something real about what's going
on your data and kind of to give you a
concrete example of what I mean by these data dependent
choices. Let's go back to the example of,
you know, looking at the difference in height between
men and women. So let's say I have my
study and I initially decided I'm going to have two
dependent variables. I'm going to have people self report
what their heart is. And I'm also going to
get an R. A. To actually measure them
with a tape measure. So I run my study
Get my two dependent variables and one of them is
significant and one of them is not. So the
self reported height was significant. The tape measure height
was not Now. I'm sitting in my lab meeting
with um my other collaborators and we're kind of talking
about the results, talking about how we're going to
write it up and somebody says, you know,
I don't think that tape measure one is really,
you know, all that ballad of a dependent variable
. Like yeah, it seems face valid but you
know a lot of your race or women they're kind
of short. If they're trying to measure how tall
a really tall guy is. Right? They probably
won't be able to see the top. The tape
measure very well. It's gonna be like tilted at
an angle. They're gonna have a lot of trouble
accurately getting a read on that. So I don't
think that dependent variables particularly valid. You should just
throw it out and go with the more dependent one
. This can happen all the time. Right?
That doesn't seem like it's particularly focus logic or like
you're trying to make your study seem nicer. Sometimes
reviewers will actually ask you to do this and throw
out a D. V. Right? This is
just considered kind of absolutely fine behavior. The problem
is that if it turned out differently so the tape
measure was significant and the self reported height wasn't,
the researcher would probably come up with just as logical
sounding of a explanation of why they should throw out
the other D. V. So you can think
well you know self reported height. There are stereotypes
about how tall men and women are supposed to be
. And you know maybe some people are saying how
tall they'd like to be rather than hotel. They
are right to fudging the numbers. The tape measure
is far more face valid right? They can't it's
how tall they really are. They can't do any
fudging so all throughout the tape measure or want to
keep the self reported one right. The problem with
that is it was completely dependent on how your data
turned out. You didn't decide this a priori.
And so that's what's causing those false positive to spike
. And so at the bottom right. These often
feel very reasonable in the moment Because of that motivated
reasoning we talked about earlier, but they're still going
to increase this false positive rates, sometimes as high
as 61%, which is a real problem. And
so one way to tamp down those false positive rates
and keep those p values as face valid as possible
is something called a pre analysis plan. And so
what a pre analysis plan is doing is asking the
researcher to specify as specifically as possible all the analyses
they're going to do in their study, it generally
includes four parts the first talking about the target sample
. So specifying the size of the sample, you're
going to have what you're stopping rule is. So
how many people, at one point you're gonna stop
your study. How many people are you going to
have? Who are you going to have? And
who are you going to sample? The second part
is about data cleaning and processing. Um So if
you work in an area like FmRI research or er
p research or reaction time research, there's a lot
of cleaning that happens to that data before the analysis
ever starts. There's a lot of transformations that happen
. Certain trials may be thrown out for things like
head movement. Um Now and different researchers are going
to there are a lot of different ways to do
that processing. Um there's a paper by uh carp
and I think it's 2012 that looked at kind of
all the different possible processing choices you can make on
FmRI data from the published literature and you came up
with something like 30,000 different potential ways to process FmRI
data all in the published literature. So there's a
huge number of ways to do this in ways that
researchers could choose from. And so it's important to
say which ones you're going to do, The next
one is exclusion criteria. So who are you excluding
? And why are you going to exclude people who
failed manipulation checks? Are you going to exclude people
who fell asleep during your studies? This happened to
me in ground school more than I would like to
say. Also drunk people in my studies, they
got excluded to. Um but right, going through
all the reasons potentially why you might exclude people.
And this is also talking about how you're going to
identify outliers. And the last thing is specify all
the actual analyses you're gonna conduct. So what statistical
tests are you going to conduct? Um you know
, what variables are you going to use? What
TVs are you going to use? What controls are
you going to use? And so the idea kind
of the the idea is to specify this as much
as humanly possible. There's a range right that you
could do. Some people write it out and kind
of plain english. Some people go as far as
to actually write up the entire analysis scripts so when
their data comes in, all they have to do
is highlight the script, click run and magic,
all their data analysis is done. Um Generally when
people start doing these, they start at the one
end where they're just describing and thinking about these things
and they slowly start getting more and more specific um
but kind of making as many decisions as possible.
Takes away those opportunities for researcher degrees of freedom.
And so it really keeps those p values in check
. And so because of that, right, it
decreases those researches of freedom And because you've written all
this out in advance and it's not just like sitting
around in your head, it holds you accountable to
what you said you were going to do and it
holds you accountable to others to what you said you
were going to do. So you can actually look
at the analysis you did and go back and look
at what you said you were going to do and
see, okay, how are these different, why
are they different? You know, were the choices
I made the different things that I did legitimate or
not? And other people can make those same.
You can ask the same questions and potentially come to
different conclusions. Now I talked about clinical trials dot
gov That's a public registry, right? Anybody can
go on there and see what you said you were
going to do. Um You can imagine that you
could create a private registry, You could create a
data analysis plan, completely private. Um but making
it public has two main advantages. Right? It
helps with replication attempts. Just because you said exactly
how you're going to do the study and exactly you
know what your analysis scripts are gonna be. So
somebody can then go and use that information when they
try and replicate your findings. And it also decreases
that file drawer effect, which we talked about before
. Other people know what studies you plan to do
. So we have a better estimate of what has
actually been done now. You might be wondering.
Okay, so you've just told me that I should
specify all my analyses before. I do them.
What about exploratory testing? Does this mean I'm not
allowed to do exploratory testing? No, completely not
. Exploratory analysis is completely fine and it's fantastic.
And I love exploratory analysis. Um one of my
teachers in grad school so that if he had to
do one type of analysis for the rest of his
life, it would be exploratory analysis. Right?
Exploratory analysis is incredibly important to scientific discovery. It
really is about generating new findings. The problem is
the confirmatory and exploratory analyses have different purposes. And
when you start dressing up exploratory analysis as confirmatory,
that's when science starts to get into a little bit
of trouble. So as I mentioned, exploratory analyses
is really for generating new hypothesis. Looking into what
might potentially be out there to look into in another
study to confirm it. A confirmatory analysis is when
you really want to say, okay, I have
an idea about what's going on here. I actually
want to test if that's really what's going on here
. So confirmatory analysis is about testing. Exploratory analysis
is about exploring discovery, right? Just investigating what
might be out there for the further confirmatory testing in
the future. And so clearly delineating between the two
is what the point of a pre analysis plant is
. The pre analysis plan says okay these are all
of my confirmatory tests. You can then go and
do other tests as well. But because you preregistered
the confirmatory ones it will be very clear to you
, to the editors and to the readers what the
exploratory analyses were and what the confirmatory analyses were.
So that when people are interpreting your results they can
take that into account figures. Sure. So let's
say I did my pre analysis plan and I said
you know I'm going to look for height differences between
men and women using a T. Test because that's
the simplest thing T. Test and my dependent variable
is going to be self reported height. So you
do that you find a null result. Um But
you would also because of how you know university works
you would also had a bunch of questionnaire measures that
they also had just happened to have responded to earlier
in this semester. So you can start thinking okay
I found a no result. But you know maybe
some people are more uh how the stronger eager how
they you know oh here's a better example. Um
So let's say you have a combination of male and
female raise and your confirmatory plan. You have not
taken that into account. You get the snow effect
and you kind of start thinking maybe the gender of
the R. A. Was changing how people self
reported their height, right? If a girl had
a male ari she might be more likely to decrease
her height or under report her height because she thinks
that will make her seem more attractive. Um Whereas
if she had another female ari she might be less
likely to do that. And so let's say you
do an exploratory test where you actually um do an
interaction and see how men and women who had male
versus female are a self reported their height. And
you found a significant result there. That interaction would
be the exploratory analysis. You didn't pre register it
. It was kind of post talk after the fact
after you saw that in all result it's a data
dependent decision. So that would be exploratory so that
p. Value around it is maybe not all that
face ballad. So what you actually want to look
for instead is start looking at the effect size the
confidence intervals and say okay you know I found this
during exploratory testing, Let me then do a confirmatory
test where I set out to look specifically for this
interaction and see if I find it again. Does
that make sense? Okay. Yeah. And so
the idea is not that any one type of analysis
is more important but it's more about thinking about moving
from exploratory to confirmatory analyses. Right? So directly
replicating exploratory results in a confirmatory fashion. Um You
know, I come from social psychology where we do
a lot of, we usually use undergrads. Undergrads
are cheap, it's really easy to replicate studies.
Um In a lot of other areas it may be
harder to get participants or you may be working with
large datasets like let's say the National Elections Survey,
which they do every Other every two or 4 years
. Um You can't just say, hey, could
you guys go out and collect another dataset for me
? You have no control over that. Um I
did want to mention there are still ways to do
more confirmatory tests even when you can't actually go out
and replicate a study. When you have kind of
set data sets. One is to, if the
data sets large enough take it and randomly split it
into two halves. Do your exploratory analysis on one
half, kind of figure out what you think is
going on and then take the other half and do
your confirmatory analysis on that half. Um This is
something that's really common in personality research and people that
are looking at these big complex models. There are
actually different tests. There's something called exploratory factor analysis
and confirmatory factor analysis. And a lot of times
they'll be working with these big data sets and they'll
randomly split them. Another kind of variant of this
that you see a lot in machine learning or computer
science is where you train up your statistical model on
a sample data set and then you go and kind
of confirm it on another dataset. Then you got
off the internet. Or if it was a big
data set originally You'll split it into 2/2 and have
a training data set. That exploratory data set and
then you're testing data set, the confirmatory data set
. Um And if you really are in a situation
where let's say you know you do medical research and
it's a group of patients where it's a small group
so you can split your sample and you're never going
to be able to get them again. It's not
to say that you can't still run exploratory analyses,
it's just to be clear about that so that everybody
knows that the exploratory and we can take that into
account when we're interpreting the findings. Um, and
there are also things you can do right. You
can run kind of sensitivity analysis and check for the
robustness of the results. So let's say you had
excluded some outliers and your effect pops out. It
was exploratory, but you still think it's interesting.
But you want to show that it wasn't anything particular
about the analysis I chose. You can run a
bunch of different analyses using different outlet exclusion methods and
show, hey, this keeps popping up every single
time. Maybe there is. I'm gonna trust this
more than something where it really depended on the specific
outlier effect I chose. So there are ways to
kind of increase the robustness of exploratory testing, but
it still is important to delineate between that exploratory and
confirmatory testing. Um, so before I move on
to the next section, does anybody have any questions
about kind of analysis? Plans and pre registration?
Yeah. All right. Start with this. Mm
hmm. And speak my feel like your product on
Reddit training. Okay. Mhm. Yes. Mhm
. This helps. Yeah. Methods and Yeah.
And then you have standing on. Right. That's
actually back. It is a close and I understand
. Yeah. All this process seems the same thing
that like reading the grand. Very clear. Mhm
. And all right. So I went green.
What if? Yeah. Yes. So, what
I would say is you know, a lot of
I think there is a lot of shift that can
happen during the research process. So, um you
know, when we describe the analyses we're going to
do, it's generally at a like abstract level.
I'm gonna compare these two groups and a lot of
scientific disciplines. It's not these are the actual tests
I'm going to run. These are the cauvery.
It's that I'm going to include. That's never it's
usually not actually written down that specifically before the test
happens. So that if you run something initially and
it's not turning out you can start adding in other
co very it's other tests trying other things out.
So a lot of data exploration is actually going on
but we don't realize that it's exploratory because we never
really clearly specified what we're gonna do beforehand. Um
Really the only science that works like that right now
is these phase three clinical trials where they really do
have to very clearly say exactly what they're going to
do. Um Preregistration is not use much really in
any discipline other than that it's starting to take root
a little bit in econ it's starting to take root
a little bit in psychology. Um I know from
psychology they there was I didn't put it up,
but there was a um there was a survey that
they sent out to researchers and asked them, you
know, if they engaged in some of these researcher
degrees of freedom behavior. So, you know,
running two dependent variables and only reporting one in their
final paper or you know, running extra people when
they didn't get a significant results. Um and then
when they found a significant result just publishing after the
fact and you get really high rates of people saying
yes, I engage in these behaviors and I think
it's fine because people don't realize what it's doing to
those false positive rates. Um And so I think
, I think it happens more than we realize because
we don't write down beforehand what we're going to do
. We have a general idea, but it's not
very specific and that lack of specificity allows a lot
of leeway in terms of what we actually do.
Does that make sense? Yeah. To say.
Mm. Yeah. And if that's yes and then
you would suggest. Okay. Mhm. Yes.
That so stand, yep. Much later. In
fact meeting. Uh huh Holiness Dad. Very good
for the So I'm interested in hearing to those survey
solved instead of this has been great. And then
what would be sure, interesting being made for?
It's a I use brand. I'm not sure if
you say you're right. Mhm. So let's put
it there. Fine then. Right, think.
Mhm, mhm. And yeah, how do you
see that opening? Mm hmm. Yeah. So
pre registration doesn't necessarily actually involve any, it doesn't
have to involve any feedback from colleagues. It's more
, it's more you specifying what analyses you're going to
do beforehand so that when you actually do the analyses
you can say okay what I said I was gonna
do matches what I did and then reviewers editors and
you know people who read your article can see okay
this match what they said they were going to do
or this was confirmatory. This is exploratory. Um
There is something called a registered reports format which some
journals are now adopting. Gonna talk about that a
little bit later but I'll just talk about it now
. Um Which is where you actually send your paper
in for peer review before data collection has ever happened
. Um And so the paper is just reviewed in
terms of you know theoretical soundness, methodological soundness and
statistical soundness. That is where feedback comes into the
preregistration kind of world. But you can do pre
registrations without feedback if that does that make sense?
Yeah. It sounds like, I mean it sounds
like if you wanted to keep your you know your
study a secret but wanted to ensure that your results
were done in a way that think it's valid.
It looks like they have a way to allow for
sort of private. Yeah. So you you could
say I'm committing this report. It's embargoed. Nobody
can see it when I published. I will make
it apparent that I Yeah so the osF which I'm
going to talk about a little later um allows for
for you to register studies. Do preregistration. You
can make those public or private registrations and if you
initially make it a private one you can make a
public later on. Um something a lot of scientists
are worry about is something called scooping, right?
You're afraid for those of you who don't know the
jargon term. It's, you know, if you're
working on something and then somebody else is working on
the same thing and they published before you, you've
been scooped. Um, some scientists, many scientists
are actually quite worried about that because generally if somebody
else publishes the stuff, you can't then publish it
or you have to publish it in a less prestigious
journal. And so some people have worried about,
well, if I make my registration public before I've
ever done my data, might somebody see that quickly
run the study and scoop me? Um, and
also some grants, especially if you're working with some
government agencies, you can't actually release information about the
study until after it's done. Um, and so
the OSF does allow for private registrations that can be
made public later or that you can just let reviewers
see if you're doing registered reports or something like that
. Clinical trials Gov, which is the one kind
of other main registration system out there right now,
all those have to be public, but other sites
do not require that. All right. Um,
so now I'm going to start talking about the current
scientific workflow and how that can create some problems in
terms of openness and transparency and just the accumulation of
knowledge and ways research, simple ways that researchers can
increase the documentation of their workflow, which is actually
really good for them. And it's also good for
science as a whole. Right. And so currently
the vast majority of the scientific research workflow is obscure
. If I'm not in your lab, all I
see is what you published in your paper, which
have you ever looked at somebody else's paper and trying
to figure out exactly what they did. It is
surprisingly hard to do. Um, I'm working on
a project right now where we're trying to run direct
replications of other people's papers from, you know,
six years ago and some of the analyses you read
it, and you're like, I have, I
have no idea what you did, this makes no
sense. Um, or the methods are really complicated
and they are obviously missing really important points. And
then you email them and they go, oh yeah
, we completely left this out of the paper,
but here are the materials, right? All of
that goes unreported in the paper. And so it's
actually really hard to figure out what was actually done
in practice in the lab versus what appears in the
paper. And potentially how that research idea has changed
over time with that. It's also hard to reproduce
other people's work. And it can be actually really
hard to reproduce our own work. So the way
a lot of labs work, depending on the field
urine is each graduate student kind of has some projects
that they work on. And then a graduate student
, you know, leaves if they go into academia
, maybe they keep those projects going. If they
don't go into academia, they leave. And it's
really hard to get in touch with them. And
the project kind of languishes in the corner for a
couple of years and then another grad student comes along
. It's like, hey, I might be interested
in that and you go back and look at you
like this was two years ago. I have no
idea what I was doing here. I don't remember
what these variables mean. I don't actually remember what
my specific research question was. All of this is
kind of very obscure. Um, so there's a
, there's a great little video, um about this
which I took out for time, but it's these
two panda bears talking to each other, because everything
with panda bears, it's just cuter. And one
of them is asking for the other person's data.
Like, you know, you send me the data
and these variable names don't make any sense. What
what does SAm one mean? The guy goes,
oh that's my collaborator, Sam lee, that's short
for his name. The guy is like, okay
, what does it stand for? And he says
oh it's this protein. He goes, okay,
well what does SAM to stand for? And he
goes, oh, I don't know. This was
a couple years ago, my collaborator probably knows the
guy goes, okay, how do I get in
touch with him? And he goes, well,
he's in china, and his name's Sam lee,
you can probably find him. The guy goes,
no I cannot find him. Your data is useless
to me because I have no idea what these variables
mean. Um And so right everybody who watches this
video kind of laughs at it, but then there's
this moment where you kind of realize, wait I
have some data sets from a couple years ago where
I named it like D. V. One in
D. V. Two. I probably knew what
time and at that point in time I probably do
not remember what that means now. And like I
did this um I got my PhD you know in
june and I was packing up and moving all my
stuff and I was trying to organise studies that I've
done my first year in grad school and I had
no idea what half of them meant. Which is
really sad. I mean obviously they're not going to
go anywhere now, but there was a lot of
resources, both my own resources, participant resources,
lab resources that were kind of wasted. Because those
studies can't continue to move forward and they can't be
used by other people down the line. And so
, you know, it's hard to reproduce your own
work sometimes because you don't have a good handle on
what you actually did. And so the last thing
is it's difficult to accumulate unpublished knowledge just because,
you know, if it actually doesn't get published anywhere
, we have less of a track record of it
, at least when it gets published. Maybe you
well you have to say what your research question was
, what your hypothesis was, if it doesn't get
published and maybe you did didn't do a presentation on
it. Some of that maybe scribbled on a note
where somewhere notebook somewhere and now you've moved and God
knows where it is. And so you have all
these pieces and parts that aren't really connected up,
so you can't find them, Nobody else can find
them. It becomes really hard to accumulate that knowledge
. And so one good way around some of these
problems is to increase the documentation of the workflow.
So right now, all we have is documentation in
terms of what is actually published, but there are
a lot of other places along the workflow that it's
really important to actually keep track of what is being
done. And so the first thing is knowing how
things started and how they evolved over time. And
this is really important. As I said, because
you can track how things changed, you can track
. Okay, this is what my research question originally
was after doing a review. Maybe this is how
it started. This is what my analysis plan originally
looked like. Here's where it ended up. There's
something how many of you in this room have heard
of something called Version Control? Okay. Yeah.
Good. Um You are ahead of the game.
So right for those of you have not heard a
Version Control. It's okay. Um You know,
I'm not gonna throw you out or anything. It's
much more prevalent in some areas than other areas.
If you're from computer science or biostatistics, um you're
far more likely to have heard a Version control than
some of the social sciences. Um So version control
is basically just where a website or a computer program
or something like that logs every single version of a
file that you have. And so you can always
go back and find older files of it and you
can see how things have changed over time. So
like on your regular computer, if you're working on
a manuscript and you save over the same file,
you can't then go back and find the other file
in a version control system. You can go back
to those other files and you can see how they've
changed and how they're different from one another. So
the next thing is saving and annotating syntax. This
seems really simple but it's not done nearly as much
as it should be. Um I come as I
said I come from social psychology where SPS S.
Is still kind of the statistical tool of choice and
S. P. S. S. Can be
used in a point and click fashion. Things like
data and SAs can actually be used in a point
and click fashion. They're usually not. Usually the
code is coded out and you save that syntax file
which is great. Um If you use a point
and click program like S. P. S.
S. There's a little paste button on it and
you can actually save out that syntax. So saving
the syntax is really important because you can actually go
back and reproduce the exact statistical test you ran.
But annotation is really important to, so annotation is
actually writing in some comments into your code, saying
this is what I was doing and this is why
. Um So I just wanted to give you kind
of an example of what this looks like and how
annotation can help other people but also help you figure
out what the heck you were doing a couple years
ago. Where is my screen? There we go
. Okay so this is are for those of you
who are not familiar with our studio. Um It
is a lovely shiny little program. Can you all
read this or is this too small? Okay.
Yeah. Okay. I know this week, that's
not actually doing what I wanted to do. Does
anyone remember how? Oh wait wait, this is
what I want to do. Okay, is that
better? Okay, so I have three versions of
the same code. So this is just the syntax
saved. No, this is the annotated one.
Sorry, I'm trying to work with remote screening.
It's not going well. Right, so these are
all the same three code. There's just different levels
of annotation. So as you can see this is
just a giant block of code and probably if I
went back and read through this, I can maybe
figure out what I was doing, but I probably
wouldn't know why because all it is is the code
. There's no explanation of, you know why I
did what I did, what's going on? Why
did I make some of the particular choices that I
made? Um so the next example is just a
bare bones annotation. So you can see that a
lot of the code blocks just have these little identify
, here's what they're doing, maybe why I did
them. So this is helpful to me because it
gives me a little bit more insight into what in
the world I was thinking it's helpful for somebody else
because it makes my code more readable and a little
bit more understandable. So the last one is an
example of something using a package called knit are,
which is a fantastic package. Um, it's something
a little bit more akin to what is called literate
coding or literate programming, where you actually kind of
write a little narrative about your code. And so
what you'll notice about this is that they're actually these
big kind of sections of text where I'm really being
explicit about, okay, this is what I was
doing. This is why explaining in full sentences,
making it very readable. And then you have these
code chunks. It makes it a lot easier for
other people to figure out what you're doing. It
makes it easier for you to figure out what you're
doing. And so, you know, annotation is
the more you can kind of talk through in plain
english what you're doing, the more useful it becomes
to yourself and other people when you're trying to look
back at old code and figure out, okay,
what did I do and why did I do it
with statistics? A lot of times there are many
different choices you can make. And so that why
is actually an incredibly important piece of information to know
going forward. Right, so going back to this
, you represent her right? Um Right. And
so I talked about this a little bit before,
clearly naming variables and creating code books. Um if
you're somebody who works with huge surveys, code books
are probably very familiar to you. Um you know
, I and grad school did not work with huge
surveys, but it was something where, you know
, going back if I didn't have access to my
SP Ss file where I had said, what did
I code as female and what did I could is
male. All I have is the zero ones in
my data file. And it's actually incredibly difficult to
go back, especially if it's a couple of years
back and figure out, Ok, how was this
coded? Really simple? You know, documents just
saying zero is male. One is female can be
really helpful in terms of making your dataset more understandable
for you and other people. Um so the last
second to last one is making sure you actually know
what materials you use for your study. This may
seem really silly because most people probably think, hey
, I know what materials I use for my study
um in practice, this is surprisingly sneaky, especially
when it comes to raw data and syntax, right
? Especially if your analysis, scripts and tax changes
over time. What can happen is you get like
you open up a file and you have final data
, one final data to really final data, really
final data to your like what in the world is
this, what did I use for my analyses?
This makes no sense. Can you tell this happen
to me? I had my original advisor was,
he was a very unorganised man and so I picked
up some bad organizational skills early in my um graduate
career and it's easier to start out with, you
know, good organization, but they can be changed
over time, right? But you know, really
being sure that, you know, okay, these
are the materials I use for my study. This
is my original Sin Tax, this is my raw
data, having all of those and not having the
system where you have like a little number clicking up
on the end of the data file. And so
the last thing is knowing where everything is. A
lot of times our workflow is very um spread out
. So you have sex, like something's live in
your email, something's live on servers, something's livin
notebook, something's live in power point presentations, and
it's easy to lose those pieces and parts and so
finding a way to connect all of those um and
the demo I'm going to do, which is the
osF is going to particularly show you one way to
easily connect all these pieces and parts of your data
set. Version control things. Preregister things and really
kind of organize the pieces and parts of your workflow
. Right? And so kind of the idea is
if your workflow is well documented in two years,
will you remember what you did and why? When
you're thinking about workflow documentation? This is the question
I tend to ask myself because I think when you
think about other people, some people go well I
don't care about other people, but most people do
care about if they will be able to reuse their
stuff two years later. Um but usually if you
can remember what you did two years later, it
makes it easier for other people as well. So
it can be a really useful question to ask when
you're trying to think about, okay, what do
I want my workflow to look like? Um And
so because I'm running short on time, I thought
I could get through this as an hour. I
have failed that goal. Um I'm gonna skip over
this section because a lot of you were at brian's
talking. So you heard about badges? I talked
a little bit about registered reports. Um Right,
so the osf, so right, I was talking
about this idea of trying to, you know,
track everything that goes on in the workflow, from
idea development to study design to data, to analysis
, to publishing reports. Um And this will also
tie together some of those open themes. So if
you have computers and you want to follow along in
real time, I know I'm somebody who like learns
through doing um feel free, but I'm also going
to show it up on the computer for those of
you who don't want to follow along or don't have
laptops. Um, so if you want to follow
along, go to OsF dot Io Io stands for
indian Ocean. This is a developer thing. It's
not like some sketchy going to like ask you for
your bank account, Thanks for those of you who
are leaving right now. There's actually um if you
email me, I can actually send you to a
video where um, there's like an online tour if
you want to watch that to get an idea of
kind of what the OSF is and how it can
help document and connect workflow. Um just because I
do realize there's a football game and at Virginia Tech
, that apparently means that everybody turns into a pumpkin
with cars it for, um, which is very
new to me as a thing. My school was
not very football headaches. Okay. Mhm Oh, 1116 00:54:59.940 --> A:middle L:90% mm.