WEBVTT 1 00:00:08.839 --> 00:00:12.230 A:middle L:90% Yeah. Mm. So thank you all for coming 2 00:00:12.230 --> 00:00:14.720 A:middle L:90% today. Just a quick show of hands. How 3 00:00:14.720 --> 00:00:19.100 A:middle L:90% many people's cars turn into pumpkins at four. Okay 4 00:00:19.109 --> 00:00:20.469 A:middle L:90% , so that's a good number. So what I'm 5 00:00:20.469 --> 00:00:23.390 A:middle L:90% gonna try and do, um, because I know 6 00:00:23.390 --> 00:00:25.739 A:middle L:90% some other people have to leave early because of, 7 00:00:25.750 --> 00:00:28.030 A:middle L:90% um, they have to get someplace in traffic, 8 00:00:28.039 --> 00:00:29.649 A:middle L:90% may or may not be a nightmare. Um, 9 00:00:29.660 --> 00:00:31.690 A:middle L:90% I'm going to try and get the majority of the 10 00:00:31.690 --> 00:00:35.770 A:middle L:90% content into the first hour and then have, you 11 00:00:35.770 --> 00:00:38.100 A:middle L:90% know, extended Q. And A. For the 12 00:00:38.100 --> 00:00:40.289 A:middle L:90% last half an hour so that people who have to 13 00:00:40.289 --> 00:00:43.229 A:middle L:90% leave early still get most of the content. Um 14 00:00:43.240 --> 00:00:44.810 A:middle L:90% , and also just a quick raise of hands. 15 00:00:44.810 --> 00:00:46.479 A:middle L:90% How many of you were at Brian Nosek's talk 16 00:00:46.479 --> 00:00:49.810 A:middle L:90% on Monday? Okay. So for those of you 17 00:00:49.810 --> 00:00:50.869 A:middle L:90% who are at the talk, there will not be 18 00:00:50.869 --> 00:00:53.429 A:middle L:90% too much review. The first couple of slides set 19 00:00:53.429 --> 00:00:55.990 A:middle L:90% up may sound somewhat familiar. Um, but most 20 00:00:55.990 --> 00:00:58.250 A:middle L:90% of it is going to be new material. Um 21 00:00:58.939 --> 00:01:00.159 A:middle L:90% , Right, so just to introduce myself, my 22 00:01:00.159 --> 00:01:03.730 A:middle L:90% name is Courtney Soderberg. I'm the statistical method logical 23 00:01:03.730 --> 00:01:07.450 A:middle L:90% consultant at the Center for Open Science or a nonprofit 24 00:01:07.450 --> 00:01:10.450 A:middle L:90% in Charlottesville Virginia. And today I'm gonna be talking 25 00:01:10.450 --> 00:01:12.700 A:middle L:90% to you about the reproducibility crisis in science and how 26 00:01:12.709 --> 00:01:18.040 A:middle L:90% openness can increase the validity of published results. Um 27 00:01:18.049 --> 00:01:19.950 A:middle L:90% and so as I said, just a little bit 28 00:01:19.950 --> 00:01:21.840 A:middle L:90% about the center, we are a non profit startup 29 00:01:21.840 --> 00:01:25.200 A:middle L:90% in Charlottesville and we are really committed to increasing openness 30 00:01:25.209 --> 00:01:27.849 A:middle L:90% and transparency in science and all branches of science. 31 00:01:27.859 --> 00:01:30.680 A:middle L:90% We do that through three main things. The first 32 00:01:30.689 --> 00:01:34.299 A:middle L:90% is oh no, ha ha, I thought that 33 00:01:34.299 --> 00:01:37.319 A:middle L:90% was a laser pointer. There we go. Um 34 00:01:37.329 --> 00:01:38.010 A:middle L:90% the first one of these is infrastructure. So I'll 35 00:01:38.010 --> 00:01:41.549 A:middle L:90% be talking later about the open science framework, which 36 00:01:41.549 --> 00:01:44.909 A:middle L:90% is something that we built to facilitate open collaboration among 37 00:01:44.909 --> 00:01:47.310 A:middle L:90% scientists. The next is meta science. So we 38 00:01:47.310 --> 00:01:49.260 A:middle L:90% do science about science looking at replica bility rates in 39 00:01:49.260 --> 00:01:53.719 A:middle L:90% different fields. And finally we have community efforts where 40 00:01:53.719 --> 00:01:56.469 A:middle L:90% we attempt to, you know, increase different types 41 00:01:56.469 --> 00:02:00.829 A:middle L:90% of behaviors, make scientists aware of various um new 42 00:02:00.829 --> 00:02:02.180 A:middle L:90% tools to help their workflow. And as part of 43 00:02:02.180 --> 00:02:06.480 A:middle L:90% that um is my job where I go around and 44 00:02:06.489 --> 00:02:08.949 A:middle L:90% kind of talk to scientists about workflow and practices and 45 00:02:08.949 --> 00:02:12.090 A:middle L:90% then actually help them implement these things in their own 46 00:02:12.090 --> 00:02:15.069 A:middle L:90% research. So a lot of you in this room 47 00:02:15.069 --> 00:02:19.009 A:middle L:90% come from different scientific disciplines, some of your librarian 48 00:02:19.009 --> 00:02:21.120 A:middle L:90% , some of your statisticians and then a lot of 49 00:02:21.120 --> 00:02:23.659 A:middle L:90% you are scientists from everything from psychology and health science 50 00:02:24.139 --> 00:02:27.849 A:middle L:90% . And in general scientists, no matter what they 51 00:02:27.849 --> 00:02:30.819 A:middle L:90% study, share a few common broad ideals. The 52 00:02:30.819 --> 00:02:32.680 A:middle L:90% first is innovative ideas. As scientists. We want 53 00:02:32.680 --> 00:02:35.460 A:middle L:90% to be studying new ideas, we want to be 54 00:02:35.460 --> 00:02:38.240 A:middle L:90% developing and figuring out new connections, new phenomenon that 55 00:02:38.240 --> 00:02:40.520 A:middle L:90% are out there in the real world. The second 56 00:02:40.520 --> 00:02:43.409 A:middle L:90% thing, though, is we want to have reproducible 57 00:02:43.409 --> 00:02:45.840 A:middle L:90% results, so we want to have faith that we 58 00:02:45.840 --> 00:02:46.939 A:middle L:90% found something and that will be able to find it 59 00:02:46.939 --> 00:02:49.710 A:middle L:90% again and then other people will be able to find 60 00:02:49.710 --> 00:02:52.139 A:middle L:90% it again because we take this reproducibility is kind of 61 00:02:52.139 --> 00:02:55.580 A:middle L:90% a hallmark of reality. So if I find something 62 00:02:55.580 --> 00:02:58.099 A:middle L:90% , but I'm the only one who could ever find 63 00:02:58.099 --> 00:03:00.150 A:middle L:90% it that may suggest that maybe it's not quite as 64 00:03:00.150 --> 00:03:02.069 A:middle L:90% robust as we thought it was, Maybe it was 65 00:03:02.069 --> 00:03:05.719 A:middle L:90% just noise. And so the last thing is that 66 00:03:05.719 --> 00:03:08.729 A:middle L:90% in general sciences about accumulating knowledge, so each individual 67 00:03:08.729 --> 00:03:10.909 A:middle L:90% study is a piece of evidence and over time those 68 00:03:10.909 --> 00:03:15.030 A:middle L:90% pieces of evidence accumulate into some broader understanding about how 69 00:03:15.030 --> 00:03:17.439 A:middle L:90% the world works. And so if these are the 70 00:03:17.439 --> 00:03:23.330 A:middle L:90% scientific ideals, the vast majority of scientific disciplines have 71 00:03:23.330 --> 00:03:25.610 A:middle L:90% kind of taken on this. I hate saying this 72 00:03:25.610 --> 00:03:30.300 A:middle L:90% word, hypothetical deductive scientific model, which if you 73 00:03:30.300 --> 00:03:31.530 A:middle L:90% remember way back when when you took research methods, 74 00:03:31.530 --> 00:03:35.729 A:middle L:90% probably an undergrad. Um this is the general scientific 75 00:03:35.729 --> 00:03:38.050 A:middle L:90% method circle, right? So you generate some specific 76 00:03:38.050 --> 00:03:42.210 A:middle L:90% hypotheses, you design your study to test those, 77 00:03:42.219 --> 00:03:45.460 A:middle L:90% collect your, collect your data, analyze the data 78 00:03:45.469 --> 00:03:47.840 A:middle L:90% , to test our hypothesis, then you interpret your 79 00:03:47.840 --> 00:03:51.460 A:middle L:90% data and then you publish your results or you go 80 00:03:51.460 --> 00:03:53.810 A:middle L:90% on to the next experiment and then you just repeat 81 00:03:53.810 --> 00:03:54.620 A:middle L:90% this over and over and over again. And so 82 00:03:54.620 --> 00:03:58.599 A:middle L:90% the idea is that by following this model, what 83 00:03:58.599 --> 00:04:00.599 A:middle L:90% comes out at this end point? Should this point 84 00:04:00.599 --> 00:04:02.120 A:middle L:90% , it really doesn't trouble, does it? So 85 00:04:02.120 --> 00:04:04.990 A:middle L:90% what comes out at this end point is something that 86 00:04:05.000 --> 00:04:08.500 A:middle L:90% we should have a good amount of faith in that 87 00:04:08.500 --> 00:04:11.669 A:middle L:90% it's real. So if we get a significant result 88 00:04:11.680 --> 00:04:14.250 A:middle L:90% , P is less than.5, and we went 89 00:04:14.250 --> 00:04:15.420 A:middle L:90% through this model, that should mean that that result 90 00:04:15.420 --> 00:04:18.230 A:middle L:90% is trustworthy, and we've discovered something real about how 91 00:04:18.230 --> 00:04:21.279 A:middle L:90% the world works. So the problem is that over 92 00:04:21.279 --> 00:04:25.050 A:middle L:90% the last few years there's been mounting evidence to suggest 93 00:04:25.050 --> 00:04:28.889 A:middle L:90% that perhaps the results of our scientific endeavor is not 94 00:04:28.899 --> 00:04:30.779 A:middle L:90% quite as solid as we thought it was. So 95 00:04:30.779 --> 00:04:34.509 A:middle L:90% there's mounting evidence to suggest that replica bility rates in 96 00:04:34.509 --> 00:04:38.620 A:middle L:90% a broad variety of scientific fields, everything from cancer 97 00:04:38.620 --> 00:04:43.720 A:middle L:90% biology to pharmacology, too. Psychology and neuroscience aren't 98 00:04:43.730 --> 00:04:46.189 A:middle L:90% all that high. There's a lot of failures to 99 00:04:46.189 --> 00:04:47.860 A:middle L:90% replicate, which suggests that perhaps a lot of what 100 00:04:47.860 --> 00:04:50.560 A:middle L:90% is published in the literature is a false positive. 101 00:04:50.569 --> 00:04:54.199 A:middle L:90% So you found a significant result, but it was 102 00:04:54.199 --> 00:04:56.850 A:middle L:90% actually a false result. You were just picking up 103 00:04:56.850 --> 00:04:59.060 A:middle L:90% a noise that effect isn't actually out there in the 104 00:04:59.060 --> 00:05:01.180 A:middle L:90% real world. And so these low rates of replication 105 00:05:01.189 --> 00:05:04.680 A:middle L:90% , these potentially high rates of false positives have suggested 106 00:05:04.680 --> 00:05:08.560 A:middle L:90% that the published literature isn't quite as valid as we 107 00:05:08.560 --> 00:05:11.220 A:middle L:90% thought it was, and so we're not actually accumulating 108 00:05:11.220 --> 00:05:14.209 A:middle L:90% as much knowledge as we thought. And so what's 109 00:05:14.209 --> 00:05:16.850 A:middle L:90% causing these potential problems? So there are a long 110 00:05:16.850 --> 00:05:19.589 A:middle L:90% list of things and these are in no way kind 111 00:05:19.589 --> 00:05:23.360 A:middle L:90% of a there are other problems that are not on 112 00:05:23.360 --> 00:05:25.810 A:middle L:90% this list. Um, there's a whole another set 113 00:05:25.810 --> 00:05:30.220 A:middle L:90% of problems that talks specifically about theory and theoretical specification 114 00:05:30.220 --> 00:05:32.089 A:middle L:90% . These are more methodological problems, but there's still 115 00:05:32.089 --> 00:05:35.370 A:middle L:90% an important subsection. And so the first is low 116 00:05:35.370 --> 00:05:38.879 A:middle L:90% power. I'm not going to go into great detail 117 00:05:38.879 --> 00:05:40.470 A:middle L:90% about that today, just because you could have, 118 00:05:40.470 --> 00:05:43.040 A:middle L:90% like a whole nother to our workshop on that, 119 00:05:43.050 --> 00:05:46.230 A:middle L:90% but the idea is that in general, across a 120 00:05:46.240 --> 00:05:49.709 A:middle L:90% wide variety of scientific disciplines, the power of studies 121 00:05:49.709 --> 00:05:53.699 A:middle L:90% that are being run is particularly low. So, 122 00:05:53.709 --> 00:05:56.250 A:middle L:90% statistical power for those of you who may not know 123 00:05:56.439 --> 00:06:00.620 A:middle L:90% is just the likelihood that you find a significant result 124 00:06:00.629 --> 00:06:01.829 A:middle L:90% if that effect is really out there in the real 125 00:06:01.829 --> 00:06:04.040 A:middle L:90% world. So if I'm testing whether there is a 126 00:06:04.040 --> 00:06:06.939 A:middle L:90% average height difference between men or men and women, 127 00:06:06.949 --> 00:06:10.180 A:middle L:90% the power of my study is how likely I am 128 00:06:10.180 --> 00:06:12.449 A:middle L:90% defined that there is a height difference between men and 129 00:06:12.449 --> 00:06:15.129 A:middle L:90% women. Um and so there's some research that's been 130 00:06:15.129 --> 00:06:17.250 A:middle L:90% done that shows that power in things like neuroscience and 131 00:06:17.250 --> 00:06:21.629 A:middle L:90% psychology, It's right around 31 41%. Um which 132 00:06:21.629 --> 00:06:24.629 A:middle L:90% means that if I were to take 100 samples of 133 00:06:24.629 --> 00:06:27.839 A:middle L:90% men and women with that power, only 31 of 134 00:06:27.839 --> 00:06:29.970 A:middle L:90% those samples would come back with a height difference. 135 00:06:29.970 --> 00:06:32.100 A:middle L:90% So that's a really low kind of return on investment 136 00:06:32.100 --> 00:06:33.750 A:middle L:90% in terms of how many studies I would have to 137 00:06:33.750 --> 00:06:36.500 A:middle L:90% do to get a significant result when that effect is 138 00:06:36.500 --> 00:06:40.509 A:middle L:90% actually real in the population. And so low power 139 00:06:40.509 --> 00:06:43.839 A:middle L:90% is a problem for reproducibility and replica bility because it 140 00:06:43.839 --> 00:06:46.600 A:middle L:90% can lead to results not replicating, not because the 141 00:06:46.600 --> 00:06:48.069 A:middle L:90% effect isn't real, but because you just don't have 142 00:06:48.069 --> 00:06:51.639 A:middle L:90% enough participants, subjects mice, whatever you're studying, 143 00:06:51.649 --> 00:06:55.470 A:middle L:90% um, you don't have enough statistical power to find 144 00:06:55.470 --> 00:06:59.670 A:middle L:90% that effect. Um, and so these other problems 145 00:06:59.680 --> 00:07:02.540 A:middle L:90% are more related to false positives. So when you 146 00:07:02.540 --> 00:07:05.329 A:middle L:90% find a significant result, but nothing was actually there 147 00:07:05.329 --> 00:07:08.329 A:middle L:90% in the real world. Um, and some of 148 00:07:08.329 --> 00:07:10.910 A:middle L:90% these I'm going to go into in greater detail later 149 00:07:10.910 --> 00:07:14.149 A:middle L:90% on. So there is questionable research practices, um 150 00:07:14.160 --> 00:07:15.399 A:middle L:90% , over abundance of positive results. So there are 151 00:07:15.399 --> 00:07:18.430 A:middle L:90% literally too many positive results in the literature. Um 152 00:07:18.430 --> 00:07:20.160 A:middle L:90% , if we think about how underpowered a lot of 153 00:07:20.160 --> 00:07:24.000 A:middle L:90% sciences are, and also just that everything we study 154 00:07:24.000 --> 00:07:26.029 A:middle L:90% is probably not real, we should have a higher 155 00:07:26.029 --> 00:07:29.750 A:middle L:90% proportion of non positive or no results in the literature 156 00:07:29.759 --> 00:07:31.910 A:middle L:90% and actuality. Um, it depends a little bit 157 00:07:31.910 --> 00:07:35.449 A:middle L:90% on the discipline, but it's usually around 90 95 158 00:07:35.839 --> 00:07:39.079 A:middle L:90% of, you know, published results are positive, 159 00:07:39.079 --> 00:07:44.279 A:middle L:90% which is just literally too high mathematically, um kind 160 00:07:44.279 --> 00:07:46.410 A:middle L:90% of going along with that, ignoring all results. 161 00:07:46.410 --> 00:07:48.329 A:middle L:90% As scientists, we generally, we pay a lot 162 00:07:48.329 --> 00:07:50.649 A:middle L:90% of attention to positive results. We don't really do 163 00:07:50.649 --> 00:07:53.490 A:middle L:90% anything with no results, right, They just kind 164 00:07:53.490 --> 00:07:56.269 A:middle L:90% of get pushed into a file drawer, forgotten, 165 00:07:56.269 --> 00:08:00.230 A:middle L:90% lonely and sad, unused. Um, another part 166 00:08:00.230 --> 00:08:01.459 A:middle L:90% is a lack of replication. So, because the 167 00:08:01.459 --> 00:08:05.230 A:middle L:90% scientists, we really are interested in these innovative results 168 00:08:05.240 --> 00:08:07.060 A:middle L:90% , usually somebody does a study and then nobody ever 169 00:08:07.060 --> 00:08:09.529 A:middle L:90% tries to replicate that study. And so it's hard 170 00:08:09.529 --> 00:08:13.769 A:middle L:90% to kind of accumulate knowledge about that specific study just 171 00:08:13.779 --> 00:08:16.959 A:middle L:90% because if nobody replicates it, you don't actually get 172 00:08:18.339 --> 00:08:20.959 A:middle L:90% accurate estimate of how replicable that effect is, and 173 00:08:20.959 --> 00:08:24.360 A:middle L:90% finally, the limits of null hypothesis testing. So 174 00:08:24.360 --> 00:08:26.699 A:middle L:90% null hypothesis testing is very good for some things, 175 00:08:26.709 --> 00:08:28.370 A:middle L:90% it has some problems in other areas, um but 176 00:08:28.370 --> 00:08:31.149 A:middle L:90% in general it's the only in most disciplines, it's 177 00:08:31.159 --> 00:08:33.830 A:middle L:90% really what is the only thing that is used for 178 00:08:33.830 --> 00:08:37.440 A:middle L:90% scientific investigation, which is creating some problems? Um 179 00:08:37.450 --> 00:08:41.190 A:middle L:90% So I'm going to go into kind of three specific 180 00:08:41.200 --> 00:08:45.259 A:middle L:90% areas and what researchers can do to help the problems 181 00:08:46.039 --> 00:08:48.379 A:middle L:90% to help deal with those problems in their own research 182 00:08:48.440 --> 00:08:52.419 A:middle L:90% , kind of really practical implementations of what researchers can 183 00:08:52.419 --> 00:08:54.860 A:middle L:90% do. Um But I did want to discuss a 184 00:08:54.860 --> 00:08:58.820 A:middle L:90% little bit about why people engage in these behaviors just 185 00:08:58.820 --> 00:09:00.669 A:middle L:90% because I think a lot of times when we start 186 00:09:00.669 --> 00:09:03.169 A:middle L:90% talking about lack of replica bility and all results and 187 00:09:03.169 --> 00:09:05.179 A:middle L:90% things like that, the first thing that people think 188 00:09:05.179 --> 00:09:07.320 A:middle L:90% about is fraud um and they think that, you 189 00:09:07.320 --> 00:09:11.169 A:middle L:90% know, there's nefarious behavior going on in any, 190 00:09:11.179 --> 00:09:13.370 A:middle L:90% and this can, you know, lead to a 191 00:09:13.370 --> 00:09:18.769 A:middle L:90% lot of people feeling very right. If people's results 192 00:09:18.769 --> 00:09:20.649 A:middle L:90% don't replicate, that is not necessarily an indication that 193 00:09:20.649 --> 00:09:24.629 A:middle L:90% they engaged in any fraudulent behavior, right? There 194 00:09:24.629 --> 00:09:26.159 A:middle L:90% is fraud that happens, but that is actually a 195 00:09:26.159 --> 00:09:28.750 A:middle L:90% very small proportion of these failures to replicate. Most 196 00:09:28.750 --> 00:09:31.620 A:middle L:90% of it is studies that were done in very good 197 00:09:31.620 --> 00:09:35.360 A:middle L:90% faith by by scientists who believe that they were doing 198 00:09:35.360 --> 00:09:37.210 A:middle L:90% things right and by the way their field was actually 199 00:09:37.220 --> 00:09:39.980 A:middle L:90% doing science, they were doing what their field considered 200 00:09:39.990 --> 00:09:43.909 A:middle L:90% right, but some things were going on behind the 201 00:09:43.909 --> 00:09:46.730 A:middle L:90% background that was kind of causing some unconscious motivation to 202 00:09:46.730 --> 00:09:48.990 A:middle L:90% seep into how the science was done and kind of 203 00:09:48.000 --> 00:09:52.009 A:middle L:90% alter that ideal scientific circle. Um so the first 204 00:09:52.009 --> 00:09:56.830 A:middle L:90% thing is perceived norms. Um I'm a social psychologist 205 00:09:56.830 --> 00:10:01.159 A:middle L:90% by trade. Um that was my PhD my background 206 00:10:01.169 --> 00:10:03.740 A:middle L:90% and such psychologists talk a lot about how people tend 207 00:10:03.740 --> 00:10:07.700 A:middle L:90% to engage in whatever they think the normative behavior is 208 00:10:07.710 --> 00:10:11.370 A:middle L:90% . So generally, if you see other scientists working 209 00:10:11.370 --> 00:10:13.549 A:middle L:90% in a particular fashion, you'll start to work in 210 00:10:13.549 --> 00:10:15.460 A:middle L:90% that fashion as well because you think that is what 211 00:10:15.460 --> 00:10:16.980 A:middle L:90% is normative lee done. Um so right now, 212 00:10:16.980 --> 00:10:20.820 A:middle L:90% most scientists, they kind of, they don't share 213 00:10:20.820 --> 00:10:22.379 A:middle L:90% their data, they don't really openly publish their data 214 00:10:22.379 --> 00:10:26.190 A:middle L:90% sets of material a lot of times they don't publicly 215 00:10:26.200 --> 00:10:28.470 A:middle L:90% post their code. And so as younger scientists, 216 00:10:28.470 --> 00:10:31.549 A:middle L:90% when we're trained up, we kind of see how 217 00:10:31.549 --> 00:10:33.220 A:middle L:90% it's done and go, okay, this is how 218 00:10:33.220 --> 00:10:33.789 A:middle L:90% it's being done, that is what I will do 219 00:10:33.789 --> 00:10:37.559 A:middle L:90% to and so these behaviors kind of continually go through 220 00:10:37.559 --> 00:10:41.320 A:middle L:90% the generations. Another thing is motivated reasoning. So 221 00:10:41.320 --> 00:10:43.720 A:middle L:90% what this means is that people have cognitive biases and 222 00:10:43.720 --> 00:10:48.279 A:middle L:90% we unconsciously think in ways that conform to what we 223 00:10:48.279 --> 00:10:50.629 A:middle L:90% would like to believe. Um, so this will 224 00:10:50.629 --> 00:10:52.750 A:middle L:90% come back in when we talk about those questionable research 225 00:10:52.750 --> 00:10:56.870 A:middle L:90% practices. But the idea is that if you are 226 00:10:56.879 --> 00:10:58.899 A:middle L:90% , you know, when you design a study, 227 00:10:58.899 --> 00:11:01.730 A:middle L:90% you probably have a particular hypothesis in mind, you 228 00:11:01.730 --> 00:11:03.610 A:middle L:90% are motivated to find that hypothesis. You're motivated to 229 00:11:03.610 --> 00:11:07.509 A:middle L:90% find that significant result. And because of that motivation 230 00:11:07.509 --> 00:11:09.549 A:middle L:90% , it can start causing you to unconsciously engage in 231 00:11:09.549 --> 00:11:13.200 A:middle L:90% behaviors and you don't realize you're doing it. But 232 00:11:13.210 --> 00:11:16.269 A:middle L:90% you know, after the fact you may go, 233 00:11:16.269 --> 00:11:16.590 A:middle L:90% okay, well why did I do that? And 234 00:11:16.590 --> 00:11:18.909 A:middle L:90% you can come up with a completely justifiable reason why 235 00:11:18.909 --> 00:11:22.340 A:middle L:90% you do that. But it was because you had 236 00:11:22.340 --> 00:11:24.009 A:middle L:90% that hypothesis in the first place, this will become 237 00:11:24.009 --> 00:11:26.929 A:middle L:90% a little bit more clear and a couple of slides 238 00:11:26.929 --> 00:11:28.419 A:middle L:90% when I give a concrete example. Um, so 239 00:11:28.419 --> 00:11:31.399 A:middle L:90% another thing is minimal accountability. When I'm, you 240 00:11:31.399 --> 00:11:33.139 A:middle L:90% know, working in a lab, I'm the only 241 00:11:33.139 --> 00:11:35.200 A:middle L:90% one who really kind of knows what's going on. 242 00:11:35.200 --> 00:11:39.159 A:middle L:90% Probably my lab colleagues also, no, but anybody 243 00:11:39.169 --> 00:11:41.159 A:middle L:90% buddy outside my lab, all they see is my 244 00:11:41.159 --> 00:11:43.820 A:middle L:90% research paper. They don't see the whole process. 245 00:11:43.830 --> 00:11:46.679 A:middle L:90% And so a lot of this motivated reasoning can happen 246 00:11:46.690 --> 00:11:50.360 A:middle L:90% because only the endpoint is known. Nobody knows where 247 00:11:50.360 --> 00:11:54.200 A:middle L:90% we started and nobody kind of knows how we got 248 00:11:54.210 --> 00:11:56.460 A:middle L:90% to the final product. And so there's not a 249 00:11:56.460 --> 00:12:00.279 A:middle L:90% lot of work flow documentation across the entire workflow. 250 00:12:00.289 --> 00:12:03.419 A:middle L:90% And so that can lead to minimal accountability. The 251 00:12:03.419 --> 00:12:05.379 A:middle L:90% last one or the second to last one is the 252 00:12:05.379 --> 00:12:07.009 A:middle L:90% incentive structures. Right? In terms of how 10 253 00:12:07.009 --> 00:12:09.860 A:middle L:90% years given out, in terms of how people get 254 00:12:09.860 --> 00:12:13.509 A:middle L:90% jobs. It's all about how many papers do you 255 00:12:13.509 --> 00:12:15.789 A:middle L:90% have published and what is the impact of the journals 256 00:12:15.789 --> 00:12:16.960 A:middle L:90% you have them published in? And so because of 257 00:12:16.960 --> 00:12:20.480 A:middle L:90% this incentive structure, it really motivates people to look 258 00:12:20.480 --> 00:12:22.970 A:middle L:90% for those significant results, which can set off a 259 00:12:22.970 --> 00:12:26.379 A:middle L:90% lot of this motivated reasoning. So it kind of 260 00:12:26.379 --> 00:12:28.059 A:middle L:90% feeds back into the top and goes down. And 261 00:12:28.059 --> 00:12:31.389 A:middle L:90% the final one is, I'm busy, right? 262 00:12:31.399 --> 00:12:33.679 A:middle L:90% All scientists are very busy people, we have a 263 00:12:33.679 --> 00:12:35.970 A:middle L:90% lot to do. And so any time you talk 264 00:12:35.970 --> 00:12:39.720 A:middle L:90% about maybe changing aspects of the scientific workflow, the 265 00:12:39.720 --> 00:12:41.950 A:middle L:90% first thing that pops into everybody's mind and I completely 266 00:12:41.950 --> 00:12:43.990 A:middle L:90% get this is, oh God, that sounds like 267 00:12:43.990 --> 00:12:46.149 A:middle L:90% more work, which, you know, nobody wants 268 00:12:46.149 --> 00:12:46.679 A:middle L:90% to do more work, nobody has time to do 269 00:12:46.679 --> 00:12:50.259 A:middle L:90% more work. Um but what I would like to 270 00:12:50.259 --> 00:12:52.690 A:middle L:90% argue is that a lot of these changes are small 271 00:12:52.700 --> 00:12:56.840 A:middle L:90% . They're not necessarily changing a great amount of work 272 00:12:56.840 --> 00:12:58.100 A:middle L:90% , They're just moving work around and that they can 273 00:12:58.100 --> 00:13:03.080 A:middle L:90% have really nice consequences for kind of the longevity of 274 00:13:03.080 --> 00:13:05.899 A:middle L:90% individual scientists work and the longevity of research lines as 275 00:13:05.899 --> 00:13:07.299 A:middle L:90% a whole. And so that short to long term 276 00:13:07.299 --> 00:13:11.799 A:middle L:90% trade off is potentially worth wild. So what can 277 00:13:11.799 --> 00:13:15.419 A:middle L:90% fix a lot of these problems? Many people have 278 00:13:15.419 --> 00:13:16.570 A:middle L:90% been talking about many different things. I'm going to 279 00:13:16.580 --> 00:13:20.659 A:middle L:90% specifically talk about three things in a very kind of 280 00:13:20.669 --> 00:13:22.710 A:middle L:90% concrete way in terms of how they can be implemented 281 00:13:22.720 --> 00:13:26.490 A:middle L:90% . The first is pre registering studies followed by increased 282 00:13:26.490 --> 00:13:30.759 A:middle L:90% documentation of the entire workflow, and the final thing 283 00:13:31.340 --> 00:13:33.059 A:middle L:90% is changing norms and incentive structures. So the first 284 00:13:33.059 --> 00:13:35.980 A:middle L:90% two can very much be done by an individual researcher 285 00:13:35.980 --> 00:13:39.470 A:middle L:90% or an individual lab. They're not really contingent on 286 00:13:39.470 --> 00:13:41.590 A:middle L:90% anyone else. The last one is kind of contingent 287 00:13:41.590 --> 00:13:45.360 A:middle L:90% on the greater scientific community as a whole. Um 288 00:13:45.360 --> 00:13:46.899 A:middle L:90% but I think it's particularly important and things have begun 289 00:13:46.899 --> 00:13:48.809 A:middle L:90% to shift in that area. Um And so I 290 00:13:48.809 --> 00:13:50.740 A:middle L:90% did want to talk about it just a bit at 291 00:13:50.750 --> 00:13:54.169 A:middle L:90% the end. Um but kind of, the overarching 292 00:13:54.179 --> 00:13:58.289 A:middle L:90% theme of all of this is increasing openness and transparency 293 00:13:58.289 --> 00:14:01.000 A:middle L:90% of research, by engaging in more open and transparent 294 00:14:01.000 --> 00:14:05.879 A:middle L:90% research practices that can actually fix a lot of the 295 00:14:05.879 --> 00:14:07.809 A:middle L:90% problems I was talking about on the other slides and 296 00:14:07.809 --> 00:14:11.200 A:middle L:90% make the published literature a little bit more trustworthy, 297 00:14:11.200 --> 00:14:13.100 A:middle L:90% a little bit more reproducible and a little bit more 298 00:14:13.100 --> 00:14:16.600 A:middle L:90% valid. So, the first thing I'm going to 299 00:14:16.600 --> 00:14:18.230 A:middle L:90% talk about is study preregistration. Um So, how 300 00:14:18.230 --> 00:14:20.250 A:middle L:90% many of you have heard of something called clinical trials 301 00:14:20.250 --> 00:14:24.200 A:middle L:90% dot gov. Okay, so a couple of people 302 00:14:24.200 --> 00:14:26.759 A:middle L:90% in the room. Um so clinical trials dot gov 303 00:14:26.769 --> 00:14:28.899 A:middle L:90% is a obviously it's a website run by the government 304 00:14:28.909 --> 00:14:35.129 A:middle L:90% um and all phase three clinical trials that receive funding 305 00:14:35.129 --> 00:14:37.720 A:middle L:90% from NIH and NSF before they start have to go 306 00:14:37.720 --> 00:14:41.240 A:middle L:90% to this website and say exactly what they're gonna do 307 00:14:41.250 --> 00:14:43.860 A:middle L:90% . So they say their hypotheses, they say they're 308 00:14:43.870 --> 00:14:46.730 A:middle L:90% what analyses there are going to run. They say 309 00:14:46.730 --> 00:14:48.799 A:middle L:90% what they're dependent variables are they basically right up the 310 00:14:48.809 --> 00:14:54.700 A:middle L:90% intro methods and analysis portion of a research paper and 311 00:14:54.710 --> 00:14:58.080 A:middle L:90% uploaded to the site. So clinical trials have studied 312 00:14:58.080 --> 00:15:01.429 A:middle L:90% preregistration. That is what study preregistration is, which 313 00:15:01.429 --> 00:15:05.590 A:middle L:90% is before a study is run ideally before the study 314 00:15:05.590 --> 00:15:07.330 A:middle L:90% is run. If you work in an area where 315 00:15:07.330 --> 00:15:11.269 A:middle L:90% you use large data sets that are already existent, 316 00:15:11.279 --> 00:15:13.629 A:middle L:90% um it would be before you looked at the data 317 00:15:13.629 --> 00:15:16.039 A:middle L:90% and started doing data analysis, you specify your research 318 00:15:16.039 --> 00:15:20.970 A:middle L:90% question your hypotheses, study design materials and data and 319 00:15:20.980 --> 00:15:24.570 A:middle L:90% data analysis plan. And so what this does, 320 00:15:24.570 --> 00:15:26.879 A:middle L:90% this does a couple of different things. So the 321 00:15:26.879 --> 00:15:30.860 A:middle L:90% first is a decrease of something called harking which is 322 00:15:31.340 --> 00:15:35.559 A:middle L:90% little term that stands for hypothesizing after results are known 323 00:15:35.639 --> 00:15:37.970 A:middle L:90% . Yeah, the last three letters should have been 324 00:15:37.970 --> 00:15:41.019 A:middle L:90% lower case. They don't stand for anything. And 325 00:15:41.019 --> 00:15:43.230 A:middle L:90% so basically what this means is because you've specified what 326 00:15:43.230 --> 00:15:46.610 A:middle L:90% your hypothesis is up front. If let's say your 327 00:15:46.610 --> 00:15:48.740 A:middle L:90% results turn out to be different than you expected, 328 00:15:48.750 --> 00:15:52.370 A:middle L:90% you can't then after the fact go back and change 329 00:15:52.370 --> 00:15:54.899 A:middle L:90% what your hypothesis was. That's what harking is. 330 00:15:54.909 --> 00:15:58.080 A:middle L:90% It can be problematic because what it does is it 331 00:15:58.080 --> 00:16:03.090 A:middle L:90% presents hypotheses data analytic choices as a priori when really 332 00:16:03.090 --> 00:16:04.100 A:middle L:90% they were post talk when they were made after you 333 00:16:04.100 --> 00:16:07.799 A:middle L:90% looked at your data. So the second thing it 334 00:16:07.799 --> 00:16:11.240 A:middle L:90% does is it decreases what are known as researcher degrees 335 00:16:11.240 --> 00:16:12.580 A:middle L:90% of freedom gonna go into more detail about this on 336 00:16:12.580 --> 00:16:15.789 A:middle L:90% the next slide. But this is another name for 337 00:16:15.789 --> 00:16:19.110 A:middle L:90% those questionable research practices and the reason it decreases these 338 00:16:19.120 --> 00:16:22.590 A:middle L:90% is because it locks you in a priority to a 339 00:16:22.590 --> 00:16:26.120 A:middle L:90% specific analysis plan. And the last thing it kind 340 00:16:26.120 --> 00:16:29.110 A:middle L:90% of does is it decreases that file drawer problem. 341 00:16:29.120 --> 00:16:30.730 A:middle L:90% So if most all results are kind of put in 342 00:16:30.730 --> 00:16:33.850 A:middle L:90% a drawer and never published. We as the scientific 343 00:16:33.850 --> 00:16:37.659 A:middle L:90% community don't really know they existed because of pre registration 344 00:16:37.659 --> 00:16:40.909 A:middle L:90% . If that preregistration is public like it is on 345 00:16:40.909 --> 00:16:42.669 A:middle L:90% clinical trials dot gov we have a record of every 346 00:16:42.669 --> 00:16:45.830 A:middle L:90% single study that was run, we may not know 347 00:16:45.830 --> 00:16:47.500 A:middle L:90% the results of it, but we know that hey 348 00:16:47.500 --> 00:16:48.600 A:middle L:90% , it was done and it wasn't published. So 349 00:16:48.600 --> 00:16:51.860 A:middle L:90% we can go and seek out those researchers and say 350 00:16:51.860 --> 00:16:52.559 A:middle L:90% , hey, what did your study look like? 351 00:16:53.039 --> 00:16:56.840 A:middle L:90% And so we are better able to find unpublished studies 352 00:16:56.840 --> 00:17:00.570 A:middle L:90% and better able to accumulate knowledge using those studies. 353 00:17:00.440 --> 00:17:03.309 A:middle L:90% And so I did want to talk in more detail 354 00:17:03.309 --> 00:17:04.789 A:middle L:90% about researcher degrees of freedom because they are kind of 355 00:17:04.789 --> 00:17:08.859 A:middle L:90% the particular crux of why pre analysis are. Pre 356 00:17:10.339 --> 00:17:14.009 A:middle L:90% pre registration is important. So research degrees of freedom 357 00:17:14.019 --> 00:17:17.690 A:middle L:90% are all data processing and analytical choices that are made 358 00:17:17.700 --> 00:17:19.940 A:middle L:90% after seeing and interacting with your data. So they're 359 00:17:19.940 --> 00:17:23.509 A:middle L:90% data dependent choices. And so to give you some 360 00:17:23.509 --> 00:17:26.900 A:middle L:90% examples of what counts as a researcher degree of freedom 361 00:17:26.980 --> 00:17:30.339 A:middle L:90% . These are things like should more data be collected 362 00:17:30.349 --> 00:17:33.240 A:middle L:90% , Should some observations be excluded? What conditions should 363 00:17:33.240 --> 00:17:34.849 A:middle L:90% be compared? Which variables should I control for? 364 00:17:34.940 --> 00:17:37.490 A:middle L:90% Which variables should I use as my main devi what 365 00:17:37.490 --> 00:17:41.289 A:middle L:90% is the statistical effective interest to test my hypothesis? 366 00:17:41.299 --> 00:17:42.089 A:middle L:90% Now, what you'll notice is that these are all 367 00:17:42.089 --> 00:17:45.000 A:middle L:90% very important questions, right? You cannot do a 368 00:17:45.000 --> 00:17:48.289 A:middle L:90% research study without asking and answering these questions. So 369 00:17:48.289 --> 00:17:51.230 A:middle L:90% the issue is not the question. The issue is 370 00:17:51.230 --> 00:17:53.960 A:middle L:90% when these questions are asked and answered by the researcher 371 00:17:55.740 --> 00:17:57.230 A:middle L:90% . If you think about these questions and come up 372 00:17:57.230 --> 00:18:00.049 A:middle L:90% with an answer before you started, you know, 373 00:18:00.049 --> 00:18:03.920 A:middle L:90% analyzing your data, everything is good, everything is 374 00:18:03.920 --> 00:18:07.710 A:middle L:90% fine. However, if you start answering these questions 375 00:18:07.720 --> 00:18:10.049 A:middle L:90% after you've looked at your data, all of a 376 00:18:10.049 --> 00:18:14.150 A:middle L:90% sudden the answers to these questions become data dependent. 377 00:18:14.259 --> 00:18:17.079 A:middle L:90% The way your data is turning out is going to 378 00:18:17.079 --> 00:18:21.829 A:middle L:90% probably unconsciously change how you answer these questions and that's 379 00:18:21.829 --> 00:18:25.430 A:middle L:90% going to create some problems specifically what it's going to 380 00:18:25.430 --> 00:18:30.710 A:middle L:90% do. It's really going to inflate your false positive 381 00:18:30.710 --> 00:18:33.259 A:middle L:90% rate and your p values aren't going to be informative 382 00:18:33.259 --> 00:18:36.529 A:middle L:90% anymore. So basically when you start making data dependent 383 00:18:36.529 --> 00:18:38.890 A:middle L:90% choices, the number of false positives are getting. 384 00:18:38.890 --> 00:18:41.710 A:middle L:90% So the number of times you're going to show a 385 00:18:41.710 --> 00:18:44.750 A:middle L:90% significant effect when that effect is not really there in 386 00:18:44.750 --> 00:18:48.950 A:middle L:90% the population is going to start going up Now. 387 00:18:48.960 --> 00:18:52.579 A:middle L:90% This can go up to anywhere around 61 false positive 388 00:18:52.579 --> 00:18:55.000 A:middle L:90% rate. Um so if you're from a field where 389 00:18:55.009 --> 00:18:56.950 A:middle L:90% you usually set your false positive rate, your p 390 00:18:56.950 --> 00:19:03.180 A:middle L:90% value 2.055 61 is really high. Right? That's 391 00:19:03.180 --> 00:19:06.630 A:middle L:90% better. I mean Flipping a coin would actually give 392 00:19:06.630 --> 00:19:10.819 A:middle L:90% you lower odds than 61%. And so because that 393 00:19:10.819 --> 00:19:12.900 A:middle L:90% false positive rate is so high, the p values 394 00:19:12.900 --> 00:19:17.569 A:middle L:90% you're getting from your data analysis aren't really informative anymore 395 00:19:17.569 --> 00:19:19.240 A:middle L:90% . They're not telling you something real about what's going 396 00:19:19.240 --> 00:19:22.380 A:middle L:90% on your data and kind of to give you a 397 00:19:22.380 --> 00:19:26.200 A:middle L:90% concrete example of what I mean by these data dependent 398 00:19:26.200 --> 00:19:27.359 A:middle L:90% choices. Let's go back to the example of, 399 00:19:27.640 --> 00:19:30.480 A:middle L:90% you know, looking at the difference in height between 400 00:19:30.480 --> 00:19:32.640 A:middle L:90% men and women. So let's say I have my 401 00:19:32.640 --> 00:19:34.000 A:middle L:90% study and I initially decided I'm going to have two 402 00:19:34.000 --> 00:19:37.900 A:middle L:90% dependent variables. I'm going to have people self report 403 00:19:37.900 --> 00:19:38.630 A:middle L:90% what their heart is. And I'm also going to 404 00:19:38.630 --> 00:19:41.619 A:middle L:90% get an R. A. To actually measure them 405 00:19:41.619 --> 00:19:45.059 A:middle L:90% with a tape measure. So I run my study 406 00:19:45.640 --> 00:19:48.329 A:middle L:90% Get my two dependent variables and one of them is 407 00:19:48.329 --> 00:19:52.559 A:middle L:90% significant and one of them is not. So the 408 00:19:52.559 --> 00:19:56.230 A:middle L:90% self reported height was significant. The tape measure height 409 00:19:56.240 --> 00:19:57.670 A:middle L:90% was not Now. I'm sitting in my lab meeting 410 00:19:57.670 --> 00:20:02.140 A:middle L:90% with um my other collaborators and we're kind of talking 411 00:20:02.140 --> 00:20:03.680 A:middle L:90% about the results, talking about how we're going to 412 00:20:03.680 --> 00:20:04.799 A:middle L:90% write it up and somebody says, you know, 413 00:20:04.809 --> 00:20:07.400 A:middle L:90% I don't think that tape measure one is really, 414 00:20:07.410 --> 00:20:10.619 A:middle L:90% you know, all that ballad of a dependent variable 415 00:20:10.619 --> 00:20:12.180 A:middle L:90% . Like yeah, it seems face valid but you 416 00:20:12.180 --> 00:20:14.980 A:middle L:90% know a lot of your race or women they're kind 417 00:20:14.980 --> 00:20:17.359 A:middle L:90% of short. If they're trying to measure how tall 418 00:20:17.369 --> 00:20:18.589 A:middle L:90% a really tall guy is. Right? They probably 419 00:20:18.589 --> 00:20:21.000 A:middle L:90% won't be able to see the top. The tape 420 00:20:21.000 --> 00:20:22.170 A:middle L:90% measure very well. It's gonna be like tilted at 421 00:20:22.170 --> 00:20:25.390 A:middle L:90% an angle. They're gonna have a lot of trouble 422 00:20:25.400 --> 00:20:26.680 A:middle L:90% accurately getting a read on that. So I don't 423 00:20:26.680 --> 00:20:30.380 A:middle L:90% think that dependent variables particularly valid. You should just 424 00:20:30.380 --> 00:20:33.349 A:middle L:90% throw it out and go with the more dependent one 425 00:20:33.539 --> 00:20:34.980 A:middle L:90% . This can happen all the time. Right? 426 00:20:34.980 --> 00:20:40.450 A:middle L:90% That doesn't seem like it's particularly focus logic or like 427 00:20:40.450 --> 00:20:41.809 A:middle L:90% you're trying to make your study seem nicer. Sometimes 428 00:20:41.809 --> 00:20:45.420 A:middle L:90% reviewers will actually ask you to do this and throw 429 00:20:45.420 --> 00:20:45.920 A:middle L:90% out a D. V. Right? This is 430 00:20:45.920 --> 00:20:51.410 A:middle L:90% just considered kind of absolutely fine behavior. The problem 431 00:20:51.410 --> 00:20:53.619 A:middle L:90% is that if it turned out differently so the tape 432 00:20:53.619 --> 00:20:56.859 A:middle L:90% measure was significant and the self reported height wasn't, 433 00:20:57.519 --> 00:21:02.200 A:middle L:90% the researcher would probably come up with just as logical 434 00:21:02.200 --> 00:21:04.880 A:middle L:90% sounding of a explanation of why they should throw out 435 00:21:04.890 --> 00:21:07.079 A:middle L:90% the other D. V. So you can think 436 00:21:07.089 --> 00:21:11.759 A:middle L:90% well you know self reported height. There are stereotypes 437 00:21:11.759 --> 00:21:12.930 A:middle L:90% about how tall men and women are supposed to be 438 00:21:12.930 --> 00:21:17.420 A:middle L:90% . And you know maybe some people are saying how 439 00:21:17.420 --> 00:21:18.380 A:middle L:90% tall they'd like to be rather than hotel. They 440 00:21:18.380 --> 00:21:21.990 A:middle L:90% are right to fudging the numbers. The tape measure 441 00:21:21.990 --> 00:21:23.960 A:middle L:90% is far more face valid right? They can't it's 442 00:21:23.960 --> 00:21:26.769 A:middle L:90% how tall they really are. They can't do any 443 00:21:26.769 --> 00:21:27.980 A:middle L:90% fudging so all throughout the tape measure or want to 444 00:21:27.980 --> 00:21:30.680 A:middle L:90% keep the self reported one right. The problem with 445 00:21:30.680 --> 00:21:33.390 A:middle L:90% that is it was completely dependent on how your data 446 00:21:33.390 --> 00:21:37.009 A:middle L:90% turned out. You didn't decide this a priori. 447 00:21:37.019 --> 00:21:40.059 A:middle L:90% And so that's what's causing those false positive to spike 448 00:21:41.140 --> 00:21:41.549 A:middle L:90% . And so at the bottom right. These often 449 00:21:41.549 --> 00:21:45.529 A:middle L:90% feel very reasonable in the moment Because of that motivated 450 00:21:45.529 --> 00:21:48.319 A:middle L:90% reasoning we talked about earlier, but they're still going 451 00:21:48.319 --> 00:21:52.039 A:middle L:90% to increase this false positive rates, sometimes as high 452 00:21:52.039 --> 00:21:55.400 A:middle L:90% as 61%, which is a real problem. And 453 00:21:55.400 --> 00:21:57.589 A:middle L:90% so one way to tamp down those false positive rates 454 00:21:57.589 --> 00:22:00.170 A:middle L:90% and keep those p values as face valid as possible 455 00:22:00.180 --> 00:22:03.970 A:middle L:90% is something called a pre analysis plan. And so 456 00:22:03.970 --> 00:22:06.500 A:middle L:90% what a pre analysis plan is doing is asking the 457 00:22:06.500 --> 00:22:11.170 A:middle L:90% researcher to specify as specifically as possible all the analyses 458 00:22:11.180 --> 00:22:12.920 A:middle L:90% they're going to do in their study, it generally 459 00:22:12.920 --> 00:22:17.259 A:middle L:90% includes four parts the first talking about the target sample 460 00:22:17.269 --> 00:22:18.450 A:middle L:90% . So specifying the size of the sample, you're 461 00:22:18.450 --> 00:22:21.309 A:middle L:90% going to have what you're stopping rule is. So 462 00:22:21.309 --> 00:22:22.579 A:middle L:90% how many people, at one point you're gonna stop 463 00:22:22.579 --> 00:22:25.240 A:middle L:90% your study. How many people are you going to 464 00:22:25.240 --> 00:22:26.960 A:middle L:90% have? Who are you going to have? And 465 00:22:26.960 --> 00:22:29.839 A:middle L:90% who are you going to sample? The second part 466 00:22:29.849 --> 00:22:33.240 A:middle L:90% is about data cleaning and processing. Um So if 467 00:22:33.240 --> 00:22:36.140 A:middle L:90% you work in an area like FmRI research or er 468 00:22:36.140 --> 00:22:38.150 A:middle L:90% p research or reaction time research, there's a lot 469 00:22:38.150 --> 00:22:41.789 A:middle L:90% of cleaning that happens to that data before the analysis 470 00:22:41.789 --> 00:22:45.460 A:middle L:90% ever starts. There's a lot of transformations that happen 471 00:22:45.609 --> 00:22:47.759 A:middle L:90% . Certain trials may be thrown out for things like 472 00:22:47.759 --> 00:22:52.750 A:middle L:90% head movement. Um Now and different researchers are going 473 00:22:52.750 --> 00:22:53.400 A:middle L:90% to there are a lot of different ways to do 474 00:22:53.400 --> 00:22:57.240 A:middle L:90% that processing. Um there's a paper by uh carp 475 00:22:57.250 --> 00:23:00.059 A:middle L:90% and I think it's 2012 that looked at kind of 476 00:23:00.069 --> 00:23:03.859 A:middle L:90% all the different possible processing choices you can make on 477 00:23:03.859 --> 00:23:07.960 A:middle L:90% FmRI data from the published literature and you came up 478 00:23:07.960 --> 00:23:12.039 A:middle L:90% with something like 30,000 different potential ways to process FmRI 479 00:23:12.039 --> 00:23:15.230 A:middle L:90% data all in the published literature. So there's a 480 00:23:15.230 --> 00:23:17.809 A:middle L:90% huge number of ways to do this in ways that 481 00:23:17.809 --> 00:23:19.440 A:middle L:90% researchers could choose from. And so it's important to 482 00:23:19.440 --> 00:23:22.230 A:middle L:90% say which ones you're going to do, The next 483 00:23:22.230 --> 00:23:25.890 A:middle L:90% one is exclusion criteria. So who are you excluding 484 00:23:25.890 --> 00:23:26.970 A:middle L:90% ? And why are you going to exclude people who 485 00:23:26.970 --> 00:23:30.269 A:middle L:90% failed manipulation checks? Are you going to exclude people 486 00:23:30.269 --> 00:23:32.819 A:middle L:90% who fell asleep during your studies? This happened to 487 00:23:32.819 --> 00:23:33.589 A:middle L:90% me in ground school more than I would like to 488 00:23:33.589 --> 00:23:36.910 A:middle L:90% say. Also drunk people in my studies, they 489 00:23:36.910 --> 00:23:38.750 A:middle L:90% got excluded to. Um but right, going through 490 00:23:38.750 --> 00:23:41.670 A:middle L:90% all the reasons potentially why you might exclude people. 491 00:23:41.839 --> 00:23:44.930 A:middle L:90% And this is also talking about how you're going to 492 00:23:44.930 --> 00:23:48.430 A:middle L:90% identify outliers. And the last thing is specify all 493 00:23:48.430 --> 00:23:52.750 A:middle L:90% the actual analyses you're gonna conduct. So what statistical 494 00:23:52.750 --> 00:23:55.640 A:middle L:90% tests are you going to conduct? Um you know 495 00:23:55.640 --> 00:23:56.740 A:middle L:90% , what variables are you going to use? What 496 00:23:56.750 --> 00:24:00.000 A:middle L:90% TVs are you going to use? What controls are 497 00:24:00.000 --> 00:24:02.650 A:middle L:90% you going to use? And so the idea kind 498 00:24:02.650 --> 00:24:06.650 A:middle L:90% of the the idea is to specify this as much 499 00:24:06.650 --> 00:24:08.230 A:middle L:90% as humanly possible. There's a range right that you 500 00:24:08.230 --> 00:24:11.309 A:middle L:90% could do. Some people write it out and kind 501 00:24:11.309 --> 00:24:12.849 A:middle L:90% of plain english. Some people go as far as 502 00:24:12.849 --> 00:24:15.490 A:middle L:90% to actually write up the entire analysis scripts so when 503 00:24:15.490 --> 00:24:18.269 A:middle L:90% their data comes in, all they have to do 504 00:24:18.269 --> 00:24:19.730 A:middle L:90% is highlight the script, click run and magic, 505 00:24:19.730 --> 00:24:23.049 A:middle L:90% all their data analysis is done. Um Generally when 506 00:24:23.049 --> 00:24:26.130 A:middle L:90% people start doing these, they start at the one 507 00:24:26.130 --> 00:24:27.440 A:middle L:90% end where they're just describing and thinking about these things 508 00:24:27.440 --> 00:24:30.480 A:middle L:90% and they slowly start getting more and more specific um 509 00:24:30.490 --> 00:24:34.059 A:middle L:90% but kind of making as many decisions as possible. 510 00:24:34.539 --> 00:24:38.069 A:middle L:90% Takes away those opportunities for researcher degrees of freedom. 511 00:24:38.079 --> 00:24:41.859 A:middle L:90% And so it really keeps those p values in check 512 00:24:42.940 --> 00:24:45.519 A:middle L:90% . And so because of that, right, it 513 00:24:45.519 --> 00:24:48.529 A:middle L:90% decreases those researches of freedom And because you've written all 514 00:24:48.529 --> 00:24:51.099 A:middle L:90% this out in advance and it's not just like sitting 515 00:24:51.099 --> 00:24:52.609 A:middle L:90% around in your head, it holds you accountable to 516 00:24:52.609 --> 00:24:55.210 A:middle L:90% what you said you were going to do and it 517 00:24:55.210 --> 00:24:56.140 A:middle L:90% holds you accountable to others to what you said you 518 00:24:56.140 --> 00:24:59.230 A:middle L:90% were going to do. So you can actually look 519 00:24:59.230 --> 00:25:00.319 A:middle L:90% at the analysis you did and go back and look 520 00:25:00.319 --> 00:25:02.619 A:middle L:90% at what you said you were going to do and 521 00:25:02.619 --> 00:25:03.700 A:middle L:90% see, okay, how are these different, why 522 00:25:03.700 --> 00:25:07.490 A:middle L:90% are they different? You know, were the choices 523 00:25:07.490 --> 00:25:10.140 A:middle L:90% I made the different things that I did legitimate or 524 00:25:10.140 --> 00:25:11.170 A:middle L:90% not? And other people can make those same. 525 00:25:11.539 --> 00:25:14.549 A:middle L:90% You can ask the same questions and potentially come to 526 00:25:14.549 --> 00:25:17.730 A:middle L:90% different conclusions. Now I talked about clinical trials dot 527 00:25:17.730 --> 00:25:19.019 A:middle L:90% gov That's a public registry, right? Anybody can 528 00:25:19.019 --> 00:25:21.609 A:middle L:90% go on there and see what you said you were 529 00:25:21.609 --> 00:25:23.390 A:middle L:90% going to do. Um You can imagine that you 530 00:25:23.390 --> 00:25:26.460 A:middle L:90% could create a private registry, You could create a 531 00:25:26.460 --> 00:25:30.259 A:middle L:90% data analysis plan, completely private. Um but making 532 00:25:30.259 --> 00:25:33.940 A:middle L:90% it public has two main advantages. Right? It 533 00:25:33.940 --> 00:25:37.369 A:middle L:90% helps with replication attempts. Just because you said exactly 534 00:25:37.369 --> 00:25:38.930 A:middle L:90% how you're going to do the study and exactly you 535 00:25:38.930 --> 00:25:41.359 A:middle L:90% know what your analysis scripts are gonna be. So 536 00:25:41.359 --> 00:25:44.440 A:middle L:90% somebody can then go and use that information when they 537 00:25:44.440 --> 00:25:47.369 A:middle L:90% try and replicate your findings. And it also decreases 538 00:25:47.369 --> 00:25:48.609 A:middle L:90% that file drawer effect, which we talked about before 539 00:25:48.619 --> 00:25:51.990 A:middle L:90% . Other people know what studies you plan to do 540 00:25:51.990 --> 00:25:52.690 A:middle L:90% . So we have a better estimate of what has 541 00:25:52.690 --> 00:25:56.990 A:middle L:90% actually been done now. You might be wondering. 542 00:25:56.990 --> 00:25:59.829 A:middle L:90% Okay, so you've just told me that I should 543 00:25:59.829 --> 00:26:02.250 A:middle L:90% specify all my analyses before. I do them. 544 00:26:02.259 --> 00:26:04.109 A:middle L:90% What about exploratory testing? Does this mean I'm not 545 00:26:04.109 --> 00:26:08.000 A:middle L:90% allowed to do exploratory testing? No, completely not 546 00:26:08.009 --> 00:26:11.240 A:middle L:90% . Exploratory analysis is completely fine and it's fantastic. 547 00:26:11.240 --> 00:26:14.730 A:middle L:90% And I love exploratory analysis. Um one of my 548 00:26:14.730 --> 00:26:15.740 A:middle L:90% teachers in grad school so that if he had to 549 00:26:15.740 --> 00:26:18.609 A:middle L:90% do one type of analysis for the rest of his 550 00:26:18.609 --> 00:26:19.869 A:middle L:90% life, it would be exploratory analysis. Right? 551 00:26:19.880 --> 00:26:25.309 A:middle L:90% Exploratory analysis is incredibly important to scientific discovery. It 552 00:26:25.309 --> 00:26:30.049 A:middle L:90% really is about generating new findings. The problem is 553 00:26:30.059 --> 00:26:33.170 A:middle L:90% the confirmatory and exploratory analyses have different purposes. And 554 00:26:33.170 --> 00:26:37.829 A:middle L:90% when you start dressing up exploratory analysis as confirmatory, 555 00:26:37.839 --> 00:26:40.829 A:middle L:90% that's when science starts to get into a little bit 556 00:26:40.829 --> 00:26:42.960 A:middle L:90% of trouble. So as I mentioned, exploratory analyses 557 00:26:42.960 --> 00:26:48.460 A:middle L:90% is really for generating new hypothesis. Looking into what 558 00:26:48.470 --> 00:26:52.430 A:middle L:90% might potentially be out there to look into in another 559 00:26:52.430 --> 00:26:55.299 A:middle L:90% study to confirm it. A confirmatory analysis is when 560 00:26:55.299 --> 00:26:56.750 A:middle L:90% you really want to say, okay, I have 561 00:26:56.750 --> 00:27:00.089 A:middle L:90% an idea about what's going on here. I actually 562 00:27:00.089 --> 00:27:02.660 A:middle L:90% want to test if that's really what's going on here 563 00:27:02.670 --> 00:27:06.539 A:middle L:90% . So confirmatory analysis is about testing. Exploratory analysis 564 00:27:06.549 --> 00:27:11.339 A:middle L:90% is about exploring discovery, right? Just investigating what 565 00:27:11.339 --> 00:27:15.609 A:middle L:90% might be out there for the further confirmatory testing in 566 00:27:15.609 --> 00:27:18.880 A:middle L:90% the future. And so clearly delineating between the two 567 00:27:18.890 --> 00:27:22.619 A:middle L:90% is what the point of a pre analysis plant is 568 00:27:22.630 --> 00:27:25.099 A:middle L:90% . The pre analysis plan says okay these are all 569 00:27:25.099 --> 00:27:27.299 A:middle L:90% of my confirmatory tests. You can then go and 570 00:27:27.299 --> 00:27:30.329 A:middle L:90% do other tests as well. But because you preregistered 571 00:27:30.329 --> 00:27:33.099 A:middle L:90% the confirmatory ones it will be very clear to you 572 00:27:33.109 --> 00:27:37.549 A:middle L:90% , to the editors and to the readers what the 573 00:27:37.549 --> 00:27:40.950 A:middle L:90% exploratory analyses were and what the confirmatory analyses were. 574 00:27:40.960 --> 00:27:42.759 A:middle L:90% So that when people are interpreting your results they can 575 00:27:42.759 --> 00:27:49.470 A:middle L:90% take that into account figures. Sure. So let's 576 00:27:49.470 --> 00:27:53.210 A:middle L:90% say I did my pre analysis plan and I said 577 00:27:53.210 --> 00:27:57.170 A:middle L:90% you know I'm going to look for height differences between 578 00:27:57.170 --> 00:28:02.440 A:middle L:90% men and women using a T. Test because that's 579 00:28:02.440 --> 00:28:04.269 A:middle L:90% the simplest thing T. Test and my dependent variable 580 00:28:04.940 --> 00:28:08.109 A:middle L:90% is going to be self reported height. So you 581 00:28:08.109 --> 00:28:14.029 A:middle L:90% do that you find a null result. Um But 582 00:28:14.029 --> 00:28:18.099 A:middle L:90% you would also because of how you know university works 583 00:28:18.099 --> 00:28:19.710 A:middle L:90% you would also had a bunch of questionnaire measures that 584 00:28:19.720 --> 00:28:26.009 A:middle L:90% they also had just happened to have responded to earlier 585 00:28:26.009 --> 00:28:27.470 A:middle L:90% in this semester. So you can start thinking okay 586 00:28:27.470 --> 00:28:30.910 A:middle L:90% I found a no result. But you know maybe 587 00:28:30.920 --> 00:28:40.640 A:middle L:90% some people are more uh how the stronger eager how 588 00:28:40.640 --> 00:28:44.259 A:middle L:90% they you know oh here's a better example. Um 589 00:28:44.259 --> 00:28:45.670 A:middle L:90% So let's say you have a combination of male and 590 00:28:45.670 --> 00:28:48.460 A:middle L:90% female raise and your confirmatory plan. You have not 591 00:28:48.460 --> 00:28:51.230 A:middle L:90% taken that into account. You get the snow effect 592 00:28:51.230 --> 00:28:55.250 A:middle L:90% and you kind of start thinking maybe the gender of 593 00:28:55.250 --> 00:28:57.150 A:middle L:90% the R. A. Was changing how people self 594 00:28:57.150 --> 00:29:00.920 A:middle L:90% reported their height, right? If a girl had 595 00:29:00.930 --> 00:29:03.180 A:middle L:90% a male ari she might be more likely to decrease 596 00:29:03.180 --> 00:29:07.279 A:middle L:90% her height or under report her height because she thinks 597 00:29:07.279 --> 00:29:08.950 A:middle L:90% that will make her seem more attractive. Um Whereas 598 00:29:08.950 --> 00:29:11.710 A:middle L:90% if she had another female ari she might be less 599 00:29:11.710 --> 00:29:14.380 A:middle L:90% likely to do that. And so let's say you 600 00:29:14.380 --> 00:29:18.180 A:middle L:90% do an exploratory test where you actually um do an 601 00:29:18.180 --> 00:29:22.380 A:middle L:90% interaction and see how men and women who had male 602 00:29:22.380 --> 00:29:25.019 A:middle L:90% versus female are a self reported their height. And 603 00:29:25.019 --> 00:29:27.769 A:middle L:90% you found a significant result there. That interaction would 604 00:29:27.769 --> 00:29:33.109 A:middle L:90% be the exploratory analysis. You didn't pre register it 605 00:29:33.119 --> 00:29:34.390 A:middle L:90% . It was kind of post talk after the fact 606 00:29:34.400 --> 00:29:37.440 A:middle L:90% after you saw that in all result it's a data 607 00:29:37.440 --> 00:29:40.630 A:middle L:90% dependent decision. So that would be exploratory so that 608 00:29:40.640 --> 00:29:44.269 A:middle L:90% p. Value around it is maybe not all that 609 00:29:44.279 --> 00:29:45.109 A:middle L:90% face ballad. So what you actually want to look 610 00:29:45.109 --> 00:29:48.529 A:middle L:90% for instead is start looking at the effect size the 611 00:29:48.529 --> 00:29:51.210 A:middle L:90% confidence intervals and say okay you know I found this 612 00:29:51.210 --> 00:29:53.609 A:middle L:90% during exploratory testing, Let me then do a confirmatory 613 00:29:53.609 --> 00:29:56.299 A:middle L:90% test where I set out to look specifically for this 614 00:29:56.299 --> 00:30:00.299 A:middle L:90% interaction and see if I find it again. Does 615 00:30:00.299 --> 00:30:04.289 A:middle L:90% that make sense? Okay. Yeah. And so 616 00:30:04.289 --> 00:30:08.130 A:middle L:90% the idea is not that any one type of analysis 617 00:30:08.140 --> 00:30:12.269 A:middle L:90% is more important but it's more about thinking about moving 618 00:30:12.269 --> 00:30:17.779 A:middle L:90% from exploratory to confirmatory analyses. Right? So directly 619 00:30:17.779 --> 00:30:22.150 A:middle L:90% replicating exploratory results in a confirmatory fashion. Um You 620 00:30:22.150 --> 00:30:25.079 A:middle L:90% know, I come from social psychology where we do 621 00:30:25.079 --> 00:30:27.730 A:middle L:90% a lot of, we usually use undergrads. Undergrads 622 00:30:27.730 --> 00:30:30.210 A:middle L:90% are cheap, it's really easy to replicate studies. 623 00:30:30.220 --> 00:30:33.450 A:middle L:90% Um In a lot of other areas it may be 624 00:30:33.450 --> 00:30:36.049 A:middle L:90% harder to get participants or you may be working with 625 00:30:36.049 --> 00:30:38.049 A:middle L:90% large datasets like let's say the National Elections Survey, 626 00:30:38.059 --> 00:30:41.490 A:middle L:90% which they do every Other every two or 4 years 627 00:30:41.500 --> 00:30:45.329 A:middle L:90% . Um You can't just say, hey, could 628 00:30:45.329 --> 00:30:47.480 A:middle L:90% you guys go out and collect another dataset for me 629 00:30:47.480 --> 00:30:48.779 A:middle L:90% ? You have no control over that. Um I 630 00:30:48.779 --> 00:30:52.170 A:middle L:90% did want to mention there are still ways to do 631 00:30:52.170 --> 00:30:55.480 A:middle L:90% more confirmatory tests even when you can't actually go out 632 00:30:55.480 --> 00:30:56.150 A:middle L:90% and replicate a study. When you have kind of 633 00:30:56.150 --> 00:30:59.829 A:middle L:90% set data sets. One is to, if the 634 00:30:59.829 --> 00:31:03.049 A:middle L:90% data sets large enough take it and randomly split it 635 00:31:03.059 --> 00:31:06.079 A:middle L:90% into two halves. Do your exploratory analysis on one 636 00:31:06.079 --> 00:31:07.009 A:middle L:90% half, kind of figure out what you think is 637 00:31:07.009 --> 00:31:10.269 A:middle L:90% going on and then take the other half and do 638 00:31:10.269 --> 00:31:12.220 A:middle L:90% your confirmatory analysis on that half. Um This is 639 00:31:12.220 --> 00:31:15.809 A:middle L:90% something that's really common in personality research and people that 640 00:31:15.809 --> 00:31:18.670 A:middle L:90% are looking at these big complex models. There are 641 00:31:18.670 --> 00:31:22.440 A:middle L:90% actually different tests. There's something called exploratory factor analysis 642 00:31:22.440 --> 00:31:26.339 A:middle L:90% and confirmatory factor analysis. And a lot of times 643 00:31:26.339 --> 00:31:27.029 A:middle L:90% they'll be working with these big data sets and they'll 644 00:31:27.029 --> 00:31:30.369 A:middle L:90% randomly split them. Another kind of variant of this 645 00:31:30.369 --> 00:31:33.089 A:middle L:90% that you see a lot in machine learning or computer 646 00:31:33.089 --> 00:31:37.680 A:middle L:90% science is where you train up your statistical model on 647 00:31:37.680 --> 00:31:40.680 A:middle L:90% a sample data set and then you go and kind 648 00:31:40.680 --> 00:31:42.289 A:middle L:90% of confirm it on another dataset. Then you got 649 00:31:42.289 --> 00:31:44.930 A:middle L:90% off the internet. Or if it was a big 650 00:31:44.930 --> 00:31:47.690 A:middle L:90% data set originally You'll split it into 2/2 and have 651 00:31:47.690 --> 00:31:49.730 A:middle L:90% a training data set. That exploratory data set and 652 00:31:49.730 --> 00:31:52.660 A:middle L:90% then you're testing data set, the confirmatory data set 653 00:31:53.740 --> 00:31:57.529 A:middle L:90% . Um And if you really are in a situation 654 00:31:57.529 --> 00:32:00.750 A:middle L:90% where let's say you know you do medical research and 655 00:32:00.750 --> 00:32:01.589 A:middle L:90% it's a group of patients where it's a small group 656 00:32:01.589 --> 00:32:04.809 A:middle L:90% so you can split your sample and you're never going 657 00:32:04.809 --> 00:32:06.609 A:middle L:90% to be able to get them again. It's not 658 00:32:06.609 --> 00:32:08.549 A:middle L:90% to say that you can't still run exploratory analyses, 659 00:32:08.750 --> 00:32:12.170 A:middle L:90% it's just to be clear about that so that everybody 660 00:32:12.410 --> 00:32:14.980 A:middle L:90% knows that the exploratory and we can take that into 661 00:32:14.980 --> 00:32:17.769 A:middle L:90% account when we're interpreting the findings. Um, and 662 00:32:17.769 --> 00:32:20.329 A:middle L:90% there are also things you can do right. You 663 00:32:20.329 --> 00:32:23.299 A:middle L:90% can run kind of sensitivity analysis and check for the 664 00:32:23.299 --> 00:32:28.289 A:middle L:90% robustness of the results. So let's say you had 665 00:32:28.289 --> 00:32:30.640 A:middle L:90% excluded some outliers and your effect pops out. It 666 00:32:30.640 --> 00:32:34.019 A:middle L:90% was exploratory, but you still think it's interesting. 667 00:32:34.019 --> 00:32:36.480 A:middle L:90% But you want to show that it wasn't anything particular 668 00:32:36.480 --> 00:32:37.910 A:middle L:90% about the analysis I chose. You can run a 669 00:32:37.910 --> 00:32:42.069 A:middle L:90% bunch of different analyses using different outlet exclusion methods and 670 00:32:42.069 --> 00:32:44.990 A:middle L:90% show, hey, this keeps popping up every single 671 00:32:44.990 --> 00:32:46.990 A:middle L:90% time. Maybe there is. I'm gonna trust this 672 00:32:46.990 --> 00:32:51.109 A:middle L:90% more than something where it really depended on the specific 673 00:32:51.109 --> 00:32:52.970 A:middle L:90% outlier effect I chose. So there are ways to 674 00:32:52.970 --> 00:32:57.069 A:middle L:90% kind of increase the robustness of exploratory testing, but 675 00:32:57.069 --> 00:33:00.390 A:middle L:90% it still is important to delineate between that exploratory and 676 00:33:00.400 --> 00:33:02.799 A:middle L:90% confirmatory testing. Um, so before I move on 677 00:33:02.799 --> 00:33:05.930 A:middle L:90% to the next section, does anybody have any questions 678 00:33:05.930 --> 00:33:08.670 A:middle L:90% about kind of analysis? Plans and pre registration? 679 00:33:09.640 --> 00:33:21.690 A:middle L:90% Yeah. All right. Start with this. Mm 680 00:33:21.690 --> 00:33:28.059 A:middle L:90% hmm. And speak my feel like your product on 681 00:33:29.339 --> 00:33:34.670 A:middle L:90% Reddit training. Okay. Mhm. Yes. Mhm 682 00:33:35.440 --> 00:33:50.039 A:middle L:90% . This helps. Yeah. Methods and Yeah. 683 00:33:50.539 --> 00:33:54.089 A:middle L:90% And then you have standing on. Right. That's 684 00:33:54.309 --> 00:33:58.849 A:middle L:90% actually back. It is a close and I understand 685 00:34:01.339 --> 00:34:05.569 A:middle L:90% . Yeah. All this process seems the same thing 686 00:34:05.569 --> 00:34:12.050 A:middle L:90% that like reading the grand. Very clear. Mhm 687 00:34:13.340 --> 00:34:20.860 A:middle L:90% . And all right. So I went green. 688 00:34:21.739 --> 00:34:25.760 A:middle L:90% What if? Yeah. Yes. So, what 689 00:34:25.760 --> 00:34:28.860 A:middle L:90% I would say is you know, a lot of 690 00:34:30.039 --> 00:34:34.340 A:middle L:90% I think there is a lot of shift that can 691 00:34:34.340 --> 00:34:37.050 A:middle L:90% happen during the research process. So, um you 692 00:34:37.050 --> 00:34:39.519 A:middle L:90% know, when we describe the analyses we're going to 693 00:34:39.519 --> 00:34:43.429 A:middle L:90% do, it's generally at a like abstract level. 694 00:34:43.429 --> 00:34:45.019 A:middle L:90% I'm gonna compare these two groups and a lot of 695 00:34:45.030 --> 00:34:47.639 A:middle L:90% scientific disciplines. It's not these are the actual tests 696 00:34:47.639 --> 00:34:50.880 A:middle L:90% I'm going to run. These are the cauvery. 697 00:34:50.880 --> 00:34:52.670 A:middle L:90% It's that I'm going to include. That's never it's 698 00:34:52.670 --> 00:34:57.539 A:middle L:90% usually not actually written down that specifically before the test 699 00:34:57.539 --> 00:35:00.170 A:middle L:90% happens. So that if you run something initially and 700 00:35:00.170 --> 00:35:02.179 A:middle L:90% it's not turning out you can start adding in other 701 00:35:02.179 --> 00:35:06.269 A:middle L:90% co very it's other tests trying other things out. 702 00:35:06.280 --> 00:35:08.670 A:middle L:90% So a lot of data exploration is actually going on 703 00:35:08.840 --> 00:35:12.579 A:middle L:90% but we don't realize that it's exploratory because we never 704 00:35:12.579 --> 00:35:16.090 A:middle L:90% really clearly specified what we're gonna do beforehand. Um 705 00:35:16.090 --> 00:35:20.099 A:middle L:90% Really the only science that works like that right now 706 00:35:20.110 --> 00:35:22.789 A:middle L:90% is these phase three clinical trials where they really do 707 00:35:22.789 --> 00:35:25.190 A:middle L:90% have to very clearly say exactly what they're going to 708 00:35:25.190 --> 00:35:30.349 A:middle L:90% do. Um Preregistration is not use much really in 709 00:35:30.349 --> 00:35:32.750 A:middle L:90% any discipline other than that it's starting to take root 710 00:35:32.750 --> 00:35:36.530 A:middle L:90% a little bit in econ it's starting to take root 711 00:35:36.530 --> 00:35:38.590 A:middle L:90% a little bit in psychology. Um I know from 712 00:35:38.590 --> 00:35:42.210 A:middle L:90% psychology they there was I didn't put it up, 713 00:35:42.219 --> 00:35:45.889 A:middle L:90% but there was a um there was a survey that 714 00:35:45.889 --> 00:35:47.989 A:middle L:90% they sent out to researchers and asked them, you 715 00:35:47.989 --> 00:35:51.530 A:middle L:90% know, if they engaged in some of these researcher 716 00:35:51.530 --> 00:35:52.139 A:middle L:90% degrees of freedom behavior. So, you know, 717 00:35:52.150 --> 00:35:54.699 A:middle L:90% running two dependent variables and only reporting one in their 718 00:35:54.699 --> 00:35:59.050 A:middle L:90% final paper or you know, running extra people when 719 00:35:59.050 --> 00:36:01.719 A:middle L:90% they didn't get a significant results. Um and then 720 00:36:01.719 --> 00:36:04.820 A:middle L:90% when they found a significant result just publishing after the 721 00:36:04.820 --> 00:36:07.670 A:middle L:90% fact and you get really high rates of people saying 722 00:36:07.679 --> 00:36:10.170 A:middle L:90% yes, I engage in these behaviors and I think 723 00:36:10.170 --> 00:36:13.809 A:middle L:90% it's fine because people don't realize what it's doing to 724 00:36:13.809 --> 00:36:16.050 A:middle L:90% those false positive rates. Um And so I think 725 00:36:19.429 --> 00:36:22.699 A:middle L:90% , I think it happens more than we realize because 726 00:36:22.699 --> 00:36:24.730 A:middle L:90% we don't write down beforehand what we're going to do 727 00:36:24.730 --> 00:36:27.889 A:middle L:90% . We have a general idea, but it's not 728 00:36:27.889 --> 00:36:30.880 A:middle L:90% very specific and that lack of specificity allows a lot 729 00:36:30.880 --> 00:36:34.139 A:middle L:90% of leeway in terms of what we actually do. 730 00:36:34.150 --> 00:36:42.960 A:middle L:90% Does that make sense? Yeah. To say. 731 00:36:45.929 --> 00:36:53.000 A:middle L:90% Mm. Yeah. And if that's yes and then 732 00:36:53.010 --> 00:36:59.050 A:middle L:90% you would suggest. Okay. Mhm. Yes. 733 00:37:00.030 --> 00:37:07.570 A:middle L:90% That so stand, yep. Much later. In 734 00:37:07.570 --> 00:37:13.130 A:middle L:90% fact meeting. Uh huh Holiness Dad. Very good 735 00:37:15.329 --> 00:37:21.539 A:middle L:90% for the So I'm interested in hearing to those survey 736 00:37:21.550 --> 00:37:23.369 A:middle L:90% solved instead of this has been great. And then 737 00:37:23.489 --> 00:37:30.150 A:middle L:90% what would be sure, interesting being made for? 738 00:37:30.159 --> 00:37:36.000 A:middle L:90% It's a I use brand. I'm not sure if 739 00:37:36.000 --> 00:37:42.619 A:middle L:90% you say you're right. Mhm. So let's put 740 00:37:42.619 --> 00:37:49.159 A:middle L:90% it there. Fine then. Right, think. 741 00:37:50.030 --> 00:37:54.170 A:middle L:90% Mhm, mhm. And yeah, how do you 742 00:37:54.170 --> 00:38:00.269 A:middle L:90% see that opening? Mm hmm. Yeah. So 743 00:38:00.269 --> 00:38:05.469 A:middle L:90% pre registration doesn't necessarily actually involve any, it doesn't 744 00:38:05.469 --> 00:38:08.309 A:middle L:90% have to involve any feedback from colleagues. It's more 745 00:38:08.320 --> 00:38:12.329 A:middle L:90% , it's more you specifying what analyses you're going to 746 00:38:12.329 --> 00:38:15.610 A:middle L:90% do beforehand so that when you actually do the analyses 747 00:38:15.619 --> 00:38:16.739 A:middle L:90% you can say okay what I said I was gonna 748 00:38:16.739 --> 00:38:21.960 A:middle L:90% do matches what I did and then reviewers editors and 749 00:38:21.969 --> 00:38:23.429 A:middle L:90% you know people who read your article can see okay 750 00:38:23.440 --> 00:38:25.050 A:middle L:90% this match what they said they were going to do 751 00:38:25.050 --> 00:38:29.519 A:middle L:90% or this was confirmatory. This is exploratory. Um 752 00:38:29.519 --> 00:38:32.469 A:middle L:90% There is something called a registered reports format which some 753 00:38:32.469 --> 00:38:35.480 A:middle L:90% journals are now adopting. Gonna talk about that a 754 00:38:35.480 --> 00:38:37.099 A:middle L:90% little bit later but I'll just talk about it now 755 00:38:37.110 --> 00:38:39.170 A:middle L:90% . Um Which is where you actually send your paper 756 00:38:39.170 --> 00:38:43.280 A:middle L:90% in for peer review before data collection has ever happened 757 00:38:43.289 --> 00:38:45.519 A:middle L:90% . Um And so the paper is just reviewed in 758 00:38:45.519 --> 00:38:49.949 A:middle L:90% terms of you know theoretical soundness, methodological soundness and 759 00:38:49.949 --> 00:38:54.329 A:middle L:90% statistical soundness. That is where feedback comes into the 760 00:38:54.340 --> 00:38:59.119 A:middle L:90% preregistration kind of world. But you can do pre 761 00:38:59.119 --> 00:39:01.449 A:middle L:90% registrations without feedback if that does that make sense? 762 00:39:01.820 --> 00:39:04.920 A:middle L:90% Yeah. It sounds like, I mean it sounds 763 00:39:04.920 --> 00:39:07.210 A:middle L:90% like if you wanted to keep your you know your 764 00:39:07.210 --> 00:39:10.690 A:middle L:90% study a secret but wanted to ensure that your results 765 00:39:10.699 --> 00:39:14.199 A:middle L:90% were done in a way that think it's valid. 766 00:39:14.210 --> 00:39:16.010 A:middle L:90% It looks like they have a way to allow for 767 00:39:16.010 --> 00:39:20.389 A:middle L:90% sort of private. Yeah. So you you could 768 00:39:20.400 --> 00:39:22.300 A:middle L:90% say I'm committing this report. It's embargoed. Nobody 769 00:39:22.300 --> 00:39:24.880 A:middle L:90% can see it when I published. I will make 770 00:39:24.880 --> 00:39:29.809 A:middle L:90% it apparent that I Yeah so the osF which I'm 771 00:39:29.809 --> 00:39:32.730 A:middle L:90% going to talk about a little later um allows for 772 00:39:32.739 --> 00:39:37.090 A:middle L:90% for you to register studies. Do preregistration. You 773 00:39:37.090 --> 00:39:39.030 A:middle L:90% can make those public or private registrations and if you 774 00:39:39.030 --> 00:39:40.809 A:middle L:90% initially make it a private one you can make a 775 00:39:40.809 --> 00:39:44.960 A:middle L:90% public later on. Um something a lot of scientists 776 00:39:44.960 --> 00:39:46.349 A:middle L:90% are worry about is something called scooping, right? 777 00:39:46.349 --> 00:39:49.250 A:middle L:90% You're afraid for those of you who don't know the 778 00:39:50.019 --> 00:39:52.750 A:middle L:90% jargon term. It's, you know, if you're 779 00:39:52.750 --> 00:39:54.059 A:middle L:90% working on something and then somebody else is working on 780 00:39:54.059 --> 00:39:55.630 A:middle L:90% the same thing and they published before you, you've 781 00:39:55.639 --> 00:39:59.500 A:middle L:90% been scooped. Um, some scientists, many scientists 782 00:39:59.500 --> 00:40:00.809 A:middle L:90% are actually quite worried about that because generally if somebody 783 00:40:00.809 --> 00:40:04.289 A:middle L:90% else publishes the stuff, you can't then publish it 784 00:40:04.300 --> 00:40:06.409 A:middle L:90% or you have to publish it in a less prestigious 785 00:40:06.409 --> 00:40:07.980 A:middle L:90% journal. And so some people have worried about, 786 00:40:07.980 --> 00:40:10.440 A:middle L:90% well, if I make my registration public before I've 787 00:40:10.440 --> 00:40:14.090 A:middle L:90% ever done my data, might somebody see that quickly 788 00:40:14.090 --> 00:40:15.510 A:middle L:90% run the study and scoop me? Um, and 789 00:40:15.510 --> 00:40:19.050 A:middle L:90% also some grants, especially if you're working with some 790 00:40:19.050 --> 00:40:22.659 A:middle L:90% government agencies, you can't actually release information about the 791 00:40:22.659 --> 00:40:25.170 A:middle L:90% study until after it's done. Um, and so 792 00:40:25.179 --> 00:40:29.480 A:middle L:90% the OSF does allow for private registrations that can be 793 00:40:29.480 --> 00:40:32.030 A:middle L:90% made public later or that you can just let reviewers 794 00:40:32.030 --> 00:40:35.440 A:middle L:90% see if you're doing registered reports or something like that 795 00:40:35.809 --> 00:40:37.170 A:middle L:90% . Clinical trials Gov, which is the one kind 796 00:40:37.170 --> 00:40:40.539 A:middle L:90% of other main registration system out there right now, 797 00:40:40.550 --> 00:40:44.369 A:middle L:90% all those have to be public, but other sites 798 00:40:44.369 --> 00:40:47.130 A:middle L:90% do not require that. All right. Um, 799 00:40:47.130 --> 00:40:50.639 A:middle L:90% so now I'm going to start talking about the current 800 00:40:50.639 --> 00:40:52.550 A:middle L:90% scientific workflow and how that can create some problems in 801 00:40:52.550 --> 00:40:55.320 A:middle L:90% terms of openness and transparency and just the accumulation of 802 00:40:55.320 --> 00:41:00.719 A:middle L:90% knowledge and ways research, simple ways that researchers can 803 00:41:00.730 --> 00:41:04.760 A:middle L:90% increase the documentation of their workflow, which is actually 804 00:41:04.760 --> 00:41:06.590 A:middle L:90% really good for them. And it's also good for 805 00:41:06.590 --> 00:41:07.849 A:middle L:90% science as a whole. Right. And so currently 806 00:41:07.849 --> 00:41:13.019 A:middle L:90% the vast majority of the scientific research workflow is obscure 807 00:41:13.030 --> 00:41:15.010 A:middle L:90% . If I'm not in your lab, all I 808 00:41:15.010 --> 00:41:16.179 A:middle L:90% see is what you published in your paper, which 809 00:41:16.179 --> 00:41:19.610 A:middle L:90% have you ever looked at somebody else's paper and trying 810 00:41:19.610 --> 00:41:21.610 A:middle L:90% to figure out exactly what they did. It is 811 00:41:21.610 --> 00:41:24.280 A:middle L:90% surprisingly hard to do. Um, I'm working on 812 00:41:24.280 --> 00:41:27.239 A:middle L:90% a project right now where we're trying to run direct 813 00:41:27.250 --> 00:41:29.730 A:middle L:90% replications of other people's papers from, you know, 814 00:41:29.730 --> 00:41:31.840 A:middle L:90% six years ago and some of the analyses you read 815 00:41:31.840 --> 00:41:34.469 A:middle L:90% it, and you're like, I have, I 816 00:41:34.469 --> 00:41:36.159 A:middle L:90% have no idea what you did, this makes no 817 00:41:36.159 --> 00:41:38.809 A:middle L:90% sense. Um, or the methods are really complicated 818 00:41:38.809 --> 00:41:42.769 A:middle L:90% and they are obviously missing really important points. And 819 00:41:42.769 --> 00:41:44.880 A:middle L:90% then you email them and they go, oh yeah 820 00:41:44.889 --> 00:41:45.469 A:middle L:90% , we completely left this out of the paper, 821 00:41:45.469 --> 00:41:47.579 A:middle L:90% but here are the materials, right? All of 822 00:41:47.579 --> 00:41:51.079 A:middle L:90% that goes unreported in the paper. And so it's 823 00:41:51.079 --> 00:41:53.070 A:middle L:90% actually really hard to figure out what was actually done 824 00:41:53.070 --> 00:41:55.670 A:middle L:90% in practice in the lab versus what appears in the 825 00:41:55.670 --> 00:42:00.119 A:middle L:90% paper. And potentially how that research idea has changed 826 00:42:00.130 --> 00:42:05.139 A:middle L:90% over time with that. It's also hard to reproduce 827 00:42:05.139 --> 00:42:07.110 A:middle L:90% other people's work. And it can be actually really 828 00:42:07.110 --> 00:42:08.469 A:middle L:90% hard to reproduce our own work. So the way 829 00:42:08.469 --> 00:42:10.369 A:middle L:90% a lot of labs work, depending on the field 830 00:42:10.369 --> 00:42:14.039 A:middle L:90% urine is each graduate student kind of has some projects 831 00:42:14.039 --> 00:42:15.789 A:middle L:90% that they work on. And then a graduate student 832 00:42:15.800 --> 00:42:19.019 A:middle L:90% , you know, leaves if they go into academia 833 00:42:19.019 --> 00:42:21.159 A:middle L:90% , maybe they keep those projects going. If they 834 00:42:21.159 --> 00:42:23.920 A:middle L:90% don't go into academia, they leave. And it's 835 00:42:23.920 --> 00:42:25.469 A:middle L:90% really hard to get in touch with them. And 836 00:42:25.469 --> 00:42:28.760 A:middle L:90% the project kind of languishes in the corner for a 837 00:42:28.760 --> 00:42:30.059 A:middle L:90% couple of years and then another grad student comes along 838 00:42:30.059 --> 00:42:31.829 A:middle L:90% . It's like, hey, I might be interested 839 00:42:31.829 --> 00:42:34.699 A:middle L:90% in that and you go back and look at you 840 00:42:34.699 --> 00:42:37.039 A:middle L:90% like this was two years ago. I have no 841 00:42:37.039 --> 00:42:38.280 A:middle L:90% idea what I was doing here. I don't remember 842 00:42:38.280 --> 00:42:40.829 A:middle L:90% what these variables mean. I don't actually remember what 843 00:42:40.829 --> 00:42:44.719 A:middle L:90% my specific research question was. All of this is 844 00:42:44.719 --> 00:42:49.320 A:middle L:90% kind of very obscure. Um, so there's a 845 00:42:49.800 --> 00:42:52.409 A:middle L:90% , there's a great little video, um about this 846 00:42:52.409 --> 00:42:53.500 A:middle L:90% which I took out for time, but it's these 847 00:42:53.510 --> 00:42:55.989 A:middle L:90% two panda bears talking to each other, because everything 848 00:42:55.989 --> 00:42:58.989 A:middle L:90% with panda bears, it's just cuter. And one 849 00:42:58.989 --> 00:43:00.989 A:middle L:90% of them is asking for the other person's data. 850 00:43:00.000 --> 00:43:01.610 A:middle L:90% Like, you know, you send me the data 851 00:43:01.619 --> 00:43:05.199 A:middle L:90% and these variable names don't make any sense. What 852 00:43:05.210 --> 00:43:07.130 A:middle L:90% what does SAm one mean? The guy goes, 853 00:43:07.130 --> 00:43:08.849 A:middle L:90% oh that's my collaborator, Sam lee, that's short 854 00:43:08.849 --> 00:43:10.559 A:middle L:90% for his name. The guy is like, okay 855 00:43:10.559 --> 00:43:12.619 A:middle L:90% , what does it stand for? And he says 856 00:43:12.619 --> 00:43:14.340 A:middle L:90% oh it's this protein. He goes, okay, 857 00:43:14.349 --> 00:43:15.980 A:middle L:90% well what does SAM to stand for? And he 858 00:43:15.980 --> 00:43:16.019 A:middle L:90% goes, oh, I don't know. This was 859 00:43:16.019 --> 00:43:19.550 A:middle L:90% a couple years ago, my collaborator probably knows the 860 00:43:19.550 --> 00:43:20.559 A:middle L:90% guy goes, okay, how do I get in 861 00:43:20.559 --> 00:43:21.679 A:middle L:90% touch with him? And he goes, well, 862 00:43:21.690 --> 00:43:23.280 A:middle L:90% he's in china, and his name's Sam lee, 863 00:43:23.280 --> 00:43:24.949 A:middle L:90% you can probably find him. The guy goes, 864 00:43:24.949 --> 00:43:29.110 A:middle L:90% no I cannot find him. Your data is useless 865 00:43:29.110 --> 00:43:30.059 A:middle L:90% to me because I have no idea what these variables 866 00:43:30.059 --> 00:43:35.110 A:middle L:90% mean. Um And so right everybody who watches this 867 00:43:35.110 --> 00:43:36.940 A:middle L:90% video kind of laughs at it, but then there's 868 00:43:36.940 --> 00:43:38.329 A:middle L:90% this moment where you kind of realize, wait I 869 00:43:38.329 --> 00:43:40.210 A:middle L:90% have some data sets from a couple years ago where 870 00:43:40.210 --> 00:43:42.320 A:middle L:90% I named it like D. V. One in 871 00:43:42.320 --> 00:43:44.260 A:middle L:90% D. V. Two. I probably knew what 872 00:43:44.260 --> 00:43:45.260 A:middle L:90% time and at that point in time I probably do 873 00:43:45.260 --> 00:43:49.139 A:middle L:90% not remember what that means now. And like I 874 00:43:49.150 --> 00:43:51.900 A:middle L:90% did this um I got my PhD you know in 875 00:43:51.900 --> 00:43:53.159 A:middle L:90% june and I was packing up and moving all my 876 00:43:53.159 --> 00:43:57.190 A:middle L:90% stuff and I was trying to organise studies that I've 877 00:43:57.190 --> 00:44:00.230 A:middle L:90% done my first year in grad school and I had 878 00:44:00.230 --> 00:44:01.210 A:middle L:90% no idea what half of them meant. Which is 879 00:44:01.210 --> 00:44:02.969 A:middle L:90% really sad. I mean obviously they're not going to 880 00:44:02.969 --> 00:44:05.570 A:middle L:90% go anywhere now, but there was a lot of 881 00:44:05.570 --> 00:44:07.420 A:middle L:90% resources, both my own resources, participant resources, 882 00:44:07.420 --> 00:44:10.440 A:middle L:90% lab resources that were kind of wasted. Because those 883 00:44:10.440 --> 00:44:14.010 A:middle L:90% studies can't continue to move forward and they can't be 884 00:44:14.010 --> 00:44:16.420 A:middle L:90% used by other people down the line. And so 885 00:44:17.590 --> 00:44:20.880 A:middle L:90% , you know, it's hard to reproduce your own 886 00:44:20.880 --> 00:44:22.519 A:middle L:90% work sometimes because you don't have a good handle on 887 00:44:22.519 --> 00:44:24.909 A:middle L:90% what you actually did. And so the last thing 888 00:44:24.909 --> 00:44:30.000 A:middle L:90% is it's difficult to accumulate unpublished knowledge just because, 889 00:44:30.690 --> 00:44:32.199 A:middle L:90% you know, if it actually doesn't get published anywhere 890 00:44:32.199 --> 00:44:34.789 A:middle L:90% , we have less of a track record of it 891 00:44:34.789 --> 00:44:37.159 A:middle L:90% , at least when it gets published. Maybe you 892 00:44:37.170 --> 00:44:38.380 A:middle L:90% well you have to say what your research question was 893 00:44:38.380 --> 00:44:40.420 A:middle L:90% , what your hypothesis was, if it doesn't get 894 00:44:40.420 --> 00:44:43.780 A:middle L:90% published and maybe you did didn't do a presentation on 895 00:44:43.780 --> 00:44:45.570 A:middle L:90% it. Some of that maybe scribbled on a note 896 00:44:45.570 --> 00:44:47.599 A:middle L:90% where somewhere notebook somewhere and now you've moved and God 897 00:44:47.599 --> 00:44:49.699 A:middle L:90% knows where it is. And so you have all 898 00:44:49.699 --> 00:44:52.110 A:middle L:90% these pieces and parts that aren't really connected up, 899 00:44:52.690 --> 00:44:53.570 A:middle L:90% so you can't find them, Nobody else can find 900 00:44:53.570 --> 00:44:57.309 A:middle L:90% them. It becomes really hard to accumulate that knowledge 901 00:44:58.690 --> 00:45:00.599 A:middle L:90% . And so one good way around some of these 902 00:45:00.599 --> 00:45:04.130 A:middle L:90% problems is to increase the documentation of the workflow. 903 00:45:04.130 --> 00:45:06.159 A:middle L:90% So right now, all we have is documentation in 904 00:45:06.159 --> 00:45:07.960 A:middle L:90% terms of what is actually published, but there are 905 00:45:07.960 --> 00:45:10.409 A:middle L:90% a lot of other places along the workflow that it's 906 00:45:10.409 --> 00:45:13.719 A:middle L:90% really important to actually keep track of what is being 907 00:45:13.730 --> 00:45:15.800 A:middle L:90% done. And so the first thing is knowing how 908 00:45:15.800 --> 00:45:20.329 A:middle L:90% things started and how they evolved over time. And 909 00:45:20.329 --> 00:45:22.219 A:middle L:90% this is really important. As I said, because 910 00:45:22.219 --> 00:45:23.369 A:middle L:90% you can track how things changed, you can track 911 00:45:23.369 --> 00:45:25.260 A:middle L:90% . Okay, this is what my research question originally 912 00:45:25.260 --> 00:45:28.190 A:middle L:90% was after doing a review. Maybe this is how 913 00:45:28.190 --> 00:45:30.789 A:middle L:90% it started. This is what my analysis plan originally 914 00:45:30.789 --> 00:45:32.079 A:middle L:90% looked like. Here's where it ended up. There's 915 00:45:32.079 --> 00:45:34.699 A:middle L:90% something how many of you in this room have heard 916 00:45:34.699 --> 00:45:37.780 A:middle L:90% of something called Version Control? Okay. Yeah. 917 00:45:37.789 --> 00:45:39.130 A:middle L:90% Good. Um You are ahead of the game. 918 00:45:39.139 --> 00:45:42.130 A:middle L:90% So right for those of you have not heard a 919 00:45:42.130 --> 00:45:44.829 A:middle L:90% Version Control. It's okay. Um You know, 920 00:45:44.829 --> 00:45:46.059 A:middle L:90% I'm not gonna throw you out or anything. It's 921 00:45:46.070 --> 00:45:50.320 A:middle L:90% much more prevalent in some areas than other areas. 922 00:45:50.320 --> 00:45:52.329 A:middle L:90% If you're from computer science or biostatistics, um you're 923 00:45:52.329 --> 00:45:54.980 A:middle L:90% far more likely to have heard a Version control than 924 00:45:54.980 --> 00:45:58.710 A:middle L:90% some of the social sciences. Um So version control 925 00:45:58.710 --> 00:46:02.570 A:middle L:90% is basically just where a website or a computer program 926 00:46:02.570 --> 00:46:07.869 A:middle L:90% or something like that logs every single version of a 927 00:46:07.869 --> 00:46:09.070 A:middle L:90% file that you have. And so you can always 928 00:46:09.070 --> 00:46:12.820 A:middle L:90% go back and find older files of it and you 929 00:46:12.820 --> 00:46:14.750 A:middle L:90% can see how things have changed over time. So 930 00:46:14.750 --> 00:46:15.650 A:middle L:90% like on your regular computer, if you're working on 931 00:46:15.650 --> 00:46:19.800 A:middle L:90% a manuscript and you save over the same file, 932 00:46:19.809 --> 00:46:22.030 A:middle L:90% you can't then go back and find the other file 933 00:46:22.039 --> 00:46:23.449 A:middle L:90% in a version control system. You can go back 934 00:46:23.449 --> 00:46:27.289 A:middle L:90% to those other files and you can see how they've 935 00:46:27.289 --> 00:46:30.889 A:middle L:90% changed and how they're different from one another. So 936 00:46:30.889 --> 00:46:34.289 A:middle L:90% the next thing is saving and annotating syntax. This 937 00:46:34.300 --> 00:46:37.349 A:middle L:90% seems really simple but it's not done nearly as much 938 00:46:37.349 --> 00:46:38.179 A:middle L:90% as it should be. Um I come as I 939 00:46:38.179 --> 00:46:42.030 A:middle L:90% said I come from social psychology where SPS S. 940 00:46:42.039 --> 00:46:45.130 A:middle L:90% Is still kind of the statistical tool of choice and 941 00:46:45.139 --> 00:46:46.170 A:middle L:90% S. P. S. S. Can be 942 00:46:46.170 --> 00:46:47.940 A:middle L:90% used in a point and click fashion. Things like 943 00:46:47.949 --> 00:46:51.130 A:middle L:90% data and SAs can actually be used in a point 944 00:46:51.130 --> 00:46:52.280 A:middle L:90% and click fashion. They're usually not. Usually the 945 00:46:52.280 --> 00:46:54.989 A:middle L:90% code is coded out and you save that syntax file 946 00:46:54.989 --> 00:46:58.000 A:middle L:90% which is great. Um If you use a point 947 00:46:58.000 --> 00:46:59.110 A:middle L:90% and click program like S. P. S. 948 00:46:59.110 --> 00:47:00.650 A:middle L:90% S. There's a little paste button on it and 949 00:47:00.650 --> 00:47:02.690 A:middle L:90% you can actually save out that syntax. So saving 950 00:47:02.690 --> 00:47:05.829 A:middle L:90% the syntax is really important because you can actually go 951 00:47:05.829 --> 00:47:07.590 A:middle L:90% back and reproduce the exact statistical test you ran. 952 00:47:07.980 --> 00:47:12.150 A:middle L:90% But annotation is really important to, so annotation is 953 00:47:12.150 --> 00:47:15.360 A:middle L:90% actually writing in some comments into your code, saying 954 00:47:15.360 --> 00:47:17.110 A:middle L:90% this is what I was doing and this is why 955 00:47:17.119 --> 00:47:19.369 A:middle L:90% . Um So I just wanted to give you kind 956 00:47:19.369 --> 00:47:22.559 A:middle L:90% of an example of what this looks like and how 957 00:47:22.570 --> 00:47:25.940 A:middle L:90% annotation can help other people but also help you figure 958 00:47:25.940 --> 00:47:28.599 A:middle L:90% out what the heck you were doing a couple years 959 00:47:28.599 --> 00:47:30.059 A:middle L:90% ago. Where is my screen? There we go 960 00:47:30.130 --> 00:47:34.599 A:middle L:90% . Okay so this is are for those of you 961 00:47:34.599 --> 00:47:36.960 A:middle L:90% who are not familiar with our studio. Um It 962 00:47:36.960 --> 00:47:38.320 A:middle L:90% is a lovely shiny little program. Can you all 963 00:47:38.320 --> 00:47:40.909 A:middle L:90% read this or is this too small? Okay. 964 00:47:42.880 --> 00:47:51.059 A:middle L:90% Yeah. Okay. I know this week, that's 965 00:47:51.059 --> 00:47:52.869 A:middle L:90% not actually doing what I wanted to do. Does 966 00:47:52.869 --> 00:48:00.960 A:middle L:90% anyone remember how? Oh wait wait, this is 967 00:48:00.960 --> 00:48:05.099 A:middle L:90% what I want to do. Okay, is that 968 00:48:05.099 --> 00:48:08.300 A:middle L:90% better? Okay, so I have three versions of 969 00:48:08.300 --> 00:48:10.340 A:middle L:90% the same code. So this is just the syntax 970 00:48:10.510 --> 00:48:14.579 A:middle L:90% saved. No, this is the annotated one. 971 00:48:14.579 --> 00:48:15.389 A:middle L:90% Sorry, I'm trying to work with remote screening. 972 00:48:15.389 --> 00:48:17.429 A:middle L:90% It's not going well. Right, so these are 973 00:48:17.429 --> 00:48:20.420 A:middle L:90% all the same three code. There's just different levels 974 00:48:20.420 --> 00:48:22.619 A:middle L:90% of annotation. So as you can see this is 975 00:48:22.619 --> 00:48:25.110 A:middle L:90% just a giant block of code and probably if I 976 00:48:25.110 --> 00:48:27.840 A:middle L:90% went back and read through this, I can maybe 977 00:48:27.840 --> 00:48:29.940 A:middle L:90% figure out what I was doing, but I probably 978 00:48:29.940 --> 00:48:30.880 A:middle L:90% wouldn't know why because all it is is the code 979 00:48:30.880 --> 00:48:34.130 A:middle L:90% . There's no explanation of, you know why I 980 00:48:34.130 --> 00:48:36.039 A:middle L:90% did what I did, what's going on? Why 981 00:48:36.039 --> 00:48:37.929 A:middle L:90% did I make some of the particular choices that I 982 00:48:37.929 --> 00:48:39.599 A:middle L:90% made? Um so the next example is just a 983 00:48:39.599 --> 00:48:42.840 A:middle L:90% bare bones annotation. So you can see that a 984 00:48:42.840 --> 00:48:45.219 A:middle L:90% lot of the code blocks just have these little identify 985 00:48:45.219 --> 00:48:47.090 A:middle L:90% , here's what they're doing, maybe why I did 986 00:48:47.090 --> 00:48:51.099 A:middle L:90% them. So this is helpful to me because it 987 00:48:51.099 --> 00:48:52.710 A:middle L:90% gives me a little bit more insight into what in 988 00:48:52.710 --> 00:48:53.989 A:middle L:90% the world I was thinking it's helpful for somebody else 989 00:48:53.989 --> 00:48:55.989 A:middle L:90% because it makes my code more readable and a little 990 00:48:55.989 --> 00:48:59.849 A:middle L:90% bit more understandable. So the last one is an 991 00:48:59.849 --> 00:49:02.019 A:middle L:90% example of something using a package called knit are, 992 00:49:02.030 --> 00:49:05.380 A:middle L:90% which is a fantastic package. Um, it's something 993 00:49:05.389 --> 00:49:07.670 A:middle L:90% a little bit more akin to what is called literate 994 00:49:07.670 --> 00:49:09.610 A:middle L:90% coding or literate programming, where you actually kind of 995 00:49:09.610 --> 00:49:13.409 A:middle L:90% write a little narrative about your code. And so 996 00:49:13.409 --> 00:49:15.050 A:middle L:90% what you'll notice about this is that they're actually these 997 00:49:15.050 --> 00:49:20.820 A:middle L:90% big kind of sections of text where I'm really being 998 00:49:20.820 --> 00:49:22.130 A:middle L:90% explicit about, okay, this is what I was 999 00:49:22.130 --> 00:49:24.460 A:middle L:90% doing. This is why explaining in full sentences, 1000 00:49:24.460 --> 00:49:27.619 A:middle L:90% making it very readable. And then you have these 1001 00:49:27.619 --> 00:49:30.329 A:middle L:90% code chunks. It makes it a lot easier for 1002 00:49:30.340 --> 00:49:31.360 A:middle L:90% other people to figure out what you're doing. It 1003 00:49:31.360 --> 00:49:34.360 A:middle L:90% makes it easier for you to figure out what you're 1004 00:49:34.360 --> 00:49:38.130 A:middle L:90% doing. And so, you know, annotation is 1005 00:49:38.139 --> 00:49:40.789 A:middle L:90% the more you can kind of talk through in plain 1006 00:49:40.789 --> 00:49:44.699 A:middle L:90% english what you're doing, the more useful it becomes 1007 00:49:44.699 --> 00:49:45.699 A:middle L:90% to yourself and other people when you're trying to look 1008 00:49:45.699 --> 00:49:47.699 A:middle L:90% back at old code and figure out, okay, 1009 00:49:47.710 --> 00:49:51.300 A:middle L:90% what did I do and why did I do it 1010 00:49:51.309 --> 00:49:52.400 A:middle L:90% with statistics? A lot of times there are many 1011 00:49:52.400 --> 00:49:54.800 A:middle L:90% different choices you can make. And so that why 1012 00:49:54.800 --> 00:49:59.289 A:middle L:90% is actually an incredibly important piece of information to know 1013 00:49:59.369 --> 00:50:04.280 A:middle L:90% going forward. Right, so going back to this 1014 00:50:05.760 --> 00:50:09.860 A:middle L:90% , you represent her right? Um Right. And 1015 00:50:09.860 --> 00:50:12.389 A:middle L:90% so I talked about this a little bit before, 1016 00:50:12.389 --> 00:50:15.110 A:middle L:90% clearly naming variables and creating code books. Um if 1017 00:50:15.110 --> 00:50:16.989 A:middle L:90% you're somebody who works with huge surveys, code books 1018 00:50:16.989 --> 00:50:21.050 A:middle L:90% are probably very familiar to you. Um you know 1019 00:50:21.059 --> 00:50:22.400 A:middle L:90% , I and grad school did not work with huge 1020 00:50:22.409 --> 00:50:24.820 A:middle L:90% surveys, but it was something where, you know 1021 00:50:24.820 --> 00:50:27.639 A:middle L:90% , going back if I didn't have access to my 1022 00:50:27.639 --> 00:50:29.559 A:middle L:90% SP Ss file where I had said, what did 1023 00:50:29.559 --> 00:50:30.639 A:middle L:90% I code as female and what did I could is 1024 00:50:30.639 --> 00:50:32.150 A:middle L:90% male. All I have is the zero ones in 1025 00:50:32.150 --> 00:50:35.739 A:middle L:90% my data file. And it's actually incredibly difficult to 1026 00:50:35.739 --> 00:50:37.400 A:middle L:90% go back, especially if it's a couple of years 1027 00:50:37.400 --> 00:50:38.389 A:middle L:90% back and figure out, Ok, how was this 1028 00:50:38.389 --> 00:50:42.000 A:middle L:90% coded? Really simple? You know, documents just 1029 00:50:42.000 --> 00:50:44.389 A:middle L:90% saying zero is male. One is female can be 1030 00:50:44.389 --> 00:50:46.880 A:middle L:90% really helpful in terms of making your dataset more understandable 1031 00:50:46.880 --> 00:50:51.480 A:middle L:90% for you and other people. Um so the last 1032 00:50:51.960 --> 00:50:53.659 A:middle L:90% second to last one is making sure you actually know 1033 00:50:53.659 --> 00:50:57.070 A:middle L:90% what materials you use for your study. This may 1034 00:50:57.070 --> 00:51:00.980 A:middle L:90% seem really silly because most people probably think, hey 1035 00:51:00.989 --> 00:51:02.349 A:middle L:90% , I know what materials I use for my study 1036 00:51:02.360 --> 00:51:07.440 A:middle L:90% um in practice, this is surprisingly sneaky, especially 1037 00:51:07.440 --> 00:51:09.119 A:middle L:90% when it comes to raw data and syntax, right 1038 00:51:09.130 --> 00:51:13.269 A:middle L:90% ? Especially if your analysis, scripts and tax changes 1039 00:51:13.269 --> 00:51:15.409 A:middle L:90% over time. What can happen is you get like 1040 00:51:15.420 --> 00:51:20.119 A:middle L:90% you open up a file and you have final data 1041 00:51:20.119 --> 00:51:24.000 A:middle L:90% , one final data to really final data, really 1042 00:51:24.000 --> 00:51:27.320 A:middle L:90% final data to your like what in the world is 1043 00:51:27.320 --> 00:51:29.389 A:middle L:90% this, what did I use for my analyses? 1044 00:51:29.389 --> 00:51:31.539 A:middle L:90% This makes no sense. Can you tell this happen 1045 00:51:31.539 --> 00:51:37.369 A:middle L:90% to me? I had my original advisor was, 1046 00:51:37.380 --> 00:51:40.110 A:middle L:90% he was a very unorganised man and so I picked 1047 00:51:40.110 --> 00:51:45.679 A:middle L:90% up some bad organizational skills early in my um graduate 1048 00:51:45.690 --> 00:51:47.829 A:middle L:90% career and it's easier to start out with, you 1049 00:51:47.829 --> 00:51:51.150 A:middle L:90% know, good organization, but they can be changed 1050 00:51:51.150 --> 00:51:52.639 A:middle L:90% over time, right? But you know, really 1051 00:51:52.650 --> 00:51:55.110 A:middle L:90% being sure that, you know, okay, these 1052 00:51:55.110 --> 00:51:58.230 A:middle L:90% are the materials I use for my study. This 1053 00:51:58.230 --> 00:52:00.210 A:middle L:90% is my original Sin Tax, this is my raw 1054 00:52:00.210 --> 00:52:01.739 A:middle L:90% data, having all of those and not having the 1055 00:52:01.739 --> 00:52:06.139 A:middle L:90% system where you have like a little number clicking up 1056 00:52:06.139 --> 00:52:07.460 A:middle L:90% on the end of the data file. And so 1057 00:52:07.460 --> 00:52:10.119 A:middle L:90% the last thing is knowing where everything is. A 1058 00:52:10.119 --> 00:52:15.400 A:middle L:90% lot of times our workflow is very um spread out 1059 00:52:15.409 --> 00:52:17.360 A:middle L:90% . So you have sex, like something's live in 1060 00:52:17.360 --> 00:52:20.780 A:middle L:90% your email, something's live on servers, something's livin 1061 00:52:20.780 --> 00:52:23.230 A:middle L:90% notebook, something's live in power point presentations, and 1062 00:52:23.230 --> 00:52:25.880 A:middle L:90% it's easy to lose those pieces and parts and so 1063 00:52:25.880 --> 00:52:29.800 A:middle L:90% finding a way to connect all of those um and 1064 00:52:29.800 --> 00:52:31.380 A:middle L:90% the demo I'm going to do, which is the 1065 00:52:31.380 --> 00:52:34.960 A:middle L:90% osF is going to particularly show you one way to 1066 00:52:34.969 --> 00:52:37.260 A:middle L:90% easily connect all these pieces and parts of your data 1067 00:52:37.260 --> 00:52:39.349 A:middle L:90% set. Version control things. Preregister things and really 1068 00:52:39.349 --> 00:52:43.760 A:middle L:90% kind of organize the pieces and parts of your workflow 1069 00:52:45.349 --> 00:52:46.570 A:middle L:90% . Right? And so kind of the idea is 1070 00:52:47.150 --> 00:52:51.110 A:middle L:90% if your workflow is well documented in two years, 1071 00:52:51.119 --> 00:52:52.010 A:middle L:90% will you remember what you did and why? When 1072 00:52:52.010 --> 00:52:54.559 A:middle L:90% you're thinking about workflow documentation? This is the question 1073 00:52:54.570 --> 00:52:58.619 A:middle L:90% I tend to ask myself because I think when you 1074 00:52:58.619 --> 00:53:00.820 A:middle L:90% think about other people, some people go well I 1075 00:53:00.820 --> 00:53:01.949 A:middle L:90% don't care about other people, but most people do 1076 00:53:01.949 --> 00:53:04.730 A:middle L:90% care about if they will be able to reuse their 1077 00:53:04.730 --> 00:53:07.440 A:middle L:90% stuff two years later. Um but usually if you 1078 00:53:07.440 --> 00:53:08.159 A:middle L:90% can remember what you did two years later, it 1079 00:53:08.159 --> 00:53:10.550 A:middle L:90% makes it easier for other people as well. So 1080 00:53:10.550 --> 00:53:13.550 A:middle L:90% it can be a really useful question to ask when 1081 00:53:13.550 --> 00:53:15.420 A:middle L:90% you're trying to think about, okay, what do 1082 00:53:15.420 --> 00:53:17.079 A:middle L:90% I want my workflow to look like? Um And 1083 00:53:17.079 --> 00:53:20.920 A:middle L:90% so because I'm running short on time, I thought 1084 00:53:20.920 --> 00:53:22.380 A:middle L:90% I could get through this as an hour. I 1085 00:53:22.380 --> 00:53:25.710 A:middle L:90% have failed that goal. Um I'm gonna skip over 1086 00:53:25.719 --> 00:53:28.639 A:middle L:90% this section because a lot of you were at brian's 1087 00:53:28.639 --> 00:53:30.909 A:middle L:90% talking. So you heard about badges? I talked 1088 00:53:30.909 --> 00:53:34.170 A:middle L:90% a little bit about registered reports. Um Right, 1089 00:53:34.179 --> 00:53:37.099 A:middle L:90% so the osf, so right, I was talking 1090 00:53:37.099 --> 00:53:39.289 A:middle L:90% about this idea of trying to, you know, 1091 00:53:39.289 --> 00:53:43.780 A:middle L:90% track everything that goes on in the workflow, from 1092 00:53:43.780 --> 00:53:45.980 A:middle L:90% idea development to study design to data, to analysis 1093 00:53:45.989 --> 00:53:50.190 A:middle L:90% , to publishing reports. Um And this will also 1094 00:53:50.190 --> 00:53:54.489 A:middle L:90% tie together some of those open themes. So if 1095 00:53:54.489 --> 00:53:57.579 A:middle L:90% you have computers and you want to follow along in 1096 00:53:57.579 --> 00:53:59.670 A:middle L:90% real time, I know I'm somebody who like learns 1097 00:53:59.670 --> 00:54:01.139 A:middle L:90% through doing um feel free, but I'm also going 1098 00:54:01.139 --> 00:54:02.780 A:middle L:90% to show it up on the computer for those of 1099 00:54:02.780 --> 00:54:05.690 A:middle L:90% you who don't want to follow along or don't have 1100 00:54:05.690 --> 00:54:07.610 A:middle L:90% laptops. Um, so if you want to follow 1101 00:54:07.610 --> 00:54:09.519 A:middle L:90% along, go to OsF dot Io Io stands for 1102 00:54:09.519 --> 00:54:12.440 A:middle L:90% indian Ocean. This is a developer thing. It's 1103 00:54:12.440 --> 00:54:14.619 A:middle L:90% not like some sketchy going to like ask you for 1104 00:54:14.619 --> 00:54:15.829 A:middle L:90% your bank account, Thanks for those of you who 1105 00:54:15.829 --> 00:54:21.300 A:middle L:90% are leaving right now. There's actually um if you 1106 00:54:21.309 --> 00:54:22.260 A:middle L:90% email me, I can actually send you to a 1107 00:54:22.269 --> 00:54:27.019 A:middle L:90% video where um, there's like an online tour if 1108 00:54:27.019 --> 00:54:29.190 A:middle L:90% you want to watch that to get an idea of 1109 00:54:29.190 --> 00:54:30.559 A:middle L:90% kind of what the OSF is and how it can 1110 00:54:30.559 --> 00:54:34.789 A:middle L:90% help document and connect workflow. Um just because I 1111 00:54:34.789 --> 00:54:37.230 A:middle L:90% do realize there's a football game and at Virginia Tech 1112 00:54:37.230 --> 00:54:38.670 A:middle L:90% , that apparently means that everybody turns into a pumpkin 1113 00:54:38.670 --> 00:54:42.380 A:middle L:90% with cars it for, um, which is very 1114 00:54:42.380 --> 00:54:44.820 A:middle L:90% new to me as a thing. My school was 1115 00:54:44.820 --> 00:54:58.550 A:middle L:90% not very football headaches. Okay. Mhm Oh, 1116 00:54:59.940 --> A:middle L:90% mm.