Same data, different analysts: variation in effect sizes due to analytical decisions in ecology and evolutionary biology

Simple item page
dc.contributor.authorGould, Ellioten
dc.contributor.authorFraser, Hannah S.en
dc.contributor.authorParker, Timothy H.en
dc.contributor.authorNakagawa, Shinichien
dc.contributor.authorGriffith, Simon C.en
dc.contributor.authorVesk, Peter A.en
dc.contributor.authorFidler, Fionaen
dc.contributor.authorHamilton, Daniel G.en
dc.contributor.authorAbbey-Lee, Robin N.en
dc.contributor.authorAbbott, Jessica K.en
dc.contributor.authorAguirre, Luis A.en
dc.contributor.authorAlcaraz, Carlesen
dc.contributor.authorAloni, Irithen
dc.contributor.authorAltschul, Drewen
dc.contributor.authorArekar, Kunalen
dc.contributor.authorAtkins, Jeff W.en
dc.contributor.authorAtkinson, Joeen
dc.contributor.authorBaker, Christopher M.en
dc.contributor.authorBarrett, Meghanen
dc.contributor.authorBell, Kristianen
dc.contributor.authorBello, Suleiman K.en
dc.contributor.authorBeltrán, Ivánen
dc.contributor.authorBerauer, Bernd J.en
dc.contributor.authorBertram, Michael G.en
dc.contributor.authorBillman, Peter D.en
dc.contributor.authorBlake, Charlie K.en
dc.contributor.authorBlake, Shannonen
dc.contributor.authorBliard, Louisen
dc.contributor.authorBonisoli-Alquati, Andreaen
dc.contributor.authorBonnet, Timothéeen
dc.contributor.authorBordes, Camille N. M.en
dc.contributor.authorBose, Aneesh P. H.en
dc.contributor.authorBotterill-James, Thomasen
dc.contributor.authorBoyd, Melissa A.en
dc.contributor.authorBoyle, Sarah A.en
dc.contributor.authorBradfer-Lawrence, Tomen
dc.contributor.authorBradham, Jenniferen
dc.contributor.authorBrand, Jack A.en
dc.contributor.authorBrengdahl, Martin I.en
dc.contributor.authorBulla, Martinen
dc.contributor.authorBussière, Lucen
dc.contributor.authorCamerlenghi, Ettoreen
dc.contributor.authorCampbell, Sara E.en
dc.contributor.authorCampos, Leonardo L. F.en
dc.contributor.authorCaravaggi, Anthonyen
dc.contributor.authorCardoso, Pedroen
dc.contributor.authorCarroll, Charles J. W.en
dc.contributor.authorCatanach, Therese A.en
dc.contributor.authorChen, Xuanen
dc.contributor.authorChik, Heung Y. J.en
dc.contributor.authorChoy, Emily S.en
dc.contributor.authorChristie, Alec P.en
dc.contributor.authorChuang, Angelaen
dc.contributor.authorChunco, Amanda J.en
dc.contributor.authorClark, Bethany L.en
dc.contributor.authorContina, Andreaen
dc.contributor.authorCovernton, Garth A.en
dc.contributor.authorCox, Murray P.en
dc.contributor.authorCressman, Kimberly A.en
dc.contributor.authorCrotti, Marcoen
dc.contributor.authorCrouch, Connor D.en
dc.contributor.authorD’Amelio, Pietro B.en
dc.contributor.authorde Sousa, Alexandra A.en
dc.contributor.authorDöbert, Timm F.en
dc.contributor.authorDobler, Ralphen
dc.contributor.authorDobson, Adam J.en
dc.contributor.authorDoherty, Tim S.en
dc.contributor.authorDrobniak, Szymon M.en
dc.contributor.authorDuffy, Alexandra G.en
dc.contributor.authorDuncan, Alison B.en
dc.contributor.authorDunn, Robert P.en
dc.contributor.authorDunning, Jamieen
dc.contributor.authorDutta, Trishnaen
dc.contributor.authorEberhart-Hertel, Lukeen
dc.contributor.authorElmore, Jared A.en
dc.contributor.authorElsherif, Mahmoud M.en
dc.contributor.authorEnglish, Holly M.en
dc.contributor.authorEnsminger, David C.en
dc.contributor.authorErnst, Ulrich R.en
dc.contributor.authorFerguson, Stephen M.en
dc.contributor.authorFernandez-Juricic, Estebanen
dc.contributor.authorFerreira-Arruda, Thalitaen
dc.contributor.authorFieberg, Johnen
dc.contributor.authorFinch, Elizabeth A.en
dc.contributor.authorFiorenza, Evan A.en
dc.contributor.authorFisher, David N.en
dc.contributor.authorFontaine, Amélieen
dc.contributor.authorForstmeier, Wolfgangen
dc.contributor.authorFourcade, Yoanen
dc.contributor.authorFrank, Graham S.en
dc.contributor.authorFreund, Cathryn A.en
dc.contributor.authorFuentes-Lillo, Eduardoen
dc.contributor.authorGandy, Sara L.en
dc.contributor.authorGannon, Dustin G.en
dc.contributor.authorGarcía-Cervigón, Ana I.en
dc.contributor.authorGarretson, Alexis C.en
dc.contributor.authorGe, Xuezhenen
dc.contributor.authorGeary, William L.en
dc.contributor.authorGéron, Charlyen
dc.contributor.authorGilles, Marcen
dc.contributor.authorGirndt, Antjeen
dc.contributor.authorGliksman, Danielen
dc.contributor.authorGoldspiel, Harrison B.en
dc.contributor.authorGomes, Dylan G. E.en
dc.contributor.authorGood, Megan K.en
dc.contributor.authorGoslee, Sarah C.en
dc.contributor.authorGosnell, J. S.en
dc.contributor.authorGrames, Eliza M.en
dc.contributor.authorGratton, Paoloen
dc.contributor.authorGrebe, Nicholas M.en
dc.contributor.authorGreenler, Skye M.en
dc.contributor.authorGriffioen, Maaikeen
dc.contributor.authorGriffith, Daniel M.en
dc.contributor.authorGriffith, Frances J.en
dc.contributor.authorGrossman, Jake J.en
dc.contributor.authorGüncan, Alien
dc.contributor.authorHaesen, Stefen
dc.contributor.authorHagan, James G.en
dc.contributor.authorHager, Heather A.en
dc.contributor.authorHarris, Jonathan P.en
dc.contributor.authorHarrison, Natasha D.en
dc.contributor.authorHasnain, Sarah S.en
dc.contributor.authorHavird, Justin C.en
dc.contributor.authorHeaton, Andrew J.en
dc.contributor.authorHerrera-Chaustre, María L.en
dc.contributor.authorHoward, Tanner J.en
dc.contributor.authorHsu, Bin-Yanen
dc.contributor.authorIannarilli, Fabiolaen
dc.contributor.authorIranzo, Esperanza C.en
dc.contributor.authorIverson, Erik N. K.en
dc.contributor.authorJimoh, Saheed O.en
dc.contributor.authorJohnson, Douglas H.en
dc.contributor.authorJohnsson, Martinen
dc.contributor.authorJorna, Jesseen
dc.contributor.authorJucker, Tommasoen
dc.contributor.authorJung, Martinen
dc.contributor.authorKačergytė, Inetaen
dc.contributor.authorKaltz, Oliveren
dc.contributor.authorKe, Alisonen
dc.contributor.authorKelly, Clint D.en
dc.contributor.authorKeogan, Katharineen
dc.contributor.authorKeppeler, Friedrich W.en
dc.contributor.authorKillion, Alexander K.en
dc.contributor.authorKim, Dongminen
dc.contributor.authorKochan, David P.en
dc.contributor.authorKorsten, Peteren
dc.contributor.authorKothari, Shanen
dc.contributor.authorKuppler, Jonasen
dc.contributor.authorKusch, Jillian M.en
dc.contributor.authorLagisz, Malgorzataen
dc.contributor.authorLalla, Kristen M.en
dc.contributor.authorLarkin, Daniel J.en
dc.contributor.authorLarson, Courtney L.en
dc.contributor.authorLauck, Katherine S.en
dc.contributor.authorLauterbur, M. E.en
dc.contributor.authorLaw, Alanen
dc.contributor.authorLéandri-Breton, Don-Jeanen
dc.contributor.authorLembrechts, Jonas J.en
dc.contributor.authorL’Herpiniere, Kiaraen
dc.contributor.authorLievens, Eva J. P.en
dc.contributor.authorde Lima, Daniela O.en
dc.contributor.authorLindsay, Shaneen
dc.contributor.authorLuquet, Martinen
dc.contributor.authorMacLeod, Rossen
dc.contributor.authorMacphie, Kirsty H.en
dc.contributor.authorMagellan, Kiten
dc.contributor.authorMair, Magdalena M.en
dc.contributor.authorMalm, Lisa E.en
dc.contributor.authorMammola, Stefanoen
dc.contributor.authorMandeville, Caitlin P.en
dc.contributor.authorManhart, Michaelen
dc.contributor.authorManrique-Garzon, Laura M.en
dc.contributor.authorMäntylä, Elinaen
dc.contributor.authorMarchand, Philippeen
dc.contributor.authorMarshall, Benjamin M.en
dc.contributor.authorMartin, Charles A.en
dc.contributor.authorMartin, Dominic A.en
dc.contributor.authorMartin, Jake M.en
dc.contributor.authorMartinig, April R.en
dc.contributor.authorMcCallum, Erin S.en
dc.contributor.authorMcCauley, Marken
dc.contributor.authorMcNew, Sabrina M.en
dc.contributor.authorMeiners, Scott J.en
dc.contributor.authorMerkling, Thomasen
dc.contributor.authorMichelangeli, Marcusen
dc.contributor.authorMoiron, Mariaen
dc.contributor.authorMoreira, Brunoen
dc.contributor.authorMortensen, Jenniferen
dc.contributor.authorMos, Benjaminen
dc.contributor.authorMuraina, Taofeek O.en
dc.contributor.authorMurphy, Penelope W.en
dc.contributor.authorNelli, Lucaen
dc.contributor.authorNiemelä, Petrien
dc.contributor.authorNightingale, Joshen
dc.contributor.authorNilsonne, Gustaven
dc.contributor.authorNolazco, Sergioen
dc.contributor.authorNooten, Sabine S.en
dc.contributor.authorNovotny, Jessie L.en
dc.contributor.authorOlin, Agnes B.en
dc.contributor.authorOrgan, Chris L.en
dc.contributor.authorOstevik, Kate L.en
dc.contributor.authorPalacio, Facundo X.en
dc.contributor.authorPaquet, Matthieuen
dc.contributor.authorParker, Darren J.en
dc.contributor.authorPascall, David J.en
dc.contributor.authorPasquarella, Valerie J.en
dc.contributor.authorPaterson, John H.en
dc.contributor.authorPayo-Payo, Anaen
dc.contributor.authorPedersen, Karen M.en
dc.contributor.authorPerez, Grégoireen
dc.contributor.authorPerry, Kayla I.en
dc.contributor.authorPottier, Patriceen
dc.contributor.authorProulx, Michael J.en
dc.contributor.authorProulx, Raphaëlen
dc.contributor.authorPruett, Jessica L.en
dc.contributor.authorRamananjato, Veronarindraen
dc.contributor.authorRandimbiarison, Finaritra T.en
dc.contributor.authorRazafindratsima, Onja H.en
dc.contributor.authorRennison, Diana J.en
dc.contributor.authorRiva, Federicoen
dc.contributor.authorRiyahi, Sepanden
dc.contributor.authorRoast, Michael J.en
dc.contributor.authorRocha, Felipe P.en
dc.contributor.authorRoche, Dominique G.en
dc.contributor.authorRomán-Palacios, Cristianen
dc.contributor.authorRosenberg, Michael S.en
dc.contributor.authorRoss, Jessicaen
dc.contributor.authorRowland, Freya E.en
dc.contributor.authorRugemalila, Deusdedithen
dc.contributor.authorRussell, Avery L.en
dc.contributor.authorRuuskanen, Suvien
dc.contributor.authorSaccone, Patricken
dc.contributor.authorSadeh, Asafen
dc.contributor.authorSalazar, Stephen M.en
dc.contributor.authorSales, Krisen
dc.contributor.authorSalmón, Pabloen
dc.contributor.authorSánchez-Tójar, Alfredoen
dc.contributor.authorSantos, Leticia P.en
dc.contributor.authorSantostefano, Francescaen
dc.contributor.authorSchilling, Hayden T.en
dc.contributor.authorSchmidt, Marcusen
dc.contributor.authorSchmoll, Timen
dc.contributor.authorSchneider, Adam C.en
dc.contributor.authorSchrock, Allie E.en
dc.contributor.authorSchroeder, Juliaen
dc.contributor.authorSchtickzelle, Nicolasen
dc.contributor.authorSchultz, Nick L.en
dc.contributor.authorScott, Drew A.en
dc.contributor.authorScroggie, Michael P.en
dc.contributor.authorShapiro, Julie T.en
dc.contributor.authorSharma, Nitikaen
dc.contributor.authorShearer, Caroline L.en
dc.contributor.authorSimón, Diegoen
dc.contributor.authorSitvarin, Michael I.en
dc.contributor.authorSkupien, Fabrício L.en
dc.contributor.authorSlinn, Heather L.en
dc.contributor.authorSmith, Grania P.en
dc.contributor.authorSmith, Jeremy A.en
dc.contributor.authorSollmann, Rahelen
dc.contributor.authorWhitney, Kaitlin S.en
dc.contributor.authorStill, Shannon M.en
dc.contributor.authorStuber, Erica F.en
dc.contributor.authorSutton, Guy F.en
dc.contributor.authorSwallow, Benen
dc.contributor.authorTaff, Conor C.en
dc.contributor.authorTakola, Elinaen
dc.contributor.authorTanentzap, Andrew J.en
dc.contributor.authorTarjuelo, Rocíoen
dc.contributor.authorTelford, Richard J.en
dc.contributor.authorThawley, Christopher J.en
dc.contributor.authorThierry, Hugoen
dc.contributor.authorThomson, Jacquelineen
dc.contributor.authorTidau, Svenjaen
dc.contributor.authorTompkins, Emily M.en
dc.contributor.authorTortorelli, Claire M.en
dc.contributor.authorTrlica, Andrewen
dc.contributor.authorTurnell, Biz R.en
dc.contributor.authorUrban, Laraen
dc.contributor.authorVan de Vondel, Stijnen
dc.contributor.authorvan der Wal, Jessica E. M.en
dc.contributor.authorVan Eeckhoven, Jensen
dc.contributor.authorvan Oordt, Francisen
dc.contributor.authorVanderwel, K. M.en
dc.contributor.authorVanderwel, Mark C.en
dc.contributor.authorVanderwolf, Karen J.en
dc.contributor.authorVélez, Julianaen
dc.contributor.authorVergara-Florez, Diana C.en
dc.contributor.authorVerrelli, Brian C.en
dc.contributor.authorVieira, Marcus V.en
dc.contributor.authorVillamil, Noraen
dc.contributor.authorVitali, Valerioen
dc.contributor.authorVollering, Julienen
dc.contributor.authorWalker, Jeffreyen
dc.contributor.authorWalker, Xanthe J.en
dc.contributor.authorWalter, Jonathan A.en
dc.contributor.authorWaryszak, Pawelen
dc.contributor.authorWeaver, Ryan J.en
dc.contributor.authorWedegärtner, Ronja E. M.en
dc.contributor.authorWeller, Daniel L.en
dc.contributor.authorWhelan, Shannonen
dc.contributor.authorWhite, Rachel L.en
dc.contributor.authorWolfson, David W.en
dc.contributor.authorWood, Andrewen
dc.contributor.authorYanco, Scott W.en
dc.contributor.authorYen, Jian D. L.en
dc.contributor.authorYoungflesh, Caseyen
dc.contributor.authorZilio, Giacomoen
dc.contributor.authorZimmer, Cédricen
dc.contributor.authorZimmerman, Gregory M.en
dc.contributor.authorZitomer, Rachel A.en
dc.date.accessioned2025-02-10T19:45:10Zen
dc.date.available2025-02-10T19:45:10Zen
dc.date.issued2025-02-06en
dc.date.updated2025-02-09T04:18:37Zen
dc.description.abstractAbstract Although variation in effect sizes and predicted values among studies of similar phenomena is inevitable, such variation far exceeds what might be produced by sampling error alone. One possible explanation for variation among results is differences among researchers in the decisions they make regarding statistical analyses. A growing array of studies has explored this analytical variability in different fields and has found substantial variability among results despite analysts having the same data and research question. Many of these studies have been in the social sciences, but one small “many analyst” study found similar variability in ecology. We expanded the scope of this prior work by implementing a large-scale empirical exploration of the variation in effect sizes and model predictions generated by the analytical decisions of different researchers in ecology and evolutionary biology. We used two unpublished datasets, one from evolutionary ecology (blue tit, Cyanistes caeruleus, to compare sibling number and nestling growth) and one from conservation ecology (Eucalyptus, to compare grass cover and tree seedling recruitment). The project leaders recruited 174 analyst teams, comprising 246 analysts, to investigate the answers to prespecified research questions. Analyses conducted by these teams yielded 141 usable effects (compatible with our meta-analyses and with all necessary information provided) for the blue tit dataset, and 85 usable effects for the Eucalyptus dataset. We found substantial heterogeneity among results for both datasets, although the patterns of variation differed between them. For the blue tit analyses, the average effect was convincingly negative, with less growth for nestlings living with more siblings, but there was near continuous variation in effect size from large negative effects to effects near zero, and even effects crossing the traditional threshold of statistical significance in the opposite direction. In contrast, the average relationship between grass cover and Eucalyptus seedling number was only slightly negative and not convincingly different from zero, and most effects ranged from weakly negative to weakly positive, with about a third of effects crossing the traditional threshold of significance in one direction or the other. However, there were also several striking outliers in the Eucalyptus dataset, with effects far from zero. For both datasets, we found substantial variation in the variable selection and random effects structures among analyses, as well as in the ratings of the analytical methods by peer reviewers, but we found no strong relationship between any of these and deviation from the meta-analytic mean. In other words, analyses with results that were far from the mean were no more or less likely to have dissimilar variable sets, use random effects in their models, or receive poor peer reviews than those analyses that found results that were close to the mean. The existence of substantial variability among analysis outcomes raises important questions about how ecologists and evolutionary biologists should interpret published results, and how they should conduct analyses in the future.en
dc.description.versionPublished versionen
dc.format.mimetypeapplication/pdfen
dc.identifier.citationBMC Biology. 2025 Feb 06;23(1):35en
dc.identifier.doihttps://doi.org/10.1186/s12915-024-02101-xen
dc.identifier.urihttps://hdl.handle.net/10919/124561en
dc.language.isoenen
dc.rightsCreative Commons Attribution 4.0 Internationalen
dc.rights.holderThe Author(s)en
dc.rights.urihttp://creativecommons.org/licenses/by/4.0/en
dc.titleSame data, different analysts: variation in effect sizes due to analytical decisions in ecology and evolutionary biologyen
dc.title.serialBMC Biologyen
dc.typeArticle - Refereeden
dc.type.dcmitypeTexten

Files

Original bundle
Now showing 1 - 1 of 1
Thumbnail Image
Name:
12915_2024_Article_2101.pdf
Size:
2.61 MB
Format:
Adobe Portable Document Format
Description:
Published version
Download
License bundle
Now showing 1 - 1 of 1
Name:
license.txt
Size:
1.5 KB
Format:
Item-specific license agreed upon to submission
Description:
Download

Collections

Journal Articles, BioMed Central and SpringerOpen
Scholarly Works, Food Science and Technology