Item request has been placed! ×
Item request cannot be made. ×
loading  Processing Request

Same data, different conclusions: Radical dispersion in empirical results when independent analysts operationalize and test the same hypothesis

Item request has been placed! ×
Item request cannot be made. ×
loading   Processing Request
  • معلومة اضافية
    • بيانات النشر:
      Linköpings universitet, Handikappvetenskap
      Linköpings universitet, Filosofiska fakulteten
      Linköpings universitet, Institutet för handikappvetenskap (IHV)
      ESMT Berlin, Germany
      Univ Zurich, Switzerland
      Tilburg Univ, Netherlands
      Tilburg Univ, Netherlands; Univ Utrecht, Netherlands
      Univ Washington, WA 98195 USA
      ESMT Berlin, Germany; Indiana Univ, IN USA
      ESMT Berlin, Germany; Humboldt Univ, Germany
      INSEAD, France
      Univ Southern Calif, CA 90007 USA
      Imperial Coll Business Sch, England
      Indian Sch Business, India
      Univ Sussex, England
      Univ Haifa, Israel
      INSEAD, Singapore
      Waseda Univ, Japan
      Stockholm Sch Econ, Sweden
      Katholieke Univ Leuven, Belgium
      IBM Corp, Germany
      Wake Forest Univ, NC 27101 USA
      Univ Texas Austin, TX 78712 USA
      SUNY Stony Brook, NY 11794 USA
      Univ Nairobi, Kenya
      Univ Groningen, Netherlands
      Univ Calif Berkeley, CA 94720 USA
      MIT, MA 02139 USA
      Univ Kentucky, KY USA
      Calif State Univ Dominguez Hills, CA 90747 USA
      Prague Coll Psychosocial Studies, Czech Republic
      Copenhagen Business Sch, Denmark
      Univ Utah, UT USA
      Univ Pittsburgh, PA USA
      Univ Glasgow, Scotland
      Texas A&M Univ, TX USA
      IIT, IL 60616 USA
      Eskisehir Osmangazi Univ, Turkey
      Imperial Coll London, England
      Northwestern Univ, IL USA
      Edinburgh Napier Univ, Scotland
      Univ Navarra, Spain
      Univ Michigan, MI 48109 USA
      Yale Univ, CT USA
      Univ Oberta Catalunya, Spain
      Michigan State Univ, MI 48824 USA
      Univ Toronto, Canada
      Univ Nove Julho, Brazil
      Lucerne Univ Appl Sci & Arts, Switzerland
      Univ Lancaster, England
      Univ Fed Santa Catarina, Brazil
      Indiana Univ, IN USA
      Swiss Fed Inst Technol, Switzerland
      Pacific Lutheran Univ, WA 98447 USA
      Rutgers State Univ, NJ USA
      Leibniz Inst Educ Trajectories, Germany; Johannes Kepler Univ Linz, Austria
      Okayama Univ, Japan
      Univ Notre Dame, IN 46556 USA
      Univ Bath, England; Univ Essex, England
      Univ Chicago, IL USA
      Xiamen Univ, Peoples R China
      Elmhurst Coll, NY USA
      Columbia Univ, NY USA
      Univ Washington, WA 98195 USA; Indiana Univ, IN USA
      Univ Oxford, England; Coventry Univ, England
      Harvard Univ, MA 02138 USA
      Univ Vienna, Austria
      UCL, England
      York Univ, Canada
      Univ Utrecht, Netherlands; Katholieke Univ Leuven, Belgium
      Univ Ljubljana, Slovenia
      Nanyang Technol Univ, Singapore
      Univ Colorado, CO 80907 USA
      Queen Mary Univ London, England
      Univ Copenhagen, Denmark
      Singapore Management Univ, Singapore
      McD Tech Labs, CA USA
      Univ Nottingham, England
      Novo Nordisk, Denmark
      Univ Idaho, ID 83843 USA
      Univ St Thomas, MN USA
      Heidelberg Univ, Germany
      Univ Edinburgh, Scotland
      Eli Lilly, IN USA
      Univ Nis, Serbia
      George Washington Univ, DC USA
      Karolinska Inst, Sweden; Stockholm Univ, Sweden
      Univ Oxford, England
      Univ Maryland Baltimore Cty, MD 21228 USA
      Univ Western Australia, Australia
      Open Univ Cyprus, Cyprus
      Natl Res Univ, Russia
      Univ Calif Santa Barbara, CA 93106 USA
      McGill Univ, Canada
      NIMH, MD 20892 USA
      Univ Sheffield, England
      Charles Univ Prague, Czech Republic; Univ Presov, Slovakia
      Univ Tennessee, TN USA
      Univ Bern, Switzerland; Max Planck Inst Human Dev, Germany
      Washington Univ, MO 63110 USA
      Dalhousie Univ, Canada
      Univ Illinois, IL USA
      Colorado State Univ, CO 80523 USA
      CSU, CA USA
      Rasa Technol, CA USA
      Oberlin Coll, OH 44074 USA
      Stanford Univ, CA 94305 USA
      Univ Granada, Spain
      Radboud Univ Nijmegen, Netherlands
      Univ Hawaii, HI 96822 USA
      Microsoft, WA USA; Rice Univ, TX USA
      Univ Amsterdam, Netherlands
      Coll Charleston, SC 29401 USA
      Univ Calif Los Angeles, CA USA
      Univ Texas Dallas, TX 75083 USA
      KTH Royal Inst Technol, Sweden; SciLifeLab, Sweden
      ACADEMIC PRESS INC ELSEVIER SCIENCE
      Dartmouth Coll, NH 03755 USA
    • الموضوع:
      2021
    • Collection:
      Linköping University Electronic Press (LiU E-Press)
    • نبذة مختصرة :
      In this crowdsourced initiative, independent analysts used the same dataset to test two hypotheses regarding the effects of scientists gender and professional status on verbosity during group meetings. Not only the analytic approach but also the operationalizations of key variables were left unconstrained and up to individual analysts. For instance, analysts could choose to operationalize status as job title, institutional ranking, citation counts, or some combination. To maximize transparency regarding the process by which analytic choices are made, the analysts used a platform we developed called DataExplained to justify both preferred and rejected analytic paths in real time. Analyses lacking sufficient detail, reproducible code, or with statistical errors were excluded, resulting in 29 analyses in the final sample. Researchers reported radically different analyses and dispersed empirical outcomes, in a number of cases obtaining significant effects in opposite directions for the same research question. A Boba multiverse analysis demonstrates that decisions about how to operationalize variables explain variability in outcomes above and beyond statistical choices (e.g., covariates). Subjective researcher decisions play a critical role in driving the reported empirical results, underscoring the need for open data, systematic robustness checks, and transparency regarding both analytic paths taken and not taken. Implications for orga-nizations and leaders, whose decision making relies in part on scientific findings, consulting reports, and internal analyses by data scientists, are discussed. ; Funding Agencies|INSEAD; Swiss National Science FoundationSwiss National Science Foundation (SNSF)European Commission [143411]
    • File Description:
      application/pdf
    • Relation:
      Organizational Behavior and Human Decision Processes, 0749-5978, 2021, 165, s. 228-249; orcid:0000-0002-0446-0827; http://urn.kb.se/resolve?urn=urn:nbn:se:liu:diva-178471; ISI:000674429500016
    • الرقم المعرف:
      10.1016/j.obhdp.2021.02.003
    • الدخول الالكتروني :
      http://urn.kb.se/resolve?urn=urn:nbn:se:liu:diva-178471
      https://doi.org/10.1016/j.obhdp.2021.02.003
    • Rights:
      info:eu-repo/semantics/openAccess
    • الرقم المعرف:
      edsbas.C6C03EEF