The reliability of replications

Autor(en)

Nate Breznau, Eike Mark Rinke, Alexander Wuttke, Muna Adem, Jule Adriaans, Esra Akdeniz, Amalia Alvarez-Benjumea, Henrik K Andersen, Daniel Auer, Flavio Azevedo, Oke Bahnsen, Ling Bai, Dave Balzer, Paul C Bauer, Gerrit Bauer, Markus Baumann, Sharon Baute, Verena Benoit, Julian Bernauer, Carl Berning, Anna Berthold, Felix S Bethke, Thomas Biegert, Katharina Blinzler, Johannes N Blumenberg, Licia Bobzien, Andrea Bohman, Thijs Bol, Amie Bostic, Zuzanna Brzozowska, Katharina Burgdorf, Kaspar Burger, Kathrin Busch, Juan-Carlos Castillo, Nathan Chan, Pablo Christmann, Roxanne Connelly, Christian S Czymara, Elena Damian, Eline A de Rooij, Alejandro Ecker, Achim Edelmann, Christina Eder, Maureen A Eger, Simon Ellerbrock, Anna Forke, Andrea Forster, Danilo Freire, Chris Gaasendam, Konstantin Gavras, Vernon Gayle, Theresa Gessler, Timo Gnambs, Amélie Godefroidt, Max Grömping, Martin Groß, Stefan Gruber, Tobias Gummer, Andreas Hadjar, Verena Halbherr, Jan Paul Heisig, Sebastian Hellmeier, Stefanie Heyne, Magdalena Hirsch, Mikael Hjerm, Oshrat Hochman, Jan H Höffler, Andreas Hövermann, Sophia Hunger, Christian Hunkler, Nora Huth-Stöckle, Zsófia S Ignácz, Sabine Israel, Laura Jacobs, Jannes Jacobsen, Bastian Jaeger, Sebastian Jungkunz, Nils Jungmann, Jennifer Kanjana, Mathias Kauff, Salman Khan, Sayak Khatua, Manuel Kleinert, Julia Klinger, Jan-Philipp Kolb, Marta Kołczyńska, John Kuk, Katharina Kunißen, Dafina Kurti Sinatra, Alexander Langenkamp, Robin C Lee, Philipp M Lersch, David Liu, Lea-Maria Löbel, Philipp Lutscher, Matthias Mader, Joan E Madia, Natalia Malancu, Luis Maldonado, Helge Marahrens, Nicole Martin, Paul Martinez, Jochen Mayerl, Oscar J Mayorga, Robert McDonnell, Patricia McManus, Kyle McWagner, Cecil Meeusen, Daniel Meierrieks, Jonathan Mellon, Friedolin Merhout, Samuel Merk, Daniel Meyer, Leticia Micheli, Jonathan Mijs, Cristóbal Moya, Marcel Neunhoeffer, Daniel Nüst, Olav Nygård, Fabian Ochsenfeld, Gunnar Otte, Anna Pechenkina, Mark Pickup, Christopher Prosser, Louis Raes, Kevin Ralston, Miguel Ramos, Frank Reichert, Arne Roets, Jonathan Rogers, Guido Ropers, Robin Samuel, Gregor Sand, Constanza Sanhueza Petrarca, Ariela Schachter, Merlin Schaeffer, David Schieferdecker, Elmar Schlueter, Katja Schmidt, Regine Schmidt, Alexander Schmidt-Catran, Claudia Schmiedeberg, Jürgen Schneider, Martijn Schoonvelde, Julia Schulte-Cloos, Sandy Schumann, Reinhard Schunck, Julian Seuring, Henning Silber, Willem Sleegers, Nico Sonntag, Alexander Staudt, Nadia Steiber, Nils D Steiner, Sebastian Sternberg, Dieter Stiers, Dragana Stojmenovska, Nora Storz, Erich Striessnig, Anne-Kathrin Stroppe, Jordan W Suchow, Janna Teltemann, Andrey Tibajev, Brian Tung, Giacomo Vagni, Jasper Van Assche, Meta van der Linden, Jolanda van der Noll, Arno Van Hootegem, Stefan Vogtenhuber, Bogdan Voicu, Fieke Wagemans, Nadja Wehl, Hannah Werner, Brenton M Wiernik, Fabian Winter, Christof Wolf, Cary Wu, Yuki Yamada, Björn Zakula, Nan Zhang, Conrad Ziller, Stefan Zins, Tomasz Żółtak, Hung H V Nguyen

Abstrakt

This study investigates researcher variability in computational reproduction, an activity for which it is least expected. Eighty-five independent teams attempted numerical replication of results from an original study of policy preferences and immigration. Reproduction teams were randomly grouped into a ‘transparent group’ receiving original study and code or ‘opaque group’ receiving only a method and results description and no code. The transparent group mostly verified original results (95.7% same sign and p-value cutoff), while the opaque group had less success (89.3%). Second-decimal place exact numerical reproductions were less common (76.9 and 48.1%). Qualitative investigation of the workflows revealed many causes of error, including mistakes and procedural variations. When curating mistakes, we still find that only the transparent group was reliably successful. Our findings imply a need for transparency, but also more. Institutional checks and less subjective difficulty for researchers ‘doing reproduction’ would help, implying a need for better training. We also urge increased awareness of complexity in the research process and in ‘push button’ replications.

Organisation(en)

Internationale Beziehungen, Institut für Staatswissenschaft, Institut für Germanistik, Institut für Lehrer*innenbildung, Dekanat der Fakultät für Psychologie, Core Facility Botanischer Garten, Zentraler Informatikdienst, Institut für Soziologie, Institut für Demografie

Externe Organisation(en)

Organization and Program Planning, German Institute for Adult Education-Leibniz Center of Lifelong Learning, Bonn 53175, Germany., University of Leeds, Ludwig-Maximilians-Universität München, University of Maryland, College Park, Universität Bielefeld, Marmara University, The Institute of Public Goods and Policies (IPP), Centro de Ciencias Humanas y Sociales-Consejo Superior de Investigaciones Científicas, Madrid 28003, Spain., Technische Universität Chemnitz, Collegio Carlo Alberto, Utrecht University, Universität Mannheim, Universität Bremen, Johannes Gutenberg-Universität Mainz, GESIS – Leibniz-Institut für Sozialwissenschaften, Universität Konstanz, Otto-Friedrich-Universität Bamberg, Leibniz-Institut Hessische Stiftung Friedens- und Konfliktforschung, London School of Economics and Political Science, Universität Potsdam, Umeå University, University of Amsterdam (UvA), University of Texas Rio Grande Valley, Österreichische Akademie der Wissenschaften (ÖAW), Gesundheit Österreich GmbH, University College London, Independent researcher, Universidad de Chile, Loyola Marymount University, University of Edinburgh, Johann Wolfgang Goethe-Universität Frankfurt am Main, Sciensano, Simon Fraser University, Universität Heidelberg, Fondation Nationale des Sciences Politiques, Center for Advanced Study in the Behavioral Sciences, Stanford, CA 94305, USA., Emory University, Government of Flanders, Europa-Universität Viadrina Frankfurt (Oder), Leibnitz-Institut für Bildungsverläufe, Katholieke Universiteit Leuven, Griffith University, Eberhard Karls Universität Tübingen, SHARE BERLIN Institute, Université de Fribourg, Association for Doctoral Studies Baden-Wuerttemberg 70174, Germany., Wissenschaftszentrum Berlin für Sozialforschung gGmbH, Universidad Espíritu Santo, Wirtschafts- und Sozialwissenschaftliches Institut in der Hans-Böckler-Stiftung (WSI), Humboldt-Universität zu Berlin, Bergische Universität Wuppertal, University of Antwerp, Deutsches Zentrum für Integrations- und Migrationsforschung E.V., Tilburg University, Rheinische Friedrich-Wilhelms-Universität Bonn, Medical School Hamburg, University of Chicago, Oregon State University, Justus-Liebig-Universität Gießen (JLU), Statistisches Bundesamt, Polish Academy of Sciences, Michigan State University, Universität zu Köln, Princeton University, Deutsches Institut für Wirtschaftsforschung (DIW), University of Oslo, Universität Witten/Herdecke, University of Oxford, Université de Neuchâtel, Pontificia Universidad Católica de Chile, Georgetown University, University of Manchester, Western Governors University, Director of Data for Freedom, Equity Research Cooperative, 19107., Indiana University Bloomington, University of Wisconsin, Milwaukee, United States Military Academy, University of Copenhagen, Pädagogische Hochschule Karlsruhe, BBSR - Bundesinstitut für Bau-, Stadt- und Raumforschung, Leiden University, Boston University, Technische Universität Dresden, Linköping University, Administrative Headquarters, Max Planck Society, Munich 80539, Germany., Utah State University (USU), University of Birmingham, Ghent University , University of California, Los Angeles, University of Luxembourg, Australian National University, Washington University in St. Louis, Freie Universität Berlin (FU), DIPF | Leibniz-Institut für Bildungsforschung und Bildungsinformation, University of Groningen, Philipps Universität Marburg, University of Michigan, Expert Council on Intergration and Migration, Berlin 10178, Germany., Polish Academy of Sciences (PAS), Stevens Institute of Technology, Stiftung Universität Hildesheim, Uppsala University, North-West University, FernUniversität in Hagen, IHS - Institut für Höhere Studien und wissenschaftliche Forschung, Lucian Blaga University of Sibiu, The Netherlands Institute for Social Research, Universität Zürich (UZH), Max-Plank Institut zur Erforschung von Gemeinschaftsgütern, York University, Kyushu University, Universität Duisburg-Essen, Bundesagentur für Arbeit

Journal

Royal Society Open Science

Band

12

Seiten

241038

ISSN

2054-5703

DOI

https://doi.org/10.1098/rsos.241038

Publikationsdatum

03-2025

Peer-reviewed

Ja

ÖFOS 2012

504007 Empirische Sozialforschung

Sustainable Development Goals

SDG 10 – Weniger Ungleichheiten

Link zum Portal

https://ucrisportal.univie.ac.at/de/publications/30d1f05a-eaea-425a-a42a-be4ff7f24669