The reliability of replications

Author(s)

Nate Breznau, Eike Mark Rinke, Alexander Wuttke, Muna Adem, Jule Adriaans, Esra Akdeniz, Amalia Alvarez-Benjumea, Henrik K Andersen, Daniel Auer, Flavio Azevedo, Oke Bahnsen, Ling Bai, Dave Balzer, Paul C Bauer, Gerrit Bauer, Markus Baumann, Sharon Baute, Verena Benoit, Julian Bernauer, Carl Berning, Anna Berthold, Felix S Bethke, Thomas Biegert, Katharina Blinzler, Johannes N Blumenberg, Licia Bobzien, Andrea Bohman, Thijs Bol, Amie Bostic, Zuzanna Brzozowska, Katharina Burgdorf, Kaspar Burger, Kathrin Busch, Juan-Carlos Castillo, Nathan Chan, Pablo Christmann, Roxanne Connelly, Christian S Czymara, Elena Damian, Eline A de Rooij, Alejandro Ecker, Achim Edelmann, Christina Eder, Maureen A Eger, Simon Ellerbrock, Anna Forke, Andrea Forster, Danilo Freire, Chris Gaasendam, Konstantin Gavras, Vernon Gayle, Theresa Gessler, Timo Gnambs, Amélie Godefroidt, Max Grömping, Martin Groß, Stefan Gruber, Tobias Gummer, Andreas Hadjar, Verena Halbherr, Jan Paul Heisig, Sebastian Hellmeier, Stefanie Heyne, Magdalena Hirsch, Mikael Hjerm, Oshrat Hochman, Jan H Höffler, Andreas Hövermann, Sophia Hunger, Christian Hunkler, Nora Huth-Stöckle, Zsófia S Ignácz, Sabine Israel, Laura Jacobs, Jannes Jacobsen, Bastian Jaeger, Sebastian Jungkunz, Nils Jungmann, Jennifer Kanjana, Mathias Kauff, Salman Khan, Sayak Khatua, Manuel Kleinert, Julia Klinger, Jan-Philipp Kolb, Marta Kołczyńska, John Kuk, Katharina Kunißen, Dafina Kurti Sinatra, Alexander Langenkamp, Robin C Lee, Philipp M Lersch, David Liu, Lea-Maria Löbel, Philipp Lutscher, Matthias Mader, Joan E Madia, Natalia Malancu, Luis Maldonado, Helge Marahrens, Nicole Martin, Paul Martinez, Jochen Mayerl, Oscar J Mayorga, Robert McDonnell, Patricia McManus, Kyle McWagner, Cecil Meeusen, Daniel Meierrieks, Jonathan Mellon, Friedolin Merhout, Samuel Merk, Daniel Meyer, Leticia Micheli, Jonathan Mijs, Cristóbal Moya, Marcel Neunhoeffer, Daniel Nüst, Olav Nygård, Fabian Ochsenfeld, Gunnar Otte, Anna Pechenkina, Mark Pickup, Christopher Prosser, Louis Raes, Kevin Ralston, Miguel Ramos, Frank Reichert, Arne Roets, Jonathan Rogers, Guido Ropers, Robin Samuel, Gregor Sand, Constanza Sanhueza Petrarca, Ariela Schachter, Merlin Schaeffer, David Schieferdecker, Elmar Schlueter, Katja Schmidt, Regine Schmidt, Alexander Schmidt-Catran, Claudia Schmiedeberg, Jürgen Schneider, Martijn Schoonvelde, Julia Schulte-Cloos, Sandy Schumann, Reinhard Schunck, Julian Seuring, Henning Silber, Willem Sleegers, Nico Sonntag, Alexander Staudt, Nadia Steiber, Nils D Steiner, Sebastian Sternberg, Dieter Stiers, Dragana Stojmenovska, Nora Storz, Erich Striessnig, Anne-Kathrin Stroppe, Jordan W Suchow, Janna Teltemann, Andrey Tibajev, Brian Tung, Giacomo Vagni, Jasper Van Assche, Meta van der Linden, Jolanda van der Noll, Arno Van Hootegem, Stefan Vogtenhuber, Bogdan Voicu, Fieke Wagemans, Nadja Wehl, Hannah Werner, Brenton M Wiernik, Fabian Winter, Christof Wolf, Cary Wu, Yuki Yamada, Björn Zakula, Nan Zhang, Conrad Ziller, Stefan Zins, Tomasz Żółtak, Hung H V Nguyen

Abstract

This study investigates researcher variability in computational reproduction, an activity for which it is least expected. Eighty-five independent teams attempted numerical replication of results from an original study of policy preferences and immigration. Reproduction teams were randomly grouped into a 'transparent group' receiving original study and code or 'opaque group' receiving only a method and results description and no code. The transparent group mostly verified original results (95.7% same sign and p-value cutoff), while the opaque group had less success (89.3%). Second-decimal place exact numerical reproductions were less common (76.9 and 48.1%). Qualitative investigation of the workflows revealed many causes of error, including mistakes and procedural variations. When curating mistakes, we still find that only the transparent group was reliably successful. Our findings imply a need for transparency, but also more. Institutional checks and less subjective difficulty for researchers 'doing reproduction' would help, implying a need for better training. We also urge increased awareness of complexity in the research process and in 'push button' replications.

Organisation(s)

International Office, Department of Government, Department of German Studies, Department for Teacher Education, Dean's office of the Faculty of Psychology, Core Facility Botanical Garden, Vienna University Computer Center, Department of Sociology, Department of Demography

External organisation(s)

Organization and Program Planning, German Institute for Adult Education-Leibniz Center of Lifelong Learning, Bonn 53175, Germany., University of Leeds, Ludwig-Maximilians-Universität München, University of Maryland, College Park, Universität Bielefeld, Marmara University, The Institute of Public Goods and Policies (IPP), Centro de Ciencias Humanas y Sociales-Consejo Superior de Investigaciones Científicas, Madrid 28003, Spain., Technische Universität Chemnitz, Collegio Carlo Alberto, Utrecht University, Universität Mannheim, Universität Bremen, Johannes Gutenberg-Universität Mainz, GESIS – Leibniz-Institut für Sozialwissenschaften, Universität Konstanz, Otto-Friedrich-Universität Bamberg, Research Department on Intrastate Conflict, Peace Research Institute Frankfurt, 60329 Frankfurt, Germany., London School of Economics and Political Science, Universität Potsdam, Umeå University, University of Amsterdam (UvA), University of Texas Rio Grande Valley, Österreichische Akademie der Wissenschaften (ÖAW), Austrian National Public Health Institute, Competence Centre on Climate and Health, Vienna, Austria., University College London, Independent researcher, Universidad de Chile, Loyola Marymount University, University of Edinburgh, Johann Wolfgang Goethe-Universität Frankfurt am Main, Sciensano, Simon Fraser University, Scientific Software Center, Fondation Nationale des Sciences Politiques, Center for Advanced Study in the Behavioral Sciences, Stanford, CA 94305, USA., Emory University, Government of Flanders, Europa-Universität Viadrina Frankfurt (Oder), Educational Measurement, Leibniz Institute for Educational Trajectories, 96047 Bamberg, Germany., Katholieke Universiteit Leuven, Griffith University, Eberhard Karls Universität Tübingen, SHARE BERLIN Institute, Université de Fribourg, Association for Doctoral Studies Baden-Wuerttemberg 70174, Germany., Center for Civil Society Research, Universidad Espíritu Santo, Wirtschafts- und Sozialwissenschaftliches Institut in der Hans-Böckler-Stiftung (WSI), Humboldt-Universität zu Berlin, Bergische Universität Wuppertal, University of Antwerp, German Center for Integration and Migration Research (DeZIM), Tilburg University, Rheinische Friedrich-Wilhelms-Universität Bonn, Medical School Hamburg, University of Chicago, Oregon State University, Justus-Liebig-Universität Gießen (JLU), Federal Statistics Office Germany, Polish Academy of Sciences, Michigan State University, Universität zu Köln, Princeton University, Deutsches Institut für Wirtschaftsforschung (DIW), University of Oslo, Universität Witten/Herdecke, University of Oxford, Université de Neuchâtel, Pontificia Universidad Católica de Chile, Georgetown University, University of Manchester, Western Governors University, Director of Data for Freedom, Equity Research Cooperative, 19107., Indiana University Bloomington, University of Wisconsin, Milwaukee, United States Military Academy, University of Copenhagen, Pädagogische Hochschule Karlsruhe, BBSR - Bundesinstitut für Bau-, Stadt- und Raumforschung, Leiden University, Boston University, Technische Universität Dresden, Linköping University, Administrative Headquarters, Max Planck Society, Munich 80539, Germany., Utah State University (USU), University of Birmingham, Ghent University , University of California, Los Angeles, University of Luxembourg, Australian National University, Washington University in St. Louis, Freie Universität Berlin (FU), Teacher and Teaching Quality, Leibniz Institute for Research and Information in Education, Frankfurt 60323, Germany., University of Groningen, Philipps Universität Marburg, University of Michigan, Expert Council on Intergration and Migration, Berlin 10178, Germany., Polish Academy of Sciences (PAS), Stevens Institute of Technology, Stiftung Universität Hildesheim, Uppsala University, North-West University, FernUniversität in Hagen, IHS - Institut für Höhere Studien und wissenschaftliche Forschung, Lucian Blaga University of Sibiu, Behaviors, Universität Zürich (UZH), Max-Plank Institut zur Erforschung von Gemeinschaftsgütern, York University, Kyushu University, Universität Duisburg-Essen, Institute for Employment Research, Federal Employment Agency, Nuremberg 90478, Germany.

Journal

Royal Society Open Science

Volume

12

Pages

241038

ISSN

2054-5703

DOI

https://doi.org/10.1098/rsos.241038

Publication date

03-2025

Peer reviewed

Yes

Austrian Fields of Science 2012

504007 Empirical social research

Sustainable Development Goals

SDG 10 - Reduced Inequalities

Portal url

https://ucrisportal.univie.ac.at/en/publications/30d1f05a-eaea-425a-a42a-be4ff7f24669