Large-Scale Study Finds Conclusions Often Diverge When Hundreds of Researchers Reanalyze the Same Data

A large-scale study published in Nature finds that scientific conclusions can vary significantly depending on how data are analyzed. The paper, which includes participation from our researcher Andreu Arenas, highlights how different analytical choices can lead to different results—even when using the same data.

The study, “Investigating the analytical robustness of the social and behavioral sciences”, was led by Balázs Aczél and Barnabás Szászi (Eötvös Loránd University and Corvinus University), and conducted within the SCORE program funded by DARPA. A total of 457 independent analysts carried out 504 reanalyses of data from 100 previously published studies across the social and behavioral sciences.

All teams worked with the same datasets and research questions but were free to choose their own analytical approaches. The results show substantial variation: although most reanalyses broadly supported the original findings, effect sizes, statistical estimates, and levels of uncertainty often differed. Only about one third of analyses reached the exact same conclusions as the original studies.

Importantly, these differences were not driven by lack of expertise—experienced researchers were just as likely to diverge in their results. The study also found that observational studies tend to be less robust than experimental ones, likely due to greater analytical flexibility.

Rather than undermining trust in science, the authors argue that these findings underscore the importance of transparency and suggest broader use of multi-analyst and “multiverse” approaches to better capture the true uncertainty behind scientific conclusions.

---