Reproducible Research: A primer for the social sciences

Ben Marwick
March 2014

Overview

  • Definitions, motives, history, spectrum
  • Current practices
  • A selection of tools to improve reproducibility
  • Challenges, standards & our role in the future of reproducible research

Definitions

Replicable refers to the ability to produce exactly the same results as published. Other people get exactly the same results when doing exactly the same thing. Technical: cf. validation and verification

Reproducible refers to the ability to create a workflow that independently upholds the published results using the information provided. Checking the results from the fixed digital form of data and code from the original study. Something similar happens in other people's hands. Substantive: possibly by a new implementation

“The goal of reproducible research is to tie specific instructions to data analysis and experimental data so that scholarship can be recreated, better understood and verified.” - Max Kuhn, CRAN Task View: Reproducible Research

History of reproducible research

  • Mathematics (400 BC?)
  • Write scientific paper, Galileo, Pasteur, etc. (1660s?)
  • Publish a pidgin algorithm and describe simulation datasets (1950s?)
  • Sell magtape of code and data (1970s?)
  • Place idiosyncratic dataset & software at website (1990s?)
  • Publish datasets and scripts at website, eg. biology, political science, genetics, statistics (2000s?)
  • Hosted integrated code and data (2020s?)

Gavish & Gonoho AAAS 2011, Oxberry 2013

Motivations: Claerbout's principle

“An article about computational result is advertising, not scholarship. The actual scholarship is the full software environment, code and data, that produced the result.” - Claerbout and Karrenbach, Proceedings of the 62nd Annual International Meeting of the Society of Exploration Geophysics. 1992

“When we publish articles containing figures which were generated by computer, we also publish the complete software environment which generates the figures” - Buckheit & Donoho, Wavelab and Reproducible Research, 1995.

Benefits are straightforward

  • Verification & Reliability: Easier to find and fix bugs. The results you produce today will be the same results you will produce tomorrow.
  • Transparency: Leads increased citation count, broader impact, improved institutional memory
  • Efficiency: Reuse allows for de-duplication of effort. Payoff in the (not so) long run
  • Flexibility: When you don’t 'point-and-click' you gain many new analytic options.

But the limitations are substantial

Technical

  • Classified/sensitive/big data
  • Nondisclosure agreements & intellectual property
  • Software licensing issues
  • Competition
  • Neither necessary nor sufficient for correctness (but essential for dispute resolution)

Cultural & personal

  • Very few researchers follow even minimal reproducibility standards.
  • No-one expects or requires reproducibility
  • No uniform standards of reproducibility, so no established user base
  • Inertia & embarassment

Our work exists on a spectrum of reproducibility

alt text Peng 2011, Science 334(6060) pp. 1226-1227

Goal is to expose the reader to more of the research workflow

Current practices, or ethnographic observations of social science research workers

  • Enter data in Excel
  • Use Excel for data cleaning & descriptive statistics
  • Import data into SPSS/SAS/Stata for further analysis
  • Use point-and-click options to run statistical analyses
  • Copy & paste output to Word document, repeatedly