Reproducibility Tutorial

The goal of this first exercise is to fill an adapted version of the MICCAI reproducibility checklist for different fake papers.

Content of the papers

All the papers have the same content: their objective is to train a deep learning network to perform a binary classification, Alzheimer's disease patients VS cognitively normal participants, using brain T1-MRI. Then they explain their algorithm using an explainability method generating an attribution map which is showing what the network uses in the image to take a decision.

They also all have the same organisation. The abstract and introduction sections are all identical and do not include any useful information, except the link to the corresponding GitHub repo at the end of the introduction.

Content of the checklist

For all papers, the participants have to fill the same checklist. Each section of the checklist is related to one or more sections of the article or the GitHub repo.

1. Data set - Sections 2 + 3.1, GitHub repo
   • Availability - Open-source / Semi-public / On request / Not findable
   • Reference (citation / link) - Yes / No
   • Definition of the labels (AD & CN) - Not described ↔ Reproducible
   • Description of the images (acquisition protocol/preprocessing) - Not described ↔ Reproducible
2. Code - Sections 3.2 + 3.3, GitHub repo
   • Availability - Open-source / Semi-public / On request / Not findable
   • Dependencies - Fixed versions / No versioning / None
   • Preprocessing code - None ↔ All experiments
   • Training code - None ↔ All experiments
   • Evaluation code - None ↔ All experiments
   • Explainability code - None ↔ All experiments
   • Pre-trained models - Open-source / Semi-public / On request / None
   • Minimal list of commands to run the code - None ↔ All experiments
3. Computational setup - Section 3.2
   • Runtime - Yes / No / Not applicable
   • Memory footprint - Yes / No / Not applicable
   • Computing infrastructure - Yes / No / Not applicable
4. Training - Section 3.2
   • Hyperparameter search - None ↔ All experiments
   • Hyperparameter values - None ↔ All experiments
5. Evaluation - Sections 3.3 + 4
   • Validation procedure - Not described ↔ Reproducible
   • Error bars / confidence intervals - None ↔ All experiments
   • Statistical significance - None ↔ All experiments

Links to the materials

The papers, codes and checklists can be found here:

The goal of this second exercise is to replicate an experiment from a research paper that adheres to reproducibility standards. The intention is to demonstrate how straightforward this process can be when following a reproducibility checklist. This step is crucial as it illustrates how a well-documented and reproducible methodology can greatly simplify the replication of research findings, thereby enhancing the verifiability and reliability of scientific work.

This time, you do not have to fill out a form, but you have to run the code (which should be really easy 😉). The final goal is to reproduce the tables and figures expliciting the results in the paper:

Content of the paper

The objective of the paper is to explore the transferability of a CNN trained on the ADNI dataset to the OASIS-1 dataset for Alzheimer's disease detection. This paper is applying a pre-trained network to a data set. We decided to make attendees train the network to limit the need for computational resources.

• Paper
• Code

To limit download time, we redistributed the part of the OASIS data set we will be using here.

Please also make sure that you have Anaconda or Miniconda installed, as this tutorial will ask you to create a conda environment.