Reproducibility in deep learning algorithms for digital pathology applications: a case study using the CAMELYON16 datasets

Abstract

Reproducibility as a cornerstone of science has in recent years attracted significant attention in a variety of applications that involve big data and complex algorithms. Artificial Intelligence (AI) applications, in particular, have been referred by some authors as having a “reproducibility crisis”. The application of deep learning algorithms in digital pathology, which has been demonstrated in some studies to achieve pathologist-level performance, involves both big data and complex algorithms. It is important to identify technical factors influencing AI performance for such studies to be reproducible. In this work, we conducted a reproducibility study using the public datasets shared by the CAMELYON16 challenge, which aimed to develop and assess algorithms for the detection of breast cancer metastasis using whole-slide images (WSIs) of lymph node sections. We used the two-stage classification framework of the CAMELYON16 challenge’s top-performing algorithm, i.e., a convolutional neural network (Inception V1) to generate heatmaps followed by feature extraction from the heatmaps and a random forest classifier for classification. We investigated the effects of variations in training/testing procedures on the performance of AI/ML algorithms including color augmentation methods, color normalization methods, and the resolution of heatmaps. Our results showed that, despite the differences in color augmentation and color normalization methods, the utilization of these techniques improved the classification performance by an AUC value of 0.07 compared to without using them, which is consistent with the CAMELYON16 findings. We concluded that sufficient details of technical description should be provided for a study to be fully reproducible.