Ahead of Print article · Journal article
From detection of individual metastases to classification of lymph node status at the patient level: the CAMELYON17 challenge
Radboud University Nijmegen1
Huazhong University of Science and Technology2
Department of Applied Mathematics and Computer Science, Technical University of Denmark3
Visual Computing, Department of Applied Mathematics and Computer Science, Technical University of Denmark4
Chinese University of Hong Kong5
ContextVision AB6
Middle East Technical University7
Proscia Inc.8
Karlsruhe Institute of Technology9
Canisius Wilhelmina Hospital10
Radboud University Medical Center11
University Medical Centre Utrecht12
Rijnstate Hospital13
Lunit, Inc.14
Massachusetts General Hospital/Harvard Medical School15
Eindhoven University of Technology16
The University of Tokyo17
Tokyo Medical and Dental University18
Indica Labs19
...and 9 moreAutomated detection of cancer metastases in lymph nodes has the potential to improve assessment of prognosis for patients. To enable fair comparison between the algorithms for this purpose, we set up the CAMELYON17 challenge in conjunction with the IEEE International Symposium on Biomedical Imaging 2017 conference in Melbourne.
Over 300 participants registered on the challenge website, of which 23 teams submitted a total of 37 algorithms before the initial deadline. Participants were provided with 899 whole-slide images for developing their algorithms. The developed algorithms were evaluated based on the test set encompassing 100 patients and 500 whole-slide images.
The evaluation metric used was a quadratic weighted Cohen’s kappa. We discuss the algorithmic details of the ten best pre-conference and two post-conference submissions. All these participants used convolutional neural networks in combination with pre-and postprocessing steps. Algorithms differed mostly in neural network architecture, training strategy and pre-and postprocessing methodology.
Overall, the kappa metric ranged from 0.89 to -0.13 across all submissions. The best results were obtained with pre-trained architectures such as ResNet. Confusion matrix analysis revealed that all participants struggled with reliably identifying isolated tumor cells, the smallest type of metastasis, with detection rates below 40%.
Qualitative inspection of the results of the top participants showed categories of false positives, such as nerves or contamination, which could be targets for further optimization. Last, we show that simple combinations of the top algorithms result in higher kappa metric values than any algorithm individually, with 0.93 for the best combination.
| Language: | English |
|---|---|
| Publisher: | IEEE |
| Year: | 2018 |
| Pages: | 550-560 |
| ISSN: | 1558254x and 02780062 |
| Types: | Ahead of Print article and Journal article |
| DOI: | 10.1109/TMI.2018.2867350 |
| ORCIDs: | Thagaard, Jeppe and Dahl, Anders Bjorholm |
Breast cancer Grand challenge Lymph node metastases SDG 3 - Good Health and Well-being Sentinel lymph node Whole-slide images
Algorithms Biomedical imaging Breast Neoplasms CAMELYON17 challenge Female Histological Techniques Hospitals Humans Image Interpretation, Computer-Assisted Lymph nodes Lymphatic Metastasis Metastasis Pathology ResNet Sentinel Lymph Node Tumors WSI automated detection cancer cancer metastases challenge website convolutional neural nets convolutional neural networks evaluation metric grand challenge higher kappa metric values image classification image segmentation individual metastases initial deadline lymph node metastases lymph node status lymph nodes medical image processing neural network architecture patient diagnosis patient level post-conference submissions pre-trained architectures quadratic weighted Cohens kappa sentinel lymph node tumours whole-slide images
