Brain tumor classification with tumor segmentations and a dual path residual convolutional neural network from MRI and pathology images

Brain tumor classification plays an important role in brain cancer diagnosis and treatment. Pathologists typically have to work through numerous pathology images that can be in the order of hundreds or thousands which takes time and is prone to manual error. Here we investigate automating this task given pathology images as well as 3D MRI volumes without lesion maps. We use data provided by the CPM-RadPath 2019 MICCAI challenge. We first evaluate accuracy on the validation dataset with MRI and pathology images separately. We predict the 3D tumor mask with our custom developed tumor segmentation model that we used for the BraTS 2019 challenge. We show that the predicted tumor segmentations give a higher validation accuracy of 77.1% vs. 69.8% with MRI images when trained by a 3D residual convolutional neural network. For pathology images we train a 2D residual network and obtain a 66.2% validation accuracy. In both cases we find high training accuracies above 95% which suggests overfitting. We propose a dual path residual convolutional neural network model that trains simultaneously from both MRI and pathology images and we use a simple method to prevent overfitting. One path of our network is fully 3D and considers 3D tumor segmentations as input while the other path considers pathology images. To prevent overfitting we stop training after 90% training accuracy at the epoch number where our network loss increases in the following one. With this approach we achieve a validation accuracy of 84.9% showing that indeed combining the two image sources yields a better overall accuracy.