Medical Image Analysis

Nuclei segmentation is a fundamental task that is critical for various computational pathology applications including nuclei morphology analysis, cell type classification, and cancer grading. Conventional vision-based methods for nuclei segmentation struggle in challenging cases, but fortunately deep learning approaches have proven to be more robust and generalizable.

One disadvantage however is that CNNs require large amounts of labeled histopathology data. Moreover, conventional CNN-based approaches lack structured prediction capabilities which are required to distinguish overlapping and clumped nuclei for instance segmentation. To address these problems we present an approach to nuclei segmentation that overcomes these challenges by utilizing a cycle generative adversarial network trained with synthetic and real data. We generate a large dataset of H&E (the combination of two histological stains: hematoxylin and eosin) training images with perfect nuclei segmentation labels using an unpaired GAN framework. We put special emphasis on the variant geometric features of the nuclei and transform the mask shape according to statistical distribution. Based on the augmented mask shape we synthesize images to improve generalization of the network and avoid overfitting.

Nuclei instances are segmented based on MaskRCNN. The networks of GAN and MaskRCNN are trained separately and then fine-tuned end-to-end to add variability to the the synthetic images.

This synthetic data along with real histopathology data from four different organs are used to train a cycle GAN with spectral normalization and gradient penalty for nuclei segmentation. This adversarial regression framework enforces higher order consistency when compared to conventional CNN models. We demonstrate that this nuclei segmentation approach generalizes across different organs, sites, patients and disease states, and outperforms conventional approaches, especially in isolating individual and overlapping nuclei.