Deep Learning for Digital Image Analysis with Whole Slide Imaging for Lymphoma Diagnosis: Challenges and Promises

Citation: Professor. Andy N.D. Nguyen and Kareem A. Allam (2022) Deep Learning for Digital Image Analysis with Whole Slide Imaging for Lymphoma Diagnosis: Challenges and Promises, 21st Century Pathology, Volume 2 (2): 116

Abstract

The advent of whole slide imaging in digital pathology has introduced the computer-aided diagnosis of tissue via digital image analysis. Digitized slides can now be analyzed via a variety of machine learning algorithms, including deep learning as the most significant one. This commentary reviews recent machine learning studies for lymphoma diagnosis. We will look at potential analysis strategies, technical considerations, advantages, limitations of such approaches, and future research directions in this emergent field.

Keywords:

Deep learning; Whole slide imaging; Lymphoma diagnosis

Description

Modern pathology practice is moving toward a digital image format, cumulating in utilizing computer monitor screens to examine scanned histology slides. This process of digitization of glass slides, in combination with the development of specialized software tools to identify and measure features previously observed via a microscope, has brought about digital image analysis on tissue sections. Tissue image analysis, when performed optimally, can result in highly precise and reproducible results. We will briefly cover the timelines of the digital analysis for scanned tissue slides and outline the current state of available software tools.

Digital Image Analysis

Analyzing images with objective tools is as old as microscopy itself. Leeuwenhoek developed a system to measure microscopic objects in the 17th century [1]. When digital images became available with digital cameras, measurements could be extracted from digitized tissue slides such as nuclear shape, nuclear size, cellular circumference, cellular texture, area of certain stain chemical or immunohistochemistry (IHC) stain, and a number of cells in a selected area, etc. [2]. They are obtained by edge detection and cell segmentation [3]. When undertaking quantification of biomarkers such as IHC stains, image analysis tools can be of great value to standardize the analysis as well as minimize bias, subjectivity, and variability in the measured data. Typical examples include PD-L1 scoring and HER2 scoring.

Digital Image Analysis and the Introduction of Whole Slide Imaging

Despite many incremental advances throughout the decades, digital image analysis remained unchanged until the advent of whole slide imaging (WSI) in the early 2000s [4]. With WSI, the traditional histology glass slide is digitized via a slide scanner to be displayed on a monitor screen at a similar resolution as light microscopy. Compared to the traditional analog workflow of tissue sections being prepared and viewed under a microscope, the new digital workflow requires additional equipment (slide scanner, image storage, and digital viewing workstation), trained personnel, and specific quality control steps for quality control of scans, all of which require increased laboratory resources [3]. However, there are multiple advantages of digital workflow, including ease of case sharing between pathologists, ease in organizing teaching slides, and extraction of complex data in a highly reproducible fashion via specialized digital image analysis software [2]. The market is currently rapidly expanding with new companies that develop software for extracting relevant information from WSI images using artificial intelligence (AI) algorithms, a step beyond the traditional digital image analysis.

Evolution of Artificial Intelligence

AI provides automated methods for data analysis. Its technique is based on the ability of the machine to learn information from previously saved data in databases and improve itself for better diagnostic purposes [5]. The AI frameworks have evolved throughout the decades. The first conventional AI algorithms included support vector machine (SVM) and neural network (NN). These techniques were followed by the new sophisticated deep learning (DL) algorithms such as convolutional neural network (CNN), recurrent neural network (RNN), long short term memory (LSTM), and extreme learning model (ELM) [5]. DL, the newest type of AI, has largely demonstrated itself as the most effective and reliable technique when applied to the medical field. It is a growing innovation trend in data analysis and has been termed one of the ten breakthrough technologies of 2013 [6]. Since DL presents in many algorithmic formats, it is not considered a single technique. Instead, DL can be described as the latest generation of artificial neural networks, with the neuron being the fundamental unit. DL consists of multiple layers of neurons lying between input and output layers that permit higher levels of abstraction and improved predictions from data input [7]. Each neuron receives the input data from multiple neurons of the previous layer and then uses unsupervised learning to find certain characteristic features that will be filtered and added together to ultimately generate an output to be communicated to the next layer. Increasing the number of layers allows for more features to be detected and more complex patterns to be learned [8]. DL has been applied to a wide range of domains, from speech recognition [9-13] to image analysis [14-16] and natural language processing [17-19]. In recent years, DL techniques have become the state of the art in computer vision. A specific DL neural network subtype, the convolutional neural network or CNN [20, 21], has become the de-facto standard in image recognition and has been shown to approach human performance in various tasks [7]. Their level of performance has far exceeded that of the traditional technique of digital analysis of measurements by edge detection and cell segmentation. These CNN systems excel by learning relevant features directly from raw data in large image databases; this contrasts with the more traditional pattern recognition techniques, which rely on detecting manually crafted quantitative features [22].

CNN systems for digital image analysis have greatly benefited from parallel processing since most image operations are based on matrix manipulations [8]. Parallel processing significantly decreases computing time by performing all similar matrix operations at the same time instead of in a linear sequence. The computer graphics cards, known as graphics processing units (GPUs), contain hundreds or thousands of processing cores and significantly increase the computational speed. The core element of the CNN algorithm is convolution [23], an operation in image processing using kernels (filters), to detect certain characteristics of an image. Mathematically, a convolution is done by multiplying the pixels’ values in the image patch by a kernel matrix; this effectively enhances the value of an image patch by adding the weighted values of all the neighboring pixels together. By moving the kernel across the input image, one obtains the feature map as a filtered image. As shown in Figure 1, the CNN model [24] has the following processing pipeline for the detection of visual categories: the convolutional layers perform feature extraction consecutively from the image patch to higher-level features, followed by the max-pooling layers’ down-sampling to reduce the amount of computation in the network, finally, the fully-connected layers provide a prediction based on the given features.

Figure 1: Processing pipeline of a convolutional neural network for the detection of visual categories in images ([25], with permission).

 

Recent studies showed that CNNs are extremely effective in object recognition for digital images. Many medical image studies have started to apply CNNs to a wide range of applications [14-16], and promising results have been emerging from recent studies [7, 20-22]. The study domains included: metastatic breast cancer in WSI of sentinel lymph node biopsies [26], skin cancers [27], HER2 status in breast cancer [28], Gleason scoring for prostate cancer [29], Ki-67 labeling index for meningiomas [30], among others. We will discuss the application of CNNs to the diagnosis of lymphoma.

Convolutional Neural Networks and Diagnosis of Lymphoma

Lymphoma is a clonal malignancy of lymphocytes, either T cells or B cells. The different lymphoma entities are typically first suspected by their pattern of growth and the cytologic features of the abnormal cells via light microscopy of Hematoxylin and Eosin stained tissue sections. Immunophenotyping is typically required for diagnosis with flow cytometry and/or immunohistochemical stains. In addition, cytogenetics, molecular pathology results, and clinical features are often needed in finalizing the diagnosis of certain lymphoma types [31]. Lymphoid malignancies are diagnosed in 280,000 people annually worldwide and include at least 38 entities according to the World Health Organization (WHO) Classification of Lymphoid Malignancies [31]. Due to subtle differences in histologic findings between various types of lymphomas, histopathologic screening often presents a challenge to practicing pathologists. Moreover, lymph node diseases are not restricted to malignancies; reactive and inflammatory changes due to infections which can have similar clinical and pathological presentations as lymphomas should always be part of the differentials. Thus, there is a need to relieve the workload on pathologists by obtaining automated software for screening purposes.

The recent introduction of digital Whole Slide Imaging (WSI) opens an opportunity for automated identification of histopathologic features of lymphomas [32]. The quality of the images is pivotal for optimal microscopic interpretation. Fortunately, digital image acquisition has improved substantially in recent years with the implementation of instrumentation capable of acquiring data at very high rates and with excellent resolution [32]. Only two WSI platforms have so far received the Food and Drug Administration (FDA) approval for primary surgical pathology in the US beyond the scope of research [33]. The first approval was granted in 2017 to the Philips IntelliSite Digital Pathology Solution. It is a closed system that comprises a scanner/image management system and display. The approval does not extend to frozen sections, cytology, or non-formalin-fixed paraffin-embedded specimens. In 2020, the second platform to have FDA approval for primary diagnosis is the Leica Biosystem’s Scanner AT2 DX.

It has been noted that color variations in the tissue exist between various staining techniques and stain color normalization techniques are needed to alleviate such variations in training and testing images for the CNN models [34]. It is important to have the results confirmed in multi-center studies using different staining protocols and different WSI instruments.

As mentioned above, the image interpretation process of digital slides is actively studied in diagnostic medicine, particularly with the advent of DL which made considerable contributions to the realm of diagnostic pathology. Hematopathology has also earned its part in this digitalization movement. Recent projects have shown promising results using DL to detect lymphoma with WSI.

Recent Deep Learning Studies on Lymphoma Diagnosis

Although DL is an active research field, its application to the microscopic diagnosis of tumors is relatively new. Most published work has focused on diagnosis between two disease entities, between benign tissue and one specific tumor, or grading for a known tumor, making it difficult to assess the practical value of the designed CNNs. Fauzi FA, et al. (2015) [35] conducted a project for automated grading of follicular lymphoma and confirmed the usefulness of the method in tissue grading. Another study using the Aperio AT2 instrument for WSI scanning for image analysis showed 82.5% concordance between the pathologists and the trained algorithms for subtyping of DLBCL [36]. A previous study was conducted by Orlov NV, et al. (2010) to classify lymphomas in one of the following three types: small lymphocytic lymphoma, follicular lymphoma, or mantle cell lymphoma using spectral analysis with a weighted-neighbor distance (WND) algorithm [37]. This study reported a high accuracy rate of 99%. However, only a small number of 30 lymphoma cases were used which did not provide an adequately rigorous validation for the model.

A study by El Achi H, et al. (2019) [38], was the first robust one to get closer to actual practice by exploring how DL can be used to accurately classify a test case as one of the four non-Hodgkin lymphoma (NHL) entities: benign lymph node, diffuse large B-cell lymphoma (DLBCL), Burkitt lymphoma (BL), or small lymphocytic lymphoma (SLL). This study provided a proof of concept for incorporating automated lymphoma diagnostic screening into future pathology workflow to augment the pathologists’ productivity. CNN was used to build a lymphoma diagnostic model. The software was written in the Python language, together with TensorFlow [25] and Keras [39], two important Python libraries particularly useful in DL modelling. Parallel processing was based on NVIDIA GPU with Compute Unified Device Architecture (CUDA) [40]. WSI images of Hematoxylin and Eosin slides were obtained for 128 cases including 32 cases for each diagnostic category [41, 42]. Four sets of 5 representative images, 40 × 40 pixels in dimension, were taken for each case. A total of 2,560 images were obtained from which 1,856 were used for training, 464 for validation, and 240 for testing. For each test set of 5 images, the predicted diagnosis was combined with predictions of five images in the selected set. The test results showed excellent diagnostic accuracy at 95% for image-by-image prediction and 100% for set-by-set prediction (Table 1 and Table 2, respectively). Out of 240 test images, a total of 228 images were correctly diagnosed by the CNN model, and the remaining 12 images were given incorrect diagnoses, yielding an overall 95% accuracy for diagnostic prediction (Table 1). Among the 12 images with a lack of concordance between the observed and the predicted diagnosis: 4 SLL images were predicted as benign, 4 other SLL images were predicted as DLBCL, and 4 benign images were predicted as BL. It appears that diagnosis based solely on one image is too stringent to be of practical value. Instead, the final diagnosis needs to be based on all the five representative images to exclude outliers for a given set.

Table 1: Accuracy in predicting diagnosis using one single image at a time ([25], with permission).

Observed Diagnosis
Predicted Diagnosis		Benign	DLBCL	BL	SLL
	Benign	56	0	0	4
	DLBCL	0	60	0	4
	BL	4	0	60	0
	SLL	0	0	0	52
Accuracy: 228/240=95%. Legends: DLBCL: diffuse large B cell lymphoma; BL: Burkitt lymphoma; SLL: small lymphocytic lymphoma

Table 2: Accuracy in predicting diagnosis for sets of 5 images using majority voting (3 out of 5 images for each set must agree) ([25], with permission).

Observed Diagnosis
Predicted Diagnosis		Benign	DLBCL	BL	SLL
	Benign	12	-	-	-
	DLBCL	-	12	-	-
	BL	-	-	12	-
	SLL	-	-	-	12
Accuracy: 48/48=100% Legends: DLBCL: diffuse large B cell lymphoma; BL: Burkitt lymphoma; SLL: small lymphocytic lymphoma

Strength and Limitation of Deep Learning in Lymphoma Diagnosis

The strength of the most robust study by El Achi H, et al. (2019) [38] lies in the inclusion of 4 lymphoid diseases and in focusing on the more frequent NHL types, taking DL a step closer to practical pathology work. Moreover, it included 128 cases collected from two databases generated at different institutions. This variety of cases from different populations and institutions combined with the successful results confirmed that the algorithm surpasses the inter-laboratory variations in the tissue processing as well as the quality and type of slides staining. This contrasts with the human eyes that must adapt to any modification of the staining, a difficult and time-consuming process.

On the other hand, the current limitations of this preliminary study consist first in including only four histologic categories, not yet practical for actual clinical use in hematopathologic diagnosis. The number of cases included in that study is 128, a substantial number that generates 2,560 digital images but may still be considered limited for DL projects which typically include many more cases [8]. Since DL performs better with a large sample volume, the database can be increased in size in the future by applying the “Data Augmentation” methods such as random cropping, image rotation, image inversion, etc. [43]. Finally, the future design of CNN models could benefit from a process known as “transfer learning” that helps improve the training method. Transfer learning is based on exploiting a pre-trained algorithm and calibrating it for a new application. The rationale behind applying the technique resides in the fact that a pre-trained network (such as one for gynecology or gastroenterology) has already learned to extract abstract features from the images and this network can be expanded to hematopathology; a process that will speed up training the model [44, 45].

Current Status of Deep Learning in Lymphoma Diagnosis

Transfer learning, in a globally-optimized platform with multiple pre-trained CNNs, has been recently applied and has achieved 100% accuracy in DLBCL diagnosis [46]. Some architectures, such as efficient net [47], use a compound scaling method, which allows the smaller size and higher speed without loss of accuracy. CNNs have also been used to predict the probability of large cell transformation in follicular lymphoma and CLL from bone marrow biopsies [48]. Another technique that has been used recently is “weakly supervised learning”. In most supervised learning techniques, the training slides have to be labeled in detail at the level of patches or pixels, which is a very painstaking process. In weakly supervised learning, the training slides can be labeled as a whole such as “cancer” if there is any cancer on the slide, or as “not cancer” if there is no cancer anywhere on the slide [49]. Other supervised learning techniques include regression, which can be used to locate the position of the tumor on the slide, and segmentation, which can be used to digitally highlight the tumor area. Recently, some “attention-based” models have been tried which use a region selection mechanism to focus on the most relevant areas for diagnosis. Some models use multi-magnification networks which incorporate image patches at different magnifications to better capture the context. There are also variants of CNNs that use a dense scanning mechanism that shares computations in overlapping regions. This can greatly increase the speed of inference. A new pooling layer, called the “anchor layer”, can also be used which can reconstruct the loss from the max-pooling layers. Unsupervised transfer learning has also been tried, which uses learned mapping functions, rather than directly applying learned features to the target task [49].

Conclusion

In summary, recent deep learning studies provided a proof of concept for incorporating automated lymphoma diagnostic screening using digital microscopic images into the pathology workflow to augment the pathologists’ productivity. Future studies will need to include far more histologic entities and many more cases for training, validation, and testing. Once this has been achieved, the CNN models would be potentially suitable to improve the efficiency of the diagnostic process in histopathology. This could in turn lead to adapted protocols, where pathologists would perform a more thorough analysis on difficult cases, as the straightforward cases have already been handled by a DL system. Most researchers believe that within the next 15 years, DL-based applications will play an essential role in the pathology laboratory, working alongside pathologists to provide a more timely and accurate diagnosis. The pathologists will continue to be instrumental in both the use and operation of image analysis workflows, which will continue to evolve and transform both clinical and research activities.