CANCOL, a Computer-Assisted Annotation Tool to Facilitate Colocalization and Tracking of Immune Cells in Intravital Microscopy

During the last two decades, two-photon intravital microscopy (2P-IVM) has been established as one of the most popular techniques to study immune cell migration and cell interaction in living organisms (1). This technique has been used to reveal hitherto unexplored immune processes in different experimental settings, including infection, autoimmunity, and cancer (2–4).

The standard 2P-IVM pipeline involves the exposition of the organ with minimally invasive surgery, followed by the acquisition of four-dimensional (three-dimensional images over time) videos (5). Subsequently, videos are quantified by performing cell detection, cell tracking, and computation of cell motility metrics such as speed and displacement among other parameters that capture leukocyte behavior (6, 7). However, analyzing 2P-IVM data is a complex process that requires a significant investment of time. Indeed, intravital imaging of immune cells is often associated with specific artifacts that hamper cell detection and tracking, leading to the introduction of errors in the final readout, if not corrected manually (8).

Typically, in a 2P-IVM experiment, multiple detectors are used to acquire several acquisition channels. Each of these channels is associated with a fluorophore used to label a specific cell type or population (9, 10). However, the broad emission spectrum of commonly used fluorophores might result in cells becoming visible in multiple acquisition channels. Moreover, nonspecific emissions from autofluorescent objects in the background and crystalline structures such as collagen fibers may appear in the acquired channels, further complicating the tracking process (8). Therefore, to separate the fluorescence emitted by the cells of interest from the background or other overlapping signals, spectral unmixing techniques were developed. These included both linear and nonlinear unmixing algorithms (11, 12), as well as software for facilitated unmixing (13). The output of these methods is an additional virtual channel in which only the cells of interest are visible. However, challenges may arise when analyzing groups of pixels with highly non-convex shapes either in the xyz space or in the color space (14). Moreover, the fluorescence intensity may undergo large shifts over time, making cells poorly visible. These factors would require precise adjustment of thresholds both across different areas on the same acquisition and over time. In addition, in 2P-IVM acquisitions, cells migrate in a complex three-dimensional environment, undergoing large shape deformations. This can result in cells closely resembling other objects in the background (8).

To account for similar challenges, new methods based on computer vision, which enabled the separation of the foreground from the background by employing supervised machine learning, have been applied (15, 16). These methods rely on a process called brushing, during which the user provides examples of the cells of interest and examples of background by drawing lines on the image. The provided examples are then used to train a pixel classifier that generates a binary image where only the cells of interest are visible. However, the binary images generated by these methods are largely influenced by the annotations provided by the user. Altogether, these problems introduce research bias and reduce the usability of specialized imaging software packages for 2P-IVM.

Therefore, in this work, we developed a tool, called CANCOL (Computer Assisted Annotation Tool to Facilitate Colocalization), to perform pixel classification in an optimized way for 2P-IVM videos of immune cells. This tool assists the user during the annotation process, employing a user interface (UI) that guides the brushing of relevant parts of the video. Then, CANCOL uses these annotations to compute a virtual imaging channel that is specific for the cell of interest and optimized to perform cell detection and tracking.

To validate CANCOL, we employed a public database of 2P-IVM videos [Leukocyte Tracking Database (LTDB) (8)]. The database included 20 videos with multiple types of immune cells (neutrophils, B cells, T cells, and NK cells) that were acquired in different anatomical regions (i.e., in the lymph node and spleen) under inflammatory or tumor conditions. Cells were labeled with different fluorophores and acquired with different microscopy platforms. All of the videos from the LTDB collection were used to validate the method presented in this paper except LTDB018 and LTDB020, in which errors in the provided ground truth were detected.

Each pixel was classified as either foreground or background using the annotations provided by the users. For the results included in this study, 20–50 individual points per video were used. Results were generated without the usage of the preview tool of our method to account only for the quality of annotations and exclude the improvement conferred by repeated training processes. A support vector machine with a radial basis function kernel was used as a binary classifier. The following features were used for pixel classification: the color of each pixel (one value per channel) and the average color in a circular neighborhood of two sizes (5 and 9 µm, one per channel). These features accounted for the brightness of a single pixel, of a cell (small neighborhood), and the average brightness in the area of the organ (large neighborhood). To maximize cell detection using spot-tracking tools, the output of the generated channel was the classification probability for class 1, rather than a binary mask.

To provide the user with real-time feedback on the quality of the annotations, the UI displays two indicators: balancing and coverage. Balancing is computed as (N1 – N0)/(N1 + N0), where N0 is the total number of points annotated of class 0 (background) and N1 is the total number of points annotated of class 1 (foreground). Its optimal value is 0, indicating that the dataset is balanced with the same number of annotations in the two classes. Negative values indicate that the user has provided more annotations on the background, whereas positive values indicate that the user has provided more annotations on the foreground. Coverage estimates how well the annotations provided by the user cover the distribution of the pixels in the color space. Its value is computed as the sum of the distance between K = 1000 random points from the images, and the closest points in the annotations.

To automatically compute cell tracks, the spots detection and tracking functions of bioimaging software Imaris (Oxford Instruments, tested versions 7.7.2–9.6.0) were used. Cells were initially detected as spots of radius 8 µm using the automatically computed thresholds for brightness intensity (no manual tuning). Spots were tracked automatically by using the autoregressive motion algorithm, with a maximum linking distance of 20 µm and without gap tolerance (gap size = 0). All of the tracks included in Figs. 2–5 were generated automatically without manual correction. The tracks used for Fig. 2D–F were manually edited by three trained image analysts using the Imaris track editing functionality.

To evaluate the accuracy of tracking when using the proposed method, the ground truth included in LTDB (8) was employed. The tracking accuracy (TRA) measure (17) was used to compare the ground truth (manually generated from the consensus of three independent operators) with respect to the tracks generated with automatic tracking. To evaluate the quality of the virtual channel, the signal-to-noise ratio (SNR) and the contrast ratio (CR) were estimated by comparing the brightness of points inside cells (FG), with respect to the brightness of points outside cells (BG). Respectively, SNR = (mean(FG) – mean(BG))/SD(BG) and CR = mean(FG)/mean(BG), where mean refers to the average pixel intensity in one channel, and SD refers to the SD in one channel. The raw imaging channels with higher SNR and CR were selected for the comparison.

The proposed method was written in MATLAB (MathWorks) r2019b, and the UI was built with App Designer.

CANCOL is released under the Open Source General Public License v3. Code is available at https://github.com/pizzagalli-du/CANCOL/ along with an Imaris plugin (XTCANCOL) and a standalone version (CANCOL-STANDALONE) that does not require Imaris to be executed.

For statistical analysis and data visualization, Prism 7 (GraphPad Software, La Jolla, CA) was used. Results are expressed as the mean ± SD. All statistical comparisons were performed with two-tailed nonparametric tests (Mann–Whitney). Statistical significance was defined as *p < 0.05, **p < 0.01, ***p < 0.001, and ****p < 0.0001.

With the aim of identifying recurrent problems affecting the detection and tracking of immune cells in 2P-IVM datasets, we analyzed the videos from the LTDB (8) by means of the classical image analysis pipeline (Fig. 1A–D) and identified recurrent sources of errors. Among these, we observed that on many occasions the acquisition channels were not specific to a single cell type. This artifact comes from the combined usage of fluorophores having some degree of emission spectrum overlap, the simultaneous excitation of these fluorophores with a single Ti:Sa laser, and their detection with fixed filter sets, which together contribute to a spectral crossover in more than one detection channel. For instance, the emission spectrum of the commonly used cyan fluorescent protein (CFP) was captured by both the green channel and the blue channel (Fig. 1E). Moreover, the fluorescence emitted by cells displayed a brightness variation across the field of view (Fig. 1F). This was due to a difference in depth between portions of the field of view and non-homogenous diffraction throughout the organ. In addition, brightness variations also occurred over time. This was associated with photodamage, loss of the objective-sample interface, occasional saturation of the detector (Fig. 1G), or upon the internalization of fluorescent material, that is, via phagocytosis (Fig. 1H). Moreover, we frequently noticed the presence of autofluorescent objects in the background, such as collagen fibers (via second harmonic generation), or debris resulting from the death of cells (Fig. 1I). Altogether, these problematics hamper the automatic detection of the cells of interest, as the cells can be missed or confused with other cells or background, resulting in poor TRA.

We developed a UI (Supplemental Video 1) to assist the user during video annotation. This interface consists of a set of buttons and navigation controls designed to guide the user toward the annotation of specific elements in 2P-IVM videos of immune cells, which are problematic for the analysis when not annotated. These elements are summarized in Table I. Moreover, the UI provides two indicators regarding the quality of annotations: the balancing of the training dataset, and the coverage of the pixel distribution in the color space. Finally, the UI computes a preview of the virtual channel to elicit visual feedback in real time while annotating. By doing this, the UI allows one to maximize the improvement achieved in the new virtual channel generated by CANCOL and to overcome the common limitations associated with intravital imaging.

CANCOL was applied to facilitate the automatic tracking of immune cells in challenging 2P-IVM videos from LTDB. CANCOL generated an imaging channel that was significantly more specific for the cells of interest than the RAW channels acquired by the microscope (Fig. 2A), showing on average a 2-fold increase of the SNR (p < 0.05), and a 2 order-of-magnitude increase in the CR (p < 0.01). After that, the imaging channel generated by the proposed method was used to compute automatic tracking. The obtained tracks were significantly more accurate (increased TRA score) with respect to the automatic tracking using the raw data or the colocalization functionality of Imaris (p < 0.001). Moreover, the usage of CANCOL enabled us to generate tracks that were significantly more accurate than the tracks obtained without user assistance (p < 0.0001) (Fig. 2B). The tracks obtained with the usage of CANCOL were further compared with the tracks annotated manually by three independent imaging experts. The accuracy of the tracks generated by CANCOL was comparable to the accuracy of an average operator (n.s.), while the variance was reduced. In addition, the time needed by the user to track the videos was evaluated. The usage of the proposed tool allowed us to reduce the processing time with respect to manual tracking when videos are more complex than two cells and 100 frames (Fig. 2C).

Lastly, we evaluated the usage of CANCOL in a semiautomated tracking setup, in which tracks were automatically generated and then curated by a trained analyst. To this end, three operators independently reviewed and edited 185 tracks from LTDB004 (135 generated via automatic tracking, 50 generated via automatic tracking+CANCOL). A comparison of the time required to complete track editing with or without CANCOL showed a significant decrease when CANCOL was used (Fig. 2D). This was associated with a steeper precision versus time curve (Fig. 2E) and fewer editing operations required to achieve the maximum accuracy (Fig. 2F).

We evaluated the facilitation of tracking conferred by CANCOL in the presence of different challenges associated with 2P-IVM imaging: fluorescence variation, poor channel specificity, and a high background or the presence of autofluorescent objects.

CANCOL was tested on 10 videos from LTDB (LTDB004–LTDB006, LTDB009–LTDB014) characterized by low SNR, or by high SNR and areas with scarce brightness. A quantitative comparison using 307 tracks from these videos (Fig. 3A) confirmed that the usage of CANCOL allowed generating significantly more accurate tracks than those generated using the raw imaging data, the colocalization tool of Imaris (p < 0.001), or without user assistance (p < 0.05). As a representative example, we presented a case in which B cells labeled with CellTracer Violet underwent fluorescence intensity shifts while migrating in the spleen (Fig. 3B, Supplemental Video 2). In this video, although the SNR of the raw acquisition channels was very high (>43), the brightness of some cells decreased when migrating in areas of the organ that were less permeable to light. This variability introduced tracking errors when cells temporarily appeared with lower brightness, giving rise to multiple track fragments of short duration (Fig. 3B, 3C, blue). Conversely, CANCOL allowed the distinction of poorly visible cells from the background, thus generating longer tracks (Fig. 3B, 3C, yellow).

CANCOL was tested on 10 videos from LTDB with poor channel specificity (LTDB001–LTDB006, LTDB008, LTDB009, LTDB016, LTDB017). A quantitative comparison using 219 tracks from these videos (Fig. 4A) confirmed that the use of CANCOL enabled the generation of tracks that were significantly more accurate than those generated using the raw imaging data or colocalization of Imaris (p < 0.05) or without user assistance (p < 0.001). As a representative example, we presented a case in which the fluorescence of CFP-expressing neutrophils was captured by two imaging channels centered on the green and blue wavelengths (Fig. 4B, Supplemental Video 3). This example was particularly challenging because the video also included objects captured by the blue channel (collagen fiber imaged via second harmonic generation) and other cell types emitting in the green spectrum. Moreover, the fluorescence of CFP neutrophils further varied across the field (Fig. 4B, dashed line), resulting in an overlap between the fluorescence emitted by the different cell populations and the collagen. The usage of CANCOL enabled the generation of a virtual channel that was specific to the CFP neutrophils (Fig. 4C). Automated cell detection using a CANCOL-generated channel showed the correct detection of CFP cells with low brightness (Fig. 4D–G, white arrows) as well as the exclusion of the collagen fibers (Fig. 4D–G, red arrows). In contrast, cells were not correctly identified either using the raw imaging channel, the same pixel classifier with random annotations on the background, or the coloc functionality of Imaris.

CANCOL was tested on 10 videos from LTDB with high background and in the presence of autofluorescence objects (LTDB001, LTDB002, LTDB005, LTDB006, LTDB012–LTDB014, LTDB16, LTDB017). A quantitative comparison using 390 tracks from these videos (Fig. 5A) confirmed that the usage of CANCOL allowed the generation of tracks that were significantly more accurate than those generated using the raw imaging data or colocalization of Imaris (p < 0.05), or without user assistance (p < 0.05). As a representative example, we presented a case of study in which CFP-expressing neutrophils were imaged in an environment with multiple autofluorescent particles in the background, associated with the debris generated by dying cells (Fig. 5B, red arrows). A visual comparison between the channel generated by CANCOL and the channel generated using the coloc functionality of Imaris showed the correct detection of CFP and the exclusion of autofluorescent particles (Fig. 5C, 5D). Lastly, we provided a visual comparison between the tracks generated manually, the tracks generated using the colocalization functionality of Imaris, and the tracks generated using CANCOL in a low-magnification video. The use of CANCOL allowed the proper exclusion of autofluorescent objects and the generation of more specific tracks (Fig. 5E–G, Supplemental Video 4).

Data annotation is a critical aspect of applying machine learning methods to bioimage analysis. Indeed, the most effective tools available to date to separate the cells of interest from the background and other cells are based on brushing. However, this process is entirely dependent on the decision of the user, who may not be aware of the key elements that are needed to be annotated for the successful application of machine learning methods.

This is particularly relevant when data are analyzed by operators with minimal expertise such as new researchers or non-specialized operators performing data annotation via crowdsourcing.

To solve this problem, CANCOL assists the users during the annotation process by specifically asking them to annotate key elements that we identified as responsible for introducing errors. Among these elements, we included regions in the outer border of the cell of interest. We observed that the annotation of such regions obtained a sharper response of the pixel classifier at the boundaries of the cells. This result confirms the recently obtained high accuracy of three-class classification methods for cell segmentation (18).

Recently, approaches based on deep learning enabled the extraction of remarkable insights from microscopy data (19, 20). Among these, neuronal networks with convolutional layers such as U-NET (21) were employed for cell segmentation. However, the training of U-NET models is a complex process that requires datasets of images with all the cells of interest annotated. To overcome this limitation, transfer learning approaches were applied to detect cells using a reduced number of annotations from the user (22). With respect to these approaches, CANCOL uses hand-crafted features that may generate virtual channels with a lower SNR. However, the generated channel can be easily tuned by the user in nearly real-time, and without the requirement of deep learning frameworks. Moreover, the annotations created via CANCOL can be used along with a variety of supervised machine learning methods for semantic segmentation, including transfer learning approaches. In the context of unsupervised machine learning, instead, CANCOL can be applied to optimize the accuracy of superpixels (by providing annotations on cell borders) (23), or along with graph-based methods to deal with highly non-convex shapes (by extracting connectivity information from the annotated paths) (14).

The increased SNR of the channel generated by our method is associated with its ability to distinguish the cells of interest from the background. Higher values of SNR could be obtained if a threshold is applied for binary background/foreground segmentation. However, we chose to generate a channel with the class 1 probability, rather than generating binary masks, to optimize the performances of the subsequent spot detection and tracking. Indeed, when using a binary mask rather than the class 1 probability, the tracking performances decreased due to the difficulty of separating objects in touch using only binary values, and the impossibility of tuning results by the user. Therefore, we considered class 1 probability to be more appropriate than binary classification for the output values of the generated channel.

CANCOL was optimized to use a minimal set of features (color and Gaussian blur with two levels), which enables us to distinguish cells from the background and other objects in realistic and common 2P-IVM setups. We excluded features that were too variable among different experimental setups. These include morphological features (due to the high plasticity of immune cells and their textureless appearance) and three-dimensional features (due to possibly high differences in the point spread function along the z-axis using different objectives, microscope setups, and imaging sites).

CANCOL can also be applied to batch process multiple videos with similar characteristics, which is an advantage when the settings cannot be adjusted from video to video. This is possible by saving the training points from multiple videos and training a common pixel classifier.

Despite the use of CANCOL, which enabled us to improve TRA, residual tracking errors introduced by problems not targeted by CANCOL were observed. However, the correction of these errors via track editing required a decreased amount of time with respect to the correction of the errors introduced by automatic tracking without CANCOL. Moreover, considering that the tracking algorithm used in this study was the default of Imaris software, without any adjustment of the parameters (Hungarian linking, with autoregressive motion coefficients, any gap correction, any exclusion of short tracks), these errors could be improved by integrating into bioimaging software more advanced automatic tracking algorithms (17), which will benefit from but are beyond the scope of this work.

In conclusion, computer-assisted annotation provided an effective strategy for improving the automated tracking of immune cells in 2P-IVM. Indeed, meaningful annotations on both the cells of interest and on the problematic objects in the background were pivotal to training a pixel classifier. These results support the importance of the data annotation process for the application of supervised machine learning to bioimage analysis, which should include both positive and negative annotations with relevant biological meanings.

We thank Yagmur Farsakoglu, Miguel Palomino-Segura, Tommaso Virgilio, Daniel Molina Romero, and Mauro Di Pilato for imaging data generation and software testing, Rocco D’Antuono and Diego Morone for microscopy assistance, and Amir Davoodi for technical support in programming.

The authors have no financial conflicts of interest.