Beyond Morphology: Whole Slide Imaging, Computer-Aided Detection, and Other Techniques
Michael D. Feldman · Archives of Pathology & Laboratory Medicine · 2008
Just 3 years ago, I do not think we would have entertained putting a conference like this together. We are literally on the exciting part of a development curve based around the discipline of digital imaging and it is accelerating with the adoption and development of new digital imaging technologies. We have gone beyond some of the early left-hand “S” portion of the development curve, and what you are beginning to see is exponential growth in the middle of the S curve of this new business and scientific cycle. Although the presentations today are exciting, they should also challenge you to rethink the entire paradigm of what you do in pathology and how you practice. A number of these technologies are commercially available and more will become available in the very near future and you will have the opportunity to start using them in the not too distant future.Our group has been working to develop new imaging technologies, software tools, and staining protocols. We have played around with a number of names for what we have been doing. It started out as slide cytometry and has now morphed into the term slide-based histocytometry. In its current form, the reality of it is that it is a very labor-intensive and slow process right now. So unlike a flow cytometer that analyzes tens of thousands of events in seconds, what I am going to be showing you is much slower, so I jokingly call it slow cytometry. It is not all that much different from the early beginnings of flow cytometry. Since those early days, we have gone from flow cytometry of 1 parameter on blood cells and are now able to perform 10- and 20-parameter experiments in minutes. So what you are going to see I think with slide-based histocytometry are some of the first instantiations of a nascent technology and you should realize that it is very early in a development cycle and will change very rapidly, much like flow cytometry did during its early years. What takes hours to do today will take minutes in the future.An additional observation/caveat that I like to point out is that none of the work I am going to show you is meant to be a pathologist replacement. I think the worry that pathology and morphology is going to go away is really incorrect. What we are really talking about here are tools to assist and extend the functionality of a pathologist: I liken them to “pathology helpers.” They are meant to allow you to perform measures with greater reproducibility, to do things that the human eye cannot do (multiplex antibodies), but to keep what you have all learned and trained on, the understanding of disease at the cellular and the tissue level, and to facilitate your ability to do more of it and to do it better.Now, the background of why we ventured into the development of a histocytometry application in the first place. The real use case at Penn was the need for quantitative approaches to tissue analysis in our clinical trials research efforts. There are many targeted therapeutics that are now being developed and tested in preclinical and clinical trial settings. The obvious question is whether the drug is working as planned. The classic paradigm is you look at your in vivo imaging modalities before and after a drug is given to determine if the lesion got bigger or smaller or stayed the same, and you might take a piece of tissue and then give it to a pathologist who would analyze the tissue as 3+ before and a 2+ after. Because of the imprecision and lack of reproducibility of manual reading methodologies, they left us wanting a better tool to explore to determine if a targeted therapeutic was really affecting the biologic pathway that it was designed to interpret.The literature has many studies that show that immunohistochemistry (IHC) or immunofluorescence scoring by human visual inspection is fraught with problems, including interobserver reliability, imprecision, and low sensitivity, not to mention the bandwidth problem of how many slides a single pathologist can manually score in an 8-hour day. Frankly, there are more clinical trialists at Penn than there are pathologists, and they can enroll folks faster than I can cut, stain, and read the slides.For purposes of presentation, it is helpful to think of histocytometry as a 7-step method. I will point those steps out to you where the technology is working and moving very quickly, as well as some of the gaps where engineering and science breakthroughs have to come through. Some of these problems are not as easy as they seem.We decided to start out with brightfield because we could not get some of the proteins we were studying to work by immunofluorescent techniques. Some of the antibodies just do not work as well using fluorescence as a reporter in our hands, and we have not been able to overcome some of those limitations. We know we are going to eventually have to get to the fluorescence for higher order multiplexing but for now we are using brightfield IHC.For instrumentation, we have standardized on the Nuance platform from CRI instruments (CRI, Inc, Woburn, Mass). Our collaborator at Rutgers University (Piscataway, NJ) (Anant Madhabushi, PhD) has developed image-processing algorithms for the tumor segmentation. A second collaborator at Rensselaer Polytechnic Institute (Troy, NY) (Badri Roysam, PhD) is developing machine vision algorithms for cellular and subcellular segmentation events. Together these image-processing tools will allow us to find tumor regions within a field of view and then colocalize one or more antibody signals to a cell as well as subcellular localization for discovery of biologic information. Our collaborator at Drexel University (Philadelphia, Pa) (Youngmoo Kim, PhD) has developed data processing algorithms for analysis and display of the cytometric data borrowing from a familiar paradigm of flow cytometry because most pathologists and biologists are comfortable with this form of data display.Other novelties of the process we have developed include background subtraction. Do any of you in your routine practices of IHC do background subtraction? In routine practice, the ability of the human operator to perform background subtraction to determine what is real signal and what is nonspecific signal is very subjective. We have developed algorithms that allow for background subtraction based on cytometric measurements.Our studies have used brightfield IHC. The staining protocols require sequential staining with blocking in between each primary antibody sequence, followed by additional rounds of antibody staining. Some of these stains will colocalize. Currently, colocalization is one of the big limitations. Staining for 3 targets within the same spatial compartment and spectrally unmixing these 3 antibodies has not been possible in our hands at the current time.Our current system allows staining and spectral unmixing of 2 antibodies in a compartment as well as allowing us to stain multiple compartments at the same time. So we can do 2 on a membrane, 2 on a nucleus, 2 on the membrane, and 2 on the nucleus. That gives you a pretty significant ability to begin interrogating biologic pathways in the context of the morphology.Trying to perform that type of multiplexed staining with the human eye is futile. It appears as a muddy mixture of colors that are hard to visually separate and impossible to quantitate. You are at best guesstimating. This is one example of where histocytometry acts as a pathology helper and extender by providing you the pathologist with the ability to do things that you would not be able to do otherwise.Different chromogens that we all use in our laboratories actually all had different spectral profiles and that allows you to take one of these multiplexed, immunohistochemically experiments and separate the individual chromogens once you know what the pure color spectra looks like for each of the chromogens (Figure 1).Once you have done your stain and have spectrally extracted information, how do you localize that information to the targets of interest? Right now, we are dealing with manually drawing in areas. Here is the tumor area. Here is the lymphocyte area. Richard Levenson, MD, showed you MIDAS, which is a machine-learning algorithm that Cambridge Research and Instrumentation, Inc (CRI, Inc) has been developing. We have been developing some other machine-learning algorithms for identifying tumors and I would like to walk you through just a little bit of a different approach. We have been using a Bayesian classifier model. This is work that starts out with a completely digitized virtual slide from Aperio (Vista, Calif) scanning system, then recapitulates the process that human pathologists go through when we look at a slide. We extract information at different magnifications and then different information at each of those magnifications to create a large information space, which we then use manifold learning to reduce and classify the information into tumor or not tumor. For example, in prostate cancer, we cannot only find areas with cancer but we can apply a Gleason grade to identified cancer (Gleason pattern 3, Gleason pattern 4, and Gleason pattern 5).Together, using machine vision, we can identify tumor regions and once they are found can then colocalize antibody signals to cells and subcellular regions within found cells. I am not suggesting we remove the pathologist from this process. At the end of any machine vision process, once the software and hardware have performed their function, the pathologist will be presented with a view of data to allow a review of the information, much like we do in flow cytometry. What the system has done is to remove the pathologist from the grunt work of having to do all of the manual segmentation work and manual scoring. This is not so foreign a workflow for the pathologist. We do this all the time in hematopathology, why not adopt a similar mechanism in surgical pathology.This is what you do when you submit a sample for flow cytometry. Pathologists do not walk to the flow cytometer. They give it to technologists. Technologists run it and segment it. They put the thresholds down. Then hematopathologists get the results and say, you got close but move this one to the right and this one up a little bit. I am talking about being able to provide the pathologist with data that has been predigested and then fed back for subject matter, expert review, and interpretation.This is just a quick walk-through of that approach of using machine learning in a way that recapitulates what we know works in the human mind. When you look at a prostate biopsy, you do not look at every little area at ×40. You spend most of your time at ×2 or ×4 and within moments, you know exactly where the important areas are that you need to examine at ×10 and maybe even ×20. In addition, you quickly decide which slides need further immunostaining to confirm a minimal cancer.That is exactly the approach we are taking with the machine learning. With a fully digitized slide, I can represent the information at different magnifications, and I can quickly focus the algorithm to only look at certain areas of the slide, where there is high probability. We extract the information using scale-based representation. So at very low magnifications, we ask where is the highest probability in those images of cancer being found. We identify those areas based on shape, texture, and color and then only pass those regions up to the higher magnifications where we apply a different set of filters and a different set of textures, to come up with a different feature space. This is exactly what you do with your microscope. We just apply a different set of features. You have trained your optical brain on this, but what we have to do is tell the computer, here is a whole bunch of different metrics, and then go in with the machine learning and say, how do those separate out.We extract hundreds if not thousands of features from the low power and then classify those into cancer or not cancer. Once that is done, we then move up to higher scale, ×4, ×10, ×20, and then go back and say okay is there more information here that then allows me to classify the images into grade 3, grade 4, or grade 5.The end result is a machine classification algorithm, which allows more reproducibility in Gleason pattern recognition with lower interobserver reliability than between pathologists doing the same work in a manual mode. With just a limited data set of a few dozen virtual slides, there is about 90% to 95% reliable between patterns 3, 4, and 5.We think this approach can be generalized to different diseases, much like MIDAS would be. You can train it on a hiker image, or you can train it on a prostate cancer image, or you can train it on breast. I think what you are going to see out of the machine-learning space is that for each of these different diseases, there are going to be algorithms developed, not perhaps 1 model that can be generalized but disease-specific algorithms that can be plugged in for analysis of different disease types. These different machine models will be applied to do the segmentation routines. What is the best algorithm remains to be determined. We are at a very early stage and frankly we can all learn from each other. I have no idea which will be best. Will it be MIDAS, our Bayesian approach, or maybe a hybrid between the two? Maybe there are some features that we use in our Bayesian algorithm that might be added to the MIDAS algorithm CRI uses.The upshot of this is that I do think machine vision is going to be able to provide a high quality of robust segmentation. I do not think it is ever going to be perfect. I do not think it has to be perfect. What do you mean it does not have to be perfect? How could you produce diagnostic information on something that is not perfect? Well if something is 95% accurate and I miss a few nuclei, who cares? As long as I capture enough nuclei to produce a statistically meaningful result that is what matters. You do not have to overengineer something to have the device and the software to be highly functional and useful.After either machine vision or manual segmentation has identified the tumor area, we use a separate set of algorithms to identify nuclei. We have a separate segmentation algorithm to identify cell membranes next to those nuclei. Once you have identified the nucleus and the cell membrane, you have got the cytoplasm by subtraction. That combination of elements (cells, nuclei, membrane, and cytoplasm with intensity information of stains for each region) is exactly the type of data that you need to generate a data file that can be graphically represented like a flow cytometer does.Figure 2 is a breast cancer slide stain for phosphorylated ERK. After it is gone through the image segmentation, finding all the nuclei, with manual tumor segmentation, the data can be converted into a histogram display (Figure 3). Instead of 1+ and 2+, we have actually got 15 different bins here and a much finer gradation of intensity values to compare.Figure 4 is another example of what you can achieve analyzing events with a single antibody. These are data from a clinical trial that is ongoing at Penn, looking at a drug that is targeting mitogen-activated protein (MAP) kinase pathway. What we want to know is, did this drug work? Seeing whether the tumor got bigger or smaller by in vivo radiology imaging is only a surrogate of that. What you really want to know is, is the MAP-kinase signal active before you give therapy, and after you give therapy, did you affect it? You want to be able to directly measure some surrogate marker within that MAP-kinase pathway to know whether you did something. This is phosphorylated ERK before therapy and then on therapy. I mean you can see that there is a very dramatic difference and we can use this as a way of monitoring.What we are beginning to do is to create a system that allows us to provide quantitative reproducible scoring system for one or more antigens colocalized to subcellular compartments. It is based on cytometric detail and it is independent of the bias we all bring to the table when we read slides.Dr Levenson showed you colocalization of estrogen receptor and progesterone receptor. We can also add HER-2 to this analysis, but what does it mean? We are submitting a grant proposal to address that question. It is going to be a HER-2, estrogen receptor, and progesterone receptor multiplex linked to 5-year outcome data in a cohort of breast cancer patients. Because HER-2 status does not always predict who is going to respond to trastuzumab therapy (40% response rate in HER-2–positive patients), we can then ask the question in responders versus nonresponders of why by dissecting signaling pathway downstream of HER-2 (Figure 5).There are some other applications for which we have used the spectral imaging. Several researchers have been interested in looking at the prognostic significance of lymphovascular invasion. To study lymphovascular invasion, we use a triple stain to look for tumor cells in one color. Lymphatic vessels (podoplanin positive) are shown in a second color and blood vessels (CD34) in a third color (Figure 6). The offshoot of this is that we have a much more sensitive stain now for identifying lymphovascular events. We also discovered, because we could look at the blood and vessels in that not too vessels for This is tumor and case and from to This finding that many studies that at as an marker have been in studying not just blood vessels but also of the early data using to study is now that we can identify on of know we need to from brightfield IHC to immunofluorescence in the future to be able to perform higher order What is the best way to use immunofluorescence for What are the best to use for multiplexing are a are not the by because the way we use right now is an You cannot go and put on your primary antibodies that have antigens and to be able to see You can get away with antigens as but the antigens like the signal are just not enough to see with directly are to find that can be directly to antibodies to move to immunofluorescent to one of the that spectral imaging is the ability to get of that This is important because it the signal to by 2 to 3 is from some research we were doing on looking at the of in The in the is but once you get of it you can really see where the up The ability to this should not be of can the signal to by or very become is the We are with a of at Penn, and we are other developing different The first one who something out is going to have a in to these will be spectral imaging and tissue and machine vision tools are going to become really significant in the With all of these new targeted they all represent to a with a drug to predict outcome and therapy. Our clinical trialists are very by this new which with and reproducible staining As we move this technology we will be able to study more than one pathway at a time. being able to study or more pathways that with one another and them to cell has been one of the most I have been with at of the things that have it is not just that it is but the that I have This is a really large We have clinical vision at Rensselaer Polytechnic a signal processing expert at vision at and our and at pathologists, we be of our eye too much on the We are too much on the and not enough on and who can really do this into our The have us to the radiology have and vision has to start doing this as are but it is hard to work in a with different out between and It it when it but it takes a of I think we need to start that back into our and our This is going to be a real challenge for the to start into our pathology I think it is if we are going to be with applied