Are image analysis algorithms for digital pathology compliant with regulatory requirements?
Image analysis (IA) as part of digital pathology has been around for a while. In fact, the first systems for remote pathologic diagnosis (video-based) were demonstrated in the USA in 1968. Since then enormous advancements in technology have been made, but the penetration of digital pathology applications in the clinical space remains limited.
One reason is compliance with regulatory requirements. The tissue image analysis space for research is pretty dynamic, and companies competing with one another use different software solutions to support pathology. Many pharmaceutical companies have their own computational pathology divisions that support their drug development.
But how many offered solutions and developed image analysis algorithms that met regulatory standards? How many will make it into the clinic and improve healthcare? And is this even the intention?
It all started with cytology
The Papanicolaou test (also known as the Pap test, Pap smear or smear test) is the most widely used screening method for uterine cervical cancer in women worldwide. It consists of an evaluation of a cytology smear from the uterine cervix in search of morphologically abnormal cells indicative of precancerous lesions or cancer. It is performed in women 21-65 years of age every 3- 5 years if the results of the tests are normal. In the US only, ca. 50 million women receive the test every year.
With so many patients to screen, it’s no wonder doctors and scientists started to search for a more efficient method of evaluation than visual assessment under the microscope, in which cytotechnologists and pathologists are intensely involved. According to the Clinical Laboratory Improvement Act (CLIA) they should not exceed 100 slides a day, as above this number the accuracy of the diagnosis decreases significantly.
The work on automating the Pap test started in the 1950’s, but it wasn’t until the 1990’s that the first commercial computer-assisted pathology solutions based on image analysis algorithms received FDA approval. The approved devices included the AutoPap 300 QC (NeoPath, Redmond, WA, USA) and the PapNet (Neuromedical Systems Inc., Suffern, NY, USA).
The PapNet was based on neural network processing and pattern recognition, but the company went bankrupt and the device was discontinued. The AutoPap remained and is now sold by BD as BD FocalPoint™ GS Imaging System, but is still only allowed to screen 25% of the slides automatically without the necessity for human revision.
Nice, but not too impressive (I personally would expect a lot more in the digital era we are living in, but the return on investment must not be so great if it didn’t really take off.)
Tissue-based biomarker quantification
Oncology is the hottest area in drug development. Some of the cancer drugs rely on tissue biomarkers quantified by immunohistochemistry (IHC) for patient stratification. This method is rather qualitative than quantitative, nevertheless quantitative pathology scores relying on visual estimation of the percentage of positivity and intensity of staining of tumor cells have been developed.
One would think that to avoid adding the variability of pathologist’s interpretation to the process, consisting of tissue processing, IHC assay, and interpretation, biomarker quantification would be a perfect field for applying image analysis and make the quantification objective and reproducible.
However, of the five predictive IHC biomarker tests approved by the FDA as companion diagnostics, only one – human epidermal growth factor receptor 2 (Her2) for breast and gastric cancers have image analysis quantification algorithms, has been FDA cleared. The fact that most of the scoring guidelines rely on visual estimation (or better said “guestimation”) of the amount of positive tumor tissue should be an argument strong enough for the quick development of reliable image analysis solutions. The task of estimating the amount of positive stain in an image simply is not a strength of the human brain. We are very good in recognizing different patterns and structures, but miserable at quantifying them, especially around the cut-off values, where inter-observer variability is extremely high and may lead to misclassification of as many as 50% of the patients.
Apart from the predictive biomarkers which are part of companion diagnostic tests, there are a few other biomarkers used in the clinical practice, which quantificationbenefits from the use of an image analysis algorithm.
Research use only
Many of the pathology supporting algorithms will never enter the regulated environment, meaning they will never comply with the strict regulations. They will remain for research use only. This is not necessarily bad, and it doesn’t mean that there should be total freedom in their design and their interpretation. As with each scientific method supporting bio-technological discoveries, image analysis for IHC quantification should always be tested for its specificity, sensitivity, and accuracy, before data is generated.
What about preclinical?
In the preclinical phase of drug development, the drug candidate compounds are evaluated for their toxicological safety in animal models. The number of candidate compounds is high, resulting in a very large number of histology slides needing evaluation. Recently the toxicologic pathology community started tapping into the potential of tissue image analysis to accelerate some aspects of the toxicologic safety assessment. The processes are conducted in compliance with principles of Good Laboratory Practice (GLP), so image analysis algorithms would need to be validated according to these principles.
The advantage is that the algorithms do not need to have the status of an FDA approved medical device to be used, so there is no submission process for the validation. Image analysis algorithms or software packages are considered a part of a digital pathology system consisting of IT infrastructure, Whole Slide Scanner, image acquisition system and annotation software. Together in the GLP environment the components constitute a computerized system and should be validated as such.
The extent of the validation always depends on the intended use of the system, and when image analysis for histologic data generation is the intended use the validation needs to be extensive. This is a difficult process involving several teams in the organization: the system owners, the end users, IT and quality assurance teams. Any well-functioning image analysis solution can be validated. However, since the validation process is very time and resource consuming a clear added value of the image analysis algorithm for the preclinical pathology evaluation must be demonstrated in advance, in order to engage all responsible parties and achieve support from the organization management.
As image analysis becomes more ubiquitous in the toxicologic pathology world and as algorithms mature the incentive to validate will be greater and we will be able to benefit from the image analysis speed, precision and objectivity also in the GLP compliant environment.
A longer version of this post originally appeared on Digital Pathology Consulting (https://digitalpathologyconsulting.com/are-image-analysis-algorithms-for-digital-pathology-compliant-with-regulatory-requirements/) a blog Dr. Zuraw launched to connect pathologists and computer scientists who are designing image analysis algorithms for pathology images.