World-First Artificial Intelligence study being delivered at Hampshire Hospitals NHS Foundation Trust

Interviews with:

PAIR 1 is a retrospective study looking to examine the effectiveness of Artificial Intelligence (AI) software in the analysis of Magnetic Resonance Imaging (MRIs), and subsequent diagnosis, for prostate cancer. It is the first large study to be sponsored by Hampshire Hospitals NHS Foundation Trust (HHFT).

Collecting and analysing data from eight hospitals across the country, we caught up with the PAIR 1 Study Team to hear about the goals of the study and their experience of leading a multi-site trial.

ANTONY RIX: The study came about more than two years ago when we were discussing the challenge of validating the new type of AI software that Lucida Medical is developing for prostate cancer. It’s especially difficult to get hold of real-world data that’s representative of how systems like this would be used in the NHS, but crucially important that the software is validated.

Richard and Aarti suggested that HHFT would be interested in this research and might be willing to sponsor it, with Lucida covering its costs. We supported them with the design and application process, collaborating with Tory and Clare to make use of their Research and Development Expertise. Once the study was approved, we worked with Aarti, Richard and Sophie who reviewed 300 patient cases to extract scans and data, with the other eight hospitals doing the same.

This data allows us to do a headto-head comparison between how a radiologist and the AI software score the MRI scans. We take the biopsy as the absolute truth and look at patient follow up to see if the patient came back for further investigations. We want to know how well the AI algorithm holds up to the yardstick of the original radiologist. At the moment, the radiologist sensitivity is close to 100% and the software is around 90-95%.

AARTI: In the short time I’ve been a consultant, the way we use MRI and diagnostics for prostate cancer has changed dramatically. We have good data now that shows that you have to place MRI at the beginning of the prostate cancer diagnostic pathway for every patient with suspected prostate cancer. That’s a huge change.

RICHARD: The diagnostic pathway is complicated for prostate cancer detection and we also use MRI for monitoring. We are very busy and if we can get consistent high-quality reports in a timely manner, it will help with diagnostic pathways, treatment planning and monitoring.

If all radiologists need to do is look at a report and sign it off, that’s good all round. The software can generate a very detailed report with images and, because we don’t all have the same standard of scanner, it has the potential to turn a sub-optimal scanner into a slightly better one.

AARTI: The pressures on these pathways are immense. So that’s why we’re trying every small thing we can to speed the process up. The need to have good MR imaging and reports that help to select which patients will benefit from further investigations, including biopsies, and which would be best left alone, is crucial for us. What we’re trying to do is score each scan against a robust criteria which is central to our decision making. The thing that’s quite clever about the technology is that everyone’s populations are very different, so you can set thresholds for all the results that can flag potential cases for further investigation. So, you can decide as a hospital whether you’re going to take a very cautious approach based on your local population data or you can decide whether you will target the more aggressive cases. We don’t want to end up over-diagnosing prostate cancer in older patients where it won’t necessarily impact on their quality of life.

RICHARD: There’s a natural variation between radiologists and their reporting so it will hopefully improve consistency, the quality of reporting and save time – potentially from 40 minutes looking at scans to just five minutes looking at the report and recommendations.

ANTONY: There’s no intention or evidence that AI will replace radiologists – this study will be one of the first in the world to evaluate how accurate the software is with real-world data. Then hopefully, on the basis of this, it will be able to set thresholds that hospitals might use to speed up the diagnostic process. The study will hopefully provide us with some guidance on that, workload prioritisation and triaging driven by the algorithm.

AARTI: No scan is the same, for example people may have a hip replacement and we can’t yet use AI for this. There will always be ones that the software struggles with which will need our input.

CLARE: Doing a retrospective trial has been a massive change to the usual clinical trial with patient participation. It’s allowed us to be quite functional, it allows Sophie to come in and work at weekends or at night, there’s no consent, no reassuring patients – it’s been a data gathering exercise which is really different. The biggest challenge for the coordinating team was the variation in process between different NHS trusts, which was staggering. It was so complicated around permissions and information governance teams, R&D teams, IT teams and making sure they were transferring their Dicom images in the right format. But we got there and because there were solid processes, SOPS and instructions behind everything from Lucida, the study did then fly.

AARTI: It is not always easy to do research in the NHS particularly involving large volumes of data. But the power and the future of the NHS is in the amount of data we generate, particularly in imaging. We are very behind in how we translate that into making research happen. A lot more could be done to help people participate in research such as this.

Game-changing research processes 

SOPHIE: We were able to get through the data quickly as it already existed. This helped hugely with the trial set up. If we’d had to collect 300 patients’ data from the start of the trial, it would have taken us a couple of years.

CLARE: We just entered date parameters and sites assessed each patient file on their suitability for the study.

ANTONY: The study isn’t complete yet but because it’s been providing progressive amounts of data, we’ve been able to do a number of interim analyses that are showing very exciting results. I found it uplifting to be sat down at a conference a few weeks ago with one of the leading radiologists internationally in prostate MRI. He had a look at the results we were presenting and suggested that we strengthen our conclusions, as the software is matching the performance of expert radiologists. That’s quite exciting.

CLARE: It was a game-changer to realise how much you can do on Teams. I’ve spent nearly two years working with people from Cambridge, Cornwall, Oxford and London – I hadn’t met Aarti until about a month ago. The connectivity that we have achieved, I think we were all surprised at the ease with which we developed a comfortable working relationship. Taking it forward within the NHS, the fact that we’ve got Teams and NHS staff can now work from home – that is ground-breaking for hospital and multi-site studies.

AARTI: Yes, previously you’d have had to do all the site visits in person – I think it’s the new face of how research will be done.

CLARE: This was the first study HHFT has done that had an Electronic Trial Master File so we have no paper. That gave me palpitations at first but it’s the way forward – it saves so much time. We all need to start coming up that learning curve really quickly around what is data, what will be archived and where will it be stored? I wouldn’t do another paper-based site file.

ANTONY: 18 months ago we made a decision to go for electronic CRF and electronic transfer of files. There is no way we’d go back to paper. For the NIHR, it’s a resource that everybody doing this type of work should have. We’ve created processes because there wasn’t an existing infrastructure in place that we could use. It could be easily and cost-effectively implemented on a national scale. There’s a lot that could be learned from here.

The study has created a fantastic data resource across eight NHS trusts with over 2,000 patients processed. With this we can look at the patient journey through to diagnosis and look at a lot beyond the narrow scope of AI. We would love to see our software being used. But I hope from the clinicians’ side, it will help ease the pressures of resourcing, staffing and variations in consistency – helping with standardisation and automation and making diagnosis more accurate wherever you are.

RICHARD: If it rolled out countrywide from all our work at HHFT it would be particularly fulfilling.

The PAIR 1 study began at Basingstoke in August 2021 and data transfers concluded at the end of December 2022.