Uncategorized

Q: What problem were you aiming to address with this research?

Most current evaluations of artificial intelligence (AI) / automated retinal image analysis systems (ARIAS) are carried out by the vendors themselves, often using datasets and testing environments of their own choosing. These single-algorithm, vendor-delivered studies tend to overestimate performance and rarely reflect how systems might behave in real-world clinical settings. Differences in patient populations, image capture systems, and evaluation protocols make it nearly impossible to compare algorithms fairly particularly when key details like pre-selection or preprocessing of images are unclear. This approach limits transparency, comparability, and ultimately, trust in AI-driven healthcare tools.

Q: How did your study take a different approach?

Recognising these shortcomings, this study took a fundamentally different path. It created a vendor-independent evaluation platform using the largest, most ethnically diverse NHS Diabetic Eye Screening dataset in North-East London. Multiple state-of-the-art ARIAS each certified as a medical device (with/pending CE Class IIa) were assessed on the same dataset, under identical computational conditions. This ensured fair, direct, and reproducible comparisons while also testing for algorithmic fairness across diverse population subgroups (e.g. ethnicity and age). The result is a transparent, sustainable model for evaluating medical AI systems that mirrors real-world use.

Q: What are the implications for policy and future practice?

This work sets a new benchmark for independent, population-based evaluation of AI in healthcare. Demonstrating how multi-vendor comparisons can be done impartially and at scale, in the intended healthcare setting. It provides a blueprint for regulators, policymakers, and health systems to demand higher standards of evidence before deployment. It provides a more predictable adoption pathway for manufacturers. Beyond eye screening, the same principles could inform policy frameworks for AI evaluation across other disease areas, helping ensure that future health AI technologies are not only effective and safe for patients but also equitable and trustworthy.

Read the full research paper here: Automated retinal image analysis systems to triage for grading of diabetic retinopathy: a large-scale, open-label, national screening programme in England – The Lancet Digital Health

This study exemplifies the type of rigorous, independent research that CERSI-AI champions. By setting new standards for transparency and fairness in AI evaluation, it helps pave the way for safer, more equitable adoption of digital health technologies.

Want to learn more? Explore our latest projects, policy insights, and collaborative opportunities at CERSI-AI and join us in shaping the future of trustworthy AI in healthcare.

Highlights included:

• Stakeholder prioritisation sessions revealing critical regulatory needs which CERSI-AI can address.
• A showcase of HaRi – Hardian Health‘s Regulatory Intelligence database for AI-enabled medical device transparency.
• International insights from Joseph Ross (Yale University/Mayo Clinic CERSI).

The discussions reinforced that sustainable AI regulation requires collaboration across the entire healthcare ecosystem. Over the coming weeks, we’ll be sharing key insights and recommendations from these important conversations. A huge thank you to all speakers, facilitators, and attendees for your energy, ideas, and commitment. We look forward to continuing the journey together.

CERSI-AI Director Professor Alastair Denniston has been filming with The Lancet Group, discussing how AI and machine learning are already transforming healthcare and research.

It’s a fascinating conversation about where we are now, and what’s next. We can’t wait to share the full recording with you this September!

Co-hosted by the ARIAS Research Group, the NIHR Incubator for AI and Digital Healthcare

• Current evidence on human grader QA standards
• Results from a recent head-to-head evaluation of multiple ARIAS tools
• Key requirements to support safe, evidence-informed implementation in the NHS

Our goal was to share insights on performance, safety, and regulatory needs, and to collaboratively shape early recommendations for implementation pathways.

Thank you to everyone who contributed to such a constructive and impactful discussion, we’re excited about the next steps!

Images and LinkedIn post here: https://www.linkedin.com/posts/cersi-uk_aiinhealthcare-nhsinnovation-diabeticeyescreening-activity-7357041126198894594-LYpK?utm_source=share&utm_medium=member_desktop&rcm=ACoAAFRfVxcBHqGcyb7Pe-QXQWWRVZ1T7jiZA-U

Funded by The Health Foundation and supported by CERSI-AI, we brought together regulatory experts from the Medicines and Healthcare products Regulatory Agency, notified bodies, and AI verification specialists to explore one of digital health’s most urgent challenges.

Together, we analysed real-world LLM use cases across healthcare, mapped regulatory pathways, and assessed device classifications and risk levels under existing frameworks. The conversations highlighted both the practical hurdles and exciting opportunities facing innovators and regulators alike. A huge thank you to everyone who attended and contributed to the discussions, and to our brilliant team for helping to shape such a thought-provoking and productive day.

This work reflects our commitment to enabling smarter, more adaptive regulation that supports safe and effective innovation in digital health – a commitment we’ll continue advancing through our ongoing collaborative workshops.

Hosted by the University of Birmingham and University Hospitals Birmingham NHS Foundation Trust‘s AI and Digital Health Research & Policy group, with support from NHS England and The Health Foundation, the workshop focused on four core aims:

• Characterise the organisational risks and opportunities of AI adoption
• Develop a readiness framework grounded in real-world NHS contexts
• Co-design practical tools to help provider organisations assess and build readiness
• Signpost existing tools, frameworks and evidence that can support NHS organisations on their AI journey

The day included implementation case studies, collaborative risk-mapping, and lively group discussions on what success should look like, all rooted in NHS realities and future ambitions.

A huge thank you to our Chairs Jeff Hogg and Alastair Denniston, our brilliant speakers Robin Carpenter, Andy Mayne and Kevin Percival who shared valuable lessons from across the system, and to everyone who contributed their time, insight and expertise.

This work is a vital step toward supporting safe, responsible and scalable AI adoption in the NHS. We’re looking forward to sharing the project’s outputs as they develop.

“The system of regulation must keep up with that, so that we can quickly and safely seize the economic and social benefits that new innovations could unlock…that is why we are launching CERSIs. They will make a valuable contribution to regulatory innovation – and will complement wider efforts to make the UK’s regulation fit for the future…”

Prof Denniston, Executive Director of CERSI-AI, said “Working across the health and technology ecosystem, the Centre will identify and address current and future needs and opportunities in the regulation of AI and digital healthcare products and services … Key to the Centre’s role will be balancing the needs of innovators, such as speed and market certainty, with those of the end-users, such as cost-effectiveness, safety, equity and sustainability, to ensure resulting technologies are able to truly improve people’s lives.”

CERSI-AI brings together a distinguished consortium of founding partners—including the University of York; industry innovators Hardian Health, Newton’s Tree, Romilly Life Sciences, and the Association of British HealthTech Industries; and NHS organisations like University Hospitals Birmingham NHS Foundation Trust and NHS Greater Glasgow and Clyde University of Birmingham, newtonstree.ai & birminghamhealthpartners.co.uk.

Guided by Executive Director Professor Alastair Denniston, the Centre aims to balance the pace of innovation with rigorous regulatory science—ensuring technologies are safe, effective, equitable, and sustainable, while supporting innovators with rapid development and regulatory clarity University of BirminghamUK Research and Innovation.

Part of a wider UK initiative, CERSI-AI is one of seven new CERSIs funded through a collaborative effort led by Innovate UK, the MHRA, the Office for Life Sciences, and the Medical Research Council. Collectively, they will generate tools, frameworks, and guidance to streamline the pathway for medical innovation in areas ranging from advanced therapies to digital health UK Research and Innovation University of Birmingham.

Full article here: New national Centre of Excellence in AI and digital health awarded to Birmingham – University of Birmingham