Published on Jan 13, 2025
Generative AI is transforming healthcare by introducing chatbots capable of providing instant medical advice to healthcare professionals and patients. The ability to integrate large language models with prompt layers and knowledge bases, and create functional products in hours, is revolutionary. However, navigating the journey toward market authorisation, building customer trust, protecting your reputation and managing liability is fraught with challenges.
AI-powered question-and-answer tools that draw on clinical guidelines or user manuals offer significant potential for clinical decision support. These tools can also support medical device usage or ensure safe medication administration. Under the Medical Device Regulations (MDR), many of these tools qualify as medical devices or accessories and may also fall under the EU Artificial Intelligence Act (AIA) as high-risk AI systems. This article explores the key challenges and potential solutions for bringing these AI-driven tools to market.
AI systems, particularly large language models (LLMs), can sometimes produce misleading or inaccurate information while presenting it as factual. Research shows that popular models like GPT-4, PalmChat, and Claude 2 can generate incorrect results in text summarisation tasks, with error rates ranging from 3% (GPT-4) to 27% (PalmChat), depending on the model and task.
Given that even an almost correct answer can pose significant risks, manufacturers must build guardrails into these systems and apply rigorous testing protocols. However, there are currently no universally accepted test protocols for natural language processing (NLP) in healthcare. While achieving 100% accuracy might not be feasible, the human-AI team must strive for performance levels that surpass human capabilities.
Developing and implementing robust test protocols such as text summarisation, question answering, language error correction, sensitive data treatment and voice anonymisation, are critical to ensuring the safety, accuracy and reliability of healthcare chatbots1. These protocols foster trust in the technology, driving its adoption and helping unlock AI’s full potential in healthcare.
To secure investment and customer confidence, manufacturers must build trust with users, regulatory bodies and other stakeholders. A collaborative effort is needed to establish acceptable test protocols and metrics. AI regulatory sandboxes, which are part of the AIA framework, provide an ideal platform for such collaboration. These sandboxes allow diverse stakeholders to work together under controlled conditions to develop, test and validate AI systems in real-world scenarios.
AI regulatory sandboxes also offer an opportunity for faster consensus-building that traditional standardisation processes. TEF Health, one such EU Testing and Experimentation Facility for healthcare, supports technology developers by providing access to real-world conditions, including testing with patients. TEF Health ensures intellectual property protection and collaborates with various stakeholders, including procurement teams, health insurers, and CE-certification advisors. Also, Notified Bodies under the umbrella of TÜV Verband partner with TEF Health to foresee their test capabilities needed for AIA designation.
Funded at 50% for five years by the European Commission’s Digital Europe Programme, TEF Health must become self-sustaining. To attract innovators, it focuses on high-impact use cases, and healthcare chatbots driven by generative AI have immense potential in this area.
Generative AI promises to revolutionise healthcare by enhancing chatbot capabilities, enabling efficient triage, and providing personalised patient education. However, AIA introduces obligations that require careful navigation. Collaboration between regulators, healthcare providers, and patient organisations is essential to foster trust in generative AI applications.
By working together, we can ensure that AI enhances healthcare delivery responsibly, ensuring a brighter future for patient care. With the right strategy, we can turn the promise of generative AI into a reality.
To gain a deeper understanding of medical device software and its regulatory landscape, you can explore our portfolio of specialized courses. We offer courses such as Market Authorisation of Software as a Medical Device (SaMD) and Medical Device Software (MDSW), Classification of Software as a Medical Device (SaMD) and Medical Device Software (MDSW), Clinical Evaluation of Medical Device Software, and Cybersecurity for Medical Devices. Each course is designed to equip professionals with the knowledge needed to navigate these complex areas effectively.
1 See ISO/IEC AWI TR 23281 Artificial intelligence — Overview of AI tasks and functionalities related to natural language processing. Currently under development.
Published on Jan 13, 2025 by Koen Cobbaert