Summary
Medical Device QMS requirements are evolving as third-party large language models (LLMs) become part of AI-enabled medical devices. While these models can accelerate innovation and introduce new capabilities, they also create quality management challenges that extend far beyond software integration. Manufacturers must now consider how supplier relationships, documentation, validation, cybersecurity, and post-market monitoring fit within an established Quality Management System.
As regulatory frameworks continue to mature, organisations developing Medical Device AI solutions are expected to demonstrate greater oversight of third-party AI providers. The European AI Act introduces new obligations around technical documentation and supplier cooperation, while existing medical device regulations continue to require manufacturers to maintain full responsibility for the safety and performance of the final product. This means that relying on an external LLM does not reduce a manufacturer’s regulatory responsibilities – it expands them.
A robust Medical Device QMS should therefore address supplier selection, contractual agreements, software lifecycle management, verification, validation, and change management with the unique behaviour of LLMs in mind. Unlike traditional software components, LLMs may change over time, produce non-deterministic outputs, and receive updates from external providers. These characteristics require manufacturers to establish appropriate acceptance criteria, regression testing strategies, and procedures for monitoring performance throughout the product lifecycle.
The article also explores how standards such as ISO 13485 and IEC 62304 apply when integrating third-party AI models, together with the importance of treating these models as Software of Unknown Provenance (SOUP) or Off-the-Shelf (OTS) software where applicable. It further discusses supplier documentation, cybersecurity expectations, post-market surveillance, and the growing relevance of Predetermined Change Control Plans (PCCPs) for managing future model updates without compromising compliance. These considerations are becoming increasingly important as Medical Device AI technologies continue to mature and regulatory expectations evolve.
Ultimately, an effective Medical Device QMS is no longer limited to internal development processes. It must also provide the governance needed to manage third-party AI providers, respond to frequent model changes, and maintain confidence in device performance over time. Organisations that proactively update their quality management processes will be better positioned to adopt AI responsibly while meeting both current and emerging regulatory requirements.
Read Leon Doorn’s full article, “Third-Party LLMs in Medical Devices – QMS Implications,” for a detailed discussion on regulatory expectations, supplier management, verification strategies, and practical recommendations for integrating third-party LLMs into compliant medical device development
