An overview of the first episode of NEN's podcast on AI in healthcare
AI keeps turning up across everyday care, from ChatGPT in a client case to tools that summarise consultations or read radiology images. That raises a practical question for the people building and buying these tools: when is an AI application more than a useful aid, and when do the rules of the EU AI Act apply?
Our CEO & co-founder Leon Doorn took this up in the first episode of NEN’s new podcast series on AI in healthcare. Host Erik Peekel spoke with Leon and Cécile van der Heijden, a lawyer at Axon Lawyers and specialist in AI regulation, working through where the line sits.
What decides whether the rules apply
The purpose an AI tool is put to, how it is used, and the risks that come with it determine which rules apply. The technology itself is only part of the assessment. The same application can fall under the AI Act (Regulation (EU) 2024/1689) or the Medical Device Regulation (Regulation (EU) 2017/745) in one situation and sit outside them in another.
The episode works through examples that most care organisations will recognise: ChatGPT used in a client case, AI transcription tools, symptom checkers, logistical analyses, and image recognition in radiology. Each one shows how the same technology is judged differently once you look at what it is for and what could go wrong.
For manufacturers, there is a further layer. Under the AI Act, a medical device that already needs a notified body for its conformity assessment generally also counts as a high-risk AI system. That places the AI Act’s obligations on top of the MDR or IVDR, rather than replacing them.
Start with an overview of the AI already in use
For care organisations, the speakers say the first step is getting a clear view of the AI tools already running across the organisation. Only once that picture exists can sensible arrangements be made for responsible use, data processing, and oversight.
They also stress that professionals need to stay critical of what AI produces. AI can support a decision, but it does not take over professional responsibility. Any output has to be read in the context of the patient and the clinical situation.
Responsible use takes more than compliance
Responsible use rests on clear internal agreements, enough knowledge among the people using the tools, and a sound grasp of the applicable rules, alongside meeting the letter of the law. This is also where standards earn their keep: Leon chairs the NEN working group on AI in healthcare and medical devices, and standards give organisations a worked-out way to show they meet the AI Act’s requirements. We covered one such standard in our post on prEN 18286.
Listen and read more
For more on how these rules interact, see our posts on the AI Act’s move from Section A to B and the legal uncertainty facing AI/ML-enabled medical devices in 2026.
MedQAIR helps manufacturers work out where their software sits across the AI Act, MDR, and IVDR, and what that means for their documentation. If you are weighing up how these rules apply to your product, our regulatory strategy team can talk it through, or you can get in touch.
