AI in Medical Devices: Why a Working Model Isn’t Enough

AI is on nearly every medical device roadmap. Getting a model to work in a demo is no longer the hard part. The hard part is everything a demo can’t show: that the product behaves safely and predictably, in a way a regulator can approve and a patient can trust.

Most teams are strong on the model. Far fewer can answer, at any point in the project, what the AI decides, who signs off on it, and what evidence a reviewer will ask for. Those questions rarely get asked during development. They get asked at the end, when the answers are most expensive to change.

Clients are asking, too. Medical device companies increasingly want to know where AI was used during development, which tools were involved, and how the outputs were reviewed and verified. Delivering working software is no longer enough. Teams must show it stays safe, reliable, and trustworthy throughout its lifecycle.

It starts with a distinction many teams skip. “AI in medical devices” covers at least two things: AI used to help build the software, and AI running inside the device, producing answers in real time. They sound similar. They are not, and the verification approach that works for one does not carry over to the other. Knowing which kind you are building is the first decision, and it shapes everything that follows.

Traditional medical device software tends to fail loudly: a crash, an error, a number that traces back to a line of code. AI doesn’t always work that way. It can return an answer that looks perfectly reasonable and is simply wrong.

That changes the question teams should be asking. “How accurate is it?” has a tidy answer, and that answer says very little about safety. The question that matters is what happens when the model is wrong, and who catches it. Better models don’t make this go away. They make the remaining failures harder to spot.

This is why a model cannot be proven safe the way deterministic software can, and why reviewers know it. What teams can prove is that the system around the model behaves safely whatever the model does. That takes deliberate choices about architecture, human oversight, testing, and post-release monitoring, made early rather than discovered late.

Teams that reach regulatory approval tend to treat evidence as something they build as they go. Kept up to date, it is routine bookkeeping. Reconstructed after the build, it becomes an archaeology project, and a slow, expensive one.

The same is true of expertise. Three kinds of knowledge decide whether an AI feature ships: clinical, lifecycle, and the model itself. Troubled projects are rarely short on talent. They are short on overlap. Getting those three perspectives into the same room in week one costs a day. Getting them together for the first time after the build can cost a rebuild.

If you already have a working prototype, there are three questions worth answering before the next sprint:

1. What decision is the AI actually making?

2. What happens when it’s wrong?

3. What evidence will you need at the end, and can you produce it as you go?

Most prototypes are silent on all three. Closing that gap early is the cheapest de-risking available, and the difference between a delay and a rebuild.

Our new white paper, From Prototype to Approval: Proving AI in Medical Devices, is written for engineering, quality, and regulatory teams who want to settle these questions early. It shows where AI belongs in a safety-critical design, what changes when a component can be confidently wrong, and what it takes to prove the system rather than the model.

For more insights, watch the full discussion on YouTube and explore the challenges of bringing AI into medical devices.

Critical Software builds and certifies software for safety- and mission-critical systems across regulated industries, including medical devices. The company supports manufacturers in achieving Software as a Medical Device certification readiness, aligning architecture, verification, and governance with FDA, EU MDR, and EU AI Act requirements. Its work spans AI medical device architecture, GMLP-aligned development workflows, and post-market frameworks such as the Predetermined Change Control Plan (PCCP), helping engineering and regulatory teams move from model performance to certifiable, scalable systems.

Building AI into a medical device? Here’s more on how we support medical device teams.