AI in Medical Devices: Takeaways from Agentic AI Summit Boston

At this year’s Agentic AI Summit Boston, more than 500 engineers, architects, and technology leaders gathered around a single question: How do we move AI agents from impressive demos into systems we can actually trust in the real world? 

Across industries, from software engineering and finance to biopharma and healthcare, the conversation has matured rapidly. Early enthusiasm for "chatbots that save a few minutes" is giving way to a much deeper engineering reality. As a framework from the Stanford HAI 2026 report, highlighted at the summit, put it: “Time saved is an option, not a cash saving.” In healthcare and medical devices, true value isn't measured by how fast an AI generates text; it is measured by effectiveness: better patient outcomes, safer clinical decisions, and faster, audit-ready product validation. 

Looking ahead, one conclusion stood out above all others. When every company has access to the same frontier AI models, the model itself is no longer a differentiator. Instead, the organizations that lead the next chapter of AI (especially in high-stakes sectors like Medical Devices) will differentiate themselves through three core pillars 

One of the biggest myths in AI today is that larger context windows, feeding more raw documents or code into a model at once, will solve complex enterprise problems. Engineering benchmarks presented at the summit showed that answer quality drops sharply once context grow beyond roughly 300,000 tokens. Why? Because raw text is linear, while real-world systems are relational. 

What will set successful implementations apart is how organizations structure their proprietary data and build domain ontologies. An ontology is more than a database. It is a structured map of relationships, rules, and domain meaning. It might map how a medical device’s embedded software modules interact with its hardware sensors, or how clinical risk controls (ISO 14971) link to software requirements (IEC 62304). Either way, a verified ontology turns guesswork into precision: it gives AI agents exact context and can be used to reject any AI output that violates the rules of the system. 

In a regulated environment, an AI model can never be treated as a trusted black box. As the regulatory panelists in Boston stressed, an AI model can never be held accountable for a mistake. Accountability always rests with the engineers and organizations deploying the system. 

That means our true engineering moat lies in Governance, Verification, and Validation (V&V) around everything that enters and leaves the model: 

• On the Input Side: Ensuring the data and context fed into an agent are accurate, traceable, and stripped of unauthorized sensitive patient information (PHI) before the model ever sees it. 
• On the Output Side: Continuously verifying that every recommendation, code change, or clinical summary meets strict safety, compliance, and quality thresholds before it can trigger a downstream action or reach a user. 

AI models are inherently non-deterministic. Given the same prompt twice, they can take slightly different reasoning paths. That flexibility makes them powerful for synthesizing unstructured clinical notes, spotting anomalies, or exploring complex datasets. In a safety-critical environment, however, "mostly right" is not enough. 

The future of dependable AI belongs to hybrid architectures that pair non-deterministic AI with deterministic engineering tools: 

• Deterministic tools (rule-based & predictable): Static code analysis, strict ontology schemas, architectural isolation (IEC 62304 patterns), automated CI/CD regression test suites, and hard security guardrails. These tools enforce the "zero patient harm" boundary with mathematical certainty. 
• Non-deterministic tools (AI-powered & adaptive): Large language models (LLMs) for semantic analysis, multi-agent workflows for complex problem-solving, and secondary "Judge" models (LLM-as-a-Judge) that act as a second opinion to filter and score candidate outputs. 

By wrapping non-deterministic AI inside deterministic guardrails, and keeping both tools working in tandem within our V&V pipelines, we get the speed of AI without compromising the certainty required by certifying authorities. 

For medical device manufacturers and Software as a Medical Device (SaMD) innovators, this shift brings both urgent challenges and transformative opportunities. 

The challenge: navigating the deployment and certification gap: Regulatory frameworks like the EU AI Act, FDA Quality Management System Regulation (QMSR), and IEC 62304 demand full traceability, explainability, and Predetermined Change Control Plans (PCCP). Teams that bolt AI onto legacy workflows without clear input/output validation stall in "pilot purgatory" when facing regulatory review. 

• Opportunity 1: accelerating V&V and regulatory traceability: By mapping device codebases, requirements, and hazard analyses into a relational knowledge graph, engineering teams can use governed AI agents to generate test cases, trace requirements end-to-end, and produce audit-ready verification evidence—shrinking months of manual documentation and regression testing into days. 
• Opportunity 2: safer clinical decision support & diagnostics: In diagnostic imaging and patient monitoring, hybrid architectures allow AI to surface early clinical insights and reconcile complex patient histories faster, while deterministic safety limits and human-in-the-loop sign-offs ensure clinicians remain in full control. 
• Opportunity 3: continuous post-market vigilance: Automated workflows that extract, validate, and reconcile field data against compliance criteria can help device makers monitor real-world performance and manage post-market updates safely within approved risk limits. 

At Critical Software, we have spent nearly 30 years building and certifying safety-critical systems, and we are a founding signatory of the AI Deployment Gap Initiative. This evolution plays directly to our strengths. The winners in the next era of medical technology won't be those who simply plug in the biggest AI model, but those who master the data, the ontologies, and the deterministic and non-deterministic V&V needed to make AI truly dependable.his evolution plays directly to our strengths. The winners in the next era of medical technology won't be those who simply plug in the biggest AI model, but those who master the data, the ontologies, and the deterministic and non-deterministic V&V needed to make AI truly dependable. 

Our medical devices team helps manufacturers and SaMD innovators build AI-enabled systems that are traceable, verifiable, and ready for certification. Discover how we can support your next device.