What you will learn
By the end of this topic, you should be able to integrate AI-specific failure modes into medical-device risk management, define meaningful human oversight, address automation bias and foreseeable misuse, design safe fallback behaviour and connect model limitations to product risk controls and verification.
Human oversight must change the risk
A person in the workflow is not automatically a control. The user must have the information, competence, time and authority to recognise a problem and act effectively.
Define the claim, control the complete system, generate independent evidence and monitor the product in real use.
Core concepts
AI contribution
Identify whether AI detects, predicts, recommends, prioritises, generates, controls or filters information.
Failure condition
Consider wrong, missed, delayed, unstable, unavailable, overconfident and out-of-scope outputs.
Automation bias
Users may accept an output too readily, especially when workload is high or the system appears authoritative.
Human–AI team
Clinical performance arises from the model, interface, user, workflow, escalation route and organisational context together.
Uncertainty
Model confidence is not necessarily a calibrated probability and does not by itself communicate safe reliance.
Fallback
The defined behaviour when inputs are unsuitable, confidence is insufficient or the AI service is unavailable.
A practical lifecycle
Risk analysis should follow the complete chain from input acquisition to the clinical action influenced by the output.
Define the decision
Describe the clinical action, available alternatives, urgency and who remains accountable.
Typical evidence: Decision pathway and use scenarios.Identify failure modes
Analyse data, model, interface, integration, user and environment failures—including combinations.
Typical evidence: Hazard analysis and misuse scenarios.Estimate consequences
Connect output failure to delayed, omitted or incorrect decisions and resulting harm.
Typical evidence: Risk analysis with clinical rationale.Design layered controls
Use input checks, bounded outputs, independent information, interface design, workflow and fallback.
Typical evidence: Safety concept and control requirements.Validate oversight
Demonstrate that representative users detect, understand and respond correctly under realistic conditions.
Typical evidence: Human-factors and clinical validation results.Monitor control effectiveness
Track overrides, disagreement, delayed actions, workarounds, near misses and abnormal reliance.
Typical evidence: Monitoring indicators and review records.Controls to build in
Prefer controls that prevent or contain failure before relying only on warning users.
- Reject or flag unsuitable inputs using verified quality criteria.
- Present output purpose, limitations and supporting information at the point of decision.
- Provide an independent route when AI output is unavailable or questionable.
- Avoid interface cues that imply certainty beyond the validated evidence.
- Define escalation, override and recovery without penalising appropriate user challenge.
- Verify that logging supports incident reconstruction without creating unnecessary privacy risk.
Evidence to retain
AI safety concept
Safety objectives, hazardous situations, control layers, safe states and monitoring assumptions.
Use-related risk analysis
Information needs, time pressure, automation bias, misuse and foreseeable workarounds.
Control verification
Objective evidence that input checks, boundaries, fallback and alerts work as specified.
Oversight validation
Representative-user evidence that intervention is feasible and effective.
Common pitfalls
“Clinician in the loop.”
This phrase is not evidence that the clinician can detect or correct a plausible AI failure.
Warnings as architecture
A warning cannot compensate for an unnecessarily hazardous product design.
Model-only FMEA
The hazardous situation often emerges from interaction between data, software, interface and workflow.
Ignoring non-use
Unavailable or rejected AI output can create delay, workload and unsafe improvised alternatives.
Action checklist
- Map every AI output to the decision or action it can influence.
- Analyse wrong, missing, delayed, unstable and out-of-scope behaviour.
- Define layered controls and safe fallback routes.
- Specify what users need to know and do when the AI may be wrong.
- Validate oversight with representative users and realistic pressure.
- Monitor whether controls remain effective in practice.
Translate AI failure into a controllable clinical risk
Trace the sequence from output to harm
‘The model is wrong’ is a failure description, not a complete risk analysis. Describe the circumstances in which the error reaches a user, influences a decision and exposes a person to harm. A missed alert might delay review, but the consequence depends on existing surveillance, response times and whether users assume unflagged cases are safe. Distinguish the hazard, sequence of events, hazardous situation and harm. Include correct outputs delivered late, to the wrong patient or in a misleading interface. These failures can be clinically serious even when model accuracy is unchanged.
Human oversight must have working conditions
A person can detect and correct an error only if they have the information, time, competence and authority to do so. Ask what evidence the reviewer sees, whether contradictory findings are accessible, how workload affects attention and whether they can override or stop the system. Test automation bias: users may defer to a confident-looking result even when told to use judgement. An approval button records an action; it does not establish that meaningful review occurred. Evaluate representative tasks under realistic pressure, including misleading outputs and unavailable-system conditions.
Choose controls across the system
Reducing risk may require better acquisition checks, an enforceable intended-use boundary, an independent plausibility check, a redesigned interface or a dependable fallback workflow. Training and warnings can support those controls but should not carry an unrealistic burden. Define requirements that can be tested: for example, an invalid input must not produce an apparently valid score, and the user must be told what action to take. Trace each control to verification and, where appropriate, validation in use. Assess residual risk after controls and consider whether one common failure could defeat several controls together.
An unflagged scan becomes a forgotten scan
A fictional triage system moves suspected urgent scans higher in a worklist. Clinicians remain responsible for reviewing every scan. During a simulation, staff start treating the unflagged list as low priority and a false-negative scan waits beyond the service’s clinical target.
1. Write the risk sequence
The model misses an urgent case; the interface places it in a low-priority list; staff infer that absence of a flag means low risk; review is delayed; needed treatment may be delayed. The human reviewer is present, but the proposed control does not prevent the hazardous situation. Analyse queue design and service capacity as part of the product context.
2. Redesign and challenge the controls
The team introduces a maximum-wait escalation independent of the AI score, makes the absence of a flag visibly distinct from a negative diagnosis, and defines manual operation when the service fails. It tests whether alert overload causes staff to ignore escalation and whether the timing mechanism remains available during the same outage that disables AI.
3. Evaluate the human–system result
Representative users handle deliberately missed and incorrect alerts within realistic worklists. Measure time to review, missed escalations, interpretation and recovery, alongside algorithm metrics. Release reviewers examine the residual delay risk and operational assumptions. Labelling alone is not accepted as evidence that clinicians will maintain the intended workflow.
Example risk-control trace
This abbreviated teaching example shows the reasoning to capture. Adapt it to the product, risk and quality-system procedures, and link to the underlying evidence.
- Risk sequence
- False-negative triage output plus workflow reliance leads to delayed assessment of an urgent scan.
- Control requirement
- Maximum-wait escalation independent of model score; invalid or unavailable output must be visibly identified.
- Verification and validation
- Timing, outage and queue tests; representative-user evidence showing recognition and timely response.
- Residual risk
- Clinical reviewer assesses remaining delay scenarios, shared dependencies and assumptions about staffing.
Make the decision yourself
A supplier reports that showing a confidence score will prevent automation bias. What evidence would you request before accepting this as a risk control?
Write down your decision, the missing evidence and the next action before opening the answer.
Read the model answer
First establish what the score measures and whether it is calibrated for the intended population; a model score is not automatically a probability of correctness. Then test whether representative users understand it, act appropriately on uncertainty and detect confidently wrong results under real workload. Assess presentation, available independent information and escalation options. Compare the observed behaviour with the risk-control requirement. A plausible explanation or a supplier usability claim does not demonstrate that the control reduces the identified harm in your product.
Apply this to your project
Use the example record above to document one real decision. Identify the assumption most likely to change the conclusion, the evidence needed to test it and the person responsible for the next step.
Read alongside this lesson: IMDRF: Good Machine Learning Practice — examine performance of the human–AI team
Authoritative starting points
- FDA — Good Machine Learning Practice for Medical Device Development: Guiding Principles
- IMDRF — Machine Learning-enabled Medical Devices: Key Terms and Definitions
- NIST — Artificial Intelligence Risk Management Framework
- Regulation (EU) 2024/1689 — Artificial Intelligence Act
This module provides educational guidance, not a product-specific regulatory determination. Confirm the legislation, guidance and submission expectations applicable to each intended market.
Design oversight as a verified safety function
Safe AI combines bounded technology, understandable information, realistic workflow and demonstrated human ability to intervene.
Continue through the MedTechLearning AI-enabled medical-device pathway to connect this topic with the wider lifecycle.