What you will learn
By the end of this topic, you should be able to build a layered AI evidence strategy, distinguish software verification from model and clinical validation, select clinically meaningful metrics and operating points, protect independent evaluation data, quantify uncertainty and connect acceptance criteria to intended use and risk.
One test set cannot answer every question
AI evidence must show that the implementation is correct, the model performs for the defined population and the complete product supports safe and effective clinical use.
Define the claim, control the complete system, generate independent evidence and monitor the product in real use.
Core concepts
Software verification
Evidence that preprocessing, inference, postprocessing, interfaces and error handling meet specified requirements.
Model performance
Evaluation of discrimination, error, calibration, robustness and subgroup behaviour on independent data.
Clinical association
Evidence that inputs and outputs relate meaningfully to the clinical condition or physiological state.
Clinical validation
Evidence that the output achieves its intended purpose in the target population and use context.
Operating point
The threshold or policy translating a model score into an output, alert, class or action.
Uncertainty
The precision and stability of the performance estimate, not merely the model's own confidence value.
A practical lifecycle
Plan evidence from the claim and risk backwards, then keep final evaluation independent from development decisions.
Define claims and criteria
State the population, setting, task, comparator, metrics, operating point and minimum acceptable performance.
Typical evidence: Evaluation plan and acceptance criteria.Verify the pipeline
Challenge data ingestion, transformations, model execution, output mapping, interfaces and exceptions.
Typical evidence: Software verification protocols and reports.Evaluate model performance
Use protected data representative of intended use; report uncertainty, calibration and subgroup results.
Typical evidence: Locked model-performance report.Test robustness
Assess plausible acquisition variation, missingness, artefacts, equipment, site and out-of-distribution inputs.
Typical evidence: Robustness and stress-test report.Validate clinical use
Evaluate the complete user–AI workflow and its effect on decisions, outcomes or accepted performance endpoints.
Typical evidence: Clinical validation and usability evidence.Confirm release equivalence
Demonstrate that the deployed configuration matches the evaluated model, software and operating point.
Typical evidence: Traceability, release checks and configuration baseline.Controls to build in
Select metrics that expose the error that matters rather than the number that makes the model look strongest.
- Predefine hypotheses, metrics, operating thresholds, sample size and acceptance criteria.
- Keep tuning decisions out of the final test dataset and document every access to protected data.
- Report confusion matrices, predictive values, calibration and uncertainty where relevant.
- Use external sites, time periods, equipment or prospective data when needed to test generalisability.
- Analyse failures qualitatively as well as statistically to identify systematic unsafe behaviour.
- Validate the complete workflow when the claim depends on user interpretation or action.
Evidence to retain
Evidence map
Claims and risk controls mapped to software, model, analytical and clinical evidence.
Evaluation dataset report
Source, independence, representativeness, reference method, sample size and limitations.
Performance report
Predefined results, confidence intervals, calibration, subgroup analysis and failure review.
Traceability baseline
Exact model, data, code, threshold, dependencies and product configuration evaluated.
Common pitfalls
Metric shopping
Choosing the most favourable metric after seeing results undermines the credibility of the evidence.
Test-set reuse
Repeated development feedback turns a supposedly independent test set into another tuning resource.
Internal data only
Performance at the development site may not survive new populations, equipment or workflows.
Model equals product
A good standalone model can fail when preprocessing, latency, interface or user response is added.
Action checklist
- Translate intended use and risk into explicit evidence questions.
- Predefine metrics, operating point, sample size and acceptance criteria.
- Verify the implemented pipeline and challenge failure behaviour.
- Protect independent test data and document dataset representativeness.
- Report uncertainty, calibration, subgroup performance and failure analysis.
- Validate the complete product and confirm release configuration equivalence.
Build an evidence argument, not a collection of scores
Give each study a question to answer
Software verification asks whether requirements are implemented correctly, including interfaces, preprocessing, calculations and fault handling. Analytical or technical evaluation asks how accurately and reliably the system processes inputs into the intended output. Clinical evaluation addresses the relationship to the clinical condition and performance for the intended purpose. Terminology and required evidence differ by product and jurisdiction. The useful discipline is to link each claim to a question, a method and an acceptance decision. A retrospective benchmark may support technical performance without demonstrating improved clinical decisions or patient outcomes.
Prespecify the operating point and analysis
Before final evaluation, define the population, setting, reference standard, endpoints, threshold, exclusions, handling of missing or indeterminate results, subgroup analyses and statistical methods. Sample size depends on the claim, event count, precision and study design; there is no universal number of images that makes an AI study sufficient. Repeated images from one patient do not provide the same information as independent patients. Use statistical expertise to account for clustering and uncertainty. Prespecification limits the temptation to select whichever metric or subgroup looks favourable after seeing the results.
Evaluate the system that will be supplied
A model tested in a notebook may receive cleaner inputs than the deployed application and omit latency, acquisition failures or user interpretation. Freeze the complete configuration and test realistic end-to-end conditions. External evaluation probes performance beyond the development setting, but a new hospital using the same restricted acquisition protocol may still leave important uncertainty. Prospective or workflow studies may be needed to answer questions that retrospective data cannot. The study design should follow the intended claim and risk rather than treating any single method as universally sufficient.
The point estimate passes, but the criterion does not
A fictional protocol requires sensitivity of at least 90% with the lower bound of a two-sided 95% Wilson confidence interval above 88%. These are invented teaching criteria, not a regulatory benchmark. The frozen model detects 92 of 100 independent positive cases.
1. Apply the actual acceptance rule
Observed sensitivity is 92%. Its approximate 95% Wilson interval is 85.0% to 95.9%; the lower bound is below the required 88%. The result therefore fails the uncertainty criterion even though the point estimate exceeds 90%. The point estimate alone does not satisfy the protocol.
2. Investigate without rewriting the result
Review the eight missed cases, reference-standard quality, site distribution and protocol deviations. Record the failed criterion. Do not remove difficult cases after the fact or adjust the threshold on these observations and call the revised analysis confirmatory. Any exploratory analysis is clearly labelled and informs a future plan.
3. Define the next evidence step
Determine whether more independent evidence under a justified statistical plan, a model change or a narrower claim is appropriate. Additional sampling is not an automatic route to passing; account for the original analysis and repeated testing. Reassess specificity and clinical workload too, because increasing sensitivity by lowering a threshold can create additional harm.
Example claim-to-evidence record
This abbreviated teaching example shows the reasoning to capture. Adapt it to the product, risk and quality-system procedures, and link to the underlying evidence.
- Claim
- Identify the defined condition in the specified population and acquisition setting at a frozen operating point.
- Protocol
- Versioned endpoints, reference standard, sample-size rationale, analysis method and acceptance criteria approved before testing.
- Result
- Counts and unrounded statistical decision; deviations, missing results and subgroup limitations retained.
- Conclusion
- Criterion not met; clinical, statistical and quality reviewers agree the next controlled evidence activity.
Make the decision yourself
A team claims a model reduces treatment delay because it has excellent retrospective sensitivity. What additional question is unanswered, and how might it be studied?
Write down your decision, the missing evidence and the next action before opening the answer.
Read the model answer
The unanswered question is whether using the product in the intended workflow actually changes time to appropriate treatment without unacceptable adverse consequences. Consider a justified comparative workflow study with prespecified timing and safety endpoints, representative users and appropriate controls for confounding. The exact design depends on the claim and risk. Evaluate false-alert burden, missed cases and downstream actions as well as time. Retrospective sensitivity supports one part of the argument but does not directly measure clinical workflow benefit.
Apply this to your project
Use the example record above to document one real decision. Identify the assumption most likely to change the conclusion, the evidence needed to test it and the person responsible for the next step.
Read alongside this lesson: IMDRF: SaMD Clinical Evaluation — distinguish clinical association, analytical validation and clinical validation
Authoritative starting points
- FDA — Good Machine Learning Practice for Medical Device Development: Guiding Principles
- IMDRF — Machine Learning-enabled Medical Devices: Key Terms and Definitions
- NIST — Artificial Intelligence Risk Management Framework
- Regulation (EU) 2024/1689 — Artificial Intelligence Act
- IMDRF — Software as a Medical Device: Clinical Evaluation
This module provides educational guidance, not a product-specific regulatory determination. Confirm the legislation, guidance and submission expectations applicable to each intended market.
Build an evidence chain from code to clinical use
Credible AI validation connects correct implementation, independent model performance, robustness, human interaction and clinical purpose in one traceable argument.
Continue through the MedTechLearning AI-enabled medical-device pathway to connect this topic with the wider lifecycle.