Medical AI — From Claim to Controlled Use

A useful alert can still create an unsafe workflow

A fictional manufacturer is developing ClearQueue, software that analyses adult chest X-rays and flags images with features suggestive of pneumothorax for earlier radiologist review. The opportunity is faster attention to potentially urgent images within a busy hospital worklist. The main concern is that staff may treat the absence of a flag as reassurance and delay reviewing a missed case.

The project team includes a clinical lead, product owner, data scientist, software engineer, human-factors specialist, quality and regulatory leads, and a security engineer. A named hospital service owner is responsible for the agreed local workflow. The manufacturer retains responsibility for the supplied product.

Bound the claim before collecting evidence

The team documents supported image types, patient population, acquisition configurations, users, timing and workflow. Paediatric images, unsupported views and unvalidated scanner configurations are outside this initial scope. Enforceable eligibility checks and user information must agree with these boundaries; a restriction in a document alone is insufficient.

Marketing proposes “Never miss an urgent chest finding”. The team rejects it: the model targets one finding and will have false negatives. Such wording would extend the claim beyond both the design and the planned evidence. The regulatory lead treats the function as a medical-device candidate requiring market-specific assessment. Clinician involvement does not automatically remove it from device regulation; this case does not assign a universal risk class or market route.

Decision record CP-01: approve the bounded claim for development, preserve example outputs, record qualification and classification questions separately for each intended market, and require review when the population, output or clinical reliance changes.

Explore MTL-130 — Is My Health Software a Medical Device?

Define the product around the model

The model is a locked image classifier. It does not learn from individual patients during use. A controlled release can replace it with a new version, but the model, preprocessing and decision threshold remain fixed within each released baseline.

1. The receiving service checks study identity, image type and supported acquisition configuration.
2. The preprocessing component applies the validated transformations and records rejected inputs.
3. The model produces a score; a frozen threshold determines the triage flag.
4. The worklist interface displays the flag or a clear failed, unsupported or pending status.
5. A separate queue-age function escalates overdue studies independently of the AI score.
6. Audit records identify the image reference, relevant configuration, processing status and timing without unnecessary patient information.

An unsupported image must not silently become an unflagged result. Similarly, a timeout must not look like successful processing. Requirements cover identity binding, retries, duplicate messages, stale results, access control and degraded operation, as well as model performance.

Baseline BL-01: application 1.0, model A, preprocessing P1, threshold T1 and the approved configuration list are versioned together. The software lead owns reproducibility; the clinical lead owns the rationale for the operating point.

Explore MTL-131 — AI and Machine-learning Foundations for Medical Devices

Design the dataset around the intended use

The team initially receives repeated studies from several hospitals. An image-level random split would place some patients in both development and evaluation sets. It instead groups all studies from a patient into the same partition and checks for duplicates across sources. Data availability does not establish permission: source agreements, permitted purposes, access, retention and transfer restrictions are reviewed before use.

The fictional initial programme allocates 6,000 patients to training, 2,000 to development tuning and 2,000 different patients at an external site to the first independent evaluation. These counts illustrate separation, not a sample-size prescription. The actual study design needs justification for event counts, precision, subgroup coverage and dependence.

Two suitably qualified readers independently assess evaluation images under a documented reference-standard procedure, with disagreement adjudication. The protocol specifies access to supporting clinical information, blinding to AI output and handling of unresolved cases. It does not quietly discard ambiguous images to make the task easier. Eligibility and exclusions are reported against the flow of all submitted studies.

Dataset record DS-01: source and permission records link to patient grouping, acquisition metadata, label instructions, adjudication, exclusion reasons and immutable partition manifests. The final evaluation custodian controls access. Once an evaluation has informed redesign, its data are no longer presented as untouched confirmation of the revised model.

Explore MTL-132 — Data Governance for Medical AI

Look for clinically meaningful gaps

The team prespecifies analyses by portable versus fixed acquisition, supported scanner family and relevant patient characteristics. A demographic count alone cannot show representation of image quality, disease severity or clinical setting. It records where small samples prevent a reliable conclusion and examines intersections when there is a plausible failure mechanism.

Explore MTL-133 — Bias, Fairness and Representativeness in Medical AI

Make human oversight an evidenced control

The central risk sequence is explicit: an urgent image is not flagged; the worklist presentation encourages staff to assume it is low risk; review is delayed; appropriate clinical assessment or treatment may be delayed. A radiologist being somewhere in the workflow does not, by itself, interrupt that sequence.

The clinical and human-factors leads evaluate representative users under realistic workload, including confidently wrong flags, missed cases, timeouts and conflicting information. They examine recognition, interpretation, response and recovery. The fictional workflow study reveals that “AI negative” is interpreted as a diagnostic conclusion. The team replaces that wording with processing and flag status, revises the worklist presentation and repeats the affected tasks.

Risk record RM-07: link delayed-review harm to the controls, requirements, verification and user-study evidence. Record residual risk and staffing assumptions. Any assumed maximum review interval requires a justified, operationally feasible specification; this lesson does not prescribe a clinical timing limit.

Explore MTL-134 — AI Risk Management and Human Oversight

A good headline is not a release decision

The evidence plan separates correct software implementation, model performance and the clinical workflow. Interface, identity, timeout and invalid-input tests verify requirements. A frozen-model evaluation measures technical performance against the reference standard. Representative-use and appropriately designed clinical studies address whether the triage function supports its intended clinical use. Retrospective sensitivity alone does not demonstrate reduced treatment delay.

First evaluation: hold release

In the invented 2,000-patient independent cohort, 200 patients have the target finding. Model A flags 184 of these and misses 16. It also flags 180 of 1,800 non-cases. Sensitivity is 92%, specificity is 90%, and positive predictive value is approximately 50.5% (184 / 364). The 10% study prevalence is illustrative; predictive value and workload must be assessed for the intended deployment setting.

The prespecified acquisition analysis changes the interpretation. Of 150 positive fixed-system cases, 148 are flagged; of 50 positive portable-system cases, only 36 are flagged. Portable-system sensitivity is therefore 72%, despite the pooled 92%. Investigation identifies underrepresented portable acquisitions and image-processing differences. This is a hypothesis to test, not a reason to dismiss the poor result.

Revised model: use new independent evidence

The team improves acquisition coverage and preprocessing, then freezes model B with P2 and T2. A second, independently assembled cohort contains 2,000 different patients: 200 positive and 1,800 negative. Results are shown below. These two cohorts are not a paired comparison, so their difference alone cannot prove superiority.

The protocol requires uncertainty intervals, not just percentages. The statistical reviewer assesses the prespecified overall and subgroup criteria, sample-size rationale and other analyses. The portable subgroup still has only 50 positive cases; the team explicitly retains that limitation. Specificity by subgroup, input rejection, severity and site analyses are also needed and cannot be inferred from this abbreviated table.

Gate V-02: the teaching scenario assumes the complete model-B evidence meets its justified protocol, the revised workflow controls are validated and residual risks are accepted by the authorised reviewers. Deployment still requires the applicable market authorisation, approved configuration, installation checks, training and monitoring readiness. No real device is cleared or approved by the figures in this example.

Explore MTL-135 — Verification, Validation and Clinical Evidence for Medical AI

Secure the same system that was evaluated

The threat model covers received image files, metadata, model packages, software dependencies, service accounts, hospital interfaces and administrative access. A manipulated study identifier could associate a correct score with the wrong patient; a substituted model could change every prediction. Security failures connect to the clinical risk analysis as well as confidentiality and availability.

The design uses least-privilege service identities, approved network connections, integrity-checked release packages and controlled administrative access. Tests exercise malformed files, identity manipulation, unauthorised model replacement and service disruption using synthetic records. They verify rejection, safe workflow behaviour and investigation records. Security review also checks whether the independent queue-age control shares a dependency that could disable both safeguards.

Release baseline BL-02: the deployment record identifies model B, preprocessing P2, threshold T2, application and dependency versions, authorised settings and the recoverable prior configuration. A security patch receives an impact assessment; it is not assumed irrelevant to performance simply because the model weights are unchanged.

Explore MTL-139 — AI Security, Robustness and Adversarial Resilience

Decide whether a language model belongs in this release

Product management proposes a language model that explains each flag and drafts a radiology report. That would introduce outputs and clinical reliance not covered by the triage evidence. A plausible explanation might invent findings, misstate anatomy or imply a diagnostic certainty that the classifier cannot establish.

The team defers patient-specific report generation from ClearQueue’s initial release. This is an architectural decision with an evidence rationale, not a general rejection of generative AI. A future proposal would need a defined purpose, source-grounding and citation checks, evaluation of unsupported additions and omissions, effective user review, prompt-injection controls, supplier-change management and a fresh regulatory assessment.

Change request CR-02: deferred pending its own requirements, risk and evidence plan. A non-patient-specific help function restricted to approved instructions could be assessed separately; it is not quietly added through the same change request.

Explore MTL-138 — Generative AI and Large Language Models in MedTech

Distinguish a planned update from a new claim

For subsequent development, the team proposes bounded retraining using additional adult images from defined compatible acquisition systems, with the population, output and triage role held fixed. Its plan states eligible data, labelling, retraining methods, independent evaluation, subgroup criteria, risk assessment, rollout, communication and rollback. It also explains how repeated updates avoid progressively overfitting to a reused evaluation set.

The FDA PCCP framework addresses planned modifications, methods for developing, validating and implementing them, and impact assessment. Whether a particular change can use a PCCP depends on the applicable authorised plan and market conditions. Preparing this fictional document is not equivalent to obtaining authorisation, and acceptance in one jurisdiction does not settle another.

Gate C-01: quality and regulatory reviewers confirm eligibility before the release board examines implementation evidence. Approval identifies the precise baseline and monitoring plan; it is never “approve all future AI improvements”.

Explore MTL-136 — Predetermined Change Control Plans for Medical AI

A scanner update tests the whole lifecycle

For this simulated field phase, assume the required market and release gates have been completed. The monitoring plan tracks input eligibility and rejection, processing failures and latency, result volumes, review delays, overrides, complaints and outcome-linked performance. Results are stratified by site and acquisition configuration where justified. Each signal has an owner, denominator, review frequency, justified trigger and response procedure.

Confirmed outcomes can arrive late and may be selectively available. If only flagged images receive follow-up, false negatives can remain invisible. The clinical lead therefore plans a justified way to evaluate unflagged cases and reports outcome coverage and delay. Stable average scores or an absence of complaints cannot establish continued performance.

The field signal

After scanner maintenance at one site, the rejected-input rate rises from around 2% to 12%. These are invented observations, not suggested universal alarm limits. The monitoring owner checks patient volume, logging integrity, scanner versions and failure reasons. The investigation associates the change with a new export format; it does not immediately label the event model drift or prove that all accepted results are wrong.

1. Contain: assess risk promptly, activate the approved manual workflow for the affected configuration where warranted, inform users and preserve relevant records.
2. Investigate: determine whether the change only caused visible rejection or also produced plausible but unreliable outputs. Review affected studies and the need for clinical follow-up through the responsible clinical process.
3. Assess obligations: quality and regulatory staff evaluate complaint, corrective-action and applicable vigilance requirements. A technical signal and a reportable event are separate decisions.
4. Correct under control: verify and evaluate the proposed compatibility fix, assess its regulatory route and update the validated configuration list before reactivation.
5. Close the loop: check post-correction behaviour and later outcomes, update supplier-change controls and revise the risk file and monitoring plan.

Incident record PM-03: link the signal to the affected baseline, containment, clinical impact assessment, change record and effectiveness review. The hospital and manufacturer agree who may restore the service. The field experience changes the next development cycle’s inputs.

Explore MTL-137 — Post-market Monitoring, Drift and Model Performance

Connect the claim, control, test and field signal

CP-01 · Clinical need and claim

Help prioritise the defined adult chest X-rays while preserving appropriate review of every study.

RM-07 · Risk sequence

A missed urgent finding plus reliance on an absent flag causes delayed assessment.

REQ-014 · Control requirement

The worklist must distinguish absent flags from unprocessed or unsupported studies and implement the specified independent overdue-study escalation.

VER-014 and USE-03 · Evidence

Identity, timing, invalid-input and outage tests demonstrate implementation; representative users demonstrate interpretation and recovery in the intended workflow.

BL-02 · Released configuration

Model B, P2, T2, software, dependencies, validated acquisition list and user information are approved together.

PM-03 and CR-03 · Lifecycle feedback

The scanner-format incident triggers containment, compatibility reassessment, a controlled correction and an effectiveness review.

These illustrative identifiers show traceability, not required document names. The link must point to actual evidence and accountable decisions, rather than merely to another empty template.

Ten topics, one product story

• MTL-130 — Is My Health Software a Medical Device?: define the triage function and assess each market.
• MTL-131 — AI and Machine-learning Foundations for Medical Devices: distinguish the locked classifier from the complete product.
• MTL-132 — Data Governance for Medical AI: control permissions, labels, patient separation and lineage.
• MTL-133 — Bias, Fairness and Representativeness in Medical AI: investigate weak portable-acquisition results.
• MTL-134 — AI Risk Management and Human Oversight: test whether the workflow controls delayed-review risk.
• MTL-135 — Verification, Validation and Clinical Evidence for Medical AI: hold release when the full evidence argument fails.
• MTL-136 — Predetermined Change Control Plans for Medical AI: distinguish bounded retraining from a new population or output.
• MTL-137 — Post-market Monitoring, Drift and Model Performance: turn a scanner-change signal into accountable action.
• MTL-138 — Generative AI and Large Language Models in MedTech: assess the new evidence burden of generated reports.
• MTL-139 — AI Security, Robustness and Adversarial Resilience: protect inputs, identity, configuration and recovery.

Would you authorise the next release?

The team proposes model C. Its pooled sensitivity is higher than model B’s, but portable-system performance has fallen. It was evaluated on the same cohort repeatedly used to select candidate models. The supplier also changed an image-processing dependency, and the hospital has not tested the revised fallback workflow. Product management requests release because “the model is better”.

Before opening the answer, identify the missing evidence, who should decide, what you would hold and what you could usefully do next.

Authoritative context for the fictional decisions

The following documents provide context for the methods discussed. None endorses ClearQueue or the invented data and decisions above.

• IMDRF N88 FINAL:2025 — Good Machine Learning Practice for Medical Device Development
• FDA — PCCP guidance for AI-enabled device software functions, August 2025
• IMDRF N41 — Software as a Medical Device: Clinical Evaluation