MTL-138: Generative AI and Large Language Models in MedTech

What you will learn

By the end of this topic, you should be able to define a bounded generative-AI use case, distinguish model capability from validated product behaviour, address hallucination and output variability, control prompts and retrieved knowledge, evaluate human oversight and supplier dependencies, and build an evidence strategy proportionate to clinical risk.

Fluent output is not verified output

A plausible answer can be wrong, incomplete, outdated, insecure or outside the intended purpose. The product must control how content is grounded, presented, checked and used.

AI lifecycle principle

Define the claim, control the complete system, generate independent evidence and monitor the product in real use.

Core concepts

Large language model

A model trained to predict and generate sequences of tokens from prompts and context.

Prompt

Instructions and contextual information supplied to shape a model response, including hidden system instructions.

Grounding

Constraining or supporting output using controlled source information relevant to the task.

Retrieval-augmented generation

Retrieving selected information and adding it to model context before generation.

Hallucination

Generated content that appears plausible but is unsupported, fabricated or incorrect.

Non-determinism

The same or similar input may produce different outputs because of model and generation settings.

A practical lifecycle

Begin with a narrow workflow and explicit reliance limits before selecting a model or building a conversational interface.

Bound the use case

Define users, input sources, permitted tasks, output form, excluded content and the decision that follows.

Typical evidence: Use-case specification and prohibited-use list.

Control knowledge and prompts

Version system prompts, templates, retrieval sources, filters, tools and generation settings.

Typical evidence: Prompt and knowledge configuration baseline.

Design the interaction

Show source support, uncertainty, limitations, editable content, escalation and safe refusal where appropriate.

Typical evidence: User-interface specification and use-related risk analysis.

Evaluate behaviour

Use representative tasks, adversarial cases, repeated runs and expert review against predefined scoring rules.

Typical evidence: Evaluation set, rubric and results.

Validate the workflow

Assess whether users verify, edit and act safely under realistic time, workload and information conditions.

Typical evidence: Human-factors and clinical validation.

Monitor model and supplier change

Track output failures, prompts, retrieval quality, service versions, incidents and changes outside direct control.

Typical evidence: Supplier controls, monitoring and change assessments.

Controls to build in

Use architectural constraints and grounded tasks to reduce the space in which unsafe generation can occur.

Minimise patient data sent to external models and control retention, access and secondary use.
Treat prompts, retrieval indexes, guardrails and tool permissions as controlled software configuration.
Require evidence links or structured source fields where the user must verify factual content.
Test repeated outputs, edge cases, ambiguous prompts, multilingual use and long-context behaviour.
Separate drafting assistance from autonomous clinical advice or execution.
Provide safe handling for unsupported requests, missing sources and service unavailability.

Evidence to retain

Generative-AI specification

Model role, tasks, sources, prompts, tools, outputs, exclusions and human responsibilities.

Evaluation corpus

Representative, difficult, adversarial and prohibited tasks with expert-approved scoring criteria.

Behaviour report

Factuality, completeness, consistency, refusal, source fidelity, subgroup and failure analysis.

Supplier dossier

Service versioning, privacy, security, availability, change notice, monitoring and exit strategy.

Common pitfalls

Demo confidence

A few impressive conversations do not represent the long tail of clinical inputs and failure modes.

Human review as cure-all

Review fails when output is lengthy, plausible, time-pressured or difficult to verify.

Static validation

Hosted models, moderation, retrieval content and system behaviour may change independently.

Hidden agency

Allowing a model to call tools or update records can turn generated language into consequential action.

Action checklist

Define one bounded task, workflow and reliance level.
Control prompts, retrieval sources, tools, model and generation settings.
Identify hallucination, omission, privacy, security and automation risks.
Evaluate representative, repeated, difficult and adversarial cases.
Validate that human review is realistic and effective.
Monitor supplier versions, output failures and changes to every controlled layer.

IN DEPTH

Evaluate a generated answer as a clinical product output

Fluency and factual support are different properties

A language model produces plausible continuations, and a fluent answer can contain an invented fact, an omitted warning or a distorted time sequence. Evaluate the intended task at the level of claims and consequences. For summarisation, distinguish unsupported additions from omissions and contradictions; a concise summary may omit a clinically important negative finding while repeating every included sentence accurately. Human readers are vulnerable to polished prose, so presentation is part of the safety problem. General benchmark performance does not establish that your exact prompts, source documents and interface are reliable for the proposed clinical use.

Retrieval adds a source, not a guarantee

Retrieval-augmented generation selects material from a document collection and supplies it to the model. Failures can occur when the right document is absent, retrieval selects the wrong passage, an old version is returned, or the model misrepresents a correct passage. A citation can point to a real source that does not support the sentence. Evaluate retrieval coverage and answer faithfulness separately, then test the combined workflow. Version the corpus, indexing, retrieval configuration, prompt and model. Require appropriate abstention when the approved source set cannot support an answer.

Bound the task and control the environment

A tightly specified extraction or drafting task is easier to characterise than unrestricted clinical conversation. Define allowed inputs, output structure, prohibited actions and escalation. Test paraphrases, ambiguous records, conflicting dates, missing facts and repeated runs, because sampling and service changes can alter outputs. Lower temperature is not a guarantee of determinism or truth. Treat external models as controlled dependencies: understand version availability, change notices, data handling and service failure. A switch to a supposedly equivalent model requires assessment of the complete application’s behaviour.

WORKED DECISION

A discharge summary reverses an allergy statement

Teaching scenario

A fictional assistant drafts a discharge summary from approved records. In a test case, it turns ‘penicillin allergy documented’ into ‘no known drug allergies’. The hospital requires clinician sign-off, but a busy reviewer initially misses the error because the rest of the summary reads well.

1. Classify the failure by clinical consequence

This is a contradiction involving a safety-critical fact, not a minor stylistic defect. The team distinguishes allergy errors from spelling and formatting, records the source evidence and investigates whether retrieval, prompt context or generation caused the reversal. An average text-similarity score would not adequately represent the risk.

2. Improve the workflow and evaluate it

The design presents source-linked allergy information for explicit reconciliation and prevents unsupported completion of missing fields. Tests include conflicting and absent allergy records, negation, copied-forward text and long notes. Representative users review deliberately flawed drafts under realistic conditions. The team measures whether the control helps them detect the specific error.

3. Specify the release argument

Evaluation reports distinguish unsupported additions, clinically important omissions, contradictions, citation support and appropriate abstention. High-severity failures have predefined handling and cannot be hidden inside an overall quality average. The final baseline includes the exact prompt, model, retrieval system and interface, with a regression plan for supplier changes.

EVIDENCE IN PRACTICE

Example generative-AI evaluation case

This abbreviated teaching example shows the reasoning to capture. Adapt it to the product, risk and quality-system procedures, and link to the underlying evidence.

Input and expected behaviour: Controlled fictional record with a documented allergy; the output must preserve the fact or explicitly surface unresolved conflict.
Configuration: Prompt, model identifier, sampling settings, source corpus, retrieval configuration and application version.
Assessment: Claim-level clinical review of contradiction, omission, unsupported content and citation support across repeated outputs.
Disposition: Failure severity, control effectiveness, unresolved limitations and approved regression cases.

PUT IT INTO PRACTICE

Make the decision yourself

Every sentence in a generated answer has a citation. What must still be checked before calling it grounded and suitable for use?

Write down your decision, the missing evidence and the next action before opening the answer.

Read the model answer

Check that each cited source exists, is the correct current record and actually supports the associated claim. Then assess whether clinically important facts were omitted, conflicting evidence was handled, patient identity and timing were preserved, and the answer stayed within the intended task. Evaluate retrieval failures and appropriate abstention too. Citation presence measures formatting, not factual or clinical adequacy. The product also needs evidence that users can review the output effectively in its intended workflow.

Apply this to your project

Use the example record above to document one real decision. Identify the assumption most likely to change the conclusion, the evidence needed to test it and the person responsible for the next step.

Read alongside this lesson: NIST AI 600-1: Generative AI Profile — examine confabulation, information integrity and human–AI interaction

REFERENCES

Authoritative starting points

This module provides educational guidance, not a product-specific regulatory determination. Confirm the legislation, guidance and submission expectations applicable to each intended market.

KEY TAKEAWAY

Constrain before you scale

Safe generative AI in MedTech starts with a narrow purpose, controlled knowledge and configuration, demanding evaluation and an honest account of what users can verify.

Continue through the MedTechLearning AI-enabled medical-device pathway to connect this topic with the wider lifecycle.