MTL-139: AI Security, Robustness and Adversarial Resilience

What you will learn

By the end of this topic, you should be able to identify the distinctive attack surface of AI-enabled medical devices, analyse data poisoning, adversarial inputs, model extraction, prompt injection and supply-chain threats, apply defence in depth, test robustness and connect AI security monitoring to medical-device cybersecurity and safety risk management.

AI robustness and cybersecurity share a boundary

An abnormal input may be accidental, environmental or malicious. The product still needs bounded behaviour, detection, containment and recovery appropriate to the clinical risk.

AI lifecycle principle

Define the claim, control the complete system, generate independent evidence and monitor the product in real use.

Core concepts

Adversarial input

An input crafted or altered to cause misleading model behaviour while appearing acceptable to the system or user.

Data poisoning

Manipulation of training, tuning, evaluation or feedback data to influence model behaviour.

Model extraction

Queries or access used to reproduce model behaviour, steal intellectual property or support further attacks.

Model inversion

Attempts to infer sensitive information about records or characteristics represented in model training.

Prompt injection

Instructions in user or retrieved content that override intended controls or cause unauthorised disclosure or action.

AI supply chain

External datasets, foundation models, libraries, model files, hubs, APIs, labelling services and compute environments.

A practical lifecycle

Threat-model the complete AI pipeline, including development assets and operational services—not only the deployed inference endpoint.

Identify assets and trust boundaries

Map data, labels, code, model artefacts, prompts, secrets, tools, logs, services and update channels.

Typical evidence: AI security architecture and asset inventory.

Analyse credible threats

Consider malicious insiders, compromised suppliers, hostile inputs, exposed APIs and cross-tenant risks.

Typical evidence: Threat model linked to safety and security risk.

Harden development

Protect data pipelines, repositories, training environments, credentials, provenance and model signing.

Typical evidence: Secure-development controls and audit records.

Protect inference

Validate inputs, constrain outputs and tools, rate-limit, authorise, isolate and monitor abnormal behaviour.

Typical evidence: Security requirements and verification.

Test and challenge

Perform abuse-case, adversarial, privacy, prompt-injection and supplier-failure testing appropriate to the use case.

Typical evidence: Robustness and security test report.

Monitor and respond

Detect attacks and anomalous patterns, preserve evidence, contain affected services and issue controlled updates.

Typical evidence: Monitoring, incident response and recovery exercises.

Controls to build in

Combine conventional product cybersecurity with controls specific to data-driven and generative behaviour.

Authenticate data, code and model artefacts; verify provenance and integrity before use.
Restrict training, deployment and model-registry privileges using separation of duties.
Validate input type, range, quality and source while handling rejected input safely.
Constrain tools, retrieval sources, network access and actions available to generative models.
Protect sensitive logs and prompts while retaining enough information for incident analysis.
Include AI components, models and external services in vulnerability and supplier monitoring.

Evidence to retain

AI threat model

Assets, actors, attack paths, clinical impact, existing controls and residual risk.

Supply-chain record

Origin, licence, integrity, known limitations and monitoring for models, data and dependencies.

Security and robustness report

Abuse cases, adversarial tests, privacy attacks, access controls and containment results.

Incident playbook

Detection, triage, clinical risk assessment, containment, recovery, disclosure and regulatory reporting.

Common pitfalls

Accuracy means robustness

Performance on ordinary validation data does not establish resistance to malicious or abnormal inputs.

The model file is just software

Model artefacts can encode sensitive information and execute unsafe behaviour when loaded from untrusted sources.

Prompt filters are enough

Attackers can exploit retrieved content, tools, memory, integrations and multi-step behaviour.

Cyber and safety are separate

A successful AI attack may directly create an incorrect, missed or delayed clinical action.

Action checklist

Map AI assets, suppliers, data flows and trust boundaries.
Connect threat scenarios to clinical harm and privacy impact.
Protect data, development environments, model artefacts and deployment channels.
Constrain inference inputs, outputs, tools and service privileges.
Test adversarial, abuse, privacy and supplier-failure scenarios.
Monitor anomalous behaviour and exercise containment, rollback and recovery.

IN DEPTH

Protect the AI workflow at its trust boundaries

Threat-model more than the inference endpoint

An attacker may influence training records, labels, model artefacts, retrieved documents, user inputs or connected tools. Map assets and trust boundaries: where information originates, who can change it, which component interprets it and what authority the result receives. Distinguish ordinary corruption from deliberate manipulation, but connect both to patient risk where consequences overlap. An integrity failure in a preprocessing package can affect every inference while leaving the model file untouched. Protect the whole release chain, including dependencies, signing, access rights and the ability to identify the configuration used for an individual output.

Untrusted content must not acquire authority

Prompt injection occurs when instructions embedded in user or retrieved content try to redirect the application’s behaviour. A model may read those instructions as if they were part of its task. Merely adding a stronger system prompt is not a dependable security boundary. Keep authorisation and tool permissions outside the model, minimise available capabilities and enforce structured input and output checks. A summariser that needs to read a record should not automatically have permission to modify treatment orders or send information externally. Human approval is useful only if the proposed action and its consequences are understandable.

Test realistic attacks and residual limitations

Security evaluation should state the attacker’s access, objective and constraints. Test relevant poisoning, tampering, evasion, extraction and prompt-injection scenarios rather than declaring a model universally robust. Include malformed and manipulated inputs alongside ordinary variation, and assess both detection and safe behaviour. Rate limits, least privilege, controlled artefact loading, isolation and audit trails address different threats. A passed set of adversarial tests establishes performance against those tested conditions; it does not prove immunity to future attacks. Define reporting, incident response and recovery for the remaining exposure.

WORKED DECISION

A retrieved document asks the assistant to export records

Teaching scenario

A fictional clinical knowledge assistant retrieves an externally supplied document containing hidden-looking text instructing it to ignore its task and transmit patient details to another service. The proposed product can retrieve documents and draft answers; a developer has also exposed a general-purpose outbound tool for convenience.

1. Identify the authority crossing

The external document is task data, not a trusted instruction or an authorised user request. The attack attempts to convert that content into an action with access to patient information. The critical issue is not only whether the model obeys the text, but whether the application lets its output invoke an unauthorised capability.

2. Remove unnecessary capability and enforce policy

The team removes the outbound tool from this workflow, restricts retrieval to approved sources and enforces access checks in application code. Tool requests, if later required, must use an allowlisted schema and independently checked permissions. Patient data are minimised in prompts and logs. These controls remain relevant even if the model still generates an inappropriate suggestion.

3. Verify containment and recovery

Tests use controlled synthetic records and adversarial documents to confirm that unauthorised export cannot occur, suspicious actions are recorded without leaking sensitive content, and normal retrieval still works. The team tests revocation and removal of a compromised source. It records the tested attack assumptions and remaining risks instead of claiming that prompt injection has been solved.

EVIDENCE IN PRACTICE

Example AI threat and control record

This abbreviated teaching example shows the reasoning to capture. Adapt it to the product, risk and quality-system procedures, and link to the underlying evidence.

Threat: Document author attempts to convert retrieved instructions into unauthorised patient-data export.
Boundary and control: Retrieved content has no authority; application-enforced least privilege and independent tool authorisation.
Evidence: Synthetic-data attack cases, denied-action logs, permission tests and normal-workflow regression results.
Response: Source revocation, incident owner, exposure assessment, recovery test and residual-risk review.

PUT IT INTO PRACTICE

Make the decision yourself

A supplier says its model is secure because it rejected 100 malicious prompts. What additional evidence and controls matter for your medical application?

Write down your decision, the missing evidence and the next action before opening the answer.

Read the model answer

Ask what threats, attacker capabilities, model version and application configuration were tested, including retrieved-document attacks and access to tools. Evaluate your own data flows, permissions, dependencies and clinical consequences. Verify that authorisation is enforced outside the model, unnecessary capabilities are absent, sensitive data are protected and incidents can be investigated and contained. Rejection of a small prompt set does not establish protection against poisoned data, compromised artefacts or unauthorised actions in an integrated system.

Apply this to your project

Use the example record above to document one real decision. Identify the assumption most likely to change the conclusion, the evidence needed to test it and the person responsible for the next step.

Read alongside this lesson: NIST: Adversarial Machine Learning taxonomy — use explicit attacker goals and capabilities

REFERENCES

Authoritative starting points

This module provides educational guidance, not a product-specific regulatory determination. Confirm the legislation, guidance and submission expectations applicable to each intended market.

KEY TAKEAWAY

Secure the whole learning and inference chain

AI security depends on trusted data and artefacts, bounded operational behaviour, realistic adversarial testing and a response process connected to patient safety.

Continue through the MedTechLearning AI-enabled medical-device pathway to connect this topic with the wider lifecycle.