MTL-132: Data Governance for Medical AI

What you will learn

By the end of this topic, you should be able to define a governed medical-AI dataset, establish provenance and permitted use, organise labelling and quality controls, prevent leakage between development and evaluation, and maintain traceability from source data to the released model.

A dataset is a controlled product input

Its origin, meaning, permitted use, transformations, labels, exclusions and limitations must be known well enough to support the claims made for the model.

AI lifecycle principle

Define the claim, control the complete system, generate independent evidence and monitor the product in real use.

Core concepts

Provenance

Where data originated, how it was collected, under which protocol, using which equipment and in which clinical setting.

Data specification

The required population, modalities, formats, quality, labels, coverage and exclusions for a defined AI purpose.

Lineage

The traceable path from source records through cleaning, labelling, transformation, partitioning and model use.

Ground truth

The reference used to establish a label or outcome, including its uncertainty, timing and adjudication method.

Partition integrity

Controls that keep patients, sites or related samples from leaking across development, tuning and test datasets.

Permitted use

The legal, ethical, contractual and consent basis that defines how data may be accessed, combined, retained and reused.

A practical lifecycle

Govern data before acquisition, not after the model has already become dependent on it.

Define the need

Translate the intended purpose and performance claims into population, input, label and coverage requirements.

Typical evidence: Dataset requirements and sampling rationale.

Approve the source

Assess provenance, permissions, privacy, contracts, collection methods and source-system limitations.

Typical evidence: Source assessment, data agreement and approval.

Acquire and inventory

Typical evidence: Data inventory and lineage record.

Curate and label

Apply controlled cleaning, transformations, annotation instructions, reviewer competence and adjudication.

Typical evidence: Curation log and labelling-quality report.

Partition and release

Create patient- and site-aware partitions; freeze evaluation data and approve dataset baselines.

Typical evidence: Partition record, release note and checksums.

Maintain and retire

Control corrections, additions, new uses, retention, withdrawal and secure disposal.

Typical evidence: Change history, impact assessment and retirement record.

Controls to build in

Good governance makes both model performance and its limitations reproducible.

Assign accountable owners for each source dataset and derived dataset release.
Record inclusion, exclusion and missingness rather than silently dropping inconvenient cases.
Prevent patient, encounter, device, image-series and site leakage across partitions.
Measure inter-rater agreement and adjudicate labels where expert interpretation varies.
Protect personal data through minimisation, access control, separation and appropriate de-identification.
Document transformations that may alter clinical meaning, image quality or signal characteristics.

Evidence to retain

Dataset specification

Purpose, required population, source types, labels, quality and acceptance rules.

Data sheet

Composition, provenance, permissions, collection context, processing, limitations and intended uses.

Labelling record

Instructions, annotator competence, tools, agreement, adjudication and uncertainty.

Dataset release

Version, contents, partitions, quality results, approvals and immutable identifiers.

Common pitfalls

Convenience sampling

Available data may systematically exclude the settings and patients where the device will be used.

Hidden duplicates

Related records in training and test data can create a convincing but false performance result.

Weak labels

A large dataset with an unreliable reference can train and validate the wrong behaviour.

Purpose creep

Data lawfully collected for one purpose is not automatically suitable or permitted for another.

Action checklist

Translate the intended purpose into dataset requirements.
Confirm provenance, permissions, ownership and permitted use.
Define labelling, uncertainty and adjudication methods.
Control transformations and preserve end-to-end lineage.
Demonstrate partition independence and test-set protection.
Approve, version and monitor each dataset used in the product lifecycle.

IN DEPTH

Build a dataset that can support the intended claim

The unit of independence comes before the split

A dataset may contain files, images, encounters, patients and hospitals at different levels. Randomly dividing files can put images from one patient into training and testing, allowing the model to benefit from shared characteristics. Decide what kind of generalisation the evaluation must demonstrate. Patient-level grouping addresses repeated-patient leakage; holding out a hospital or later time period probes other forms of transfer. No single split proves every claim. Preserve grouping keys under appropriate access control, document the partition algorithm and inspect overlap before any model evaluation begins.

A label is a measurement with limitations

A diagnosis code, pathology result and expert judgement are not interchangeable ground truth. They differ in timing, error and relevance to the intended target. Define the reference standard and its relationship to the prediction time. A feature collected after the clinical outcome can leak the answer into a supposedly prospective prediction. For expert labels, specify instructions, reviewer qualifications, blinding, disagreement handling and adjudication. Keep uncertain cases visible: excluding every difficult example can create an unrealistically easy dataset and make performance look stronger than it will be in practice.

Trace permitted use and transformations

Dataset governance combines scientific fitness with control of access and permitted use. Record who supplied data, the purposes for which it may be used, restrictions on transfers and retention, and who has authorised access. Removing direct identifiers does not by itself establish anonymity or permission for every use. Trace source records through cleaning, exclusions, transformations and derived labels. A dataset version should identify its contents and processing history, while the actual sensitive data remain in a controlled store. This enables investigation without copying patient information into ordinary development logs.

WORKED DECISION

Forty thousand images are not forty thousand independent patients

Teaching scenario

A fictional team receives 40,000 wound images from 5,000 patients at three clinics. Many patients have weekly photographs. An initial image-level split gives excellent performance, but the intended claim concerns first assessment of new patients at clinics with different cameras.

1. Audit leakage before celebrating the result

The team discovers photographs of the same wound in both training and test sets. It withdraws the initial result as evidence of new-patient generalisation. Perceptual duplicates and patient identifiers are checked because renamed files can conceal overlap. The training set is not simply cleaned while leaving the previously inspected test result as the headline.

2. Rebuild the evaluation design

All observations from a patient stay in one partition. A separate clinic evaluation investigates camera and workflow differences, while first-visit inclusion rules match the proposed claim. Clinicians define the reference diagnosis and handle uncertain labels. The team documents which settings remain unrepresented and plans additional collection where necessary.

3. Establish the controlled baseline

The final manifest records source versions, permitted uses, patient counts, visit counts, exclusions and transformation history. Data-quality checks detect missing scale references and corrupted images. An independent custodian controls final test access; the development team receives results only according to the agreed evaluation procedure.

EVIDENCE IN PRACTICE

Example dataset release record

This abbreviated teaching example shows the reasoning to capture. Adapt it to the product, risk and quality-system procedures, and link to the underlying evidence.

Scope: First-visit wound photographs; defined patient population, acquisition settings and reference-standard procedure.
Lineage: Source agreement and batch IDs linked to cleaning code, exclusion log, label version and immutable manifest.
Partition assurance: Patient grouping and duplicate checks completed; clinic coverage and temporal limitations recorded.
Approval: Clinical, data-governance and development reviewers approve fitness and permitted use for the specified purpose.

PUT IT INTO PRACTICE

Make the decision yourself

A hospital offers a new dataset with twice as many images but cannot establish whether patients overlap with your final test cohort. What should the team do?

Write down your decision, the missing evidence and the next action before opening the answer.

Read the model answer

Quarantine the dataset from model training until overlap can be assessed using an approved linkage method. Document the uncertainty, permissions and proposed purpose. If independence cannot be established, do not claim the existing final set remains an independent evaluation after training on the new data; plan a demonstrably independent evaluation instead. More images do not compensate for loss of a trustworthy test. Also examine whether the extra observations add new patients, settings and clinically relevant variation rather than just repeat the same examples.

Apply this to your project

Use the example record above to document one real decision. Identify the assumption most likely to change the conclusion, the evidence needed to test it and the person responsible for the next step.

Read alongside this lesson: IMDRF: Good Machine Learning Practice — review data representativeness and independence

REFERENCES

Authoritative starting points

This module provides educational guidance, not a product-specific regulatory determination. Confirm the legislation, guidance and submission expectations applicable to each intended market.

KEY TAKEAWAY

Data governance is design control for AI

If the dataset cannot be explained, reproduced and defended, neither can the resulting model or its performance claim.

Continue through the MedTechLearning AI-enabled medical-device pathway to connect this topic with the wider lifecycle.