Medical Device AI Bias Testing and Algorithmic Fairness: Validation Methods, Regulatory Requirements, and Submission Documentation

The Bias Problem in Medical AI: Why Regulators Are Paying Attention

The FDA has authorized over 1,000 AI-enabled medical devices as of early 2025, with radiology accounting for roughly three-quarters of all authorizations. These devices are increasingly used in clinical decisions that directly affect patient outcomes -- detecting tumors, flagging abnormalities, triaging urgent cases, and guiding treatment. But a critical question is only now receiving the regulatory attention it demands: do these devices perform equally well for all patients?

The evidence so far is not reassuring. A landmark study by Wu and colleagues published in Nature examined the subset of FDA-reviewed AI/ML medical devices that considered demographic subgroup performance during their evaluation. The finding: only 13% of these devices included any subgroup analysis in their regulatory submissions. The vast majority of AI-enabled medical devices on the market today reached patients without systematic evidence that they work equitably across racial, ethnic, age, sex, and socioeconomic groups.

This is not a theoretical concern. Algorithmic bias in medical AI can produce tangible patient harm: missed diagnoses in underrepresented populations, inappropriate treatment recommendations, delayed interventions, and systematic under-referral for critical care. As AI-enabled devices proliferate across clinical specialties, the risk that bias becomes embedded at scale -- silently and pervasively -- has made algorithmic fairness one of the defining regulatory challenges of this generation.

Regulators have taken notice. The FDA's January 2025 draft guidance on AI-enabled device lifecycle management explicitly requires manufacturers to address "transparency and bias throughout the life cycle." The EU AI Act classifies medical device AI as "high-risk," triggering mandatory requirements for bias testing, transparency, human oversight, and ongoing performance monitoring. Together, these developments signal that algorithmic fairness is no longer optional -- it is a regulatory expectation that manufacturers must address with the same rigor they apply to safety and effectiveness.

This guide covers how to test AI-enabled medical devices for algorithmic bias, which statistical methods to use, how to document bias analysis for FDA and EU submissions, and how to implement ongoing post-market monitoring to detect performance degradation across subgroups.

Real-World Examples of AI Bias in Medical Devices

Understanding algorithmic bias in medical devices requires looking at documented failures. These cases illustrate the mechanisms by which bias enters AI systems and the clinical consequences that result.

Pulse Oximetry: Overestimating Oxygen Saturation in Darker-Skinned Patients

Pulse oximetry is the most prominent real-world example of algorithmic bias in a medical device. Pulse oximeters estimate arterial oxygen saturation (SpO2) by passing light through a patient's fingertip and analyzing the absorption spectrum. The underlying algorithms were developed and validated primarily on lighter-skinned populations.

Multiple studies, including a large 2020 analysis published in the New England Journal of Medicine, demonstrated that pulse oximeters systematically overestimate oxygen saturation in patients with darker skin pigmentation. The clinical consequences were significant: patients with dangerously low actual oxygen levels were incorrectly categorized as having adequate oxygenation, leading to delayed treatment. During the COVID-19 pandemic, this bias had disproportionate effects on Black and Hispanic patients, who were more likely to receive delayed or inadequate oxygen supplementation. The FDA issued a safety communication in 2023 acknowledging the issue and convened an advisory panel meeting in 2024 to evaluate corrective actions.

Dermatology AI: Performance Gaps Across Skin Tones

AI-based dermatology tools trained to assess skin lesion malignancy risk have shown consistent performance disparities across skin tones. Training datasets for these systems historically overrepresent lighter skin phototypes (Fitzpatrick types I-III), leading to reduced sensitivity for detecting melanoma and other malignancies in darker skin. Given that melanoma is frequently diagnosed at later stages in patients with darker skin -- in part because of lower clinical suspicion and training gaps -- an AI tool that performs worse on these populations would compound existing disparities rather than reduce them.

Care Management Algorithms: Systematic Under-Referral

A widely cited 2019 study in Science examined a commercial algorithm used by large U.S. health systems to identify patients who would benefit from high-risk care management programs. The algorithm used healthcare costs as a proxy for health needs, but Black patients had historically incurred lower healthcare costs than equally sick White patients due to structural barriers to access. The result: Black patients had to be significantly sicker than White patients to be recommended for the same level of care. The algorithm was less biased than the data it learned from, but it faithfully amplified an existing disparity.

The Common Thread: Dataset Bias

These examples share a common root cause: the training or calibration data did not adequately represent the full range of patients on whom the device would be used. Dataset shift -- the difference between training and test data distributions -- is the primary driver of algorithmic bias in medical AI. When development data overrepresents certain demographics, clinical settings, or disease presentations, the resulting model encodes those imbalances as systematic performance differences.

FDA Regulatory Expectations for Bias Testing

The FDA's regulatory framework for AI-enabled medical devices has evolved rapidly, and bias testing has moved from a peripheral consideration to an explicit requirement. Two guidance documents are particularly relevant.

The January 2025 TPLC Draft Guidance

In January 2025, the FDA published "Artificial Intelligence-Enabled Device Software Functions: Lifecycle Management Considerations and Marketing Submission Recommendations" -- commonly referred to as the TPLC (Total Product Lifecycle) draft guidance. This document is the FDA's most comprehensive statement on how AI-enabled devices should be developed, validated, and monitored throughout their lifecycle.

The TPLC guidance explicitly requires manufacturers to develop strategies that address "transparency and bias throughout the life cycle." This is not a suggestion -- it is listed as a core element of the device's lifecycle management approach, alongside clinical validation, cybersecurity, and real-world performance monitoring.

Specifically, the TPLC guidance recommends that manufacturers:

• Ensure validation data sufficiently represents the intended use population, including relevant demographic subgroups
• Reference existing FDA guidances on collection of race/ethnicity data, age/race/ethnicity-specific data, and sex-specific data
• Describe the representativeness of training, validation, and test datasets relative to the target clinical population
• Include strategies for identifying and mitigating sources of bias in the data collection and model development process
• Provide subgroup performance analyses in marketing submissions

The guidance also includes an example "model card" -- a structured metadata format for reporting AI device performance characteristics. The model card concept, originally proposed by Mitchell et al. at Google, provides a standardized framework for documenting model architecture, training data characteristics, evaluation metrics, and performance across subgroups. The FDA's inclusion of a model card example signals its expectation that manufacturers will provide structured, transparent reporting of AI device performance.

The PCCP Final Guidance

The FDA's final guidance on Predetermined Change Control Plans (PCCP), finalized in December 2024, addresses how manufacturers can implement planned modifications to AI-enabled devices without submitting new marketing applications for each change. While the PCCP guidance focuses primarily on change management, it includes requirements relevant to bias testing.

The PCCP guidance recommends that manufacturers include descriptive statistics for all datasets used in development and validation, and confirm that these datasets reflect the intended use population. For modifications that affect model behavior, the guidance expects manufacturers to evaluate whether the modification introduces or exacerbates performance disparities across subgroups.

Referenced FDA Guidances on Demographic Data

The TPLC guidance explicitly cross-references several existing FDA guidances that provide detailed expectations for demographic subgroup data collection:

• Collection of race and ethnicity data in clinical trials: Establishes standards for self-identified race and ethnicity categories
• Evaluation of sex-specific data in medical device clinical studies: Requires that clinical studies include sufficient representation of both sexes and that results are analyzed by sex
• Evaluation of age/race/ethnicity-specific data: Addresses the need for adequate representation across age groups and racial/ethnic categories

These guidances, originally developed for traditional medical device clinical trials, now apply with particular force to AI-enabled devices where the relationship between training data demographics and device performance is direct and measurable.

What This Means for Submissions

For manufacturers preparing 510(k), De Novo, or PMA submissions that include AI-enabled device functions, the FDA's current expectations for bias analysis can be summarized as follows:

1. Describe training data composition: Report demographic characteristics of the training dataset, including race, ethnicity, age, sex, and any other clinically relevant subgroup dimensions. Identify gaps or imbalances relative to the intended use population.

2. Validate on representative test data: Ensure that the test dataset used for clinical validation is independent of training data and adequately represents the diversity of the intended use population.

3. Report subgroup performance: Present performance metrics (sensitivity, specificity, AUC, positive predictive value, etc.) stratified by relevant demographic subgroups. Where sample sizes permit, conduct formal statistical tests for performance differences across subgroups.

4. Document bias mitigation steps: Describe the methods used during development to identify and mitigate sources of bias, including data collection strategies, preprocessing techniques, and algorithmic fairness interventions.

5. Provide a model card: Include structured metadata documenting model characteristics, training data, evaluation results, and known limitations.

Recommended Reading

EU Breakthrough Device (BtX) Designation: Complete Guide to MDCG 2025-9 and the EMA Pilot Program

Regulatory EU MDR / IVDR2026-05-08 · 11 min read

EU Requirements: AI Act High-Risk Obligations and MDR Clinical Evidence

The European Union's regulatory approach to AI bias in medical devices operates on two parallel tracks: the EU AI Act and the Medical Device Regulation (MDR). Understanding how these frameworks intersect is essential for manufacturers seeking CE marking.

EU AI Act: Medical Device AI as High-Risk

The EU AI Act, which entered into force in August 2024, classifies AI systems that are safety components of, or are themselves, medical devices as "high-risk" under Annex II. This classification triggers a comprehensive set of mandatory requirements:

• Risk management system: Manufacturers must establish a continuous risk management process that identifies, analyzes, and mitigates risks throughout the AI system's lifecycle, including risks related to bias and discrimination.
• Data governance requirements: High-risk AI systems must be developed using high-quality datasets that are relevant, sufficiently representative, and as free of errors as possible. The training, validation, and testing datasets must be examined for possible biases.
• Transparency: Users must be able to interpret the AI system's output and use it appropriately. This includes providing information about the system's capabilities, limitations, and expected performance.
• Human oversight: The AI system must be designed to allow effective human oversight, including mechanisms for users to understand and override automated decisions.
• Accuracy, robustness, and cybersecurity: The system must achieve appropriate levels of accuracy, robustness, and cybersecurity, and must perform consistently in these respects throughout its lifecycle.
• Quality management system: Manufacturers must establish a quality management system covering all aspects of the AI system's lifecycle.

Timeline of EU AI Act Obligations

The EU AI Act's obligations for high-risk AI systems enter force on a phased timeline:

Date	Milestone
August 2, 2025	Prohibitions on certain AI practices (social scoring, real-time biometric identification in public spaces) take effect
August 2, 2026	Most high-risk AI system obligations take effect, including requirements for risk management, data governance, transparency, human oversight, accuracy, and cybersecurity
August 2, 2027	Obligations specific to high-risk AI systems that are medical devices (or safety components of medical devices) take effect, aligning with MDR conformity assessment timelines

The additional year for medical device-specific obligations acknowledges that these products already undergo conformity assessment under the MDR and provides time for manufacturers to integrate AI Act requirements into their existing regulatory processes.

The EU AI Act Omnibus Amendment

In November 2025, the European Commission proposed a Digital Omnibus on AI to simplify compliance with the AI Act. On May 7, 2026, EU co-legislators reached a provisional agreement that confirms AI-based medical devices remain classified as high-risk under the AI Act -- the product safety exemption some member states sought was not adopted. The deal introduces practical simplifications: a single conformity assessment procedure can be used across both the AI Act and MDR/IVDR, and Notified Bodies can simultaneously evaluate AI Act compliance during MDR/IVDR assessments.

The compliance timeline for AI-enabled medical devices has been extended from August 2, 2027 to August 2, 2028 (or 12 months after the Commission confirms support measures are available, whichever is earlier). This gives manufacturers additional time but does not reduce the substance of the requirements. Bias testing expectations remain firmly in place under both frameworks, and the fundamental expectation -- that AI medical devices must perform safely and effectively for all patients in the intended use population -- is unchanged. For a detailed analysis of the Omnibus deal, see the EU AI Act Omnibus Amendment Guide.

MDR Clinical Evidence for AI Devices

Under the MDR, manufacturers must establish clinical evidence through clinical evaluation as defined in Article 61 and Annex XIV. For AI-enabled devices, this means:

• The clinical evaluation must address the device's performance across clinically relevant subgroups
• Post-market clinical follow-up (PMCF) must include mechanisms to detect performance degradation in underrepresented populations
• The technical file must include evidence that the training data is representative of the European patient population, which has different demographic characteristics than U.S. datasets commonly used in AI development
• Notified bodies may request subgroup analyses as part of conformity assessment, particularly for devices used in diagnostic or treatment decisions

Statistical Methods for Bias Detection

Testing AI-enabled medical devices for bias requires selecting appropriate fairness metrics, applying them to performance data stratified by demographic subgroups, and interpreting the results in the context of clinical risk. This section covers the primary statistical methods used in medical AI fairness evaluation.

Defining the Problem Formally

Consider an AI model that produces a prediction or classification for each patient. Let:

• Y = the true outcome (e.g., disease present or absent)
• Y-hat = the model's prediction
• A = a protected attribute (e.g., race, sex, age group)

Bias exists when the model's performance -- measured by some relevant metric -- varies systematically across values of A in a way that disadvantages certain groups. The question is not whether performance differences exist (they almost always do), but whether those differences are clinically acceptable given the device's intended use and risk profile.

Key Fairness Metrics

The following fairness metrics are drawn from the computer science literature and have been applied to medical AI evaluation in peer-reviewed research, including the Nature Biomedical Engineering review by Chen and colleagues (2023).

Demographic Parity (Statistical Parity)

Demographic parity requires that the model's positive prediction rate is equal across demographic groups. Formally: P(Y-hat = 1 | A = a) = P(Y-hat = 1 | A = b) for all groups a and b.

• What it measures: Whether the model allocates positive outcomes (e.g., referrals, diagnoses, treatment recommendations) at equal rates across groups
• When to use: Appropriate when the base rate of the condition should be similar across groups, or when the goal is to ensure equitable allocation of resources
• Limitations: Ignores differences in actual prevalence across groups. If a disease genuinely occurs at different rates in different populations, demographic parity may require the model to make systematically incorrect predictions in one group to satisfy the fairness constraint. This is often inappropriate for medical devices where accuracy is paramount.

Equalized Odds

Equalized odds requires that the model's true positive rate and false positive rate are equal across demographic groups. Formally: P(Y-hat = 1 | A = a, Y = y) = P(Y-hat = 1 | A = b, Y = y) for y in {0, 1} and all groups a and b.

• What it measures: Whether the model makes errors at equal rates across groups, conditioned on the true outcome. A model satisfying equalized odds has the same sensitivity and specificity across groups.
• When to use: This is one of the most appropriate metrics for medical AI because it directly measures whether the model is equally accurate for all groups. It captures both missed diagnoses (false negatives) and false alarms (false positives).
• Limitations: Can be difficult to satisfy simultaneously with maximum overall accuracy. May require accepting lower overall performance to achieve equitable performance across groups.

Equal Opportunity

Equal opportunity is a relaxation of equalized odds that requires only that the true positive rate is equal across groups. Formally: P(Y-hat = 1 | A = a, Y = 1) = P(Y-hat = 1 | A = b, Y = 1) for all groups a and b.

• What it measures: Whether the model identifies positive cases equally well across groups, without requiring that false positive rates are also equal.
• When to use: Appropriate when the primary concern is ensuring that all groups benefit equally from the model's ability to detect the condition (e.g., equal sensitivity for detecting cancer across racial groups), and unequal false positive rates are considered acceptable.
• Limitations: Does not protect against disproportionate false positives in any group, which can lead to overtreatment or unnecessary follow-up procedures.

Calibration Across Subgroups

A model is calibrated across subgroups if, for any given predicted probability p, the actual probability of the event is the same across groups. Formally: P(Y = 1 | Y-hat = p, A = a) = P(Y = 1 | Y-hat = p, A = b) for all groups a and b.

• What it measures: Whether the model's confidence scores are equally reliable across groups. If the model predicts a 70% probability of malignancy, that prediction should be equally accurate regardless of the patient's race.
• When to use: Critical for medical AI applications where clinicians rely on probability scores to make treatment decisions. Miscalibration across groups can lead to systematic under-treatment or over-treatment.
• Limitations: A model can be well-calibrated across groups while still having different error rates (a model can assign lower predicted probabilities to one group, leading to fewer positive classifications even when calibrated). Calibration should be evaluated alongside other metrics.

Counterfactual Fairness

Counterfactual fairness requires that the model's prediction for an individual would be the same if that individual's protected attribute were different, holding all other attributes constant. Formally: P(Y-hat = 1 | do(A = a), X) = P(Y-hat = 1 | do(A = b), X).

• What it measures: Whether the model's predictions are causally independent of protected attributes. This is a stronger condition than the correlation-based metrics above.
• When to use: Appropriate when there is a clear causal model linking protected attributes to other features, and the goal is to ensure that the model does not use protected attributes (or their proxys) in its decision-making.
• Limitations: Requires causal modeling assumptions that may not be testable from observational data alone. Difficult to implement for complex medical AI systems where the relationship between demographics and clinical features is deeply intertwined.

Comparison of Bias Testing Methods

Metric	What It Measures	When to Use	Key Limitation
Demographic parity	Equal positive prediction rates across groups	Resource allocation, screening decisions where equal access is the priority	Ignores genuine prevalence differences
Equalized odds	Equal TPR and FPR across groups	Diagnostic AI where both missed diagnoses and false alarms matter equally	May reduce overall accuracy
Equal opportunity	Equal TPR across groups	Detection tasks where missed positive cases are the primary concern	Does not address FPR disparities
Calibration across subgroups	Equal reliability of predicted probabilities	Risk scoring and clinical decision support where probability estimates guide care	Can coexist with unequal classification rates
Counterfactual fairness	Predictions invariant to protected attributes	When causal fairness is required and a causal model is available	Requires untestable causal assumptions

Planning a Bias Testing Program

A structured bias testing program should be integrated into the device development lifecycle from the earliest stages, not bolted on as a final check before submission.

Step 1: Identify Protected Subgroups

Determine which demographic dimensions are relevant for bias testing. At minimum, these should include:

• Race and ethnicity: Use self-identified categories consistent with OMB standards (U.S.) or the target population demographics for other markets
• Sex: Include male, female, and where data permits, broader gender categories
• Age: Stratify into clinically relevant age bands (e.g., pediatric, adult, geriatric)
• Additional clinically relevant factors: Comorbidity status, disease severity, socioeconomic proxies (insurance type, geographic region), imaging equipment type, and clinical setting

The choice of subgroups should be driven by clinical knowledge of the condition and known disparities in the relevant clinical domain. For a cardiac monitoring AI, race and sex are critical because cardiovascular disease manifests differently across these groups. For a dermatology AI, skin phototype is the primary dimension of concern.

Step 2: Select Fairness Metrics

Choose fairness metrics based on the clinical context and risk profile of the device:

• For diagnostic AI (e.g., detecting pathology on imaging): Equalized odds is typically the most clinically relevant metric, as both missed diagnoses and false alarms have patient safety consequences
• For screening AI (e.g., prioritizing cases for review): Equal opportunity may be sufficient, ensuring that positive cases are detected at equal rates across groups
• For risk prediction AI (e.g., estimating disease probability): Calibration across subgroups is essential, as clinicians base treatment decisions on probability estimates
• For triage or resource allocation AI: Demographic parity may be relevant to ensure equitable distribution

In most cases, manufacturers should evaluate multiple metrics rather than relying on a single one. A model that satisfies equal opportunity but has poor calibration across groups could still produce clinically significant bias.

Step 3: Define Acceptable Performance Bounds

Regulators have not prescribed numerical thresholds for acceptable performance disparities across subgroups. Manufacturers must define their own risk-based criteria:

• Establish minimum acceptable performance for each subgroup based on clinical risk analysis
• Define the maximum acceptable disparity between the best-performing and worst-performing subgroups for each key metric
• Document the clinical rationale for these thresholds, referencing the severity of harm from incorrect predictions and the availability of human oversight

For a device that detects life-threatening conditions (e.g., pneumothorax on chest X-ray), even small disparities in sensitivity across racial groups may be unacceptable. For a device that provides supportive information with mandatory physician review, larger disparities may be tolerable if documented and mitigated through labeling.

Step 4: Ensure Adequate Sample Sizes

Subgroup analyses require sufficient sample sizes to produce meaningful results. A fairness metric computed on 15 patients from an underrepresented group has a confidence interval so wide that the metric is essentially uninformative. Before conducting subgroup analyses, perform a power analysis to determine the minimum sample size needed to detect a clinically meaningful performance disparity with adequate statistical power. If existing data is insufficient, this is a signal that additional data collection is needed before the device can be adequately validated.

Recommended Reading

Medical Device Clinical Trial Cost: Complete 2026 Budget Breakdown from Early Feasibility Through Pivotal Studies

Clinical Evidence Regulatory2026-05-13 · 31 min read

Data Requirements: Ensuring Representative Datasets

The quality and representativeness of the data used to train and validate an AI-enabled medical device are the single most important determinants of its fairness. No post-hoc statistical correction can fully compensate for training data that systematically excludes or underrepresents patient populations.

Assessing Dataset Representativeness

For each dataset used in development (training, validation, and test), report:

• The total number of patients or samples
• Demographic breakdown by race, ethnicity, sex, age, and other relevant subgroups
• Geographic and institutional sources of data (number of sites, countries, and clinical settings)
• Inclusion and exclusion criteria applied during data collection
• Comparison of dataset demographics to the known epidemiology of the target condition and the demographics of the intended use population

When the dataset demographics deviate from the target population, document the deviation and its potential impact on model performance. Describe any steps taken to address the imbalance, such as oversampling underrepresented groups, stratified sampling, or reweighting during training.

Addressing Dataset Shift

Dataset shift -- differences between the data distribution on which the model was trained and the distribution it encounters in clinical use -- is a primary cause of algorithmic bias in medical AI. Several types of shift are relevant:

• Covariate shift: The distribution of input features changes between training and deployment (e.g., the model was trained on data from academic medical centers but is deployed in community hospitals with different patient populations and imaging equipment)
• Prior probability shift: The prevalence of the target condition changes (e.g., a model trained in a specialist referral population is used in a general screening population with lower disease prevalence)
• Concept shift: The relationship between input features and the target outcome changes (e.g., new disease variants, updated diagnostic criteria, or changes in treatment protocols)

To address dataset shift:

• Evaluate model performance on external datasets from institutions and populations not represented in training
• Monitor input data distributions in post-market deployment and flag when distributions diverge from training data
• Establish retraining protocols triggered by detected shifts, with appropriate regulatory documentation (PCCP or new submission, as applicable)

The MEDFAIR Benchmark Findings

The MEDFAIR benchmark, which evaluated 11 fairness techniques across 10 medical imaging datasets, provides a sobering data point for manufacturers. The study found that current state-of-the-art fairness methods do not significantly outperform "fairness through unawareness" -- the naive approach of simply removing protected attributes from the model input. This finding suggests that algorithmic fairness interventions are not a substitute for representative data. The most effective strategy for reducing bias remains collecting and curating training data that adequately represents the intended use population.

Bias Mitigation Strategies

Bias mitigation techniques are categorized by the stage of the machine learning pipeline at which they are applied: pre-processing, in-processing, and post-processing. Each category has distinct trade-offs.

Pre-Processing Methods

Pre-processing methods modify the training data before model training to reduce bias:

• Resampling: Oversample underrepresented groups or undersample overrepresented groups to create a more balanced training dataset. Simple but can introduce overfitting (oversampling) or information loss (undersampling).
• Reweighting: Assign higher training weights to examples from underrepresented groups or underrepresented combinations of features. The model's loss function gives more emphasis to getting these examples right without discarding data.
• Disparate impact remover: Transform features to remove correlation with protected attributes while preserving rank-ordering within groups. Useful when protected attributes are proxies for other features in the dataset.
• Learning fair representations: Use representation learning to create a transformed feature space that encodes the information needed for prediction but minimizes information about protected attributes.

Pre-processing methods are attractive because they do not require modifying the model architecture or training procedure. However, they may not fully address bias that arises from the model's learning dynamics, and they cannot correct for data that is fundamentally missing.

In-Processing Methods

In-processing methods modify the model training procedure to incorporate fairness constraints:

• Adversarial debiasing: Train a second model (adversary) to predict the protected attribute from the main model's predictions, and train the main model to maximize prediction accuracy while minimizing the adversary's ability to infer the protected attribute. This forces the model to learn representations that are predictive of the outcome but not of group membership.
• Fairness-constrained optimization: Add fairness constraints directly to the model's objective function. For example, the model minimizes prediction loss subject to the constraint that equalized odds is satisfied within some tolerance.
• Prejudice remover regularizer: Add a regularization term that penalizes the model for relying on protected attributes or their statistical proxies.

In-processing methods are more powerful than pre-processing methods because they directly shape what the model learns. However, they require more engineering effort and may affect model convergence, training time, and architecture complexity.

Post-Processing Methods

Post-processing methods modify the model's outputs after prediction to improve fairness:

• Threshold adjustment: Apply different decision thresholds for different demographic groups to equalize a chosen fairness metric (e.g., adjusting the threshold for each group so that the false positive rate is equal across groups). This is one of the simplest and most transparent bias mitigation approaches.
• Calibration adjustment: Apply group-specific calibration corrections to ensure that predicted probabilities are equally reliable across groups.
• Reject option classification: For predictions near the decision boundary (where the model is least confident), defer to human review rather than making an automated classification. This is particularly useful in clinical settings where human oversight is already expected.

Post-processing methods are easy to implement and do not require retraining the model. However, they address symptoms rather than root causes -- the underlying model may still produce systematically biased intermediate representations, and post-hoc adjustments may not generalize to new data distributions.

Choosing a Mitigation Strategy

In practice, most manufacturers should combine approaches:

1. Start with pre-processing to ensure the training data is as representative and balanced as possible
2. Apply in-processing methods during model development to learn fairer representations
3. Use post-processing (particularly threshold adjustment and calibration) to fine-tune subgroup performance
4. Validate the combined approach on independent test data stratified by subgroup

Document all mitigation methods used, their parameters, and their impact on both overall and subgroup performance. Regulators expect transparency about what was attempted and what was achieved.

Documenting Bias Analysis for Regulatory Submissions

Bias analysis documentation should be integrated into the device's technical file, not isolated in a separate appendix. The following framework outlines what to include in FDA and EU submissions.

FDA Submission Content

For 510(k), De Novo, and PMA submissions containing AI-enabled device functions, include the following in the software and clinical sections:

Software Documentation (per IEC 62304 and TPLC guidance):

• Description of the AI/ML model architecture, training methodology, and feature engineering
• Training data characteristics: number of samples, demographic composition, data sources, inclusion/exclusion criteria, and comparison to intended use population
• Validation data characteristics: same parameters as training data, confirming independence from training data and representativeness
• Bias analysis methodology: fairness metrics selected, rationale for selection, statistical methods used, and pre-specified thresholds for acceptable performance disparities
• Subgroup performance results: performance metrics stratified by race, ethnicity, sex, age, and other relevant subgroups, with confidence intervals
• Bias mitigation documentation: description of all pre-processing, in-processing, and post-processing techniques applied, with before-and-after performance comparisons
• Model card: structured metadata summary following the FDA's example format from the TPLC guidance
• Known limitations: explicit statement of subgroups for which performance data is insufficient or where performance disparities exceed predefined thresholds

Clinical Evidence Section:

• Clinical study design, including stratification strategy for subgroup enrollment
• Pre-specified subgroup analysis plan
• Results by subgroup with appropriate statistical tests
• Clinical significance assessment of any observed performance disparities

Labeling:

• Description of the populations for which the device has been validated
• Statement of any known limitations in performance for specific subgroups
• Instructions for use that account for potential bias (e.g., recommendation for additional clinical evaluation for patients in underrepresented groups)

EU Technical File Content

For CE marking under the MDR with AI Act compliance:

• Clinical evaluation report: Subgroup performance analysis integrated into clinical evidence, demonstrating safety and performance across the European patient population
• Risk management file (ISO 14971): Algorithmic bias documented as a hazard, with risk analysis considering the severity and probability of harm from biased predictions for each subgroup
• AI Act conformity assessment documentation: Evidence of compliance with data governance requirements, risk management, transparency, human oversight, and accuracy/robustness requirements
• Statement of representativeness: Explicit comparison of training data demographics to the European target population (which may differ significantly from U.S. training data demographics)
• Post-market surveillance plan: Includes specific provisions for monitoring subgroup performance in the European market

Practical Submission Tips

• Present subgroup performance in clearly formatted tables with confidence intervals, not just point estimates
• Use forest plots to visualize performance across subgroups, making disparities immediately apparent to reviewers
• If subgroup sample sizes are too small for meaningful analysis, state this explicitly and describe the plan for collecting additional data post-market
• Do not bury unfavorable subgroup results -- regulators will find them, and transparency builds credibility
• Include a narrative interpretation of bias analysis results written for clinical reviewers, not just data scientists

Recommended Reading

NGS Diagnostic Devices Regulatory Guide: FDA, EU IVDR, Companion Diagnostics, and Bioinformatics Pipelines

IVD & Diagnostics Regulatory2026-04-30 · 13 min read

Post-Market Bias Monitoring

Bias testing does not end at the time of market authorization. The FDA's TPLC framework and the EU AI Act both require ongoing monitoring throughout the device's lifecycle. Post-market bias monitoring serves two purposes: detecting performance degradation across subgroups (algorithmic drift) and identifying emerging biases that were not present during development.

Monitoring for Algorithmic Drift

Algorithmic drift occurs when a model's performance changes over time due to shifts in the input data distribution. To monitor for drift:

• Track input feature distributions: Monitor the statistical distribution of input features in production data compared to training data. Significant shifts in demographic composition, clinical characteristics, or imaging parameters may indicate emerging bias risk.
• Track prediction distributions: Monitor the distribution of model predictions across subgroups over time. A sudden change in the positive prediction rate for one demographic group, without a corresponding change in disease prevalence, may indicate drift.
• Collect outcome data where possible: Where ground truth labels are available (e.g., through follow-up clinical assessment, biopsy results, or adjudicated outcomes), compare model predictions to actual outcomes stratified by subgroup. This is the most direct measure of performance drift but requires clinical data infrastructure.
• Set statistical thresholds: Establish pre-specified thresholds for acceptable changes in subgroup performance metrics, with escalation procedures when thresholds are exceeded.

Reporting Adverse Bias Events

Under both FDA and EU requirements, clinically significant performance disparities that constitute patient safety risks must be reported through existing adverse event reporting mechanisms:

• FDA: Performance disparities that result in missed diagnoses or inappropriate treatment recommendations for specific demographic groups may meet the threshold for Medical Device Reports (MDRs) under 21 CFR 803
• EU MDR: Significant bias-related performance issues should be captured in Periodic Safety Update Reports (PSURs) and addressed through the post-market surveillance plan
• EU AI Act: The AI Act requires reporting of serious incidents involving AI systems, which includes harm resulting from biased predictions

Post-Market Data Collection

Post-market clinical follow-up (PMCF) for AI-enabled devices should include specific provisions for bias monitoring:

• Enroll patients from demographic subgroups that were underrepresented in pre-market validation
• Collect structured demographic data to enable subgroup performance analysis
• Establish partnerships with diverse clinical sites to ensure broad population coverage
• Use the PCCP framework to implement corrective actions (such as model retraining with more representative data) when post-market data reveals clinically significant bias

The Accuracy-Fairness Trade-Off

One of the most challenging aspects of algorithmic fairness in medical AI is the potential trade-off between overall accuracy and fairness across subgroups. In many cases, maximizing overall accuracy produces a model that performs best for the majority group at the expense of minority groups. Conversely, optimizing for strict fairness may reduce overall accuracy.

Why the Trade-Off Exists

The trade-off arises because fairness constraints restrict the set of models that can be selected. If the optimal model (in terms of overall accuracy) happens to satisfy the fairness constraint, there is no trade-off. But in practice, the unconstrained optimal model often has higher accuracy for the majority group than for minority groups, and imposing fairness constraints requires accepting slightly lower overall accuracy to achieve more equitable performance.

How to Navigate It

• Make the trade-off explicit: Document the impact of fairness constraints on overall accuracy and on each subgroup's performance. Present this analysis to clinical stakeholders and regulatory reviewers with a clear explanation of the trade-off.
• Use clinical risk to set priorities: For a device that detects a life-threatening condition, equity in sensitivity (missed diagnosis rate) across groups may be worth a small reduction in overall specificity. The clinical consequences of a missed cancer diagnosis in an underrepresented group typically outweigh the consequences of an additional false positive.
• Consider the device's role: For devices with mandatory physician oversight, small accuracy trade-offs may be more acceptable because the human clinician provides a safety net. For autonomous AI devices, stricter fairness requirements are warranted.
• Document the decision: Regulators do not expect perfect fairness. They expect a transparent, reasoned analysis of the trade-off and a documented decision about where to land on the accuracy-fairness spectrum.

Building a Compliance-Ready Bias Testing Program

For manufacturers preparing to bring AI-enabled medical devices to market in 2026 and beyond, the following checklist summarizes the key elements of a compliance-ready bias testing program:

Element	Description
Subgroup identification	Define protected and clinically relevant subgroups based on the intended use population
Representative data	Ensure training, validation, and test datasets reflect the demographic composition of the target population
Metric selection	Choose fairness metrics appropriate to the clinical context (equalized odds for diagnostic AI, calibration for risk prediction, etc.)
Threshold definition	Establish risk-based thresholds for acceptable performance disparities, with clinical rationale
Power analysis	Confirm adequate sample sizes for subgroup-level performance estimation
Bias mitigation	Apply pre-processing, in-processing, and/or post-processing methods as appropriate
Subgroup performance reporting	Present stratified performance metrics with confidence intervals for all relevant subgroups
Model card	Provide structured metadata per FDA TPLC guidance example format
Technical file integration	Embed bias analysis in software documentation, clinical evidence, and risk management file
Labeling	Include subgroup-specific performance information and limitations in device labeling
Post-market monitoring	Establish ongoing surveillance for algorithmic drift with subgroup-stratified analysis
PCCP provisions	Include bias-related retraining triggers and corrective action protocols in the predetermined change control plan

Recommended Reading

Medical Device Cybersecurity Patch Management: Regulated Update Deployment Under EU MDR, FDA Section 524B, and the Cyber Resilience Act (2026)

Cybersecurity EU MDR / IVDR2026-05-13 · 31 min read

Looking Ahead

The regulatory expectations for AI bias testing in medical devices will continue to tighten. Several developments are likely in the near term:

• Finalization of the FDA TPLC guidance: The January 2025 draft guidance is expected to be finalized, potentially with additional specificity on bias testing requirements and model card content.
• EU AI Act enforcement: As the August 2026 and August 2027 compliance deadlines approach, notified bodies will increasingly scrutinize bias analysis in AI-enabled device technical files.
• International harmonization: The IMDRF and other international bodies are developing guidance on AI/ML medical device evaluation that will likely include bias testing expectations.
• Advancing methodology: Fairness metrics and bias mitigation techniques are active areas of research. The current limitations identified by benchmarks like MEDFAIR will drive development of more effective methods, but the fundamental importance of representative data is unlikely to change.
• Post-market evidence: Real-world performance data from deployed AI devices will increasingly inform regulatory expectations, as regulators move from theoretical concerns to documented evidence of bias in clinical practice.

Manufacturers who invest in robust bias testing programs now -- not as a compliance checkbox but as a core element of clinical validation -- will be better positioned for regulatory success and, more importantly, will produce devices that serve all patients equitably. The cost of building representative datasets and conducting thorough subgroup analyses is measured in development time and data acquisition. The cost of deploying biased AI in clinical practice is measured in patient harm.