AI Bias: Where It Enters and How to Test It

A hiring model trained on past decisions can reproduce the company’s past discrimination. Removing a gender field does not necessarily solve the problem: other features may carry similar information, and the historical labels may already reflect unequal treatment.

Bias can enter through the data, the target being predicted, the training objective, or how a result is used. Improving one metric is only part of addressing it.

Why the choice of metric matters

A model with good average accuracy may still make costly errors for a particular group. In hiring, lending, or healthcare, those errors affect access to opportunities and services.

Before choosing a fairness metric, specify the harm you want to detect. Equal selection rates, similar error rates, and well-calibrated scores answer different questions. They are not interchangeable definitions of fairness.

Three places to intervene

Bias mitigation methods are commonly grouped by where they act in the pipeline:

1. Before training

Audit and change the training data. This may involve collecting missing examples, correcting labels, or reweighting samples. Rebalancing cannot repair a target that measures the wrong thing.

2. During training

Change the learning objective or add constraints. A fairness penalty can discourage a measurable disparity while the model learns to predict the target.

3. After training

Adjust how scores become decisions, for example by changing thresholds. The practical and legal suitability of a method depends on its use and jurisdiction; a mathematically attractive adjustment is not automatically appropriate to deploy.

The method has to fit the setting

Madeleine Waller, Odinaldo Rodrigues, Michelle Seng Ah Lee, and Oana Cocarascu examine these practical and legal limitations in Bias Mitigation Methods: Applicability, Legality, and Recommendations for Development.

Missing populations, biased labels, and proxy features can each require a different response. Removing protected attributes may also make disparities harder to measure. Teams need to consider what data they may collect and use, which harms their tests can detect, and what the deployment process does with the model’s output.

A small fairness-penalty example

Demographic parity compares selection rates across groups. The following toy example uses a differentiable proxy: the squared difference in mean predicted probabilities between two groups. It trains the same logistic model with and without that penalty, then measures accuracy and the selection-rate gap on held-out data.

The penalty uses probabilities during training because thresholded predictions do not provide a useful gradient. Matching mean probabilities does not guarantee equal selection rates after thresholding.

import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

rng = np.random.default_rng(42)
n = 200
sensitive = rng.integers(0, 2, n)
X = rng.normal(size=(n, 3)) + sensitive[:, None]
y = (X[:, 0] + X[:, 1] + 0.5 * sensitive
     + rng.normal(scale=0.5, size=n) > 0).astype(int)

X_train, X_test, y_train, y_test, s_train, s_test = train_test_split(
    X, y, sensitive, test_size=0.3, random_state=42, stratify=sensitive
)

def sigmoid(z):
    return 1 / (1 + np.exp(-np.clip(z, -50, 50)))

def fit(strength, steps=500, learning_rate=0.1):
    weights = np.zeros(X_train.shape[1])
    bias = 0.0
    group0, group1 = s_train == 0, s_train == 1

    for _ in range(steps):
        p = sigmoid(X_train @ weights + bias)
        error = p - y_train
        grad_w = X_train.T @ error / len(y_train)
        grad_b = error.mean()

        # Loss: cross-entropy + strength * (mean(p1) - mean(p0))**2
        gap = p[group1].mean() - p[group0].mean()
        slope = p * (1 - p)
        gap_grad_w = (
            (slope[group1, None] * X_train[group1]).mean(axis=0)
            - (slope[group0, None] * X_train[group0]).mean(axis=0)
        )
        gap_grad_b = slope[group1].mean() - slope[group0].mean()
        grad_w += 2 * strength * gap * gap_grad_w
        grad_b += 2 * strength * gap * gap_grad_b

        weights -= learning_rate * grad_w
        bias -= learning_rate * grad_b

    return weights, bias

def selection_rate_gap(predictions, groups):
    return abs(predictions[groups == 1].mean()
               - predictions[groups == 0].mean())

for label, strength in [("Baseline", 0.0), ("Penalized", 0.5)]:
    weights, bias = fit(strength)
    predictions = (sigmoid(X_test @ weights + bias) >= 0.5).astype(int)
    print(label)
    print("  Accuracy:", round(accuracy_score(y_test, predictions), 3))
    print("  Selection-rate gap:",
          round(selection_rate_gap(predictions, s_test), 3))

What this demonstrates

Both runs use the same data, initialization, optimizer, and number of steps. The second adds the gradient of the squared group-probability gap to the prediction-loss gradient.

This demonstrates one intervention on one synthetic dataset. It does not establish that either model is fair or suitable for a real decision. The outcome labels themselves contain group dependence, and the test sample is small. Inspect group-level errors and uncertainty before drawing conclusions from the aggregate scores.

Limits to keep in view

A lower disparity on one metric can coincide with worse results on another. Overall accuracy can hide group-specific harms, and matching an imperfect historical label may reinforce them.

The relevant question is whether the method reduces the harm identified in the actual setting, within the constraints that apply there. No single penalty or dashboard score answers that question.

A practical review

1. Identify who could be harmed and how the model’s output affects them.
2. Audit coverage, labels, and proxy features in the data.
3. Choose metrics that address those harms, including results for relevant subgroups.
4. Compare interventions on held-out data and review their operational and legal suitability.
5. Monitor outcomes after deployment and provide a route to challenge or correct decisions.

Testing outcomes also needs a way for people to challenge them. People Don’t Fear AI. They Fear Losing Agency. connects evaluation to control, evidence, and recourse.

Resources

• Waller and colleagues’ paper on mitigation methods and their limitations.
• Fairlearn’s guide to fairness metrics and the assumptions behind them.
• AI Fairness 360 for assessment and mitigation tools.
• Algorithmic Justice League and AI Now Institute for research and advocacy on the effects of automated systems.