A job applicant applied to more than 100 roles through the same recruiting platform and was rejected from every one, many within the hour. The applicant's case anchors a federal case in California's Northern District, arguing that AI-based hiring systems unlawfully screened out qualified applicants in violation of the Age Discrimination in Employment Act (ADEA). The court rejected the vendor's core defense that the ADEA doesn't apply to autonomous software.

How AI bias became an operational risk

Most AI conversations over the past five years have focused on what a model would say, how accurate it was, and whether it hallucinated. LLM-based technology now drives a growing population of AI agents, which make API calls, modify records, trigger payments, and escalate cases, sometimes to a person and sometimes to another agent. None of those actions get reviewed before they take effect.

This isn't new. In 2023, another company's recruiting software drew a similar complaint. A job applicant tested the software by submitting two nearly identical applications, differing only in one field, their date of birth. The version listing an older birth date was auto-rejected within seconds. The version listing a younger birth date got an interview. According to the U.S. Equal Employment Opportunity Commission (EEOC), the software had been set to automatically reject female applicants 55 and older and male applicants 60 and older, screening out more than 200 people. The company settled the EEOC's claim for $365,000 without admitting wrongdoing, in what's widely reported as the EEOC's first AI-related discrimination settlement.

That case makes two things concrete that used to feel theoretical. The first is bias, which AI doesn't create but rather industrializes the patterns, judgments, and assumptions embedded in the data and decisions that inform it. The second is accountability. Who answers for the outcome when a system acts on the organization's behalf, and no one signed off on the action before it happened?

"Technically, AI is not inventing bias. It's scaling and industrializing it."

Sri Krishnamurthy, Quant University

AI models train on such large volumes of data that removing every trace of bias is close to impossible. Anyone using AI needs to assume bias is present and manage it for their specific use case, rather than expect it to be handled for them. There's a general assumption that model providers have already addressed bias at the source. They haven't.

In our webinar on AI ethics and bias, one of our speakers summed up this shift as garbage in, disaster out. In other words, the input problem no longer stays contained inside the model but instead travels downstream to the action.

Why fairness metrics don't make AI explainable

A fairness score and a disparate-impact score both describe patterns across a dataset, not why an applicant was rejected.

Picture that 2023 case again, but with a different defense. Imagine the company had argued that its fairness metrics showed acceptable outcomes across the applicant pool. That still wouldn't explain why one person changed only their date of birth and received a different outcome. Fairness metrics describe how a system behaves in aggregate; they don't explain why a specific decision was made.

With that in mind, a model can appear fair at the population level while still producing outcomes that need to be justified one decision at a time. Audits, therefore, need to target the decision that was made, since the model on its own might not carry the explanation for it. In practice, that means checking the input distribution the decision drew on, which features drove the outcome, how the outcome would have changed under a counterfactual input, how the system holds up under stress and scenario testing, and what's happening in production right now, all at the point of decision rather than the point of training.

The more decisions an organization automates, the more of them it must be able to defend on demand, whether to satisfy compliance, win board approval, or account for the growth and risk those decisions carry. AI governance, both within organizations and through regulation, is adjusting to that reality. While some rules are tightening and some regulators have softened their stance in the past year, few of these frameworks, old or new, were built to assess decisions made this fast, or defended this quickly when challenged.

How knowledge graphs help make an AI decision auditable

Auditing a decision without the relationships, history, and constraints it was made against is hard. A model output without context is just a number, and a number is harder to defend than an answer grounded in why it was produced.

Knowledge graphs resolve scattered relationships, informing a context layer that's navigable and traceable underneath the AI, rather than something an analyst needs to reconstruct after the fact. 

Our ebook, How to Build Trustworthy Intelligence in a Connected World, illustrates this. Ask a retrieval-augmented large language model (LLM) who the ultimate beneficial owner of a company is, and it goes hunting through documents to find an answer. Ask the same question of a knowledge graph, and it follows the ownership connections directly, leaving a path you can walk back through. But a knowledge graph is only plausibly correct by using entity resolution first. If the same customer, prospect, business or identifier shows up as three different records, the graph fragments, and the audit built on top of it fails with it.

What it takes to make an AI action defensible

Organizations need actions they can stand behind in public, tested at the same level of scrutiny as the model itself.

Your AI could already be industrializing the bias sitting in your data. What determines whether your organization can stand behind its AI is what happens the moment that bias becomes an action.

Our ebook walks you through five layers in which trust must live for an AI-driven action to hold up, shows where most organizations are already exposed, and how to fix.

Read the ebook to learn more: How to Build Trustworthy Intelligence in a Connected World