Which AI systems can have their logic audited, and which cannot?

Rule-based systems are deterministic. You can read the rule, point at the line that produced a result, and argue about whether the rule is right.

Learned systems derived their logic from data during training, and the result is stored as weights: billions of numbers that nobody, including the people who trained the model, can read as reasons. The behavior can be observed and tested. The reasoning cannot be inspected, because no written reasoning exists.

Where the law draws the same line

The word doing the work is infers. The test looks at what a system does, and it is the line the Act uses to separate AI from conventional software. California AB 316, in force since 1 January 2026, defines AI by the same test. For an organization, this decides which of its systems those laws reach: an engine applying a fixed eligibility table is ordinary software, and a model trained to score the same applications is an AI system.

Most real systems are hybrids

Most production deployments wrap a learned model in deterministic guardrails. The model is learned; the fence around it is written. So an audit splits in two. The guardrails can be read line by line: what the system may attempt, what it refuses, and what it records. The model inside can only be tested. See AI guardrails best practices, from proof of concept to enterprise for how guardrails are set up from proof of concept through production.

Sentinel, an AI gateway, is an example of the written fence. An admin sets its rule, either replacing the values it catches or blocking the request, and it records counts, which rule fired, and latency for each request. The model behind it can still only be tested.

That split is the useful one. "No customer account numbers leave the organization in a prompt" is a rule someone wrote, and it can be audited. "Explain why the model drafted this paragraph" cannot be answered from the model at all.

What this means for explainability requirements

A requirement to explain a decision can be met on a rule system: produce the rule. On a learned system there is no rule to produce, so the same words become a requirement nobody can meet. The answer that gets written is a document describing the model in general terms, and that document is what most explainability mandates produce.

• Sort your systems first. For each AI system in use, record whether its decision logic was written or learned. A policy that asks both kinds for an explanation is asking for two different things.
• Ask vendors which part is which. Which behavior comes from rules you can read, and which from a model you can only test? A vendor who cannot separate the two cannot tell you what an audit would find.
• Write requirements against the fence. For learned components, ask for test results and a record of what the system did. Keep line-by-line explanation for the parts a person wrote.