How AI works
Which AI systems can have their logic audited, and which cannot?
Short answerRule-based systems, yes: a person wrote the logic down, so you can read the rule that produced a result. Learned systems, no. There is no written reasoning to read, only behavior to measure and test.
| Rule-based | Learned | |
|---|---|---|
| Who wrote the logic | A person | Nobody; the system derived it from data |
| Where it lives | Rules, formulas, and tables you can read | Billions of numbers no one can read |
| Examples | An amortization formula, a tax table, an eligibility rule | A fraud model, a language model, an image classifier |
| What an audit can do | Point at the line that produced a result | Observe and test behavior |
Rule-based systems are deterministic. You can read the rule, point at the line that produced a result, and argue about whether the rule is right.
Learned systems derived their logic from data during training, and the result is stored as weights: billions of numbers that nobody, including the people who trained the model, can read as reasons. The behavior can be observed and tested. The reasoning cannot be inspected, because no written reasoning exists.
Where the law draws the same line
The word doing the work is infers. The test looks at what a system does, and it is the line the Act uses to separate AI from conventional software. California AB 316, in force since 1 January 2026, defines AI by the same test. For an organization, this decides which of its systems those laws reach: an engine applying a fixed eligibility table is ordinary software, and a model trained to score the same applications is an AI system.
Most real systems are hybrids
Most production deployments wrap a learned model in deterministic guardrails. The model is learned; the fence around it is written. So an audit splits in two. The guardrails can be read line by line: what the system may attempt, what it refuses, and what it records. The model inside can only be tested.
That split is the useful one. "No customer account numbers leave the organization in a prompt" is a rule someone wrote, and it can be audited. "Explain why the model drafted this paragraph" cannot be answered from the model at all.
What this means for explainability requirements
A requirement to explain a decision can be met on a rule system: produce the rule. On a learned system there is no rule to produce, so the same words become a requirement nobody can meet. The answer that gets written is a document describing the model in general terms, and that document is what most explainability mandates produce.
- Sort your systems first. For each AI system in use, record whether its decision logic was written or learned. A policy that asks both kinds for an explanation is asking for two different things.
- Ask vendors which part is which. Which behavior comes from rules you can read, and which from a model you can only test? A vendor who cannot separate the two cannot tell you what an audit would find.
- Write requirements against the fence. For learned components, ask for test results and a record of what the system did. Keep line-by-line explanation for the parts a person wrote.
Sources
- European Union, Artificial Intelligence Act, Article 3(1), definition of an AI system; Articles 14 and 15.
- California Assembly Bill 316 (Krell), Artificial intelligence: defenses, chaptered 13 October 2025, effective 1 January 2026. Defines AI by whether it infers from input how to generate outputs.
- Orrick, EU Commission Clarifies the Definition of AI Systems, April 2025. On "infers" as the line separating AI from conventional software.
- National Institute of Standards and Technology, AI Risk Management Framework 1.0. Trustworthiness characteristics.
Reviewed