Judging AI governance

How do you tell real AI governance from theater?

Short answerAsk five questions, in order: where the data goes, who built the model, whether the controls are enforced while the system runs, what you can see while it runs, and whether you can prove afterwards who did what. The third is the one almost everyone fails.

The five work on any AI system: a vendor's product, an internal project, a tool a team bought on a card. The first two are asked before deployment and answered from documents. The last three are asked of the running system and answered with evidence.

Two questions you are told the answer to

  1. Where does the data go? Where do prompts and documents go, how long are they kept, and are they used for training? An evasion sounds like "your data is secure." Security is a different question; destination is the one being asked.
  2. Who built this model, and on what? Which provider, open or closed weights, and what is known about the training data. An evasion sounds like "our own proprietary AI." Almost nobody trains their own frontier model.

Three questions you can test

  1. Are the controls enforced, or just written down? A policy document says what people are supposed to do. An enforced control stops what should not happen, whether or not anyone remembers the policy. The follow-up: show me it being enforced.
  2. What can you see while it is running? Live usage, cost, and rule triggers, on a screen your own people can open. A quarterly report is the past, described by the party being evaluated.
  3. Can you prove afterwards who did what? A complete, tamper-resistant record tying every interaction to an identified person or agent. The follow-up: if this went to litigation, what could you produce?
The line between the two halvesEverything in the first two answers is a claim you are told. Everything in the last three is something you can check. The first two tell you what a vendor intends; only the last three tell you what is true.

What the five do together

A system that answers the first two well and fails the last three is one you understand and cannot govern. You know the model, the provider, and where the data goes, and you have no way to stop anything, see anything, or prove anything afterwards. This is the common case, and it presents best in a meeting.

A system that answers the last three well is governable more or less whatever the first two say. If rules are enforced while it runs, if you can see it running, and if the record is yours, a change of model is a change of input.

That is the property underneath the whole test: governance that lives where the system runs survives the system changing. Governance written into each application has to be rewritten every time. It is also what decides whether one system can satisfy several regulatory regimes at once.

Sources

  1. National Institute of Standards and Technology, AI Risk Management Framework 1.0. Trustworthiness characteristics.

Reviewed