Judging AI governance

Who built the AI model, and on what?

Short answerA named model from a named provider. Almost nobody trains their own frontier model, so most AI products are built on someone else's, and whoever buys one inherits that provider's training data, terms, and update schedule along with it.

The question tests whether a supplier understands its own supply chain. A company building on another firm's frontier model inherits three things from it.

  • Training data. What the model learned from, and how much of that is known.
  • Terms. The provider's conditions, which sit underneath the supplier's contract with you.
  • Update schedule. When the provider changes the model, the system behaves differently, with nothing in the supplier's software or yours changing at all.

The update nobody shipped

The third is the one that matters once a system is in production. A test passed last month describes last month's model. If the provider can swap what sits underneath, the product you approved and the product your people use today can quietly diverge.

A supplier who cannot answer is usually hiding nothing. More often they have never had to look. Either way, the answer tells you how much of the system is yours to govern.

Open or closed weights

Open weights means the trained numbers are published, so you can download the model, run it, and host it yourself. What stays out of view is what it was trained on and exactly how. Nearly every model marketed as "open" is open weights only, so the training data is closed in both cases. The difference is worked through in open weights vs. open source, and what a model weight is covers what the numbers themselves are.

How to ask it

A good answer
A named model, a named provider, a statement of whether the weights are open or closed, and whatever is actually known about the training data.
An evasion
"Proprietary technology." "Our own advanced AI." Almost nobody trains their own frontier model; most build on someone else's.
The follow-up
If the provider changed their model tomorrow, would I know?

What to check

  • For each AI product in use, write down the model and the provider underneath it.
  • Ask each supplier how they learn of a model change, and how and when they tell you.
  • Find whose terms govern your data once it reaches the model provider. That is half of where your data goes.

Sources

  1. National Institute of Standards and Technology, AI Risk Management Framework 1.0. Trustworthiness characteristics.
  2. National Telecommunications and Information Administration, Dual-Use Foundation Models with Widely Available Model Weights, July 2024.

Reviewed