Reporting and observability are routinely sold as the same thing. A report is retrospective and self-authored: the party being evaluated chooses the period, the metrics, and the framing. Observability shows what is happening now, and nobody under evaluation gets to choose what it shows.
| Reporting | Observability |
|---|---|
| When | After the period ends |
| Who chooses what is shown | The party being evaluated |
| What it tells you | What happened, as described |
Who holds the instrument
The question also settles who holds the instrument. If the only view into a system is the vendor's dashboard, oversight exists at the vendor's discretion. That is a commercial arrangement, and it lasts as long as the contract says it does. A control is something your own people can use when they decide to.
Three things worth seeing live
- Usage. Who is using which AI, and how much.
- Cost. What that use costs. Running a model is the permanent cost, so the view has to keep pace with use.
- Rule triggers. Where a rule fired and what it caught.
Sentinel, an AI gateway, shows part of this for the prompts it handles. It records counts, which rule fired, and latency for each request, and requests are measured and shown in the account.
Watching a running system matters more than it first appears. Some properties of AI can be guaranteed in advance and some never can, and for the second kind, watching what the system actually does is the only verification there is.
How to ask it
- A good answer
- Live visibility into who is using what, at what cost, and where rules were triggered, on a screen the organization can open itself.
- An evasion
- "We provide quarterly reports." A report is a description of the past, delivered by the party being evaluated.
- The follow-up
- Can my own people see that without asking you?
What to check
- Find out who in your organization can open a live view of AI use today without filing a request.
- Check whether that view shows all three of usage, cost, and rule triggers.
- If the view belongs to the vendor, read what the contract says about your access to it, and what happens to it at renewal.
Frequently asked questions
- What is the difference between reporting and observability?
- A report is retrospective and self-authored: the party being evaluated chooses the period, the metrics, and the framing. Observability shows what is happening now, and nobody under evaluation gets to choose what it shows. Reporting arrives after the period ends, while observability is live and independent of the vendor.
- What should you be able to see live about your AI?
- Three things: usage, meaning who is using which AI and how much; cost, meaning what that use costs; and rule triggers, meaning where a rule fired and what it caught. Running a model is the permanent cost of AI, so the view has to keep pace with use.
- Why does it matter who holds the monitoring dashboard?
- If the only view into a system is the vendor's dashboard, oversight exists at the vendor's discretion. That is a commercial arrangement, and it lasts as long as the contract says it does. A control is something your own people can use when they decide to.
- How should you ask a vendor about visibility?
- Ask whether your own people can see live usage, cost, and rule triggers without asking the vendor. "We provide quarterly reports" is the evasion, since a report is a description of the past delivered by the party being evaluated. A good answer names a live screen the organization can open itself.
- What should you check in your own organization?
- Find out who can open a live view of AI use without filing a request. Check whether that view shows all three of usage, cost, and rule triggers. If the view belongs to the vendor, read what the contract says about your access to it and what happens to it at renewal.