Skip to content
Layer 4The governance stack/ Applications

Application Governance

Does the system behave — and can we prove it?

The governance of AI systems, not just models: grounding and citation discipline, prompts and configurations under change control, output validation and egress handling, application identity and entitlements, release gates with safety regression, and scope fences that decide what the system may not do. The unit of validation moved from the model to the pipeline — retrieval, prompt, model, tools, and UI together.

The risk, in one line

Your customers, courts, and regulators judge the application's words — not the model card behind them.

Why leadership should care

  • Courts hold companies liable for what their AI applications say: Air Canada lost on a policy its chatbot invented; NYC's business bot dispensed illegal advice under official branding for months.
  • A one-line prompt change is a production change to behavior — the Grok incident shipped extremist output for hours because a prompt edit skipped safety regression.
  • Scope is the cheapest safety control: Wendy's fenced its drive-thru AI to the menu with confidence-based human handoff and hit 86% autonomous completion at ~99% accuracy with assists.
86%
of Wendy's drive-thru orders completed with no human intervention — scope fences plus handoff design, not model magic
Decisions only leadership can make
  • Which decisions may AI applications make alone, and which require a human at the point of consequence?
  • Do prompts and AI configs go through the same change control as code? (They are code.)
  • What may our customer-facing AI never discuss — price, contract, advice — and where is that enforced?
Read the claims right
PracticeOWASP Top 10 for LLM Applications (2025)RegulationEU AI Act Art. 50 transparencyPracticeNIST MEASURE / MANAGERegulationHuman-oversight design (Art. 26 shape)
How Google Cloud approaches this

Cards link to official documentation. Status is a snapshot (August 2026) — verify per component before contractual commitments. Full mapping and honest gaps: 08 · Google Cloud.

In the room — discussion points

The model is 20% of the system; govern the pipeline that wraps it — that's where incidents actually start.

  • Deutsche Bank publishes its eval regime — citation precision and recall, false-rejection rates, git-pinned baselines. That's what audit-ready gen AI concretely looks like.
  • Treat prompts as production code. Half of the public AI incidents trace to an untested prompt or config change.
  • The strongest deployments constrain scope first: menu-only, policy-corpus-only, intent-extraction-only. Capability follows trust.
  • Ask where output validation happens. If the answer is 'the model is aligned', there is no application governance.
Questions to ask your organization
  • For your most visible AI application: what may it never say or do, and where is that enforced?
  • What happens between a prompt edit and production today?
  • Can users and reviewers see sources for AI answers in high-stakes flows?
  • How do you sample and review production AI conversations for quality?
  • Who owns an AI behavior incident at 2am, and what is the kill/fix path?