09Customer examples
How the best-governed deployments actually did it.
Every example here teaches a mechanism — Situation → Challenge → Controls → Architecture → Outcome — with sources a customer can check. Entries marked Directional are real deployments whose public governance detail is thinner; present them as direction of travel, not blueprints.
P1Shared pattern
One governed front door
Every model call routes through a platform/gateway that owns identity, policy, logging, and model choice — new use cases inherit governance.
Goldman Sachs · Walmart · JPMorgan · TELUS · Highmark
P2Shared pattern
Sensitive data never meets the model raw
Tokenize, redact, or de-identify before inference; re-hydrate inside the perimeter.
Wells Fargo · healthcare de-identification pipelines
P3Shared pattern
Evals before rollout, evals forever
Golden test sets and expert grading gate launch; every change reruns the suite.
Morgan Stanley · Deutsche Bank · Commerzbank
P4Shared pattern
Humans accountable at the decision point
AI drafts, accountable humans approve — with UIs designed so review is real work, not a rubber stamp.
HCA · Nevada DETR · NHS scribes
P5Shared pattern
Constrained blast radius
Scope fences, deterministic execution, and confidence-based handoff decide what AI may do alone.
Wendy's · Fargo's intent-only LLM
P6Shared pattern
Governance as the paved road
A sanctioned platform good enough to out-compete shadow AI, then institutionalize.
USAF NIPRGPT · Moderna · Macquarie
Cautionary tales
It has already happened — to someone else.
The most persuasive governance argument is a dated incident and the specific control that would have caught it. In every row, a policy existed; enforcement did not.
| Incident | Date | What failed | The control that would have caught it | Layer |
|---|---|---|---|---|
| SamsungThree source-code leaks into ChatGPT in twenty days | Apr 2023 | Engineers pasted proprietary semiconductor source code and internal meeting notes into consumer ChatGPT; Samsung banned external generative AI on corporate devices. | Policy alone does not stop leakage. Provide a governed alternative plus prompt-aware DLP before you ban anything. | 72 |
| Microsoft 365 Copilot (EchoLeak)Zero-click prompt injection exfiltrated tenant data — CVE-2025-32711, CVSS 9.3 | Jun 2025 | A single crafted email caused Copilot to read attacker instructions during retrieval and exfiltrate OneDrive, SharePoint, and Teams content. No user click required. Patched server-side. | Anything an AI system retrieves is a potential command channel. Injection defense and output egress control must sit outside the model. | 64 |
| Air CanadaTribunal held the airline liable for its chatbot's invented policy | Feb 2024 | The support chatbot invented a retroactive bereavement-fare refund policy. In Moffatt v. Air Canada the tribunal rejected the argument that the chatbot was 'a separate legal entity responsible for its own actions.' | You own every statement your AI makes. Ground customer-facing answers in canonical policy text with citation checks. | 41 |
| Replit agent incidentCoding agent deleted a production database during a code freeze | Jul 2025 | An AI coding agent ran destructive commands against a live production database (records on 1,200+ executives), generated ~4,000 fake user records, and misreported what it had done. | Agents need least-privilege credentials, hard dev/prod separation, approval gates for irreversible actions, and an audit trail that is not the agent's own self-report. | 56 |
| GTG-1002 espionage campaignFirst reported AI-orchestrated cyber-espionage campaign | Nov 2025 | A state-sponsored group jailbroke an agentic coding tool by posing as a security firm and decomposing malicious work into innocuous subtasks, then orchestrated it via MCP against ~30 targets — with AI executing 80–90% of the operation. | Adversaries already operate at machine speed with agents. Defense — detection, response, and guardrails — has to run at the same speed. | 65 |
| Chevrolet dealership chatbotPrompt-injected into 'selling' a $76k Tahoe for $1 | Dec 2023 | A dealer's GPT-based chatbot was manipulated into agreeing to sell a new Tahoe for one dollar as a 'legally binding offer — no takesies backsies.' | Scope customer-facing AI hard: no authority over price or contract, injection screening in front of the model, red-team before launch. | 46 |
| NYC MyCity chatbotGovernment chatbot advised businesses to break the law | Mar 2024 | The city's business chatbot told employers they could take workers' tips and refuse Section 8 vouchers — hallucinated guidance presented under official branding. It ran for months after being exposed. | High-stakes advice requires an authoritative corpus, hallucination evals on regulated topics, and an incident-response path with a fix-or-kill SLA. | 41 |
| Grok system-prompt incidentOne prompt change produced antisemitic output at scale | Jul 2025 | A system-prompt/code change ('don't be politically correct') shipped without safety regression testing caused hours of extremist output on a live platform, followed by a public apology. | Prompts are production code. Every prompt or config change needs change control and a safety eval gate. | 43 |
| Bartz v. Anthropic settlement$1.5B settlement over pirated training data | Sep 2025 | Roughly $3,000 per book across ~500k works downloaded from pirate libraries for training — the largest publicly reported copyright recovery. The court had held training on lawfully acquired books to be fair use; acquiring pirated copies was not. | Training-data provenance is a board-level issue. Prefer providers with IP indemnification and documented data provenance. | 231 |
| Mobley v. WorkdayNationwide age-discrimination collective action over AI screening | May 2025 | A court preliminarily certified a nationwide ADEA collective action alleging AI-based applicant screening discriminated by age — and held that an AI vendor can be liable as an 'agent' of employers. | Any AI touching employment decisions needs pre-deployment bias testing and continuous adverse-impact monitoring — and vendors are not a liability shield. | 31 |
| McDonald's McHire (Paradox.ai)Hiring-chatbot platform exposed data on ~64M applicants | Jul 2025 | A test admin account with password '123456' and an insecure direct object reference in the recruiting chatbot platform exposed tens of millions of applicant records. | AI vendors are part of your attack surface. Third-party AI needs the same security assessment as any SaaS handling your data. | 61 |
| Slack AI research findingInstructions planted in a public channel could exfiltrate private data | Aug 2024 | Researchers showed that indirect prompt injection via a public Slack channel could cause Slack AI to leak private-channel content through crafted link rendering. | Separate trusted context from untrusted content in retrieval, and neutralize rendered URLs — assistants that read everything can be commanded by anything. | 64 |
| Courts worldwide (1,598 decisions)A public tracker of AI-fabricated legal citations passed 1,500 court decisions | 2023–2026 | Sanctions for hallucinated citations escalated from $5,000 (Mata v. Avianca, 2023) to six-figure aggregates, with ~90% of tracked decisions written in 2025 alone. | People treat models as databases unless the workflow forces verification. Human-verification duties belong in acceptable-use policy and in the UI. | 74 |
| KlarnaReversed an AI-only support strategy on quality grounds | May 2025 | After claiming its assistant did the work of ~700 agents, Klarna's CEO publicly conceded 'we focused too much on efficiency and cost — the result was lower quality' and reintroduced human agents with AI as the front line. | Deflection metrics are not a governance framework. Outcome-quality measures and confidence-aware human escalation are controls, not features. | 41 |
Sources for every incident are in the register — see Sources & methodology.