In February 2024, a Canadian tribunal ruled that Air Canada was liable for its own customer service chatbot’s invented bereavement fare policy. The airline argued the chatbot was “a separate legal entity” responsible for its own words. The tribunal rejected that argument outright. No one governed what the model could say, and Air Canada paid for it.
CISA added CVE-2026-64849, a Server-Side Request Forgery (SSRF) vulnerability in MLflow, to its Known Exploited Vulnerabilities catalogue on August 19, 2026, confirmed as actively exploited in the wild. MLflow tracks model experiments and manages model registries, exactly the infrastructure organizations rely on to govern their AI systems. When the governance tooling itself has an unpatched, actively exploited vulnerability, the framework built on top of it inherits that risk.
An AI model governance framework exists to close both gaps: the behavioural gap that lets a chatbot invent policy, and the technical gap that lets attackers exploit the infrastructure governing the model itself.
Talk to Qualysec about validating your AI model governance framework.
What actually counts as an AI model governance framework?
An AI governance framework is the specific set of policies, technical controls, and oversight processes that govern individual AI models across their lifecycle, from development through retirement. It operates at the model level, not the organizational policy level, which is the distinction that determines whether governance catches a problem before or after it becomes a tribunal ruling.
An organization can have a well-written AI ethics policy and still have no idea which models are running in production, what data trained them, or whether their outputs have been tested against adversarial manipulation. A model governance framework closes that gap by treating each model as an asset: named owner, documented risk tier, defined behaviour boundaries, and an active monitoring plan.
What are the core pillars of enterprise AI governance?
Five pillars hold up a working framework: ownership, risk classification, data lineage, technical security validation, and continuous monitoring. Skip any one, and the framework has a blind spot an attacker or a regulator will eventually find.
- Ownership and accountability. Every model needs a named owner. “The AI did it” was not a defense that held up for Air Canada, and it will not hold up for anyone else.
- Risk-based classification. A model summarizing meeting notes and a model approving loans do not belong under the same scrutiny.
- Data lineage and training transparency. Document what trained the model and whether its provenance can be defended.
- Technical security validation. Policy states what a model should do. Testing confirms whether it stays within that boundary under attack.
- Continuous monitoring. A model validated at launch can behave differently within months as inputs and integrations shift.
How do you build an AI model governance framework, step by step?
Building a framework follows four steps in sequence: find every model running in the enterprise, classify each one by risk, write policy mapped to that risk, then monitor continuously after deployment. Skipping the order, especially trying to write policy before completing discovery, produces rules that miss the systems that matter most.
How do you find shadow AI hiding in your enterprise?
Start with a complete inventory covering approved platforms, custom-built models, and unsanctioned tools employees adopted without approval. Shadow AI is usually the largest category, and it is the one most CTOs underestimate until they run a real discovery pass.
Recent studies report that a majority of enterprises now see unsanctioned AI use, with some finding around 80% of workers using unapproved tools and many security leaders unsure how much more exists undetected. Asset discovery is not a one-time inventory. New tools appear faster than an annual review catches them, which is why continuous discovery, not a periodic audit, is what mature programs actually run.
How do you classify AI model risk correctly?
Classify every model as low, medium, high, or unacceptable based on what it does and who it affects, before it reaches production, not after an incident forces the question. A model generating internal marketing copy carries a different risk profile than one influencing hiring decisions or customer-facing financial guidance, the exact category Air Canada’s chatbot fell into.
This step should also flag which models fall under specific regulatory categories: EU AI Act risk tiers, or sector rules like the Federal Reserve’s SR-11-7 for financial models. Classification that does not map to a real regulatory category is an internal exercise, not compliance evidence.
How do you turn risk tiers into enforceable AI policy?
Write policy that maps directly to each risk tier: what a model can say or do, what data it can access, and what human review is required before a high-risk output reaches a customer. Policy written without reference to risk tiers tends to be either too vague to enforce or too rigid to survive contact with how engineering teams actually ship.
Compliance mapping matters here too. Tie each policy explicitly to the regulatory requirement it satisfies, so the framework produces audit evidence instead of a general assurance that governance exists.
How do you monitor AI models after deployment?
Track behavioral drift, log what a model actually did in production, and flag anomalous patterns that might indicate manipulation or failure, on a recurring schedule, not an annual one. Deployment is not the finish line. The MLflow vulnerability above is the clearest possible illustration: the tooling meant to track and govern a model’s lifecycle became an attack vector precisely because monitoring assumed the infrastructure itself was safe.
Vulnerable pattern, seen in unpatched model registry and tracking servers:
CVE-2026-64849 is a critical, unauthenticated SSRF in MLflow’s webhook test endpoint (POST /api/2.0/mlflow/webhooks/{id}/test). In versions before 3.15.0, MLflow validates the configured webhook URL once but then follows redirects and re-resolves the hostname without pinning the validated address, allowing attackers to reach internal services or cloud metadata endpoints and exfiltrate response bodies. CISA added this CVE to its Known Exploited Vulnerabilities catalog on August 19, 2026, confirming active exploitation.
An attacker can supply a URL that redirects to an internal or cloud metadata endpoint. Because MLflow follows redirects and re-resolves the hostname without pinning the originally validated address, the webhook test endpoint becomes an SSRF vector, exactly the class of flaw behind CVE-2026-64849.
Fixed pattern:
Restricting webhook URLs to an explicit allowlist and either disabling redirects or re-validating the hostname after each redirect closes the specific SSRF vector. Patch management for governance infrastructure itself, not just the models it tracks, has to be part of Step 4.
Which is better: NIST AI RMF or ISO 42001?
NIST AI RMF gives you a voluntary internal structure for model risk; ISO 42001 gives you a certifiable standard an outside party independently verifies. Most mature programs use both: NIST to organize internal thinking, ISO 42001 once that program is ready for third-party proof.
| Dimension | NIST AI RMF 1.0 | ISO/IEC 42001 |
| Status | Voluntary, no certification | Certifiable via accredited third-party audit |
| Structure | Four functions: Govern, Map, Measure, Manage | Clauses 4 to 10, 38 Annex A controls |
| Origin | US, published January 2023 | International, published December 2023 |
| Evidence produced | Internal risk documentation | Independently verified certificate |
| Best fit | Building an internal AI risk program | Needing provable, external validation |
Why does AI model governance require penetration testing?
Because prompt injection, ranked LLM01:2025 in OWASP’s Top 10 for LLM Applications for the third edition running, is not something a written policy can test for. A policy stating a model will not leak sensitive data is an assertion until someone actually tries to make it leak sensitive data.
NIST’s Generative AI Profile, NIST AI 600-1, names adversarial inputs as a primary risk category precisely because standard governance documentation does not catch them. A model governance framework that has never been tested against real attack techniques is running on assumed security, not proven security, and that distinction is exactly what separates a framework that looks solid on paper from one that has been verified.
How does Qualysec validate AI model security?
Qualysec manually attempts the attack classes named in OWASP LLM01 through LLM04, direct and indirect prompt injection, training data extraction, and API attacks against the infrastructure surrounding a model, then documents exactly which guardrails held. This produces evidence a Technical Team Head can hand directly to a reproduction ticket, not a generic finding that needs translation before anyone can act on it.
We test model registries and tracking infrastructure the same way we test the model itself, since CVE-2026-64849 shows the governance tooling is as much a target as the model it governs. Findings come with severity ratings, reproduction steps, and remediation guidance mapped to NIST AI RMF and ISO 42001 documentation requirements.
Schedule an AI model security assessment with Qualysec.
What mistakes do enterprises make building AI governance frameworks?
The most common mistake is treating the policy as the finished product, exactly the gap that turned into a tribunal ruling for Air Canada. Three other patterns show up just as often.
Governing the model while ignoring the infrastructure around it leaves the easier attack path unexamined, since APIs and model registries are often less scrutinized than the model getting all the attention. Skipping risk classification and applying one policy to everything either over-restricts low-risk tools until teams route around them, or under-protects the systems that actually need scrutiny. Treating shadow AI as a training problem instead of a visibility problem means a memo goes out and the underlying behavior does not change, since the reason people reach for unapproved tools, speed, does not disappear because a policy says not to.
What should be on your AI governance implementation checklist?
- Complete, continuously updated inventory of every AI model, including shadow AI
- Named owner assigned to each model, not a committee
- Risk classification completed and mapped to applicable regulatory categories
- Documented data lineage for all training and fine-tuning data
- Policies written specifically against each risk tier
- Adversarial penetration testing completed before high-risk models reach production
- Model registry and tracking infrastructure patched and access-controlled, not just the models themselves
- Monitoring in place to catch behavioral drift after deployment
- Audit cycle scheduled quarterly at minimum
Conclusion
Ai model governance framework is what stands between an organization and its own version of the Air Canada ruling or the MLflow SSRF exposure, a model saying something nobody approved, or governance infrastructure itself becoming the entry point. The four-step process, discovery, classification, policy, monitoring, only works with technical validation behind it, confirming the guardrails hold and the infrastructure governing them is patched.
Contact Qualysec to test whether your AI model governance framework holds up under real pressure.
FAQ
1. What is an AI model governance framework?
A set of policies, technical controls, and oversight processes governing individual AI models across their lifecycle. It gives each model a named owner, documented risk classification, and monitoring plan, rather than relying on general organizational AI policy alone.
2. What is the difference between the NIST AI RMF and ISO 42001?
NIST AI RMF is a voluntary framework organized around four functions used to structure an internal AI risk program. ISO 42001 is a certifiable international standard audited by an accredited third party. NIST helps build the program; ISO 42001 proves to an outside party it works.
3. How do you identify and control Shadow AI in an enterprise?
Identify it through continuous discovery across network, browser, and endpoint layers, not an annual audit, since unsanctioned tools appear faster than periodic reviews catch them. Control it through governed alternatives that meet the same need employees are solving with unapproved tools, since outright bans alone rarely stop the underlying behavior.
4. Why is penetration testing required for AI model governance?
Penetration testing provides the technical evidence that a model’s guardrails actually work. A policy claiming a model will not leak sensitive data is an assertion until someone attempts exactly that under controlled conditions and confirms whether it holds, which is what OWASP‘s LLM01 through LLM04 attack classes specifically test for.
5. What is the first step to building an AI governance framework?
Asset discovery: a complete, continuously updated inventory of every AI model, including shadow AI running without formal approval. Every later step, risk classification, policy, monitoring, depends on knowing what actually needs governing.
