AI governance glossary

AI red-teaming

Also: adversarial testing, LLM red team

Structured adversarial testing of an AI system by security researchers or automated tooling attempting to make it behave in unintended or harmful ways - for example, bypassing safety guardrails through prompt injection, or extracting data the system should not reveal. Distinct from routine QA testing because it explicitly adopts an attacker's mindset rather than testing for expected behavior.

Algorithmic bias audit

Also: AI fairness audit

A focused assessment testing whether an AI system produces systematically different outcomes for different demographic groups in ways that are not justified by legitimate factors. A narrower discipline than a full AI governance audit - typically one component of a broader governance or compliance review, but sometimes commissioned as a standalone engagement, especially in hiring, lending, and insurance use cases with direct fair-treatment exposure.

Related: model validation

Guardrails

Also: AI guardrails

Runtime controls placed around an LLM or AI agent to constrain its inputs and outputs - filtering disallowed content, blocking known attack patterns, or restricting what actions an agent can take. A common product feature among LLM security vendors, and something an RFP for LLM security testing should ask a vendor to test the real-world effectiveness of, not just confirm exists.

Hallucination

An output from a generative AI model that is fabricated, factually incorrect, or unsupported by its inputs, but presented with the same fluency and confidence as an accurate response. A core risk category in generative-AI-specific model validation, distinct from bias or security vulnerabilities, since a hallucinating model can be perfectly "secure" and still unreliable.

Related: model validation

Jailbreaking

A category of prompt-injection attack that specifically attempts to bypass an LLM's built-in safety guardrails, tricking it into producing content or taking actions its provider explicitly designed it to refuse. A standard test category in LLM red-team engagements, distinct from prompt injection aimed purely at data exfiltration or instruction override.

OWASP LLM Top 10

Also: OWASP Top 10 for LLM Applications

A ranked list of the most critical security vulnerabilities specific to large language model applications, published and maintained by the Open Worldwide Application Security Project (OWASP). Widely used by LLM security testing vendors as a common taxonomy so that findings and severity are comparable across different vendors and engagements.

Prompt injection

Also: jailbreaking (related)

An attack technique where crafted input text manipulates an LLM into ignoring its original instructions or safety constraints, potentially causing it to reveal confidential system prompts, execute unintended actions, or produce harmful output. One of the most common findings in LLM red-team engagements and the top entry on the OWASP LLM Top 10.

Frameworks & Standards 11

AI literacy

A requirement under Article 4 of the EU AI Act that providers and deployers of AI systems ensure their staff have a sufficient level of understanding of AI to use it responsibly - one of the first EU AI Act obligations to take effect (February 2025), ahead of the Act's later, more technical compliance deadlines.

AI management system (AIMS)

Also: AIMS

The formal governance structure ISO/IEC 42001 requires an organization to establish, implement, and continually improve, following the same Plan-Do-Check-Act cycle used in other ISO management-system standards like ISO 9001 and ISO/IEC 27001. An AIMS is what gets audited and certified when an organization pursues ISO 42001 certification.

Related: Statement of Applicability

See: ai compliance consultants, ai governance audit

Conformity assessment

The formal process under Article 43 of the EU AI Act by which a high-risk AI system is checked against the Act's requirements before it can be placed on the EU market, either through internal control or, for certain systems, an accredited third-party notified body. Distinct from ISO/IEC 42001 certification, which is a voluntary management-system standard rather than a legal market-entry requirement.

Related: high-risk AI system

European AI Office

Also: EU AI Office

The European Commission body established to oversee EU AI Act enforcement, particularly for general-purpose AI models, and to support consistent application of the regulation across member states. Works alongside national market surveillance authorities that handle enforcement for other AI systems under the Act.

General-purpose AI (GPAI) model

Also: GPAI

A category defined in the EU AI Act for AI models capable of performing a wide range of tasks (most large foundation/language models qualify). Providers of GPAI models carry a distinct chapter of obligations under the Act, with additional requirements for GPAI models classified as posing 'systemic risk.'

Related: model risk tiering

High-risk AI system

A category defined in Article 6 and Annex III of the EU AI Act covering AI systems used in specific contexts (employment, education, credit, law enforcement, and others) that must meet requirements for risk management, data governance, technical documentation, human oversight, and conformity assessment before entering the EU market. Not every AI system is high-risk under the Act - most fall into limited or minimal risk tiers instead.

Human oversight

A requirement under Article 14 of the EU AI Act that high-risk AI systems be designed so a human can effectively monitor, intervene in, and if necessary stop the system's operation. Also a common control area in internal AI governance programs generally, independent of any specific legal requirement.

Related: high-risk AI system

Post-market monitoring

An ongoing obligation under Article 72 of the EU AI Act for providers of high-risk AI systems to actively collect and review data on the system's real-world performance after deployment, and report serious incidents. The regulatory analog to the general governance practice of continuous model monitoring for drift.

Statement of Applicability

Also: SoA

A required ISO/IEC 42001 document in which an organization records which Annex A controls apply to its AI management system, selected in proportion to its identified AI risks, and justifies any exclusions. Accredited certification auditors review the Statement of Applicability directly as part of a certification audit.

Related: AI management system (AIMS)

Systemic risk (GPAI)

A classification under the EU AI Act for general-purpose AI models whose scale or capability creates risk with the potential for significant, widespread impact - triggering additional obligations beyond the Act's baseline GPAI provider requirements, such as model evaluation, adversarial testing, and incident reporting to the European AI Office.

Technical documentation

The specific documentation set required under Annex IV of the EU AI Act for high-risk AI systems - covering the system's design, development process, capabilities, limitations, and risk-management measures - that must exist before the system enters the EU market and be kept current afterward. A heavier, more prescriptive requirement than a general model card.

Governance Fundamentals 13

AI agent governance

Also: agentic AI governance

Governance controls specific to autonomous or tool-using AI agents, which can take actions (API calls, data access, transactions) rather than just generating text. Agent governance typically adds permissioning and action-scoping controls on top of standard model governance, since an agent's ability to act independently introduces risk that a purely conversational model does not carry.

Related: AI inventory, AI red-teaming

See: ai compliance consultants, llm security testing

AI inventory

Also: AI system inventory, model inventory

A documented, maintained list of every AI/ML system an organization runs or is developing, including each system's purpose, owner, and risk tier. An AI inventory is the foundational input almost every governance framework requires before any other control can be applied - you cannot govern, audit, or assess risk for a system you haven't documented exists.

Related: model risk tiering, shadow AI

See: ai governance consultants, ai governance audit

Chief AI Officer (CAIO)

Also: CAIO

An emerging executive role with accountability for an organization's AI strategy and governance program. Several large consultancies have created adjacent named roles for their own AI governance practices (for example, Accenture named its first Chief Responsible AI Officer in 2024) - a real, if still-forming, signal that AI governance accountability is consolidating into a named leadership position rather than staying diffuse across existing functions.

Related: three lines model

Explainability (XAI)

Also: explainable AI, XAI

The degree to which an AI system's decision-making process can be understood and articulated in terms a human can evaluate. One of NIST AI RMF's seven characteristics of trustworthy AI ('explainable and interpretable') and a common audit finding area, since a system that cannot be explained is difficult to validate or challenge.

Related: model validation

Governance, Risk, and Compliance (GRC)

Also: GRC

An umbrella term for the combined discipline of setting policy (governance), identifying and managing exposure (risk), and meeting external obligations (compliance). Several AI governance platforms (OneTrust, for example) build their AI governance module on top of a broader existing GRC platform rather than as a standalone product.

Related: model risk tiering

Model card

A short, standardized document accompanying a trained model that records its intended use, training data characteristics, known limitations, and evaluation results. A common artifact in an AI inventory, and something an AI governance audit will typically sample to check documentation completeness.

Related: AI inventory, model validation

Model drift

Also: data drift, concept drift

The gradual degradation of an AI model's real-world accuracy or fairness over time as the data it encounters in production diverges from the data it was trained or validated on. Continuous monitoring for drift is a core capability of most AI governance platforms and a standard control point in AI governance audits.

Related: AI inventory, model validation

Model risk tiering

Also: AI risk classification, use-case risk tiering

The practice of classifying each AI system by potential impact (e.g. low, medium, high risk) so that governance controls can be applied proportionally rather than uniformly. The EU AI Act formalizes this into legal risk tiers (unacceptable, high, limited, minimal); most enterprise AI governance programs adopt a similar tiered approach even where no specific law requires it.

Related: AI inventory

Model validation

The documented process of testing an AI model for accuracy, fairness, and robustness before deployment, and re-testing on a defined cadence afterward. Model validation records are a common evidence artifact an AI governance auditor will sample directly, since a written validation policy without corresponding validation records is a frequent audit finding.

Related: model drift

Responsible AI

Also: RAI, trustworthy AI

An umbrella term for the practices and principles an organization applies to develop and deploy AI ethically and safely - typically covering fairness, transparency, accountability, privacy, and safety. Several major consultancies (Deloitte's Trustworthy AI, PwC's Responsible AI Toolkit) have published named proprietary frameworks under this banner, generally mapped to external standards like the NIST AI RMF.

Shadow AI

Also: unsanctioned AI, unauthorized AI use

AI tools or models adopted by teams without going through an organization's formal intake, review, or governance process - for example, an employee using an unapproved consumer LLM tool on internal data. Shadow AI is a common finding in AI governance audits and a primary reason AI-discovery capability is a core feature of most AI governance platforms.

Related: AI inventory

Third-party risk management (TPRM)

Also: TPRM, vendor risk management

The discipline of assessing and monitoring risk introduced by external vendors and suppliers, extended in AI governance to cover the specific case of third-party or vendor-supplied AI models and platforms - which carry their own inventory, validation, and monitoring requirements distinct from internally-built models.

See: ai governance consultants, ai compliance consultants

Three Lines Model

Also: Three Lines of Defense (prior name)

An IIA governance model assigning distinct risk-management roles: first line (teams that build and operate a system and own its risk directly), second line (risk/compliance functions providing expertise, monitoring, and challenge), and third line (independent internal audit providing objective assurance to the governing body). Applied to AI governance to delineate who owns AI risk, who challenges it, and who independently verifies it's being managed.