How to Assess Generative AI Risk

Begin with a simple system map: user or upstream system, input and system prompts, retrieved sources, model and version, provider, tenant settings, plugins or tools, output, reviewer, destination, and downstream action. Include logs, memory, feedback, and fallback. This exposes risks that disappear when the assessment records only “uses a large language model.”

When the workflow can select tools or execute actions, continue with the AI agent risk assessment because permission, delegation, compounding error, and recovery become central.

Useful scenarios connect a technical or human cause to an operational event and consequence. Relevant themes may include fabricated or omitted facts, biased treatment, disclosure of confidential or personal data, prompt injection, insecure generated code, harmful content, intellectual-property uncertainty, manipulated retrieval, over-reliance, unauthorised tool use, supplier change, service failure, and weak provenance. Select the scenarios that can materially affect the defined workflow.

Evaluation should reflect real users, languages, data, prompts, retrieval sources, and consequences. Begin with normal tasks, then add edge cases, adversarial prompts, conflicting sources, missing context, sensitive inputs, and high-consequence outputs. Define acceptance criteria before reviewing results. A high average score can conceal one failure mode that makes the use unacceptable.

Design the evaluation set as a controlled asset. Record where cases came from, which populations they represent, who reviewed them, how sensitive data was handled, and which version of the workflow was tested. Separate benchmark construction from final evaluation where practical so prompt tuning does not quietly optimise against the test. Preserve failed cases because they are often more useful for monitoring than the aggregate score.

Red teaming complements evaluation by looking for unexpected or adversarial paths; it does not replace representative performance testing. Model cards, system cards, provider evaluations, security tests, and external benchmarks can inform the assessment, but their scope and configuration must match the deployed workflow before management relies on them.

Set thresholds by consequence and segment rather than demanding one universal accuracy number. A drafting assistant may tolerate errors that a benefits, safety, legal, or customer-remedy workflow cannot. Define which failure types block deployment, which require a warning or human check, and which can be monitored after launch. Record the rationale before results are known so delivery pressure does not quietly move the acceptance line.

Reusable safeguards should map to the generative AI, copilot, and agent controls guide, while the risk register preserves scenario, control evidence, residual exposure, and acceptance.

Define fallback and continuity before deployment. If the provider is unavailable, the retrieval source is corrupted, monitoring cannot run, or quality falls below threshold, users need a safe alternative and a clear stop decision. For important processes, test manual fallback, queued work, customer communication, rollback, and recovery rather than assuming the AI service is an optional convenience.

A provider can change a model, safety system, retention setting, subprocessor, feature, or service boundary. Internal teams can change prompts, retrieval sources, plugins, users, integrations, output destinations, or automation. Version and change monitoring should identify which evaluations, risk scenarios, approvals, controls, and user communications must reopen.

Monitor outcomes that correspond to the original scenarios: grounded-answer rate, material error, reviewer override, sensitive-data event, blocked prompt injection, unsupported citation, user escalation, service interruption, and prohibited-use attempts where relevant. Choose a small set that can trigger action. Token counts and adoption may describe usage while saying little about whether residual risk remains acceptable.

Use the third-party AI risk assessment for provider governance, contractual rights, assurance scope, incidents, continuity, and exit. Keep supplier assurance and use-case acceptance linked but separately owned.