Skip to content
AI Safety & Security

Module 7 · AI security

How AI features fail

The OWASP Top 10 for LLM Applications, the risk frameworks above it, and a threat model for an AI feature that touches student data.

Schedule
Weeks 7 and 8
Status
Draft outline
Sources checked

This module is a first draft. It sets out the scope so it can be reviewed before it is written in full.

Overview

The OWASP Top 10 for LLM Applications is the shared vocabulary for how LLM features go wrong. The 2025 edition lists prompt injection, sensitive information disclosure, supply chain risks, data and model poisoning, improper output handling, excessive agency, system prompt leakage, vector and embedding weaknesses, misinformation, and unbounded consumption. For a school product, the first two and excessive agency are where most real risk sits: a crafted input that makes the model ignore its instructions, a response that reveals another student's data, or an assistant that can take actions nobody intended it to take.

Prompt injection deserves the most attention because there is no complete fix. Any text the model reads, including a student essay, an uploaded document or a web page, can carry instructions. The defenses are architectural: give the model the least data and the fewest permissions it needs, treat its output as untrusted input to the rest of the system, and keep a human in the loop for anything consequential.

Above the OWASP list sit management frameworks. The NIST AI Risk Management Framework organizes the work into govern, map, measure and manage, and its Generative AI Profile (NIST AI 600-1) applies that to generative systems. Google's Secure AI Framework (SAIF) offers a practitioner's view of the same ground. The output of this module is a threat model: a diagram of where student data flows through an AI feature, with each OWASP risk marked where it applies and a mitigation next to it.

  • The 2025 OWASP Top 10 for LLM Applications, one risk at a time
  • Prompt injection: direct, indirect, and why it cannot be fully filtered
  • NIST AI RMF and the Generative AI Profile, in practical terms
  • Google SAIF as a second lens

Lab

Threat model an AI feature

  1. 1Draw the data flow for one AI feature: inputs, retrieval sources, model provider, outputs, logs.
  2. 2Mark where each OWASP risk could occur on the diagram.
  3. 3Try a few prompt injection exercises on Lakera's Gandalf to see how easily instructions leak.
  4. 4Write one mitigation per marked risk and who owns it.

You finish with: A threat model diagram with risks and owners.

Open questions for the next draft

  • Split into two modules, one for OWASP and one for NIST and SAIF, or keep it as one?
  • Should the threat model use a real Lumina feature as the worked example?