Testing for data leaks

A product that serves many districts is multi-tenant, and the worst AI failure for such a product is cross-tenant leakage: a user in one district receiving another district's student data. It happens through shared retrieval indexes without tenant filters, caches keyed only on the prompt, fine-tuned models that memorized one customer's records, and agents with tools that can query beyond the caller's scope. The rule that prevents it is to enforce tenant and role checks in code at every data access, never by asking the model to respect them.

Testing has to try to break that rule. Seed two test tenants with distinct synthetic records containing unique canary strings, then use one tenant's account to try to retrieve the other's canaries, directly, through prompt injection, through uploaded documents and through every tool the assistant can call. Microsoft's AI Red Teaming 101 course and open-source tools such as PyRIT and garak help structure and automate these attempts. Any canary that crosses tenants is a release-blocking bug.

This module's second output is for the other side of the table: an AI tool vetting checklist a district can use on any vendor. About twenty yes or no questions cover FERPA and state law terms, COPPA consent, no advertising or profiling, where data is stored and processed, retention, whether training on student data is excluded, whether injection and leakage testing is done, and whether PII is redacted before model calls.