A safety check students can’t talk their way around

Problem

A chatbot follows its rules until a student pushes

Schools want AI helpers for open-ended work. The usual safeguard is to write the rules into the chatbot’s prompt. That puts the rules inside the same conversation the student is typing into, so a determined 12-year-old can probe them, argue with them, or talk around them, and nobody can easily audit what happened.

VETTING moves the rules out of the conversation. A chat model drafts each reply without seeing the policy. A separate verifier, which the student never talks to, checks the draft against it. Failed drafts are rewritten with feedback up to a fixed limit, then fall back to a safe reply.

Step through the example trace from the lab’s API documentation to see one reply get caught and rewritten:

Round 1

Round 1: students didn’t ask the way we expected

In Fall 2024 we tested the first version in an undergraduate AI course (published at AIED 2025). Students practiced midterm questions with a chatbot and were randomly assigned to the verified version or a general one. I built the study app end to end: consent, random assignment, and telemetry for every hint, answer reveal, and chat turn.

The outcome comparison was null: 41 students were randomized, attrition was heavy, and no differences were significant. The interaction logs were more useful. Across 218 messages, only 3 students ever triggered a direct-answer revision. Students rarely asked “tell me the answer.” They pasted the question in, or broke it into small pieces. The verifier was tuned for the question we imagined, not the one students asked.

The next version reflects both lessons. Policies became explicit criteria about what a reply may contain, instead of guesses about what a student might ask. And they moved behind a server, because a prompt the browser can read is a prompt a student can read.

Round 2

Round 2: 151 middle-school students, one 45-minute writing task

190 students in grades 6–8 wrote a short essay about AI in their own school, with the chatbot available and no limits on what they could ask. 151 used it. Every draft, verdict, rewrite, and timing was logged.

Every flagged turn was audited by hand, along with a random sample of 250 passing turns to estimate what the verifier missed. Each failure was then coded into an eight-category taxonomy of boundary-testing behavior.

Findings

What we learned about students, and about the system

• A few students test limits, repeatedlyOnly 4.1% of turns triggered verification, but they came from 16.6% of conversations. Within those conversations, nearly a quarter of turns were flagged.
• Boundary-testing is social, not academicRomantic and mature themes made up more than half of failures. Prompts with risky keywords had 4.6 times the odds of failing verification.
• The prompt alone would have leakedReplaying the 51 flagged turns with the policy written into a single model’s prompt, 18 of them (35.3%) still produced violations students would have seen.
• Safety costs time, within a boundMedian reply time rose from 2.9 s to 6.6 s on turns that needed a rewrite. The rewrite cap keeps the worst case predictable.

Impact

From one study app to shared infrastructure

The framework is published in Computers & Education: Artificial Intelligence (2026), released as an open-source Python package (vetting-python), and runs as verified endpoints in the lab’s AI gateway, so other products can turn it on without rebuilding it. The eight-category taxonomy gives schools and developers a shared vocabulary for what students actually test.