KnowBe4 test: an AI-generated email draft leaked a login code
A lab test shows why an email assistant's model and its surrounding automation must be understood together. The results differed between Gmail's sidebar and a custom workflow.
In brief: An AI email assistant can mistake instructions inside a message for instructions from its user. The consequences depend on both the information the assistant can read and what the surrounding software does with its answer.
When an email becomes an instruction
An assistant asked to summarise or translate an email needs to read the sender’s words. Those words may also contain commands aimed at the model.
OWASP calls this indirect prompt injection: instructions reach the model through external content, rather than through the user’s request. A message can therefore serve two purposes at once: ordinary correspondence for its recipient and an attempt to redirect the assistant reading it.
For example, asking for a summary does not mean authorising the sender to decide what other information the assistant should include.
What KnowBe4’s test showed
In research published on 23 September, KnowBe4 reported different outcomes across two setups. Gemini’s Gmail sidebar exposed unrelated inbox content in its answer but blocked external transmission.
In the researchers’ custom Apps Script workflow, Gemini 3.5 Flash received recent inbox context and produced an HTML draft. Opening that draft triggered an external image request carrying a login code from another email. Gemini 3.6 Flash and 3.7 Flash resisted the tested injection.
This was a laboratory demonstration of that workflow, not evidence that every Gmail account is affected.
Why the software around the model matters
A model generates an answer. An application then decides how to use it: display plain text, create a formatted document, or pass it to another tool. Those are different operations, even when the words came from the same model.
For a business using automated correspondence, that distinction explains why a draft is more than a writing convenience. Its content may be interpreted by the application displaying it. Assessing the model’s answer alone does not describe the whole process.
The same separation between a model and its tools helps explain how AI can carry out actions. Here the issue is an incoming message influencing an assistant’s work, followed by software processing its output.
