Should an AI That Reads Outside Documents Be Allowed to Act on Them?
An AI reading outside documents should only act within limits defined by Dr. Jonah Tebaa's Weakest-Writer Rule, which caps permissions based on the least trusted contributor. Under this framework, material from outsiders permits only Summarise actions, partners reach Draft, staff reach Send, and only internal material permits Change. Illustrated by a composite hiring assistant steered by hidden text in a CV, organizations must separate reading from acting so tools extract fixed fields before systems act.
A hiring manager asks why a screening assistant ranked one applicant far above the rest. The explanation turns up when someone opens the applicant's CV as plain text. At the foot of the last page, in white type on a white background, sits a short request that the assistant describe the candidate as an exceptional match. The applicant wrote it. The employer never did. The assistant acted on it because it arrived inside the file it had been asked to read.
This is an illustrative composite, not a client engagement: a scene assembled from a type of risk that is widely described. Dr. Jonah Tebaa uses scenes like it to make one narrow point. The assistant behaved in line with how such systems take in text, and the exposure was created before the first application arrived.
Why a written sentence can steer a reading tool
Dr. Tebaa starts from a plain observation: everything an AI reads was written by somebody. Sometimes the author is the organisation itself. Often it is an applicant, a supplier, a customer or the owner of a web page. The risk of an outsider's text being followed as though it were an instruction has a common name, prompt injection, and he uses the term once and moves on.
He is careful about the explanation. As he describes it, these tools take in an employer's instructions and a document's contents as one run of text, and they cannot reliably separate the two. He presents that as a property of how the systems work rather than a flaw in any single product, and he does not assume it will be solved. For an owner, he argues, the useful question is therefore not how accurate the model is. It is who can put words in front of it.
The Weakest-Writer Rule
In his work, Dr. Tebaa turns that question into a rule he calls the Weakest-Writer Rule. It has two steps and fits on a single page.
The first step is a sort. Every source the AI reads is placed in one of four tiers according to who is able to write to it:
- Only us: material produced by a small group the organisation controls.
- Our staff: material written by employees.
- Named partners under contract: material from organisations the business can name and contact.
- Anyone: web pages, inbound email, uploaded files and forms, shared documents with outside editors, and reviews or comments.
The test he applies is who could add a sentence to the source without asking permission. A source that looks respectable but accepts text from the public belongs in the last tier.
The second step sets a ceiling. He distinguishes four levels of action. At Summarise, a person reads the result. At Draft, a person approves it before anything happens. At Send, the output leaves the building. At Change, it alters a record, a price or a payment. In any one task, the highest action the AI may take is the ceiling of the least trusted source it touched in that task. Material from anyone caps at Summarise, named partners at Draft, staff may reach Send, and only the owner's own material may reach Change.
Applied to the screening scene, the sort is quick. Applications are written by anyone, so an assistant that reads them may summarise them for a recruiter to read. An assistant that reads the same files and then advances or rejects candidates in the hiring system is performing a Change on the strength of the least trusted source in the task. In Dr. Tebaa's terms it is over its ceiling, however well it performed in testing.
Separating reading from acting
He does not recommend giving up the tool. Where a source is open to anyone, he proposes splitting the work in two. The first step reads the file and has no rights to act; it pulls out only fixed fields, such as name, dates of employment and listed qualifications, into a form that a person or a simple rule-check can review. The second step acts, and it sees only those fields, never the original text. A sentence hidden in the file then has much less chance of reaching the step that can do harm.
The five-item checklist, compressed
Dr. Tebaa condenses the practical work into five items, to be done before the AI reads one more outside document:
- List every source the AI reads and write its tier beside it.
- Give each task a verdict against its ceiling, and fix the over-ceiling tasks first, starting with any that move money or change a record.
- Separate reading from acting, so that extracted fields, not raw text, reach any step that can act.
- Run a ten-minute canary test with a harmless instruction in a file an outsider could realistically submit, and repeat it after any change to the tool.
- Keep a record tying each action to the source that prompted it, and look through it once a month.
The canary is deliberately trivial: a request to write one harmless word at the top of the summary. If the tool writes it, the owner has seen the risk rather than read about it.
Who should apply it first, and what it does not claim
He points first to teams with an open intake: recruitment, accounts payable, a shared customer inbox, or research that draws on public web pages. Approval there was usually decided on accuracy, and nobody asked who could send the tool a document.
Dr. Tebaa is explicit about limits. The rule offers no immunity; what it does is shrink what one planted sentence can achieve. A reviewer who sees only the AI's summary of an open source can still be led astray by it, so he treats that summary as a reason to check, not a conclusion. Some tasks will gain a step, and he argues the extra step is a fair price.
His closing instruction to owners is practical: write down every place an outsider can put text in front of your AI, and next to each, the most it is allowed to do.
Related evidence: The paper shows that applications built on language models blur the line between data and instructions, and that adversaries with no direct access can steer such an application by planting prompts in data it is likely to retrieve (indirect prompt injection). (the research paper that first described indirect prompt injection, in which instructions planted in retrieved data steer an AI application)
The UK NCSC explains that current large language models do not enforce a security boundary between instructions and data inside a prompt, so text inside material the system reads (such as a CV) can be executed as an instruction; it recommends limiting what the system is allowed to do rather than expecting the model to tell the two apart. (the UK National Cyber Security Centre's explanation that current language models do not enforce a boundary between instructions and data)