The agent does not need your whole laptop to summarize one document.
That sounds obvious until productivity tooling gets involved. A local assistant wants to search files, summarize meeting notes, draft replies, inspect screenshots, read browser context, pull from document folders, and coordinate across apps. The operating system permission prompt asks for broad access because narrow access is harder to design. The user clicks allow because the workflow is blocked. The product team calls it convenience.
Security and privacy teams should call it what it is: a data access decision.
Not all local AI access is bad. Some of it will be useful. The tradeoff is not AI versus no AI. The tradeoff is whether the organization can explain what the agent was allowed to read, why it needed that access, how long the permission lasted, and what evidence exists after the fact.
Full access is a lazy product boundary
Broad file permissions are seductive because they reduce friction. The assistant works in more places. Fewer prompts interrupt the user. Support tickets go down. The demo looks better.
But broad access also collapses purpose boundaries.
A folder can contain customer exports, legal drafts, payroll fragments, incident notes, screenshots, downloaded attachments, API keys in forgotten text files, board materials, medical documents, source code, and personal files. The fact that all of those objects live on one machine does not mean they belong to one risk category.
That is the mistake. Endpoint access is often treated as a device trust question: is this managed, encrypted, patched, and enrolled? Useful questions, but incomplete. AI agents add a second question: which parts of the device can be used as model context, tool input, retrieved evidence, or automation fuel?
Device trust does not answer that.
Give the agent a budget, not the building master key
A file access budget is a simple operating idea: define the maximum data the agent may touch for a purpose before anyone argues about prompts, vendors, or model behavior.
The budget should answer practical questions.
Which locations can the agent read by default? Which file types are excluded? Can it inspect hidden folders, downloads, screenshots, browser storage, synced drives, chat exports, or local mail archives? Can it follow links into cloud storage? Can it read files the user can access but the current task does not need? Can it retain snippets? Can it send content to a remote model? Can it create embeddings from local material? Can it keep an index after permission is revoked?
That list is not bureaucracy. It is the difference between scoped assistance and quiet collection.
A useful budget is purpose based. Drafting a reply from a selected email is not the same as indexing the user’s entire mailbox. Summarizing a selected contract is not the same as reading every legal folder on the laptop. Answering from a project workspace is not the same as searching the whole home directory because it might be helpful.
The product may need escalation paths. Fine. But escalation should be visible, time bound, and evidenced. “Allow for this folder until the task completes” is a different control than “grant full disk access forever.”
Local context still needs retrieval permissions
A lot of teams already understand that enterprise AI search needs source aligned permissions. If the user cannot access a document, the assistant should not retrieve it. That principle still applies locally, but it is not enough.
The harder issue is purpose alignment. A user may technically have access to a file and still not have a good reason to feed it into an agent for the current task. Local AI assistants blur that line because the user and the agent appear to share a desktop. They do not share judgment, accountability, or retention obligations.
The same lesson shows up in enterprise retrieval systems: retrieval permissions are the missing control in enterprise AI search. Permissions need to travel with the source, but the agent also needs limits on what it is allowed to ask the source to provide.
Otherwise, “the user could open it” becomes a blank check for the agent to process it.
Evidence should show access, not vibes
If an agent reads local files, the organization needs evidence that can survive more than a product demo.
Useful evidence is not a giant transcript of every prompt and file body. That can create its own privacy problem. The evidence should show which permission was granted, by whom, for what purpose, what sources were in scope, what categories were excluded, whether content left the device, what retention rule applied, and when access ended.
This is where AI security and privacy engineering need to work together. Security wants enough evidence to investigate abuse, misconfiguration, or unexpected behavior. Privacy wants to avoid turning every assistant interaction into a retained copy of sensitive work. Both are right.
The answer is not “log everything” or “log nothing.” It is a designed evidence layer. The same logic applies to runtime observability: AI runtime telemetry needs a privacy boundary.
Tool access and file access are different risks
Agent governance often focuses on actions: can the agent send email, create tickets, approve refunds, update records, run code, or call APIs? That matters. We have written before that AI agent tool access needs a transaction boundary.
File access is quieter. The agent may not change anything. It may only read. That still creates risk.
Reading can expose regulated data. Reading can move confidential context into a model call. Reading can populate an index. Reading can create summaries that lose original handling rules. Reading can combine unrelated fragments into a new sensitive output. Reading can give a later tool action bad context.
Treating read access as harmless is how organizations end up surprised by their own convenience features.
What to decide before rollout
Before approving local AI agents with broad file capabilities, force a few decisions into the open.
Name the owner of the file access policy. Define default denied locations. Separate selected file access from folder access and full device access. Require a reason for expanded scope. Decide whether indexing is allowed, where it lives, and how it is deleted. Make remote processing visible. Preserve evidence about permissions without retaining unnecessary content. Give users and admins a way to revoke access that actually stops future reads.
Also decide who can accept the residual risk. Not every team should be able to grant an assistant access to sensitive local material just because the prompt box is convenient.
If your organization is trying to turn AI governance into operating controls rather than policy theater, Zero Drama Security services can help structure the access, evidence, and privacy decisions without slowing every experiment to a crawl.
The goal is not to make agents useless. It is to stop pretending a helpful assistant should inherit every file boundary the operating system was too blunt to express.
Give the agent enough context to do the job. Not enough to become the unofficial archive of everything the user has ever touched.