Local AI agents for business: what they are and what they can do on your own server

The jobs a business wants an AI agent for are everyday ones: sort the inbox, search the document store, open a ticket. A local AI agent does them on a server you control, so your data never reaches a model provider.

What an AI agent is and how it works

An AI agent is a language model that uses tools in a loop. In a workflow, your code fixes which step comes next. An agent decides its next step and its tools itself, and for that it has four parts:

1. The model picks the next step.
2. The tools look something up or change it: a database query, a file, an email.
3. The loop runs the tools and passes the results back.
4. The limits: maximum steps, allowed tools, permissions, human approval.

AI agent or workflow: examples for your business

The simplest solution that works is usually the right one, because every extra step costs GPU time and waiting time. Our rules of thumb:

Running an AI agent locally: what fits on your own server

All four parts fit on one machine of your own:

• The model. Open-weight models such as Qwen3 and gpt-oss come with tool calling built in.
• The model server. vLLM and Ollama pass tool calls through their APIs (Ollama vs vLLM).
• The tools. Database, file server, ticket system and document search are internal anyway. The Model Context Protocol (MCP) connects them all the same way.
• The loop is ordinary code that checks every call before it runs it.

How to keep your data in-house

An agent is as local as its tools. The model, the loop, the log and your internal tools run on your server. Search across your own documents stays in-house too when it uses a local embedding model, the small model that makes text searchable.

Tools that point outwards, such as web search or email, send what their job needs: the search query, the message. Two rules keep that tight:

1. The server connects only to a fixed list of addresses, so a manipulated agent can’t send data anywhere else.
2. Outward tools take narrow inputs: a search tool that accepts only a short search term can’t carry a document out.

AI agent risks and the controls against them

Two risks matter most, and both are on the OWASP Top 10 list of risks for LLM applications. Prompt injection: instructions hidden in an email, a web page or a file the agent reads. Excessive agency: more tools, rights or autonomy than the job needs. In the OWASP example, an email summariser’s mail tool can also send, and a crafted email gets it to forward sensitive mail to an attacker. A read-only tool would have stopped it.

These controls belong in any agent, most of them in steps 03, 04 and 06 of the loop above:

1. Only the tools it needs, with the rights of the person it works for.
2. Permission checks in the target system, not in the model. A model saying no is no security control.
3. Human approval before anything irreversible: sending, deleting, paying, publishing.
4. A step limit and rate limits on tools.
5. A sandbox first, because errors can add up over many steps.
6. A lean log: which tool ran when, whether it worked, who approved it. Full content only with a clear purpose and a deletion period.

AI agents and GDPR

On your own server, prompts, documents and answers are processed with no model provider in between, which takes care of the biggest data protection question. We sign a data processing agreement with every client. Three rules apply wherever the model runs: decisions with legal effects, such as hiring, are made by people, with the agent preparing them. People should know when they are dealing with an AI. And many AI projects need a data protection impact assessment, so bring in your data protection officer early. More in Self-hosted LLM and GDPR.

What hardware a local AI agent needs

Mainly GPU memory (VRAM). An agent resends its whole history at every step, tool results included, so it needs more context than a chat, and context takes memory.

A 20 GB GPU server runs fast mid-size models. For agents that read long documents, the 96 GB server gives more room: it runs Qwen3.6 35B-A3B, which our model catalogue lists for agents, tools and long context, and holds gpt-oss-120b completely. Details in Which LLMs fit 20, 24 or 96 GB of VRAM and the VRAM calculator.

Want an agent that works with your data on a server of your own? That is what we build with Private AI: your model on a dedicated GPU server in Germany, an API for your tools, the automations and agents on top, and a data processing agreement. The first call is free.