Agent, Local LLM, MCP Server: Who Decides, Who Executes?

The need: internal reports instantly, without the data leaving

Most of the reports people ask for every day in a fintech follow the same pattern: “how many dollars were deposited today?”, “how many withdrawals are waiting?”, “how many accounts are long on EURUSD, how many are short?”, “how many lots were traded on gold today?”, “how many KYC applications are pending?” The answer sits in a system; the question arrives in natural language. In between there is a person: someone asks for the report, someone writes the SQL, someone sends a screenshot. A cloud model closes that gap nicely; but sending customer balances and position data to the cloud means, in a fintech, a meeting, a contract and a legal opinion — and usually a “no”.

The internal solution had three conditions:

• Data never leaves the company. The model and the tools both live on a server inside the company; not a single HTTP call goes outside.
• The model can call our own tools. Deposits, withdrawals, position split, volume, balance, KYC. Adding a new report should be as easy as writing a Python function.
• A single GPU has to be enough. The server has one card with 8 GB of VRAM. That means models with 4–8 billion parameters, compressed to 4 bits (q4); no 70-billion-parameter reasoning.

The third condition shaped the design. A small LLM cannot make many decisions with little guidance. So the decision load has to be taken away from the model and put somewhere else. That place is the agent.

Architecture: three parts, three responsibilities

The one decision made on day one: the agent manages, not the model. The model only answers the question “is a tool needed, and which one?” It does not run the tool, does not collect the result, does not decide when to stop. All of that is the agent’s job.

You --> Client (chat UI) --HTTP--> AGENT (FastAPI, OpenAI-compatible API)
                                     |
                                     +--> Local LLM (qwen3:4b) --> plain text  or  tool_calls
                                     |
                                     +--> MCP Server (our tools) --> result --> back to the LLM --> final answer

This separation paid off in two places. First: the client can change. Because the agent exposes an OpenAI-compatible API, an internal chat screen, Open WebUI or curl can connect; the agent and the tools stay the same. Second: when something breaks, the question “which box?” has four answers, not forty. Wrong tool selected: the docstring. Tool returned an error: the MCP Server. Invented answer: the text the agent gave to the model.

Stack

• Internal server, single GPU (8 GB VRAM)
• Python 3.12 (uv), mcp SDK 2.2.0, FastAPI + uvicorn, httpx
• Ollama 0.34 (local LLM runtime), model qwen3:4b

MCP Server: the hands

A small server built with the mcp SDK, talking over stdio transport. The tools are ordinary Python functions; the SDK takes the docstring and the type signature and turns them into the schema that goes to the model. Adding a new capability = writing one function and registering it in server.py. Nothing changes on the agent side.

@mcp.tool()
def depozit_ozeti(tarih: str = "today") -> str:
    """Daily deposit summary: total amount (USD), number of transactions, average, split by payment method.

    WHEN TO USE: If the question contains "deposit", "deposited", "how many dollars came in",
    "funding", ALWAYS call this tool.
    Example: "how many dollars were deposited today?", "how many deposits came in yesterday?"
    WHEN NOT TO USE: When withdrawals are asked (cekim_ozeti), or a single customer's
    balance is asked (musteri_bakiyesi).
    """

The most important lesson of the project is inside this block: the only thing the LLM uses to decide whether to call a tool is the docstring. The agent sends these texts to the model with every request, and the model compares the question with that text. The “when to use / when not to use + example question” form raised accuracy a lot with a small LLM. We measured it: 96% with 11 tools (26/27), 20/20 with 5 tools. Nothing changed in the code; only the text changed.

There are two groups of tools. The real ones look at the server: sistem_bilgisi (system info), dosya_ara (file search), dosya_oku (read file), not_kaydet (save note), notlari_getir (get notes). The business tools are mocks for now, returning a fixed response; they exist to test the architecture. When the real integration arrives (payment system, trading platform, KYC), only the function body changes; the docstring and the agent stay as they are.

The borders between these six tools are close: deposit and withdrawal, position and volume, balance and deposit. That is why the “WHEN NOT TO USE” line in every tool points to its neighbour by name.

Local LLM: it decides, but it cannot touch anything

The agent sends requests to Ollama’s /api/chat with httpx and the tools parameter; native tool calling is supported. temperature=0 so that tool selection is deterministic: the same question asked twice should not go to two different tools.

qwen3 is a reasoning model. Sending think: false did not turn thinking off; it mixed the thinking into the content. When the parameter is not sent at all, the runtime puts the thinking into a separate field. Even then, leftover </think> tags arrive; sometimes the opening tag is swallowed and only the closing one remains. So everything after the last </think> is treated as the answer.

Model selection: no decision without measuring

A test set of 20 questions: 10 that need a tool (“what time is it?”, “take a note: buy milk tomorrow”) and 10 that do not (“what is 2+2?”, “what is a list comprehension in Python?”). The unrelated ones count too: a model that calls a tool when it should not is also wrong.

For accuracy, qwen3:4b is the default; if you want speed, OLLAMA_MODEL=qwen2.5:3b. 11 seconds is long. But three wrong tool selections cost more than 11 seconds: an answer that went to the wrong tool contains invented numbers, and you do not notice.

Agent: the manager

The agent has four jobs: manage the loop, decide what goes to the model, expose a standard API to the outside, and bring itself up.

1. The loop: five rounds, then a final answer without tools

for rnd in range(1, MAX_TOOL_ROUNDS + 1):          # default 5
    resp = await llm.chat(convo, tools=tools)
    if not resp.wants_tools:
        return resp.content                             # plain text came back: done
    convo.append(resp.raw_message)                      # keep the model's tool_call message in history
    for call in resp.tool_calls:
        result = await registry.call(call.name, call.arguments)   # execute on MCP
        convo.append({"role": "tool", "tool_name": call.name, "content": result})

# round limit reached: ask for one final answer, without tools, using the results so far
resp = await llm.chat(convo, tools=None)

Four steps: prompt + tool list go to the model; if plain text comes back, return it; if tool_calls come back, run each one on MCP, add the results to the history as role: tool and go back to the start; if MAX_TOOL_ROUNDS is reached, ask for a final answer without tools. The infinite-loop protection lives in the loop; the rule “if a tool returned an error, do not call the same tool again and again” lives in the system prompt.

2. What goes to the model: the instructions are ours

The agent’s system prompt is short and strict: if the question matches a tool’s “WHEN TO USE” case, call the tool, do not invent the information; things specific to this machine (time, files, notes) can only be learned through a tool; do not use tools for general knowledge and maths; summarise the tool result, do not paste the raw output; if an error came back, explain it, do not retry; if you are not sure, calling a tool is better than inventing.

There are also two rules against what the client sends. System messages from the client are dropped (KEEP_CLIENT_SYSTEM=false); some chat clients add thousands of tokens of their own instructions to every request, and a small LLM cannot see our 11 tools inside that pile. If the user message arrives inside a wrapper (such as <userRequest>), only the inside is taken. Without a wrapper the message passes as it is; curl and Open WebUI go that way.

def _normalize_messages(messages):
    out = []
    for m in messages:
        role = m.get("role", "user")
        if role == "system" and not settings.keep_client_system:
            continue                                   # instructions come from us
        content = m.get("content", "")
        if role == "user":
            content = _unwrap_copilot_user(content)    # wrapper present? take only the inside
        out.append({"role": role, "content": content})
    return out

For debugging, every incoming request is written to .last_request.json. “What exactly does the client send?” is as easy as opening a file. Both rules above were written after looking at that file.

3. HTTP API: OpenAI format

POST /v1/chat/completions and GET /v1/models. Streaming (SSE) exists; for now the response is sent in chunks once it is ready, not as a real token stream. As a backup there is also an Ollama-compatible /api/chat + /api/tags: a client with an Ollama provider can be pointed at the agent’s port. Under every answer, the agent adds which tools were called:

Total deposits today: $184,250 (312 transactions, average $590)
Card 61%, bank transfer 34%, crypto 5%

---
Tools used: depozit_ozeti

4. Bootstrap: zero manual setup

When you run uv run local-agent, it does the following in order: install Ollama if it is missing → start the service if it is stopped → pull the model if it is not downloaded → load it into VRAM with a warm-up → start the MCP Server as a subprocess → listen on :8000. Every step is idempotent; on the second run they are all skipped and the agent is up in 1–2 s. If a step fails, the agent still starts; the error shows up in the chat response, not at the bottom of a log.

From the field: from 5 tools to 11

The first measurement was done with the five real tools: 20/20 on 20 questions. Then six mock business tools were added — deposits, withdrawals, positions, volume, balance, KYC — and the tool count became 11. Same model, same code; the only thing that changed was the length of the tool list in front of the model. Choosing between five short descriptions was easy; between eleven, the borders became blurred: deposit or withdrawal, position or volume, balance or deposit?

The fix was looked for in the text, not in the model: every docstring got “when to use”, “when not to use” and example questions, and the “not to use” line points to the neighbouring tool by name (cekim_ozeti, musteri_bakiyesi). Same model, same code, 26/27 with 11 tools. The border is drawn by the “WHEN NOT TO USE” line, not by code.

The loop teaches a similar lesson. The round limit alone is not enough: a model that gets an error can call the same tool for five rounds, and each round takes ~11 s. The limit only guarantees that it ends in 55 seconds, not that it stops spinning for nothing. That is why the system prompt has a separate line: “if a tool returns an error, explain the error to the user, do not call the same tool again.” The limit stops the loop; the rule keeps it from entering the loop at all.

What to watch

Checklist

Conclusion

The need was one sentence, and the answer was written in one day: agent, MCP Server, bootstrap, benchmark. Now I ask “how many dollars were deposited today?”, the answer comes 11 seconds later, it says underneath which tool ran, and not a single byte leaves the company. No chain of someone asking for the report, someone writing the SQL, someone sending a screenshot; there is a question and an answer.

The hardest part was not sending requests to Ollama or speaking the MCP protocol. The hard part was deciding honestly what a small LLM can do: only “is a tool needed, and which one?” The rest — order, limits, errors, history, instructions — is in the agent. When the model gets bigger, this separation will not break; it will just need less guidance.

The answer to the question in the title: the LLM decides, the MCP Server executes, the agent manages. The agent determines which text goes in front of the model, where the tool result is added, and when the loop ends. The model is only one step of that decision.