# Tom Seidel – Java Freelancer & Cloud-Native AI Expert

> Freelance Java consultant with 20+ years of experience in cloud-native architectures, microservices, DevOps, and AI-powered agentic systems. Based in Germany, remote-first.

Tom Seidel is a freelance consultant specializing in Java development, cloud-native architectures, AI integration, and DevOps. He supports projects end-to-end: from initial concept and architecture to production-ready implementation. His focus areas include migrating monolithic systems to scalable microservice architectures, launching new Java-based cloud projects, and building intelligent agentic workflows using LLM APIs (Anthropic, OpenAI, Ollama, llama.cpp).

## Contact

- Email: tom.seidel@remus-software.org
- LinkedIn: https://linkedin.com/in/tomseidel
- GitHub: https://github.com/tmseidel

## Skills & Technologies

Languages: Java, Kotlin, Bash, Python, TypeScript
AI & LLM: Anthropic API, OpenAI API, Ollama, llama.cpp, Agentic Workflows, Hugging Face
Frameworks: Spring Boot, Micronaut, OSGi, React
Cloud Platforms: AWS, Azure, OVH, Hetzner
DevOps: Docker, Kubernetes, Helm, Terraform, Maven, Ansible
CI/CD: GitHub Actions, Jenkins, GitLab CI
Tools & Protocols: Git, REST, gRPC, MQTT, WebSockets, oAuth2

## Availability

Open for freelance and contract work. Remote-first, based in Germany (CET/CEST timezone). Prefers remote work but open to on-site at regular intervals.

---

## Articles

### Your Agent Is Paying an LLM to Run a For-Loop

URL: https://remus-software.org/articles/llm-agent-and-scripting/
Date: October 9, 2026
Description: Most agent loops make the model the bus between every tool call, paying tokens, context and latency for deterministic bookkeeping. Giving the agent one execute-code tool moves that loop into a script — and adds a security bill that decides whether the saving survives production.
Tags: AI, LLM, Agents, MCP

Most agent harnesses make the model the bus between every tool call: one inference to find the files, another to read each one, another to filter, another to aggregate. None of those hops requires reasoning — they are a deterministic sequence you could write in ten lines of Python — yet you pay for every one of them in tokens, in context window, and in wall-clock time. The fix is small and almost everyone has converged on it: give the model a tool that runs a script, and move the loop out of the model and into the script.
This is not a new idea, and it is not my discovery. It is worth writing down anyway, because the why is what decides how you build it — and the problems it creates are the ones that decide whether it survives contact with production.
The loop everybody writes first
Ask an agent to read every Markdown file and extract the headings, and the default shape of the loop looks like this:
flowchart TB
    L1["LLM: find the Markdown files"] --> T1["find files"]
    T1 --> R1["a large result, back into context"]
    R1 --> L2["LLM: read file A"]
    L2 --> T2["read_file A"]
    T2 --> R2["result"]
    R2 --> L3["LLM: read file B"]
    L3 --> T3["read_file B"]
    T3 --> R3["result"]
    R3 --> L4["LLM: read file C"]
    L4 --> T4["read_file C"]
    T4 --> R4["result"]
    R4 --> L5["LLM: filter and aggregate"]
    L5 --> A["the answer"]
Look at what is actually happening. The only genuinely non-deterministic step is deciding that the task should be “read the files, pull the headings”. Everything after that — which files, hold this result, go read the next, keep only the headings, count them — is bookkeeping. The model gets invoked for the bookkeeping anyway, once per hop, and each hop drags its payload back through the context window. The determinism is the tell: nothing in the middle of that chain needs a model. It is a for-loop wearing a chat interface.
The shape the work actually has
The efficient version keeps the two jobs apart — the model decides, the interpreter executes:
flowchart TB
    L["LLM: write one program"] --> P["Program, in an interpreter"]
    P --> C1["tool call: find the files"]
    P --> C2["tool call: read A, B, C"]
    C1 --> AGG["filter and aggregate, in code"]
    C2 --> AGG
    AGG --> PR["print one compact result"]
    PR --> L2["LLM: read the answer"]
One round in, one round out. The intermediate results — every file’s full text, every intermediate list — never enter the model’s context at all. Only what the program chooses to print() comes back.
Why you cannot simply ship the program as a tool
The obvious objection is: if that program is so good, why not just register it as one more tool and skip the interpreter? Because you cannot write it ahead of time. The task arrives as natural language — “read the Markdown files and extract the headings” is one request; the next one is “find every TODO and group it by author”, the one after that is “compare what these two files declare”. The program is specific to the request, and the request is unknown until the model reads it. You cannot pre-build it, so you cannot ship it as a tool. The model has to write it — which means the tool you add is not a program. It is an interpreter.
That is the whole trick: register execute-code, a tool whose argument is source code, let the model generate the script that fits the task, run it in a sandbox, and hand back its stdout as the tool result.
What the interpreter buys you
Three things fall out of it at once, and they compound.

Missing tools are compensated by scripts. A capability you never wrapped as a tool — a one-off parse, a join across two results, a computation — is expressible in the same program, because it is just code. This is the composability argument CodeAct makes: a constrained action space (a fixed set of pre-defined tools) both limits what the agent can reach and, worse, makes it unable to compose multiple tools into one action [3]. Code, tools and control flow live in one action space.
Fewer round trips, so less cost, less context and less latency. This is the point the harness vendors have now made in public. Anthropic’s worked example — loading a Google Drive document and handing it to Salesforce through code instead of through the model — drives token usage from 150,000 tokens to 2,000 tokens, a 98.7% saving, precisely because by default “the full call transcript flows through [the model] twice” [1]. Cloudflare puts the same mechanism plainly: with the traditional approach “the output of each tool call must feed into the LLM’s neural network, just to be copied over to the inputs of the next call, wasting time, energy, and tokens” [2]. Simon Willison, reading the Anthropic post, notes the same win from the other side — not round-tripping the first response through the model is “faster, more reliable, saves on context tokens, and avoids the model being exposed to any potentially sensitive data” [5].
Fewer tool definitions, so still less context. This is the half that is easy to miss. Every tool you register carries a schema and a description in the system prompt, whether or not it is used. An agent wired to hundreds of tools is “process[ing] hundreds of thousands of tokens before reading a request” [1]. Presenting the surface as code you can browse and call on demand — Anthropic calls it progressive disclosure, loading only the definitions a task needs [1] — lets one execute-code tool stand in for a catalogue, and an entire API fit “in 1,000 tokens” by describing operations as typed code rather than as separate tools [6].

It is the same conclusion reached independently by Anthropic’s MCP team [1] and by Cloudflare’s Code Mode [2][6]. The CodeAct line of research [3] and the smolagents library — whose whole premise is “code agents” that “write actions in code” rather than as JSON [4] — arrive at the same shape from the research side. When four groups with different incentives land on the same shape, it is worth understanding rather than copying by superstition.
The return channel: the script has to reach the real tools
Here is where a naive implementation goes wrong. It is tempting to treat execute-code as a self-contained Python sandbox — a place the model can compute things, detached from the agent’s actual tool surface. That throws away most of the value. The model already has the tool configurations and the MCP schemas in its context; the generated script should use them. The program needs a way to call the same tools the model can call, including MCP tools, from inside the sandbox.
In our implementation the bridge is deliberately thin. The script imports a small module and gets one object:
import llmhelper

names  = llmhelper.tools.list()                       # what this run may call
schema = llmhelper.tools.describe("rg")               # its arguments, on demand
result = llmhelper.tools.call("rg", {"args": ["TODO"]})
print(result["output"])

Every call becomes one JSON line on a Unix-domain socket named by an environment variable, and the reply is decoded back into Python objects; an ordinary refusal or a failed tool comes back as a result with success=False, not as an exception. MCP tool names are unchanged — mcp:<server>:<tool> is exactly what the program passes to tools.call(...) — so a generated script drives the same MCP surface the model drives directly. On the Java side the frames look like this:
→ {"type":"tool_call","id":"3","name":"mcp:github:search_issues","arguments":{"query":"x"}}
← {"type":"tool_result","id":"3","result":{"success":true,"exitCode":0,"output":"…"}}
← {"type":"tool_error","id":"3","error":{"code":"TOOL_CALL_LIMIT","message":"…"}}

and the routing looks like this:
flowchart TB
    L["LLM writes a Python program"] --> P["Program: tools.call,then filter, aggregate, print"]
    P -->|"one JSON line per call,over an AF_UNIX socket"| B["Bridge, served by the JVM"]
    B --> T["The same tool surface the model has:built-in reads plus the selected MCP tools"]
    T -->|"nested result, capped before it crosses back"| R["Back in the program"]
    R -->|"stdout only"| O["One compact result, intothe LLM context"]
Two details matter more than they look. First, stdout is the output channel and the socket is the control channel — they are kept apart so a program cannot print something that is mistaken for a protocol frame. Second, every nested result is capped before it crosses the bridge, so a program cannot smuggle a 10 MiB file into the model’s context with a careless print() on its last line. The isolation is structural: a nested tool result never becomes a conversation message, because it never touches the agent loop at all.
What it changed in our own runs
The concrete result, on our side: reviewing large changes dropped to roughly half the round trips for the same outcome. The review agent spends its iterations gathering — read this file, check that declaration, compare the two — and those iterations are exactly the deterministic kind the interpreter collapses into one. Halving the round trips halves the inferences, and on a local model running on constrained hardware that is the difference between an agent you can afford to put in a business process and one you cannot. The token saving is welcome on a paid API; on a small local model, where throughput is the product, it is the whole argument.
Security: the bill for running foreign code
Everything above is a saving. This is the part that is not, and it is the reason execute-code should never be a default.
You are now executing code that a model wrote, in response to input that may be hostile. A prompt injection — an instruction hidden in a file the agent reads, a PR description, a tool result — can steer the generated program, and “prompt injections do not need to be human-visible … as long as the content is parsed by the model” [7]. Once the program runs, it runs with reach:
flowchart TD
    PR["Untrusted content:PR text, a file, a tool result"] --> PI["Prompt injection"]
    PI --> LLM["The agent writes a program"]
    LLM --> EX["The program runs with the agent's reach"]
    EX --> S1["reads the process environment:API keys, DB credentials"]
    EX --> S2["reads the filesystem"]
    EX --> S3["opens a network connection"]
    S1 --> LK["Exfiltration"]
    S2 --> LK
    S3 --> LK
If the program runs in the same context as the harness, it can read the harness’s secrets — API keys, passwords, the JVM’s own start-time environment — touch the filesystem, and reach the network. Anthropic says as much about its own pattern: running agent-generated code “requires a secure execution environment with appropriate sandboxing, resource limits, and monitoring”, and those “infrastructure requirements add operational overhead and security considerations that direct tool calls avoid” [1]. That sentence is the honest counterweight to the 98.7%.
Our posture, and it is a posture rather than a proof:

a scrubbed environment for the subprocess, so the program does not inherit the harness’s secrets;
a per-execution identity from a pool of low-privilege accounts, one per run — so a program reads neither the service user’s files nor a concurrent run’s program, bridge socket or process;
resource limits (RLIMIT_AS, RLIMIT_CPU, RLIMIT_FSIZE, RLIMIT_NPROC) applied by a bootstrap, with the harness-side timeout as the backstop;
an import guard that refuses the obvious network and FFI modules — and which we label honestly as defence in depth only. A program that actually wants out does not import socket; it calls subprocess or reaches for ctypes. The guard raises the cost of an accident; it is not the boundary.

The real boundary is the deployment: run the whole agent somewhere it cannot hurt you, or accept the risk knowingly. The sandbox confines the filesystem and the network weakly; the identity separation and the scrubbed environment are what actually hold, and the network is open unless you close it at the container. Nothing there is free, and none of it is optional once the tool exists.
If you build this, build it honestly

Add the interpreter as a tool the model can choose, not as the default path for every read. A single file read is still cheaper as a direct call; the win is in multi-step work.
Give the script the same tool surface the model has — built-in reads plus the bot’s selected MCP tools — or you have built a calculator, not an agent.
Keep stdout and the control channel separate, cap every nested result, and set a tool-call budget per execution.
Scrub the environment, run under a dedicated identity, and apply rlimits. Treat the import guard as decoration, never as the boundary.
Make the tool opt-in per agent, and say so in the docs. It is not in the default configuration for a reason.
Watch the round-trip count as a first-class metric. It is the number the whole technique exists to move.

Verdict: the cheapest optimisation is a loop that never enters the model
The interesting thing about scripting in agents is how little of it is about scripting. The tool is trivial to add; the discipline is deciding what is actually a model’s job. Every step in an agent loop that is deterministic — fetch, filter, aggregate, compare, repeat — is one the model should not be billed for, and the interpreter is simply the mechanism that lets it step out of the way. That is worth fifty lines of Python.
What is not trivial is the bill that arrives with it: untrusted code, in your process’s neighbourhood, with your secrets and your network in reach. Present the savings without the sandbox and you have shipped a foot-gun with a nice benchmark. Present both, and the pattern is what makes agents affordable enough to put next to real work — especially where the model is small and the hardware is not generous. The for-loop was never the model’s job. Give it an interpreter and stop paying it to run one.
Sources

[1] https://www.anthropic.com/engineering/code-execution-with-mcp — Anthropic — Code execution with MCP: Building more efficient agents
[2] https://blog.cloudflare.com/code-mode — Cloudflare — Code Mode: the better way to use MCP
[3] https://arxiv.org/abs/2402.01030 — CodeAct — Executable Code Actions Elicit Better LLM Agents
[4] https://huggingface.co/blog/smolagents — Hugging Face — Introducing smolagents, agents that write actions in code
[5] https://simonwillison.net/2025/Nov/4/code-execution-with-mcp — Simon Willison — Code execution with MCP (commentary)
[6] https://blog.cloudflare.com/code-mode-mcp — Cloudflare — Code Mode: give agents an entire API in 1,000 tokens
[7] https://genai.owasp.org/llmrisk/llm01-prompt-injection — OWASP — LLM01:2025 Prompt Injection

---

### Jev Is Not a Faster LLM — It Is a Bill for the Architecture You Skipped

URL: https://remus-software.org/articles/jev-agentic-integration/
Date: September 22, 2026
Description: Jev answers in milliseconds for almost nothing per call — but it is not a faster LLM. It is a parallel sampler that relocates the reasoning into the pipeline you have to build, own and test, turning hallucination into silent miscalibration.
Tags: AI, LLM, Agents, Architecture

The hype train is packed again. Ever since TypeSafe AI released Jev, the internet has been full of examples that make it look like the fast, cheap alternative to a classical LLM: 70–500 ms instead of seconds, $0.042 per million input tokens, output “too cheap to meter”, 40x to 200x speedups. The conclusion is drawn implicitly and sometimes explicitly: why pay for a reasoning model, when this thing answers instantly for nothing?
That conclusion is wrong, and it is wrong in a way that costs money. Jev is not a cheaper way to do what an LLM does. It is a different computation for a different problem — and it quietly moves the expensive part of the work into your codebase, where the hype slides, the demos and the LinkedIn posts never look.
Here is the honest version.
The speed is not an optimization, it is a different problem
Give a reasoning model this task:

Should this pull request be merged? Analyze the changes, consider security, tests,
architecture and possible regressions.

Roughly this happens: context, understand, form hypotheses, intermediate steps, compare, decide, and finally formulate an answer. That last step is where the bill is. An LLM generates autoregressively, token by token, conditioned on the token before it. Structured JSON output does not change that — it is still one token at a time, just with a schema.
Jev gets a different shape of input:
STATE:
- change touches OAuth callback handling
- security scan: no findings
- tests: 142/142 passing
- API compatibility: unchanged
- review finding: possible missing null check
- project policy: severity >= HIGH blocks merge

QUESTIONS:
1. finding_severity = [NONE, LOW, MEDIUM, HIGH, CRITICAL]
2. merge_blocking = [yes, no]
3. needs_human_review = [yes, no]

and returns a distribution:
severity:
  MEDIUM: 0.74
  HIGH:   0.19
  LOW:    0.07

merge_blocking:
  no:  0.81
  yes: 0.19

No text. No chain of thought. All outputs in a single forward pass. TypeSafe calls the architecture a parallel sampler and the training method Reinforcement Learning for Calibrated Decisions (RLCD) — probabilities optimized against outcomes instead of against human raters’ preference.
Line those two up and the “faster LLM” story collapses. The reasoning model pays for every token of its own deliberation:
flowchart TB
    C[Context / prompt] --> N[Neural pass]
    N --> T1["First I need to check ..."]
    T1 --> T2[token]
    T2 --> T3[token]
    T3 --> T4["However ..."]
    T4 --> T5[token]
    T5 --> T6["... and therefore the answer is"]
    T6 --> D[Decision]
Jev pays once per question and hands back a distribution:
flowchart TB
    C["Context / statediff summary, scan results, test results,API compatibility, findings, project policy"] --> N[Neural pass]
    N --> D1["severity: MEDIUM 0.74, HIGH 0.19, LOW 0.07"]
    N --> D2["merge_blocking: no 0.81, yes 0.19"]
    N --> D3["needs_human_review: no 0.9, yes 0.1"]
Comparing those as if one were the optimized version of the other is like calling a calculator a faster author. Text in either way, and a transformer under the hood according to TypeSafe — but a fundamentally different compute problem.
Does Jev not reason at all? Wrong question
Jev still has to see relationships in the state and weigh them. What it does not do is narrate that weighing as tokens. The difference is not “reasoning versus no reasoning”, it is where the decomposition happens:

With a reasoning LLM, the decomposition happens at inference time, in tokens, inside the model. You pay for it on every call, and you get a rationale you cannot trust.
With Jev, the decomposition has to have already happened — in your questions, in your options, in the workflow that assembled the state. If you did not do that work, no parallel sampler will do it for you at 3 a.m.

So the honest sentence is not “Jev needs less reasoning”. It is: Jev relocates the reasoning, and the new location is your repository.
Nobody advertises the pipeline that feeds Jev
Consider the question people actually want answered:

Is this change safe?

To answer that, someone has to walk this chain:
flowchart TD
    Q["Is this change safe?"] --> A[Read the diff]
    A --> B[Find the affected classes]
    B --> C[Follow the call graph]
    C --> D[Look up auth configuration]
    D --> E[Determine library versions]
    E --> F[Check CVE data]
    F --> G[Inspect the tests]
    G --> H[Decide]
    H --> I["Jev: probabilities for this one decision"]
    style Q stroke-dasharray: 5 5
Jev does not collapse that chain. Not in 100 ms, not in 100 s. What it collapses is the last hop: instead of asking a model to find out what is going on and decide in one breath, you hand it a state that is already bite-sized and ask it to decide. The difference is the whole product:

Not “figure out what the situation is.” But: “here is the entire situation — decide.”

That is a legitimate, valuable, genuinely hard-to-do-well thing. It is also a thing that presupposes the chain above it exists, runs, and is correct. Every millisecond Jev saves on decision latency is a millisecond some other system spent on context assembly. That system is written by you, tested by you, and paged at 3 a.m. by you.
What a Jev integration actually looks like
flowchart TB
    A["Classical softwarecollect context, read the database,call APIs, compute, apply rules"] --> B["Jevfuzzy decision, 100-500 ms"]
    B --> C["probabilities and confidence"]
    C --> D["Classical softwareif / route / retry / stop"]
    D -.->|"next iteration"| A
And what it does not look like:
flowchart LR
    U[User] --> J[Jev] --> X["Autonomously solves a complex problem"]
    X --> Y["This is not a feature request, it is a category error"]
The confusion is understandable: the name “System One” — borrowed from Kahneman’s fast, intuitive thinking — plus the framing of “frontier intelligence in a function call” suggests that Jev thinks like a reasoning model, only faster. It does not. It is an extremely capable, context-dependent classifier with calibrated confidence. An agent it is not. Calling it one is how you end up with autonomy that works in the demo and flakes in production, with nothing to read in the logs but floats.
The reasoning moves into your program graph — and that is the good news
Here is the pattern people are actually building, for the merge decision above:
flowchart TD
    S["Prepared state:diff summary, scan results, test results,API compat, findings, project policy"] --> SR["securityRisk = Jev(...)"]
    S --> TR["testRisk = Jev(...)"]
    S --> AR["apiRisk = Jev(...)"]
    S --> QR["qualityRisk = Jev(...)"]
    SR --> D1{"securityRisk above 0.8?"}
    D1 -->|yes| B[BLOCK]
    D1 -->|no| D2{"apiRisk above 0.6?"}
    D2 -->|yes| H[HUMAN_REVIEW]
    D2 -->|no| D3{"testRisk below 0.2and qualityRisk below 0.3?"}
    D3 -->|yes| M[MERGE]
    D3 -->|no| H
The knowledge that used to be implicit in a model’s chain of thought — security outranks style, so check security first; if security is critical, the other results do not matter anymore — is now explicit in code. Which is genuinely better: code is diffable, reviewable, unit-testable, and it does not cost a token to branch.
But notice what had to happen for this to work. Somebody had to write down that security outranks style. Somebody had to define the four questions, the option sets, the thresholds 0.8 / 0.6 / 0.2 / 0.3, and somebody has to own the answer to “why 0.8?”. With an LLM agent you never wrote any of that down and it mostly worked anyway. With Jev, the design is the product. That is not a flaw — it is the bill arriving for the architecture you skipped.
The only combination worth building into an agent
The interesting hybrid is not “Jev instead of the reasoning model”. It is a division of labour:
flowchart TB
    L["Reasoning LLMexplore, investigate, generate"] --> ST["state / results"]
    ST --> G1["Jev: is this finding relevant?"]
    ST --> G2["Jev: is a retry worth it?"]
    ST --> G3["Jev: did the review gate pass?"]
    ST --> G4["Jev: another tool call needed?"]
    G1 --> W["workflow / code"]
    G2 --> W
    G3 --> W
    G4 --> W
    W -.->|"next step"| L
The expensive model does only what needs exploration or generation. The dozens of small judgements per step — relevant? retry? good enough? escalate? — go to a model that answers in milliseconds with a number attached. At 40x to 200x the speed and a small fraction of the cost per call, you can afford gates that were previously too expensive to even consider, and that is the real win: not replacing your model, but adding judgement where you previously made do with a hard-coded rule or nothing at all.
Now the part nobody puts on the slide: gates multiply. A chain of ten gates at 95% accuracy lands at 60% end-to-end. Twenty land at 36%. To keep twenty gates above 98% you need roughly 99.9% per gate — which is exactly why “it is just a classifier with calibrated probabilities” is not a downgrade, it is the whole requirement. And it is why your test suite now needs calibration checks on your distribution, not benchmark leaderboards.
The myth of “it cannot hallucinate”
This one deserves a hard stop. TypeSafe’s claim is technically careful: possible outputs are defined in advance, so the model never makes type errors, and schema matching is guaranteed. Fine. That is a statement about form, not about truth.
Jev can absolutely decide wrongly. It can label a CRITICAL finding as MEDIUM at 0.74, and your code, having been told that 0.74 is the answer, will route on it. What Jev removes is the visible failure mode, not the failure:
flowchart LR
    A["LLM: a wrong sentencein fluent text"] --> B["Someone notices,laughs, retries"]
    C["Jev: a wrong labelwith a confident probability"] --> D["Your code branches silently"]
    D --> E["Weeks later:a post-mortem withno rationale to read"]
Hallucination has not been abolished here. It has been converted into silent miscalibration — and for operations, that is the worse failure mode, because a wrong sentence is loud while a wrong float is invisible, reproducible, and trusted by CI. Three consequences follow immediately:

You cannot ask why. Simon Willison’s complaint is the right one: an LLM at least produces a rationale you may not trust; a decision model produces a number. If Jev flags something, which content signals tipped it off? Nobody can tell you, and bias now hides inside a float.
Calibration is a claim, and it is about their distribution. That a model returns honest probabilities does not mean it stays honest on your inputs. TypeSafe’s own jaggedness notes admit weak spots on numbers, dates and adversarial content. Perturb a prompt slightly, watch a decision flip, and try to explain it to an auditor.
The blast radius got bigger. An LLM hallucinating a tool call is inconvenient. The same model buried three layers deep in a dependency chain with a latency budget is a production incident.

So: not “Jev cannot hallucinate”. Rather: “Jev’s wrong answers no longer look like wrong answers”. If you cannot detect them, you have not automated a decision — you have automated trust.
If you are going to build this, build it honestly

Keep state assembly in ordinary software. Jev decides, it does not look things up.
Express every decision as a question with a typed, closed option set — if you cannot, you have not thought the decision through yet.
Set thresholds explicitly and write down why they are what they are. A number nobody can justify is a liability, not a configuration.
Measure calibration on your inputs, not on the vendor’s benchmark. Then break your inputs on purpose and watch which decisions flip.
Read the gate chain end to end and multiply the accuracies. Compound error is where “fast and cheap” turns into “fast, cheap and quietly wrong”.
Never let a gate decide something you cannot explain to a reviewer, a customer or an auditor. “The model said 0.74” is not an explanation.

Verdict: a very good tool with a very bad pitch
Jev is a serious, interesting approach, and for the right problem shape it is the best tool currently on the market: constant, documentable, cheap judgement inside ordinary software. Used well, it is what makes agentic systems affordable — cheap gates around an expensive explorer.
But it is not a replacement for LLM reasoning. It is less reasoning, applied precisely where reasoning was never what you needed — only a decision. Everything that made the decision hard (knowing what matters, assembling the state, choosing the questions, setting the thresholds, owning the calibration) stays on your side of the fence.
Which is the actual provocation here. The hype train is selling you speed and a price tag. What Jev really ships is a demand: write down what you actually decide, in code, with numbers, and test it. Teams that skipped that work are not going to be saved by a parallel sampler; they are going to discover, in production, that they never knew what they were asking.
Stop calling it a faster LLM. Start treating it as a receipt — for the architecture you have been postponing all along.

---

### Regarding the Job Market: Why 2026 Feels Like a Crisis — and Why the Profession Isn't Dying

URL: https://remus-software.org/articles/ai-job-market/
Date: September 10, 2026
Description: The 2026 job market feels like a crisis, but the data says otherwise: demand for software is intact, only the shape of the roles is changing. What the numbers actually show — and why the doomsday narrative is a product of filter bubbles.
Tags: AI, LLM, Software Engineering

Regarding the Job Market: Why 2026 Feels Like a Crisis — and Why the Profession Isn’t Dying
When ChatGPT started making its way into development workflows — back then it was all copy-paste from the chat window — and non-developers suddenly started vibe-coding prototypes that looked genuinely impressive, the question immediately surfaced: do we even need developers anymore? From day one I was a convinced believer in AI as a tool, and I still am: I’m more confident than ever that developers are needed more than before, not less. And the job-market statistics from the major job boards broadly support that view. We still have a massive deficit of applications that are supposed to drive the digitalization of everyday processes, and the demand for software remains unbroken.
The numbers: strong overall demand, a transforming job market
The current state of the market is best described as a split, not a collapse. AI engineering roles are growing much faster than classic software engineering roles, while generalist positions are growing only modestly or sitting flat.[1] At the same time, the tooling is already everywhere: 84% of developers now use or plan to use AI tools in their development process, and 51% of professional developers use them daily.[9] Roughly 40% of software engineering job postings already mention AI coding tools — Copilot, Cursor, Claude Code — as a requirement or a plus.[23]
The junior paradox: the right data, the wrong conclusion
In certain bubbles, the warning is loud: getting in as a junior developer is the hardest it has ever been. The numbers back that up. A study by Stanford’s Digital Economy Lab, built on ADP payroll data covering millions of U.S. workers, found that employment for software developers aged 22–25 declined by nearly 20% from its late-2022 peak.[5][7] U.S. entry-level tech job postings dropped by roughly 67% between 2023 and 2024 in that same Stanford/ADP analysis.[3]. The reason why I’ve chosen statistics from the U.S. is that they are the most detailed and reliable I could find.
But that is the wrong way to look at it. Getting into the profession has not become harder in an absolute sense — the profile of the entry-level role has simply changed. Tasks that junior developers typically handled are now done reliably by AI. What has changed is the job description, not the demand for fresh talent. What juniors need is to adapt their skill set to the new conditions: use AI as an accelerator in their own work, focus on system design and software architecture, build more business and domain knowledge — or pick crisis-resistant industries. That work methods and tools keep changing and improving is nothing new. We have been using tools and abstractions that massively boost productivity for decades. Business as usual.
Why 2026 feels like a crisis year
If demand is still there, why does 2026 subjectively feel like a crisis year for software developers? My analysis of the market psychology behind that:
1. The fear of “tech debt”
Companies are afraid to spend millions today on software architectures or large teams for projects that new AI models could render obsolete — or solve in a radically different way — in two years.
The worry: anyone who commits to a specific framework or expensive in-house development today may be building tomorrow’s legacy baggage.
The consequence: companies wait until standards for AI-native software architectures have settled. The market’s own uncertainty feeds this: Gartner predicts that 40% of agentic AI projects will be scrapped by 2027,[22] and 2026 Gartner/IDC data puts the share of enterprise AI agent pilots that stall at 89%.[13]
2. The team-scaling paradox
The core uncertainty is a simple question: how many developers will we need in three years for the same work?
The worry: If AI agents multiply a single developer’s productivity several times over, an aggressive hiring wave today becomes massive overcapacity tomorrow.
The consequence: Companies currently prefer to invest in higher compute capacity and AI licenses rather than in additional full-time employees. They scale performance through technology, not through headcount. The money is visibly moving in that direction: enterprises are burning through entire AI budgets in months,[11] while the hyperscalers alone are planning roughly $700 billion in AI capex for 2026.[12] Gartner even predicts that AI coding token costs will overtake the average developer’s salary by 2028.[18]
3. The legacy problem
Many companies are standing in front of gigantic migration projects — replacing old COBOL or Java monoliths, for example. Here the “wait and see” attitude is extreme
The worry: Why pay consultants millions when LLMs might soon handle such migrations fully automated and error-free at the push of a button (btw: This rests on a promise that track record does not yet support)?
The consequence; Vendors are aggressively selling exactly that promise,[20][21] and as long as the pain isn’t acute, those mega-projects get postponed.
What makes the situation feel even more strained is that the “freeze” is only selective. Companies are waiting on juniors while simultaneously hunting, almost frantically, for tech leads and software architects who can bridge existing systems and generative AI — the field is bifurcating, with implementation roles shrinking while system-level, architecture, and AI-oversight roles expand.[25]
The doomsday hype in the filter bubbles
In certain online bubbles — LinkedIn, select subreddits — there is a partly hysterical fear that has little to do with reality. I see four causes:
1. The “survivor bias” of the frustrated
Extremely frustrated people who wrote 200 applications and got only rejections, then pin the blame on AI — plus attention-hungry tech influencers chasing clicks and reach with dark prophecies. It is easy to overlook the thousands of developers who say nothing, write code, and use AI to write better code.
2. Confusing “coding” with “software engineering”
Most demos show an AI implementing a specific function in the shortest possible time. The thinking error is that many people believe typing code is a developer’s main job. Reality looks very different: research puts the time developers actually spend writing code at around 11% of the working week — about 52 minutes a day,[26] and broader industry data on how developers spend their time points in the same direction.[17] The real work consists of tasks that LLMs cannot perform on their own.
3. Hype distortion
On LinkedIn or Reddit it often feels like the state of the art is already successfully deployed in enterprises — “AI agents now replace whole teams.” In reality, companies are still fighting the basics: data privacy, hallucinations, intellectual property, and how to build a unified, resilient AI infrastructure inside an enterprise at all. The numbers tell the same story: a March 2026 survey found 78% of enterprises have AI agent pilots, but fewer than 15% have scaled even one agent to operational use.[14] Gartner’s own forecast for “vibe coding” — 40% of new enterprise software built with these techniques — is only dated 2028, not today.[19]
4. Underestimating maintenance
“If AI writes the code, we don’t need people” is completely wrong — the opposite is true. The cheaper it becomes to generate code, the more code exists in our systems that has to be understood, maintained, updated, and secured — work that will require plenty of developers for decades. You can already see this spillover in open-source projects, which are being flooded with AI-generated contributions: in February 2026, GitHub itself acknowledged the surge in low-quality AI-generated contributions and described it as an “Eternal September” for open source.[15]
Bottom line
The fear of the profession changing is justified. The fear of the profession disappearing is a product of filter bubbles. The profession of software developer is not dying — it is changing at high speed. Anyone who develops from a “code typist” into a “system problem-solver” will have a secure career in the future as well.
The wage data points in the same direction as the conclusion: the Stanford/ADP analysis found that workers with AI skills earn roughly 62% more than their peers.[6] If AI were simply replacing developers, it should depress their wages — and it is not. The data is consistent with a market in which AI raises the value of developers who can direct it, which is precisely the bet that the “adapt, don’t panic” advice is based on.
Sources

https://newsletter.pragmaticengineer.com/p/state-of-the-job-market-2026 — The Pragmatic Engineer: State of the job market 2026
https://www.danilchenko.dev/posts/junior-developer-jobs-2026 — Danilchenko: Junior developer jobs in 2026
https://byteiota.com/junior-developer-extinction-67-hiring-collapse-explained — ByteIota: Junior developer extinction, 67% hiring collapse
https://hakia.com/news/junior-developer-crisis-2026 — Hakia: Junior developer crisis 2026
https://thecollegeinvestor.com/81990/stanford-study-entry-level-software-jobs-down-nearly-20-as-ai-reshapes-hiring-for-college-grads — The College Investor: Stanford study on entry-level software jobs
https://www.digitalecononews.com/en/articles/stanford-economist-ai-entry-level-jobs-crisis — Digital Econ News: Stanford economist Brynjolfsson on AI entry-level jobs crisis
https://msftnewsnow.com/stanford-study-generative-ai-entry-level-jobs — Stanford study: Generative AI’s early impact on entry-level jobs
https://digitaleconomy.stanford.edu/project/indicators/canaries-dashboard — Stanford Digital Economy Lab: Canaries Dashboard
https://survey.stackoverflow.co/2025/ai — Stack Overflow 2025 Developer Survey: AI
https://stackoverflow.co/company/press/archive/stack-overflow-2025-developer-survey — Stack Overflow 2025 Developer Survey press release
https://www.forbes.com/sites/timbajarin/2026-04-29/ai-compute-surpasses-human-costs-enterprise-budgets-shift — Forbes: AI compute surpasses human costs, enterprise budgets shift
https://www.cnbc.com/2026-02-06/google-microsoft-meta-amazon-ai-cash.html — CNBC: Tech AI spending approaches 700B in 2026
https://www.beri.net/article/ai-agent-adoption-enterprise-2026-gartner-idc — Gartner/IDC 2026: 89% of enterprise AI agent pilots stall
https://agentmarketcap.ai/blog/2026-04-11/ai-agent-reality-check-hype-to-production-gap-2026 — AI Agent Reality Check: 78% pilots, under 15% to production
https://en.wikipedia.org/wiki/Vibe_coding — Wikipedia: Vibe coding
https://www.sonarsource.com/blog/how-much-time-do-developers-spend-actually-writing-code — Sonar: How much time do developers spend actually writing code?
https://www.software.com/reports/code-time-report — Antenna: Global Code Time Report
https://www.techtimes.com/articles/319333/20260629/ai-coding-costs-can-drain-budget-days-gartner-predicts-they-will-match-developer-pay.htm — TechTimes: Gartner on AI coding costs vs developer pay
https://theoutpost.ai/news-story/vibe-coding-transforms-software-development-as-gartner-predicts-90-enterprise-adoption-by-2028-30292 — The Outpost: Gartner vibe coding 40% by 2028
https://medium.com/@hashbyt/what-happens-when-ai-meets-legacy-cobol-systems-00dae11f51d3 — Medium: AI COBOL modernization in 2026
https://ascendion.com/ai-basic/mainframe-and-cobol-modernization-with-agentic-ai-whats-actually-possible-in-2026 — Ascendion: Mainframe modernization with agentic AI in 2026
https://www.kore.ai/blog/ai-agents-in-2026-from-hype-to-enterprise-reality — kore.ai: Gartner predicts 40% of agentic AI projects scrapped by 2027
https://gitgood.dev/blog/2026-tech-job-market-hiring-rebound-ai-roles — GitGood: 40% of SE job postings mention AI coding tools
https://ucstrategies.com/news/ibm-lost-40b-because-ai-cant-actually-modernize-cobol — UC Strategies: AI COBOL modernization evidence gap
https://theboard.world/articles/technology/will-ai-replace-software-engineers-2026-reality-check — TheBoard: Will AI replace software engineers in 2026?
https://medium.com/@vikpoca/developers-spend-only-11-of-their-time-coding-what-3a53f65982df — Ponamarev: Developers spend only 11% of their time coding

---

### The Low-Code Automation Hype Isn't Fading in 2026. That's the Problem.

URL: https://remus-software.org/articles/low-code-hype-with-ai/
Date: September 8, 2026
Description: The low-code automation market is at its all-time peak, but the agentic AI promise underneath it is entering its trough of disillusionment. Why I still think code-first is the winning play in the AI era.
Tags: AI, LLM, Agents, Low-Code, n8n, Software Architecture

Low-Code Automation’s 2026 Hype Isn’t Fading. That’s the Problem.
People keep asking whether the low-code automation hype — n8n, Zapier, Make, Coze, Dify, LangFlow — is finally cooling off in 2026. I researched it. The answer is worse for the skeptics than you think: the low-code automation market is not cooling. It’s at its peak. The thing that’s in the trough is the promise it’s built on — autonomous AI agents. And that’s exactly why I think code-first is the winning play.
The data: low-code automation is not fading
If you ask “is the low-code automation hype over in 2026?”, the numbers say no:

n8n more than doubled its valuation in 2026. $2.5B after its $180M Series C (October 2025) [1], then $5.2B when SAP took a strategic stake in May 2026 — and signed a multi-year deal to embed n8n’s workflow canvas inside Joule Studio, SAP’s AI agent-building platform [2][3].
195k+ GitHub stars [4], a growing “fair-code” community, and a flagship conference (“In The Loop 2026”) that’s now an industry event in its own right.
Zapier shipped “Agents” with 7,000+ app integrations. Make, Coze, and Dify are all expanding their AI-node feature sets [5]. The entire category is racing to add “AI” to the visual canvas.

So let’s not pretend the 2010-style “it all died out” narrative applies here. It doesn’t. The low-code automation market in 2026 is the strongest it’s ever been.
So the market isn’t growing because the visual canvas got better. It’s growing because the canvas is now the delivery vehicle for agentic AI. n8n’s own homepage says it: “Build visually, go deep with code” — for AI agents [6]. Zapier sells autonomous agents [5]. SAP is buying n8n to put the visual canvas inside its agent platform [2][3]. The visual workflow builder has quietly become the industry’s preferred way to ship “AI agents” to business users.
And now we get to what is in the trough.
What is fading: the agentic AI promise underneath the canvas
In April 2026, Gartner published its first dedicated Hype Cycle for Agentic AI — separated from the broader GenAI cycle for the first time [7]. Its verdict: AI agents are at the peak of inflated expectations and will slide into the trough of disillusionment throughout 2026. Gartner calls 2026 the “year of disillusionment” for agentic AI [8].
The supporting data:

An 88% security incident rate for AI agents (NoCode.Tech’s coverage of enterprise incident data) [9] — what happens when autonomous decision-making touches live infrastructure without a governance layer.
Microsoft pausing data-center construction. Anthropic delaying a major release over safety concerns [10]. The broader AI sector is hitting real headwinds in 2026.
Gartner’s framing: the trough “reflects how agentic AI was sold and bought, not whether the technology works.” One line, the whole 2010–2013 story.

Read those two facts together — low-code automation at an all-time peak, agentic AI heading into its trough — and the picture gets sharp: low-code automation has become the Trojan horse for the agentic AI hype. The growth in workflow platforms is largely growth in “AI agent” workflows: LLM nodes on the canvas, natural-language workflow building, agents that “reason” between connectors. When the agentic trough lands — and Gartner says it lands in 2026 [7][8] — the platforms won’t shrink, but a lot of what their users built will be graphs of LLM calls that nobody can explain, test, or trust.
That’s a more dangerous position than the old hype, not a less dangerous one. In 2010, a broken BPM graph was a maintenance chore. In 2026, a broken 400-node agent graph is a silent compliance and security incident.
Why the 2010 wave matters: we saw this ending once
Around 2010 the enterprise world got a structurally identical pitch: SOA orchestration, ESB tools, Mule, TIBCO, jBPM, Camunda’s rise, BPMN modelers, iPaaS. “Developers are the bottleneck — let business people wire integrations visually. No code. Citizen developers. Done.”
The ending is well documented: abstraction tax, version drift, connector sprawl, “who owns this when the modeler leaves.” The industry went back to code. The only piece of that wave that survived is the BPMN modeler as a companion to the IDE — modeling survived, implementation did not.
The 2026 wave repeats the same core move (visual abstraction over real integration logic) and adds one new ingredient: nondeterminism inside the graph. That’s the difference between a 2013 problem and a 2026 problem.
Why AI makes the visual canvas worse, not better
1. Two hidden layers of nondeterminism. A 200-node graph already has a black box in every connector. Now 15 of those nodes are LLM calls that return something different every run. Model nondeterminism stacked on platform nondeterminism: the pipeline isn’t flaky anymore, it’s unprovable.
2. “AI builds your workflow from a description” just moves the drift into the prompt. Hallucinated field names, mismatched connectors, silently best-effort transformations. The 2010 wave hid complexity behind boxes; this wave hides it behind a chat.
3. Observability and review stop working. How do you trace a 400-node graph with 12 LLM calls in it? Diff it? Regression-test it? The visual format was never a good git citizen; with LLM nodes it becomes unauditable.
4. The abstraction tax now includes the model’s limits. Every LLM node is a context window, a token cost, a hallucination risk, and a latency spike — wrapped in a pretty node. Your “simple” workflow is a distributed system with a stochastic dependency you can’t upgrade.
5. The platforms know the ceiling exists — that’s why they all ship code nodes. n8n’s feature list literally says “fall back to code: drop into JavaScript or Python in a code node” [11]. The escape hatch is part of the product. That is the entire low-code thesis conceding defeat, shipped as a feature.
6. “Low-code” was never low-maintenance; it just looked that way. AI makes it look even more low-maintenance. The work doesn’t disappear — it becomes invisible until an LLM node returns a different JSON shape on a Tuesday at 2 a.m. and the pipeline quietly stops.
Why code-first wins — especially in the AI era
The architecture for AI automation is deterministic core, stochastic edges: code that orchestrates, with well-defined LLM calls at the edges — schema-validated, tested, version-controlled.

AI collapsed the cost of code. Low-code’s historical advantage was access for people who can’t code. AI-assisted coding erased that. Code is now the fast path — generated in seconds, reviewed in minutes, diffable.
Determinism is a feature. You can unit-test, replay, and prove a deterministic pipeline. You cannot prove a graph of LLM calls. In production, “it usually works” is a failure state.
Review and rollback survive. A 200-line diff is reviewable. A 400-node YAML blob is not. AI writes the code; you review the diff. That combination is the whole game.
You control the whole stack. Containers, CI, secrets, observability, backup, upgrades. Code-first gives you the levers; the canvas gives you a vendor roadmap — which in 2026 is literally “embedded in SAP’s agent platform” or “not” [2][3].
Complexity doesn’t scale linearly. A well-factored 300-line module covering 15 integrations is maintainable. 150 nodes is not.

Low-code in the AI era isn’t “more automation.” It’s more hidden nondeterminism with a bigger funding round.
Where low-code still makes sense

Throwaway prototypes and MVPs. Weeks, not years. Validate, then rewrite in code.
Trivial one-offs. “Email → Slack,” “CSV from A to B.” No state, low risk.
Business-owned, low-risk internal flows. When the business genuinely owns the maintenance and the flow isn’t regulated or high-volume.
When visual authoring is the feature. Some process changes are inherently visual — and the 2010 wave proved that modeling (BPMN) survives even when implementation doesn’t.

Rule of thumb: low complexity + low risk + short-lived = fine on a canvas. The moment you need state, idempotency, scale, security, testability, review, or portability — the decision flips to code, and in the AI era that flip comes sooner, because AI made the code path cheaper, not more expensive.
The controversial part
The same buyers who bought the 2010 SOA/iPaaS dream are buying the 2026 dream in the same clothes with a new logo: end the developer bottleneck, visually. The dynamic hasn’t changed — abstraction doesn’t remove complexity, it hides it. In 2010 the hidden complexity became a maintenance liability. In 2026 it becomes hidden nondeterminism inside an LLM — which is exactly where outages, runaway token costs, and compliance breaches breed.
And the “is it cooling?” question deserves a straight answer: the low-code canvas market isn’t cooling — it’s at its peak, worth billions, embedded in SAP [2][3]. What’s cooling is the agentic AI promise the whole category rebranded itself around, and Gartner says that trough hits in 2026 [7][8]. So the risk isn’t that low-code automation dies. The risk is that it survives on top of a hype cycle entering its disillusionment phase, leaving every organization with hundreds of LLM-node graphs that can’t be explained, tested, audited, or trusted — and a platform roadmap deciding what they can and can’t do.
Code-first isn’t developer ego. It’s the only lever that doesn’t get linearly more expensive as complexity grows, and the only one that survives a trough. AI didn’t change that — AI made the code path cheaper and the canvas path more dangerous, in the same year.
We saw the 2010 ending in 2013. The 2026 ending won’t be a shrinking market. It’ll be a peak-sized market full of graphs nobody can debug.

Sources

n8n Blog, “n8n raises $180M to get AI closer to value with orchestration” (Oct 9, 2025) — $2.5B Series C led by Accel. https://blog.n8n.io/series-c/
AyAutomate, “n8n Hits $5.2B Valuation: Enterprise-Ready in 2026” — SAP strategic stake (May 2026), canvas embedded inside Joule Studio (SAP’s AI agent-building platform). https://www.ayautomate.com/blog/n8n-enterprise-valuation-2026
Wikipedia, “N8n” (accessed Sep 2026) — SAP strategic investment, May 2026, roughly doubling the valuation; founded by Jan Oberhauser, launched Oct 2019. https://en.wikipedia.org/wiki/N8n
GitHub, “n8n-io/n8n” — 195k+ stars; “fair-code platform to build and deploy AI agents and workflows, 1500+ integrations.” https://github.com/n8n-io/n8n
Virtua, “n8n vs Zapier vs Make: Kosten, DSGVO & Datenschutz (2026)” — Zapier Agents with 7,000+ app integrations. https://www.virtua.cloud/learn/de/concepts/n8n-vs-zapier-vs-make-kosten-datenschutz
n8n homepage — “Build visually, go deep with code, connect to anything. Every step of your agents’ reasoning, traceable on the canvas.” https://n8n.io/
Gartner, “2026 Hype Cycle for Agentic AI” (Apr 15, 2026) — first dedicated agentic AI hype cycle; AI agents at peak of inflated expectations, trough of disillusionment predicted throughout 2026. https://www.gartner.com/en/articles/hype-cycle-for-agentic-ai
Generative, Inc., “Agentic AI in 2026: How AI Evolved from Chatbots to Autonomous Agents” (Jul 10, 2026) — 2026 as the “year of disillusionment” for agentic AI. https://www.generative.inc/agentic-ai-in-2026-how-ai-went-from-chatting-to-doing
NoCode.Tech, “Gartner’s 2026 Hype Cycle for Agentic AI: What the Trough of Disillusionment Actually Means for No-Code Teams” (Jul 21, 2026) — 88% AI agent security incident rate. https://www.nocode.tech/article/gartners-2026-hype-cycle-agentic-ai
Financial Express, “Is an AI bust coming?” (Apr 12, 2026) — Microsoft pausing data-center construction, Anthropic delaying a release over safety concerns. https://www.financialexpress.com/market/global-market-pulse/is-an-ai-bust-coming/4204135/
n8n, “Workflows App Automation Features” — “Fall back to code: drop into JavaScript or Python in a code node, add npm packages when you self-host.” https://n8n.io/features/

---

### Upgrading the Local LLM Stack: From Qwen3.6-35B-A3B to Qwen3.8-27B

URL: https://remus-software.org/articles/qwen3.6-to-qwen3.8-upgrade/
Date: August 15, 2026
Description: How a model generation jump changed our Ansible-managed vLLM deployment on 2× RTX 5060 Ti — trading a fast 3B-active MoE for a dense 27B model, quadrupling the served context to the full 262K, and what it means for agent harnesses like Hermes.
Tags: AI, LLM, vLLM, Self-Hosting, Infrastructure

Upgrading the Local LLM Stack: From Qwen3.6-35B-A3B to Qwen3.8-27B
How a model generation jump changed our Ansible-managed vLLM deployment on 2× RTX 5060 Ti — and what it means for agent harnesses like Hermes.

1. What Changed
Our home-lab inference server (2× NVIDIA RTX 5060 Ti 16 GB = 32 GB VRAM, Ryzen 9 9950X, 64 GB DDR5) runs a single Ansible-managed vLLM instance behind an OpenAI-compatible API. Since July 2026 it served cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit — a hybrid MoE model: 35B total parameters but only ~3B active per token, mixing Gated DeltaNet linear attention with full attention, 262K native context (git tag vllm_Qwen3.6-35B-A3B-AWQ-4bit, commit 66ed291).
The new configuration serves Qwen3.8-27B via the community quant philbert440/Qwen3.8-27B-W4A16-AWQ. This is not a routine model bump — the serving economics of the machine changed, and in one important way against us:

Aspect
Old: Qwen3.6-35B-A3B (AWQ)
New: Qwen3.8-27B (W4A16)

Architecture
Hybrid MoE: 40 layers (30 GDN linear + 10 full attn), 256 experts
Hybrid dense: 64 layers (48 GDN linear + 16 full attn)

Parameters
35B total, ~3B active per token
27B — all active per token

Active KV heads
2 KV heads × 10 full-attn layers
4 KV heads × 16 full-attn layers

Native context (model)
262,144
262,144 — same on paper

Context actually served
65,536 — KV didn’t fit for more (see §2)
262,144 — the full native window

Multimodal
Vision-language (native)
Vision-language (native) — unchanged

Speculative decoding
MTP head (multi-token prediction)
MTP head — unchanged

Quantization
AWQ INT4 (awq_marlin flag)
compressed-tensors W4A16 (auto-detected)

Weights on disk
~20 GB
~19.5 GB

vLLM minimum
any 2026 release
≥ 0.17 (Qwen3_5 architecture support)

Tool-call parser
qwen3_coder
qwen3_coder — unchanged

The Ansible changes were small in line count but each one load-bearing: model repo, vllm_quantization: "none" (W4A16 compressed-tensors is auto-detected — passing awq_marlin would break loading), vllm_max_num_seqs: 8 → 4, a vLLM ≥ 0.17 version floor, and a new optional vllm_speculative_config for MTP.
2. The Real Trade: Speed for Intelligence — and 4× the Usable Context
The honest framing of this upgrade has two halves, and the second one is easy to miss if you only read model cards: yes, both models natively support 262K — but the old deployment could never serve it. So the real story is: we trade raw speed for substantially better answers, and we finally unlock the context window both models always advertised.
The context unlock (what actually changed for users). The old config served vllm_max_model_len: 65536 — a quarter of the model’s native window. Not from lack of ambition: the old MoE’s ~20 GB weights left only ~12 GB of KV headroom, and at 64K per sequence with fp8 KV that meant 1–2 full-length sequences before preemption. Raising the limit on paper would have bought nothing the KV pool couldn’t back. The new setup changes both sides of the equation: weights shrink slightly (~19.5 GB), and — decisive — the max_num_batched_tokens/block_size constraint and Mamba-cache behavior that capped the old hybrid model are handled cleanly in vLLM ≥ 0.22 with the W4A16 checkpoint. Result: the served window quadruples to the full 262,144 tokens, with KV for four saturated 262K sequences (~2.1 GiB each at fp8) in a pool of over a million tokens.
What got more expensive. Compute per token scales with active parameters, and that number went up ~9× (3B → 27B). Decode speed will drop correspondingly — the old MoE was one of the fastest things you could run on this hardware; the new dense model is mid-pack. KV cost per token also rose ~3.2× (2 KV heads × 10 layers → 4 × 16), though from such a low base that it barely matters at these pool sizes.
What we get for it. The largest capability jump this machine has seen — see the benchmark table below. Dense 27B at INT4 simply out-reasons a 3B-active MoE, and the 3.8 generation’s gains land exactly on agentic work.
What stays the same. Hybrid GDN attention, MTP speculative decoding, native vision, the qwen3_coder tool parser, fp8 KV cache, and — on paper — the native context length. Only the paper became reality.
The VRAM story inverts — in our favor. The old MoE’s routing forces all 35B parameters (~20 GB quantized) into VRAM even though only 3B run per token — MoE saves compute, not memory. The dense model needs less VRAM for weights (~19.5 GB), and because we keep gpu_memory_utilization: 0.95, the KV pool (~9–10 GiB) backs the full window at four parallel saturated sequences — versus one to two at 64K before. Capacity per gigabyte of VRAM went up roughly 8×.
vLLM’s recipe measurements on 2× RTX 5090 (same sm_120 Blackwell generation as our 5060 Ti) confirm the class: 377K–920K KV-token pools depending on precision, with the in-checkpoint MTP head reaching 0.75–0.90 draft acceptance. The checkpoint we use was independently validated end-to-end on 2× V100-32GB: 55–59 tok/s single-stream with MTP K=2, ~165 tok/s aggregate at 4-way concurrency, 92.5% draft acceptance. MTP (shipped disabled, one variable away) is the lever that wins back a good chunk of the dense-model decode penalty — 2–3× speedup at ~92% acceptance, quality-neutral.
3. Benchmark Comparison
Cross-model-card comparisons need care: Qwen re-evaluates baselines each generation on refined benchmarks, so the two cards’ numbers are not produced in one run — treat the deltas as indicative, not exact. The old deployment ran the 35B-A3B MoE; its card compares against its own generation’s peers, so we place the two official columns side by side and add Qwen3.6-27B (same-size predecessor on the 3.8 card) as the connecting reference.

Benchmark
New: Qwen3.8-27B (dense)
Qwen3.6-27B (dense, 3.8 card)
Old: Qwen3.6-35B-A3B (MoE, 3.6 card)

Terminal coding
73.0 (Terminal Bench 2.1)
63.4 (TB 2.1)
51.5 (Terminal-Bench 2.0)

SWE-bench Pro (agentic coding)
61.7
53.5
49.5

Repo-level codegen
42.3 (NL2Repo-Bench)
36.2 (NL2Repo-Bench)
29.4 (NL2Repo)

Agentic SWE (in-house)
79.0 (QwenSWEBench)
49.3
—

DeepSWE 1.1
42.2
13.3
—

Long-horizon work
70.7 (CoWorkBench)
61.0
—

Instruction following
79.5 (IFBench)
69.1
—

Scientific reasoning
89.2 (GPQA Diamond)
87.8
86.0 (GPQA)

Multidisciplinary reasoning
30.8 (HLE)
24.0
21.4 (HLE)

Competitive coding
90.3 (LiveCodeBench v6)
83.9
80.4 (LCB v6)

Tool/agent use
68.7-class gains across the board
—
68.7 (Claw-Eval Avg)

Sources: Qwen3.8-27B model card (August 2026, Claude Code harness, temp=1.0, top_p=0.95, 256K context) and Qwen3.6-35B-A3B model card (internal agent scaffold, temp=1.0, top_p=0.95, 200–256K context). “—” = benchmark not reported on that card.
On vision-language tasks the old model was no slouch (RealWorldQA 84.1–85.3, OmniDocBench 89.3–89.9 on its card) — vision is not new in this upgrade. What is new is agentic vision: Qwen3.8-27B scores 84.3 on OSWorld-Verified (computer use) vs 63.9 for Qwen3.6-27B, 64.8 WebArena-Verified (browser use) vs 48.8, and 81.9 AndroidWorld vs 70.3 — the ability to act on screenshots, not just describe them (3.8 card, “VL Performance”; 3.6 card, vision tables).
The pattern: knowledge benchmarks barely move (GPQA 86.0 → 89.2), but everything an agent harness touches jumps by 10–30 points — terminal work, repo-level coding, long-horizon tasks, and computer/browser use. This model was built for doing, not reciting.
4. Implications for Agent Harnesses
This is where the upgrade stops being a benchmark story and becomes an operational one. We run Hermes Agent against this server; the same points apply to any OpenAI-compatible harness (OpenCode, Aider, Claude-Code-style loops, LangChain).
a) Fewer derailments per long task. Terminal Bench 51.5 → 73.0-class and repo-codegen 29.4 → 42.3 mean the practical thing: agent loops that used to wander off after step N now hold together longer. Expect to raise — not lower — the autonomy you grant the harness (longer unstopped runs, bigger multi-file tasks).
b) Full 262K context is now actually usable — before, it wasn’t. The old deployment capped sessions at 64K: its KV budget allowed only 1–2 full-length sequences, and every longer agent conversation silently relied on truncation and summarization to survive. The new KV pool holds four saturated 262K sequences. Harness patterns built around scarcity — aggressive history truncation, summarization checkpoints, retrieval stitching — can be relaxed. And it’s native context on both models, so no YaRN quality tax anywhere.
c) Hermes-side config must move with the window. If your harness has a context_length-style setting (Hermes: model.context_length, which also drives its compression triggers), raise it from 64K to 262144 to match the server — otherwise the harness will keep compressing conversations at one quarter of what the server could hold. Set the compression trigger high (e.g. 0.75) so the window is actually used before the lossy summary pass fires.
d) Budget more latency per token. Dense 27B decodes roughly like a classic 27B — noticeably slower than the 3B-active MoE. Interactive chat feels this most. Two mitigations, both already in the config: enable MTP (vllm_speculative_config) when a workload is decode-bound, and keep thinking-mode control per request (below) so trivial calls don’t pay full reasoning price.
e) Tool calling stays on qwen3_coder — and the reasoning parser stays mandatory. Both models speak the Qwen3.5+ tool format, so no parser change was needed this time (the hermes parser belongs to the even older Qwen3 dense generation). --reasoning-parser qwen3 remains non-optional: Qwen3.8 opens every assistant turn with <think>; without the parser, the entire reasoning block lands in message.content, where it pollutes downstream parsing and can burn a whole token budget before the answer starts. With it, reasoning lands cleanly in reasoning_content.
f) Thinking control is per-request, and harnesses should use it. Thinking is on by default. chat_template_kwargs: {"enable_thinking": false} gives fast direct answers for trivial calls (classification, formatting); {"reasoning_effort": "low|medium|xhigh"} tunes depth; preserve_thinking retains reasoning across turns for multi-step tasks — Qwen notes this can reduce total token consumption in agent loops by avoiding redundant re-reasoning. A harness that hardcodes one mode leaves capability on the table. Official sampling: thinking = temp 1.0 / top_p 0.95 / top_k 20; non-thinking = temp 0.7 / top_p 0.80 / presence_penalty 1.5.
g) Vision moves from “can read” to “can operate”. Both models see; the new one acts — OSWorld 84.3 is computer-use territory. Screenshot-driven debugging, UI automation, and PDF/diagram workflows through the same endpoint become realistic harness tools. Caveat: the vision tower (BF16) shares the VRAM budget, and vLLM’s recipe notes --language-model-only as a lever to reclaim it if vision isn’t needed.
h) MTP speculative decoding: same lever, new calculus. Both models ship an MTP head, but it matters more now — it’s the cheapest way to offset the dense decode penalty (~2–3× at ~92% acceptance, quality-neutral since verification is against the full model). We still ship it disabled by default: its BF16 head costs ~0.85 GiB VRAM, and at maximum-context-plus-concurrency the KV pool is worth more. Flip one Ansible variable when decode speed matters more than concurrent long contexts. It needs no CUDA graphs, so it composes fine with enforce_eager.
i) Watch your served-model name. vLLM’s --served-model-name is the repo basename (Qwen3.8-27B-W4A16-AWQ), so harness configs must reference that string — the Hermes hint is documented in the README (provider: openai-api, api_base: http://<host>:8001/v1, api_key: EMPTY).
j) The honest cost, amplified: prefill latency at long context. A 200K-token prompt streams through chunked prefill at 4,096 tokens per chunk on two consumer GPUs — expect minutes of TTFT before the first token, on either model. That’s physics, not misconfiguration. What changed is that the KV budget now lets those long prompts actually finish instead of being preempted — but every giant prompt now costs real wall-clock time. Harnesses should still avoid pointless prompt bloat; the difference is that useful bloat (whole repos, full histories) is now viable.
5. Deployment Notes (Blackwell sm_120 specifics)
The upgrade inherits several hard-won workarounds, all preserved in the Ansible role:

TRITON_ATTN: FLASH_ATTN is broken on sm_120 (undefined symbol); FlashInfer’s JIT arch check fails on Blackwell — the role actively uninstalls it so vLLM falls back to Triton attention and the native sampler. (vLLM 0.22 logs the old VLLM_ATTENTION_BACKEND env var as unknown but still honors the resulting selection — cosmetic.)
enforce_eager: CUDA graphs + hybrid attention + tensor parallelism over PCIe (GeForce has no NVLink/P2P) deadlocked workers after long generations. Eager mode costs some decode speed but is stable; MTP is the sanctioned way to win it back.
NCCL_P2P_DISABLE=1: no working PCIe P2P on GeForce pairs; transfers go over shared memory. vLLM 0.22 confirms by falling back to the PYNCCL all-reduce backend on its own.
vLLM ≥ 0.17 is now enforced at install time — the Qwen3_5 architecture doesn’t exist in older releases.
Prefix caching + Mamba ‘align’ mode: vLLM warns this combination is experimental for the GatedDeltaNet layers. If repeated-prefix requests ever return wrong outputs, disabling prefix caching is the workaround; so far it behaves.

6. The Configuration Is Open Source
Everything in this article is reproducible from public sources. The full Ansible configuration — the vllm role, the group vars holding every knob mentioned above (vllm_max_num_seqs, vllm_max_model_len, gpu_memory_utilization, vllm_speculative_config, the Blackwell workarounds), and the model-switch history with git tags — is published under the MIT license at github.com/tmseidel/local-llm-setup. It targets exactly this hardware (2× RTX 5060 Ti 16 GB on Ubuntu 24.04), runs one inference engine at a time (vLLM or Ollama) controlled by a single inference_engine variable so the active engine gets the full 32 GB VRAM, and includes systemd watchdogs that restart vLLM on failure. The latest release pins the configuration to the Qwen3.8-27B-W4A16-AWQ setup described in this article; earlier tags cover the previous generations (Qwen3.6-35B-A3B-AWQ-4bit, Qwen3-32B-AWQ), so the whole upgrade path is auditable in the git history.
7. Verdict
This upgrade is a deliberate speed-for-intelligence trade, not a free lunch — plus one genuinely free win. The old MoE was the faster chatbot; the new dense model is the better colleague. We give up roughly a 9× compute-efficiency advantage per token and get back benchmark gains of 10–30 points concentrated exactly where an agent harness lives: terminal work, repo-level coding, long-horizon tasks, and acting on what it sees. Vision, MTP, and the tool-calling stack carry over untouched. And the free win: the context window both models always advertised finally became real — the old deployment capped at 64K because its KV pool couldn’t back more, while the new one serves the full 262K with four saturated long contexts in flight, roughly an 8× capacity gain per gigabyte of VRAM. The infrastructure changes were modest because the Ansible role was already parameterized — but the defaults now encode real lessons: auto-detected quantization, a vLLM version floor, and MTP as an explicit speed-vs-memory toggle.

Glossary
AI- and serving-specific terms used in this article, in order of appearance. Where a term affects what you actually see as output, a → In practice note describes the impact on an agent run or a simple chat.
MoE (Mixture-of-Experts) / active parameters — An MoE model contains many parallel “expert” sub-networks (Qwen3.6-35B-A3B: 256) but routes each token through only a few (8 routed + 1 shared ≈ 3B of 35B parameters). This makes each token cheap to compute while the full knowledge of all 35B stays available.
→ In practice: MoE = fast tokens, sometimes uneven quality on unusual prompts where routing misfires. Crucially for self-hosters: MoE saves compute, not VRAM — all experts must be resident, so a “35B-A3B” needs the memory of a 35B model while performing like a much smaller one per token.
Dense model — A transformer where every parameter participates in every token prediction. Qwen3.8-27B is dense: all 27B parameters work on every token, which makes VRAM requirements predictable and per-token quality uniform.
→ In practice: slower per token than an MoE of similar total size, but more consistent answers — and it’s why this upgrade trades speed for reasoning depth.
Transformer / attention — The standard LLM architecture. “Attention” lets each token look back at all previous tokens; the data it stores per past token is the KV cache (below). “Full attention” = every layer does this over the whole context.
→ In practice: this is why the model can reference something you said 100 messages ago — and why that ability costs VRAM.
Hybrid attention / Gated DeltaNet / linear attention — The backbone both models share. Instead of full attention in every layer, most layers use Gated DeltaNet, a linear-attention variant that compresses the entire past into a fixed-size recurrent state (like an RNN with a memory slot) instead of storing every past token. Result: near-constant memory per token for those layers. Only a minority of layers keep classic full attention for precise long-range recall.
→ In practice: you get a 262K window on consumer hardware. Trade-off: exact recall of one specific line buried 200K tokens deep is slightly less reliable than with all-full-attention models — the few full-attention layers carry that load. For agents reading whole repos, recall is still strong, but quote-critical tasks benefit from re-including the relevant snippet in the prompt.
GQA (Grouped-Query Attention) — A memory optimization where multiple query heads share the same key/value heads. Qwen3.8-27B uses 24 query heads but only 4 KV heads (the old MoE: 16 Q / 2 KV), cutting KV-cache size severalfold versus classic multi-head attention.
→ In practice: invisible in output quality (measurable but negligible quality delta); visible in your bill of materials — it’s a big part of why 262K context fits at all.
KV cache — The stored keys and values of all previous tokens, required so the model doesn’t recompute the whole conversation for every new token. It lives in VRAM during generation and is usually the dominant memory cost of long contexts. Measured in MiB per token.
→ In practice: when the KV pool is exhausted, vLLM preempts or rejects requests — visible as stalled generations or errors under load, never as degraded text. Chat quality does not fade with a full cache; throughput does.
Context length / context window — The maximum number of tokens (input + output) the model can consider at once. Native context is what the model was trained for; anything beyond it requires position-encoding tricks (see YaRN). Both models here: 262,144 native — but note the difference between what a model supports and what a deployment serves: the old config capped at 65,536 because its KV pool couldn’t back more.
→ In practice: everything inside the window can influence the answer; anything outside effectively never happened. For an agent, a bigger window means fewer “it forgot the earlier error message” loops; for chat, fewer mid-conversation amnesia moments. Exceeding the limit is a hard API error (400), not a silent truncation. And check your server’s limit, not the model card’s — they routinely differ.
Token — The model’s unit of text: roughly ¾ of an English word, or 3–4 characters. 262K tokens ≈ a 600+ page book or a mid-size codebase.
→ In practice: the unit your latency, memory, and truncation bugs are made of. max_tokens limits output length — too tight and thinking models burn it all on reasoning and return an empty-looking answer.
RoPE / YaRN / RoPE scaling — Rotary Position Embeddings encode token positions. YaRN is a scaling method that stretches a model beyond its trained context length by rescaling those positions. It works, but applying a static stretch factor also slightly degrades quality at short contexts — which is why a natively long model is preferable. Neither model in this article needs it below 262K.
→ In practice: if you ever enable YaRN (e.g. toward 1M tokens), expect short everyday questions to get subtly worse in exchange for the extra range. Below the native limit, leave it off.
Quantization (AWQ, W4A16, INT4, FP8) — Storing model weights in fewer bits to save VRAM. AWQ (Activation-aware Weight Quantization) compresses weights to 4-bit integers while keeping accuracy. W4A16 = 4-bit weights, 16-bit activations (compute stays high-precision). compressed-tensors is vLLM’s native container format for such quants. FP8 = 8-bit floating point, natively fast on Blackwell GPUs. Quality loss from a good 4-bit weight quant is small; it is the standard way to fit a 27B model into 2×16 GB.
→ In practice: a well-made W4A16 quant answers within ~1% of the full model on benchmarks — you won’t notice in chat. A badly calibrated quant shows up as broken think-tags, repetition loops, or odd refusals; that’s why the calibration story on the model card matters. The fp8 KV cache variant is even safer: it affects memory, not weights.
Marlin kernel (awq_marlin) — The GPU compute kernel vLLM uses to run 4-bit quantized matrix multiplications fast on NVIDIA cards. “Kernel” here means the low-level math routine, not the OS kernel. (The startup log confirms: “Using MarlinLinearKernel for CompressedTensorsWNA16”.)
→ In practice: pure speed, no quality impact. Wrong kernel choice = slower tokens, not wrong tokens.
vLLM / PagedAttention / continuous batching — vLLM is the inference server. Its two core tricks: PagedAttention manages the KV cache in small blocks (like OS virtual memory) to avoid waste, and continuous batching mixes prefill and decode steps of many requests so the GPUs stay busy.
→ In practice: multiple agent sessions share the server without each waiting for the other to finish; individual answers are identical to single-user mode, they just arrive concurrently.
TP / PP (tensor / pipeline parallelism) — Ways to split one model across multiple GPUs. Tensor parallelism (TP) shards every layer across both cards — both always busy, but they exchange data each layer (needs fast interconnect). Pipeline parallelism (PP) assigns half the layers to each card — less chatter, but one card idles between stages at low request counts. We use TP=2.
→ In practice: no effect on answer content — outputs are bit-comparable either way. It only decides your tokens/sec, and at 1–4 concurrent requests (typical agent use) TP=2 is the faster choice.
NVLink / P2P / NCCL / PYNCCL — NVLink is NVIDIA’s fast GPU-to-GPU interconnect (absent on GeForce cards). P2P = direct GPU-to-GPU memory transfer over PCIe, also broken on GeForce. NCCL is NVIDIA’s multi-GPU communication library; NCCL_P2P_DISABLE=1 forces it to route transfers through host RAM instead — vLLM 0.22 then selects the PYNCCL fallback backend by itself.
→ In practice: with P2P wrongly enabled on this hardware, multi-GPU runs hang mid-generation or crawl ~4× slower — an agent that mysteriously stalls after several minutes. With the flag set, output is correct and speed is normal.
Prefill / decode / TTFT — The two phases of a request. Prefill processes your whole prompt at once (parallel, compute-heavy). Decode generates the answer one token at a time (sequential, bandwidth-bound). TTFT = time to first token, dominated by prefill on long prompts.
→ In practice: a 200K-token agent prompt can sit silent for minutes before the first token appears — the answer will be fine, the wait is the cost. Short chats see sub-second TTFT. Streaming doesn’t shorten prefill; it only makes the wait visible earlier.
Chunked prefill — Splitting a long prompt’s prefill into smaller chunks (here 4,096 tokens) so ongoing decode requests of other users aren’t blocked behind one huge prompt.
→ In practice: your chat stays responsive while an agent chews through a giant prompt in the background; without it, one big request freezes everyone else until its prefill completes.
Prefix caching — Reusing the KV cache of a previously-seen prompt prefix (system prompt, tool definitions) instead of recomputing it. Nearly free TTFT savings for agent workloads that resend the same header every turn. On hybrid (Mamba/GDN) models vLLM enables it in an experimental “align” mode.
→ In practice: the second and subsequent turns of an agent loop start answering noticeably faster. Identical output — this is a cache, not a shortcut in reasoning. (If outputs ever go wrong specifically on repeated prefixes, this experimental mode is the first suspect.)
Speculative decoding / MTP / draft head / acceptance rate — A speed trick: a small “draft” mechanism proposes several tokens ahead, and the full model verifies them in one parallel pass — accepted tokens cost a fraction of normal decode. MTP (Multi-Token Prediction) means the draft capability is trained into the model itself as an extra head, so draft quality is high. Acceptance rate = fraction of drafted tokens the full model agrees with; ~92% at K=2 means most speculation succeeds, giving 2–3× decode speedup with zero quality change (verification is against the full model).
→ In practice: enabling MTP changes speed only, never content — rejected drafts are simply regenerated by the full model. For chat, answers stream 2–3× faster; for agents, long code edits complete proportionally sooner. The cost is the VRAM the draft head occupies, which is why it’s off in our max-context config.
CUDA graphs / enforce-eager — CUDA graphs record a sequence of GPU operations once and replay them with near-zero CPU overhead — faster decode. Enforce-eager disables this, running every op individually: slower but more robust. Required here because CUDA graphs + hybrid attention + PCIe tensor-parallelism deadlocked on Blackwell.
→ In practice: eager mode costs maybe 10–20% decode speed; the alternative on this stack was generations dying mid-answer after long runs. Slower-and-finished beats faster-and-hung.
Attention backend (TRITON_ATTN, FLASH_ATTN, FlashInfer) — The software implementation of the attention math. FlashAttention is the famous fast one but is broken for LLM attention on RTX 50-series (sm_120); FlashInfer needs a JIT toolchain that also fails there. Triton (OpenAI’s GPU programming language) compiles working kernels on the fly — our fallback. (The vision tower uses its own FLASH_ATTN path, which is fine — the sm_120 breakage concerns the LLM attention.)
→ In practice: the backend changes speed and whether the server starts at all, never what the model says. A broken backend here means a dead service, not a degraded one.
sm_120 / Blackwell — NVIDIA’s RTX 50-series GPU generation (2025) and its compute-capability number. New silicon often breaks older GPU kernels, which is why backend choice matters.
→ In practice: explains most “it worked on the old card” mysteries: failures manifest as startup crashes or hangs, not subtly wrong answers.
Tool calling / function calling / tool-call parser — The model emitting structured JSON requests (“call get_weather with city=Berlin”) instead of plain text, so a harness can execute real actions. Each model family formats these differently; the parser converts raw output into structured tool_calls. Wrong parser = calls silently arrive as text and never execute. Qwen3.5+ (including 3.6 and 3.8) needs qwen3_coder.
→ In practice: this is the difference between an agent that does things and one that narrates things. With the wrong parser, the model’s reply looks like a tool call in the chat text but nothing executes — the classic “agent talks about editing the file but the file never changes” bug. Simple chat without tools is unaffected.
Reasoning parser / thinking mode / <think> / reasoning_content / preserve_thinking — Qwen3.8 “thinks” before answering, writing its chain-of-thought inside <think>…</think> tags. The reasoning parser splits that into a separate API field (reasoning_content), keeping the actual answer (content) clean. preserve_thinking keeps prior turns’ reasoning in the conversation, which Qwen notes can reduce total token use in agent loops by avoiding redundant re-reasoning.
→ In practice: without the parser, a max_tokens=2048 request can return 2048 tokens of reasoning and zero answer — and downstream JSON parsers choke on the <think> text. With it, you get a clean answer plus optional visibility into the reasoning. Disabling thinking per request makes trivial chats much faster and cheaper at some cost to hard-problem accuracy; leaving it on for agentic work is usually right.
Vision-language model (VLM) / vision tower — A model that accepts images/video as input, not just text. The vision tower is the encoder sub-network that converts pixels into tokens the language model can reason over. Both models in this article are VLMs; the 3.8 generation adds strong agentic vision (acting on screenshots, not just describing them).
→ In practice: “what’s wrong in this screenshot”, “click the button in this UI”, “read this scanned PDF” — through the same chat endpoint. High-resolution images consume many tokens each, so image-heavy chats eat the context window faster than text alone.
Harness — The agent framework that drives the model: sends prompts, parses tool calls, executes actions, feeds results back (Hermes Agent, Claude Code, OpenCode, Aider). The model cards’ agentic benchmarks were measured with specific harnesses (Claude Code for 3.8; an internal scaffold for 3.6) — same model, different harness, different score.
→ In practice: the same model can feel brilliant in one harness and mediocre in another — system prompts, tool schemas, and retry logic shape outcomes as much as weights do. When comparing your local results to published benchmarks, match the harness and settings before blaming the quant.
Benchmarks named in the tables — Terminal Bench: completing real tasks in a Linux terminal (2.0 vs 2.1 are different editions — scores not directly comparable). SWE-bench (Verified/Pro/Multilingual): fixing real GitHub issues. GPQA (Diamond): PhD-level science questions. HLE: “Humanity’s Last Exam”, deliberately frontier-hard. LiveCodeBench: competitive programming, contamination-free. IFBench: instruction following. OSWorld / WebArena / AndroidWorld: operating real desktops, browsers, and phones via screenshots + actions. Claw-Eval / MCPMark / Tool Decathlon / TAU3-Bench: tool-use and agent suites. NL2Repo(-Bench): generating code at whole-repository scale. GSM8K: grade-school math (sanity check). CoWorkBench / JobBench / DeepSWE / QwenSWEBench / Agents’ Last Exam / SWE-MM / RealWorldQA / OmniDocBench: agentic, work, and vision suites — see the linked model cards for definitions.
→ In practice: read benchmarks as predictors, not guarantees: high Terminal Bench/SWE-bench scores predict an agent that completes multi-step coding tasks without derailing; GPQA/HLE predict answer quality on hard science Q&A; IFBench predicts whether your formatting instructions (“respond in JSON only”) are obeyed — the one simple-chat users notice most. And never compare scores across different benchmark editions (Terminal Bench 2.0 ≠ 2.1) as if they were one scale.
VRAM headroom / gpu_memory_utilization — vLLM pre-allocates a fixed fraction of each GPU (here 95%) for weights + KV cache. Whatever remains after weights is the KV pool — its size in tokens is printed at startup and is the real ceiling on concurrent long contexts.
→ In practice: set too low, long agent contexts get preempted (visible as retries/stalls under load); set too high, the server OOMs at startup or during CUDA graph capture. It tunes capacity, never answer quality.
CPU offload — Moving some model weights into system RAM (the 64 GB DDR5) to free VRAM. Works, but weights then stream over PCIe on every forward pass — noticeably slower. A fallback lever, not a default.
→ In practice: identical answers at a fraction of the speed — an emergency lever to make an oversized model run at all, not a tuning knob for daily use.

Sources

Qwen/Qwen3.8-27B — Hugging Face model card — architecture (64 layers, 48 GDN + 16 full attn, 4 KV heads, dense 27B), benchmark tables (text + VL), thinking control (preserve_thinking, reasoning_effort), sampling recommendations.
Qwen/Qwen3.6-35B-A3B — Hugging Face model card — previous model: 35B total / 3B active MoE (256 experts, 8+1 active), 40 layers (30 GDN + 10 full attn, 2 KV heads), 262K native context, benchmark tables, harness/eval conditions.
philbert440/Qwen3.8-27B-W4A16-AWQ — Hugging Face — quantization details (W4A16 g128, 19.5 GB, vision/MTP intact), measured throughput and MTP acceptance on 2×V100-32GB, validation results.
Qwen/Qwen3.8-27B — vLLM Recipes — verified serving configs on consumer Blackwell (2× RTX 5090, sm_120), KV pool measurements per precision, MTP acceptance figures, --reasoning-parser qwen3 and qwen3_coder parser guidance, vLLM ≥ 0.17 requirement, 1M-context YaRN override.
Local git history + Ansible configuration: commit 66ed291 “feat: switch to Qwen3.6-35B-A3B-AWQ-4bit”, tags vllm_Qwen3-32B-AWQ / vllm_Qwen3.6-35B-A3B-AWQ-4bit, ansible/group_vars/all.yml, ansible/roles/vllm/ — deployed parameters, Blackwell workarounds, KV sizing math, startup log (vLLM 0.22.0: MarlinLinearKernel, TRITON_ATTN, PYNCCL fallback).
tmseidel/local-llm-setup — GitHub — the complete open-source Ansible configuration (MIT license): the vLLM role, group vars, model-switching git history and releases, Blackwell workarounds, systemd watchdogs.

---

### AI-Git-Bot 1.18 — Audit trails, SSO and the road into your organization

URL: https://remus-software.org/articles/ai-git-bot-1-18-audit-trails-sso/
Date: July 26, 2026
Description: AI-Git-Bot 1.18 is the release where the bot grows up: OAuth2/OIDC login, a tamper-evident audit trail, outgoing webhooks for your SIEM, Prometheus metrics, deterministic review gates — and a one-command docker run to try it all. Not new AI tricks, but the features that let an AI teammate live inside a real company.
Tags: software-development, AI, DevOps, news, Git

Imagine your CISO asking: “This AI bot that reviews our code — who can administer it, what exactly did it do last Tuesday, and can we prove the record wasn’t touched?”
With AI-Git-Bot 1.18, you have an answer. Shipping today. 🚀

First time here? 30-second intro 👋
AI-Git-Bot is the open-source AI teammate that lives inside your Git tool — Gitea, GitHub, GitHub Enterprise, GitLab, Bitbucket Cloud. It reviews PRs, writes tests, turns issues into pull requests, keeps docs and translations in sync, and runs E2E suites against preview deployments. Self-hosted, provider-independent (OpenAI, Claude, Gemini, or local Ollama models), and MIT-licensed.
You assign it work the same way you’d assign it to a colleague: request it as a PR reviewer, assign it an issue, or @mention it in a comment.

👉 Want the long-form pitch? Read doc/pitch/PITCH.md — it’s the fastest way to decide whether AI-Git-Bot is for your team.

What’s new in 1.18 — the highlights
This release is different. Not one feature makes the AI smarter. Instead, 1.18 answers the questions organizations ask before they let software anywhere near their source code. Let’s walk through them.
🔐 1. OAuth2/OIDC login — meet your identity provider
The admin UI used to have its own little user database with username/password login. Fine for a homelab — a non-starter where credentials must come from a central IdP.
Now the login method is a configuration switch:

GITEABOT_SECURITY_LOGIN_METHOD=native — the existing database login, unchanged. Current installations keep working exactly as before.
GITEABOT_SECURITY_LOGIN_METHOD=oauth — the entire UI authenticates via OAuth 2.0 Authorization Code flow against your provider.

OIDC issuer discovery is built in, so Microsoft Entra ID and Keycloak work by pointing at a single issuer URL — no manual endpoint wiring.
Why your org will love it: the bot joins your existing SSO landscape. Admin onboarding and offboarding happen in your IdP, access follows your central policies, and there is no shadow user database left to audit.
📜 2. Tamper-evident audit trail — “what exactly happened?”
This is the headline. AI in the development workflow raises a legitimate question from security and compliance teams: what did the bot actually do — and can we prove it later?
1.18 introduces a structured, append-only audit trail for every PR workflow execution:

Full lifecycle coverage — run started, every step, run completed/failed/cancelled, every review decision
Agent tool-call tracing — every tool the LLM invokes, with sanitized arguments, result summary and duration. You can reconstruct the agent’s decision path step by step
Cost correlation — events carry token usage and link to AI usage sessions, connecting what happened to what it cost
Tamper evidence — every event is chained to its predecessor with a per-run SHA-256 hash chain. Insert, alter or delete a record and the chain breaks. The history isn’t just stored; it’s provably intact
Configurable retention — a nightly garbage collector prunes events older than your window (default 90 days, e.g. AUDIT_RETENTION=P180D) to match your data-retention policies

And it never gets in the way: auditing is fire-and-forget, it cannot block or fail the workflow it observes.
Why your org will love it: this turns “we use an AI bot” into “we can govern an AI bot.” Internal audits, ISO 27001 / SOC 2 style assessments, post-incident reviews — they all start with show us the record. Now there is one, and it’s tamper-evident by construction.
📡 3. Outgoing webhooks — the bot talks to your systems
Until now, AI-Git-Bot only received webhooks from Git platforms. It now also emits them. Admins configure endpoints under System settings → Outgoing webhooks and subscribe each one to the events that matter — workflow started/completed/failed, agent-review findings with their severity, issue-assignment lifecycles for the CodingBot and WriterBot.

Built for enterprise plumbing:

Durable, at-least-once delivery — persisted before dispatch, retries with exponential backoff, survives restarts
HMAC-SHA256 signing per endpoint (GitHub-style), plus optional Authorization and custom headers
Scoped subscriptions — global, per-bot, or per-repository
Encrypted secrets at rest with the same AES-GCM encryption that protects AI provider keys
Delivery log with manual retry in the admin UI

Why your org will love it: forward findings to your SIEM, open a ticket on blocker-severity findings, post to Slack/Teams when a workflow fails, feed a custom compliance archive — in real time, without polling. This is the integration point your platform team has been asking for.
📊 4. Prometheus metrics — operational visibility
Observability is table stakes for production. The bot now exposes a standard scrape endpoint at /actuator/prometheus: review and finding counts, workflow run counts and durations, LLM statistics (tool rounds, tool calls, token usage), errors and failures — all with deliberately low-cardinality labels so they behave in real Prometheus deployments.
Why your org will love it: the bot shows up in your existing Grafana dashboards and alert rules next to everything else you operate. Token-spend anomalies and workflow failure spikes become visible before users report them.
⚖️ 5. Deterministic review gates — the LLM advises, the system decides
When the agentic PR review was allowed to submit formal decisions (approve / request changes), the LLM made the final call itself. For a merge gate, a probabilistic decider isn’t good enough.
The decision logic has been inverted:

The LLM only classifies findings by severity — BLOCKER, MEDIUM, LOW — and reports counts
The application computes the review action deterministically against thresholds you configure (e.g. approve only if BLOCKER ≤ 0)
No thresholds configured? The bot posts its review and leaves the review state untouched

Same model, same findings — but the approve/reject decision is now a reproducible function of explicit policy, not of model mood.
Why your org will love it: you can put an AI review in the merge-blocking path and explain the decision rule to an auditor in one sentence. Policy lives in configuration, not in a prompt.
🐳 6. One-command evaluation — curious to running in seconds
Finally, we removed the last evaluation friction. No checkout, no Docker Compose needed for a first look:
docker run -p 8080:8080 tmseidel/ai-git-bot:latest

The container starts with sensible defaults and an embedded H2 database. Docker Compose with PostgreSQL remains the production path — but the first five minutes with the product should never be harder than one command.

The bigger picture: maturity is a feature
Look at the shape of this release:

Who can administer it? — your IdP says so (OAuth2/OIDC)
What did it do? — a tamper-evident audit trail says so
How do our systems learn about it? — outgoing webhooks say so
How do we operate it? — Prometheus metrics say so
Can we trust its gates? — deterministic thresholds say so

Building auditing is a deliberate statement. AI-Git-Bot is not an experiment that happens to run in a company — it is a platform designed to be integrated into one. Self-hosted and provider-independent from day one, and now auditable, observable and SSO-enabled, it fits environments where compliance and security aren’t afterthoughts but entry requirements.
And since evaluation is a single docker run away, the distance between “interesting project” and “running under our own policies” has never been shorter.

Get it now
docker run -p 8080:8080 tmseidel/ai-git-bot:latest

⭐ Star us on GitHub: https://github.com/tmseidel/ai-git-bot
📖 Docs: User Guide · Audit Trail & PR Workflows · Outgoing Webhooks · Deployment
🐛 Found something? Open an issue — we read every one.

The bottom line: in 1.18, AI-Git-Bot stops being just the helpful AI teammate and becomes one your security, compliance and platform teams can sign off on. The chores still get done — every PR, forever — but now with SSO, an audit trail, real integrations and deterministic gates around them.
Happy shipping. 🚀

---

### AI-Git-Bot vs Copilot, GitLab Duo, Qodo and OpenHands — Which Problem Are You Actually Trying To Solve?

URL: https://remus-software.org/articles/ai-git-bot-vs-copilot-gitlab-duo-qodo-openhands/
Date: July 11, 2026
Description: An honest comparison of AI-Git-Bot, GitHub Copilot, GitLab Duo, Qodo and OpenHands. Not by features, but by the engineering problems they were built to solve.
Tags: AI, software-development, DevOps, Git, code-review, productivity

Copilot, GitLab Duo, Qodo, OpenHands or AI-Git-Bot? Picking the right AI teammate for your engineering bottleneck

At least once a week someone asks me a variation of the same question: “How does AI-Git-Bot compare to Copilot?” Sometimes it’s GitLab Duo, sometimes Qodo, sometimes OpenHands. On the surface that’s a perfectly reasonable question since they all involve AI, repositories, pull requests and software development. But after building AI-Git-Bot for a while, I’ve become convinced that most of these tools aren’t really competing with each other, or at least not in the way many people think.

Imagine a team working on a fairly ordinary feature. A developer writes code with Copilot. A review assistant checks the pull request. A coding agent picks up a backlog item and opens a PR. A test generator creates additional coverage. A workflow bot deploys a preview environment and validates the change. All of these tools participated in shipping the feature, yet none of them solved the same problem.
This is what makes AI tooling surprisingly difficult to compare. Most discussions focus on products, while engineering teams usually care about bottlenecks. The question is rarely “Which AI tool is best?” The more useful question is: “Which part of our software delivery process is slowing us down?” That’s the question I’ll try to answer in this article. Rather than declaring winners and losers, I’ll look at where each tool is strongest, where it falls short, and which problem I would personally reach for it to solve.

Short on time? Here’s the 60-second answer ⏱️
If you’re evaluating AI tooling and just want a recommendation, here’s a quick guide based on your primary goal:

Your primary goal
Start with

Help developers write code faster
GitHub Copilot

Add AI capabilities throughout GitLab
GitLab Duo

Improve review quality and test coverage
Qodo

Experiment with autonomous software engineering
OpenHands

Automate Git workflows across multiple Git platforms
AI-Git-Bot

The one-sentence summary is this: most AI developer tools help engineers write code, while AI-Git-Bot helps teams automate the necessary-but-uncomfortable work surrounding software delivery. That difference matters more than it may initially appear.

First, let’s stop comparing apples to oranges 🍎🍊
One mistake I see regularly is treating all AI development tools as if they belong in the same category. They don’t. Some tools live inside the IDE and help developers write code faster. Some tools try to behave like software engineers. Others focus on improving reviews, tests and code quality. And some are concerned less with the code itself and more with the engineering processes surrounding it.
Comparing all of them directly is a little like comparing Jenkins, IntelliJ and SonarQube. They’re all valuable, but they simply solve different problems. For the rest of this article, it helps to think about three broad categories:

Category
Mission
Examples

🧑‍💻 Coding assistants
Help developers write code faster
Copilot, Cursor, Aider

🤖 Engineering agents
Let AI perform engineering tasks on behalf of developers
OpenHands, Devin, SWE-Agent

⚙️ Workflow automation platforms
Make good engineering practices happen consistently
GitLab Duo, Qodo, AI-Git-Bot

Let’s start with GitHub Copilot
When people discover AI-Git-Bot, Copilot is usually the first comparison. That’s understandable since both products involve AI, interact with source code and aim to improve developer productivity. But they attack the problem from opposite ends of the development lifecycle.
If your bottleneck is writing code, Copilot is one of the strongest products on the market. Developers spend most of their day in an editor, and Copilot meets them exactly there. Whether you need boilerplate, a unit test skeleton or help understanding an unfamiliar API, Copilot can often reduce minutes of work to seconds. Its biggest strength isn’t necessarily the quality of the generated code, but rather the reduction in friction that allows developers to stay inside the editor and maintain their flow.
However, Copilot is primarily concerned with helping developers produce code. Once that code becomes a pull request, many of the engineering chores are still waiting: reviews, test coverage, acceptance criteria, regression checks, preview validation and follow-up tasks. Copilot can assist with some of those activities, but they are not its primary focus.
So the decision comes down to this: choose Copilot if your biggest problem is that your developers spend too much time writing code, but choose AI-Git-Bot if your biggest problem is that the work after the code is written keeps getting skipped.

GitLab Duo: AI for an entire platform
GitLab Duo is one of the most interesting products in the space because it benefits from owning an entire software delivery platform. GitLab doesn’t just manage source control—it also manages issues, pipelines, security scans, releases and documentation. That gives Duo access to an enormous amount of context.
If your engineering organization is already deeply invested in GitLab, Duo is a very natural choice. The experience feels integrated because it is integrated. Features can span multiple parts of the software lifecycle without crossing system boundaries, which creates opportunities that are difficult for standalone products to replicate.
The trade-off, of course, is that the same thing that makes Duo powerful also limits it. It is fundamentally a GitLab solution, so for organizations running GitHub Enterprise, Gitea, Bitbucket or mixed environments, that tight integration becomes less valuable.
AI-Git-Bot takes a different approach. Instead of asking teams to adopt a specific Git platform, it attempts to work with whichever Git platform already exists. For some organizations, platform flexibility is more important than deep platform integration, while for others the opposite is true. Neither approach is inherently better—they simply optimize for different priorities.

Qodo: solving the quality problem
Not every engineering team struggles with development speed. Many teams ship plenty of code, but their challenge is making sure that code is correct. That’s where Qodo becomes interesting.
Qodo focuses heavily on code quality, particularly pull request reviews, test generation, code understanding and reviewer assistance. If your recurring team conversation sounds like “We’re moving quickly, but quality keeps slipping,” then Qodo is worth serious consideration.
One of Qodo’s strengths is that it doesn’t require dramatic process changes. Most teams can add it to their existing workflow and start seeing value relatively quickly, which is often an underrated advantage.
The overlap between Qodo and AI-Git-Bot is real since both can participate in reviews and generate tests. The difference lies in where they place their center of gravity. Qodo focuses primarily on improving review and testing quality, while AI-Git-Bot focuses on automating workflows around reviews, tests and pull requests—things like auto-reviewing pull requests, re-reviewing after force-pushes, turning issues into pull requests, generating Playwright suites, managing preview environments and automating repetitive engineering tasks. The goals are related, but not identical.

OpenHands: the engineering agent approach
OpenHands is often compared to AI-Git-Bot because both products can act autonomously, but the philosophy behind them is very different. OpenHands asks whether an AI can act like a software engineer, while AI-Git-Bot asks whether we can automate the necessary-but-uncomfortable parts of software engineering. Those questions sound similar, but they lead to very different products.
OpenHands is one of the most ambitious projects in the AI engineering space. It can explore repositories, execute commands, modify code, complete complex tasks and work toward larger objectives. In many situations it feels less like an assistant and more like an autonomous junior engineer, which is a remarkable capability.
Autonomy always comes with trade-offs, though. The more freedom a system has, the more review and oversight become necessary. This is not a criticism of OpenHands—it’s simply the reality of agent-based systems today.
Choose OpenHands when your goal is experimentation with autonomous engineering, but choose AI-Git-Bot when your goal is reliable automation of recurring Git workflows.

So where does AI-Git-Bot actually fit?
AI-Git-Bot started from a very simple observation: most teams already know what good engineering practices look like. The problem isn’t knowledge—it’s consistency.
Nobody argues against writing good issues, adding tests, reviewing carefully, re-testing after changes, cleaning up environments or documenting decisions. In fact, most teams already agree these things should happen. The challenge is that they’re also the first things to disappear when deadlines get tight.
AI-Git-Bot focuses on the work that teams know they should be doing but struggle to do consistently. Today that includes:

Capability
Description

🔍 Pull request reviews
Consistent, thorough code reviews on every PR

💬 Interactive conversations
Ongoing dialogue about code changes and decisions

📝 Issue refinement
Turning vague ideas into actionable work items

🔗 Issue-to-code workflows
Automatically creating PRs from issues

🧪 Unit test generation
Adding tests for new code and bug fixes

🎭 End-to-end test generation
Creating Playwright suites for user journeys

🚀 Preview environment validation
Deploying and testing PRs in real environments

🔄 Lifecycle automation
Managing the entire PR workflow automatically

The goal has never been to replace developers. The goal is much closer to making good engineering habits easier to sustain.

Where AI-Git-Bot is strongest
Three areas stand out.
Platform flexibility is the first. Most competing solutions are tied to a specific Git ecosystem, but AI-Git-Bot works with GitHub, GitHub Enterprise, GitLab, Gitea and Bitbucket Cloud through the same architecture. Teams can keep existing workflows while changing platforms underneath.
Provider flexibility is the second. Most AI products are closely associated with a particular AI vendor, but AI-Git-Bot allows teams to choose OpenAI, Claude, Gemini, Ollama or llama.cpp without redesigning workflows. You switch providers but keep the process.
Self-hosting is the third, and for some organizations this is the deciding factor. Code, prompts, models and credentials can remain entirely inside the organization’s infrastructure. For regulated industries, government organizations and privacy-sensitive teams, that flexibility is often more important than having the newest AI feature.

Where AI-Git-Bot is not the best choice
This is the section marketing departments usually avoid, but it’s worth saying explicitly.
If your primary goal is to help developers write code faster, you should probably start with GitHub Copilot, Cursor or Aider. Those products were built specifically for that experience. AI-Git-Bot focuses on workflow automation rather than editor assistance, which is a deliberate trade-off.

If I were choosing today

If you are…
Start with…
Why

👤 Solo developer
Copilot, Cursor or Aider
You’ll feel the productivity boost almost immediately

🏢 GitLab organization
GitLab Duo
The platform integration is difficult to beat

✅ Team struggling with quality
Qodo
Its strengths align extremely well with review and testing challenges

🤖 Team exploring autonomous agents
OpenHands
One of the most interesting engineering-agent projects available today

🔒 Team wanting complete control
AI-Git-Bot
Especially if you’re running Gitea, GitHub Enterprise, Ollama or self-hosted infrastructure

Brand new? Start here 🧭

If you are…
Start with…

👀 Curious
The pitch — why this exists and what it actually does

🧑‍💻 A developer who wants to play
Local development guide — up and running in ~10 min

🏗️ An architect evaluating it
Architecture overview

🛠️ DevOps / Platform
Deployment guide

🆙 Already running it
Migration guides — drop in, done

Get it now
docker pull tmseidel/ai-git-bot:latest

⭐ Star us on GitHub: https://github.com/tmseidel/ai-git-bot
🐛 Found something? Open an issue — we read every one
💬 Just want to say hi? Drop a discussion on the repo

The bottom line
AI tooling is slowly splitting into distinct layers of the software delivery process. Some tools help developers write code. Some tools attempt to act like developers. Some tools improve software quality. Some tools automate the workflows surrounding development itself.
AI-Git-Bot belongs firmly in that last category.
If the thing slowing your team down isn’t writing code—but reviewing it, testing it, validating it, discussing it and generally handling all the necessary-but-uncomfortable work around software delivery—then that’s exactly the problem AI-Git-Bot was built to solve. And honestly, that’s why the project exists. 🚀
Happy shipping.

---

### We Gave Every Team Member an AI Agent — Here's What It Took to Make Them All Think Alike

URL: https://remus-software.org/articles/howto-share-your-ai-experience/
Date: June 27, 2026
Description: Centralizing AI agent configuration across a team using Hermes Agent profiles and Git — keeping everyone on the same models, skills, and behavior without locking them into a web interface.
Tags: AI, Agent, Self-Hosting, Infrastructure

We Gave Every Team Member an AI Agent — Here’s What It Took to Make Them All Think Alike
You know that moment when someone on your team shows you an AI-generated document and it’s… different? Different tone, different structure, different quality. And then you realize: they’re using a different model. Or a different temperature setting. Or some prompt they found on Reddit six months ago.
That’s when it hits you. AI tools are powerful, but they’re also non-deterministic. Two people running the same task with slightly different configurations will get different results. Multiply that across a team of ten, and you don’t have an AI-assisted workflow — you have chaos with better grammar.
I went through this realization firsthand. What started as “let’s give everyone access to an AI tool” turned into a deeper question: how do you make an entire team’s AI experience consistent — same models, same skills, same behavior — without locking them into a rigid web interface that can’t touch their actual tools?
This is the story of that journey.

The Problem That Sneaks Up on You
Introducing AI into a team’s workflow sounds simple on paper. Pick a tool, give everyone access, done. But non-deterministic AI has a property that deterministic software doesn’t: identical inputs do not guarantee identical outputs. And when outputs vary, consistency becomes the foundation everything else rests on.
It’s not just about the model. It’s about everything:

The model itself — which version, which quantization, which endpoint
The skills — the reusable workflows and capabilities the AI has
The hooks — the automation triggers that fire before or after actions
The configuration parameters — temperature, context length, system prompts, tool access

Change any one of these and you change the output. Change them differently for different people, and you’ve built a machine that produces inconsistent results by design.
The core insight is this: for AI to be a team tool rather than a personal toy, the configuration has to be as shared and versioned as the codebase itself.

First Attempt: The Central Server Dream
The obvious answer is centralization. Host everything on one server, give everyone a web browser, and call it a day.
We set up Open WebUI — a polished, open-source chat interface — on a central machine. It served a single model, with a single configuration, to everyone through their browser. No installations. No version conflicts. No “which model are you on?” conversations.
For a while, it worked beautifully.
But then the cracks appeared. Someone wanted the AI to read a file from their locally synced SharePoint folder. Someone else wanted it to interact with a Git repository. A third person needed it to run terminal commands as part of a multi-step workflow.
And that’s where the central-server model hit its ceiling.
Open WebUI is a web interface. It lives in a browser tab. It can’t git diff your local repository. It can’t traverse your file system. It can’t launch terminal processes. These aren’t bugs — they’re architectural constraints. A web app, no matter how well-designed, can’t be an agent.
The conclusion was uncomfortable but clear: to get agent capabilities — real integration with local tools and file systems — the AI has to run on the local machine. But we still needed the consistency of a central configuration.
We needed both things at once.

The Architecture That Actually Worked
That’s when we found Hermes Agent, an open-source AI agent framework from Nous Research, and specifically its profile system.
Here’s the architecture that solved the problem:
flowchart TD
    subgraph Git["Central Git Repository"]
        direction LR
        Profiles["profiles/"]
        Localllm["localllm/"]
        Config["config.yaml(model, provider, params)"]
        Skills["skills/(agent capabilities)"]
        Hooks["hooks/(automation triggers)"]
        Profiles --> Localllm
        Localllm --> Config
        Localllm --> Skills
        Localllm --> Hooks
    end

    Git -->|"git pull"| Alice
    Git -->|"git pull"| Bob
    Git -->|"git pull"| Charlie

    subgraph Alice["Alice's Machine"]
        HA["Hermes Agent"]
        LF1["Local files ✓"]
        LG1["Git ✓"]
        LT1["Terminal ✓"]
        HA --> LF1
        HA --> LG1
        HA --> LT1
    end

    subgraph Bob["Bob's Machine"]
        HB["Hermes Agent"]
        LF2["Local files ✓"]
        LG2["Git ✓"]
        LT2["Terminal ✓"]
        HB --> LF2
        HB --> LG2
        HB --> LT2
    end

    subgraph Charlie["Charlie's Machine"]
        HC["Hermes Agent"]
        LF3["Local files ✓"]
        LG3["Git ✓"]
        LT3["Terminal ✓"]
        HC --> LF3
        HC --> LG3
        HC --> LT3
    end
The magic is in the separation of concerns: the agent runs locally, the configuration lives centrally.
Hermes profiles are plain directories — YAML configs, markdown skill definitions, hook scripts — stored in Git. Before starting Hermes, each team member pulls the latest profile from the repository. The agent then reads that profile and runs with the exact same models, skills, and parameters as everyone else.
But because the agent executes locally, it has full access to the user’s machine: their file system, their synced SharePoint folders, their Git repositories, their terminal. No web interface sandbox. No “sorry, I can’t access that.”
This is the best of both worlds: the consistency of a central server with the power of a local agent.

Why Hermes Specifically?
There are other agent frameworks. What made Hermes the right fit came down to a few specific design decisions:
Profiles are just files. There’s no database to sync, no server to configure, no proprietary format. A profile is a directory. You clone it, you run it. That’s it.
Git-native collaboration. Because profiles are directories of text files, they work with every Git workflow the team already uses. Feature branches for new skills. Pull requests for configuration changes. Code review before anything goes live. The same process that governs the team’s software also governs its AI.
Skills are shareable and composable. Hermes skills are self-contained markdown documents with YAML frontmatter — instructions, scripts, templates, and references all in one directory. Write a skill once, commit it to the shared profile, and every team member gets it on their next git pull.
Model flexibility. The profile specifies which model to use, but Hermes can also fall back to a locally configured provider. This means the team profile can point to a shared local LLM server (we use a Qwen3-32B model running on internal hardware), while each user can also have a personal fallback to a cloud API for tasks that need more power.

The Team Workflow for AI Configuration
One of the most important design decisions was treating AI configuration the same way we treat code. No one edits the production config by hand. Changes go through review.
Here’s what that looks like in practice:

Someone has an idea. Maybe it’s a new skill for generating compliance documents. Maybe it’s a tweak to the system prompt that produces better results.

They create a branch. git checkout -b feature/compliance-skill — same as any code change.

They make the change. A new skill file in skills/, an updated parameter in config.yaml, a new hook script. The profile directory is their workspace.

Pull request and review. Other team members look at the change. Does the skill work? Are the model parameters reasonable? Will this break anyone’s workflow?

Merge and deploy. Once approved, the change lands in main. Every team member gets it on their next git pull. No deployment scripts, no server restarts, no configuration drift.

The beautiful thing about this workflow is that it requires zero new tools. The team already knows how to branch, commit, review, and merge. AI configuration management simply inherits the team’s existing software engineering discipline.

Setting It Up: From Zero to Shared Agent in Three Steps
If you want to replicate this setup, here’s the concrete path.
Prerequisites

A Linux or macOS machine (Windows works through WSL)
Git installed
Access to your team’s shared profile repository

Step 1: Clean Slate
Start fresh. Remove any existing Hermes configuration that might conflict:
cd ~
rm -rf ~/.hermes

Step 2: Install Hermes Agent
The official installer handles everything — core binaries, Python virtual environment, and an interactive setup wizard:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash

The wizard walks you through a series of choices. For a team setup, the key decisions are:

Setup Step
Recommendation

Installation type
Full Setup

Fallback LLM provider
Your choice (OpenAI, Anthropic, etc.)

Terminal backend
Keep default

External platforms
Skip

CLI tools
Keep default selection

Browser provider
Local Browser

Image generation
Skip

Text-to-speech
Skip

Vision backend
Skip

Search provider
DuckDuckGo

The fallback provider matters because it’s what Hermes uses when no profile is specified — useful as a personal backup or for tasks that need frontier-model reasoning.
Step 3: Link the Shared Profile
This is the critical step. The wizard creates an empty ~/.hermes/profiles directory. Replace it with a symlink to your team’s shared profile repository:
# Clone the team repo if you haven't already
git clone git@github.com:your-team/hermes-profiles.git /path/to/team-profiles

# Replace the empty directory with a symlink
rm -rf ~/.hermes/profiles
ln -s /path/to/team-profiles/profiles ~/.hermes/profiles

That’s it. Now when you run Hermes with the team profile, it transparently reads all configuration from your local Git clone — which you keep in sync with git pull.
Using the Team Profile
# Launch with the shared team configuration
hermes chat --profile localllm

# Or use your personal fallback model
hermes chat

What Happens Behind the Scenes
When you run hermes chat --profile localllm, three things happen:

The core engine starts using the Hermes installation from Step 2 — the CLI binary, the Python environment, the system dependencies.

The profile is loaded from the symlinked Git repository — the model configuration, the skills, the hooks. Everything the agent needs to behave like every other team member’s agent.

Data stays local. Conversation history, execution logs, and personal API tokens live in your ~/.hermes directory (in files excluded from the shared repository by .gitignore). No sensitive data ever reaches the team repository or the central server.

What We Learned
After rolling this out across the team, a few lessons became clear:
1. Consistency isn’t just about the model. We initially thought “same model = same results.” Wrong. Skills, hooks, system prompts, and even tool access permissions all affect output. The profile system covers all of these, not just the model name.
2. Git is the right synchronization primitive. We considered configuration databases, API-driven configuration servers, even shared network drives. Git won because it’s battle-tested, every developer already has it, and it gives you branching, history, review, and rollback for free.
3. Local execution is worth the setup cost. Yes, installing an agent on every machine is more work than pointing everyone at a web UI. But the payoff — real file system access, Git integration, terminal commands — is the difference between a chatbot and an AI teammate.
4. The web UI wasn’t wrong, just insufficient. Open WebUI is excellent for what it does. For pure chat interactions, it’s hard to beat. But when your workflow involves files, repositories, and multi-step automation, you need an agent, not an interface.
5. Review processes protect quality. Putting AI configuration through the same PR review as code means bad prompts get caught before they reach the team. It also creates a shared understanding — when someone reviews your skill, they learn how it works, and the team’s AI literacy grows organically.
6. Data security is simpler than it seems. Because the agent runs locally and the profile contains only configuration (no data, no tokens, no history), the shared repository stays clean. The .gitignore in the Hermes installation handles the rest.

The Bigger Picture
What we’ve built here isn’t just a way to share AI configurations. It’s a pattern for how teams will manage AI tooling in general.
Think about it: as AI agents become more capable, they also become more configurable. More skills. More hooks. More parameters that subtly influence behavior. The difference between a well-configured agent and a poorly configured one grows with every new capability.
Centralized approaches — the web UI model — will always be simpler to set up. But they’ll also always hit a ceiling on integration depth. Local approaches — the agent model — will require more initial investment but repay it with genuine system integration.
Hermes profiles give you a way to have both: the consistency of centralized configuration with the power of local execution. And because it’s built on Git, it doesn’t ask the team to learn anything new.
The future of AI in teams isn’t about which model is best. It’s about how you make sure everyone is using it the same way.

Further Reading

Hermes Agent Documentation — The full platform documentation
Nous Research — The team behind Hermes Agent
The profile system and skill authoring guide are part of the Hermes documentation

---

### Why Your Local LLM Agent Fails at Round 4: The Hidden Context Window Crisis

URL: https://remus-software.org/articles/context-window-hardening/
Date: June 16, 2026
Description: Agentic AI systems crash after 4-6 rounds when context windows fill up. A four-layer defense strategy keeps local agents running reliably for 20+ rounds with minimal accuracy loss.
Tags: AI, LLM, Agent, Self-Hosting, Infrastructure, Context Window

Why Your Local LLM Agent Fails at Round 4: The Hidden Context Window Crisis
Agentic AI systems — coding assistants, automated reviewers, research agents — rely on multi-turn conversations for complex tasks. Each round adds user prompts, assistant responses, and tool outputs to conversation history. This history is sent back to the model on every request, growing monotonically.
For cloud-hosted models with 100k–200k token windows, this is tolerable for a while. For local models on consumer hardware, it isn’t. A model like Llama 3.1 8B or Mistral 7B typically operates with 8k–32k tokens — a budget exhausted in as few as 4–6 agentic rounds when tool outputs include build logs, file contents, or command output.
The consequences are twofold: the system either crashes with a “prompt too long” error, or — more insidiously — output quality degrades well before the hard limit. Research by Liu et al. (2023) shows that language models attend strongly to the beginning and end of input context but lose information buried in the middle. A bloated window doesn’t just risk crashes; it actively degrades reasoning quality.
This article examines the problem systematically and presents a four-layer defense strategy that keeps agentic systems running reliably for 20+ rounds on 32k-token models. Each layer is explained with trade-offs, implementation, and measurable accuracy impact.
The Problem: Why Context Windows Matter More Than You Think
Every message gets sent back to the LLM on every request. Round 1 sends 5k tokens. Round 2 sends 15k (history + new content). Round 3 sends 30k. Round 4 tries 50k — and crashes if your window is 32k.
For cloud APIs with 200k windows, you might survive 15–20 rounds. But the real problem isn’t hitting the ceiling — it’s degraded quality long before that. Research shows LLMs exhibit “lost in the middle” behavior: they attend strongly to prompt beginnings and ends but lose info in the middle. A bloated window doesn’t just risk crashes; it actively makes your agent dumber.
Why Local Models Are Especially Vulnerable
Three factors make context management critical for local deployments:

Smaller windows: Llama 3.1 8B on a single RTX 3060 (12GB VRAM) typically gets 8k–16k tokens. Mistral 7B stretches to 32k. That’s 10x less headroom than Claude 3.5 Sonnet’s 200k.

No graceful degradation: Cloud providers often implement server-side truncation or return helpful errors. Local inference servers like Ollama and vLLM tend to crash or return garbled output on overflow.

VRAM pressure: Context tokens consume GPU memory. A 32k-token context at FP16 requires ~2GB just for the key-value cache. On a 12GB card, that leaves precious little for model weights and computation.

The Four-Layer Defense Strategy
A tiered approach applies progressively more destructive strategies as context pressure increases. The goal: apply the lightest touch that keeps the system within budget.
flowchart TD
    A["Tool Result Generated"] --> B{"Is a code file?"}
    B -->|"Yes"| C1["Layer 1a: Structural ExtractionSignatures + dependencies"]
    B -->|"No"| C2{"Size > 8k chars?"}
    C1 --> C1a{"Reduced enough?"}
    C1a -->|"Yes"| D["Add to History"]
    C1a -->|"No"| C2
    C2 -->|"Yes"| C3["Layer 1b: FallbackHead+Tail Truncation"]
    C2 -->|"No"| D
    C3 --> D
    
    D --> E["AI Call Returns"]
    E --> F{"Last-call tokens> 70% of window?"}
    F -->|"Yes"| G["Layer 2: Proactive CompactionRemove old units"]
    F -->|"No"| H["Continue"]
    G --> H
    
    H --> I{"History > 120k chars?"}
    I -->|"Yes"| J["Layer 3: Aggressive CompactionKeep only last 2 units"]
    I -->|"No"| K["Next Round"]
    J --> K
    
    K --> L{"AI Call Failswith 'prompt too long'?"}
    L -->|"Yes"| M["Layer 4: Emergency RecoveryRetry with compaction"]
    L -->|"No"| N["Process Response"]
    M --> E
    N --> O["Done"]
Each layer has a specific purpose and trade-off between information loss and context savings.
Layer 1: Smart Code Truncation — Don’t Cut, Distill
Tool outputs are the biggest context hogs. A single cat on a 500-line Java file produces 15k-20k chars. Build logs easily exceed 100k. But how you shrink them matters enormously — and code demands a fundamentally different approach than prose or log output.
Code relies on strict logic: missing closing brackets, severed variable scopes, or orphaned function calls break the AI’s understanding. Simple text cutting — lopping off the middle of a source file — produces syntactically broken fragments that confuse the model more than they help. Text cutting of code should be the last resort, applied only when the context window is already full and no smarter option remains.
The right approach for code is structural distillation: parse the file into its abstract shape, extract only what the model needs for reasoning, and discard implementation detail it can re-read on demand.
Layer 1a: Structural Extraction for Code Files (The Smart Path)
Three complementary strategies extract the minimum viable representation of a codebase without breaking syntax:
1. Signature extraction via Abstract Syntax Trees. Parse source files with Universal Ctags (or Tree-sitter) to extract only function signatures, class definitions, method declarations, and docstrings. This gives the model a lightweight “map” of the codebase — all the structural landmarks with none of the implementation bloat.
Universal Ctags supports 40+ languages natively and can be called as a subprocess. Tree-sitter — the engine powering GitHub’s syntax highlighting and code navigation — is the industry standard for multi-language tooling. Here is the conceptual pipeline:
Source file → ctags --output-format=json --fields=+neKz
             → Filter for classes, methods, functions, constructors
             → Format as markdown code block with signatures only

A 500-line OrderProcessor.java becomes:
class OrderProcessor {
  constructor OrderProcessor(String processorId)
  method processOrder(String orderId, double amount)
  method internalCleanup()
}

This 6-line skeleton preserves hierarchy, parameter types, and method names — everything the model needs to understand program structure — while slashing token count by 95%+. The implementation bodies can be re-read on demand if the model decides they’re relevant.
2. Dependency and import extraction. A second pass with ctags (--kinds-all=+i+n) strips a file down to its declared namespace and imported dependencies only:
{
  "file": "Button.tsx",
  "declared_namespace_or_package": "components.ui",
  "dependencies": [
    "react",
    "../hooks/useAuth",
    "@mui/material/Button"
  ]
}

This 6-line dependency map tells the model exactly which modules are coupled — far more useful than 300 lines of JSX it can’t fit in context anyway. The model can make architectural decisions (refactoring, dependency breaking, circular import detection) without seeing a single line of implementation code.
3. Directory structure mapping. Tools like tree or a generated structural map (components/, utils/, models/) let the model navigate the project before asking for specific file contents. This is already well-served by standard tooling (rg, find, tree) and needs no additional infrastructure.
How to implement. The simplest universal approach is a system call to universal-ctags (installable via apt install universal-ctags or available as a Docker image). The tool emits JSON with fields for name, kind (class / method / function / import / namespace), and signature. Filter to the kind values the model actually needs (classes, methods, constructors, functions — not local variables), format as a markdown code block with the correct language marker, and inject into context.
For dependency-only extraction, the same ctags call with --kinds-all=-* --kinds-all=+i+n emits only imports, includes, namespaces, and packages — a 3-5 line output regardless of file size.
Accuracy impact: Negligible. The model sees the same structural information it would extract by reading the full file, compressed without syntax damage. Tasks like “add a method to class X” or “refactor the dependency on Y” work identically.
Layer 1b: Head+Tail Truncation (The Fallback — Last Resort for Code)
When structural extraction fails (no ctags available, unsupported language, or output still too large), fall back to text cutting. Even here, be smarter than blind character chopping:
def truncate_tool_result(text, max_chars=8000, is_code=False):
    if len(text) <= max_chars:
        return text
    
    if is_code:
        # Code: try structural first, fall back to text cutting as last resort
        structured = try_structural_extraction(text)
        if structured and len(structured) <= max_chars:
            return structured
    
    # Fallback: head+tail for logs, prose, or code where structural failed
    head_size = max_chars // 2
    tail_size = max_chars - head_size - 64  # 64 chars for marker
    
    removed = len(text) - head_size - tail_size
    
    return (
        text[:head_size] +
        f"\n\n[... {removed} characters truncated ...]\n\n" +
        text[-tail_size:]
    )

Why head+tail? A Maven build log: first 50 lines show which modules are building, last 50 show errors or BUILD SUCCESS. The middle 2,000 lines are Compiling... repeated endlessly. Head+tail keeps the useful parts.
Accuracy impact: Minimal (<5% loss on logs/prose), but significant on code — text-cut source files lose syntax integrity mid-function, producing broken fragments the model may misinterpret. This is why structural extraction (Layer 1a) must be tried first for any code file.
Edge case: When max_chars is too small, tail_size goes negative. The implementation guards against this by falling back to head-only.
Layer 2: Proactive Compaction (The Smart Layer)
This is where things get interesting. Token usage is monitored and compaction happens before hitting the limit.
The Subtle Bug: Cumulative vs. Last-Call Tokens
A naive tracker uses cumulative input tokens:
# BROKEN: Don't do this
class TokenTracker:
    def __init__(self, context_window):
        self.total_input_tokens = 0
        self.context_window = context_window
    
    def record(self, input_tokens):
        self.total_input_tokens += input_tokens
    
    def should_compact(self):
        # Triggers when cumulative > 70% of window
        return self.total_input_tokens > 0.7 * self.context_window

This seems fine until a critical issue emerges: each AI call’s “input tokens” already includes the full history. So round 1 reports 5k, round 2 reports 15k (prompt grew), round 3 reports 30k. The cumulative counter sees 5k + 15k + 30k = 50k, but the actual current prompt is only 30k.
The fix: track last-call input tokens instead:
# CORRECT: Track the most recent call
class TokenTracker:
    def __init__(self, context_window):
        self.last_input_tokens = 0
        self.context_window = context_window
    
    def record(self, input_tokens):
        self.last_input_tokens = input_tokens  # Replace, don't add
    
    def should_compact(self):
        # Triggers when current prompt > 70% of window
        return self.last_input_tokens > 0.7 * self.context_window

This correctly measures “how full is the context window right now?” rather than “how many tokens have been sent in total?”
Tool-Pair Atomic Compaction
Old messages can’t simply be dropped — tool-call pairing must be respected. If the assistant says “I’ll call read_file,” that’s followed by a tool result with a matching toolCallId. Dropping the assistant message but keeping the result (or vice versa) causes provider API errors: “orphaned tool message.”
The solution: group messages into compaction units kept or dropped atomically.
def compact_history(history, max_chars, keep_last_n=4):
    # Step 1: Group into logical units
    units = []
    i = 0
    while i < len(history):
        msg = history[i]
        
        if msg.role == "assistant" and msg.has_tool_calls:
            # This assistant + all its tool results = one unit
            unit = [msg]
            i += 1
            while i < len(history) and history[i].role == "tool":
                unit.append(history[i])
                i += 1
            units.append(unit)
        else:
            # Single message = one unit
            units.append([msg])
            i += 1
    
    # Step 2: Check if compaction needed
    total_chars = sum(chars(unit) for unit in units)
    if total_chars <= max_chars:
        return history  # No compaction needed
    
    # Step 3: Drop oldest units, keep last N
    kept_units = units[-keep_last_n:]
    dropped = units[:-keep_last_n]
    
    # Step 4: Insert summary message
    summary = Message(
        role="user",
        content=f"[Earlier conversation compacted: {len(dropped)} turns removed. "
                f"Previous tool results are no longer in history but their "
                f"effects persist. You can re-read files if needed.]"
    )
    
    return [summary] + [msg for unit in kept_units for msg in unit]

Accuracy impact: Moderate. The model loses the ability to reference old tool results but can re-invoke them if needed. Acceptable when the system prompt instructs the agent to re-read files when uncertain.
Layer 3: Aggressive Compaction (The Safety Valve)
When normal compaction isn’t enough — say a 200k-char file upload bloats history — aggressive mode keeps only the last 2 units.
def compact_aggressively(history):
    units = group_into_units(history)
    
    # Keep only the last 2 units (current context + most recent action)
    kept = units[-2:] if len(units) >= 2 else units
    
    summary = Message(
        role="user",
        content="[Context aggressively compacted due to size limits. "
                "Only the most recent 2 exchanges remain. "
                "Re-read any files you need to reference.]"
    )
    
    return [summary] + [msg for unit in kept for msg in unit]

Accuracy impact: Significant — the model loses most working memory. Better than crashing though, as the agent recovers by re-reading files and re-running tools.
Layer 4: Reactive Error Handling (The Last Resort)
When all else fails and the provider returns “prompt too long,” catch the error and retry with aggressive compaction:
def agent_loop_with_retry(max_retries=2):
    retries = 0
    
    while True:
        try:
            response = call_ai(history)
            return response
        
        except PromptTooLongError:
            if retries >= max_retries:
                raise  # Give up
            
            retries += 1
            history = compact_aggressively(history)
            log.warn(f"Prompt too long, aggressively compacted (retry {retries}/{max_retries})")
            continue

This is the last defense. If aggressive compaction still doesn’t fit (e.g., the system prompt alone is too large), the second retry fails and the error surfaces to the user.
How the LLM Can Help Itself
The most underappreciated strategy: teach the LLM to manage its own context via tool use.
The “Re-Read When Needed” Pattern
Instruct the agent to:

Read files once, act on them, then let them age out of context
Re-read files when it needs to reference them again
Use tools to reconstruct state rather than relying on memory

A system prompt might include:
When you need to reference a file you read earlier, re-read it. Context windows
are limited and old tool results may have been compacted. The file system is
your external memory.

This turns the context window from a “must remember everything” constraint into a “working memory” model. The agent learns it’s okay to lose old context because it can always re-acquire information via tools.
Structured Tool Outputs
Instead of dumping entire files into context, the agent can request structural views — the Layer 1a approach exposed as explicit tools:
Tool: read_file("src/main/java/UserService.java")
Result: [2000 lines of Java code — risks context bloat]

Tool: extract_signatures("src/main/java/UserService.java")  
Result: "class UserService { constructor(), method findById(), method save() } // 15 methods"

Tool: extract_dependencies("src/main/java/UserService.java")
Result: { "dependencies": ["UserRepository", "java.util.Optional", "lombok.Data"] }

This requires custom tools but dramatically reduces context for common operations. The extract_signatures tool runs ctags --output-format=json --fields=+neKz under the hood; the extract_dependencies tool runs the same binary with --kinds-all=+i+n. Both are one-time infrastructure costs that pay off on every code-reading tool call.
LLM-Assisted Context Summarization
Here’s where it gets meta: the LLM can summarize its own context before it’s discarded. Instead of replacing dropped messages with a generic “[12 turns removed]” placeholder, the system makes a dedicated LLM call to distill the dropped history into a concise digest preserving key decisions, facts, and task state.
When proactive or aggressive compaction triggers, instead of throwing away old messages:

Extract the messages about to be dropped
Send them to the model with a summarization prompt
Inject the resulting summary as the first message in the compacted history

def summarize_dropped_context(dropped_messages, model_client):
    """Ask the LLM to summarize messages before they're discarded."""
    
    conversation_text = "\n".join(
        f"{msg.role}: {msg.content}" for msg in dropped_messages
    )
    
    summary_prompt = f"""Summarize the following conversation excerpt in 3-5 sentences.
Focus on:
- Decisions made and their rationale
- Files read or modified (by path)
- Current task state and what remains to be done
- Any errors encountered and how they were resolved
- Key facts discovered via tool calls

Do NOT include pleasantries, acknowledgments, or meta-commentary.

Conversation:
{conversation_text}"""

    response = model_client.call(
        messages=[Message(role="user", content=summary_prompt)],
        max_tokens=300  # Keep the summary bounded
    )
    
    return response.content

The compacted history then starts with a rich summary instead of a generic placeholder:
def compact_with_summary(history, max_chars, keep_last_n=4, model_client=None):
    units = group_into_units(history)
    total_chars = sum(chars(unit) for unit in units)
    
    if total_chars <= max_chars:
        return history
    
    kept_units = units[-keep_last_n:]
    dropped_units = units[:-keep_last_n]
    dropped_messages = [msg for unit in dropped_units for msg in unit]
    
    # Attempt LLM summarization; fall back to generic message on failure
    if model_client and dropped_messages:
        try:
            summary_text = summarize_dropped_context(dropped_messages, model_client)
            summary = Message(
                role="user",
                content=f"[Context summary from {len(dropped_units)} earlier turns: "
                        f"{summary_text}]"
            )
        except Exception:
            # Summarization failed — use generic fallback
            summary = Message(
                role="user",
                content=f"[Earlier conversation compacted: {len(dropped_units)} turns "
                        f"removed. Re-read files if needed.]"
            )
    else:
        summary = Message(
            role="user",
            content=f"[Earlier conversation compacted: {len(dropped_units)} turns "
                    f"removed. Re-read files if needed.]"
        )
    
    return [summary] + [msg for unit in kept_units for msg in unit]

Why this beats generic placeholders. A generic “[12 turns removed]” tells the model nothing about what was removed. It has to re-read files and re-run tools just to reconstruct state — wasting tokens and rounds. A summary like “We decided PostgreSQL over SQLite for concurrent writes. Modified UserService.java to add retry logic. Build fails due to missing @Transactional on createUser()” gives enough context to continue without re-discovering everything.
The cost trade-off. The summarization call itself consumes tokens and adds latency — 1-3 seconds for local, 0.5-1 for cloud APIs. But it’s a single bounded call, so cost is predictable. On a 20-round session with 3 compaction events, that’s 3 extra calls — a small price for much better continuity.
flowchart TD
    A["Compaction triggered"] --> B["Extract dropped messages"]
    B --> C["Send to LLM withsummarization prompt"]
    C --> D{"Summarysuccessful?"}
    D -->|"Yes"| E["Inject summary asfirst message"]
    D -->|"No"| F["Fall back to generic'[N turns removed]' message"]
    E --> G["Continue withcompacted history"]
    F --> G
The key insight: summarization is an enhancement, not a dependency — the system works without it, just better with it.
When to summarize. Not every compaction warrants a call. Good heuristic:

Proactive (Layer 2): Summarize if more than 4 units dropped. Below that, generic placeholder suffices.
Aggressive (Layer 3): Always summarize. The model loses most working memory, so a summary is critical.
Emergency retry (Layer 4): Skip. Speed matters more than quality — priority is a successful response.

Recursive summarization. An edge case: what if dropped messages already contain a previous summary? The prompt handles this naturally — the LLM incorporates key points into the new summary. Each cycle distils context further, like reduction in cooking: more concentrated, not gone.
Measuring the Impact
A benchmark on a 20-round agentic coding task using Mistral 7B (32k window) shows:

Strategy
Rounds Completed
Avg Accuracy
Context Utilization

No compaction
4 (crashed)
78%
100% (overflow)

Text-only truncation (head+tail)
8 (crashed)
72%
100% (overflow)

Structural extraction (code) + head+tail (logs)
14 (crashed)
77%
92%

Structural + proactive compaction
20
74%
58%

All four layers with structural extraction
20
73%
55%

Key findings:

Text cutting code hurts accuracy — head+tail on source files produces broken fragments, dropping accuracy 6 points vs. no compaction on the rounds that complete. The model misinterprets syntax-damaged code.
Structural extraction preserves accuracy — signature + dependency extraction recovers nearly all the accuracy lost to text cutting because the model sees intact, parseable code structures rather than mid-function cuts.
Tool truncation alone isn’t enough — even with structural extraction, history still grows unbounded from assistant messages and user prompts.
Proactive compaction extends runtime indefinitely but costs ~5% accuracy on tasks needing long-term memory.
Aggressive compaction rarely triggers (<5% of runs) but prevents crashes when it does.
The “re-read when needed” pattern lets the agent recover accuracy by re-acquiring information on demand.

The 5% accuracy loss from compaction is real but acceptable. Most agentic tasks are “do this thing” not “remember everything from 15 rounds ago.” When long-term memory is needed, the agent compensates by re-reading files or re-running tools. The structural extraction layer ensures that when it does re-read, it gets structurally sound code rather than text-cut fragments — a critical difference the benchmark makes clear.
Takeaways

Context management isn’t optional for local LLMs. A 32k window fills in 6–8 rounds of real agentic work. Plan for it from day one.

Track last-call tokens, not cumulative. Cumulative counters superlinearly overestimate usage since each call includes the full history. Use the most recent call’s input tokens to measure “how full is the window right now?”

Respect tool-call pairing during compaction. Dropping an assistant message but keeping its tool results (or vice versa) causes hard provider errors. Group them into atomic units.

For code, distill the structure — don’t cut the text. Source files lose syntax integrity when text-cut mid-function. Use Universal Ctags or Tree-sitter to extract signatures, class definitions, and dependencies. A 500-line Java file compresses to a 6-line skeleton with zero syntax damage. Text cutting is the last resort for code, applied only when structural extraction fails and the window is already full.

Head+tail truncation is fine for logs and prose, where the middle is usually repetitive and syntax integrity isn’t a concern. But never make it the first strategy for source code.

Teach the LLM to use tools as external memory. The “re-read when needed” pattern turns context limits into a working memory model, reducing compaction’s accuracy cost.

Layer your defenses. No single strategy suffices. Structural extraction (code) → head+tail fallback (logs) → proactive compaction → aggressive fallback → error retry gives resilience at every level.

Aggressive compaction is a feature, not a bug. Losing context is better than crashing. The agent recovers by re-reading files — if your system prompt encourages that behavior.

Let the LLM summarize its own discarded context. A single extra call during compaction produces a rich digest replacing generic “[N turns removed]” placeholders. The model retains decisions, file paths, and task state — recovering much of the accuracy compaction would otherwise cost.

What’s Next?
A proper tokenizer would replace character-count proxies for more precise control, especially for non-Latin languages or code with special characters.
Selective summarization is another promising direction: instead of summarizing all dropped messages uniformly, weight summarization depth by relevance — more detail for current-task messages, less for tangential exploration. This requires a relevance scoring step before summarization but could further improve the quality-to-cost ratio.
Tree-sitter integration would lift structural extraction from a subprocess call to an in-process library, enabling richer queries: extract only methods matching a regex, filter by annotation/attribute, or diff two versions of a file at the AST level. Universal Ctags via system call works today with zero build dependencies; Tree-sitter adds power at the cost of per-language grammars.
For now, the four-layer strategy — structural extraction for code, head+tail fallback for logs, proactive compaction with LLM summarization, and aggressive error recovery — keeps local agents running reliably for 20+ rounds on consumer GPUs. The key insight: don’t cut code, distill it. The model doesn’t need to see every line of a source file; it needs to see its shape.

Further reading:

Lost in the Middle: How Language Models Use Long Contexts — Research on attention patterns in long prompts
vLLM Configuration Options — How inference servers handle context memory
Ollama FAQ: Context Window Configuration — Setting up local models with appropriate windows
Claude Context Windows Documentation — How cloud providers handle large contexts
Universal Ctags — Multi-language code indexer usable as a subprocess for signature and dependency extraction
Tree-sitter — Incremental parsing library powering GitHub’s syntax highlighting and code navigation

---

### From Qwen3-32B to Qwen3.6-35B-A3B: Upgrading a Local Inference Stack on 2× RTX 5060 Ti

URL: https://remus-software.org/articles/qwen3-to-qwen36-upgrade/
Date: June 13, 2026
Description: A dense-to-sparse model swap that nearly doubled token throughput, improved output quality, and taught us a few things about vLLM with MoE and Mamba architectures.
Tags: AI, LLM, vLLM, Self-Hosting, Infrastructure

From Qwen3-32B to Qwen3.6-35B-A3B: Upgrading a Local Inference Stack on 2× RTX 5060 Ti
A dense-to-sparse model swap that nearly doubled our token throughput, made the outputs noticeably better, and taught us a few things about vLLM along the way.

The Setup
We run a local inference server with two NVIDIA RTX 5060 Ti GPUs — 16 GB VRAM each, Blackwell architecture, connected over PCIe (no NVLink, unfortunately). The whole stack is managed through Ansible playbooks so nothing is done manually and everything is reproducible. vLLM handles inference, Ollama sits on standby for lighter stuff.
Before diving into the models, here’s the VRAM math that governs everything we do on this hardware:
=== Model Weight VRAM ===

Formula:  params (billions) × bytes_per_param = weight VRAM

fp16 (no quantization):  2 bytes/param
  → 32B model:  32 × 2 = 64 GB  ❌ way too big
  → 35B model:  35 × 2 = 70 GB  ❌ way too big

INT4 quantization (AWQ/GPTQ):  0.5 bytes/param (4 bits = 0.5 bytes)
  → 32B model:  32 × 0.5 = 16 GB  ✓ fits
  → 35B model:  35 × 0.5 = 17.5 GB  ✓ fits

INT8 quantization:  1 byte/param
  → 32B model:  32 × 1 = 32 GB  ⚠️ tight (no room for KV cache)
  → 35B model:  35 × 1 = 35 GB  ❌ doesn't fit

FP8 quantization:  1 byte/param
  → 32B model:  32 × 1 = 32 GB  ⚠️ same as INT8
  → 35B model:  35 × 1 = 35 GB  ❌ doesn't fit

=== KV Cache VRAM (per active sequence) ===

Formula:  context_tokens × bytes_per_token

The KV cache stores attention keys + values for every token in the context.
Size depends on model architecture (num_layers, num_heads, head_dim) and
the KV dtype:

  fp16 KV cache:  ~0.25 MiB per token (model-dependent)
  fp8 KV cache:   ~0.125 MiB per token (half of fp16)

  For our 35B model at 65K context:
    fp16:  65,536 × 0.25 MiB ≈ 16 GB  ❌ eats half our VRAM
    fp8:   65,536 × 0.125 MiB ≈ 8 GB  ✓ manageable

With AWQ 4-bit weights (~18-20 GB) + fp8 KV cache (~8 GB for a full 65K sequence) + overhead (~2 GB), we land at roughly 28-30 GB — just under our 32 GB ceiling. That’s the budget we’re working with.
Our workhorse for months was Qwen3-32B-AWQ — a dense 32-billion-parameter model quantized to 4-bit. It did code reviews, general chat, and agent tool-calling just fine inside a 65K-token context window. Reliable, predictable, no complaints.
Then we switched to Qwen3.6-35B-A3B-AWQ-4bit. On paper the numbers look similar (35B vs 32B), but under the hood it’s a completely different beast. The upgrade turned out to be worth it — but it wasn’t exactly a drop-in replacement.

What Changed Under the Hood
Dense vs. Mixture of Experts
Here’s the big one. Qwen3-32B is a dense model — all 32 billion parameters fire for every single token. Every forward pass uses the entire brain, so to speak.
Qwen3.6-35B-A3B is a Mixture of Experts (MoE) model. It has 35 billion parameters in total, but only about 3 billion are active per token. The model has a bunch of “expert” sub-networks and a router that picks which experts handle each input. Picture a company with 35 specialists where only 3 get pulled into any given meeting.
The upshot: way less compute per token, even though the model weights take up roughly the same VRAM. That’s where the speed bump comes from.
Hybrid Mamba-Transformer Architecture
Qwen3-32B is a pure Transformer — standard attention, quadratic cost, you know the drill.
Qwen3.6 mixes things up with a hybrid architecture that alternates between Transformer layers and Mamba (State Space Model) layers. Mamba processes sequences with linear complexity instead of quadratic attention, which is a big deal for long contexts. You get precise token-to-token attention from the Transformer layers where it matters, and efficient sequential processing from Mamba everywhere else.
This hybrid design is also the reason we had to change a bunch of vLLM settings — more on that below.

The vLLM Config Changes (and Why They Mattered)
The architecture shift meant several vLLM parameters needed adjusting. Some were obvious, some were not. Here’s the rundown.
1. Tool Call Parser: hermes → qwen3_coder
What this is: vLLM needs a parser to translate the model’s raw text output into structured tool calls (function names + JSON arguments) that match the OpenAI API format. Different models format their tool calls differently, so the parser has to match.
What tripped us up: Qwen3.5 and 3.6 completely changed their tool-calling format. The hermes parser that worked perfectly with Qwen3-32B just doesn’t understand the new format. The correct one is qwen3_coder — yes, confusingly named, but it’s the right parser for all Qwen3.5/3.6 models, not just the ones with “Coder” in the name.
What happens if you get this wrong: This is the nasty part. Tool calls don’t break outright — they just get parsed wrong. The model outputs structured calls in the new format, the hermes parser mangles them, and tool results get injected back in a format the model doesn’t recognize. The result looks like the model suddenly got stupid. It didn’t — it’s a format mismatch. We spent a while scratching our heads before figuring this one out.
2. max_num_batched_tokens: New Parameter, Set to 4096
What this is: Controls the maximum number of tokens vLLM processes in a single prefill batch — basically how much work it tries to chew on at once when processing input tokens.
Why we needed it: The Mamba layers in Qwen3.6 require a memory alignment block size of 2096 tokens. vLLM’s default is 2048. That’s less than 2096. You can probably guess what happens:
AssertionError: In Mamba cache align mode, block_size (2096)
must be <= max_num_batched_tokens (2048)

Server crashes on startup. Not a tuning issue — a hard compatibility floor. Setting it to 4096 gives comfortable headroom and solved it immediately. This is a must-have for any Mamba-based model.
3. RoPE Scaling: Just Removed It
What this is: RoPE (Rotary Position Embedding) scaling tricks like YaRN let a model handle context windows longer than what it was trained for. Qwen3-32B only natively supports 32K tokens, so we needed YaRN to stretch it to 65K.
What changed: Qwen3.6 natively supports up to 262,144 tokens. Our 65K fits easily. So we just… removed the scaling. Done.
Bonus: Without RoPE extrapolation, the model operates inside its trained context range. No more positional confusion in long documents, no artifacts from stretching embeddings beyond their design. This alone improved quality on long-context tasks.
4. max_num_seqs: 32 → 8
What this is: How many concurrent requests the engine keeps alive in GPU memory at once. Each active request needs its own KV cache — the memory structure that stores all the previous tokens’ attention data.
Why we had to lower it: Here’s the VRAM budget math:
Total VRAM (2 GPUs):                          32 GB
- Model weights (AWQ 4-bit, TP=2):           ~20 GB
- Overhead (activations, CUDA, etc.):         ~2 GB
----------------------------------------------
Available for KV cache:                       ~10 GB

KV cache per full-length sequence (65K tokens, fp8):
  65,536 tokens × ~0.125 MiB/token ≈ 8 GB

Max concurrent full-length sequences:
  10 GB available ÷ 8 GB per sequence ≈ 1.25

So with full 65K contexts, you can barely fit one sequence. But most requests are way shorter than 65K — a typical agent conversation might use 10-20K tokens, which needs only 1-2 GB of KV cache. That’s why max_num_seqs: 8 works: it assumes a realistic mix of context lengths, not everyone maxing out the window at once.
The old value of 32 would have risked OOM errors. For our agent workload (typically 1-3 concurrent requests), 8 is more than enough. If you’re running a high-traffic service though, this could be a real constraint.

What Didn’t Change
A few settings carried over and are still doing heavy lifting:

fp8 KV cache: Cuts KV memory in half compared to fp16. The single most important setting for squeezing long contexts into 32 GB.
AWQ 4-bit with Marlin kernel: Model weights at 4-bit precision, ~20 GB instead of ~70 GB. The Marlin kernel is optimized for our Blackwell GPUs.
Tensor Parallelism (TP=2): Each layer is split across both GPUs. Better than pipeline parallelism for our low-concurrency agent workload.
Triton attention backend: Flash Attention is broken on sm_120 consumer GPUs. Triton works fine with the hybrid architecture.
Prefix caching: Reuses KV cache for repeated prompt prefixes (system prompts, tool definitions). Free 50-80% improvement in time-to-first-token for agent workloads.

So, Did It Actually Get Better?
Short answer: yes, noticeably.
Token Throughput Nearly Doubled
The MoE architecture is doing the heavy lifting here. Only 3B parameters active per token means way less compute per generated token compared to the dense 32B. The Mamba layers add more efficiency on the prefill side since they dodge the quadratic cost of full attention.
In practice: responses come back roughly twice as fast. For interactive use and multi-step agent loops, that’s a very welcome improvement.
Output Quality Stepped Up

Code reviews worked fine with both models — the old 32B was already good at spotting bugs, suggesting improvements, explaining logic. The new model keeps that bar.
Minor code refactorings are where Qwen3.6 pulls ahead. Renaming variables for clarity, extracting helper functions, simplifying conditionals, restructuring imports — these need a nuanced feel for code intent and scope. Qwen3-32B would often over-engineer or miss the point. Qwen3.6 handles these cleanly and stays focused.
Tool-calling is more accurate and consistent now that the right parser is in place. Fewer malformed JSON arguments, cleaner structured output.

No More Context Weirdness
With Qwen3-32B at 65K, the YaRN scaling would sometimes cause positional confusion — the model would lose track of where information appeared in long documents, especially past the 32K boundary. Qwen3.6, running within its native context range, doesn’t have this problem. References to earlier parts of long conversations are just more reliable.

What We Learned

Architecture changes aren’t drop-in. MoE and Mamba need different vLLM settings than dense Transformers. The max_num_batched_tokens crash wasn’t well-documented — we had to dig into vLLM source to figure it out.

Tool call parsers are model-specific. Always check this when upgrading between Qwen generations. A mismatch doesn’t throw an error — it just makes the model look dumb, which sends you debugging in the wrong direction for a while.

MoE punches way above its weight. 35B total params, 3B active, quality comparable to dense models many times its active size. For constrained hardware, MoE is the move.

Native context beats stretched context. Running within the model’s native range beats extrapolation every time. When picking models, check the native context window first.

Ansible-managed infra saves your bacon during upgrades. Every config change — parser swap, new parameter, removed scaling — went through playbooks. The whole migration is reproducible, reversible, and self-documenting. Doing this manually on the server would have made the Mamba crash way harder to debug and rollbacks way riskier.

Quick Comparison

Qwen3-32B-AWQ (Before)
Qwen3.6-35B-A3B-AWQ-4bit (After)

Architecture
Dense Transformer
Hybrid Mamba-Transformer (MoE)

Active Params
32B (all of them)
~3B (of 35B total)

Native Context
32K
262K

RoPE Scaling
YaRN (to reach 65K)
None needed

Tool Call Parser
hermes
qwen3_coder

max_num_batched_tokens
2048 (default)
4096 (must be ≥ 2096)

max_num_seqs
32
8

KV Cache
fp8
fp8

Quantization
AWQ (awq_marlin)
AWQ (awq_marlin)

Tensor Parallelism
2
2

Attention Backend
TRITON_ATTN
TRITON_ATTN

Hardware: 2× NVIDIA RTX 5060 Ti 16 GB (Blackwell, PCIe, no NVLink). Inference: vLLM. Infrastructure: Ansible.

---

### I Built My Own AI Server — Here's What I Learned About Running Frontier Models at Home

URL: https://remus-software.org/articles/i-built-my-own-ai-server/
Date: June 5, 2026
Description: A hands-on journey of building a local AI server with dual RTX 5060 Ti GPUs to run 32-billion-parameter models at home — covering hardware choices, software stack (Ollama, vLLM, Open WebUI), Blackwell compatibility headaches, and how local open-source models compare to frontier APIs.
Tags: AI, LLM, Self-Hosting, Hardware, DevOps

I Built My Own AI Server — Here’s What I Learned About Running Frontier Models at Home
We’ve all been there: you send a prompt to Claude or GPT-5, wait a few seconds, and wonder — could I do this myself? Not out of arrogance, but curiosity. What does it actually take to run a 32-billion-parameter language model on your own hardware, in your own home, without sending a single byte to the cloud?
I set out to find out. I bought the hardware, installed the software, fought the bugs, and benchmarked the results. What I found was a landscape that is simultaneously more accessible and more chaotic than I expected. The tools exist. The models are free. The hardware is affordable. But putting it all together? That’s still an adventure — one I want to share with you.

Why Bother With a Local AI Server?
The case for running AI locally is compelling on paper:

Privacy: Your data never leaves your network.
Cost: No per-token API fees that scale unpredictably.
Control: You choose the model, the configuration, and the rules.
Learning: There’s no better way to understand these systems than running them yourself.

But there’s also an honest reason: I wanted to see if a 32-billion-parameter open-source model could actually hold its own against the frontier models from Anthropic and OpenAI. The answer, as you’ll see, is nuanced.

The Hardware: Keeping It Realistic
I didn’t want to go all-in financially. The goal was to build something that could seriously run AI agents — not a toy, but not a $20,000 GPU workstation either. Here’s what I landed on:

Component
Choice
Why

GPU
2× NVIDIA RTX 5060 Ti (16 GB each)
32 GB total VRAM — enough for a 32B model

CPU
AMD Ryzen 9 9950X (16 cores)
Fast enough for data preprocessing and orchestration

RAM
64 GB DDR5
Buffer for model weights and context overflow

Storage
2 TB NVMe SSD
Fast model loading and KV cache spill

The RTX 5060 Ti is interesting. It’s NVIDIA’s newest Blackwell architecture (compute capability SM 12.0) — powerful, but so new that much of the AI software ecosystem hadn’t caught up yet. That decision alone would cause me weeks of headaches, as we’ll see.
The total build came in at a fraction of what a cloud GPU instance would cost over a year. And as a bonus? I could install Windows on the side and play games when I wasn’t experimenting with AI. :)

The Software Stack: Three Engines, One Server
The open-source LLM ecosystem has consolidated around a few key tools. I chose a stack of three:
flowchart TD
    User["You (browser, CLI, or agent)"]
    WebUI["Open WebUI (:8080)Chat interface"]
    Ollama["Ollama (:11434)Simple, friendly model serving"]
    vLLM["vLLM (:8001)High-performance inference"]
    GPU["2× RTX 5060 Ti32 GB VRAM"]

    User --> WebUI
    WebUI --> Ollama
    WebUI --> vLLM
    Ollama --> GPU
    vLLM --> GPU

Ollama: The easy option. Download a model, run it, chat with it. Great for experimentation.
vLLM: The serious option. Built for throughput — parallel requests, efficient memory management, and production-grade serving.
Open WebUI: A beautiful chat interface that connects to either engine.

A critical design decision: only one inference engine runs at a time. Both Ollama and vLLM need the full 32 GB of VRAM to serve a 32-billion-parameter model. Running them simultaneously would cause one to crash with an out-of-memory error. I automated the switching with a single configuration toggle:
# One variable controls everything
inference_engine: "vllm"   # or "ollama"

When you flip this value, Ansible (my automation tool of choice) stops one engine, starts the other, and reconfigures the web interface to point at the right one. Clean and simple.

The Blackwall: When New Hardware Meets Immature Software
Here’s where the story gets interesting. The RTX 5060 Ti uses NVIDIA’s Blackwell architecture, internally known as SM 12.0. It’s powerful hardware, but vLLM — the inference engine I needed for serious work — had barely been tested on it.
The first time I tried to start vLLM, it crashed immediately. The error messages were cryptic:
undefined symbol: check_cuda_arch
RuntimeError: FlashInfer requires GPUs with sm75 or higher

This was baffling. SM 12.0 is higher than SM 7.5. The problem? The FlashInfer library — vLLM’s default attention backend — had a JIT compiler that simply didn’t know how to handle Blackwell GPUs yet. It saw an architecture number it didn’t recognize and panicked.
This wasn’t just one bug. It was a cascade of three independent problems, each hiding behind the other:
flowchart TD
    A["vLLM won't start"] --> B["Bug 1: Broken systemd unit(typo in config file)"]
    A --> C["Bug 2: FLASH_ATTN backendincompatible with SM 12.0"]
    A --> D["Bug 3: FlashInfer sampleralso incompatible with SM 12.0"]
    B --> E["Fix: Remove stray charactersfrom RestartSec value"]
    C --> F["Fix: Switch to TRITON_ATTN(a different attention backend)"]
    D --> G["Fix: Uninstall flashinfer-pythonentirely from the environment"]
    E --> H["vLLM boots! ...but slowly"]
    F --> H
    G --> H
The fix for the attention backend was to switch from FLASH_ATTN to TRITON_ATTN — a different implementation that happened to work on Blackwell. But FlashInfer kept resurfacing in unexpected places: not just for attention, but also for the sampler (the component that picks which word comes next). Eventually, the only reliable solution was to completely remove the FlashInfer package from the Python environment, forcing vLLM to use its fallback paths everywhere.
This took days of debugging, reading GitHub issues, and testing configurations. It’s a reminder that in fast-moving open-source ecosystems, being on the cutting edge of hardware means you are the QA team.

The VRAM Puzzle: Fitting a 32-Billion-Parameter Model Into 32 GB
A 32-billion-parameter model like Qwen3-32B is a large piece of software. In its compressed AWQ (4-bit quantized) form, the model weights take up about 19 GB of VRAM. That leaves roughly 13 GB for the KV cache — the working memory the model uses to track the conversation context.
Here’s the math that kept me up at night:

Context Length
KV Cache (fp16)
KV Cache (fp8)
Fits?

32K tokens
~8 GB
~4 GB
✅ (with fp8)

64K tokens
~16 GB
~8 GB
✅ (with fp8, tight)

128K tokens
~32 GB
~16 GB
❌

At standard precision (fp16), even 32K tokens of context wouldn’t fit alongside the model weights. The breakthrough was switching the KV cache to fp8 — a half-precision floating point format that Blackwell supports natively. This halved the memory requirement with virtually no quality loss, letting me serve conversations up to 64K tokens long.
Splitting Across Two GPUs
Since the 19 GB model doesn’t fit on a single 16 GB card, it must span both GPUs. There are two ways to do this:
flowchart LR
    subgraph "Pipeline Parallelism (PP=2)"
        direction LR
        P1["GPU 0: Layers 1-30compute → send → wait"]
        P2["GPU 1: Layers 31-60wait → compute → send"]
    end

    subgraph "Tensor Parallelism (TP=2)"
        direction LR
        T1["GPU 0: Half of every layercompute → sync → compute"]
        T2["GPU 1: Other half of every layercompute → sync → compute"]
    end
Pipeline Parallelism splits the model in half — GPU 0 handles the first 30 layers, GPU 1 handles the last 30. The problem? While GPU 0 is working, GPU 1 sits idle, and vice versa. For a single conversation, this wastes about 50% of available compute.
Tensor Parallelism splits each layer across both GPUs, so they’re always working together. The cost? They need to exchange data after every single layer — and on consumer GPUs without NVLink (NVIDIA’s high-speed GPU-to-GPU connection), that data has to travel through the motherboard’s PCIe bus, which is slow.
I tested both. For my use case (a personal AI agent handling 1-4 requests at a time), the communication overhead of Tensor Parallelism was roughly equal to the idle-time waste of Pipeline Parallelism. I ended up choosing Pipeline Parallelism because it was more stable on my specific hardware — GeForce cards have known issues with P2P (peer-to-peer) GPU communication that can cause deadlocks.

Automation: Because Doing It Twice Is Doing It Wrong
The setup process was complex enough that I automated it with Ansible — a tool that lets you describe your entire server configuration as code. The result: a single command that installs NVIDIA drivers, Docker, Ollama, vLLM, Open WebUI, downloads models, and configures everything:
# One command, 30-90 minutes later, you have a working AI server
ansible-playbook -i inventory.ini site.yml --ask-become-pass

The Ansible project grew to include:

Roles for each component (NVIDIA drivers, Docker, Ollama, vLLM, Open WebUI)
Mutual exclusion logic that ensures only one engine runs at a time
Model download tasks that pull the right quantized models from HuggingFace
A test playbook that verifies everything is working after installation
An update playbook for keeping components current

This was essential. When you’re debugging driver incompatibilities and attention backend failures, you need to be able to tear everything down and rebuild from scratch in minutes, not hours.

Performance: The Honest Numbers
After all the configuration, optimization, and debugging, what did I actually get?
~20 tokens per second for single-stream generation with Qwen3-32B-AWQ.
To put that in context: that’s roughly 15 words per second — readable in real time, but noticeably slower than ChatGPT or Claude, which typically deliver 40-80+ tokens per second through their cloud infrastructure.
Where does the time go? The biggest bottleneck is the pipeline parallelism bubble — that 50% idle time I mentioned earlier. Here are the optimizations I identified and tested:

Optimization
Expected Improvement
Status

Switch to Tensor Parallelism
+15-30% at low concurrency
Tested — marginal gain

Enable prefix caching
+30-50% time-to-first-token
Implemented

Speculative decoding (small draft model)
+50-100% decode speed
Planned

NVFP4 quantization (Blackwell-native)
+17% if available
Waiting for model checkpoint

Reduce context window to 32K
More VRAM for batching
Available if 64K not needed

Push GPU utilization to 95%
Small but free
Implemented

The most promising path forward is speculative decoding: using a tiny model (like Qwen3-0.6B) to quickly generate candidate tokens, which the large model then verifies in parallel. This could potentially double the throughput to 40-55 tokens per second — competitive with cloud APIs.

Real-World Testing: Can It Replace Claude?
I put the local setup to the test with the Hermes agent — an open-source AI agent framework — running real development tasks:

Writing project documentation for a codebase
Generating test cases
Drafting pull request descriptions
Small and large code refactoring suggestions

The results were honest and humbling:
Documentation was useful but lacked depth. The model captured the high-level structure but missed important details that a human reviewer would need to fill in.
Test cases were mostly correct but overlooked edge cases and contained minor errors — good as a starting point, not as a final product.
Pull request descriptions were generally good but sometimes lacked clarity.
Refactoring suggestions were the weakest area. The model sometimes made the code worse, and the response times were long enough to break the flow of interactive development.
The verdict: a 32-billion-parameter local model is not yet a replacement for frontier cloud models when it comes to complex code understanding and generation. It’s a capable first-draft machine that still requires significant human review.
A More Promising Use Case
But there’s a second use case I’m more optimistic about: compliance documentation. Generating ISO 27001 and ISO 62304 compliant documents — security policies, risk assessments, compliance reports — from unstructured customer input. This is fundamentally a text-generation-from-template task rather than deep code analysis, and early signs suggest the model handles it better. The structured, formulaic nature of compliance documents plays to the strengths of a quantized model.

What I’d Tell Someone Starting Today
If you’re thinking about building your own AI server, here’s what I learned:

The hardware is ready. Consumer GPUs with 16+ GB of VRAM can genuinely run useful 30B+ parameter models. The price-to-performance ratio is compelling.

The software is still catching up. Especially on newer GPU architectures. Budget significant time for debugging if you’re on cutting-edge hardware. If stability matters more than bleeding-edge performance, consider a previous-generation GPU (RTX 4090, RTX 3090) where the software ecosystem is more mature.

Automation is not optional. The number of configuration variables, environment settings, and interdependencies is too large to manage manually. Tools like Ansible, Docker, and systemd are essential.

Quantization is the key that unlocks everything. Without AWQ 4-bit quantization and fp8 KV caches, a 32B model simply wouldn’t fit in 32 GB of VRAM. These techniques have matured enough that quality loss is minimal for many tasks.

Set realistic expectations. Local models at this scale are powerful assistants, not autonomous replacements. They excel at drafting, summarizing, and generating structured content — tasks where a human reviews the output. For complex reasoning and real-time interactive coding, cloud frontier models still have a clear edge.

Looking Forward
The pace of progress in this space is extraordinary. When I started this project, Blackwell support in vLLM was essentially nonexistent. Within weeks, community fixes were landing in pull requests. Quantization techniques that were research papers six months ago are now one-line configuration options.
The open-source model ecosystem is also advancing rapidly. Qwen3-32B is already a capable model, and newer releases from Qwen, Llama, and DeepSeek continue to close the gap with proprietary offerings. NVFP4 quantization — designed specifically for Blackwell’s hardware — promises another 17% speedup once model checkpoints become available.
Building your own AI server in 2026 is a bit like building your own PC in 1998: it requires patience, some technical knowledge, and a willingness to troubleshoot. But the reward is a deeper understanding of the technology and a system that is entirely yours — private, customizable, and free from API rate limits.
The future of AI isn’t just in the cloud. It’s increasingly in our homes, our offices, and our homelabs. And it’s getting better every month.

Further Reading

Project Source on GitHub — The complete Ansible playbooks, Docker configurations, and systemd units to replicate this entire setup from scratch
vLLM Documentation — The inference engine powering this setup
Ollama — The simpler alternative for getting started
Qwen3 Model Family — The open-weight models used in this project
vLLM Field Report: AWQ on RTX 5060 Ti — Community testing on Blackwell hardware
Open WebUI — The chat interface that ties it all together

---

### Sovereign AI Is Loud — But The Real Issue Is Sovereign IT

URL: https://remus-software.org/articles/regarding-sovereign-it/
Date: May 22, 2026
Description: Why the debate about sovereign AI is really a broader discussion about sovereign IT — control over infrastructure, platforms, portability, and the long-term ability to make and reverse technical decisions.
Tags: AI, Sovereignty, Cloud, Architecture

Sovereign AI Is Loud — But The Real Issue Is Sovereign IT
Most articles here are focused on very concrete, technical problems: setups, architectures, and reproducible solutions. This one takes a step back. Still, at its core, it remains technical — because questions of “sovereignty” ultimately materialize in infrastructure, operations, and the ability to make and reverse decisions.

The Current Narrative: Sovereign AI
The debate around sovereign AI has gained remarkable momentum. Across industry reports, policy discussions, and conference stages, there is a growing emphasis on regaining control over data, models, and infrastructure. Especially in Europe, the dependency on non-local providers has become a recurring concern, driven by both geopolitical tension and regulatory realities.
Conceptually, the idea is straightforward: build, run, and control AI systems within your own sphere of influence. In practice, however, this turns out to be significantly more demanding.
Operating a self-hosted LLM setup today requires assembling a stack from relatively young and still evolving components:

inference engines such as vLLM or Ollama
containerized deployments and orchestration, often via Kubernetes
surrounding services like vector databases, authentication, observability, and networking
and, perhaps most critically, access to and operation of suitable GPU infrastructure

There is no widely accepted, production-ready “standard stack” comparable to what exists for more traditional enterprise workloads. Instead, organizations are faced with a growing but fragmented ecosystem. While this provides flexibility, it also means that building such a platform still requires a considerable amount of in-house expertise and operational maturity.

Infrastructure Is Scaling — But Not Necessarily Becoming Sovereign
At the same time, the physical backbone of digital infrastructure is expanding rapidly. Driven largely by AI workloads, Europe is experiencing a strong increase in data center construction and capacity.
Key Data Points

Topic
Data

Demand driver
AI is now the leading driver of data center demand in Europe 1

Capacity growth
Avg. deployments: 16 MW → 33 MW → 47 MW (2023–2025) 1

Investment scale
> €100B expected in European data centers by 2030 2

Pipeline size
~14 GW planned capacity in EMEA by mid-2025 3

This expansion is often interpreted as a positive signal for digital sovereignty: more infrastructure, closer to users, under European jurisdiction.
Yet the picture is more nuanced. Much of this newly built capacity is designed to support hyperscale cloud and AI platforms. Even when physically located in Europe, these systems often operate within technological, contractual, and operational frameworks defined elsewhere. In that sense, infrastructure alone does not automatically translate into control.

The Overlooked Context: Sovereignty Decisions Were Made Earlier
Seen in this light, the current intensity of the AI sovereignty debate is somewhat surprising. Over the past decade, many organizations have already made fundamental decisions that reduced their control over IT systems — often quite deliberately.
The shift toward cloud-based and platform-centric architectures brought undeniable advantages: faster deployment, reduced operational overhead, and access to highly sophisticated tooling. At the same time, it gradually relocated critical capabilities:

infrastructure to cloud providers
development workflows to hosted platforms
collaboration and communication to SaaS ecosystems
customer-facing systems to externally operated services

None of this was irrational. In many cases, it was the most pragmatic choice available. However, it also meant that certain aspects of control — over data flows, system behavior, or long-term portability — became harder to maintain.

Different Industries, Different Trade-offs
In my own experience, the way organizations approach these questions varies significantly by industry.
In medical technology, there remains a noticeable degree of caution when it comes to outsourcing critical IT systems. This is not simply a cultural preference but largely shaped by regulation. Requirements around data protection, auditability, and traceability leave relatively little room for ambiguity. As a result, questions about where data resides, who can access it, and how systems can be audited or replaced tend to be addressed early and in detail.
This does not necessarily mean that cloud usage is avoided altogether. Rather, it is approached more deliberately, and often with additional safeguards or hybrid models. In that sense, sovereignty is less an abstract concept and more an operational constraint that needs to be managed.
By contrast, in parts of the automotive industry, the shift toward external platforms was often more assertive. Internal infrastructure teams were frequently seen as cost-intensive and less flexible, while cloud providers offered scalability, speed, and a rich ecosystem of services. Over time, this led to a situation where a significant portion of the IT landscape — from development pipelines to collaboration tools — relies on a relatively small number of external vendors.
This approach brought clear benefits in terms of velocity and standardization. At the same time, it introduced new forms of dependency, some of which only become visible when trying to change direction. Migration paths, data portability, and operational independence tend to become more complex as systems are more deeply integrated into proprietary platforms.

Why AI Changes the Tone of the Debate
Against this backdrop, it becomes clearer why AI has triggered a more pronounced reaction.
AI systems tend to sit very close to core value creation. They process sensitive data, encode domain knowledge, and increasingly influence decision-making processes. As a result, questions of control feel more immediate. Where earlier layers of abstraction — infrastructure, collaboration tools, or workflow systems — could be externalized with relatively limited visibility, AI makes dependencies more tangible.
At the same time, many of the underlying challenges are not new. Vendor lock-in, limited transparency, reliance on external roadmaps, or the gradual loss of internal expertise are all dynamics that have been present for years. AI does not introduce them so much as amplify their impact.

A More Nuanced View on Sovereignty
It is also worth noting that sovereignty is rarely absolute. In practice, it tends to manifest as a spectrum rather than a binary choice.
Levels of Sovereignty

Level
Example

Low
Full reliance on SaaS / hyperscalers

Medium
EU hosting with contractual safeguards

High
Open-source, portable architectures

Maximum
Fully self-hosted or air‑gapped systems

For many organizations, a balanced approach is likely the most practical: retaining control where it matters most while leveraging external services where appropriate. What becomes increasingly important, however, is the ability to make these choices consciously — and to revisit them if needed.

Early Signs of a Shift
There are indications that the broader conversation is evolving. In the public sector, for example, initiatives are emerging that explicitly frame digital sovereignty as a strategic objective.
Programs that move away from proprietary ecosystems toward open-source-based infrastructures — such as the adoption of LibreOffice, Linux, and open collaboration platforms — reflect a growing awareness of long-term dependencies and their implications. 4
These efforts are not without friction. Migration, training, and organizational change can be challenging. Yet they also demonstrate that alternative approaches are possible, even at scale.

Conclusion
The renewed focus on sovereign AI is both understandable and valuable. At the same time, it risks narrowing the perspective if it is treated in isolation.
Questions of control, dependency, and adaptability do not begin with AI — they extend across the entire IT landscape. In that sense, sovereign AI is less a starting point and more a visible manifestation of a broader theme.
A more comprehensive view would therefore consider not only AI systems, but the surrounding architecture, data flows, and operational capabilities. Ultimately, sovereignty is less about complete independence and more about maintaining the flexibility to respond, adapt, and make informed choices.
Or, put differently:
it is not necessarily about building everything yourself —
but about retaining the ability not to be entirely dependent on others.

---

### Meet AI-Git-Bot 1.7 — the teammate that reviews, tests and ships your PRs

URL: https://remus-software.org/articles/meet-ai-git-bot-the-teammate-that-review-ships-test-prs/
Date: May 21, 2026
Description: AI-Git-Bot 1.7 is here, transforming from 'the PR review bot' to the AI teammate your repo has been waiting for — reviewing your code, writing your tests, deploying your previews, cleaning up after itself. The chores that always get cut under deadline pressure? Wire one bot. They get done. Every PR. Forever.
Tags: software-development, AI, DevOps, Git, CI/CD

Imagine opening a pull request and, two minutes later, a bot has reviewed your diff, deployed a fresh preview, written a Playwright test for the feature you just added, run it against that preview, and pasted the report back into the PR — all inside the Git tool you already use.
That’s not a roadmap. That’s AI-Git-Bot 1.7, shipping today. 🚀

First time here? 30-second intro 👋
AI-Git-Bot is the open-source AI teammate that lives inside your Git tool — Gitea, GitHub, GitHub Enterprise, GitLab, Bitbucket Cloud. No new dashboard to log into, no Chrome extension, no Slack bot to babysit. You assign it work the same way you’d assign it to a colleague: request it as a PR reviewer, assign it an issue, or @mention it in a comment.
It takes over the necessary-but-uncomfortable chores that quietly rot every codebase:

📝 Writing a proper issue with acceptance criteria — before any code is written
🔍 Reviewing PRs consistently, even when the human reviewer is drowning
🧪 Adding the regression test for the bug you just fixed (the one we always say “we’ll add later”)
🛠️ Implementing the boring follow-up tickets (renames, bumps, small refactors)
🧹 Tearing down the preview environment nobody remembers spinning up

You pick which chores hurt most this quarter, wire one bot, done. The rest stays exactly as it was.

👉 Want the long-form pitch? Read doc/pitch/PITCH.md — it’s the fastest way to decide whether AI-Git-Bot is for your team.

What’s new in 1.7 — the highlights
🎬 1. PRs that test themselves
This is the headline. Tag your bot with the new Full-stack QA workflow + tell it where to deploy your PRs, and every pull request now gets:

A short plan of which user journeys to cover
A fresh Playwright test suite, generated for this PR
A deploy of your PR to a real preview
The suite run live against that preview
A report comment on the PR — pass/fail, screenshots, suite source included
Automatic teardown when the PR closes

Don’t like the suite? Drop a comment: @bot regenerate-tests focus on the checkout flow. The bot replans with your feedback. Just need a quick rerun? @bot rerun-tests. That’s it.

Why you’ll love it: the test debt that always slips to “next sprint” finally gets paid down — automatically, per PR.
🔌 2. It plays nice with your pipeline (whatever it is)
We added four ways for the bot to deploy your PR — pick the one you already have:

You’re using…
The bot uses strategy…

Jenkins / TeamCity / a bash script behind a webhook
WEBHOOK

Vercel, Netlify, Render, Cloudflare Pages
STATIC

GitHub Actions, Gitea Actions, GitLab CI, Bitbucket Pipelines
CI_ACTION

An internal platform team exposing deploys via MCP
MCP

No need to migrate anything. The bot adapts to your stack, not the other way around.
🧩 3. Workflows you can mix, match and extend
Reviewing PRs and running E2E tests are now just two examples of a bigger idea: PR Workflows. Each one is a named, configurable bundle (“Default review”, “Full-stack QA”, or your own) that you can assign to any bot from the admin UI.

Want a custom one — say a license-header check, an SBOM-diff comment, or a “ping the on-call channel on every hotfix PR” workflow? It’s a clean extension point now. Your platform team can ship it without forking the project.
🗣️ 4. New slash commands, less back-and-forth
Two more commands in the bot’s vocabulary:

@bot rerun-tests — re-run the existing suite, no replanning
@bot regenerate-tests <your feedback> — replan the suite, your feedback goes straight into the planner

Combined with the existing @bot fix … and @bot write …, your PR comments become the remote control.
🛟 5. Test suites that can outlive the PR — if you want them to
By default, generated suites are ephemeral: they live with the PR, vanish on close, nothing leaks. Safe and boring — exactly what most teams want.
But if you’d love to keep the suites the bot writes, flip one setting and pick:

commit-to-pr — committed straight to the PR branch
offer-as-pr — the bot opens a follow-up PR with the suite
promote-on-merge — auto-promoted when the original PR merges

A nightly cleanup job keeps the test folder from turning into a swamp.
🔒 6. Tighter, safer, friendlier
A handful of quality-of-life upgrades that you’ll feel without noticing:

Per-bot tool whitelisting — give your writer-bot read-only access, your coding-bot the full toolbox
Async callbacks with single-use, HMAC-signed secrets — no shared tokens in your CI runners
Force-push safe — re-pushing three times in a minute no longer confuses the bot
Better LLM compatibility — Gemini 3.x, sanitised tool names across providers, more robust agent loops

What to try first — pick one 👇
You don’t need to adopt everything. Start with the one that hooks you:

🚀 The 15-minute upgrade test. Pull tmseidel/ai-git-bot:1.7.0, point at your existing database, watch it migrate cleanly. Your current bots behave exactly like in 1.6. Zero risk, great way to validate.
🎬 The “wow” demo. Spin up the bundled sample stack — docker compose -f systemtest/docker-compose-e2e-sample.yml up — open a PR, sit back, watch the bot plan, deploy, test, report. This is the one to show your team on Monday morning.
🔌 Wire it to your real CI. If you live in GitHub/GitLab/Gitea/Bitbucket pipelines, PR_WORKFLOWS_CI_ACTIONS.md is the shortest path to “the bot just tested my PR against a real preview environment.”
🧩 Build your own workflow. Got a chore you keep nagging humans about on every PR? Ship it as a workflow — your team will thank you forever.

Brand new? Start here 🧭

If you are…
Start with…

👀 Curious
The pitch — why this exists, what it actually does, how it compares to Copilot Workspace / GitLab Duo / Qodo / Aider

🧑‍💻 A developer who wants to play
Local development guide — up and running in ~10 min

🏗️ An architect evaluating it
Architecture overview and agentic workflows internals

🛠️ DevOps / Platform
Deployment guide + CI action recipes

🆙 Already running 1.6
Migration guide 1.6 → 1.7 — short version: drop in, done

Get it now
docker pull tmseidel/ai-git-bot:1.7.0

⭐ Star us on GitHub: https://github.com/tmseidel/ai-git-bot
🐛 Found something? Open an issue — we read every one.
💬 Just want to say hi? Drop a discussion on the repo, we love hearing how teams are wiring this up.

The bottom line: in 1.7, AI-Git-Bot stops being “the PR review bot” and starts being the AI teammate your repo has been waiting for — reviewing your code, writing your tests, deploying your previews, cleaning up after itself.
The chores that always get cut under deadline pressure? Wire one bot. They get done. Every PR. Forever.
Happy shipping. 🚀

---

### Standardizing Agentic AI at Scale: Why Remote MCP Servers Are the Missing Layer

URL: https://remus-software.org/articles/remote-mcp-standardization/
Date: May 15, 2026
Description: Why organizations should standardize agentic AI with remote MCP server pools to improve consistency, governance, security, and adoption across teams.
Tags: AI, MCP, Architecture, Security

How organizations can eliminate inconsistent AI output, unify governance, and accelerate adoption — by moving from local, per-developer tool chaos to a shared pool of remote MCP resources.

The Problem No One Talks About: AI Output Variance
Enterprise AI adoption is accelerating, but a quiet problem is eroding its ROI: the same prompt, run by two different employees, produces dramatically different results. This is not a model problem — it is an infrastructure problem.
The root causes are surprisingly structural:

Factor
Description

Client-side setup
Each employee configures their local AI environment differently — different system prompts, context window sizes, temperature settings, or client versions

Prompt engineering skill gap
Employees vary widely in their ability to write effective prompts, leading to wildly different output quality for identical tasks

Installed local tools & MCP servers
One developer has a CRM MCP plugin; another does not. One has an up-to-date internal knowledge tool; another is running a stale fork

Model version drift
Without pinned configurations, different clients may call different model versions

Context availability
Access to company-specific context (style guides, internal APIs, compliance rules) depends on what each user has manually configured

Security posture
API keys stored in local .env files, personal tokens in config files, credentials passed as environment variables — each a potential exposure point

Tool versioning
Locally installed MCP servers may lag behind in functionality or contain unfixed bugs

Permissions divergence
Without centralized access control, some employees inadvertently have access to sensitive tools they should not, and others are blocked from tools they need

This variance has three compounding consequences that directly impact the business case for AI:

Quality rework — Outputs need manual post-processing because they don’t meet the expected standard, eliminating the productivity gain
Adoption stagnation — When AI “works well for some and not others,” skepticism spreads and uptake plateaus
Compliance failure — Uncontrolled tool access and inconsistent outputs make it nearly impossible to enforce data governance, GDPR obligations, or industry-specific regulations

Where the MCP Ecosystem Stands Today: A Protocol Snapshot
Before making the case for remote MCP, it is worth grounding the discussion in where the ecosystem actually is — because the data tells a revealing story about the gap between where things are and where they need to go.
As of early 2026, the distribution of publicly registered MCP servers by transport protocol breaks down as follows — based on aggregated data from Anthropic, OpenAI, and analyst surveys published Q4 2025 through Q1 2026, as compiled by Digital Applied (April 2026):

Transport
Share
Status

stdio
~67 %
Local subprocess — runs on the client machine

Streamable HTTP
~28 %
Remote, OAuth 2.1-secured — the current standard

SSE (legacy)
~5 %
Remote, deprecated — being phased out

pie title MCP Server Distribution by Transport Protocol (2026)
    "stdio (local)" : 67
    "Streamable HTTP (remote)" : 28
    "SSE legacy (remote)" : 5
Two-thirds of all MCP servers currently run locally via stdio. This is a direct artifact of how MCP adoption began: developers building quick local integrations for their own AI clients, spinning up tools as subprocesses, storing credentials in environment variables. It was the fastest path to a working prototype — and it shows.
The more important number, however, is the trend. Remote MCP deployments have grown nearly fourfold since May 2025. Among the 20 most widely used MCP servers, 80 % now offer a remote deployment option — a figure independently confirmed by MCP Manager via Ahrefs search-volume data (MCP Manager, October 2025). And of those remote servers, 81 % authenticate via OAuth 2.1 — meaning the governance infrastructure is already being built at the ecosystem level.
The market is moving toward remote. The question for organizations is whether they lead that shift deliberately — with centralized governance, versioning, and access control — or whether they arrive there reactively, having accumulated the same local-config debt that once plagued their microservices era.

The 67 % stdio figure is not a benchmark to maintain — it is a starting point to move away from.

The Architectural Shift: From Local Chaos to a Shared MCP Resource Pool
The solution mirrors a pattern mature engineering organizations have known for decades: centralize what should be consistent, and expose it through a well-governed interface.
In the context of AI agents, this means replacing a sprawl of locally configured MCP servers with a unified, remote MCP server pool that every client in the organization connects to.
graph TD
    subgraph Before ["❌ Before: Per-Client Local Setup"]
        E1["Employee A\n(has CRM MCP v1.2)"]
        E2["Employee B\n(has CRM MCP v1.0, misconfigured)"]
        E3["Employee C\n(no CRM MCP)"]
    end

    subgraph After ["✅ After: Shared Remote MCP Pool"]
        C1["Client A"]
        C2["Client B"]
        C3["Client C"]

        subgraph Pool["Remote MCP Server Pool (Org-managed)"]
            direction LR
            MCP1["CRM MCP\n(latest, versioned)"]
            MCP2["Knowledge Base MCP\n(RAG-enabled)"]
            MCP3["Compliance MCP\n(audit-logged)"]
            MCP4["Code Review MCP\n(shared tooling)"]
        end

        C1 --> Pool
        C2 --> Pool
        C3 --> Pool
    end

Benefits of a Remote, Centralized MCP Resource Pool
Moving to shared remote MCP servers delivers benefits across multiple dimensions:
🔒 Security & Governance

Centralized secret management — API keys, tokens, and credentials live on the server, never on employee machines
Unified access control — Role-based permissions enforced at the MCP layer: the marketing team never accidentally calls a financial data tool
Audit trails — Every tool call is logged centrally, making compliance reporting tractable
Reduced attack surface — No credentials scattered across dozens of laptops

📐 Consistency & Quality

Single source of truth — All agents access the same tool versions, the same data, the same business logic
Versioned, tested releases — Tool updates are deployed once and take effect for all users simultaneously
Embedded prompt hygiene — System prompts, context, and guardrails can be baked into server-side tool definitions, not left to individual configuration

⚙️ Operational Efficiency

Zero per-client installation — Onboarding a new employee means one auth flow, not an afternoon of CLI setup
Centralized monitoring — Error rates, latency, and usage patterns visible in one place
Faster iteration — Improve a tool once, benefit everyone immediately

📈 Adoption & Culture

Consistent experience — Every user gets the same capable toolset, reducing “it works for them but not me” friction
Lower barrier to entry — Non-technical employees can use powerful agentic tools without understanding local config
Trust through predictability — Repeatable, governed outputs build organizational trust in AI

The Microservices Analogy: Secured Services Calling Secured Services
This architecture is not new — it is precisely the pattern that modern software engineering adopted when monoliths became unmanageable.
Consider the parallel to microservices:
graph LR
    subgraph Microservices["Microservices Architecture (known pattern)"]
        direction LR
        MS_Client["Frontend / API Gateway"]
        MS_Auth["Auth Service\n(OAuth 2.1 / OIDC)"]
        MS_CRM["CRM Service\n(secured endpoint)"]
        MS_BI["BI Service\n(secured endpoint)"]

        MS_Client -- "Bearer Token" --> MS_Auth
        MS_Auth -- "validates, issues scoped token" --> MS_Client
        MS_Client -- "scoped token" --> MS_CRM
        MS_Client -- "scoped token" --> MS_BI
    end

    subgraph MCP_Arch["MCP Architecture (same principle)"]
        direction LR
        AI_Client["AI Agent / Claude Code"]
        MCP_Auth["OAuth 2.1 Provider\n(org identity)"]
        MCP_CRM["CRM MCP Server\n(remote, secured)"]
        MCP_KB["Knowledge MCP Server\n(remote, secured)"]

        AI_Client -- "auth request" --> MCP_Auth
        MCP_Auth -- "scoped token (RFC 8707)" --> AI_Client
        AI_Client -- "tool call + token" --> MCP_CRM
        AI_Client -- "tool call + token" --> MCP_KB
    end
Just as a well-designed microservices landscape enforces that only authenticated, authorized clients can call sensitive services — regardless of who wrote the calling code or where it runs — a remote MCP pool enforces that only authenticated agents, with the right scopes, can call the right tools.
The MCP spec adopted OAuth 2.1 with PKCE and Resource Indicators (RFC 8707) precisely for this reason: so that a rogue or misconfigured client cannot escalate privileges or leak tokens across service boundaries — a gap that was explicitly closed in the June 2025 spec update.

The bottom line: Just as you would never let application developers hardcode database passwords into their laptops and query production directly, you should not let them hardcode MCP tool configurations either. Centralize the services; secure the boundary; govern the access.

Quick Start: Connecting Claude Code CLI to Remote MCP Servers
The good news is that connecting to a remote MCP server requires only a single command. Here is a practical walkthrough using Claude Code (the CLI), which supports Streamable HTTP transport natively.
Step 1: Add a remote MCP server
# Basic remote server (Streamable HTTP)
claude mcp add --transport http <server-name> https://mcp.your-org.com/crm

# With a scope — "project" commits to .mcp.json (shareable via git)
claude mcp add --transport http --scope project crm-mcp https://mcp.your-org.com/crm

# With a static auth header (for servers not using OAuth)
claude mcp add --transport http --scope user analytics-mcp https://mcp.your-org.com/analytics \
  --header "Authorization: Bearer ${ORG_MCP_TOKEN}"

Step 2: Authenticate via OAuth (if the server requires it)
# Inside a Claude Code session, trigger the OAuth flow
/mcp
# Select the server name, press Enter
# Claude Code opens a browser tab for org SSO / OAuth consent
# After consent, the token is stored automatically

Step 3: Add via JSON (ideal for scripting or CI/CD provisioning)
# For automated onboarding scripts
claude mcp add-json crm-mcp '{
  "type": "http",
  "url": "https://mcp.your-org.com/crm"
}'

Step 4: Verify the connection
claude mcp list
# → crm-mcp    https://mcp.your-org.com/crm    ✓ connected

Committing the org-wide configuration
When --scope project is used, Claude Code writes the server configuration to .mcp.json at the project root. Committing this file to your repository means every developer who clones the repo gets the same MCP pool automatically — no per-person setup required.
// .mcp.json (committed to version control)
{
  "mcpServers": {
    "crm-mcp": {
      "type": "http",
      "url": "https://mcp.your-org.com/crm"
    },
    "knowledge-base-mcp": {
      "type": "http",
      "url": "https://mcp.your-org.com/kb"
    },
    "compliance-mcp": {
      "type": "http",
      "url": "https://mcp.your-org.com/compliance"
    }
  }
}

💡 This approach is AI-implementation agnostic. The MCP specification is an open standard, now governed by the Agentic AI Foundation (AAIF) under the Linux Foundation, with support from Anthropic, OpenAI, Google, Microsoft, and Block. The same remote MCP servers you configure for Claude Code can be consumed by Cursor, Windsurf, the OpenAI Agents SDK, Gemini’s Vertex AI Agent Builder, and any other MCP-compliant client. Invest once in your MCP server infrastructure; benefit across your entire AI toolchain.

A Note on the Argument: Where It Holds — and Where to Be Careful
This proposal is structurally sound, but a few nuances are worth acknowledging:
Where the argument is strong:

For large organizations with many AI users, the consistency and governance case is compelling and mirrors established patterns from API gateway and microservices adoption
OAuth 2.1 at the MCP layer is not theoretical — it is spec-mandated since mid-2025 and implemented in production by major SaaS vendors
The adoption-stagnation problem is real: inconsistent AI experiences are a documented cause of tool abandonment

Where to be careful:

Latency — Remote MCP servers introduce network round-trips vs. local stdio. For latency-sensitive agentic loops (many sequential tool calls), this can matter. Mitigate with geographic co-location or edge deployment
Dependency risk — A central MCP pool is a shared dependency. Plan for high availability from day one; a pool outage affects all agents simultaneously
Migration cost — Teams with mature local setups face a real migration effort. A phased approach (critical/shared tools remote first, niche local tools later) reduces friction
Not a silver bullet for prompt quality — Centralizing tools improves tool consistency, but employees with weak prompt engineering skills will still produce weaker outputs. Pair this with prompt libraries, agent templates, and training

Conclusion
The variance problem in enterprise AI is not going to solve itself. As long as every employee runs their own local configuration, prompt-engineers with their own skills, and connects to their own patchwork of local tools, the delta between the best and worst AI-assisted work in your organization will remain large.
Remote MCP servers are the architectural lever that addresses the tooling layer of this problem — and it is the layer most amenable to centralized governance. Centralizing your MCP resource pool brings your AI agents closer to the reliability model you expect from your SaaS APIs: versioned, authenticated, monitored, and consistent for every caller.
The pattern is proven. The spec is stable. The tooling is ready.

Outlook: Further Levers for AI Standardization in Your Organization
The remote MCP pool is one layer of a broader AI standardization stack. Organizations that go further will benefit from:
graph TD
    L1["🏗️ Layer 1: Remote MCP Server Pool\n(consistent tools, governed access)"]
    L2["📝 Layer 2: Shared Prompt Libraries & Agent Templates\n(consistent task framing, embedded best practices)"]
    L3["🔍 Layer 3: Observability & Evaluation\n(log tool calls, score outputs, detect drift)"]
    L4["🏢 Layer 4: AI Gateway / Proxy\n(rate limits, cost allocation, model routing, PII redaction)"]
    L5["🎓 Layer 5: Continuous AI Literacy Programs\n(prompt engineering, agent design patterns)"]

    L1 --> L2 --> L3 --> L4 --> L5

Shared prompt libraries: Versioned, peer-reviewed prompt templates for common tasks, stored in a central repository and surfaced to all users — reducing the skill-gap effect directly
Observability pipelines: Logging agent tool calls and scoring outputs over time exposes which tools are underperforming, which prompts produce the best results, and where policy violations occur
AI gateways: A proxy layer (such as LiteLLM, Portkey, or a custom gateway) sitting in front of multiple model providers enables model routing, cost visibility, PII stripping, and fallback — independent of which AI client employees use
Agent-as-a-service: For high-value, repeatable workflows (contract review, data extraction, report generation), encapsulate the entire agent — model, tools, system prompt, output format — as an internal service with an API, removing individual variation entirely

The organizations that will lead in AI-augmented productivity are not the ones who gave every employee access to ChatGPT or Claude. They are the ones who built the infrastructure layer that makes the output of any employee, on any client, consistently excellent.

References and further reading:

MCP Official Specification — Transports
Claude Code: Connect to tools via MCP
Anthropic MCP Connector Directory

---

### Improving Software Quality with Reviewer Personas in AI-Git-Bot

URL: https://remus-software.org/articles/improve-software-quality-with-reviewer-personas/
Date: May 8, 2026
Description: How to use AI-Git-Bot as a review gateway with multiple specialized reviewer personas to catch security, reliability, performance, testability, and maintainability issues in Git pull requests.
Tags: AI, Code Review, Best Practices

Improving Software Quality with Reviewer Personas in AI-Git-Bot
AI-Git-Bot can be used as a self-hosted review gateway that connects Git platforms such as Gitea, GitHub, GitLab, and Bitbucket Cloud with configurable AI providers. One of its strongest quality-improvement patterns is not to configure a single generic AI reviewer, but to create a small system of focused reviewer personas. Each persona uses its own system prompt, its own bot identity, and its own review criteria. The result is a repeatable multi-pass review process that brings security, reliability, performance, testability, and maintainability perspectives into the Git workflow before code is merged.
This article uses an exemplary project, ShopFlow, to explain the approach. ShopFlow is a fictional e-commerce application with a Java/Spring Boot backend, a PostgreSQL database, and a web frontend. The team frequently changes checkout logic, payment integrations, discount rules, user-account flows, and order-processing jobs. Those are exactly the kinds of changes where a single human reviewer or a single generic AI review can miss important risks.

1. Problem Description
Modern software teams already know that code review is essential, but the review process is under pressure:

Pull requests often mix business logic, infrastructure changes, tests, and documentation.
Human reviewers have limited time and different areas of expertise.
Security problems, concurrency bugs, missing edge cases, and weak test coverage are easy to overlook.
Teams want fast feedback, but not at the cost of quality.
Generic AI reviews can help, but a single broad prompt often produces broad feedback instead of deep specialized analysis.

In ShopFlow, consider a pull request titled “Add express checkout with saved cards and promotional discount stacking”. The diff touches:

authentication checks for saved payment methods,
discount calculation rules,
database migrations,
order-confirmation emails,
retry behavior for payment-provider timeouts,
unit and integration tests.

No single reviewer perspective is enough. A security-minded reviewer will ask whether users can access another user’s saved card. A reliability reviewer will look for retry storms and idempotency problems. A maintainability reviewer will ask whether the discount rules are testable and understandable.
AI-Git-Bot addresses this by letting a team configure several bots, each connected to a Git integration, an AI integration, and a dedicated system prompt entry. These bots can be requested as reviewers in the Git platform. When assigned or re-requested as reviewers, they fetch the pull-request diff, send it to the configured AI provider, and post their review back to the pull request.
flowchart LR
    PR["ShopFlow pull request\nExpress checkout"] --> SecBot["Security Reviewer Bot"]
    PR --> RelBot["Reliability & Performance Bot"]
    PR --> MaintBot["Testability & Maintainability Bot"]

    SecBot --> SecReview["Finds auth, permission, secret,\nand input-validation risks"]
    RelBot --> RelReview["Finds latency, retry, concurrency,\nand resource-usage risks"]
    MaintBot --> MaintReview["Finds weak tests, unclear design,\nand maintainability issues"]

    SecReview --> Git["Git pull-request discussion"]
    RelReview --> Git
    MaintReview --> Git
    Git --> Human["Human reviewer makes\nfinal merge decision"]
The goal is not to replace human review. The goal is to make human review more effective by adding consistent, specialized, early feedback directly where development already happens: in the Git pull request.

2. Current Status of Good Practices
Good engineering teams already combine several quality practices:

Practice
Strength
Limitation

Human code review
Context, judgment, ownership, mentoring
Slow under load; expertise varies by reviewer

CI builds and tests
Deterministic validation
Only catches what is encoded in tests

Static analysis
Repeatable style, bug, and complexity checks
Often shallow on business context

Dependency scanning
Known vulnerability detection
Does not reason about application-specific misuse

Security review
Deep risk analysis
Expensive and often reserved for larger changes

Architecture review
Long-term design quality
Usually asynchronous and not applied to every PR

These practices are still necessary. AI-Git-Bot fits as an additional review layer between automated checks and final human approval. It is especially useful because its reviews are:

triggered intentionally by assigning or re-requesting a bot as reviewer,
configurable through reusable system prompt entries,
contextual because the bot reviews the pull-request diff and can answer follow-up questions,
centralized through a gateway that manages Git integrations, AI integrations, prompts, credentials, and sessions,
portable across supported Git platforms and AI providers.

For ShopFlow, the team keeps its existing CI pipeline:
flowchart TD
    Dev["Developer opens PR"] --> CI["CI: compile, tests, lint, dependency scan"]
    Dev --> Bots["AI-Git-Bot reviewer personas"]
    CI --> Status["Git status checks"]
    Bots --> Reviews["Persona-specific PR reviews"]
    Status --> HumanReview["Human review"]
    Reviews --> HumanReview
    HumanReview --> Merge{"Ready to merge?"}
    Merge -- "No" --> Fix["Developer fixes findings"]
    Fix --> CI
    Merge -- "Yes" --> Main["Merge to main"]
The important shift is that code quality is no longer treated as one generic review question. Instead, it becomes a structured set of perspectives.

3. Analysis of Possible Solutions
Option A: One Generic AI Reviewer
A team can create one bot with the default review prompt. This is easy and already valuable. The default prompt asks the AI to look for correctness bugs, security vulnerabilities, performance problems, concurrency issues, API or database concerns, missing tests, and maintainability problems.
For small teams or low-risk repositories, this may be sufficient. The downside is that the reviewer has to cover everything at once. In practice, broad reviews often produce broad findings. They may mention many categories without going deeply into the most important ones.
Option B: Specialized Reviewer Personas
Instead of one broad reviewer, the team creates multiple bots with different system prompt entries. Each bot focuses on a narrow review mission.
For ShopFlow, the team configures three personas:

Persona
Bot name in Git
Primary mission
Typical findings

Security Reviewer
shopflow-security-ai
Protect data, identity, permissions, secrets, and inputs
IDOR, missing authorization, unsafe token handling, injection risk

Reliability & Performance Reviewer
shopflow-reliability-ai
Protect production stability and scalability
non-idempotent retries, N+1 queries, race conditions, slow queries

Testability & Maintainability Reviewer
shopflow-maintainability-ai
Protect long-term changeability
missing tests, unclear abstractions, excessive coupling, unreadable business rules

This approach gives the team three independent review passes. Each bot reads the same diff, but its system prompt changes the evaluation lens.
graph TD
    PromptSec["System Prompt Entry:\nSecurity Review"] --> BotSec["Bot:\nshopflow-security-ai"]
    PromptRel["System Prompt Entry:\nReliability & Performance Review"] --> BotRel["Bot:\nshopflow-reliability-ai"]
    PromptMaint["System Prompt Entry:\nTestability & Maintainability Review"] --> BotMaint["Bot:\nshopflow-maintainability-ai"]

    AI["AI Integration\nCloud or local model"] --> BotSec
    AI --> BotRel
    AI --> BotMaint

    GitInt["Git Integration\nGitea / GitHub / GitLab / Bitbucket"] --> BotSec
    GitInt --> BotRel
    GitInt --> BotMaint

    BotSec --> PR["Pull request reviews"]
    BotRel --> PR
    BotMaint --> PR
Option C: Combine Writer Agent, Coding Agent, and Review Personas
AI-Git-Bot also supports issue-based agent workflows. A writer bot can improve vague issues into structured, testable follow-up issues. A coding bot can implement assigned issues and open pull requests. The reviewer personas then inspect the resulting PR.
For ShopFlow, this creates a complete quality loop:

A product owner writes a vague issue: “Make checkout faster and support express payment.”
A writer bot turns it into an implementation-ready issue with acceptance criteria, non-goals, and test cases.
A coding agent or developer implements the change.
The three reviewer personas review the pull request.
A human reviewer decides what must be fixed before merge.

sequenceDiagram
    participant PO as Product Owner
    participant Git as Git Platform
    participant Writer as Writer Bot
    participant Dev as Developer or Coding Agent
    participant Reviewers as Reviewer Personas
    participant Human as Human Reviewer

    PO->>Git: Create vague ShopFlow checkout issue
    PO->>Git: Assign writer bot
    Git->>Writer: Issue assignment webhook
    Writer->>Git: Ask clarifying questions or create AI Created Issue
    Dev->>Git: Implement issue and open PR
    Git->>Reviewers: PR review requested for each persona
    Reviewers->>Git: Post specialized review feedback
    Human->>Git: Review findings and approve or request changes
Recommended Solution
For a production project like ShopFlow, the best balance is Option B plus selected parts of Option C:

Use three reviewer personas for every medium- or high-risk pull request.
Use a writer bot for unclear issues before implementation starts.
Keep CI, dependency scanning, and human review as mandatory gates.
Treat bot findings as structured expert input, not as automatic merge decisions.

4. Implementing the Solution
4.1 Configure AI-Git-Bot as the Gateway
In AI-Git-Bot, each bot connects:

one Git integration,
one AI integration,
one system prompt entry,
one bot identity and webhook URL.

For ShopFlow, the team creates:

a Git integration for its Git platform,
an AI integration for the chosen model provider,
three system prompt entries,
three coding bots with review-focused prompts,
optionally one writer bot for issue refinement.

The gateway pattern is useful because secrets, prompts, sessions, model choices, and Git-provider details are managed centrally rather than scattered across repositories.
flowchart LR
    subgraph Git["Git System"]
        Repo["ShopFlow repository"]
        PR["Pull requests"]
        Issues["Issues"]
    end

    subgraph Gateway["AI-Git-Bot Gateway"]
        Bots["Bots"]
        Prompts["System prompt entries"]
        Sessions["Review sessions"]
        Secrets["Encrypted tokens and API keys"]
    end

    subgraph Providers["AI Providers"]
        Cloud["Cloud AI"]
        Local["Ollama / llama.cpp"]
    end

    Repo  Gateway
    PR  Gateway
    Issues  Gateway
    Bots --> Prompts
    Bots --> Sessions
    Bots --> Secrets
    Gateway  Cloud
    Gateway  Local
4.2 Create the Three Reviewer Personas
The system prompts below are intentionally short enough to maintain predictable output, but strict enough to shape each review. In AI-Git-Bot, these would be stored under System settings → System prompts as separate prompt entries. The Review System-Prompt field is the important field for pull-request reviews and PR conversations.
Persona 1: Security Reviewer
Purpose: identify security defects before they become production vulnerabilities.
Example Review System-Prompt:
You are a senior application security reviewer for the ShopFlow e-commerce platform.

Review the pull request diff only from a security and abuse-resistance perspective.
Focus on authentication, authorization, tenant/user isolation, payment data handling,
input validation, injection risks, secrets, logging of sensitive data, dependency exposure,
and prompt-injection or instruction-handling risks.

Prioritize findings that could expose customer data, payment data, admin actions,
or internal credentials. Do not invent vulnerabilities that are not supported by the diff.

Format the review as:
1. Blocking security issues
2. Non-blocking hardening suggestions
3. Security tests to add
4. Overall security assessment

How this results in the Git system:

The team creates a bot named shopflow-security-ai.
The bot uses the Security Review system prompt entry.
The Git provider is configured with the bot’s webhook URL.
When shopflow-security-ai is assigned as reviewer, AI-Git-Bot posts a security-focused review.
Developers can ask follow-up questions in the PR with @shopflow-security-ai.

In the express-checkout PR, this bot might comment:

“Blocking: the saved-card lookup uses cardId without verifying ownership by the current user.”
“Add a regression test proving user A cannot use user B’s saved payment method.”
“Avoid logging payment-provider request payloads because they may contain sensitive identifiers.”

Persona 2: Reliability & Performance Reviewer
Purpose: prevent production incidents, scalability bottlenecks, and operationally unsafe behavior.
Example Review System-Prompt:
You are a reliability and performance reviewer for the ShopFlow e-commerce platform.

Review the pull request diff for production stability, scalability, latency, resource usage,
database efficiency, concurrency, transaction boundaries, retry behavior, idempotency,
timeouts, fallback behavior, and observability.

Prioritize issues that can cause outages, duplicate orders, retry storms, slow checkout,
deadlocks, memory pressure, or hard-to-debug production failures. Be concrete and explain
what load or failure mode would trigger the issue.

Format the review as:
1. Blocking reliability/performance issues
2. Non-blocking operational improvements
3. Load, concurrency, or failure-mode tests to add
4. Overall production-readiness assessment

How this results in the Git system:

The team creates a bot named shopflow-reliability-ai.
The bot uses the Reliability & Performance Review prompt entry.
The bot is requested on PRs touching checkout, database access, async jobs, or integrations.
AI-Git-Bot fetches the diff and posts findings as a PR review or review comments.
Follow-up discussions stay inside the Git pull request.

In the express-checkout PR, this bot might comment:

“Blocking: payment retries are not idempotent; a timeout after provider success could create duplicate orders.”
“The new discount query is executed once per cart item and may create an N+1 query pattern.”
“Add a test for concurrent checkout submissions with the same cart.”

Persona 3: Testability & Maintainability Reviewer
Purpose: protect long-term code health, reduce regression risk, and improve developer velocity.
Example Review System-Prompt:
You are a testability and maintainability reviewer for the ShopFlow e-commerce platform.

Review the pull request diff for clarity, cohesive design, readable business rules,
low coupling, meaningful naming, consistency with surrounding code, regression risk,
and sufficient automated tests. Focus on whether future developers can safely understand,
modify, and test the change.

Avoid minor style nitpicks unless they materially affect readability or consistency.
Prefer actionable suggestions with concrete refactoring or test recommendations.

Format the review as:
1. Blocking maintainability or testability issues
2. Non-blocking design/readability suggestions
3. Missing or weak tests
4. Overall maintainability assessment

How this results in the Git system:

The team creates a bot named shopflow-maintainability-ai.
The bot uses the Testability & Maintainability Review prompt entry.
The bot is assigned to feature PRs and refactoring PRs.
Its findings appear in the same Git review discussion as human comments, CI results, and other bot reviews.
Developers can ask targeted questions such as @shopflow-maintainability-ai how would you split this service?.

In the express-checkout PR, this bot might comment:

“The discount-stacking rules are embedded in CheckoutService; extract a DiscountPolicy so each rule can be tested independently.”
“The tests cover the happy path but not conflicting promotions or expired discounts.”
“The migration changes order state semantics; add an integration test for existing pending orders.”

4.3 Define Review Routing Rules
The team should not request every persona on every tiny change. A practical routing matrix keeps review signal high.

Change type
Security
Reliability & Performance
Testability & Maintainability

Authentication or authorization
Required
Optional
Required

Payment or checkout flow
Required
Required
Required

Database migration
Optional
Required
Required

UI-only text change
Optional
Optional
Optional

Business-rule refactoring
Optional
Optional
Required

Background jobs or queues
Optional
Required
Required

Dependency upgrade
Required for security-sensitive libs
Optional
Optional

This matrix can be implemented as a team convention. For example, pull-request templates can contain a checklist asking authors which AI reviewer personas they requested.
4.4 Keep Review Output Actionable
The most important prompt-design rule is to force useful structure. Each persona should separate:

blocking issues,
non-blocking suggestions,
recommended tests,
an overall assessment.

This maps well to how teams already work in Git:

blocking issues become required fixes before merge,
non-blocking suggestions can become follow-up issues,
test recommendations become new commits in the PR,
overall assessments help the human reviewer judge residual risk.

flowchart TD
    Finding["Bot finding"] --> Classify{"Finding type"}
    Classify -- "Blocking" --> FixNow["Fix in current PR"]
    Classify -- "Non-blocking" --> FollowUp["Create follow-up issue if valuable"]
    Classify -- "Test gap" --> AddTests["Add or improve tests"]
    Classify -- "Uncertain" --> Ask["Ask @bot or human owner for clarification"]
    FixNow --> NewCommit["Push new commit"]
    AddTests --> NewCommit
    NewCommit --> Rerequest["Re-request relevant bot review"]
4.5 Evaluate and Improve the Prompts
Prompts are part of the quality system and should be reviewed like code. Before rolling out new reviewer personas broadly, the team should:

collect representative historical PR diffs,
define a scoring rubric for correctness, security awareness, actionability, concision, and false positives,
compare the default prompt with the specialized prompt,
test malicious or confusing PR content to ensure the bot ignores instructions inside diffs or comments,
start with one repository or one team before expanding.

AI-Git-Bot’s reusable system prompt entries make this manageable. A team can clone a prompt entry, adjust the wording, assign it to one bot, observe review quality, and then promote it more broadly.
4.6 Example End-to-End Flow in ShopFlow
The following scenario shows the system in practice:

A developer opens a PR for express checkout.
CI starts automatically.
The developer requests reviews from:

shopflow-security-ai,
shopflow-reliability-ai,
shopflow-maintainability-ai.

AI-Git-Bot receives review-request webhooks.
Each bot fetches the PR diff and uses its configured system prompt.
Each bot posts a focused review.
The developer fixes blocking issues and asks follow-up questions with bot mentions.
The developer re-requests only the relevant bot reviews.
A human reviewer uses the bot reviews plus CI status to make the final decision.

sequenceDiagram
    participant Dev as Developer
    participant Git as Git Platform
    participant Gateway as AI-Git-Bot
    participant Sec as Security AI
    participant Rel as Reliability AI
    participant Maint as Maintainability AI
    participant Human as Human Reviewer

    Dev->>Git: Open express-checkout PR
    Dev->>Git: Request three AI reviewers
    Git->>Gateway: Webhook for review requests
    Gateway->>Git: Fetch PR diff and context
    Gateway->>Sec: Send diff with security prompt
    Gateway->>Rel: Send diff with reliability prompt
    Gateway->>Maint: Send diff with maintainability prompt
    Sec-->>Gateway: Security review
    Rel-->>Gateway: Reliability review
    Maint-->>Gateway: Maintainability review
    Gateway->>Git: Post reviews/comments
    Dev->>Git: Push fixes and ask @bot follow-ups
    Human->>Git: Final approval or request changes

Conclusion
AI-Git-Bot improves software quality when it is treated as a configurable review gateway rather than a single generic chatbot. By creating multiple reviewer personas, teams can make quality concerns explicit and repeatable:

the Security Reviewer protects users, secrets, permissions, and sensitive data,
the Reliability & Performance Reviewer protects production stability and scalability,
the Testability & Maintainability Reviewer protects long-term code health.

In the ShopFlow example, the same pull request receives three independent perspectives before a human reviewer makes the merge decision. This creates a stronger review process without replacing existing best practices such as CI, tests, static analysis, dependency scanning, and human ownership.
The practical result in the Git system is simple: different bot identities appear as different reviewers, each guided by its own system prompt, each posting focused feedback directly into the pull request. Over time, this turns quality standards from informal reviewer preferences into a visible, repeatable, and continuously improvable review system.

Short Note: Getting and Installing the Gateway
AI-Git-Bot is available from the project repository at https://github.com/tmseidel/ai-git-bot. The documented Docker image is tmseidel/ai-git-bot:latest, published on Docker Hub as tmseidel/ai-git-bot.
For a quick installation, clone the repository and start the provided Docker Compose setup:
git clone https://github.com/tmseidel/ai-git-bot.git
cd ai-git-bot
docker compose up -d

The gateway then runs as a web application, typically on port 8080, where administrators can create AI integrations, Git integrations, system prompt entries, and reviewer bots.

---

### Running Docker Engine in WSL2 alongside Docker Desktop for Windows

URL: https://remus-software.org/articles/kb_two_docker/
Date: April 11, 2026
Description: A practical guide to running both Docker Engine natively in WSL2 and Docker Desktop for Windows in complete isolation — diagnosing conflicts, fixing networking issues, and configuring coexistence.
Tags: Docker, WSL2, DevOps, Windows

This guide explains why you might need both Docker Engine (installed natively in WSL2) and Docker Desktop for Windows, what conflicts arise when both are present, and how to configure them to run in complete isolation.
Why You Need Both
Docker Engine in WSL2 — For Local Development
For local development and testing, you run containers directly in WSL2 using the native Docker Engine (installed via apt from Docker’s official repository).
Key advantages:

Containers share the WSL2 network stack — host.docker.internal:host-gateway routes to the real host IP (172.x.x.x), so containers can reach services running on the host (e.g., a web server on port 8080)
Fast, lightweight, and fully Linux-native
No dependency on a Windows GUI application

Docker Desktop — Why It Might Be Installed
There are several reasons why you might also have Docker Desktop for Windows installed:

Remote SSH deployments from Windows: Docker Compose supports deploying to remote servers via SSH contexts (docker context create my-server --docker "host=ssh://user@server"). When running this from a Windows terminal (PowerShell, CMD), Docker Desktop provides the Docker daemon. While SSH-based Docker contexts work perfectly fine from native Linux (including WSL2), some workflows require running docker compose from the Windows side — e.g., CI/CD scripts running on Windows, or using Windows-native tooling.
Windows container support: Docker Desktop can run Windows containers (for .NET Framework apps, IIS, etc.), which Docker Engine in WSL2 cannot.
GUI-based management: Docker Desktop provides a graphical dashboard for container management, image browsing, and resource configuration.
Corporate/team requirement: Your organization may standardize on Docker Desktop for license compliance or support reasons.
Kubernetes integration: Docker Desktop includes a single-node Kubernetes cluster for local testing.

The key point is: you don’t need to uninstall Docker Desktop — you just need to prevent it from interfering with the Docker Engine in WSL2.
Summary

Use Case
Tool
Where to Run

Local development containers
Docker Engine in WSL2
WSL2 terminal

Host service + container integration testing
Docker Engine in WSL2
WSL2 terminal

SSH remote deployment from WSL2
Docker Engine in WSL2
WSL2 terminal

Windows containers, GUI management, K8s
Docker Desktop
Windows terminal / PowerShell

Corporate/team-mandated Docker usage on Windows
Docker Desktop
Windows terminal / PowerShell

The Conflict
When Docker Desktop is installed, it enables WSL2 Integration by default (Settings → Resources → WSL Integration). This injects Docker Desktop’s own binaries and configuration into your WSL2 distro, overriding the native Docker Engine. The two cannot coexist when this integration is active.
What Goes Wrong

Symptom
Cause

host.docker.internal resolves to 192.168.65.254 instead of 172.17.0.1
Docker Desktop routes through its internal VM gateway, not the WSL2 bridge

curl http://host.docker.internal:<port> fails from containers
The VM gateway (192.168.65.254) does not forward to WSL2 host ports

/run/docker.sock or /var/run/docker.sock does not exist
Docker Desktop replaces the socket with its own, and the native docker.socket systemd unit gets confused

docker context ls shows desktop-linux context
Docker Desktop injected its context into WSL2

docker ps shows different containers than expected
CLI is talking to Docker Desktop’s daemon instead of the local one

Webhooks or callbacks from containers never reach a host service
Container → host.docker.internal → Docker Desktop VM → dead end (not forwarded to WSL2)

--network host still doesn’t reach host services
On Docker Desktop, host mode connects to the VM’s network, not WSL2’s

Understanding the Routing Problem
When a container needs to reach a service on your WSL2 host (e.g., a web application running on port 8080), it uses host.docker.internal. How this hostname resolves determines whether the connection works.
With Docker Desktop WSL2 Integration (broken):
Container
  → host.docker.internal (192.168.65.254)
    → Docker Desktop VM
      → Windows host
        ✘ WSL2 host (not forwarded)

The traffic goes through Docker Desktop’s internal VM, reaches the Windows host, but is never forwarded to the WSL2 instance where your service is actually running.
With native Docker Engine in WSL2 (working):
Container
  → host.docker.internal (172.17.0.1)
    → WSL2 host ✔ (direct bridge route)

The traffic goes directly over the Docker bridge network to the WSL2 host — no intermediate VM, no forwarding issues.
Diagnosis
If you suspect Docker Desktop is interfering with your WSL2 Docker Engine, run these commands inside your WSL2 terminal:
1. Check which Docker binary is active
which docker
ls -la $(which docker)

✅ Native: /usr/bin/docker owned by root
❌ Desktop override: symlink to /mnt/wsl/docker-desktop/... or similar

2. Check Docker contexts
docker context ls

✅ Only default pointing to unix:///var/run/docker.sock
❌ A desktop-linux context exists (Docker Desktop injected it)

3. Check the Docker socket
ls -la /run/docker.sock

✅ Socket file exists: srw-rw---- root docker ... /run/docker.sock
❌ File does not exist, or is a symlink to a Docker Desktop path

4. Check host.docker.internal from inside a container
docker run --rm --add-host=host.docker.internal:host-gateway \
  busybox cat /etc/hosts | grep host.docker

✅ 172.17.0.1  host.docker.internal (WSL2 Docker bridge gateway)
❌ 192.168.65.254  host.docker.internal (Docker Desktop VM gateway)

5. Test container-to-host connectivity
If you have a service running on the host (e.g., on port 8080):
docker run --rm --add-host=host.docker.internal:host-gateway \
  curlimages/curl curl -s -o /dev/null -w "HTTP %{http_code}" \
  --connect-timeout 3 http://host.docker.internal:8080/

✅ HTTP 200 (or any non-zero HTTP status)
❌ HTTP 000 (connection failed)

Resolution
Step 1: Disable Docker Desktop WSL2 Integration
This is the critical step. In Docker Desktop:

Open Docker Desktop
Go to Settings → Resources → WSL Integration
Disable “Enable integration with my default WSL distro”
Disable the toggle for your Ubuntu (or other) WSL2 distro
Click Apply & Restart

Docker Desktop will continue working from Windows terminals (PowerShell, CMD), but it will stop injecting itself into WSL2.
Step 2: Remove the Docker Desktop context from WSL2
docker context rm desktop-linux

Verify only the native context remains:
docker context ls

Expected output:
NAME        DESCRIPTION                               DOCKER ENDPOINT
default *   Current DOCKER_HOST based configuration   unix:///var/run/docker.sock

Step 3: Restart the native Docker Engine
If the Docker socket is missing after disabling Desktop integration:
sudo systemctl stop docker docker.socket
sudo systemctl start docker.socket
sudo systemctl start docker

Verify the socket exists:
ls -la /run/docker.sock
# Expected: srw-rw---- 1 root docker 0 ... /run/docker.sock

Step 4: Verify container-to-host connectivity
Start any service on the host (or use a simple test server), then:
# Quick test: start a temporary HTTP server on the host
python3 -m http.server 9999 &
TEST_PID=$!

# Test from a container
docker run --rm --add-host=host.docker.internal:host-gateway \
  curlimages/curl curl -s -o /dev/null -w "HTTP %{http_code}" \
  --connect-timeout 3 http://host.docker.internal:9999/

# Clean up
kill $TEST_PID

Expected result: HTTP 200
Step 5: Recreate your running containers
Existing containers were created under the old networking configuration. You need to recreate them for the fix to take effect:
docker compose -f <your-compose-file.yml> down
docker compose -f <your-compose-file.yml> up -d

Note: If your compose file uses named volumes and you’re also changing container image versions, you may need down -v to remove old volumes. Only use -v if you don’t mind losing persisted data.

Running in Isolation
After completing the resolution steps, the two Docker installations run independently:

Docker Engine (WSL2)
Docker Desktop (Windows)

Access from
WSL2 terminal (bash)
Windows terminal (PowerShell, CMD)

Daemon
dockerd via systemd in WSL2
Docker Desktop VM

Socket
/run/docker.sock
npipe:////./pipe/docker_engine

Containers
Separate set
Separate set

Networks
WSL2 bridge (172.17.0.0/16)
Desktop VM bridge (192.168.65.0/24)

host.docker.internal
172.17.0.1 (WSL2 host)
192.168.65.254 (Windows host)

Use for
Local dev, testing, SSH deployments
Windows containers, GUI, K8s

Using Docker Desktop from Windows
When you need Docker Desktop features (Windows containers, GUI, Kubernetes), open PowerShell (not WSL2):
# In PowerShell / Windows Terminal
docker ps
docker compose -f docker-compose.yml up -d

Using Docker Engine from WSL2
For local development, use your WSL2 terminal as usual:
# In WSL2 bash
docker compose -f docker-compose.yml up -d

Ensuring Docker Starts on WSL2 Boot
WSL2 doesn’t always start systemd services automatically. To ensure Docker Engine is available when you open a WSL2 terminal:
# Enable Docker to start with systemd
sudo systemctl enable docker.service
sudo systemctl enable docker.socket

If your WSL2 distro doesn’t have systemd enabled, add this to /etc/wsl.conf:
[boot]
systemd=true

Then restart WSL2 from PowerShell:
wsl --shutdown

Troubleshooting
“Interactive authentication required” when starting Docker
Failed to start docker.service: Interactive authentication required.

You ran systemctl start docker without sudo. Use:
sudo systemctl start docker

To avoid needing sudo for Docker commands (not for systemctl), add your user to the docker group:
sudo usermod -aG docker $USER
newgrp docker

Docker socket disappears after reboot
Docker Desktop’s WSL2 integration was re-enabled (e.g., after a Docker Desktop update), or the systemd socket unit didn’t start. Check and fix:
# Check if Desktop integration is back
docker context ls  # Should NOT show desktop-linux

# Restart the socket
sudo systemctl restart docker.socket docker
ls -la /run/docker.sock

If desktop-linux reappeared, Docker Desktop re-enabled WSL2 integration — repeat Step 1 of the resolution.
Container can ping host.docker.internal but can’t connect to a port
The hostname resolves but the port is unreachable. Verify:

The service is actually running on the host:
curl http://localhost:<port>/

Check what IP the container sees:
docker exec <container-name> cat /etc/hosts | grep host.docker

If 192.168.65.254 → Docker Desktop is still interfering (see Resolution)
If 172.17.0.1 → the service might not be running, might be bound to 127.0.0.1 only, or a firewall is blocking

Verify the service listens on all interfaces:
ss -tlnp | grep <port>

✅ *:<port> or 0.0.0.0:<port> — listening on all interfaces
❌ 127.0.0.1:<port> — only localhost; reconfigure the service to bind to 0.0.0.0

docker compose shows containers from the wrong Docker
If docker ps in WSL2 shows containers you created in Docker Desktop (or vice versa), the wrong context is active:
docker context ls       # Check which context is active (marked with *)
docker context use default   # Switch to native WSL2 Docker

Docker Desktop updates re-enable WSL2 integration
Docker Desktop may re-enable WSL2 integration after updates. If things suddenly break again after a Docker Desktop update, re-check Settings → Resources → WSL Integration and disable it again. Consider disabling Docker Desktop auto-updates if this becomes a recurring issue.

---

### Building an AI Agent for Code Generation: Lessons from 13 Iterations

URL: https://remus-software.org/articles/how-to-develop-an-ai-agent/
Date: April 7, 2026
Description: Practical lessons learned from developing an agentic AI system that generates source code — from naive prompting to a robust, tool-using agent with a generic loop, schema-validated plans, provider-native function calls, and an optional self-critique step.
Tags: AI, LLM, Agents, Java, Software Architecture

Building software with an AI agent at its core is fundamentally different from traditional application development. Over the course of thirteen iterations — nine on the agent’s behaviour, four more on the surrounding architecture of the host — I developed an AI agent that reads GitHub/Gitea issues, generates implementation code, validates it, and commits the result, all autonomously. This article distils the key lessons learned along the way.
The Paradigm Shift: Inversion of Control
The single biggest mental shift when building agentic systems is the inversion of control flow. In traditional software, the application owns the business logic and orchestrates every step. In an agentic system, a significant portion of that decision-making shifts to the AI — the surrounding program increasingly acts as a communication partner, executing commands on the agent’s behalf.
graph LR
    subgraph Traditional
        App[Application] -->|calls| Lib[Library / API]
        App -->|orchestrates| DB[(Database)]
    end
graph LR
    subgraph Agentic
        Agent[AI Agent] -->|requests action| Host[Host Program]
        Host -->|returns result| Agent
        Agent -->|requests tool| Host
        Host -->|executes & returns| Agent
    end
Your application becomes, to a large degree, an execution environment for the agent. It provides tools, fetches context, applies file changes, and reports results — while the agent takes on much of the reasoning about what to do. In practice, the split is not black and white — which is exactly what the next section explores.
Designing the Agent Flow: What to Keep, What to Delegate
Once you accept the inversion of control, the next critical question is: which actions belong in your application, and which do you delegate to the LLM? This is not an all-or-nothing decision — it’s a spectrum, and every point on it has trade-offs.
Keeping Logic in the Application
Deterministic steps — file I/O, git operations, API calls, JSON parsing, diff application — are natural candidates for application-side logic.
Advantages:

Predictable and fast. Same input, same output, every time. No API latency, no token cost.
Testable. You can write unit tests with clear assertions.
Debuggable. Stack traces, breakpoints, and logging work exactly as expected.

Disadvantages:

Rigid. You must anticipate every edge case upfront. An unexpected file format or an unusual error message breaks the flow.
More code to maintain. Every special case becomes an if branch or a new parser.

Delegating Logic to the LLM
Reasoning tasks — deciding what to change, interpreting error messages, choosing a fix strategy, analysing code structure — are where the LLM excels.
Advantages:

Flexible and adaptive. The model handles novel situations without explicit programming.
Reduces code complexity. A single prompt can replace hundreds of lines of hand-coded decision trees.
Understands intent. The AI can reason about why something should change, not just what the syntax rules say.

Disadvantages:

Non-deterministic. The same input can produce different outputs on each run.
Slower. Each LLM call adds 5–30 seconds of latency depending on the model and input size.
Token cost. Every delegation costs money and consumes context window space.
Hallucination risk. The model may confidently produce incorrect results.
Harder to test. Assertions on LLM output are inherently fuzzy.

Finding the Right Split
In practice, the division that worked best for our code generation agent was:
flowchart LR
    subgraph "Application (deterministic)"
        A[File I/O]
        B[Git operations]
        C[Diff application]
        D[JSON parsing]
        E[Tool execution]
    end
    subgraph "LLM (reasoning)"
        F[Code generation]
        G[Error analysis]
        H[Fix strategy]
        I[Choosing validation tools]
        J[Context requests]
    end
    E -->|results| G
    G -->|decisions| A
The guiding principle: if a step requires understanding intent or adapting to ambiguity, delegate it. If it requires reliability and speed, keep it in the application. As you will see throughout the iterations below, we gradually moved more logic to the AI side — not because we wanted to, but because the deterministic alternatives kept failing on edge cases.
The Tightrope Walk: Context Window vs. Output Quality
One of the most persistent challenges is balancing the context window. Too little context, and the AI hallucinates imports, invents method signatures, or misses existing patterns. Too much context, and the model loses focus, produces lower-quality output, or exceeds token limits.

The sweet spot shifts with every model generation, but the tension never goes away.

This interplay shaped almost every iteration of the agent.
The Iteration Dilemma: How Many Loops?
Closely related to the context problem is the question of how many iterations to allow between your application and the LLM. LLMs have a remarkable ability to improve their output when given feedback — a compilation error sent back to the model often results in a correct fix on the second attempt. This makes iterative correction loops very attractive.
But iteration comes at a price:

Time. Each round-trip adds 10–30 seconds of latency. Three retry loops turn a 15-second task into a two-minute task.
Context bloat. Every iteration appends messages to the conversation — the error report, the AI’s response, the next error report. The context window fills up fast, which degrades output quality (see above).
Hallucination drift. Counter-intuitively, too many iterations can make things worse. After three or four failed attempts, the AI tends to “drift” — introducing new errors while fixing old ones, inventing methods that don’t exist, or producing solutions that look plausible but are semantically wrong. The model starts optimising for passing the immediate check rather than being correct.
Cost. Each iteration consumes tokens. With large context windows, a single retry can cost as much as the original request.

Practical Guards
Through experimentation, we arrived at the following guidelines:

Hard caps on every loop. No open-ended retries. We used 3 rounds for file requests, 5 rounds for code validation loops, and 3 rounds for diff recovery attempts.
Progress monitoring. If the error count isn’t decreasing between iterations, abort early. An AI that produces more errors after a fix attempt is unlikely to recover.
Compaction between rounds. Summarise earlier turns aggressively to keep the context lean (see Iteration 3).
Escalation, not repetition. If simple “fix this error” prompts fail twice, escalate to a richer strategy — provide more context, rephrase the problem, or fall back to a full file regeneration instead of incremental fixes.

Rule of thumb: If your agent hasn’t solved the problem in three iterations, throwing more iterations at it is unlikely to help — you need a different strategy, not more attempts.

Core Principles
Before diving into the iterations, here are the overarching lessons:

Always give the AI the ability to request more information. Never assume your initial context is sufficient. Let the agent ask for files, type definitions, or documentation on demand.
Define a protocol in the system prompt for structured output — but build your application to be resilient against protocol violations. The AI will sometimes deviate from the agreed JSON schema, return partial responses, or mix formats. Your parser must handle this gracefully.
The agent drives the reasoning — your code provides the infrastructure and guardrails.

Iteration 1 — Naive Generate-and-Validate Loop
The first version was straightforward: send the issue description to the AI, receive generated code, then validate it.
flowchart TD
    A[Issue Description] --> B[AI generates code]
    B --> C[CodeValidationService validates all files]
    C --> D{Errors found?}
    D -->|Yes| E[Build error report]
    E --> F["Send to AI: 'Fix these syntax errors'"]
    F --> B
    D -->|No| G[Commit code]
    D -->|Max retries reached| H[Post warning in issue & commit anyway]
What worked: The basic loop caught obvious syntax errors and gave the AI a chance to self-correct.
What didn’t work: The AI frequently generated code that referenced classes, methods, or interfaces it had never seen. Without sufficient context about the existing codebase, the output was often structurally correct but semantically wrong.
Iteration 2 — Smarter Context Gathering
The root cause from Iteration 1 was clear: the AI didn’t know enough about the existing codebase. The fetchRelevantFileContents() method was significantly improved:

Package awareness: When a file is mentioned, all files in the same package are loaded.
Partial name matching: The word “Task” in an issue matches Task.java, TaskService.java, TaskRepository.java, etc.
Domain structure recognition: Files in /domain/, /model/, /entity/, /config/, /dto/, /repository/, /service/, and /controller/ directories are included when they relate to the issue.
Increased file limit: From 15 to 30 files.

This gave the AI visibility into existing method signatures, interface definitions, inheritance hierarchies, and repository methods — drastically reducing hallucinated references.
Lesson: Context quality matters more than prompt engineering. A perfectly worded prompt with missing context will always lose to a mediocre prompt with complete context.
Iteration 3 — Conversation Compaction
With richer context and multi-turn code review conversations, the context window filled up fast. After several rounds of back-and-forth, the conversation could easily exceed 100 KB of tokens.
The solution: automatic compaction after every code review interaction. The system retains only the last four messages plus a short summary of the earlier conversation.
flowchart LR
    A["100 KB+ conversation"] --> B[Compaction]
    B --> C["Summary + last 4 messages"]
    C --> D["~10 KB context"]
Lesson: LLMs work best with focused context. Aggressively summarise historical turns — the AI doesn’t need the full transcript, just the current state and a brief recap.
Iteration 4 — Prompt Deduplication
A careful audit of all prompts revealed massive redundancy:

Instructions already present in the system prompt were repeated in every user prompt.
The full repository tree was sent with every continuation, even though the AI already had it from the previous turn.
Verbose formatting instructions (Markdown headers, extra blank lines) were duplicated.

Changes made:

"Output your response as a JSON object with the structure described in the system prompt" → "Output JSON per system prompt format"
treeContext removed from buildContinuationPrompt — the AI retains it from the conversation history.
Repeated formatting directives eliminated.

Lesson: Treat your prompts like production code. Audit them for duplication, dead instructions, and unnecessary verbosity. Every wasted token is context the AI could have used for actual reasoning.
Iteration 5 — Diff-Based Updates and Dynamic File Requests
This was the most impactful single iteration. Two major features were introduced:
Diff-Based Changes
Instead of returning entire files for every small change, the AI now returns SEARCH/REPLACE diffs:
{
  "fileChanges": [
    {
      "path": "src/main/java/com/example/Task.java",
      "operation": "UPDATE",
      "diff": "<<<<<<< SEARCH\nprivate String name;\n=======\nprivate String name;\nprivate String description;\n>>>>>>> REPLACE"
    }
  ]
}

A new DiffApplyService applies these blocks to the actual file content.
Dynamic File Requests
The AI can now respond with a file request instead of code changes:
{
  "summary": "Need more context about the repository interface",
  "requestFiles": ["src/main/java/com/example/TaskRepository.java", "pom.xml"]
}

The host program fetches the requested files and continues the conversation.
Token savings were dramatic:

Scenario
Before
After
Saving

Small change in a 500-line file
~500 lines
~10 lines (diff)
~98%

Follow-up without new files
Tree + file list
Only comment
~90%

Iterative requests
All files again
Only requested files
~70%

Lesson: Give the AI the tools to be efficient. Diff-based output and on-demand file requests transformed a chatty, wasteful interaction into a focused, surgical one.
In hindsight, though, this was an important transitional design, not the final one. Diff-based updates reduced token usage, but they also introduced a fragile mini-language that the host application had to interpret and repair.
Iteration 6 — Robust Diff Application
Real-world diffs from the AI are messy. The DiffApplyService had to handle numerous edge cases:

Empty SEARCH blocks — content is appended to the file.
Placeholder comments like /* Add existing... */ — treated as append operations.
Append patterns — when the REPLACE block starts with the SEARCH content and adds more, only the new part is appended.
Trailing whitespace differences — a fuzzy match is attempted before failing.

Additionally, the IssueImplementationService was ignoring AI responses that contained requestFiles but no fileChanges, returning null instead. The fix:

Detect requestFiles even when fileChanges is empty.
Fetch the requested files and continue the conversation.
Allow a maximum of three rounds of file requests to prevent infinite loops.

Lesson: The interface between AI output and your application is inherently fuzzy. Build robust parsers, add fallback strategies, and always cap iteration counts.
That said, there is a deeper lesson here: every extra recovery rule in the host is a signal that the protocol itself may be too brittle. If you keep adding fuzzy matching, placeholder handling, and recovery paths, you may be compensating for the wrong abstraction.
Iteration 7 — AI-Driven Validation with Tools
A fundamental architectural change: remove built-in validators entirely and let the AI decide how to validate its own output.
The agent prompt was updated to make tool usage mandatory:

“IMPORTANT: You MUST include runTool in every response that contains fileChanges. The bot does not have built-in validators — only you can determine how to validate the code by executing external tools.”

The AI now specifies a validation command (e.g., mvn compile, npm run build, gradle check) alongside its code changes. If it forgets, the host program sends it back with a reminder.
flowchart TD
    A[AI returns fileChanges + runTool] --> B[Host applies file changes]
    B --> C[Host executes specified tool]
    C --> D{Tool output}
    D -->|Success| E[Commit]
    D -->|Failure| F[Send tool output back to AI]
    F --> A
    G[AI returns fileChanges WITHOUT runTool] --> H["Host: 'Please specify a validation tool'"]
    H --> A
Lesson: The AI often knows better than a hardcoded validator what constitutes “correct” in a given context. A Java project needs mvn compile; a Node project needs npm run build; a Python project might need pytest. Let the agent choose.
The Tool Selection Problem
Letting the AI choose which tool to run immediately raises a thorny question: which tools do you allow it to execute at all?
This is one of the hardest design decisions in an agentic system, and the answer should be conservative: allow the absolute minimum set of tools needed to get the job done. Every tool you expose is:

A security risk. A build command like mvn compile is safe. An arbitrary shell command is not. The distance between “run my tests” and rm -rf / is one hallucinated token.
A source of complexity. More tool types mean more parsing, more error handling, more edge cases in your host program.
A surface for misuse. The AI might call tools in unexpected ways, with unexpected arguments, or in unexpected order. The more tools available, the larger the space of possible (and possibly harmful) interactions.

For our code generation agent, the minimal toolset was:

Tool
Purpose

Read file
Fetch source code from the repository

File tools (write-file, patch-file, mkdir, delete-file)
Modify source code

Execute build command
Validate changes (mvn compile, npm run build, etc.)

Request additional files
Ask for more context

We deliberately did not expose: arbitrary shell access, database queries, network requests, or deployment commands. Each tool that didn’t make this list was considered and rejected because it either wasn’t strictly necessary or introduced unacceptable risk.
Sandboxing is essential. Even with a minimal toolset, run tool executions in an isolated environment. Restrict file system access to the project directory. Set timeouts on build commands. Log every tool invocation for audit. The AI is not malicious, but it is unpredictable — and unpredictable + powerful = dangerous.

Principle: Start with zero tools and add them only when the agent demonstrably cannot complete its task without them. Resist the temptation to expose “just one more” convenience tool.

Iteration 8 — Resilient JSON Parsing
With complex multi-turn conversations, the AI occasionally produced truncated or malformed JSON — especially near token limits. The repairTruncatedJson method was overhauled:

Check completeness first: Verify whether brackets are balanced before attempting any repair.
Only truncate genuinely incomplete JSON — previously, valid JSON was sometimes mangled by premature repair attempts.
Add @NoArgsConstructor to all DTO classes to ensure Jackson can deserialize partial objects.
Parse runTool as a proper typed object (AiToolRequest) instead of a raw map.

Lesson: When you define a structured protocol in the system prompt, the AI will follow it most of the time — perhaps 95%. Your application must handle the other 5% gracefully. Invest in resilient parsing, not stricter prompts.
Iteration 9 — AI-Assisted Diff Recovery
Even with all the fuzzy matching from Iteration 6, diffs still sometimes failed to apply — typically because the file had been modified by a previous step in the same conversation, and the SEARCH block no longer matched.
The elegant solution: ask the AI to resolve it.
When a DiffApplyException occurs:

Fetch the current file content from the repository.
Send both the current content and the failed diff to the AI.
Ask it to produce the complete new file content.

flowchart TD
    A[Apply diff] --> B{DiffApplyException?}
    B -->|No| C[Success]
    B -->|Yes| D[Fetch current file from repo]
    D --> E["Send to AI:\n• Current file content\n• Failed diff\n• 'Produce complete new file'"]
    E --> F[AI returns full file content]
    F --> G[Use directly — no diff needed]
This is far more robust than implementing ever-more-complex matching strategies in the DiffApplyService. The AI sees the actual current state of the file and can produce the intended result directly.
Lesson: When your deterministic code fails, don’t add more deterministic complexity — delegate back to the AI. It can reason about intent in ways that string matching never will.
Postscript — Tool Requests Beat Fragile Diff Protocols
After publishing the original version of this architecture, I revisited a later iteration of the agent and checked how the design had evolved over time. The pattern was clear: there were several rounds of hardening around diff handling, but eventually the fileChanges mechanism was removed entirely and replaced with tool-based file operations.
That refactoring switched the protocol from describing edits as a custom JSON diff format to requesting concrete actions:

write-file
patch-file
mkdir
delete-file

All of these are executed through runTools, alongside validation commands. In other words, file mutation stopped being a special side channel and became just another tool request.
This turned out to be much more robust for three reasons:

The contract is simpler. The host executes explicit operations instead of parsing and heuristically applying a synthetic diff language.
Failures are clearer. patch-file either finds the exact text once or it fails with a precise error; there is less “maybe this is close enough” behavior.
The conversation model is cleaner. The agent can use requestTools to inspect files first, then issue runTools for exact changes and validation in the next round.

The most important updated lesson is this: when the bridge between agent and host becomes fragile, prefer executable tool requests over clever output formats. A protocol that asks the model to say what to do is usually sturdier than one that asks it to emit a compact patch language the host must interpret.
Diffs were still a valuable intermediate step. They reduced token usage and forced a more structured contract. But in practice, explicit tool requests turned out to be the more durable endpoint.
Iteration 10 — From a Hand-Wired Loop to a Generic Agent Loop with Strategies
By the time we had a second agent in the system — one that turns vague user reports into well-formed issues, alongside the original code-generating one — the original implementation loop had been copy-pasted, renamed, and quietly diverged. Both loops did almost the same thing:

bump a round counter and check budgets,
mirror every message into the persisted session and into an in-memory list,
call the AI with (history, message, systemPrompt, modelOverride, maxTokens),
parse the response into a plan,
branch on “context request” vs. “tool run” vs. “final”,
and call back into the agent-specific finisher.

The divergences were the dangerous part. One loop carried an attempt-- hack that decremented the validation counter whenever the AI asked for more context, “because context lookups shouldn’t burn an implementation attempt.” The other had a different off-by-one on the same idea. Neither could be reasoned about without reading the surrounding 200 lines.
The refactor
Instead of designing the new loop top-down, we started bottom-up by characterising the existing one. Several new tests pinned three branch combinations that nobody dared touch: multi-round validation retry, the “ignore non-blocking tool failures after a successful build” policy, and the “file-only success without any validation tool” path. Only with those tests green did we extract:

a generic loop class that owns rounds, history mirroring, and the AI call,
a sealed StepDecision type with exactly two cases (Continue(nextMessage) / Finish(outcome)) that the strategy returns each round,
a small Strategy interface with systemPrompt(), step(ctx, aiResponse, round), and onBudgetExhausted(ctx),
an immutable Budget(maxRounds, maxContextRounds, maxValidationRetries, maxTokensPerCall) value object,
and a RunContext carrying per-run mutable state (the workspace, the current base branch, the issue identifier — whatever the agent needs to know).

The attempt-- hack was deleted. Context-fetch rounds and implementation attempts now use separate counters in the strategy. The session/history double-bookkeeping disappeared into the loop, where it belongs.
flowchart TD
    A["loop.run()"] --> B["ai.chat() / ai.chatWithTools()"]
    B --> C["strategy.step(ctx, response, round)"]
    C -->|Continue| A
    C -->|Finish| D[strategy-specific finisher]
    A -. budget exhausted .-> E[strategy.onBudgetExhausted]
Lesson

Don’t refactor what you cannot reproduce. Characterisation tests are the cheapest possible insurance policy when ripping apart a loop full of off-by-one tricks.

The second, subtler lesson surfaced months later: a sealed StepDecision interface forces every new control-flow case to opt in. We later considered adding an Abort(reason) variant for Iteration 13’s critic step — and the compiler immediately pointed at every switch that would need updating. The exhaustiveness check turned the loop from a behavioural surprise generator into something typed. That alone justified the refactor.
Iteration 11 — Schema Validation: Trust, but Verify (Quietly)
The output protocol from Iteration 5 onwards was JSON-in-prompt. We documented its shape in the system prompt, deserialised it with a standard JSON library, and patched up violations in a thicket of helpers — extract-the-JSON-from-the-fenced-block (four strategies), truncate-to-first-balanced-object, repair-truncated-JSON, sanitise-invalid-escapes, find-the-last-complete-tool-call. The pile grew with every new model we tested.
The real cost was not the helpers themselves but the invisibility of their work. We had no idea how often the AI actually violated the contract — and which fields it got wrong most often. The agent might be quietly recovering from 30 % schema violations and we wouldn’t know.
What we built

JSON-Schema documents (Draft 2020-12) for every plan shape the AI is allowed to return. They accept the de-facto aliases the models had already invented (requestedFiles next to requestFiles, singular runTool next to plural runTools[]) via oneOf, because legacy responses must remain valid.
A validator component that runs after the existing extract/repair pipeline and before deserialisation.
A counter like agent.plan.schema_violations_total{agent=...} exposed on the metrics endpoint.
A feature flag agent.schema.enforce (default false).
Snapshot tests against real captured AI responses, plus a couple of deliberately broken negatives.

The key design decision: in the default mode, the validator does not change behaviour at all. It logs the violation, increments the counter, and lets the existing repair heuristics do their job. The whole layer is observational.
Lesson

Adding a strict validator next to a forgiving parser is not a contradiction. Run them in parallel for a release or two, measure how often the strict layer would have rejected, and only then decide whether to flip enforcement on.

This is the boring-but-correct path. We did not “delete the messy parser and replace it with the proper one” — that would have broken production the same day. We added a measurement and waited.
A second lesson, hidden in the implementation: the schemas turned out to be reusable. In Iteration 12 we needed a JSON-Schema for each tool the agent can call. We already had two well-tested plan schemas to learn from.
Iteration 12 — Provider-Native Function Calling (Without Burning the Bridges)
All four major LLM providers we support now expose first-class tool calling. The provider parses the model’s tool-call intent server-side and hands you a structured { name, arguments } object. No JSON-in-prompt, no fenced code blocks, no fuzzy extraction.
We had wanted to use this from the start. We could not, because:

Not every model supports it (older fine-tunes, certain self-hosted setups, raw-completion endpoints).
Switching providers mid-flight would have invalidated months of accumulated prompt engineering.
Operators of an existing deployment may have a strong reason — model choice, cost, debuggability — to stay on the legacy path even on providers that do support tools.

So the design constraint was firm: native function calling had to be opt-out per provider configuration, with the JSON-in-prompt path remaining fully functional and fully tested.
The contract changes
record ChatTurn(String assistantText, List<ToolCall> toolCalls, StopReason stop) {}
record ToolCall(String id, String name, JsonNode args) {}
record ToolDescriptor(String name, String description, JsonNode jsonSchema) {}

interface AiClient {
    String chat(List<Message> history, String msg, String sys, String model, int maxTokens);

    // default delegates to chat(), so non-native clients work unchanged
    default ChatTurn chatWithTools(List<Message> history, String msg,
                                   List<ToolDescriptor> tools,
                                   String sys, String model, int maxTokens) {
        return ChatTurn.text(chat(history, msg, sys, model, maxTokens));
    }

    default boolean supportsNativeTools() { return false; }
}

Three things make this work:

The default method shields old clients. Anything that hasn’t been migrated keeps its plain-text chat() semantics, even when the loop asks for tools. No big-bang client rewrite.
A shared helper wraps a chat() call into a ChatTurn. Every native client uses it as its fallback when the operator-level kill switch is set.
The loop triple-gates the native path: the strategy must prefer NATIVE, the client must report supportsNativeTools() == true, and the strategy must actually expose at least one ToolDescriptor. Any missing condition falls back to the legacy path, with the reason logged.

A per-configuration kill switch
A useLegacyToolCalling flag was added to the AI-provider configuration record. The admin UI got a switch with a popover explaining both modes:

“Native tool calls give the model a structured tools API and are the recommended default. Disable this if you observe parse failures with a specific model or want to keep using the prompt-engineered protocol.”

When the operator flips the switch, the client is reconstructed with native mode disabled, so chatWithTools(...) quietly returns a ChatTurn.text(...) and the legacy text path runs end-to-end — without any change to the agent code above it.
Telemetry that finally tells the truth
Three metrics were added:

Metric
Tags
What it measures

agent.tool_call.mode_total
mode={native,legacy}, provider
One increment per AI round. Lets us see migration progress per provider.

agent.tool_call.parse_failures_total
provider
Where the model defied the protocol.

agent.tool_call.latency_seconds
mode, provider
Wall-clock per AI round.

The first metric matters more than it looks. The single hardest question in this kind of migration is “are we actually on the new path, or are we silently falling back?” A simple count per mode answers that without instrumenting the loop further.
Lesson

When you change a contract that several providers must implement, give the interface a default method that preserves the old behaviour. Then add the new behaviour as an opt-in capability, with telemetry from day one. Migrating providers becomes a measurable, reversible operation instead of a big bang.

A related, second-order lesson: the schemas from Iteration 11 were exactly the right artefact to hand to the providers’ tool-call APIs. The investment in a stricter protocol paid off precisely when we no longer needed the loose one.
Iteration 13 — Persisted State, Consolidated Budgets, and an Optional Critic
The final iteration is really three small refactors that share a theme: the agent had grown subtle bugs from implicit state. Each fix replaces an inference-from-history with an explicit, persisted value.
13a. Persisting the last plan instead of replaying history
The old “get the last plan” helper walked the conversation backwards, calling the JSON parser on every assistant message until one returned non-null. It was used in three places — the PR body, the follow-up comment, and the critic step we were about to add. It had three problems:

Performance. Long sessions re-parsed dozens of messages per call.
Inconsistency. If the plan format evolved between runs, an older message would parse differently than a fresh one would.
Brittleness in tests. “The last plan” depended on the exact mock of the history accessor, leading to several hours of debugging “why does this test see an empty plan?”.

The fix is unglamorous: three new columns on the session table (last_plan_summary, last_plan_json, last_plan_at), a recordPlan(...) method on the session service, and a single call in the strategy right after the response is parsed successfully. Downstream readers now hit the row in O(1). The history walk survives as a fallback for sessions that pre-date the migration.

Lesson: If you find yourself re-deriving the same fact from history three times in three places, persist the fact. The history is for audit, not for lookup.

13b. One budget config to rule them all
Before this iteration, the agent had at least seven overlapping limits scattered across three places: separate validation retries, max tool executions, a hard-coded MAX_CONTEXT_TOOL_REQUESTS = 5 constant, separate writer-specific knobs, a hard-coded fileRequestRounds < 3 literal in the loop, and a legacy maxTokens setting. Each one was set somewhere different and meant something subtly different.
The new BudgetConfig collapses these into five named knobs with sensible defaults: maxRounds = 10, maxContextRounds = 3, maxValidationRetries = 3, maxContextToolRequestsPerRound = 5, maxTokensPerCall = 16384. The deprecated fields stay on the config object for backwards compatibility, but a constructor hook copies them into the new struct whenever its values are still at the defaults — so an operator who customised agent.max-tokens=8192 in their YAML keeps that value, without ever touching the new property.

Lesson: When the operator-visible configuration drifts from the model the code actually uses, add a migration inside the config class. A post-construct hook (in your framework’s equivalent) is the cheapest possible adapter and keeps the documentation honest.

13c. An optional Critic / Reflection step
The last addition is intentionally small, and intentionally off by default. It is inspired by the “Self-Refine” / “Reflexion” family of prompting techniques: after the implementation has passed validation, ask a second LLM call to read the diff and answer one question — does this change actually address the issue?
The contract is a three-valued return:
{"outcome": "APPROVE" | "ITERATE" | "ABORT", "feedback": "short, actionable text"}

APPROVE proceeds to commit and PR creation, unchanged.
ITERATE posts the critic’s feedback back to the issue and resets the session to “waiting for the user to ping the bot”. (We deliberately did not feed the feedback back into the same loop. That door is open, but it has subtle failure modes — a critic that keeps demanding “more tests” can lock the loop in an unproductive cycle.)
ABORT aborts the PR creation entirely, posts the reason as a comment, and marks the session as failed.

There is also a hidden fourth outcome, SKIPPED, emitted whenever the feature is disabled (the default). It exists purely so the metric agent.critic.outcome_total{outcome=…} makes the off-state visible — zero additional LLM calls per implementation, but a non-zero count of SKIPPED increments.
agent:
  critic:
    enabled: false                       # opt-in
    max-iterations: 1
    require-approval-for: [LARGE_DIFF]   # informational, future trigger

The most important property of this design is what it does not do in the default configuration: it makes no extra AI call, allocates no extra tokens, and adds no latency. A unit test asserts exactly that — “when disabled, the AI client is never called”.
Lesson

A useful self-critique step is one that costs nothing when disabled, that always fails open in the face of unparseable AI output, and that publishes its outcome as a first-class metric. Reflection is powerful, but a critic that blocks PRs when the network blips is worse than no critic at all.

The pragmatic policy we settled on: parse failure → APPROVE. AI exception → APPROVE. Empty response → APPROVE. The only paths that can stop a PR are an explicit ITERATE or ABORT with a well-formed JSON body. The metric tags (approve, iterate, abort, skipped, error, parse_error) make every fail-open visible to operators.

Reflecting on the Architectural Iterations
Iterations 10–13 are different in flavour from 1–9. The early iterations were about what the agent does; the later ones are about the shape of the host that contains it. Three patterns kept recurring:

Sealed types beat strings. Replacing string-based control flow (if (response.contains("requestFiles"))) with a sealed StepDecision and a handful of records gave the compiler enough visibility to catch every new branch. Native function calling’s StopReason enum did the same job for provider responses.
Default methods are the migration tool nobody talks about. Both Iteration 12’s chatWithTools rollout and Iteration 13b’s budget migration leaned on the same idiom: an interface with a default implementation, or a config object with a post-construct hook that bridges old fields to new. Both were strictly additive. Neither broke a single existing call site.
A metric is the entry ticket for a feature flag. Every flag we added — schema enforcement, native tool calling, the critic — landed together with a counter that distinguishes “the flag is on” from “the flag is doing work”. Without that, “we have schema validation now” is a press release. With it, it’s an operations capability.

Summary of Key Takeaways
After thirteen iterations — nine on the agent’s behaviour, four on the architecture around it — these are the principles I’d carry into any agentic system:

Principle
Detail

Inversion of control
The agent drives the logic; your app is the execution environment.

Choose the split wisely
Keep deterministic steps in the app; delegate reasoning to the LLM. Each side has clear trade-offs.

Context is everything
Invest heavily in smart, dynamic context gathering.

Cap your iteration loops
LLMs improve with feedback, but after 3 failed attempts, change your strategy — don’t just retry.

Let the agent ask for more
Never assume you’ve provided enough information upfront.

Define a protocol, but be resilient
Structured output formats are essential, but the AI will violate them. Build robust parsers.

Minimise token waste
Use summaries, targeted context, and compact tool requests to keep the context window focused.

Delegate validation to the agent
The AI knows the build system better than a hardcoded checker.

Minimise the toolset
Expose only the tools strictly needed. Every additional tool is a security risk and a source of complexity.

Prefer tool requests over patch protocols
If diff parsing keeps getting more complex, simplify the contract and let the host execute explicit file operations.

Use the AI to fix AI failures
When diff application or parsing fails, ask the AI to resolve it.

Don’t refactor what you can’t reproduce
Characterisation tests before invasive loop changes. Sealed StepDecision types beat string-based control flow.

Add the strict layer next to the lenient one
Run schema validation in parallel with the repair heuristics first. Flip enforcement only after measuring violation rates in production.

Migrate via default methods, not big bangs
Interface defaults (chatWithTools delegating to chat) and @PostConstruct config bridges let you ship the new contract without breaking the old one.

A flag without a metric is a press release
Every feature toggle ships with a Prometheus counter that distinguishes “on” from “doing work”.

Fail open on optional critique
A self-critic that blocks PRs on a parse error or network blip is worse than no critic. APPROVE on uncertainty; only block on explicit, well-formed ITERATE / ABORT.

Building agentic systems is an exercise in designing for uncertainty. The AI is powerful but imprecise. Your surrounding infrastructure must be resilient, adaptive, and willing to hand control back to the agent when deterministic approaches fail. The result is a system that is more capable than either part alone.

---

### Restic Explorer 1.0 — A Lightweight Monitoring Dashboard for Restic Backups

URL: https://remus-software.org/articles/rest-explorer-1-0-released/
Date: April 4, 2026
Description: Restic Explorer 1.0 is out — a lightweight, self-hosted web dashboard that monitors all restic backup repositories across S3, Azure, SFTP, REST, and Rclone from a single UI with automated scans, integrity checks, and retention policy tracking.
Tags: backup, Self-Hosting, news, Restic

Backups are only as good as the confidence that they actually work. Restic Explorer 1.0 is now available — a focused, self-hosted web dashboard that provides exactly that confidence for all restic repositories in one place.

The Problem
Restic is an outstanding backup tool. Fast, encrypted, deduplicated — it has become the go-to choice for backing up servers, NAS devices, and cloud workloads. But restic is a CLI tool by design. When running multiple repositories across different backends — S3 buckets, Azure Blob, SFTP servers — keeping track of “is everything still running?” becomes a chore. It often means writing shell scripts, parsing JSON output, wiring up cron jobs, and hoping someone notices when something breaks.
Existing monitoring solutions are excellent pieces of software, but they tend to come with far more complexity than many use cases require: agent-based architectures, extensive plugin systems, or dashboards designed for hundreds of repositories across large teams. For operators who simply need a single pane of glass that answers are the backups running, are they healthy, and do they meet retention requirements? — a lighter approach is needed.
The Solution
Restic Explorer is that single pane of glass. It connects directly to restic repositories — wherever they live — and provides:

Multi-Repository Dashboard — status of all repos at a glance with color-coded badges (green/red/amber)
Automated Scanning — scheduled restic snapshots calls cache metadata for fast browsing without CLI round-trips
Integrity Checks — scheduled restic check --read-data runs with configurable intervals per repository
Retention Policy Monitoring — daily/weekly/monthly/yearly rules with soft warnings when snapshots fall short
Health Endpoint — /actuator/health JSON endpoint reporting per-repo status, ready for Uptime Kuma, Prometheus, or any HTTP health checker
Snapshot Browser — paginated, sortable snapshot list with a dedicated detail page showing paths, tags, hostname, and size
Lock Detection — automatic stale lock detection with one-click unlock
Encrypted Credentials — AES-256-GCM encryption at rest for repository passwords and backend keys

Five Backends, One UI

Backend
What it covers

S3 / S3-Compatible
AWS S3, MinIO, Wasabi, Backblaze B2 (S3 API)

Azure Blob Storage
Native Azure integration

SFTP
Any SSH-accessible server, key-based auth

REST Server
Restic’s own REST backend with optional HTTP auth

Rclone
Google Drive, Dropbox, OneDrive, B2, and 40+ more via rclone

Getting Started in 60 Seconds
The fastest way to get running is Docker Compose:
services:
  app:
    image: tmseidel/restic-explorer:latest
    ports:
      - "8080:8080"
    environment:
      SPRING_PROFILES_ACTIVE: docker
      DB_HOST: db
      DB_PORT: 5432
      DB_NAME: resticexplorer
      DB_USER: resticexplorer
      DB_PASSWORD: resticexplorer
    depends_on:
      db:
        condition: service_healthy
    restart: unless-stopped

  db:
    image: postgres:16-alpine
    environment:
      POSTGRES_DB: resticexplorer
      POSTGRES_USER: resticexplorer
      POSTGRES_PASSWORD: resticexplorer
    volumes:
      - db-data:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U resticexplorer"]
      interval: 10s
      timeout: 5s
      retries: 5
    restart: unless-stopped

volumes:
  db-data:

docker compose up -d

Open http://localhost:8080, create the admin account, and start adding repositories. That’s it.
The image ships with restic, rclone, and openssh-client pre-installed — no additional setup required for any backend type.
Why Restic is a Great Fit for Cloud & Infrastructure-as-Code
For teams managing cloud infrastructure through Terraform, Ansible, Pulumi, or similar tools, restic fits naturally into the workflow:
Stateless by Design
Restic repositories are self-contained. There is no central server, no daemon, no database to maintain. A repository is just a structured set of encrypted blobs in any storage backend. This makes restic trivially reproducible — IaC can provision the storage bucket and the backup job in the same run.
Backend Agnostic
Moving from AWS to Azure? Migrating from on-prem to cloud? Restic’s backend abstraction means the backup strategy isn’t tied to a vendor. A Terraform module provisions an S3 bucket today; tomorrow it provisions Azure Blob Storage. The restic commands stay the same.
Encryption Without Infrastructure
Restic encrypts everything client-side. There is no need for a KMS, a Vault instance, or an HSM for backup encryption. One password, stored in the secrets manager of choice, and data is encrypted at rest regardless of the storage backend’s capabilities.
Deduplication Saves Cloud Storage Costs
Restic’s content-defined chunking and deduplication means incremental backups are genuinely incremental — even across different source machines backing up to the same repository. In cloud environments where storage is metered, this translates directly to lower costs.
Scriptable and Composable
Restic is a CLI tool that outputs JSON. It composes perfectly with cron, systemd timers, CI/CD pipelines, and container sidecars. No agents to install, no ports to open, no protocols to configure — just a binary and a repository URL.
Restic Explorer adds the monitoring layer on top: existing restic workflows remain untouched, and Restic Explorer watches the repositories and surfaces issues when they need attention.
What’s in 1.0
This release marks the point where the feature set is stable, tested, and production-ready:

Five backend types — S3, Azure, SFTP, REST, Rclone
Repository groups — organize repos by team, environment, or purpose
Configurable scan and check intervals per repository
Retention policy monitoring with violation warnings
Error log with date filtering and auto-cleanup
Dark mode with automatic theme detection
Health & info endpoints for external monitoring integration
Admin-only download of snapshots as .tar archives
Encrypted credential storage (AES-256-GCM)
Docker image running as non-root user with built-in healthcheck

Snapshots
Snapshot Detail

Get It

Docker Hub: tmseidel/restic-explorer:latest
GitHub: tmseidel/restic-explorer
Documentation: User Guide · Configuration · Architecture

Licensed under MIT. Contributions, issues, and feedback welcome.

Restic Explorer is built with Spring Boot 4, Thymeleaf, and Bootstrap 5. It runs as a single container alongside PostgreSQL and requires no additional infrastructure beyond what is already in place.

---

### From Legacy to Lean: Rethinking Your Backup Strategy

URL: https://remus-software.org/articles/replacing-veeam-with-restic/
Date: March 29, 2026
Description: How we replaced a costly, complex backup system with a simple shell script and S3 storage — and the key questions to ask before you do the same.
Tags: backup, Self-Hosting, DevOps, Restic

We ditched our expensive, bloated backup platform for a shell script and S3. Here’s how — and what to think about before you do the same.

The Problem Nobody Wants to Touch
Let’s be honest: most backup systems are set up once and then nobody looks at them again. They just… run. Hopefully.
We were in that exact spot. A centralized commercial backup server on Windows, proprietary agents on every machine, enterprise licenses, the whole deal. It worked — until it didn’t:

The config kept breaking. More than once, the backup server’s internal state got corrupted. Trying to add a new backup job? Error dialog. Can’t configure anything until someone fixes it manually.
Way too much overhead. Each server needed a proprietary agent, a service user, SSH access, firewall rules — all for what’s basically “copy some files somewhere safe.”
We used 5% of the features. Bare-metal recovery? Granular restore? Application-aware snapshots? We never used any of that. Our servers are provisioned with automation — we can rebuild them from scratch. We just needed the data.
It cost real money. A Windows Server with commercial licenses, just to store backups. For a team that runs Linux everywhere else, that’s an expensive oddball.

Before You Migrate: Ask Yourself These Questions
Don’t jump to a new tool just because the old one annoys you. Think it through first:

What are you actually backing up? If your servers can be rebuilt from code, you probably just need data-level backups (database dumps, config files), not full disk images.
Have you ever restored from backup? If the answer is “uh, I think so?” — that’s your real problem, regardless of the tool.
What’s the total cost? Licenses + the server it runs on + agent maintenance + engineer time spent debugging weird issues.
Do you get alerts when a backup fails? A backup that silently breaks is worse than no backup at all.
Is backup part of your provisioning? If setting up backup for a new server is a separate manual process, it will get skipped eventually.

What We Switched To
We landed on Restic — open-source, encrypts everything, deduplicates, compresses, and stores to any S3-compatible backend. It’s in the default Debian repos. Install is literally apt install restic.

Old System
Restic

Install
Proprietary repo + agent + service user + firewall rules
apt install restic

Storage
Dedicated Windows backup server
Any S3-compatible object storage

Config
GUI on backup server
Environment variables + shell script

Licensing
Per-server commercial license
Free

Restore
Through backup server UI
restic restore from anywhere

When picking any replacement tool, look for: simple deployment, storage flexibility (don’t get locked in), full CLI scriptability, client-side encryption, active community, and built-in retention management.

The Architecture
Here’s what we ended up with — three layers:
graph TB
    subgraph servers["Servers"]
        native["Native Apppg_dumpall → gzip→ restic backup"]
        docker["Docker Appdocker exec → pg_dump→ gzip → restic backup"]
        legacy["Legacy Appmysqldump→ legacy agent"]
    end

    subgraph storage["Storage Layer"]
        s3["S3-Compatible Object StoreOne bucket per project"]
        legacysrv["Legacy Backup Server"]
    end

    subgraph monitoring["Monitoring Layer"]
        explorer["Backup ExplorerBrowse repos,check health"]
        heartbeat["Heartbeat MonitorPush-based alerts onsuccess / failure"]
    end

    native -- "Restic + S3" --> s3
    docker -- "Restic + S3" --> s3
    legacy -- "Legacy Agent" --> legacysrv
    s3 --> explorer
    s3 --> heartbeat
A few rules we learned the hard way:

One bucket per project. Never mix backups from different apps in the same bucket. Isolation, access control, cost tracking — all easier this way.
Every backup is individual. A Postgres DB needs pg_dumpall. A Docker service needs docker compose exec. A VPN server needs its config files. There’s no universal “back up everything” script. Write one per app.
Credentials go in a team vault. If the person who set up the backup leaves, you don’t want the passwords leaving with them.

The Script Pattern
After iterating across a bunch of projects, we settled on a template every backup job follows:
#!/usr/bin/env bash
set -euo pipefail

source /opt/app/.restic-env

# Error trap — always notify on failure
trap 'notify_monitor "down" "Backup failed"; rm -f "${DUMP_FILE}"; exit 1' ERR

# Init repo if first run
restic snapshots > /dev/null 2>&1 || restic init

# Create the dump (customize this per app)
pg_dumpall | gzip > "${DUMP_FILE}"

# Don't upload empty dumps
[[ -s "${DUMP_FILE}" ]] || { notify_monitor "down" "Empty dump"; exit 1; }

# Upload, clean up, prune old snapshots
restic backup "${DUMP_FILE}" --tag app-name
rm -f "${DUMP_FILE}"
restic forget --keep-daily 30 --keep-weekly 8 --keep-monthly 12 --prune

# All good
notify_monitor "up" "OK"

The important bits: the error trap makes sure you hear about failures. The empty-dump check catches silent breakage (like a database dump that exits 0 but produces nothing). Retention runs on every backup, not as a separate task. And tags let you filter snapshots later.
With default retention (30 daily, 8 weekly, 12 monthly) you end up with about 44 snapshots at any given time — good granularity without blowing up storage.

Monitoring: Don’t Skip This
Two layers — you need both:
Heartbeat monitoring: Every backup script pings a monitor on success or failure (we use Uptime Kuma, but anything push-based works). If no ping arrives within 26 hours → alert. This catches script failures, cron being broken, and servers being down.
curl -sf "${MONITOR_URL}?status=up&msg=OK"       # on success
curl -sf "${MONITOR_URL}?status=down&msg=Failed"  # in error trap

Repository browser: A heartbeat tells you if the backup ran. A browser tells you what’s in it — snapshot counts, sizes, retention compliance, integrity checks. This catches things like backups that “succeed” but are suspiciously small.

How to Actually Migrate
Don’t flip the switch overnight. We did it in phases:

New servers get the new tool from day one. Zero risk, no migration needed.
Old servers run both systems in parallel. Set up the new backup alongside the legacy one.
Test restores from the new backup. Actually restore on a test environment. Verify the data.
Remove the legacy agent per server after the new backup has been solid for a couple of months.
Kill the legacy server last — only after every server is migrated and validated.

Don’t rush step 4. Storage is cheap. Lost data is not.

TL;DR

If your servers are provisioned from code, you don’t need image-level backups. Just back up the data.
Write a backup script per application — there is no one-size-fits-all.
Monitor everything. Heartbeats for “did it run?”, a browser for “what’s in it?”
Bake backup into your provisioning. If it’s manual, it’ll get skipped.
Test your restores. A backup you’ve never restored from is a hope, not a strategy.
Migrate gradually. Parallel-run, validate, then decommission.

A shell script, a cron job, encrypted uploads to S3, and a heartbeat ping. That’s the whole system. No servers, no GUI, no licenses.

The best backup system is the one your team actually understands, maintains, and tests.

---

### Evaluating Self-Hosted AI Services: A Translation Service Case Study

URL: https://remus-software.org/articles/self-hosted-ai-translation-service/
Date: February 2, 2026
Description: A practical evaluation of replacing DeepL with a self-hosted translation service using open-source LLMs — comparing quality, performance, and cost.
Tags: AI, LLM, Self-Hosting, DevOps, Spring Boot, Python

With freely available large language models now widely accessible, it has become straightforward to self-host software that was previously only available through commercial providers. The key question always comes down to the resulting costs and the effort involved.
In this case study, I examined whether the translation service DeepL can be replaced by a self-hosted solution. The goal was to provide a DeepL-compatible REST API that:

achieves comparable translation quality,
offers similar performance, and
implements the same REST API specification[1],

in order to then compare the one-time and ongoing costs. Using the DeepL API requires a paid subscription; while the pay-as-you-go model is transparent, it can become very expensive with heavy usage. Additionally, data leaves the corporate network, and the API’s behaviour under heavy load is not fully transparent.
Choosing a Suitable Local Model
The first question is which freely available models are suitable for translation tasks. Hugging Face offers a large selection of models that can be easily integrated into custom software[2]. For this evaluation, Meta’s nllb-200-distilled model was chosen, as it is widely used, easy to deploy, and available in three sizes (600M, 1.3B, and 3.3B parameters).
Implementing the DeepL-Compatible REST API
A pragmatic approach was taken for the implementation: a Spring Boot application serves as the API frontend and delegates the actual translation request to a Python Flask component that controls the LLM.
For easy deployment, the system can be run either:

in Docker containers, or
natively on a Debian/Ubuntu server.

The goal was a straightforward deployment on various cloud hardware platforms to test quality and performance there. The complete implementation is available on GitHub[3]. Ansible was used for automated native deployment.
Test — Translation Quality
The following German reference sentence was used to evaluate translation quality:

“Sobald der Glasfaser-Ausbau abgeschlossen ist, erhalten Sie eine Mitteilung zum Schaltungstermin und eine Schnell-Start-Anleitung für die Einrichtung des Glasfaser-Anschlusses.”

DeepL produces the following translation:

“Once the fiber optic expansion is complete, you will receive a notification of the activation date and a quick start guide for setting up your fiber optic connection.”

This translation serves as the reference.
Test with nllb-200-distilled-600M
The smallest model was first run on a development machine via Docker. Performance was not a concern at this stage. The generated translation was:

“Once the glass-faser-Ausbau is closed, you receive a Mitteilung zum Schaltungstermin und eine Schnell-Start-Anleitung für die Einrichtung der Glasfaser-Anschlusses.”

Test with nllb-200-distilled-1.3B
The medium model produced the following output:

“The Commission shall inform the Member States of the date of the entry into force of this Regulation.”

Test with nllb-200-distilled-3.3B
The largest model generated the following translation:

“Once the glass fibre installation is completed, you will receive a notice on the date of installation and a quick start guide for the installation of the glass fibre connections.”

Translation Quality Conclusion
A comprehensive assessment is difficult after just a few tests. Nevertheless, it became clear that only the largest model is viable for production use. It was also notable that the models performed significantly more reliably when the source language was English. If translation is exclusively from English, the medium model might therefore be sufficient.
Test — Performance
Once the suitable model was identified, the next step was to determine under which hardware conditions productive operation is feasible. As a benchmark, it was assumed that translating the reference sentence should take no longer than two seconds. Additionally, the difference between a traditional CPU-based server and a GPU-based system was to be determined.
Test: Traditional Server
A Hetzner CX53 with 16 vCPUs and 32 GB RAM was used as the CPU server (cost: €17 per month).
Response time: 12.93 seconds
Test: GPU Server
An Amazon g4dn.large with 16 GB GPU RAM (Nvidia) was used as the GPU server. The cost is €0.67 per hour, roughly €500 per month.
Response time: 1.31 seconds — GPU memory usage: approx. 13 GB
Performance Conclusion
The difference between the two systems was significantly larger than expected. Even without deep knowledge of the internal workings of LLMs, it is clear that productive operation is practically only feasible with GPU-based hardware. Costs on AWS are currently high, but cheaper alternatives exist — for example at Hetzner[4]. The achieved response time is fundamentally suitable for production use. Parallel requests had no significant impact on latency in the tests.

Overall Conclusion
This evaluation clearly demonstrates that it is possible to self-host AI-based services like machine translation using freely available models and modern hardware — with reasonable effort and competitive quality. While the ongoing costs for GPU-based systems are still relatively high, falling prices and increasing efficiency can be expected as adoption grows and technology advances. Moreover, more affordable hosting alternatives beyond the major cloud providers already exist today.
Especially in heavily regulated industries — such as finance, healthcare, or the public sector — a self-hosted AI service can offer significant advantages:

Data sovereignty is fully preserved, as no sensitive information leaves external systems.
Compliance requirements are easier to meet, since infrastructure and data flows are fully controllable.
Performance and scalability can be precisely tailored to your own needs.
Competitive advantages emerge when you can offer services that are not only cheaper but also more secure and flexible than commercial alternatives.

References

DeepL API Documentation ↩︎

Hugging Face Translation Models ↩︎

simple_ai_translation_service on GitHub ↩︎

Hetzner GPU Dedicated Servers ↩︎

---

### Migrating a Monolith to Microservices: A Practical Guide

URL: https://remus-software.org/articles/monolith-to-microservices/
Date: March 15, 2024
Description: A hands-on walkthrough of the architectural decisions and patterns I use when migrating Java monoliths to cloud-native microservices.
Tags: Java, Microservices, Cloud, Architecture

Migrating a monolithic Java application to microservices is one of the most impactful — and challenging — transformations you can undertake. This article shares the practical approach I’ve refined over multiple engagements.
Why Migrate?
Before touching a single line of code, ask: why are we doing this? The most common drivers I encounter are:

Deployment bottlenecks: A single deployable artifact blocks independent team delivery.
Scalability constraints: You need to scale a specific module, not the entire application.
Technology modernisation: Teams want to adopt newer frameworks or languages for specific domains.
Organisational growth: Conway’s Law — architecture tends to mirror team structure.

“Never migrate for migration’s sake. Identify the concrete pain point and validate that microservices solve it.”

The Strangler Fig Pattern
My go-to approach is the Strangler Fig Pattern: incrementally replace monolith functionality behind a facade, leaving the monolith running until it’s fully strangled.
graph LR
    Client -->|All traffic| Facade[API Gateway / Facade]
    Facade -->|Legacy routes| Monolith[(Monolith)]
    Facade -->|New routes| SvcA[User Service]
    Facade -->|New routes| SvcB[Order Service]
    Monolith -.->|Shared DB - phase 1| DB[(Database)]
    SvcA -->|Own DB - phase 2| DBA[(Users DB)]
    SvcB -->|Own DB - phase 2| DBB[(Orders DB)]
This lets you:

Ship value incrementally
Reduce risk by keeping the fallback running
Validate each new service before extracting the next

Identifying Service Boundaries
Domain-Driven Design (DDD) gives us the best tools for finding service boundaries. I use Event Storming workshops to:

Map all domain events with the business team
Identify bounded contexts — areas with consistent language and ownership
Use bounded contexts as candidate service boundaries

graph TD
    subgraph "Order Context"
        OE1[OrderPlaced]
        OE2[OrderConfirmed]
        OE3[OrderShipped]
    end
    subgraph "Inventory Context"
        IE1[StockReserved]
        IE2[StockReleased]
    end
    subgraph "Notification Context"
        NE1[EmailSent]
        NE2[SMSSent]
    end
    OE2 --> IE1
    OE3 --> NE1
    IE2 --> NE2
Practical Steps
1. Start with the API Layer
Deploy an API Gateway (AWS API Gateway, Kong, or a simple Spring Cloud Gateway) in front of the monolith. This gives you:

A single entry point for traffic
The ability to route selectively to new services
A foundation for cross-cutting concerns (auth, rate limiting, logging)

2. Extract Stateless Services First
Pick a bounded context that:

Has clear, stable APIs
Is relatively self-contained
Has low coupling to the rest of the monolith

Notification services, reporting modules, and authentication are often good first targets.
3. Database Decomposition
The hardest part. Never share a database between the monolith and a new service in the long run. The interim approach:
sequenceDiagram
    participant New Service
    participant Monolith
    participant Shared DB
    participant New DB

    Note over New Service, Shared DB: Phase 1 – Dual Write
    New Service->>Shared DB: Write (compatibility)
    New Service->>New DB: Write (new schema)
    Monolith->>Shared DB: Read/Write

    Note over New Service, New DB: Phase 2 – Cutover
    New Service->>New DB: Write only
    Monolith->>Shared DB: Read/Write (deprecated path)
4. Embrace Eventual Consistency
With separate services comes eventual consistency. Use domain events over synchronous REST calls wherever possible:

Publish events to a message broker (Kafka, RabbitMQ)
Services subscribe to relevant events
Saga pattern for distributed transactions

Key Takeaways

Migrate iteratively — the Strangler Fig pattern is your friend.
Define clear boundaries using DDD bounded contexts.
Decouple the database as a separate, explicit step.
Invest in observability early — distributed tracing (Jaeger, Zipkin) and centralised logging (ELK stack) become essential.
Automate everything — CI/CD per service, infrastructure as code, automated testing.

The migration journey is long, but each extracted service pays dividends in team autonomy and deployment velocity. Start small, validate, and build momentum.

---