Freelance Java consultant with 20+ years of experience in cloud-native architectures, microservices, DevOps, and AI-powered agentic systems. https://remus-software.org/ Tom Seidel tom.seidel@remus-software.org 2026-06-13T00:00:00.000Z https://remus-software.org/articles/qwen3-to-qwen36-upgrade/ 2026-06-13T00:00:00.000Z 2026-06-13T00:00:00.000Z

A dense-to-sparse model swap that nearly doubled token throughput, improved output quality, and taught us a few things about vLLM with MoE and Mamba architectures.

<h1 id="from-qwen3-32b-to-qwen3.6-35b-a3b%3A-upgrading-a-local-inference-stack-on-2%C3%97-rtx-5060-ti" tabindex="-1"><a class="header-anchor" href="#from-qwen3-32b-to-qwen3.6-35b-a3b%3A-upgrading-a-local-inference-stack-on-2%C3%97-rtx-5060-ti">From Qwen3-32B to Qwen3.6-35B-A3B: Upgrading a Local Inference Stack on 2× RTX 5060 Ti</a></h1> <p><em>A dense-to-sparse model swap that nearly doubled our token throughput, made the outputs noticeably better, and taught us a few things about vLLM along the way.</em></p> <hr> <h2 id="the-setup" tabindex="-1"><a class="header-anchor" href="#the-setup">The Setup</a></h2> <p>We run a local inference server with two NVIDIA RTX 5060 Ti GPUs — 16 GB VRAM each, Blackwell architecture, connected over PCIe (no NVLink, unfortunately). The whole stack is managed through Ansible playbooks so nothing is done manually and everything is reproducible. vLLM handles inference, Ollama sits on standby for lighter stuff.</p> <p>Before diving into the models, here’s the VRAM math that governs everything we do on this hardware:</p> <pre><code>=== Model Weight VRAM === Formula: params (billions) × bytes_per_param = weight VRAM fp16 (no quantization): 2 bytes/param → 32B model: 32 × 2 = 64 GB ❌ way too big → 35B model: 35 × 2 = 70 GB ❌ way too big INT4 quantization (AWQ/GPTQ): 0.5 bytes/param (4 bits = 0.5 bytes) → 32B model: 32 × 0.5 = 16 GB ✓ fits → 35B model: 35 × 0.5 = 17.5 GB ✓ fits INT8 quantization: 1 byte/param → 32B model: 32 × 1 = 32 GB ⚠️ tight (no room for KV cache) → 35B model: 35 × 1 = 35 GB ❌ doesn't fit FP8 quantization: 1 byte/param → 32B model: 32 × 1 = 32 GB ⚠️ same as INT8 → 35B model: 35 × 1 = 35 GB ❌ doesn't fit === KV Cache VRAM (per active sequence) === Formula: context_tokens × bytes_per_token The KV cache stores attention keys + values for every token in the context. Size depends on model architecture (num_layers, num_heads, head_dim) and the KV dtype: fp16 KV cache: ~0.25 MiB per token (model-dependent) fp8 KV cache: ~0.125 MiB per token (half of fp16) For our 35B model at 65K context: fp16: 65,536 × 0.25 MiB ≈ 16 GB ❌ eats half our VRAM fp8: 65,536 × 0.125 MiB ≈ 8 GB ✓ manageable </code></pre> <p>With AWQ 4-bit weights (~18-20 GB) + fp8 KV cache (~8 GB for a full 65K sequence) + overhead (~2 GB), we land at roughly 28-30 GB — just under our 32 GB ceiling. That’s the budget we’re working with.</p> <p>Our workhorse for months was <strong>Qwen3-32B-AWQ</strong> — a dense 32-billion-parameter model quantized to 4-bit. It did code reviews, general chat, and agent tool-calling just fine inside a 65K-token context window. Reliable, predictable, no complaints.</p> <p>Then we switched to <strong>Qwen3.6-35B-A3B-AWQ-4bit</strong>. On paper the numbers look similar (35B vs 32B), but under the hood it’s a completely different beast. The upgrade turned out to be worth it — but it wasn’t exactly a drop-in replacement.</p> <hr> <h2 id="what-changed-under-the-hood" tabindex="-1"><a class="header-anchor" href="#what-changed-under-the-hood">What Changed Under the Hood</a></h2> <h3 id="dense-vs.-mixture-of-experts" tabindex="-1"><a class="header-anchor" href="#dense-vs.-mixture-of-experts">Dense vs. Mixture of Experts</a></h3> <p>Here’s the big one. Qwen3-32B is a <strong>dense</strong> model — all 32 billion parameters fire for every single token. Every forward pass uses the entire brain, so to speak.</p> <p>Qwen3.6-35B-A3B is a <strong>Mixture of Experts</strong> (MoE) model. It has 35 billion parameters in total, but only about <strong>3 billion are active</strong> per token. The model has a bunch of “expert” sub-networks and a router that picks which experts handle each input. Picture a company with 35 specialists where only 3 get pulled into any given meeting.</p> <p>The upshot: way less compute per token, even though the model weights take up roughly the same VRAM. That’s where the speed bump comes from.</p> <h3 id="hybrid-mamba-transformer-architecture" tabindex="-1"><a class="header-anchor" href="#hybrid-mamba-transformer-architecture">Hybrid Mamba-Transformer Architecture</a></h3> <p>Qwen3-32B is a pure Transformer — standard attention, quadratic cost, you know the drill.</p> <p>Qwen3.6 mixes things up with a <strong>hybrid architecture</strong> that alternates between Transformer layers and <strong>Mamba</strong> (State Space Model) layers. Mamba processes sequences with linear complexity instead of quadratic attention, which is a big deal for long contexts. You get precise token-to-token attention from the Transformer layers where it matters, and efficient sequential processing from Mamba everywhere else.</p> <p>This hybrid design is also the reason we had to change a bunch of vLLM settings — more on that below.</p> <hr> <h2 id="the-vllm-config-changes-(and-why-they-mattered)" tabindex="-1"><a class="header-anchor" href="#the-vllm-config-changes-(and-why-they-mattered)">The vLLM Config Changes (and Why They Mattered)</a></h2> <p>The architecture shift meant several vLLM parameters needed adjusting. Some were obvious, some were not. Here’s the rundown.</p> <h3 id="1.-tool-call-parser%3A-hermes-%E2%86%92-qwen3_coder" tabindex="-1"><a class="header-anchor" href="#1.-tool-call-parser%3A-hermes-%E2%86%92-qwen3_coder">1. Tool Call Parser: <code>hermes</code> → <code>qwen3_coder</code></a></h3> <p><strong>What this is:</strong> vLLM needs a parser to translate the model’s raw text output into structured tool calls (function names + JSON arguments) that match the OpenAI API format. Different models format their tool calls differently, so the parser has to match.</p> <p><strong>What tripped us up:</strong> Qwen3.5 and 3.6 completely changed their tool-calling format. The <code>hermes</code> parser that worked perfectly with Qwen3-32B just doesn’t understand the new format. The correct one is <code>qwen3_coder</code> — yes, confusingly named, but it’s the right parser for <em>all</em> Qwen3.5/3.6 models, not just the ones with “Coder” in the name.</p> <p><strong>What happens if you get this wrong:</strong> This is the nasty part. Tool calls don’t break outright — they just get parsed wrong. The model outputs structured calls in the new format, the <code>hermes</code> parser mangles them, and tool results get injected back in a format the model doesn’t recognize. The result looks like the model suddenly got stupid. It didn’t — it’s a format mismatch. We spent a while scratching our heads before figuring this one out.</p> <h3 id="2.-max_num_batched_tokens%3A-new-parameter%2C-set-to-4096" tabindex="-1"><a class="header-anchor" href="#2.-max_num_batched_tokens%3A-new-parameter%2C-set-to-4096">2. <code>max_num_batched_tokens</code>: New Parameter, Set to 4096</a></h3> <p><strong>What this is:</strong> Controls the maximum number of tokens vLLM processes in a single prefill batch — basically how much work it tries to chew on at once when processing input tokens.</p> <p><strong>Why we needed it:</strong> The Mamba layers in Qwen3.6 require a memory alignment block size of <strong>2096 tokens</strong>. vLLM’s default is 2048. That’s less than 2096. You can probably guess what happens:</p> <pre><code>AssertionError: In Mamba cache align mode, block_size (2096) must be <= max_num_batched_tokens (2048) </code></pre> <p>Server crashes on startup. Not a tuning issue — a hard compatibility floor. Setting it to 4096 gives comfortable headroom and solved it immediately. This is a must-have for any Mamba-based model.</p> <h3 id="3.-rope-scaling%3A-just-removed-it" tabindex="-1"><a class="header-anchor" href="#3.-rope-scaling%3A-just-removed-it">3. RoPE Scaling: Just Removed It</a></h3> <p><strong>What this is:</strong> RoPE (Rotary Position Embedding) scaling tricks like YaRN let a model handle context windows longer than what it was trained for. Qwen3-32B only natively supports 32K tokens, so we needed YaRN to stretch it to 65K.</p> <p><strong>What changed:</strong> Qwen3.6 natively supports up to <strong>262,144 tokens</strong>. Our 65K fits easily. So we just… removed the scaling. Done.</p> <p><strong>Bonus:</strong> Without RoPE extrapolation, the model operates inside its trained context range. No more positional confusion in long documents, no artifacts from stretching embeddings beyond their design. This alone improved quality on long-context tasks.</p> <h3 id="4.-max_num_seqs%3A-32-%E2%86%92-8" tabindex="-1"><a class="header-anchor" href="#4.-max_num_seqs%3A-32-%E2%86%92-8">4. <code>max_num_seqs</code>: 32 → 8</a></h3> <p><strong>What this is:</strong> How many concurrent requests the engine keeps alive in GPU memory at once. Each active request needs its own KV cache — the memory structure that stores all the previous tokens’ attention data.</p> <p><strong>Why we had to lower it:</strong> Here’s the VRAM budget math:</p> <pre><code>Total VRAM (2 GPUs): 32 GB - Model weights (AWQ 4-bit, TP=2): ~20 GB - Overhead (activations, CUDA, etc.): ~2 GB ---------------------------------------------- Available for KV cache: ~10 GB KV cache per full-length sequence (65K tokens, fp8): 65,536 tokens × ~0.125 MiB/token ≈ 8 GB Max concurrent full-length sequences: 10 GB available ÷ 8 GB per sequence ≈ 1.25 </code></pre> <p>So with full 65K contexts, you can barely fit one sequence. But most requests are way shorter than 65K — a typical agent conversation might use 10-20K tokens, which needs only 1-2 GB of KV cache. That’s why <code>max_num_seqs: 8</code> works: it assumes a realistic mix of context lengths, not everyone maxing out the window at once.</p> <p>The old value of 32 would have risked OOM errors. For our agent workload (typically 1-3 concurrent requests), 8 is more than enough. If you’re running a high-traffic service though, this could be a real constraint.</p> <hr> <h2 id="what-didn%E2%80%99t-change" tabindex="-1"><a class="header-anchor" href="#what-didn%E2%80%99t-change">What Didn’t Change</a></h2> <p>A few settings carried over and are still doing heavy lifting:</p> <ul> <li><strong>fp8 KV cache</strong>: Cuts KV memory in half compared to fp16. The single most important setting for squeezing long contexts into 32 GB.</li> <li><strong>AWQ 4-bit with Marlin kernel</strong>: Model weights at 4-bit precision, ~20 GB instead of ~70 GB. The Marlin kernel is optimized for our Blackwell GPUs.</li> <li><strong>Tensor Parallelism (TP=2)</strong>: Each layer is split across both GPUs. Better than pipeline parallelism for our low-concurrency agent workload.</li> <li><strong>Triton attention backend</strong>: Flash Attention is broken on sm_120 consumer GPUs. Triton works fine with the hybrid architecture.</li> <li><strong>Prefix caching</strong>: Reuses KV cache for repeated prompt prefixes (system prompts, tool definitions). Free 50-80% improvement in time-to-first-token for agent workloads.</li> </ul> <hr> <h2 id="so%2C-did-it-actually-get-better%3F" tabindex="-1"><a class="header-anchor" href="#so%2C-did-it-actually-get-better%3F">So, Did It Actually Get Better?</a></h2> <p>Short answer: yes, noticeably.</p> <h3 id="token-throughput-nearly-doubled" tabindex="-1"><a class="header-anchor" href="#token-throughput-nearly-doubled">Token Throughput Nearly Doubled</a></h3> <p>The MoE architecture is doing the heavy lifting here. Only 3B parameters active per token means way less compute per generated token compared to the dense 32B. The Mamba layers add more efficiency on the prefill side since they dodge the quadratic cost of full attention.</p> <p>In practice: responses come back roughly twice as fast. For interactive use and multi-step agent loops, that’s a very welcome improvement.</p> <h3 id="output-quality-stepped-up" tabindex="-1"><a class="header-anchor" href="#output-quality-stepped-up">Output Quality Stepped Up</a></h3> <ul> <li><strong>Code reviews</strong> worked fine with both models — the old 32B was already good at spotting bugs, suggesting improvements, explaining logic. The new model keeps that bar.</li> <li><strong>Minor code refactorings</strong> are where Qwen3.6 pulls ahead. Renaming variables for clarity, extracting helper functions, simplifying conditionals, restructuring imports — these need a nuanced feel for code intent and scope. Qwen3-32B would often over-engineer or miss the point. Qwen3.6 handles these cleanly and stays focused.</li> <li><strong>Tool-calling</strong> is more accurate and consistent now that the right parser is in place. Fewer malformed JSON arguments, cleaner structured output.</li> </ul> <h3 id="no-more-context-weirdness" tabindex="-1"><a class="header-anchor" href="#no-more-context-weirdness">No More Context Weirdness</a></h3> <p>With Qwen3-32B at 65K, the YaRN scaling would sometimes cause positional confusion — the model would lose track of where information appeared in long documents, especially past the 32K boundary. Qwen3.6, running within its native context range, doesn’t have this problem. References to earlier parts of long conversations are just more reliable.</p> <hr> <h2 id="what-we-learned" tabindex="-1"><a class="header-anchor" href="#what-we-learned">What We Learned</a></h2> <ol> <li> <p><strong>Architecture changes aren’t drop-in.</strong> MoE and Mamba need different vLLM settings than dense Transformers. The <code>max_num_batched_tokens</code> crash wasn’t well-documented — we had to dig into vLLM source to figure it out.</p> </li> <li> <p><strong>Tool call parsers are model-specific.</strong> Always check this when upgrading between Qwen generations. A mismatch doesn’t throw an error — it just makes the model look dumb, which sends you debugging in the wrong direction for a while.</p> </li> <li> <p><strong>MoE punches way above its weight.</strong> 35B total params, 3B active, quality comparable to dense models many times its active size. For constrained hardware, MoE is the move.</p> </li> <li> <p><strong>Native context beats stretched context.</strong> Running within the model’s native range beats extrapolation every time. When picking models, check the native context window first.</p> </li> <li> <p><strong>Ansible-managed infra saves your bacon during upgrades.</strong> Every config change — parser swap, new parameter, removed scaling — went through playbooks. The whole migration is reproducible, reversible, and self-documenting. Doing this manually on the server would have made the Mamba crash way harder to debug and rollbacks way riskier.</p> </li> </ol> <hr> <h2 id="quick-comparison" tabindex="-1"><a class="header-anchor" href="#quick-comparison">Quick Comparison</a></h2> <table> <thead> <tr> <th></th> <th>Qwen3-32B-AWQ (Before)</th> <th>Qwen3.6-35B-A3B-AWQ-4bit (After)</th> </tr> </thead> <tbody> <tr> <td>Architecture</td> <td>Dense Transformer</td> <td>Hybrid Mamba-Transformer (MoE)</td> </tr> <tr> <td>Active Params</td> <td>32B (all of them)</td> <td>~3B (of 35B total)</td> </tr> <tr> <td>Native Context</td> <td>32K</td> <td>262K</td> </tr> <tr> <td>RoPE Scaling</td> <td>YaRN (to reach 65K)</td> <td>None needed</td> </tr> <tr> <td>Tool Call Parser</td> <td><code>hermes</code></td> <td><code>qwen3_coder</code></td> </tr> <tr> <td>max_num_batched_tokens</td> <td>2048 (default)</td> <td>4096 (must be ≥ 2096)</td> </tr> <tr> <td>max_num_seqs</td> <td>32</td> <td>8</td> </tr> <tr> <td>KV Cache</td> <td>fp8</td> <td>fp8</td> </tr> <tr> <td>Quantization</td> <td>AWQ (awq_marlin)</td> <td>AWQ (awq_marlin)</td> </tr> <tr> <td>Tensor Parallelism</td> <td>2</td> <td>2</td> </tr> <tr> <td>Attention Backend</td> <td>TRITON_ATTN</td> <td>TRITON_ATTN</td> </tr> </tbody> </table> <hr> <p><em>Hardware: 2× NVIDIA RTX 5060 Ti 16 GB (Blackwell, PCIe, no NVLink). Inference: vLLM. Infrastructure: Ansible.</em></p> Tom Seidel https://remus-software.org/articles/i-built-my-own-ai-server/ 2026-06-05T00:00:00.000Z 2026-06-05T00:00:00.000Z

A hands-on journey of building a local AI server with dual RTX 5060 Ti GPUs to run 32-billion-parameter models at home — covering hardware choices, software stack (Ollama, vLLM, Open WebUI), Blackwell compatibility headaches, and how local open-source models compare to frontier APIs.

<h1 id="i-built-my-own-ai-server-%E2%80%94-here%E2%80%99s-what-i-learned-about-running-frontier-models-at-home" tabindex="-1"><a class="header-anchor" href="#i-built-my-own-ai-server-%E2%80%94-here%E2%80%99s-what-i-learned-about-running-frontier-models-at-home">I Built My Own AI Server — Here’s What I Learned About Running Frontier Models at Home</a></h1> <p>We’ve all been there: you send a prompt to Claude or GPT-5, wait a few seconds, and wonder — <em>could I do this myself?</em> Not out of arrogance, but curiosity. What does it actually take to run a 32-billion-parameter language model on your own hardware, in your own home, without sending a single byte to the cloud?</p> <p>I set out to find out. I bought the hardware, installed the software, fought the bugs, and benchmarked the results. What I found was a landscape that is simultaneously more accessible and more chaotic than I expected. The tools exist. The models are free. The hardware is affordable. But putting it all together? That’s still an adventure — one I want to share with you.</p> <hr> <h2 id="why-bother-with-a-local-ai-server%3F" tabindex="-1"><a class="header-anchor" href="#why-bother-with-a-local-ai-server%3F">Why Bother With a Local AI Server?</a></h2> <p>The case for running AI locally is compelling on paper:</p> <ul> <li><strong>Privacy</strong>: Your data never leaves your network.</li> <li><strong>Cost</strong>: No per-token API fees that scale unpredictably.</li> <li><strong>Control</strong>: You choose the model, the configuration, and the rules.</li> <li><strong>Learning</strong>: There’s no better way to understand these systems than running them yourself.</li> </ul> <p>But there’s also an honest reason: I wanted to see if a 32-billion-parameter open-source model could actually hold its own against the frontier models from Anthropic and OpenAI. The answer, as you’ll see, is nuanced.</p> <hr> <h2 id="the-hardware%3A-keeping-it-realistic" tabindex="-1"><a class="header-anchor" href="#the-hardware%3A-keeping-it-realistic">The Hardware: Keeping It Realistic</a></h2> <p>I didn’t want to go all-in financially. The goal was to build something that could <em>seriously</em> run AI agents — not a toy, but not a $20,000 GPU workstation either. Here’s what I landed on:</p> <table> <thead> <tr> <th>Component</th> <th>Choice</th> <th>Why</th> </tr> </thead> <tbody> <tr> <td><strong>GPU</strong></td> <td>2× NVIDIA RTX 5060 Ti (16 GB each)</td> <td>32 GB total VRAM — enough for a 32B model</td> </tr> <tr> <td><strong>CPU</strong></td> <td>AMD Ryzen 9 9950X (16 cores)</td> <td>Fast enough for data preprocessing and orchestration</td> </tr> <tr> <td><strong>RAM</strong></td> <td>64 GB DDR5</td> <td>Buffer for model weights and context overflow</td> </tr> <tr> <td><strong>Storage</strong></td> <td>2 TB NVMe SSD</td> <td>Fast model loading and KV cache spill</td> </tr> </tbody> </table> <p><img src="hardware_setup.jpg" alt="Hardware setup — the completed build"></p> <p>The RTX 5060 Ti is interesting. It’s NVIDIA’s newest <strong>Blackwell</strong> architecture (compute capability SM 12.0) — powerful, but so new that much of the AI software ecosystem hadn’t caught up yet. That decision alone would cause me weeks of headaches, as we’ll see.</p> <p>The total build came in at a fraction of what a cloud GPU instance would cost over a year. And as a bonus? I could install Windows on the side and play games when I wasn’t experimenting with AI. :)</p> <hr> <h2 id="the-software-stack%3A-three-engines%2C-one-server" tabindex="-1"><a class="header-anchor" href="#the-software-stack%3A-three-engines%2C-one-server">The Software Stack: Three Engines, One Server</a></h2> <p>The open-source LLM ecosystem has consolidated around a few key tools. I chose a stack of three:</p> <div class="mermaid">flowchart TD User["You (browser, CLI, or agent)"] WebUI["Open WebUI (:8080)<br/>Chat interface"] Ollama["Ollama (:11434)<br/>Simple, friendly model serving"] vLLM["vLLM (:8001)<br/>High-performance inference"] GPU["2× RTX 5060 Ti<br/>32 GB VRAM"] User --> WebUI WebUI --> Ollama WebUI --> vLLM Ollama --> GPU vLLM --> GPU </div><ul> <li><strong><a href="https://ollama.com/">Ollama</a></strong>: The easy option. Download a model, run it, chat with it. Great for experimentation.</li> <li><strong><a href="https://docs.vllm.ai/">vLLM</a></strong>: The serious option. Built for throughput — parallel requests, efficient memory management, and production-grade serving.</li> <li><strong><a href="https://openwebui.com/">Open WebUI</a></strong>: A beautiful chat interface that connects to either engine.</li> </ul> <p>A critical design decision: <strong>only one inference engine runs at a time.</strong> Both Ollama and vLLM need the full 32 GB of VRAM to serve a 32-billion-parameter model. Running them simultaneously would cause one to crash with an out-of-memory error. I automated the switching with a single configuration toggle:</p> <pre><code class="language-yaml"># One variable controls everything inference_engine: "vllm" # or "ollama" </code></pre> <p>When you flip this value, Ansible (my automation tool of choice) stops one engine, starts the other, and reconfigures the web interface to point at the right one. Clean and simple.</p> <hr> <h2 id="the-blackwall%3A-when-new-hardware-meets-immature-software" tabindex="-1"><a class="header-anchor" href="#the-blackwall%3A-when-new-hardware-meets-immature-software">The Blackwall: When New Hardware Meets Immature Software</a></h2> <p>Here’s where the story gets interesting. The RTX 5060 Ti uses NVIDIA’s Blackwell architecture, internally known as <strong>SM 12.0</strong>. It’s powerful hardware, but vLLM — the inference engine I needed for serious work — had barely been tested on it.</p> <p>The first time I tried to start vLLM, it crashed immediately. The error messages were cryptic:</p> <pre><code>undefined symbol: check_cuda_arch RuntimeError: FlashInfer requires GPUs with sm75 or higher </code></pre> <p>This was baffling. SM 12.0 <em>is</em> higher than SM 7.5. The problem? The FlashInfer library — vLLM’s default attention backend — had a JIT compiler that simply didn’t know how to handle Blackwell GPUs yet. It saw an architecture number it didn’t recognize and panicked.</p> <p>This wasn’t just one bug. It was a cascade of three independent problems, each hiding behind the other:</p> <div class="mermaid">flowchart TD A["vLLM won't start"] --> B["Bug 1: Broken systemd unit<br/>(typo in config file)"] A --> C["Bug 2: FLASH_ATTN backend<br/>incompatible with SM 12.0"] A --> D["Bug 3: FlashInfer sampler<br/>also incompatible with SM 12.0"] B --> E["Fix: Remove stray characters<br/>from RestartSec value"] C --> F["Fix: Switch to TRITON_ATTN<br/>(a different attention backend)"] D --> G["Fix: Uninstall flashinfer-python<br/>entirely from the environment"] E --> H["vLLM boots! ...but slowly"] F --> H G --> H </div><p>The fix for the attention backend was to switch from <code>FLASH_ATTN</code> to <strong><code>TRITON_ATTN</code></strong> — a different implementation that happened to work on Blackwell. But FlashInfer kept resurfacing in unexpected places: not just for attention, but also for the <em>sampler</em> (the component that picks which word comes next). Eventually, the only reliable solution was to completely remove the FlashInfer package from the Python environment, forcing vLLM to use its fallback paths everywhere.</p> <p>This took days of debugging, reading GitHub issues, and testing configurations. It’s a reminder that in fast-moving open-source ecosystems, being on the cutting edge of hardware means you <em>are</em> the QA team.</p> <hr> <h2 id="the-vram-puzzle%3A-fitting-a-32-billion-parameter-model-into-32-gb" tabindex="-1"><a class="header-anchor" href="#the-vram-puzzle%3A-fitting-a-32-billion-parameter-model-into-32-gb">The VRAM Puzzle: Fitting a 32-Billion-Parameter Model Into 32 GB</a></h2> <p>A 32-billion-parameter model like <strong>Qwen3-32B</strong> is a large piece of software. In its compressed AWQ (4-bit quantized) form, the model weights take up about <strong>19 GB</strong> of VRAM. That leaves roughly 13 GB for the <strong>KV cache</strong> — the working memory the model uses to track the conversation context.</p> <p>Here’s the math that kept me up at night:</p> <table> <thead> <tr> <th>Context Length</th> <th>KV Cache (fp16)</th> <th>KV Cache (fp8)</th> <th>Fits?</th> </tr> </thead> <tbody> <tr> <td>32K tokens</td> <td>~8 GB</td> <td>~4 GB</td> <td>✅ (with fp8)</td> </tr> <tr> <td>64K tokens</td> <td>~16 GB</td> <td>~8 GB</td> <td>✅ (with fp8, tight)</td> </tr> <tr> <td>128K tokens</td> <td>~32 GB</td> <td>~16 GB</td> <td>❌</td> </tr> </tbody> </table> <p>At standard precision (fp16), even 32K tokens of context wouldn’t fit alongside the model weights. The breakthrough was switching the KV cache to <strong>fp8</strong> — a half-precision floating point format that Blackwell supports natively. This halved the memory requirement with virtually no quality loss, letting me serve conversations up to 64K tokens long.</p> <h3 id="splitting-across-two-gpus" tabindex="-1"><a class="header-anchor" href="#splitting-across-two-gpus">Splitting Across Two GPUs</a></h3> <p>Since the 19 GB model doesn’t fit on a single 16 GB card, it must span both GPUs. There are two ways to do this:</p> <div class="mermaid">flowchart LR subgraph "Pipeline Parallelism (PP=2)" direction LR P1["GPU 0: Layers 1-30<br/>compute → send → wait"] P2["GPU 1: Layers 31-60<br/>wait → compute → send"] end subgraph "Tensor Parallelism (TP=2)" direction LR T1["GPU 0: Half of every layer<br/>compute → sync → compute"] T2["GPU 1: Other half of every layer<br/>compute → sync → compute"] end </div><p><strong>Pipeline Parallelism</strong> splits the model in half — GPU 0 handles the first 30 layers, GPU 1 handles the last 30. The problem? While GPU 0 is working, GPU 1 sits idle, and vice versa. For a single conversation, this wastes about 50% of available compute.</p> <p><strong>Tensor Parallelism</strong> splits each layer across both GPUs, so they’re always working together. The cost? They need to exchange data after every single layer — and on consumer GPUs without NVLink (NVIDIA’s high-speed GPU-to-GPU connection), that data has to travel through the motherboard’s PCIe bus, which is slow.</p> <p>I tested both. For my use case (a personal AI agent handling 1-4 requests at a time), the communication overhead of Tensor Parallelism was roughly equal to the idle-time waste of Pipeline Parallelism. I ended up choosing Pipeline Parallelism because it was more stable on my specific hardware — GeForce cards have known issues with P2P (peer-to-peer) GPU communication that can cause deadlocks.</p> <hr> <h2 id="automation%3A-because-doing-it-twice-is-doing-it-wrong" tabindex="-1"><a class="header-anchor" href="#automation%3A-because-doing-it-twice-is-doing-it-wrong">Automation: Because Doing It Twice Is Doing It Wrong</a></h2> <p>The setup process was complex enough that I automated it with <strong>Ansible</strong> — a tool that lets you describe your entire server configuration as code. The result: a single command that installs NVIDIA drivers, Docker, Ollama, vLLM, Open WebUI, downloads models, and configures everything:</p> <pre><code class="language-bash"># One command, 30-90 minutes later, you have a working AI server ansible-playbook -i inventory.ini site.yml --ask-become-pass </code></pre> <p>The Ansible project grew to include:</p> <ul> <li><strong>Roles</strong> for each component (NVIDIA drivers, Docker, Ollama, vLLM, Open WebUI)</li> <li><strong>Mutual exclusion logic</strong> that ensures only one engine runs at a time</li> <li><strong>Model download tasks</strong> that pull the right quantized models from HuggingFace</li> <li><strong>A test playbook</strong> that verifies everything is working after installation</li> <li><strong>An update playbook</strong> for keeping components current</li> </ul> <p>This was essential. When you’re debugging driver incompatibilities and attention backend failures, you need to be able to tear everything down and rebuild from scratch in minutes, not hours.</p> <hr> <h2 id="performance%3A-the-honest-numbers" tabindex="-1"><a class="header-anchor" href="#performance%3A-the-honest-numbers">Performance: The Honest Numbers</a></h2> <p>After all the configuration, optimization, and debugging, what did I actually get?</p> <p><strong>~20 tokens per second</strong> for single-stream generation with Qwen3-32B-AWQ.</p> <p>To put that in context: that’s roughly 15 words per second — readable in real time, but noticeably slower than ChatGPT or Claude, which typically deliver 40-80+ tokens per second through their cloud infrastructure.</p> <p>Where does the time go? The biggest bottleneck is the <strong>pipeline parallelism bubble</strong> — that 50% idle time I mentioned earlier. Here are the optimizations I identified and tested:</p> <table> <thead> <tr> <th>Optimization</th> <th>Expected Improvement</th> <th>Status</th> </tr> </thead> <tbody> <tr> <td>Switch to Tensor Parallelism</td> <td>+15-30% at low concurrency</td> <td>Tested — marginal gain</td> </tr> <tr> <td>Enable prefix caching</td> <td>+30-50% time-to-first-token</td> <td>Implemented</td> </tr> <tr> <td>Speculative decoding (small draft model)</td> <td>+50-100% decode speed</td> <td>Planned</td> </tr> <tr> <td>NVFP4 quantization (Blackwell-native)</td> <td>+17% if available</td> <td>Waiting for model checkpoint</td> </tr> <tr> <td>Reduce context window to 32K</td> <td>More VRAM for batching</td> <td>Available if 64K not needed</td> </tr> <tr> <td>Push GPU utilization to 95%</td> <td>Small but free</td> <td>Implemented</td> </tr> </tbody> </table> <p>The most promising path forward is <strong>speculative decoding</strong>: using a tiny model (like Qwen3-0.6B) to quickly generate candidate tokens, which the large model then verifies in parallel. This could potentially double the throughput to 40-55 tokens per second — competitive with cloud APIs.</p> <hr> <h2 id="real-world-testing%3A-can-it-replace-claude%3F" tabindex="-1"><a class="header-anchor" href="#real-world-testing%3A-can-it-replace-claude%3F">Real-World Testing: Can It Replace Claude?</a></h2> <p>I put the local setup to the test with the <strong>Hermes agent</strong> — an open-source AI agent framework — running real development tasks:</p> <ul> <li>Writing project documentation for a codebase</li> <li>Generating test cases</li> <li>Drafting pull request descriptions</li> <li>Small and large code refactoring suggestions</li> </ul> <p>The results were honest and humbling:</p> <p><strong>Documentation</strong> was useful but lacked depth. The model captured the high-level structure but missed important details that a human reviewer would need to fill in.</p> <p><strong>Test cases</strong> were mostly correct but overlooked edge cases and contained minor errors — good as a starting point, not as a final product.</p> <p><strong>Pull request descriptions</strong> were generally good but sometimes lacked clarity.</p> <p><strong>Refactoring suggestions</strong> were the weakest area. The model sometimes made the code <em>worse</em>, and the response times were long enough to break the flow of interactive development.</p> <p>The verdict: a 32-billion-parameter local model is not yet a replacement for frontier cloud models when it comes to complex code understanding and generation. It’s a capable first-draft machine that still requires significant human review.</p> <h3 id="a-more-promising-use-case" tabindex="-1"><a class="header-anchor" href="#a-more-promising-use-case">A More Promising Use Case</a></h3> <p>But there’s a second use case I’m more optimistic about: <strong>compliance documentation</strong>. Generating ISO 27001 and ISO 62304 compliant documents — security policies, risk assessments, compliance reports — from unstructured customer input. This is fundamentally a text-generation-from-template task rather than deep code analysis, and early signs suggest the model handles it better. The structured, formulaic nature of compliance documents plays to the strengths of a quantized model.</p> <hr> <h2 id="what-i%E2%80%99d-tell-someone-starting-today" tabindex="-1"><a class="header-anchor" href="#what-i%E2%80%99d-tell-someone-starting-today">What I’d Tell Someone Starting Today</a></h2> <p>If you’re thinking about building your own AI server, here’s what I learned:</p> <ol> <li> <p><strong>The hardware is ready.</strong> Consumer GPUs with 16+ GB of VRAM can genuinely run useful 30B+ parameter models. The price-to-performance ratio is compelling.</p> </li> <li> <p><strong>The software is still catching up.</strong> Especially on newer GPU architectures. Budget significant time for debugging if you’re on cutting-edge hardware. If stability matters more than bleeding-edge performance, consider a previous-generation GPU (RTX 4090, RTX 3090) where the software ecosystem is more mature.</p> </li> <li> <p><strong>Automation is not optional.</strong> The number of configuration variables, environment settings, and interdependencies is too large to manage manually. Tools like Ansible, Docker, and systemd are essential.</p> </li> <li> <p><strong>Quantization is the key that unlocks everything.</strong> Without AWQ 4-bit quantization and fp8 KV caches, a 32B model simply wouldn’t fit in 32 GB of VRAM. These techniques have matured enough that quality loss is minimal for many tasks.</p> </li> <li> <p><strong>Set realistic expectations.</strong> Local models at this scale are powerful assistants, not autonomous replacements. They excel at drafting, summarizing, and generating structured content — tasks where a human reviews the output. For complex reasoning and real-time interactive coding, cloud frontier models still have a clear edge.</p> </li> </ol> <hr> <h2 id="looking-forward" tabindex="-1"><a class="header-anchor" href="#looking-forward">Looking Forward</a></h2> <p>The pace of progress in this space is extraordinary. When I started this project, Blackwell support in vLLM was essentially nonexistent. Within weeks, community fixes were landing in pull requests. Quantization techniques that were research papers six months ago are now one-line configuration options.</p> <p>The open-source model ecosystem is also advancing rapidly. Qwen3-32B is already a capable model, and newer releases from Qwen, Llama, and DeepSeek continue to close the gap with proprietary offerings. NVFP4 quantization — designed specifically for Blackwell’s hardware — promises another 17% speedup once model checkpoints become available.</p> <p>Building your own AI server in 2026 is a bit like building your own PC in 1998: it requires patience, some technical knowledge, and a willingness to troubleshoot. But the reward is a deeper understanding of the technology and a system that is entirely yours — private, customizable, and free from API rate limits.</p> <p>The future of AI isn’t just in the cloud. It’s increasingly in our homes, our offices, and our homelabs. And it’s getting better every month.</p> <hr> <h2 id="further-reading" tabindex="-1"><a class="header-anchor" href="#further-reading">Further Reading</a></h2> <ul> <li><a href="https://github.com/tmseidel/local-llm-setup">Project Source on GitHub</a> — The complete Ansible playbooks, Docker configurations, and systemd units to replicate this entire setup from scratch</li> <li><a href="https://docs.vllm.ai/">vLLM Documentation</a> — The inference engine powering this setup</li> <li><a href="https://ollama.com/">Ollama</a> — The simpler alternative for getting started</li> <li><a href="https://huggingface.co/Qwen">Qwen3 Model Family</a> — The open-weight models used in this project</li> <li><a href="https://www.reddit.com/r/LocalLLaMA/comments/1rwubcy/guide_awq_models_working_on_rtx_5060_ti_sm_120/?tl=de">vLLM Field Report: AWQ on RTX 5060 Ti</a> — Community testing on Blackwell hardware</li> <li><a href="https://openwebui.com/">Open WebUI</a> — The chat interface that ties it all together</li> </ul> Tom Seidel https://remus-software.org/articles/regarding-sovereign-it/ 2026-05-22T00:00:00.000Z 2026-05-22T00:00:00.000Z

Why the debate about sovereign AI is really a broader discussion about sovereign IT — control over infrastructure, platforms, portability, and the long-term ability to make and reverse technical decisions.

<h1 id="sovereign-ai-is-loud-%E2%80%94-but-the-real-issue-is-sovereign-it" tabindex="-1"><a class="header-anchor" href="#sovereign-ai-is-loud-%E2%80%94-but-the-real-issue-is-sovereign-it">Sovereign AI Is Loud — But The Real Issue Is Sovereign IT</a></h1> <p>Most articles here are focused on very concrete, technical problems: setups, architectures, and reproducible solutions. This one takes a step back. Still, at its core, it remains technical — because questions of “sovereignty” ultimately materialize in infrastructure, operations, and the ability to make and reverse decisions.</p> <hr> <h2 id="the-current-narrative%3A-sovereign-ai" tabindex="-1"><a class="header-anchor" href="#the-current-narrative%3A-sovereign-ai">The Current Narrative: Sovereign AI</a></h2> <p>The debate around <strong>sovereign AI</strong> has gained remarkable momentum. Across industry reports, policy discussions, and conference stages, there is a growing emphasis on regaining control over data, models, and infrastructure. Especially in Europe, the dependency on non-local providers has become a recurring concern, driven by both geopolitical tension and regulatory realities.</p> <p>Conceptually, the idea is straightforward: build, run, and control AI systems within your own sphere of influence. In practice, however, this turns out to be significantly more demanding.</p> <p>Operating a self-hosted LLM setup today requires assembling a stack from relatively young and still evolving components:</p> <ul> <li>inference engines such as <strong>vLLM</strong> or <strong>Ollama</strong></li> <li>containerized deployments and orchestration, often via <strong>Kubernetes</strong></li> <li>surrounding services like vector databases, authentication, observability, and networking</li> <li>and, perhaps most critically, access to and operation of suitable <strong>GPU infrastructure</strong></li> </ul> <p>There is no widely accepted, production-ready “standard stack” comparable to what exists for more traditional enterprise workloads. Instead, organizations are faced with a growing but fragmented ecosystem. While this provides flexibility, it also means that building such a platform still requires a considerable amount of in-house expertise and operational maturity.</p> <hr> <h2 id="infrastructure-is-scaling-%E2%80%94-but-not-necessarily-becoming-sovereign" tabindex="-1"><a class="header-anchor" href="#infrastructure-is-scaling-%E2%80%94-but-not-necessarily-becoming-sovereign">Infrastructure Is Scaling — But Not Necessarily Becoming Sovereign</a></h2> <p>At the same time, the physical backbone of digital infrastructure is expanding rapidly. Driven largely by AI workloads, Europe is experiencing a strong increase in data center construction and capacity.</p> <h3 id="key-data-points" tabindex="-1"><a class="header-anchor" href="#key-data-points">Key Data Points</a></h3> <table> <thead> <tr> <th>Topic</th> <th>Data</th> </tr> </thead> <tbody> <tr> <td>Demand driver</td> <td>AI is now the leading driver of data center demand in Europe <a href="https://www.rlbinsights.com/reports/data-centre-trends-report-2025/ai-hits-europe">1</a></td> </tr> <tr> <td>Capacity growth</td> <td>Avg. deployments: 16 MW → 33 MW → 47 MW (2023–2025) <a href="https://www.rlbinsights.com/reports/data-centre-trends-report-2025/ai-hits-europe">1</a></td> </tr> <tr> <td>Investment scale</td> <td>> €100B expected in European data centers by 2030 <a href="https://www.eudca.org/state-of-european-data-centres-2025">2</a></td> </tr> <tr> <td>Pipeline size</td> <td>~14 GW planned capacity in EMEA by mid-2025 <a href="https://www.allianz.com/content/dam/onemarketing/azcom/Allianz_com/economic-research/publications/specials/en/2025/october/2025-10-07-construction-AZ.pdf">3</a></td> </tr> </tbody> </table> <p>This expansion is often interpreted as a positive signal for digital sovereignty: more infrastructure, closer to users, under European jurisdiction.</p> <p>Yet the picture is more nuanced. Much of this newly built capacity is designed to support hyperscale cloud and AI platforms. Even when physically located in Europe, these systems often operate within technological, contractual, and operational frameworks defined elsewhere. In that sense, infrastructure alone does not automatically translate into control.</p> <hr> <h2 id="the-overlooked-context%3A-sovereignty-decisions-were-made-earlier" tabindex="-1"><a class="header-anchor" href="#the-overlooked-context%3A-sovereignty-decisions-were-made-earlier">The Overlooked Context: Sovereignty Decisions Were Made Earlier</a></h2> <p>Seen in this light, the current intensity of the AI sovereignty debate is somewhat surprising. Over the past decade, many organizations have already made fundamental decisions that reduced their control over IT systems — often quite deliberately.</p> <p>The shift toward cloud-based and platform-centric architectures brought undeniable advantages: faster deployment, reduced operational overhead, and access to highly sophisticated tooling. At the same time, it gradually relocated critical capabilities:</p> <ul> <li>infrastructure to cloud providers</li> <li>development workflows to hosted platforms</li> <li>collaboration and communication to SaaS ecosystems</li> <li>customer-facing systems to externally operated services</li> </ul> <p>None of this was irrational. In many cases, it was the most pragmatic choice available. However, it also meant that certain aspects of control — over data flows, system behavior, or long-term portability — became harder to maintain.</p> <hr> <h2 id="different-industries%2C-different-trade-offs" tabindex="-1"><a class="header-anchor" href="#different-industries%2C-different-trade-offs">Different Industries, Different Trade-offs</a></h2> <p>In my own experience, the way organizations approach these questions varies significantly by industry.</p> <p>In <strong>medical technology</strong>, there remains a noticeable degree of caution when it comes to outsourcing critical IT systems. This is not simply a cultural preference but largely shaped by regulation. Requirements around data protection, auditability, and traceability leave relatively little room for ambiguity. As a result, questions about where data resides, who can access it, and how systems can be audited or replaced tend to be addressed early and in detail.</p> <p>This does not necessarily mean that cloud usage is avoided altogether. Rather, it is approached more deliberately, and often with additional safeguards or hybrid models. In that sense, sovereignty is less an abstract concept and more an operational constraint that needs to be managed.</p> <p>By contrast, in parts of the <strong>automotive industry</strong>, the shift toward external platforms was often more assertive. Internal infrastructure teams were frequently seen as cost-intensive and less flexible, while cloud providers offered scalability, speed, and a rich ecosystem of services. Over time, this led to a situation where a significant portion of the IT landscape — from development pipelines to collaboration tools — relies on a relatively small number of external vendors.</p> <p>This approach brought clear benefits in terms of velocity and standardization. At the same time, it introduced new forms of dependency, some of which only become visible when trying to change direction. Migration paths, data portability, and operational independence tend to become more complex as systems are more deeply integrated into proprietary platforms.</p> <hr> <h2 id="why-ai-changes-the-tone-of-the-debate" tabindex="-1"><a class="header-anchor" href="#why-ai-changes-the-tone-of-the-debate">Why AI Changes the Tone of the Debate</a></h2> <p>Against this backdrop, it becomes clearer why AI has triggered a more pronounced reaction.</p> <p>AI systems tend to sit very close to core value creation. They process sensitive data, encode domain knowledge, and increasingly influence decision-making processes. As a result, questions of control feel more immediate. Where earlier layers of abstraction — infrastructure, collaboration tools, or workflow systems — could be externalized with relatively limited visibility, AI makes dependencies more tangible.</p> <p>At the same time, many of the underlying challenges are not new. Vendor lock-in, limited transparency, reliance on external roadmaps, or the gradual loss of internal expertise are all dynamics that have been present for years. AI does not introduce them so much as amplify their impact.</p> <hr> <h2 id="a-more-nuanced-view-on-sovereignty" tabindex="-1"><a class="header-anchor" href="#a-more-nuanced-view-on-sovereignty">A More Nuanced View on Sovereignty</a></h2> <p>It is also worth noting that sovereignty is rarely absolute. In practice, it tends to manifest as a spectrum rather than a binary choice.</p> <h3 id="levels-of-sovereignty" tabindex="-1"><a class="header-anchor" href="#levels-of-sovereignty">Levels of Sovereignty</a></h3> <table> <thead> <tr> <th>Level</th> <th>Example</th> </tr> </thead> <tbody> <tr> <td>Low</td> <td>Full reliance on SaaS / hyperscalers</td> </tr> <tr> <td>Medium</td> <td>EU hosting with contractual safeguards</td> </tr> <tr> <td>High</td> <td>Open-source, portable architectures</td> </tr> <tr> <td>Maximum</td> <td>Fully self-hosted or air‑gapped systems</td> </tr> </tbody> </table> <p>For many organizations, a balanced approach is likely the most practical: retaining control where it matters most while leveraging external services where appropriate. What becomes increasingly important, however, is the ability to make these choices consciously — and to revisit them if needed.</p> <hr> <h2 id="early-signs-of-a-shift" tabindex="-1"><a class="header-anchor" href="#early-signs-of-a-shift">Early Signs of a Shift</a></h2> <p>There are indications that the broader conversation is evolving. In the public sector, for example, initiatives are emerging that explicitly frame digital sovereignty as a strategic objective.</p> <p>Programs that move away from proprietary ecosystems toward open-source-based infrastructures — such as the adoption of LibreOffice, Linux, and open collaboration platforms — reflect a growing awareness of long-term dependencies and their implications. <a href="https://www.schleswig-holstein.de/DE/landesregierung/ministerien-behoerden/I/Presse/PI/2024/CdS/241125_cds_open-source-strategie">4</a></p> <p>These efforts are not without friction. Migration, training, and organizational change can be challenging. Yet they also demonstrate that alternative approaches are possible, even at scale.</p> <hr> <h2 id="conclusion" tabindex="-1"><a class="header-anchor" href="#conclusion">Conclusion</a></h2> <p>The renewed focus on sovereign AI is both understandable and valuable. At the same time, it risks narrowing the perspective if it is treated in isolation.</p> <p>Questions of control, dependency, and adaptability do not begin with AI — they extend across the entire IT landscape. In that sense, sovereign AI is less a starting point and more a visible manifestation of a broader theme.</p> <p>A more comprehensive view would therefore consider not only AI systems, but the surrounding architecture, data flows, and operational capabilities. Ultimately, sovereignty is less about complete independence and more about maintaining the flexibility to respond, adapt, and make informed choices.</p> <p>Or, put differently:<br> it is not necessarily about building everything yourself —<br> but about retaining the ability not to be entirely dependent on others.</p> Tom Seidel https://remus-software.org/articles/meet-ai-git-bot-the-teammate-that-review-ships-test-prs/ 2026-05-21T00:00:00.000Z 2026-05-21T00:00:00.000Z

AI-Git-Bot 1.7 is here, transforming from 'the PR review bot' to the AI teammate your repo has been waiting for — reviewing your code, writing your tests, deploying your previews, cleaning up after itself. The chores that always get cut under deadline pressure? Wire one bot. They get done. Every PR. Forever.

<blockquote> <p>Imagine opening a pull request and, two minutes later, a bot has reviewed your diff, deployed a fresh preview, written a Playwright test for the feature you just added, run it against that preview, and pasted the report back into the PR — all inside the Git tool you already use.</p> <p>That’s not a roadmap. That’s <strong>AI-Git-Bot 1.7</strong>, shipping today. 🚀</p> </blockquote> <p><img src="dashboard_ai_git_bot.PNG" alt="AI-Git-Bot dashboard"></p> <hr> <h2 id="first-time-here%3F-30-second-intro-%F0%9F%91%8B" tabindex="-1"><a class="header-anchor" href="#first-time-here%3F-30-second-intro-%F0%9F%91%8B">First time here? 30-second intro 👋</a></h2> <p><strong>AI-Git-Bot is the open-source AI teammate that lives inside your Git tool</strong> — Gitea, GitHub, GitHub Enterprise, GitLab, Bitbucket Cloud. No new dashboard to log into, no Chrome extension, no Slack bot to babysit. You assign it work the same way you’d assign it to a colleague: request it as a PR reviewer, assign it an issue, or <code>@mention</code> it in a comment.</p> <p>It takes over the <strong>necessary-but-uncomfortable</strong> chores that quietly rot every codebase:</p> <ul> <li>📝 Writing a <em>proper</em> issue with acceptance criteria — before any code is written</li> <li>🔍 Reviewing PRs <strong>consistently</strong>, even when the human reviewer is drowning</li> <li>🧪 Adding the regression test for the bug you just fixed (the one we always say “we’ll add later”)</li> <li>🛠️ Implementing the boring follow-up tickets (renames, bumps, small refactors)</li> <li>🧹 Tearing down the preview environment nobody remembers spinning up</li> </ul> <p>You pick which chores hurt most this quarter, wire one bot, done. The rest stays exactly as it was.</p> <blockquote> <p>👉 Want the long-form pitch? Read <strong><a href="https://github.com/tmseidel/ai-git-bot/blob/main/doc/pitch/PITCH.md"><code>doc/pitch/PITCH.md</code></a></strong> — it’s the fastest way to decide whether AI-Git-Bot is for your team.</p> </blockquote> <hr> <h2 id="what%E2%80%99s-new-in-1.7-%E2%80%94-the-highlights" tabindex="-1"><a class="header-anchor" href="#what%E2%80%99s-new-in-1.7-%E2%80%94-the-highlights">What’s new in 1.7 — the highlights</a></h2> <h3 id="%F0%9F%8E%AC-1.-prs-that-test-themselves" tabindex="-1"><a class="header-anchor" href="#%F0%9F%8E%AC-1.-prs-that-test-themselves">🎬 1. PRs that test themselves</a></h3> <p>This is the headline. Tag your bot with the new <strong>Full-stack QA</strong> workflow + tell it where to deploy your PRs, and every pull request now gets:</p> <ol> <li>A short <strong>plan</strong> of which user journeys to cover</li> <li>A fresh <strong>Playwright test suite</strong>, generated for <em>this</em> PR</li> <li>A <strong>deploy</strong> of your PR to a real preview</li> <li>The suite <strong>run live against that preview</strong></li> <li>A <strong>report comment</strong> on the PR — pass/fail, screenshots, suite source included</li> <li>Automatic <strong>teardown</strong> when the PR closes</li> </ol> <p>Don’t like the suite? Drop a comment: <code>@bot regenerate-tests focus on the checkout flow</code>. The bot replans with your feedback. Just need a quick rerun? <code>@bot rerun-tests</code>. That’s it.</p> <p><img src="gitea-pr-with-e2e-test-run.png" alt="PR with a Playwright report posted by the bot"></p> <p><strong>Why you’ll love it:</strong> the test debt that always slips to “next sprint” finally gets paid down — automatically, per PR.</p> <h3 id="%F0%9F%94%8C-2.-it-plays-nice-with-your-pipeline-(whatever-it-is)" tabindex="-1"><a class="header-anchor" href="#%F0%9F%94%8C-2.-it-plays-nice-with-your-pipeline-(whatever-it-is)">🔌 2. It plays nice with your pipeline (whatever it is)</a></h3> <p>We added four ways for the bot to deploy your PR — pick the one you already have:</p> <table> <thead> <tr> <th>You’re using…</th> <th>The bot uses strategy…</th> </tr> </thead> <tbody> <tr> <td>Jenkins / TeamCity / a bash script behind a webhook</td> <td><code>WEBHOOK</code></td> </tr> <tr> <td>Vercel, Netlify, Render, Cloudflare Pages</td> <td><code>STATIC</code></td> </tr> <tr> <td>GitHub Actions, Gitea Actions, GitLab CI, Bitbucket Pipelines</td> <td><code>CI_ACTION</code></td> </tr> <tr> <td>An internal platform team exposing deploys via MCP</td> <td><code>MCP</code></td> </tr> </tbody> </table> <p><strong>No need to migrate anything.</strong> The bot adapts to your stack, not the other way around.</p> <h3 id="%F0%9F%A7%A9-3.-workflows-you-can-mix%2C-match-and-extend" tabindex="-1"><a class="header-anchor" href="#%F0%9F%A7%A9-3.-workflows-you-can-mix%2C-match-and-extend">🧩 3. Workflows you can mix, match and extend</a></h3> <p>Reviewing PRs and running E2E tests are now just two examples of a bigger idea: <strong>PR Workflows</strong>. Each one is a named, configurable bundle (“Default review”, “Full-stack QA”, or your own) that you can assign to any bot from the admin UI.</p> <p><img src="ai-bot-workflow-settings.png" alt="Workflow configurations admin UI"></p> <p>Want a custom one — say a license-header check, an SBOM-diff comment, or a “ping the on-call channel on every hotfix PR” workflow? It’s a clean extension point now. Your platform team can ship it without forking the project.</p> <h3 id="%F0%9F%97%A3%EF%B8%8F-4.-new-slash-commands%2C-less-back-and-forth" tabindex="-1"><a class="header-anchor" href="#%F0%9F%97%A3%EF%B8%8F-4.-new-slash-commands%2C-less-back-and-forth">🗣️ 4. New slash commands, less back-and-forth</a></h3> <p>Two more commands in the bot’s vocabulary:</p> <ul> <li><code>@bot rerun-tests</code> — re-run the existing suite, no replanning</li> <li><code>@bot regenerate-tests <your feedback></code> — replan the suite, your feedback goes straight into the planner</li> </ul> <p>Combined with the existing <code>@bot fix …</code> and <code>@bot write …</code>, your PR comments become the remote control.</p> <h3 id="%F0%9F%9B%9F-5.-test-suites-that-can-outlive-the-pr-%E2%80%94-if-you-want-them-to" tabindex="-1"><a class="header-anchor" href="#%F0%9F%9B%9F-5.-test-suites-that-can-outlive-the-pr-%E2%80%94-if-you-want-them-to">🛟 5. Test suites that can outlive the PR — if you want them to</a></h3> <p>By default, generated suites are <strong>ephemeral</strong>: they live with the PR, vanish on close, nothing leaks. Safe and boring — exactly what most teams want.</p> <p>But if you’d love to <strong>keep</strong> the suites the bot writes, flip one setting and pick:</p> <ul> <li><em>commit-to-pr</em> — committed straight to the PR branch</li> <li><em>offer-as-pr</em> — the bot opens a follow-up PR with the suite</li> <li><em>promote-on-merge</em> — auto-promoted when the original PR merges</li> </ul> <p>A nightly cleanup job keeps the test folder from turning into a swamp.</p> <h3 id="%F0%9F%94%92-6.-tighter%2C-safer%2C-friendlier" tabindex="-1"><a class="header-anchor" href="#%F0%9F%94%92-6.-tighter%2C-safer%2C-friendlier">🔒 6. Tighter, safer, friendlier</a></h3> <p>A handful of quality-of-life upgrades that you’ll feel without noticing:</p> <ul> <li><strong>Per-bot tool whitelisting</strong> — give your writer-bot read-only access, your coding-bot the full toolbox</li> <li><strong>Async callbacks</strong> with single-use, HMAC-signed secrets — no shared tokens in your CI runners</li> <li><strong>Force-push safe</strong> — re-pushing three times in a minute no longer confuses the bot</li> <li><strong>Better LLM compatibility</strong> — Gemini 3.x, sanitised tool names across providers, more robust agent loops</li> </ul> <hr> <h2 id="what-to-try-first-%E2%80%94-pick-one-%F0%9F%91%87" tabindex="-1"><a class="header-anchor" href="#what-to-try-first-%E2%80%94-pick-one-%F0%9F%91%87">What to try first — pick one 👇</a></h2> <p>You don’t need to adopt everything. Start with the one that hooks you:</p> <ol> <li><strong>🚀 The 15-minute upgrade test.</strong> Pull <code>tmseidel/ai-git-bot:1.7.0</code>, point at your existing database, watch it migrate cleanly. Your current bots behave exactly like in 1.6. Zero risk, great way to validate.</li> <li><strong>🎬 The “wow” demo.</strong> Spin up the bundled sample stack — <code>docker compose -f systemtest/docker-compose-e2e-sample.yml up</code> — open a PR, sit back, watch the bot plan, deploy, test, report. <strong>This is the one to show your team on Monday morning.</strong></li> <li><strong>🔌 Wire it to your real CI.</strong> If you live in GitHub/GitLab/Gitea/Bitbucket pipelines, <a href="https://github.com/tmseidel/ai-git-bot/blob/main/doc/PR_WORKFLOWS_CI_ACTIONS.md"><code>PR_WORKFLOWS_CI_ACTIONS.md</code></a> is the shortest path to “the bot just tested my PR against a real preview environment.”</li> <li><strong>🧩 Build your own workflow.</strong> Got a chore you keep nagging humans about on every PR? Ship it as a workflow — your team will thank you forever.</li> </ol> <hr> <h2 id="brand-new%3F-start-here-%F0%9F%A7%AD" tabindex="-1"><a class="header-anchor" href="#brand-new%3F-start-here-%F0%9F%A7%AD">Brand new? Start here 🧭</a></h2> <table> <thead> <tr> <th>If you are…</th> <th>Start with…</th> </tr> </thead> <tbody> <tr> <td>👀 <strong>Curious</strong></td> <td><a href="https://github.com/tmseidel/ai-git-bot/blob/main/doc/pitch/PITCH.md">The pitch</a> — why this exists, what it actually does, how it compares to Copilot Workspace / GitLab Duo / Qodo / Aider</td> </tr> <tr> <td>🧑‍💻 <strong>A developer</strong> who wants to play</td> <td><a href="https://github.com/tmseidel/ai-git-bot/blob/main/doc/LOCAL_DEVELOPMENT.md">Local development guide</a> — up and running in ~10 min</td> </tr> <tr> <td>🏗️ <strong>An architect</strong> evaluating it</td> <td><a href="https://github.com/tmseidel/ai-git-bot/blob/main/doc/ARCHITECTURE.md">Architecture overview</a> and <a href="https://github.com/tmseidel/ai-git-bot/tree/main/doc/agentic-workflows">agentic workflows internals</a></td> </tr> <tr> <td>🛠️ <strong>DevOps / Platform</strong></td> <td><a href="https://github.com/tmseidel/ai-git-bot/blob/main/doc/DEPLOYMENT.md">Deployment guide</a> + <a href="https://github.com/tmseidel/ai-git-bot/blob/main/doc/PR_WORKFLOWS_CI_ACTIONS.md">CI action recipes</a></td> </tr> <tr> <td>🆙 <strong>Already running 1.6</strong></td> <td><a href="https://github.com/tmseidel/ai-git-bot/blob/main/doc/MIGRATION_1.6_TO_1.7.md">Migration guide 1.6 → 1.7</a> — short version: drop in, done</td> </tr> </tbody> </table> <p><img src="ai-bot-bot-configuration.png" alt="Bot configuration form"></p> <hr> <h2 id="get-it-now" tabindex="-1"><a class="header-anchor" href="#get-it-now">Get it now</a></h2> <pre><code class="language-bash">docker pull tmseidel/ai-git-bot:1.7.0 </code></pre> <ul> <li>⭐ <strong>Star us on GitHub:</strong> <a href="https://github.com/tmseidel/ai-git-bot">https://github.com/tmseidel/ai-git-bot</a></li> <li>🐛 <strong>Found something?</strong> <a href="https://github.com/tmseidel/ai-git-bot/issues">Open an issue</a> — we read every one.</li> <li>💬 <strong>Just want to say hi?</strong> Drop a discussion on the repo, we love hearing how teams are wiring this up.</li> </ul> <hr> <p><strong>The bottom line:</strong> in 1.7, AI-Git-Bot stops being “the PR review bot” and starts being <strong>the AI teammate your repo has been waiting for</strong> — reviewing your code, writing your tests, deploying your previews, cleaning up after itself.</p> <p>The chores that always get cut under deadline pressure? Wire one bot. They get done. Every PR. Forever.</p> <p>Happy shipping. 🚀</p> Tom Seidel https://remus-software.org/articles/remote-mcp-standardization/ 2026-05-15T00:00:00.000Z 2026-05-15T00:00:00.000Z

Why organizations should standardize agentic AI with remote MCP server pools to improve consistency, governance, security, and adoption across teams.

<p><em>How organizations can eliminate inconsistent AI output, unify governance, and accelerate adoption — by moving from local, per-developer tool chaos to a shared pool of remote MCP resources.</em></p> <hr> <h2 id="the-problem-no-one-talks-about%3A-ai-output-variance" tabindex="-1"><a class="header-anchor" href="#the-problem-no-one-talks-about%3A-ai-output-variance">The Problem No One Talks About: AI Output Variance</a></h2> <p>Enterprise AI adoption is accelerating, but a quiet problem is eroding its ROI: <strong>the same prompt, run by two different employees, produces dramatically different results</strong>. This is not a model problem — it is an infrastructure problem.</p> <p>The root causes are surprisingly structural:</p> <table> <thead> <tr> <th>Factor</th> <th>Description</th> </tr> </thead> <tbody> <tr> <td><strong>Client-side setup</strong></td> <td>Each employee configures their local AI environment differently — different system prompts, context window sizes, temperature settings, or client versions</td> </tr> <tr> <td><strong>Prompt engineering skill gap</strong></td> <td>Employees vary widely in their ability to write effective prompts, leading to wildly different output quality for identical tasks</td> </tr> <tr> <td><strong>Installed local tools & MCP servers</strong></td> <td>One developer has a CRM MCP plugin; another does not. One has an up-to-date internal knowledge tool; another is running a stale fork</td> </tr> <tr> <td><strong>Model version drift</strong></td> <td>Without pinned configurations, different clients may call different model versions</td> </tr> <tr> <td><strong>Context availability</strong></td> <td>Access to company-specific context (style guides, internal APIs, compliance rules) depends on what each user has manually configured</td> </tr> <tr> <td><strong>Security posture</strong></td> <td>API keys stored in local <code>.env</code> files, personal tokens in config files, credentials passed as environment variables — each a potential exposure point</td> </tr> <tr> <td><strong>Tool versioning</strong></td> <td>Locally installed MCP servers may lag behind in functionality or contain unfixed bugs</td> </tr> <tr> <td><strong>Permissions divergence</strong></td> <td>Without centralized access control, some employees inadvertently have access to sensitive tools they should not, and others are blocked from tools they need</td> </tr> </tbody> </table> <p>This variance has three compounding consequences that directly impact the business case for AI:</p> <ol> <li><strong>Quality rework</strong> — Outputs need manual post-processing because they don’t meet the expected standard, eliminating the productivity gain</li> <li><strong>Adoption stagnation</strong> — When AI “works well for some and not others,” skepticism spreads and uptake plateaus</li> <li><strong>Compliance failure</strong> — Uncontrolled tool access and inconsistent outputs make it nearly impossible to enforce data governance, GDPR obligations, or industry-specific regulations</li> </ol> <hr> <h2 id="where-the-mcp-ecosystem-stands-today%3A-a-protocol-snapshot" tabindex="-1"><a class="header-anchor" href="#where-the-mcp-ecosystem-stands-today%3A-a-protocol-snapshot">Where the MCP Ecosystem Stands Today: A Protocol Snapshot</a></h2> <p>Before making the case for remote MCP, it is worth grounding the discussion in where the ecosystem actually is — because the data tells a revealing story about the gap between where things are and where they need to go.</p> <p>As of early 2026, the distribution of publicly registered MCP servers by transport protocol breaks down as follows — based on aggregated data from Anthropic, OpenAI, and analyst surveys published Q4 2025 through Q1 2026, as compiled by <a href="https://www.digitalapplied.com/blog/mcp-adoption-statistics-2026-model-context-protocol">Digital Applied (April 2026)</a>:</p> <table> <thead> <tr> <th>Transport</th> <th>Share</th> <th>Status</th> </tr> </thead> <tbody> <tr> <td><strong>stdio</strong></td> <td>~67 %</td> <td>Local subprocess — runs on the client machine</td> </tr> <tr> <td><strong>Streamable HTTP</strong></td> <td>~28 %</td> <td>Remote, OAuth 2.1-secured — the current standard</td> </tr> <tr> <td><strong>SSE</strong> (legacy)</td> <td>~5 %</td> <td>Remote, deprecated — being phased out</td> </tr> </tbody> </table> <div class="mermaid">pie title MCP Server Distribution by Transport Protocol (2026) "stdio (local)" : 67 "Streamable HTTP (remote)" : 28 "SSE legacy (remote)" : 5 </div><p>Two-thirds of all MCP servers currently run locally via stdio. This is a direct artifact of how MCP adoption began: developers building quick local integrations for their own AI clients, spinning up tools as subprocesses, storing credentials in environment variables. It was the fastest path to a working prototype — and it shows.</p> <p>The more important number, however, is the <strong>trend</strong>. Remote MCP deployments have grown nearly fourfold since May 2025. Among the 20 most widely used MCP servers, 80 % now offer a remote deployment option — a figure independently confirmed by MCP Manager via Ahrefs search-volume data (<a href="https://mcpmanager.ai/blog/mcp-adoption-statistics/">MCP Manager, October 2025</a>). And of those remote servers, 81 % authenticate via OAuth 2.1 — meaning the governance infrastructure is already being built at the ecosystem level.</p> <p>The market is moving toward remote. The question for organizations is whether they lead that shift deliberately — with centralized governance, versioning, and access control — or whether they arrive there reactively, having accumulated the same local-config debt that once plagued their microservices era.</p> <blockquote> <p><strong>The 67 % stdio figure is not a benchmark to maintain — it is a starting point to move away from.</strong></p> </blockquote> <hr> <h2 id="the-architectural-shift%3A-from-local-chaos-to-a-shared-mcp-resource-pool" tabindex="-1"><a class="header-anchor" href="#the-architectural-shift%3A-from-local-chaos-to-a-shared-mcp-resource-pool">The Architectural Shift: From Local Chaos to a Shared MCP Resource Pool</a></h2> <p>The solution mirrors a pattern mature engineering organizations have known for decades: <strong>centralize what should be consistent, and expose it through a well-governed interface</strong>.</p> <p>In the context of AI agents, this means replacing a sprawl of locally configured MCP servers with a unified, remote MCP server pool that every client in the organization connects to.</p> <div class="mermaid">graph TD subgraph Before ["❌ Before: Per-Client Local Setup"] E1["Employee A\n(has CRM MCP v1.2)"] E2["Employee B\n(has CRM MCP v1.0, misconfigured)"] E3["Employee C\n(no CRM MCP)"] end subgraph After ["✅ After: Shared Remote MCP Pool"] C1["Client A"] C2["Client B"] C3["Client C"] subgraph Pool["Remote MCP Server Pool (Org-managed)"] direction LR MCP1["CRM MCP\n(latest, versioned)"] MCP2["Knowledge Base MCP\n(RAG-enabled)"] MCP3["Compliance MCP\n(audit-logged)"] MCP4["Code Review MCP\n(shared tooling)"] end C1 --> Pool C2 --> Pool C3 --> Pool end </div><hr> <h2 id="benefits-of-a-remote%2C-centralized-mcp-resource-pool" tabindex="-1"><a class="header-anchor" href="#benefits-of-a-remote%2C-centralized-mcp-resource-pool">Benefits of a Remote, Centralized MCP Resource Pool</a></h2> <p>Moving to shared remote MCP servers delivers benefits across multiple dimensions:</p> <h3 id="%F0%9F%94%92-security-%26-governance" tabindex="-1"><a class="header-anchor" href="#%F0%9F%94%92-security-%26-governance">🔒 Security & Governance</a></h3> <ul> <li><strong>Centralized secret management</strong> — API keys, tokens, and credentials live on the server, never on employee machines</li> <li><strong>Unified access control</strong> — Role-based permissions enforced at the MCP layer: the marketing team never accidentally calls a financial data tool</li> <li><strong>Audit trails</strong> — Every tool call is logged centrally, making compliance reporting tractable</li> <li><strong>Reduced attack surface</strong> — No credentials scattered across dozens of laptops</li> </ul> <h3 id="%F0%9F%93%90-consistency-%26-quality" tabindex="-1"><a class="header-anchor" href="#%F0%9F%93%90-consistency-%26-quality">📐 Consistency & Quality</a></h3> <ul> <li><strong>Single source of truth</strong> — All agents access the same tool versions, the same data, the same business logic</li> <li><strong>Versioned, tested releases</strong> — Tool updates are deployed once and take effect for all users simultaneously</li> <li><strong>Embedded prompt hygiene</strong> — System prompts, context, and guardrails can be baked into server-side tool definitions, not left to individual configuration</li> </ul> <h3 id="%E2%9A%99%EF%B8%8F-operational-efficiency" tabindex="-1"><a class="header-anchor" href="#%E2%9A%99%EF%B8%8F-operational-efficiency">⚙️ Operational Efficiency</a></h3> <ul> <li><strong>Zero per-client installation</strong> — Onboarding a new employee means one auth flow, not an afternoon of CLI setup</li> <li><strong>Centralized monitoring</strong> — Error rates, latency, and usage patterns visible in one place</li> <li><strong>Faster iteration</strong> — Improve a tool once, benefit everyone immediately</li> </ul> <h3 id="%F0%9F%93%88-adoption-%26-culture" tabindex="-1"><a class="header-anchor" href="#%F0%9F%93%88-adoption-%26-culture">📈 Adoption & Culture</a></h3> <ul> <li><strong>Consistent experience</strong> — Every user gets the same capable toolset, reducing “it works for them but not me” friction</li> <li><strong>Lower barrier to entry</strong> — Non-technical employees can use powerful agentic tools without understanding local config</li> <li><strong>Trust through predictability</strong> — Repeatable, governed outputs build organizational trust in AI</li> </ul> <hr> <h2 id="the-microservices-analogy%3A-secured-services-calling-secured-services" tabindex="-1"><a class="header-anchor" href="#the-microservices-analogy%3A-secured-services-calling-secured-services">The Microservices Analogy: Secured Services Calling Secured Services</a></h2> <p>This architecture is not new — it is precisely the pattern that modern software engineering adopted when monoliths became unmanageable.</p> <p>Consider the parallel to microservices:</p> <div class="mermaid">graph LR subgraph Microservices["Microservices Architecture (known pattern)"] direction LR MS_Client["Frontend / API Gateway"] MS_Auth["Auth Service\n(OAuth 2.1 / OIDC)"] MS_CRM["CRM Service\n(secured endpoint)"] MS_BI["BI Service\n(secured endpoint)"] MS_Client -- "Bearer Token" --> MS_Auth MS_Auth -- "validates, issues scoped token" --> MS_Client MS_Client -- "scoped token" --> MS_CRM MS_Client -- "scoped token" --> MS_BI end subgraph MCP_Arch["MCP Architecture (same principle)"] direction LR AI_Client["AI Agent / Claude Code"] MCP_Auth["OAuth 2.1 Provider\n(org identity)"] MCP_CRM["CRM MCP Server\n(remote, secured)"] MCP_KB["Knowledge MCP Server\n(remote, secured)"] AI_Client -- "auth request" --> MCP_Auth MCP_Auth -- "scoped token (RFC 8707)" --> AI_Client AI_Client -- "tool call + token" --> MCP_CRM AI_Client -- "tool call + token" --> MCP_KB end </div><p>Just as a well-designed microservices landscape enforces that <strong>only authenticated, authorized clients can call sensitive services</strong> — regardless of who wrote the calling code or where it runs — a remote MCP pool enforces that <strong>only authenticated agents, with the right scopes, can call the right tools</strong>.</p> <p>The MCP spec adopted <strong>OAuth 2.1 with PKCE and Resource Indicators (RFC 8707)</strong> precisely for this reason: so that a rogue or misconfigured client cannot escalate privileges or leak tokens across service boundaries — a gap that was explicitly closed in the June 2025 spec update.</p> <blockquote> <p><strong>The bottom line:</strong> Just as you would never let application developers hardcode database passwords into their laptops and query production directly, you should not let them hardcode MCP tool configurations either. Centralize the services; secure the boundary; govern the access.</p> </blockquote> <hr> <h2 id="quick-start%3A-connecting-claude-code-cli-to-remote-mcp-servers" tabindex="-1"><a class="header-anchor" href="#quick-start%3A-connecting-claude-code-cli-to-remote-mcp-servers">Quick Start: Connecting Claude Code CLI to Remote MCP Servers</a></h2> <p>The good news is that connecting to a remote MCP server requires only a single command. Here is a practical walkthrough using Claude Code (the CLI), which supports Streamable HTTP transport natively.</p> <h3 id="step-1%3A-add-a-remote-mcp-server" tabindex="-1"><a class="header-anchor" href="#step-1%3A-add-a-remote-mcp-server">Step 1: Add a remote MCP server</a></h3> <pre><code class="language-bash"># Basic remote server (Streamable HTTP) claude mcp add --transport http <server-name> https://mcp.your-org.com/crm # With a scope — "project" commits to .mcp.json (shareable via git) claude mcp add --transport http --scope project crm-mcp https://mcp.your-org.com/crm # With a static auth header (for servers not using OAuth) claude mcp add --transport http --scope user analytics-mcp https://mcp.your-org.com/analytics \ --header "Authorization: Bearer ${ORG_MCP_TOKEN}" </code></pre> <h3 id="step-2%3A-authenticate-via-oauth-(if-the-server-requires-it)" tabindex="-1"><a class="header-anchor" href="#step-2%3A-authenticate-via-oauth-(if-the-server-requires-it)">Step 2: Authenticate via OAuth (if the server requires it)</a></h3> <pre><code class="language-bash"># Inside a Claude Code session, trigger the OAuth flow /mcp # Select the server name, press Enter # Claude Code opens a browser tab for org SSO / OAuth consent # After consent, the token is stored automatically </code></pre> <h3 id="step-3%3A-add-via-json-(ideal-for-scripting-or-ci%2Fcd-provisioning)" tabindex="-1"><a class="header-anchor" href="#step-3%3A-add-via-json-(ideal-for-scripting-or-ci%2Fcd-provisioning)">Step 3: Add via JSON (ideal for scripting or CI/CD provisioning)</a></h3> <pre><code class="language-bash"># For automated onboarding scripts claude mcp add-json crm-mcp '{ "type": "http", "url": "https://mcp.your-org.com/crm" }' </code></pre> <h3 id="step-4%3A-verify-the-connection" tabindex="-1"><a class="header-anchor" href="#step-4%3A-verify-the-connection">Step 4: Verify the connection</a></h3> <pre><code class="language-bash">claude mcp list # → crm-mcp https://mcp.your-org.com/crm ✓ connected </code></pre> <h3 id="committing-the-org-wide-configuration" tabindex="-1"><a class="header-anchor" href="#committing-the-org-wide-configuration">Committing the org-wide configuration</a></h3> <p>When <code>--scope project</code> is used, Claude Code writes the server configuration to <code>.mcp.json</code> at the project root. <strong>Committing this file to your repository means every developer who clones the repo gets the same MCP pool automatically</strong> — no per-person setup required.</p> <pre><code class="language-jsonc">// .mcp.json (committed to version control) { "mcpServers": { "crm-mcp": { "type": "http", "url": "https://mcp.your-org.com/crm" }, "knowledge-base-mcp": { "type": "http", "url": "https://mcp.your-org.com/kb" }, "compliance-mcp": { "type": "http", "url": "https://mcp.your-org.com/compliance" } } } </code></pre> <blockquote> <p>💡 <strong>This approach is AI-implementation agnostic.</strong> The MCP specification is an open standard, now governed by the Agentic AI Foundation (AAIF) under the Linux Foundation, with support from Anthropic, OpenAI, Google, Microsoft, and Block. The same remote MCP servers you configure for Claude Code can be consumed by Cursor, Windsurf, the OpenAI Agents SDK, Gemini’s Vertex AI Agent Builder, and any other MCP-compliant client. <strong>Invest once in your MCP server infrastructure; benefit across your entire AI toolchain.</strong></p> </blockquote> <hr> <h2 id="a-note-on-the-argument%3A-where-it-holds-%E2%80%94-and-where-to-be-careful" tabindex="-1"><a class="header-anchor" href="#a-note-on-the-argument%3A-where-it-holds-%E2%80%94-and-where-to-be-careful">A Note on the Argument: Where It Holds — and Where to Be Careful</a></h2> <p>This proposal is structurally sound, but a few nuances are worth acknowledging:</p> <p><strong>Where the argument is strong:</strong></p> <ul> <li>For large organizations with many AI users, the consistency and governance case is compelling and mirrors established patterns from API gateway and microservices adoption</li> <li>OAuth 2.1 at the MCP layer is not theoretical — it is spec-mandated since mid-2025 and implemented in production by major SaaS vendors</li> <li>The adoption-stagnation problem is real: inconsistent AI experiences are a documented cause of tool abandonment</li> </ul> <p><strong>Where to be careful:</strong></p> <ul> <li><strong>Latency</strong> — Remote MCP servers introduce network round-trips vs. local stdio. For latency-sensitive agentic loops (many sequential tool calls), this can matter. Mitigate with geographic co-location or edge deployment</li> <li><strong>Dependency risk</strong> — A central MCP pool is a shared dependency. Plan for high availability from day one; a pool outage affects all agents simultaneously</li> <li><strong>Migration cost</strong> — Teams with mature local setups face a real migration effort. A phased approach (critical/shared tools remote first, niche local tools later) reduces friction</li> <li><strong>Not a silver bullet for prompt quality</strong> — Centralizing tools improves tool consistency, but employees with weak prompt engineering skills will still produce weaker outputs. Pair this with prompt libraries, agent templates, and training</li> </ul> <hr> <h2 id="conclusion" tabindex="-1"><a class="header-anchor" href="#conclusion">Conclusion</a></h2> <p>The variance problem in enterprise AI is not going to solve itself. As long as every employee runs their own local configuration, prompt-engineers with their own skills, and connects to their own patchwork of local tools, the delta between the best and worst AI-assisted work in your organization will remain large.</p> <p>Remote MCP servers are the architectural lever that addresses the <em>tooling</em> layer of this problem — and it is the layer most amenable to centralized governance. Centralizing your MCP resource pool brings your AI agents closer to the reliability model you expect from your SaaS APIs: versioned, authenticated, monitored, and consistent for every caller.</p> <p>The pattern is proven. The spec is stable. The tooling is ready.</p> <hr> <h2 id="outlook%3A-further-levers-for-ai-standardization-in-your-organization" tabindex="-1"><a class="header-anchor" href="#outlook%3A-further-levers-for-ai-standardization-in-your-organization">Outlook: Further Levers for AI Standardization in Your Organization</a></h2> <p>The remote MCP pool is one layer of a broader AI standardization stack. Organizations that go further will benefit from:</p> <div class="mermaid">graph TD L1["🏗️ Layer 1: Remote MCP Server Pool\n(consistent tools, governed access)"] L2["📝 Layer 2: Shared Prompt Libraries & Agent Templates\n(consistent task framing, embedded best practices)"] L3["🔍 Layer 3: Observability & Evaluation\n(log tool calls, score outputs, detect drift)"] L4["🏢 Layer 4: AI Gateway / Proxy\n(rate limits, cost allocation, model routing, PII redaction)"] L5["🎓 Layer 5: Continuous AI Literacy Programs\n(prompt engineering, agent design patterns)"] L1 --> L2 --> L3 --> L4 --> L5 </div><ul> <li><strong>Shared prompt libraries</strong>: Versioned, peer-reviewed prompt templates for common tasks, stored in a central repository and surfaced to all users — reducing the skill-gap effect directly</li> <li><strong>Observability pipelines</strong>: Logging agent tool calls and scoring outputs over time exposes which tools are underperforming, which prompts produce the best results, and where policy violations occur</li> <li><strong>AI gateways</strong>: A proxy layer (such as LiteLLM, Portkey, or a custom gateway) sitting in front of multiple model providers enables model routing, cost visibility, PII stripping, and fallback — independent of which AI client employees use</li> <li><strong>Agent-as-a-service</strong>: For high-value, repeatable workflows (contract review, data extraction, report generation), encapsulate the entire agent — model, tools, system prompt, output format — as an internal service with an API, removing individual variation entirely</li> </ul> <p>The organizations that will lead in AI-augmented productivity are not the ones who gave every employee access to ChatGPT or Claude. They are the ones who built the infrastructure layer that makes the output of any employee, on any client, consistently excellent.</p> <hr> <p><em>References and further reading:</em></p> <ul> <li><a href="https://modelcontextprotocol.io/specification/2025-11-25/basic/transports">MCP Official Specification — Transports</a></li> <li><a href="https://code.claude.com/docs/en/mcp">Claude Code: Connect to tools via MCP</a></li> <li><a href="https://docs.anthropic.com/en/build-with-claude/mcp/directory">Anthropic MCP Connector Directory</a></li> </ul> Tom Seidel https://remus-software.org/articles/improve-software-quality-with-reviewer-personas/ 2026-05-08T00:00:00.000Z 2026-05-08T00:00:00.000Z

How to use AI-Git-Bot as a review gateway with multiple specialized reviewer personas to catch security, reliability, performance, testability, and maintainability issues in Git pull requests.

<h1 id="improving-software-quality-with-reviewer-personas-in-ai-git-bot" tabindex="-1"><a class="header-anchor" href="#improving-software-quality-with-reviewer-personas-in-ai-git-bot">Improving Software Quality with Reviewer Personas in AI-Git-Bot</a></h1> <p>AI-Git-Bot can be used as a self-hosted review gateway that connects Git platforms such as Gitea, GitHub, GitLab, and Bitbucket Cloud with configurable AI providers. One of its strongest quality-improvement patterns is not to configure a single generic AI reviewer, but to create a small system of focused reviewer personas. Each persona uses its own system prompt, its own bot identity, and its own review criteria. The result is a repeatable multi-pass review process that brings security, reliability, performance, testability, and maintainability perspectives into the Git workflow before code is merged.</p> <p>This article uses an exemplary project, <strong>ShopFlow</strong>, to explain the approach. ShopFlow is a fictional e-commerce application with a Java/Spring Boot backend, a PostgreSQL database, and a web frontend. The team frequently changes checkout logic, payment integrations, discount rules, user-account flows, and order-processing jobs. Those are exactly the kinds of changes where a single human reviewer or a single generic AI review can miss important risks.</p> <hr> <h2 id="1.-problem-description" tabindex="-1"><a class="header-anchor" href="#1.-problem-description">1. Problem Description</a></h2> <p>Modern software teams already know that code review is essential, but the review process is under pressure:</p> <ul> <li>Pull requests often mix business logic, infrastructure changes, tests, and documentation.</li> <li>Human reviewers have limited time and different areas of expertise.</li> <li>Security problems, concurrency bugs, missing edge cases, and weak test coverage are easy to overlook.</li> <li>Teams want fast feedback, but not at the cost of quality.</li> <li>Generic AI reviews can help, but a single broad prompt often produces broad feedback instead of deep specialized analysis.</li> </ul> <p>In ShopFlow, consider a pull request titled <strong>“Add express checkout with saved cards and promotional discount stacking”</strong>. The diff touches:</p> <ul> <li>authentication checks for saved payment methods,</li> <li>discount calculation rules,</li> <li>database migrations,</li> <li>order-confirmation emails,</li> <li>retry behavior for payment-provider timeouts,</li> <li>unit and integration tests.</li> </ul> <p>No single reviewer perspective is enough. A security-minded reviewer will ask whether users can access another user’s saved card. A reliability reviewer will look for retry storms and idempotency problems. A maintainability reviewer will ask whether the discount rules are testable and understandable.</p> <p>AI-Git-Bot addresses this by letting a team configure several bots, each connected to a Git integration, an AI integration, and a dedicated system prompt entry. These bots can be requested as reviewers in the Git platform. When assigned or re-requested as reviewers, they fetch the pull-request diff, send it to the configured AI provider, and post their review back to the pull request.</p> <div class="mermaid">flowchart LR PR["ShopFlow pull request\nExpress checkout"] --> SecBot["Security Reviewer Bot"] PR --> RelBot["Reliability & Performance Bot"] PR --> MaintBot["Testability & Maintainability Bot"] SecBot --> SecReview["Finds auth, permission, secret,\nand input-validation risks"] RelBot --> RelReview["Finds latency, retry, concurrency,\nand resource-usage risks"] MaintBot --> MaintReview["Finds weak tests, unclear design,\nand maintainability issues"] SecReview --> Git["Git pull-request discussion"] RelReview --> Git MaintReview --> Git Git --> Human["Human reviewer makes\nfinal merge decision"] </div><p>The goal is not to replace human review. The goal is to make human review more effective by adding consistent, specialized, early feedback directly where development already happens: in the Git pull request.</p> <hr> <h2 id="2.-current-status-of-good-practices" tabindex="-1"><a class="header-anchor" href="#2.-current-status-of-good-practices">2. Current Status of Good Practices</a></h2> <p>Good engineering teams already combine several quality practices:</p> <table> <thead> <tr> <th>Practice</th> <th>Strength</th> <th>Limitation</th> </tr> </thead> <tbody> <tr> <td>Human code review</td> <td>Context, judgment, ownership, mentoring</td> <td>Slow under load; expertise varies by reviewer</td> </tr> <tr> <td>CI builds and tests</td> <td>Deterministic validation</td> <td>Only catches what is encoded in tests</td> </tr> <tr> <td>Static analysis</td> <td>Repeatable style, bug, and complexity checks</td> <td>Often shallow on business context</td> </tr> <tr> <td>Dependency scanning</td> <td>Known vulnerability detection</td> <td>Does not reason about application-specific misuse</td> </tr> <tr> <td>Security review</td> <td>Deep risk analysis</td> <td>Expensive and often reserved for larger changes</td> </tr> <tr> <td>Architecture review</td> <td>Long-term design quality</td> <td>Usually asynchronous and not applied to every PR</td> </tr> </tbody> </table> <p>These practices are still necessary. AI-Git-Bot fits as an additional review layer between automated checks and final human approval. It is especially useful because its reviews are:</p> <ul> <li><strong>triggered intentionally</strong> by assigning or re-requesting a bot as reviewer,</li> <li><strong>configurable</strong> through reusable system prompt entries,</li> <li><strong>contextual</strong> because the bot reviews the pull-request diff and can answer follow-up questions,</li> <li><strong>centralized</strong> through a gateway that manages Git integrations, AI integrations, prompts, credentials, and sessions,</li> <li><strong>portable</strong> across supported Git platforms and AI providers.</li> </ul> <p>For ShopFlow, the team keeps its existing CI pipeline:</p> <div class="mermaid">flowchart TD Dev["Developer opens PR"] --> CI["CI: compile, tests, lint, dependency scan"] Dev --> Bots["AI-Git-Bot reviewer personas"] CI --> Status["Git status checks"] Bots --> Reviews["Persona-specific PR reviews"] Status --> HumanReview["Human review"] Reviews --> HumanReview HumanReview --> Merge{"Ready to merge?"} Merge -- "No" --> Fix["Developer fixes findings"] Fix --> CI Merge -- "Yes" --> Main["Merge to main"] </div><p>The important shift is that code quality is no longer treated as one generic review question. Instead, it becomes a structured set of perspectives.</p> <hr> <h2 id="3.-analysis-of-possible-solutions" tabindex="-1"><a class="header-anchor" href="#3.-analysis-of-possible-solutions">3. Analysis of Possible Solutions</a></h2> <h3 id="option-a%3A-one-generic-ai-reviewer" tabindex="-1"><a class="header-anchor" href="#option-a%3A-one-generic-ai-reviewer">Option A: One Generic AI Reviewer</a></h3> <p>A team can create one bot with the default review prompt. This is easy and already valuable. The default prompt asks the AI to look for correctness bugs, security vulnerabilities, performance problems, concurrency issues, API or database concerns, missing tests, and maintainability problems.</p> <p>For small teams or low-risk repositories, this may be sufficient. The downside is that the reviewer has to cover everything at once. In practice, broad reviews often produce broad findings. They may mention many categories without going deeply into the most important ones.</p> <h3 id="option-b%3A-specialized-reviewer-personas" tabindex="-1"><a class="header-anchor" href="#option-b%3A-specialized-reviewer-personas">Option B: Specialized Reviewer Personas</a></h3> <p>Instead of one broad reviewer, the team creates multiple bots with different system prompt entries. Each bot focuses on a narrow review mission.</p> <p>For ShopFlow, the team configures three personas:</p> <table> <thead> <tr> <th>Persona</th> <th>Bot name in Git</th> <th>Primary mission</th> <th>Typical findings</th> </tr> </thead> <tbody> <tr> <td>Security Reviewer</td> <td><code>shopflow-security-ai</code></td> <td>Protect data, identity, permissions, secrets, and inputs</td> <td>IDOR, missing authorization, unsafe token handling, injection risk</td> </tr> <tr> <td>Reliability & Performance Reviewer</td> <td><code>shopflow-reliability-ai</code></td> <td>Protect production stability and scalability</td> <td>non-idempotent retries, N+1 queries, race conditions, slow queries</td> </tr> <tr> <td>Testability & Maintainability Reviewer</td> <td><code>shopflow-maintainability-ai</code></td> <td>Protect long-term changeability</td> <td>missing tests, unclear abstractions, excessive coupling, unreadable business rules</td> </tr> </tbody> </table> <p>This approach gives the team three independent review passes. Each bot reads the same diff, but its system prompt changes the evaluation lens.</p> <div class="mermaid">graph TD PromptSec["System Prompt Entry:\nSecurity Review"] --> BotSec["Bot:\nshopflow-security-ai"] PromptRel["System Prompt Entry:\nReliability & Performance Review"] --> BotRel["Bot:\nshopflow-reliability-ai"] PromptMaint["System Prompt Entry:\nTestability & Maintainability Review"] --> BotMaint["Bot:\nshopflow-maintainability-ai"] AI["AI Integration\nCloud or local model"] --> BotSec AI --> BotRel AI --> BotMaint GitInt["Git Integration\nGitea / GitHub / GitLab / Bitbucket"] --> BotSec GitInt --> BotRel GitInt --> BotMaint BotSec --> PR["Pull request reviews"] BotRel --> PR BotMaint --> PR </div><h3 id="option-c%3A-combine-writer-agent%2C-coding-agent%2C-and-review-personas" tabindex="-1"><a class="header-anchor" href="#option-c%3A-combine-writer-agent%2C-coding-agent%2C-and-review-personas">Option C: Combine Writer Agent, Coding Agent, and Review Personas</a></h3> <p>AI-Git-Bot also supports issue-based agent workflows. A writer bot can improve vague issues into structured, testable follow-up issues. A coding bot can implement assigned issues and open pull requests. The reviewer personas then inspect the resulting PR.</p> <p>For ShopFlow, this creates a complete quality loop:</p> <ol> <li>A product owner writes a vague issue: “Make checkout faster and support express payment.”</li> <li>A writer bot turns it into an implementation-ready issue with acceptance criteria, non-goals, and test cases.</li> <li>A coding agent or developer implements the change.</li> <li>The three reviewer personas review the pull request.</li> <li>A human reviewer decides what must be fixed before merge.</li> </ol> <div class="mermaid">sequenceDiagram participant PO as Product Owner participant Git as Git Platform participant Writer as Writer Bot participant Dev as Developer or Coding Agent participant Reviewers as Reviewer Personas participant Human as Human Reviewer PO->>Git: Create vague ShopFlow checkout issue PO->>Git: Assign writer bot Git->>Writer: Issue assignment webhook Writer->>Git: Ask clarifying questions or create AI Created Issue Dev->>Git: Implement issue and open PR Git->>Reviewers: PR review requested for each persona Reviewers->>Git: Post specialized review feedback Human->>Git: Review findings and approve or request changes </div><h3 id="recommended-solution" tabindex="-1"><a class="header-anchor" href="#recommended-solution">Recommended Solution</a></h3> <p>For a production project like ShopFlow, the best balance is <strong>Option B plus selected parts of Option C</strong>:</p> <ul> <li>Use three reviewer personas for every medium- or high-risk pull request.</li> <li>Use a writer bot for unclear issues before implementation starts.</li> <li>Keep CI, dependency scanning, and human review as mandatory gates.</li> <li>Treat bot findings as structured expert input, not as automatic merge decisions.</li> </ul> <hr> <h2 id="4.-implementing-the-solution" tabindex="-1"><a class="header-anchor" href="#4.-implementing-the-solution">4. Implementing the Solution</a></h2> <h3 id="4.1-configure-ai-git-bot-as-the-gateway" tabindex="-1"><a class="header-anchor" href="#4.1-configure-ai-git-bot-as-the-gateway">4.1 Configure AI-Git-Bot as the Gateway</a></h3> <p>In AI-Git-Bot, each bot connects:</p> <ol> <li>one <strong>Git integration</strong>,</li> <li>one <strong>AI integration</strong>,</li> <li>one <strong>system prompt entry</strong>,</li> <li>one bot identity and webhook URL.</li> </ol> <p>For ShopFlow, the team creates:</p> <ul> <li>a Git integration for its Git platform,</li> <li>an AI integration for the chosen model provider,</li> <li>three system prompt entries,</li> <li>three coding bots with review-focused prompts,</li> <li>optionally one writer bot for issue refinement.</li> </ul> <p>The gateway pattern is useful because secrets, prompts, sessions, model choices, and Git-provider details are managed centrally rather than scattered across repositories.</p> <div class="mermaid">flowchart LR subgraph Git["Git System"] Repo["ShopFlow repository"] PR["Pull requests"] Issues["Issues"] end subgraph Gateway["AI-Git-Bot Gateway"] Bots["Bots"] Prompts["System prompt entries"] Sessions["Review sessions"] Secrets["Encrypted tokens and API keys"] end subgraph Providers["AI Providers"] Cloud["Cloud AI"] Local["Ollama / llama.cpp"] end Repo <--> Gateway PR <--> Gateway Issues <--> Gateway Bots --> Prompts Bots --> Sessions Bots --> Secrets Gateway <--> Cloud Gateway <--> Local </div><h3 id="4.2-create-the-three-reviewer-personas" tabindex="-1"><a class="header-anchor" href="#4.2-create-the-three-reviewer-personas">4.2 Create the Three Reviewer Personas</a></h3> <p>The system prompts below are intentionally short enough to maintain predictable output, but strict enough to shape each review. In AI-Git-Bot, these would be stored under <strong>System settings → System prompts</strong> as separate prompt entries. The <strong>Review System-Prompt</strong> field is the important field for pull-request reviews and PR conversations.</p> <h4 id="persona-1%3A-security-reviewer" tabindex="-1"><a class="header-anchor" href="#persona-1%3A-security-reviewer">Persona 1: Security Reviewer</a></h4> <p><strong>Purpose:</strong> identify security defects before they become production vulnerabilities.</p> <p><strong>Example Review System-Prompt:</strong></p> <pre><code class="language-markdown">You are a senior application security reviewer for the ShopFlow e-commerce platform. Review the pull request diff only from a security and abuse-resistance perspective. Focus on authentication, authorization, tenant/user isolation, payment data handling, input validation, injection risks, secrets, logging of sensitive data, dependency exposure, and prompt-injection or instruction-handling risks. Prioritize findings that could expose customer data, payment data, admin actions, or internal credentials. Do not invent vulnerabilities that are not supported by the diff. Format the review as: 1. Blocking security issues 2. Non-blocking hardening suggestions 3. Security tests to add 4. Overall security assessment </code></pre> <p><strong>How this results in the Git system:</strong></p> <ul> <li>The team creates a bot named <code>shopflow-security-ai</code>.</li> <li>The bot uses the <code>Security Review</code> system prompt entry.</li> <li>The Git provider is configured with the bot’s webhook URL.</li> <li>When <code>shopflow-security-ai</code> is assigned as reviewer, AI-Git-Bot posts a security-focused review.</li> <li>Developers can ask follow-up questions in the PR with <code>@shopflow-security-ai</code>.</li> </ul> <p>In the express-checkout PR, this bot might comment:</p> <ul> <li>“Blocking: the saved-card lookup uses <code>cardId</code> without verifying ownership by the current user.”</li> <li>“Add a regression test proving user A cannot use user B’s saved payment method.”</li> <li>“Avoid logging payment-provider request payloads because they may contain sensitive identifiers.”</li> </ul> <h4 id="persona-2%3A-reliability-%26-performance-reviewer" tabindex="-1"><a class="header-anchor" href="#persona-2%3A-reliability-%26-performance-reviewer">Persona 2: Reliability & Performance Reviewer</a></h4> <p><strong>Purpose:</strong> prevent production incidents, scalability bottlenecks, and operationally unsafe behavior.</p> <p><strong>Example Review System-Prompt:</strong></p> <pre><code class="language-markdown">You are a reliability and performance reviewer for the ShopFlow e-commerce platform. Review the pull request diff for production stability, scalability, latency, resource usage, database efficiency, concurrency, transaction boundaries, retry behavior, idempotency, timeouts, fallback behavior, and observability. Prioritize issues that can cause outages, duplicate orders, retry storms, slow checkout, deadlocks, memory pressure, or hard-to-debug production failures. Be concrete and explain what load or failure mode would trigger the issue. Format the review as: 1. Blocking reliability/performance issues 2. Non-blocking operational improvements 3. Load, concurrency, or failure-mode tests to add 4. Overall production-readiness assessment </code></pre> <p><strong>How this results in the Git system:</strong></p> <ul> <li>The team creates a bot named <code>shopflow-reliability-ai</code>.</li> <li>The bot uses the <code>Reliability & Performance Review</code> prompt entry.</li> <li>The bot is requested on PRs touching checkout, database access, async jobs, or integrations.</li> <li>AI-Git-Bot fetches the diff and posts findings as a PR review or review comments.</li> <li>Follow-up discussions stay inside the Git pull request.</li> </ul> <p>In the express-checkout PR, this bot might comment:</p> <ul> <li>“Blocking: payment retries are not idempotent; a timeout after provider success could create duplicate orders.”</li> <li>“The new discount query is executed once per cart item and may create an N+1 query pattern.”</li> <li>“Add a test for concurrent checkout submissions with the same cart.”</li> </ul> <h4 id="persona-3%3A-testability-%26-maintainability-reviewer" tabindex="-1"><a class="header-anchor" href="#persona-3%3A-testability-%26-maintainability-reviewer">Persona 3: Testability & Maintainability Reviewer</a></h4> <p><strong>Purpose:</strong> protect long-term code health, reduce regression risk, and improve developer velocity.</p> <p><strong>Example Review System-Prompt:</strong></p> <pre><code class="language-markdown">You are a testability and maintainability reviewer for the ShopFlow e-commerce platform. Review the pull request diff for clarity, cohesive design, readable business rules, low coupling, meaningful naming, consistency with surrounding code, regression risk, and sufficient automated tests. Focus on whether future developers can safely understand, modify, and test the change. Avoid minor style nitpicks unless they materially affect readability or consistency. Prefer actionable suggestions with concrete refactoring or test recommendations. Format the review as: 1. Blocking maintainability or testability issues 2. Non-blocking design/readability suggestions 3. Missing or weak tests 4. Overall maintainability assessment </code></pre> <p><strong>How this results in the Git system:</strong></p> <ul> <li>The team creates a bot named <code>shopflow-maintainability-ai</code>.</li> <li>The bot uses the <code>Testability & Maintainability Review</code> prompt entry.</li> <li>The bot is assigned to feature PRs and refactoring PRs.</li> <li>Its findings appear in the same Git review discussion as human comments, CI results, and other bot reviews.</li> <li>Developers can ask targeted questions such as <code>@shopflow-maintainability-ai how would you split this service?</code>.</li> </ul> <p>In the express-checkout PR, this bot might comment:</p> <ul> <li>“The discount-stacking rules are embedded in <code>CheckoutService</code>; extract a <code>DiscountPolicy</code> so each rule can be tested independently.”</li> <li>“The tests cover the happy path but not conflicting promotions or expired discounts.”</li> <li>“The migration changes order state semantics; add an integration test for existing pending orders.”</li> </ul> <h3 id="4.3-define-review-routing-rules" tabindex="-1"><a class="header-anchor" href="#4.3-define-review-routing-rules">4.3 Define Review Routing Rules</a></h3> <p>The team should not request every persona on every tiny change. A practical routing matrix keeps review signal high.</p> <table> <thead> <tr> <th>Change type</th> <th style="text-align:right">Security</th> <th style="text-align:right">Reliability & Performance</th> <th style="text-align:right">Testability & Maintainability</th> </tr> </thead> <tbody> <tr> <td>Authentication or authorization</td> <td style="text-align:right">Required</td> <td style="text-align:right">Optional</td> <td style="text-align:right">Required</td> </tr> <tr> <td>Payment or checkout flow</td> <td style="text-align:right">Required</td> <td style="text-align:right">Required</td> <td style="text-align:right">Required</td> </tr> <tr> <td>Database migration</td> <td style="text-align:right">Optional</td> <td style="text-align:right">Required</td> <td style="text-align:right">Required</td> </tr> <tr> <td>UI-only text change</td> <td style="text-align:right">Optional</td> <td style="text-align:right">Optional</td> <td style="text-align:right">Optional</td> </tr> <tr> <td>Business-rule refactoring</td> <td style="text-align:right">Optional</td> <td style="text-align:right">Optional</td> <td style="text-align:right">Required</td> </tr> <tr> <td>Background jobs or queues</td> <td style="text-align:right">Optional</td> <td style="text-align:right">Required</td> <td style="text-align:right">Required</td> </tr> <tr> <td>Dependency upgrade</td> <td style="text-align:right">Required for security-sensitive libs</td> <td style="text-align:right">Optional</td> <td style="text-align:right">Optional</td> </tr> </tbody> </table> <p>This matrix can be implemented as a team convention. For example, pull-request templates can contain a checklist asking authors which AI reviewer personas they requested.</p> <h3 id="4.4-keep-review-output-actionable" tabindex="-1"><a class="header-anchor" href="#4.4-keep-review-output-actionable">4.4 Keep Review Output Actionable</a></h3> <p>The most important prompt-design rule is to force useful structure. Each persona should separate:</p> <ol> <li>blocking issues,</li> <li>non-blocking suggestions,</li> <li>recommended tests,</li> <li>an overall assessment.</li> </ol> <p>This maps well to how teams already work in Git:</p> <ul> <li>blocking issues become required fixes before merge,</li> <li>non-blocking suggestions can become follow-up issues,</li> <li>test recommendations become new commits in the PR,</li> <li>overall assessments help the human reviewer judge residual risk.</li> </ul> <div class="mermaid">flowchart TD Finding["Bot finding"] --> Classify{"Finding type"} Classify -- "Blocking" --> FixNow["Fix in current PR"] Classify -- "Non-blocking" --> FollowUp["Create follow-up issue if valuable"] Classify -- "Test gap" --> AddTests["Add or improve tests"] Classify -- "Uncertain" --> Ask["Ask @bot or human owner for clarification"] FixNow --> NewCommit["Push new commit"] AddTests --> NewCommit NewCommit --> Rerequest["Re-request relevant bot review"] </div><h3 id="4.5-evaluate-and-improve-the-prompts" tabindex="-1"><a class="header-anchor" href="#4.5-evaluate-and-improve-the-prompts">4.5 Evaluate and Improve the Prompts</a></h3> <p>Prompts are part of the quality system and should be reviewed like code. Before rolling out new reviewer personas broadly, the team should:</p> <ul> <li>collect representative historical PR diffs,</li> <li>define a scoring rubric for correctness, security awareness, actionability, concision, and false positives,</li> <li>compare the default prompt with the specialized prompt,</li> <li>test malicious or confusing PR content to ensure the bot ignores instructions inside diffs or comments,</li> <li>start with one repository or one team before expanding.</li> </ul> <p>AI-Git-Bot’s reusable system prompt entries make this manageable. A team can clone a prompt entry, adjust the wording, assign it to one bot, observe review quality, and then promote it more broadly.</p> <h3 id="4.6-example-end-to-end-flow-in-shopflow" tabindex="-1"><a class="header-anchor" href="#4.6-example-end-to-end-flow-in-shopflow">4.6 Example End-to-End Flow in ShopFlow</a></h3> <p>The following scenario shows the system in practice:</p> <ol> <li>A developer opens a PR for express checkout.</li> <li>CI starts automatically.</li> <li>The developer requests reviews from: <ul> <li><code>shopflow-security-ai</code>,</li> <li><code>shopflow-reliability-ai</code>,</li> <li><code>shopflow-maintainability-ai</code>.</li> </ul> </li> <li>AI-Git-Bot receives review-request webhooks.</li> <li>Each bot fetches the PR diff and uses its configured system prompt.</li> <li>Each bot posts a focused review.</li> <li>The developer fixes blocking issues and asks follow-up questions with bot mentions.</li> <li>The developer re-requests only the relevant bot reviews.</li> <li>A human reviewer uses the bot reviews plus CI status to make the final decision.</li> </ol> <div class="mermaid">sequenceDiagram participant Dev as Developer participant Git as Git Platform participant Gateway as AI-Git-Bot participant Sec as Security AI participant Rel as Reliability AI participant Maint as Maintainability AI participant Human as Human Reviewer Dev->>Git: Open express-checkout PR Dev->>Git: Request three AI reviewers Git->>Gateway: Webhook for review requests Gateway->>Git: Fetch PR diff and context Gateway->>Sec: Send diff with security prompt Gateway->>Rel: Send diff with reliability prompt Gateway->>Maint: Send diff with maintainability prompt Sec-->>Gateway: Security review Rel-->>Gateway: Reliability review Maint-->>Gateway: Maintainability review Gateway->>Git: Post reviews/comments Dev->>Git: Push fixes and ask @bot follow-ups Human->>Git: Final approval or request changes </div><hr> <h2 id="conclusion" tabindex="-1"><a class="header-anchor" href="#conclusion">Conclusion</a></h2> <p>AI-Git-Bot improves software quality when it is treated as a configurable review gateway rather than a single generic chatbot. By creating multiple reviewer personas, teams can make quality concerns explicit and repeatable:</p> <ul> <li>the <strong>Security Reviewer</strong> protects users, secrets, permissions, and sensitive data,</li> <li>the <strong>Reliability & Performance Reviewer</strong> protects production stability and scalability,</li> <li>the <strong>Testability & Maintainability Reviewer</strong> protects long-term code health.</li> </ul> <p>In the ShopFlow example, the same pull request receives three independent perspectives before a human reviewer makes the merge decision. This creates a stronger review process without replacing existing best practices such as CI, tests, static analysis, dependency scanning, and human ownership.</p> <p>The practical result in the Git system is simple: different bot identities appear as different reviewers, each guided by its own system prompt, each posting focused feedback directly into the pull request. Over time, this turns quality standards from informal reviewer preferences into a visible, repeatable, and continuously improvable review system.</p> <hr> <h2 id="short-note%3A-getting-and-installing-the-gateway" tabindex="-1"><a class="header-anchor" href="#short-note%3A-getting-and-installing-the-gateway">Short Note: Getting and Installing the Gateway</a></h2> <p>AI-Git-Bot is available from the project repository at <a href="https://github.com/tmseidel/ai-git-bot">https://github.com/tmseidel/ai-git-bot</a>. The documented Docker image is <code>tmseidel/ai-git-bot:latest</code>, published on Docker Hub as <code>tmseidel/ai-git-bot</code>.</p> <p>For a quick installation, clone the repository and start the provided Docker Compose setup:</p> <pre><code class="language-bash">git clone https://github.com/tmseidel/ai-git-bot.git cd ai-git-bot docker compose up -d </code></pre> <p>The gateway then runs as a web application, typically on port <code>8080</code>, where administrators can create AI integrations, Git integrations, system prompt entries, and reviewer bots.</p> Tom Seidel https://remus-software.org/articles/kb_two_docker/ 2026-04-11T00:00:00.000Z 2026-04-11T00:00:00.000Z

A practical guide to running both Docker Engine natively in WSL2 and Docker Desktop for Windows in complete isolation — diagnosing conflicts, fixing networking issues, and configuring coexistence.

<p>This guide explains why you might need both Docker Engine (installed natively in WSL2) and Docker Desktop for Windows, what conflicts arise when both are present, and how to configure them to run in complete isolation.</p> <h2 id="why-you-need-both" tabindex="-1"><a class="header-anchor" href="#why-you-need-both">Why You Need Both</a></h2> <h3 id="docker-engine-in-wsl2-%E2%80%94-for-local-development" tabindex="-1"><a class="header-anchor" href="#docker-engine-in-wsl2-%E2%80%94-for-local-development">Docker Engine in WSL2 — For Local Development</a></h3> <p>For local development and testing, you run containers directly in WSL2 using the native Docker Engine (installed via <code>apt</code> from Docker’s official repository).</p> <p>Key advantages:</p> <ul> <li>Containers share the WSL2 network stack — <code>host.docker.internal:host-gateway</code> routes to the real host IP (<code>172.x.x.x</code>), so containers can reach services running on the host (e.g., a web server on port 8080)</li> <li>Fast, lightweight, and fully Linux-native</li> <li>No dependency on a Windows GUI application</li> </ul> <h3 id="docker-desktop-%E2%80%94-why-it-might-be-installed" tabindex="-1"><a class="header-anchor" href="#docker-desktop-%E2%80%94-why-it-might-be-installed">Docker Desktop — Why It Might Be Installed</a></h3> <p>There are several reasons why you might also have Docker Desktop for Windows installed:</p> <ul> <li><strong>Remote SSH deployments from Windows:</strong> Docker Compose supports deploying to remote servers via SSH contexts (<code>docker context create my-server --docker "host=ssh://user@server"</code>). When running this from a <strong>Windows terminal</strong> (PowerShell, CMD), Docker Desktop provides the Docker daemon. While SSH-based Docker contexts work perfectly fine from native Linux (including WSL2), some workflows require running <code>docker compose</code> from the Windows side — e.g., CI/CD scripts running on Windows, or using Windows-native tooling.</li> <li><strong>Windows container support:</strong> Docker Desktop can run Windows containers (for .NET Framework apps, IIS, etc.), which Docker Engine in WSL2 cannot.</li> <li><strong>GUI-based management:</strong> Docker Desktop provides a graphical dashboard for container management, image browsing, and resource configuration.</li> <li><strong>Corporate/team requirement:</strong> Your organization may standardize on Docker Desktop for license compliance or support reasons.</li> <li><strong>Kubernetes integration:</strong> Docker Desktop includes a single-node Kubernetes cluster for local testing.</li> </ul> <p>The key point is: <strong>you don’t need to uninstall Docker Desktop</strong> — you just need to prevent it from interfering with the Docker Engine in WSL2.</p> <h3 id="summary" tabindex="-1"><a class="header-anchor" href="#summary">Summary</a></h3> <table> <thead> <tr> <th>Use Case</th> <th>Tool</th> <th>Where to Run</th> </tr> </thead> <tbody> <tr> <td>Local development containers</td> <td>Docker Engine in WSL2</td> <td>WSL2 terminal</td> </tr> <tr> <td>Host service + container integration testing</td> <td>Docker Engine in WSL2</td> <td>WSL2 terminal</td> </tr> <tr> <td>SSH remote deployment from WSL2</td> <td>Docker Engine in WSL2</td> <td>WSL2 terminal</td> </tr> <tr> <td>Windows containers, GUI management, K8s</td> <td>Docker Desktop</td> <td>Windows terminal / PowerShell</td> </tr> <tr> <td>Corporate/team-mandated Docker usage on Windows</td> <td>Docker Desktop</td> <td>Windows terminal / PowerShell</td> </tr> </tbody> </table> <h2 id="the-conflict" tabindex="-1"><a class="header-anchor" href="#the-conflict">The Conflict</a></h2> <p>When Docker Desktop is installed, it enables <strong>WSL2 Integration</strong> by default (Settings → Resources → WSL Integration). This injects Docker Desktop’s own binaries and configuration into your WSL2 distro, overriding the native Docker Engine. The two cannot coexist when this integration is active.</p> <h3 id="what-goes-wrong" tabindex="-1"><a class="header-anchor" href="#what-goes-wrong">What Goes Wrong</a></h3> <table> <thead> <tr> <th>Symptom</th> <th>Cause</th> </tr> </thead> <tbody> <tr> <td><code>host.docker.internal</code> resolves to <code>192.168.65.254</code> instead of <code>172.17.0.1</code></td> <td>Docker Desktop routes through its internal VM gateway, not the WSL2 bridge</td> </tr> <tr> <td><code>curl http://host.docker.internal:<port></code> fails from containers</td> <td>The VM gateway (<code>192.168.65.254</code>) does not forward to WSL2 host ports</td> </tr> <tr> <td><code>/run/docker.sock</code> or <code>/var/run/docker.sock</code> does not exist</td> <td>Docker Desktop replaces the socket with its own, and the native <code>docker.socket</code> systemd unit gets confused</td> </tr> <tr> <td><code>docker context ls</code> shows <code>desktop-linux</code> context</td> <td>Docker Desktop injected its context into WSL2</td> </tr> <tr> <td><code>docker ps</code> shows different containers than expected</td> <td>CLI is talking to Docker Desktop’s daemon instead of the local one</td> </tr> <tr> <td>Webhooks or callbacks from containers never reach a host service</td> <td>Container → <code>host.docker.internal</code> → Docker Desktop VM → <strong>dead end</strong> (not forwarded to WSL2)</td> </tr> <tr> <td><code>--network host</code> still doesn’t reach host services</td> <td>On Docker Desktop, <code>host</code> mode connects to the VM’s network, not WSL2’s</td> </tr> </tbody> </table> <h3 id="understanding-the-routing-problem" tabindex="-1"><a class="header-anchor" href="#understanding-the-routing-problem">Understanding the Routing Problem</a></h3> <p>When a container needs to reach a service on your WSL2 host (e.g., a web application running on port 8080), it uses <code>host.docker.internal</code>. How this hostname resolves determines whether the connection works.</p> <p><strong>With Docker Desktop WSL2 Integration (broken):</strong></p> <pre><code>Container → host.docker.internal (192.168.65.254) → Docker Desktop VM → Windows host ✘ WSL2 host (not forwarded) </code></pre> <p>The traffic goes through Docker Desktop’s internal VM, reaches the Windows host, but is never forwarded to the WSL2 instance where your service is actually running.</p> <p><strong>With native Docker Engine in WSL2 (working):</strong></p> <pre><code>Container → host.docker.internal (172.17.0.1) → WSL2 host ✔ (direct bridge route) </code></pre> <p>The traffic goes directly over the Docker bridge network to the WSL2 host — no intermediate VM, no forwarding issues.</p> <h2 id="diagnosis" tabindex="-1"><a class="header-anchor" href="#diagnosis">Diagnosis</a></h2> <p>If you suspect Docker Desktop is interfering with your WSL2 Docker Engine, run these commands inside your WSL2 terminal:</p> <h3 id="1.-check-which-docker-binary-is-active" tabindex="-1"><a class="header-anchor" href="#1.-check-which-docker-binary-is-active">1. Check which Docker binary is active</a></h3> <pre><code class="language-bash">which docker ls -la $(which docker) </code></pre> <ul> <li>✅ Native: <code>/usr/bin/docker</code> owned by <code>root</code></li> <li>❌ Desktop override: symlink to <code>/mnt/wsl/docker-desktop/...</code> or similar</li> </ul> <h3 id="2.-check-docker-contexts" tabindex="-1"><a class="header-anchor" href="#2.-check-docker-contexts">2. Check Docker contexts</a></h3> <pre><code class="language-bash">docker context ls </code></pre> <ul> <li>✅ Only <code>default</code> pointing to <code>unix:///var/run/docker.sock</code></li> <li>❌ A <code>desktop-linux</code> context exists (Docker Desktop injected it)</li> </ul> <h3 id="3.-check-the-docker-socket" tabindex="-1"><a class="header-anchor" href="#3.-check-the-docker-socket">3. Check the Docker socket</a></h3> <pre><code class="language-bash">ls -la /run/docker.sock </code></pre> <ul> <li>✅ Socket file exists: <code>srw-rw---- root docker ... /run/docker.sock</code></li> <li>❌ File does not exist, or is a symlink to a Docker Desktop path</li> </ul> <h3 id="4.-check-host.docker.internal-from-inside-a-container" tabindex="-1"><a class="header-anchor" href="#4.-check-host.docker.internal-from-inside-a-container">4. Check <code>host.docker.internal</code> from inside a container</a></h3> <pre><code class="language-bash">docker run --rm --add-host=host.docker.internal:host-gateway \ busybox cat /etc/hosts | grep host.docker </code></pre> <ul> <li>✅ <code>172.17.0.1 host.docker.internal</code> (WSL2 Docker bridge gateway)</li> <li>❌ <code>192.168.65.254 host.docker.internal</code> (Docker Desktop VM gateway)</li> </ul> <h3 id="5.-test-container-to-host-connectivity" tabindex="-1"><a class="header-anchor" href="#5.-test-container-to-host-connectivity">5. Test container-to-host connectivity</a></h3> <p>If you have a service running on the host (e.g., on port 8080):</p> <pre><code class="language-bash">docker run --rm --add-host=host.docker.internal:host-gateway \ curlimages/curl curl -s -o /dev/null -w "HTTP %{http_code}" \ --connect-timeout 3 http://host.docker.internal:8080/ </code></pre> <ul> <li>✅ <code>HTTP 200</code> (or any non-zero HTTP status)</li> <li>❌ <code>HTTP 000</code> (connection failed)</li> </ul> <h2 id="resolution" tabindex="-1"><a class="header-anchor" href="#resolution">Resolution</a></h2> <h3 id="step-1%3A-disable-docker-desktop-wsl2-integration" tabindex="-1"><a class="header-anchor" href="#step-1%3A-disable-docker-desktop-wsl2-integration">Step 1: Disable Docker Desktop WSL2 Integration</a></h3> <p>This is the critical step. In Docker Desktop:</p> <ol> <li>Open <strong>Docker Desktop</strong></li> <li>Go to <strong>Settings → Resources → WSL Integration</strong></li> <li><strong>Disable</strong> “Enable integration with my default WSL distro”</li> <li><strong>Disable</strong> the toggle for your Ubuntu (or other) WSL2 distro</li> <li>Click <strong>Apply & Restart</strong></li> </ol> <p>Docker Desktop will continue working from Windows terminals (PowerShell, CMD), but it will stop injecting itself into WSL2.</p> <h3 id="step-2%3A-remove-the-docker-desktop-context-from-wsl2" tabindex="-1"><a class="header-anchor" href="#step-2%3A-remove-the-docker-desktop-context-from-wsl2">Step 2: Remove the Docker Desktop context from WSL2</a></h3> <pre><code class="language-bash">docker context rm desktop-linux </code></pre> <p>Verify only the native context remains:</p> <pre><code class="language-bash">docker context ls </code></pre> <p>Expected output:</p> <pre><code>NAME DESCRIPTION DOCKER ENDPOINT default * Current DOCKER_HOST based configuration unix:///var/run/docker.sock </code></pre> <h3 id="step-3%3A-restart-the-native-docker-engine" tabindex="-1"><a class="header-anchor" href="#step-3%3A-restart-the-native-docker-engine">Step 3: Restart the native Docker Engine</a></h3> <p>If the Docker socket is missing after disabling Desktop integration:</p> <pre><code class="language-bash">sudo systemctl stop docker docker.socket sudo systemctl start docker.socket sudo systemctl start docker </code></pre> <p>Verify the socket exists:</p> <pre><code class="language-bash">ls -la /run/docker.sock # Expected: srw-rw---- 1 root docker 0 ... /run/docker.sock </code></pre> <h3 id="step-4%3A-verify-container-to-host-connectivity" tabindex="-1"><a class="header-anchor" href="#step-4%3A-verify-container-to-host-connectivity">Step 4: Verify container-to-host connectivity</a></h3> <p>Start any service on the host (or use a simple test server), then:</p> <pre><code class="language-bash"># Quick test: start a temporary HTTP server on the host python3 -m http.server 9999 & TEST_PID=$! # Test from a container docker run --rm --add-host=host.docker.internal:host-gateway \ curlimages/curl curl -s -o /dev/null -w "HTTP %{http_code}" \ --connect-timeout 3 http://host.docker.internal:9999/ # Clean up kill $TEST_PID </code></pre> <p>Expected result: <code>HTTP 200</code></p> <h3 id="step-5%3A-recreate-your-running-containers" tabindex="-1"><a class="header-anchor" href="#step-5%3A-recreate-your-running-containers">Step 5: Recreate your running containers</a></h3> <p>Existing containers were created under the old networking configuration. You need to recreate them for the fix to take effect:</p> <pre><code class="language-bash">docker compose -f <your-compose-file.yml> down docker compose -f <your-compose-file.yml> up -d </code></pre> <blockquote> <p><strong>Note:</strong> If your compose file uses named volumes and you’re also changing container image versions, you may need <code>down -v</code> to remove old volumes. Only use <code>-v</code> if you don’t mind losing persisted data.</p> </blockquote> <h2 id="running-in-isolation" tabindex="-1"><a class="header-anchor" href="#running-in-isolation">Running in Isolation</a></h2> <p>After completing the resolution steps, the two Docker installations run independently:</p> <table> <thead> <tr> <th></th> <th>Docker Engine (WSL2)</th> <th>Docker Desktop (Windows)</th> </tr> </thead> <tbody> <tr> <td><strong>Access from</strong></td> <td>WSL2 terminal (<code>bash</code>)</td> <td>Windows terminal (PowerShell, CMD)</td> </tr> <tr> <td><strong>Daemon</strong></td> <td><code>dockerd</code> via systemd in WSL2</td> <td>Docker Desktop VM</td> </tr> <tr> <td><strong>Socket</strong></td> <td><code>/run/docker.sock</code></td> <td><code>npipe:////./pipe/docker_engine</code></td> </tr> <tr> <td><strong>Containers</strong></td> <td>Separate set</td> <td>Separate set</td> </tr> <tr> <td><strong>Networks</strong></td> <td>WSL2 bridge (<code>172.17.0.0/16</code>)</td> <td>Desktop VM bridge (<code>192.168.65.0/24</code>)</td> </tr> <tr> <td><strong><code>host.docker.internal</code></strong></td> <td><code>172.17.0.1</code> (WSL2 host)</td> <td><code>192.168.65.254</code> (Windows host)</td> </tr> <tr> <td><strong>Use for</strong></td> <td>Local dev, testing, SSH deployments</td> <td>Windows containers, GUI, K8s</td> </tr> </tbody> </table> <h3 id="using-docker-desktop-from-windows" tabindex="-1"><a class="header-anchor" href="#using-docker-desktop-from-windows">Using Docker Desktop from Windows</a></h3> <p>When you need Docker Desktop features (Windows containers, GUI, Kubernetes), open <strong>PowerShell</strong> (not WSL2):</p> <pre><code class="language-powershell"># In PowerShell / Windows Terminal docker ps docker compose -f docker-compose.yml up -d </code></pre> <h3 id="using-docker-engine-from-wsl2" tabindex="-1"><a class="header-anchor" href="#using-docker-engine-from-wsl2">Using Docker Engine from WSL2</a></h3> <p>For local development, use your WSL2 terminal as usual:</p> <pre><code class="language-bash"># In WSL2 bash docker compose -f docker-compose.yml up -d </code></pre> <h2 id="ensuring-docker-starts-on-wsl2-boot" tabindex="-1"><a class="header-anchor" href="#ensuring-docker-starts-on-wsl2-boot">Ensuring Docker Starts on WSL2 Boot</a></h2> <p>WSL2 doesn’t always start systemd services automatically. To ensure Docker Engine is available when you open a WSL2 terminal:</p> <pre><code class="language-bash"># Enable Docker to start with systemd sudo systemctl enable docker.service sudo systemctl enable docker.socket </code></pre> <p>If your WSL2 distro doesn’t have systemd enabled, add this to <code>/etc/wsl.conf</code>:</p> <pre><code class="language-ini">[boot] systemd=true </code></pre> <p>Then restart WSL2 from PowerShell:</p> <pre><code class="language-powershell">wsl --shutdown </code></pre> <h2 id="troubleshooting" tabindex="-1"><a class="header-anchor" href="#troubleshooting">Troubleshooting</a></h2> <h3 id="%E2%80%9Cinteractive-authentication-required%E2%80%9D-when-starting-docker" tabindex="-1"><a class="header-anchor" href="#%E2%80%9Cinteractive-authentication-required%E2%80%9D-when-starting-docker">“Interactive authentication required” when starting Docker</a></h3> <pre><code>Failed to start docker.service: Interactive authentication required. </code></pre> <p>You ran <code>systemctl start docker</code> without <code>sudo</code>. Use:</p> <pre><code class="language-bash">sudo systemctl start docker </code></pre> <p>To avoid needing <code>sudo</code> for Docker commands (not for systemctl), add your user to the <code>docker</code> group:</p> <pre><code class="language-bash">sudo usermod -aG docker $USER newgrp docker </code></pre> <h3 id="docker-socket-disappears-after-reboot" tabindex="-1"><a class="header-anchor" href="#docker-socket-disappears-after-reboot">Docker socket disappears after reboot</a></h3> <p>Docker Desktop’s WSL2 integration was re-enabled (e.g., after a Docker Desktop update), or the systemd socket unit didn’t start. Check and fix:</p> <pre><code class="language-bash"># Check if Desktop integration is back docker context ls # Should NOT show desktop-linux # Restart the socket sudo systemctl restart docker.socket docker ls -la /run/docker.sock </code></pre> <p>If <code>desktop-linux</code> reappeared, Docker Desktop re-enabled WSL2 integration — repeat <a href="#step-1-disable-docker-desktop-wsl2-integration">Step 1</a> of the resolution.</p> <h3 id="container-can-ping-host.docker.internal-but-can%E2%80%99t-connect-to-a-port" tabindex="-1"><a class="header-anchor" href="#container-can-ping-host.docker.internal-but-can%E2%80%99t-connect-to-a-port">Container can ping <code>host.docker.internal</code> but can’t connect to a port</a></h3> <p>The hostname resolves but the port is unreachable. Verify:</p> <ol> <li> <p><strong>The service is actually running on the host:</strong></p> <pre><code class="language-bash">curl http://localhost:<port>/ </code></pre> </li> <li> <p><strong>Check what IP the container sees:</strong></p> <pre><code class="language-bash">docker exec <container-name> cat /etc/hosts | grep host.docker </code></pre> <ul> <li>If <code>192.168.65.254</code> → Docker Desktop is still interfering (see <a href="#resolution">Resolution</a>)</li> <li>If <code>172.17.0.1</code> → the service might not be running, might be bound to <code>127.0.0.1</code> only, or a firewall is blocking</li> </ul> </li> <li> <p><strong>Verify the service listens on all interfaces:</strong></p> <pre><code class="language-bash">ss -tlnp | grep <port> </code></pre> <ul> <li>✅ <code>*:<port></code> or <code>0.0.0.0:<port></code> — listening on all interfaces</li> <li>❌ <code>127.0.0.1:<port></code> — only localhost; reconfigure the service to bind to <code>0.0.0.0</code></li> </ul> </li> </ol> <h3 id="docker-compose-shows-containers-from-the-wrong-docker" tabindex="-1"><a class="header-anchor" href="#docker-compose-shows-containers-from-the-wrong-docker"><code>docker compose</code> shows containers from the wrong Docker</a></h3> <p>If <code>docker ps</code> in WSL2 shows containers you created in Docker Desktop (or vice versa), the wrong context is active:</p> <pre><code class="language-bash">docker context ls # Check which context is active (marked with *) docker context use default # Switch to native WSL2 Docker </code></pre> <h3 id="docker-desktop-updates-re-enable-wsl2-integration" tabindex="-1"><a class="header-anchor" href="#docker-desktop-updates-re-enable-wsl2-integration">Docker Desktop updates re-enable WSL2 integration</a></h3> <p>Docker Desktop may re-enable WSL2 integration after updates. If things suddenly break again after a Docker Desktop update, re-check Settings → Resources → WSL Integration and disable it again. Consider disabling Docker Desktop auto-updates if this becomes a recurring issue.</p> Tom Seidel https://remus-software.org/articles/how-to-develop-an-ai-agent/ 2026-04-07T00:00:00.000Z 2026-04-07T00:00:00.000Z

Practical lessons learned from developing an agentic AI system that generates source code — from naive prompting to a robust, tool-using agent with a generic loop, schema-validated plans, provider-native function calls, and an optional self-critique step.

<p>Building software with an AI agent at its core is fundamentally different from traditional application development. Over the course of thirteen iterations — nine on the agent’s behaviour, four more on the surrounding <em>architecture</em> of the host — I developed an AI agent that reads GitHub/Gitea issues, generates implementation code, validates it, and commits the result, all autonomously. This article distils the key lessons learned along the way.</p> <h2 id="the-paradigm-shift%3A-inversion-of-control" tabindex="-1"><a class="header-anchor" href="#the-paradigm-shift%3A-inversion-of-control">The Paradigm Shift: Inversion of Control</a></h2> <p>The single biggest mental shift when building agentic systems is the <strong>inversion of control flow</strong>. In traditional software, the application owns the business logic and orchestrates every step. In an agentic system, a significant portion of that decision-making shifts to the AI — the surrounding program increasingly acts as a communication partner, executing commands on the agent’s behalf.</p> <div class="mermaid">graph LR subgraph Traditional App[Application] -->|calls| Lib[Library / API] App -->|orchestrates| DB[(Database)] end </div><div class="mermaid">graph LR subgraph Agentic Agent[AI Agent] -->|requests action| Host[Host Program] Host -->|returns result| Agent Agent -->|requests tool| Host Host -->|executes & returns| Agent end </div><p>Your application becomes, to a large degree, an <strong>execution environment</strong> for the agent. It provides tools, fetches context, applies file changes, and reports results — while the agent takes on much of the reasoning about <em>what</em> to do. In practice, the split is not black and white — which is exactly what the next section explores.</p> <h2 id="designing-the-agent-flow%3A-what-to-keep%2C-what-to-delegate" tabindex="-1"><a class="header-anchor" href="#designing-the-agent-flow%3A-what-to-keep%2C-what-to-delegate">Designing the Agent Flow: What to Keep, What to Delegate</a></h2> <p>Once you accept the inversion of control, the next critical question is: <strong>which actions belong in your application, and which do you delegate to the LLM?</strong> This is not an all-or-nothing decision — it’s a spectrum, and every point on it has trade-offs.</p> <h3 id="keeping-logic-in-the-application" tabindex="-1"><a class="header-anchor" href="#keeping-logic-in-the-application">Keeping Logic in the Application</a></h3> <p>Deterministic steps — file I/O, git operations, API calls, JSON parsing, diff application — are natural candidates for application-side logic.</p> <p><strong>Advantages:</strong></p> <ul> <li><strong>Predictable and fast.</strong> Same input, same output, every time. No API latency, no token cost.</li> <li><strong>Testable.</strong> You can write unit tests with clear assertions.</li> <li><strong>Debuggable.</strong> Stack traces, breakpoints, and logging work exactly as expected.</li> </ul> <p><strong>Disadvantages:</strong></p> <ul> <li><strong>Rigid.</strong> You must anticipate every edge case upfront. An unexpected file format or an unusual error message breaks the flow.</li> <li><strong>More code to maintain.</strong> Every special case becomes an <code>if</code> branch or a new parser.</li> </ul> <h3 id="delegating-logic-to-the-llm" tabindex="-1"><a class="header-anchor" href="#delegating-logic-to-the-llm">Delegating Logic to the LLM</a></h3> <p>Reasoning tasks — deciding <em>what</em> to change, interpreting error messages, choosing a fix strategy, analysing code structure — are where the LLM excels.</p> <p><strong>Advantages:</strong></p> <ul> <li><strong>Flexible and adaptive.</strong> The model handles novel situations without explicit programming.</li> <li><strong>Reduces code complexity.</strong> A single prompt can replace hundreds of lines of hand-coded decision trees.</li> <li><strong>Understands intent.</strong> The AI can reason about <em>why</em> something should change, not just <em>what</em> the syntax rules say.</li> </ul> <p><strong>Disadvantages:</strong></p> <ul> <li><strong>Non-deterministic.</strong> The same input can produce different outputs on each run.</li> <li><strong>Slower.</strong> Each LLM call adds 5–30 seconds of latency depending on the model and input size.</li> <li><strong>Token cost.</strong> Every delegation costs money and consumes context window space.</li> <li><strong>Hallucination risk.</strong> The model may confidently produce incorrect results.</li> <li><strong>Harder to test.</strong> Assertions on LLM output are inherently fuzzy.</li> </ul> <h3 id="finding-the-right-split" tabindex="-1"><a class="header-anchor" href="#finding-the-right-split">Finding the Right Split</a></h3> <p>In practice, the division that worked best for our code generation agent was:</p> <div class="mermaid">flowchart LR subgraph "Application (deterministic)" A[File I/O] B[Git operations] C[Diff application] D[JSON parsing] E[Tool execution] end subgraph "LLM (reasoning)" F[Code generation] G[Error analysis] H[Fix strategy] I[Choosing validation tools] J[Context requests] end E -->|results| G G -->|decisions| A </div><p>The guiding principle: <strong>if a step requires understanding intent or adapting to ambiguity, delegate it. If it requires reliability and speed, keep it in the application.</strong> As you will see throughout the iterations below, we gradually moved <em>more</em> logic to the AI side — not because we wanted to, but because the deterministic alternatives kept failing on edge cases.</p> <h2 id="the-tightrope-walk%3A-context-window-vs.-output-quality" tabindex="-1"><a class="header-anchor" href="#the-tightrope-walk%3A-context-window-vs.-output-quality">The Tightrope Walk: Context Window vs. Output Quality</a></h2> <p>One of the most persistent challenges is balancing the context window. Too little context, and the AI hallucinates imports, invents method signatures, or misses existing patterns. Too much context, and the model loses focus, produces lower-quality output, or exceeds token limits.</p> <blockquote> <p>The sweet spot shifts with every model generation, but the tension never goes away.</p> </blockquote> <p>This interplay shaped almost every iteration of the agent.</p> <h2 id="the-iteration-dilemma%3A-how-many-loops%3F" tabindex="-1"><a class="header-anchor" href="#the-iteration-dilemma%3A-how-many-loops%3F">The Iteration Dilemma: How Many Loops?</a></h2> <p>Closely related to the context problem is the question of <strong>how many iterations to allow</strong> between your application and the LLM. LLMs have a remarkable ability to improve their output when given feedback — a compilation error sent back to the model often results in a correct fix on the second attempt. This makes iterative correction loops very attractive.</p> <p>But iteration comes at a price:</p> <ul> <li><strong>Time.</strong> Each round-trip adds 10–30 seconds of latency. Three retry loops turn a 15-second task into a two-minute task.</li> <li><strong>Context bloat.</strong> Every iteration appends messages to the conversation — the error report, the AI’s response, the next error report. The context window fills up fast, which degrades output quality (see above).</li> <li><strong>Hallucination drift.</strong> Counter-intuitively, too many iterations can make things <em>worse</em>. After three or four failed attempts, the AI tends to “drift” — introducing new errors while fixing old ones, inventing methods that don’t exist, or producing solutions that look plausible but are semantically wrong. The model starts optimising for <em>passing the immediate check</em> rather than <em>being correct</em>.</li> <li><strong>Cost.</strong> Each iteration consumes tokens. With large context windows, a single retry can cost as much as the original request.</li> </ul> <h3 id="practical-guards" tabindex="-1"><a class="header-anchor" href="#practical-guards">Practical Guards</a></h3> <p>Through experimentation, we arrived at the following guidelines:</p> <ul> <li><strong>Hard caps on every loop.</strong> No open-ended retries. We used 3 rounds for file requests, 5 rounds for code validation loops, and 3 rounds for diff recovery attempts.</li> <li><strong>Progress monitoring.</strong> If the error count isn’t decreasing between iterations, abort early. An AI that produces <em>more</em> errors after a fix attempt is unlikely to recover.</li> <li><strong>Compaction between rounds.</strong> Summarise earlier turns aggressively to keep the context lean (see Iteration 3).</li> <li><strong>Escalation, not repetition.</strong> If simple “fix this error” prompts fail twice, escalate to a richer strategy — provide more context, rephrase the problem, or fall back to a full file regeneration instead of incremental fixes.</li> </ul> <blockquote> <p><strong>Rule of thumb:</strong> If your agent hasn’t solved the problem in three iterations, throwing more iterations at it is unlikely to help — you need a different strategy, not more attempts.</p> </blockquote> <h2 id="core-principles" tabindex="-1"><a class="header-anchor" href="#core-principles">Core Principles</a></h2> <p>Before diving into the iterations, here are the overarching lessons:</p> <ol> <li><strong>Always give the AI the ability to request more information.</strong> Never assume your initial context is sufficient. Let the agent ask for files, type definitions, or documentation on demand.</li> <li><strong>Define a protocol in the system prompt for structured output</strong> — but build your application to be resilient against protocol violations. The AI will <em>sometimes</em> deviate from the agreed JSON schema, return partial responses, or mix formats. Your parser must handle this gracefully.</li> <li><strong>The agent drives the reasoning</strong> — your code provides the infrastructure and guardrails.</li> </ol> <h2 id="iteration-1-%E2%80%94-naive-generate-and-validate-loop" tabindex="-1"><a class="header-anchor" href="#iteration-1-%E2%80%94-naive-generate-and-validate-loop">Iteration 1 — Naive Generate-and-Validate Loop</a></h2> <p>The first version was straightforward: send the issue description to the AI, receive generated code, then validate it.</p> <div class="mermaid">flowchart TD A[Issue Description] --> B[AI generates code] B --> C[CodeValidationService validates all files] C --> D{Errors found?} D -->|Yes| E[Build error report] E --> F["Send to AI: 'Fix these syntax errors'"] F --> B D -->|No| G[Commit code] D -->|Max retries reached| H[Post warning in issue & commit anyway] </div><p><strong>What worked:</strong> The basic loop caught obvious syntax errors and gave the AI a chance to self-correct.</p> <p><strong>What didn’t work:</strong> The AI frequently generated code that referenced classes, methods, or interfaces it had never seen. Without sufficient context about the existing codebase, the output was often structurally correct but semantically wrong.</p> <h2 id="iteration-2-%E2%80%94-smarter-context-gathering" tabindex="-1"><a class="header-anchor" href="#iteration-2-%E2%80%94-smarter-context-gathering">Iteration 2 — Smarter Context Gathering</a></h2> <p>The root cause from Iteration 1 was clear: the AI didn’t know enough about the existing codebase. The <code>fetchRelevantFileContents()</code> method was significantly improved:</p> <ul> <li><strong>Package awareness:</strong> When a file is mentioned, all files in the same package are loaded.</li> <li><strong>Partial name matching:</strong> The word “Task” in an issue matches <code>Task.java</code>, <code>TaskService.java</code>, <code>TaskRepository.java</code>, etc.</li> <li><strong>Domain structure recognition:</strong> Files in <code>/domain/</code>, <code>/model/</code>, <code>/entity/</code>, <code>/config/</code>, <code>/dto/</code>, <code>/repository/</code>, <code>/service/</code>, and <code>/controller/</code> directories are included when they relate to the issue.</li> <li><strong>Increased file limit:</strong> From 15 to 30 files.</li> </ul> <p>This gave the AI visibility into existing method signatures, interface definitions, inheritance hierarchies, and repository methods — drastically reducing hallucinated references.</p> <p><strong>Lesson:</strong> Context quality matters more than prompt engineering. A perfectly worded prompt with missing context will always lose to a mediocre prompt with complete context.</p> <h2 id="iteration-3-%E2%80%94-conversation-compaction" tabindex="-1"><a class="header-anchor" href="#iteration-3-%E2%80%94-conversation-compaction">Iteration 3 — Conversation Compaction</a></h2> <p>With richer context and multi-turn code review conversations, the context window filled up fast. After several rounds of back-and-forth, the conversation could easily exceed 100 KB of tokens.</p> <p>The solution: <strong>automatic compaction</strong> after every code review interaction. The system retains only the last four messages plus a short summary of the earlier conversation.</p> <div class="mermaid">flowchart LR A["100 KB+ conversation"] --> B[Compaction] B --> C["Summary + last 4 messages"] C --> D["~10 KB context"] </div><p><strong>Lesson:</strong> LLMs work best with focused context. Aggressively summarise historical turns — the AI doesn’t need the full transcript, just the current state and a brief recap.</p> <h2 id="iteration-4-%E2%80%94-prompt-deduplication" tabindex="-1"><a class="header-anchor" href="#iteration-4-%E2%80%94-prompt-deduplication">Iteration 4 — Prompt Deduplication</a></h2> <p>A careful audit of all prompts revealed massive redundancy:</p> <ul> <li>Instructions already present in the system prompt were repeated in every user prompt.</li> <li>The full repository tree was sent with every continuation, even though the AI already had it from the previous turn.</li> <li>Verbose formatting instructions (Markdown headers, extra blank lines) were duplicated.</li> </ul> <p><strong>Changes made:</strong></p> <ul> <li><code>"Output your response as a JSON object with the structure described in the system prompt"</code> → <code>"Output JSON per system prompt format"</code></li> <li><code>treeContext</code> removed from <code>buildContinuationPrompt</code> — the AI retains it from the conversation history.</li> <li>Repeated formatting directives eliminated.</li> </ul> <p><strong>Lesson:</strong> Treat your prompts like production code. Audit them for duplication, dead instructions, and unnecessary verbosity. Every wasted token is context the AI could have used for actual reasoning.</p> <h2 id="iteration-5-%E2%80%94-diff-based-updates-and-dynamic-file-requests" tabindex="-1"><a class="header-anchor" href="#iteration-5-%E2%80%94-diff-based-updates-and-dynamic-file-requests">Iteration 5 — Diff-Based Updates and Dynamic File Requests</a></h2> <p>This was the most impactful single iteration. Two major features were introduced:</p> <h3 id="diff-based-changes" tabindex="-1"><a class="header-anchor" href="#diff-based-changes">Diff-Based Changes</a></h3> <p>Instead of returning entire files for every small change, the AI now returns <strong>SEARCH/REPLACE diffs</strong>:</p> <pre><code class="language-json">{ "fileChanges": [ { "path": "src/main/java/com/example/Task.java", "operation": "UPDATE", "diff": "<<<<<<< SEARCH\nprivate String name;\n=======\nprivate String name;\nprivate String description;\n>>>>>>> REPLACE" } ] } </code></pre> <p>A new <code>DiffApplyService</code> applies these blocks to the actual file content.</p> <h3 id="dynamic-file-requests" tabindex="-1"><a class="header-anchor" href="#dynamic-file-requests">Dynamic File Requests</a></h3> <p>The AI can now respond with a <strong>file request</strong> instead of code changes:</p> <pre><code class="language-json">{ "summary": "Need more context about the repository interface", "requestFiles": ["src/main/java/com/example/TaskRepository.java", "pom.xml"] } </code></pre> <p>The host program fetches the requested files and continues the conversation.</p> <p><strong>Token savings were dramatic:</strong></p> <table> <thead> <tr> <th>Scenario</th> <th>Before</th> <th>After</th> <th>Saving</th> </tr> </thead> <tbody> <tr> <td>Small change in a 500-line file</td> <td>~500 lines</td> <td>~10 lines (diff)</td> <td>~98%</td> </tr> <tr> <td>Follow-up without new files</td> <td>Tree + file list</td> <td>Only comment</td> <td>~90%</td> </tr> <tr> <td>Iterative requests</td> <td>All files again</td> <td>Only requested files</td> <td>~70%</td> </tr> </tbody> </table> <p><strong>Lesson:</strong> Give the AI the tools to be efficient. Diff-based output and on-demand file requests transformed a chatty, wasteful interaction into a focused, surgical one.</p> <p>In hindsight, though, this was an important <em>transitional</em> design, not the final one. Diff-based updates reduced token usage, but they also introduced a fragile mini-language that the host application had to interpret and repair.</p> <h2 id="iteration-6-%E2%80%94-robust-diff-application" tabindex="-1"><a class="header-anchor" href="#iteration-6-%E2%80%94-robust-diff-application">Iteration 6 — Robust Diff Application</a></h2> <p>Real-world diffs from the AI are messy. The <code>DiffApplyService</code> had to handle numerous edge cases:</p> <ul> <li><strong>Empty SEARCH blocks</strong> — content is appended to the file.</li> <li><strong>Placeholder comments</strong> like <code>/* Add existing... */</code> — treated as append operations.</li> <li><strong>Append patterns</strong> — when the REPLACE block starts with the SEARCH content and adds more, only the new part is appended.</li> <li><strong>Trailing whitespace differences</strong> — a fuzzy match is attempted before failing.</li> </ul> <p>Additionally, the <code>IssueImplementationService</code> was ignoring AI responses that contained <code>requestFiles</code> but no <code>fileChanges</code>, returning <code>null</code> instead. The fix:</p> <ul> <li>Detect <code>requestFiles</code> even when <code>fileChanges</code> is empty.</li> <li>Fetch the requested files and continue the conversation.</li> <li>Allow a maximum of <strong>three rounds</strong> of file requests to prevent infinite loops.</li> </ul> <p><strong>Lesson:</strong> The interface between AI output and your application is inherently fuzzy. Build robust parsers, add fallback strategies, and always cap iteration counts.</p> <p>That said, there is a deeper lesson here: every extra recovery rule in the host is a signal that the protocol itself may be too brittle. If you keep adding fuzzy matching, placeholder handling, and recovery paths, you may be compensating for the wrong abstraction.</p> <h2 id="iteration-7-%E2%80%94-ai-driven-validation-with-tools" tabindex="-1"><a class="header-anchor" href="#iteration-7-%E2%80%94-ai-driven-validation-with-tools">Iteration 7 — AI-Driven Validation with Tools</a></h2> <p>A fundamental architectural change: <strong>remove built-in validators entirely</strong> and let the AI decide how to validate its own output.</p> <p>The agent prompt was updated to make tool usage mandatory:</p> <blockquote> <p><em>“IMPORTANT: You MUST include <code>runTool</code> in every response that contains <code>fileChanges</code>. The bot does not have built-in validators — only you can determine how to validate the code by executing external tools.”</em></p> </blockquote> <p>The AI now specifies a validation command (e.g., <code>mvn compile</code>, <code>npm run build</code>, <code>gradle check</code>) alongside its code changes. If it forgets, the host program sends it back with a reminder.</p> <div class="mermaid">flowchart TD A[AI returns fileChanges + runTool] --> B[Host applies file changes] B --> C[Host executes specified tool] C --> D{Tool output} D -->|Success| E[Commit] D -->|Failure| F[Send tool output back to AI] F --> A G[AI returns fileChanges WITHOUT runTool] --> H["Host: 'Please specify a validation tool'"] H --> A </div><p><strong>Lesson:</strong> The AI often knows better than a hardcoded validator what constitutes “correct” in a given context. A Java project needs <code>mvn compile</code>; a Node project needs <code>npm run build</code>; a Python project might need <code>pytest</code>. Let the agent choose.</p> <h3 id="the-tool-selection-problem" tabindex="-1"><a class="header-anchor" href="#the-tool-selection-problem">The Tool Selection Problem</a></h3> <p>Letting the AI choose <em>which</em> tool to run immediately raises a thorny question: <strong>which tools do you allow it to execute at all?</strong></p> <p>This is one of the hardest design decisions in an agentic system, and the answer should be conservative: <strong>allow the absolute minimum set of tools needed to get the job done.</strong> Every tool you expose is:</p> <ul> <li><strong>A security risk.</strong> A build command like <code>mvn compile</code> is safe. An arbitrary shell command is not. The distance between “run my tests” and <code>rm -rf /</code> is one hallucinated token.</li> <li><strong>A source of complexity.</strong> More tool types mean more parsing, more error handling, more edge cases in your host program.</li> <li><strong>A surface for misuse.</strong> The AI might call tools in unexpected ways, with unexpected arguments, or in unexpected order. The more tools available, the larger the space of possible (and possibly harmful) interactions.</li> </ul> <p>For our code generation agent, the minimal toolset was:</p> <table> <thead> <tr> <th>Tool</th> <th>Purpose</th> </tr> </thead> <tbody> <tr> <td>Read file</td> <td>Fetch source code from the repository</td> </tr> <tr> <td>File tools (<code>write-file</code>, <code>patch-file</code>, <code>mkdir</code>, <code>delete-file</code>)</td> <td>Modify source code</td> </tr> <tr> <td>Execute build command</td> <td>Validate changes (<code>mvn compile</code>, <code>npm run build</code>, etc.)</td> </tr> <tr> <td>Request additional files</td> <td>Ask for more context</td> </tr> </tbody> </table> <p>We deliberately did <em>not</em> expose: arbitrary shell access, database queries, network requests, or deployment commands. Each tool that didn’t make this list was considered and rejected because it either wasn’t strictly necessary or introduced unacceptable risk.</p> <p><strong>Sandboxing is essential.</strong> Even with a minimal toolset, run tool executions in an isolated environment. Restrict file system access to the project directory. Set timeouts on build commands. Log every tool invocation for audit. The AI is not malicious, but it is unpredictable — and unpredictable + powerful = dangerous.</p> <blockquote> <p><strong>Principle:</strong> Start with zero tools and add them only when the agent demonstrably cannot complete its task without them. Resist the temptation to expose “just one more” convenience tool.</p> </blockquote> <h2 id="iteration-8-%E2%80%94-resilient-json-parsing" tabindex="-1"><a class="header-anchor" href="#iteration-8-%E2%80%94-resilient-json-parsing">Iteration 8 — Resilient JSON Parsing</a></h2> <p>With complex multi-turn conversations, the AI occasionally produced truncated or malformed JSON — especially near token limits. The <code>repairTruncatedJson</code> method was overhauled:</p> <ul> <li><strong>Check completeness first:</strong> Verify whether brackets are balanced before attempting any repair.</li> <li><strong>Only truncate genuinely incomplete JSON</strong> — previously, valid JSON was sometimes mangled by premature repair attempts.</li> <li><strong>Add <code>@NoArgsConstructor</code></strong> to all DTO classes to ensure Jackson can deserialize partial objects.</li> <li><strong>Parse <code>runTool</code></strong> as a proper typed object (<code>AiToolRequest</code>) instead of a raw map.</li> </ul> <p><strong>Lesson:</strong> When you define a structured protocol in the system prompt, the AI will follow it <em>most of the time</em> — perhaps 95%. Your application must handle the other 5% gracefully. Invest in resilient parsing, not stricter prompts.</p> <h2 id="iteration-9-%E2%80%94-ai-assisted-diff-recovery" tabindex="-1"><a class="header-anchor" href="#iteration-9-%E2%80%94-ai-assisted-diff-recovery">Iteration 9 — AI-Assisted Diff Recovery</a></h2> <p>Even with all the fuzzy matching from Iteration 6, diffs still sometimes failed to apply — typically because the file had been modified by a previous step in the same conversation, and the SEARCH block no longer matched.</p> <p>The elegant solution: <strong>ask the AI to resolve it</strong>.</p> <p>When a <code>DiffApplyException</code> occurs:</p> <ol> <li>Fetch the current file content from the repository.</li> <li>Send both the current content and the failed diff to the AI.</li> <li>Ask it to produce the complete new file content.</li> </ol> <div class="mermaid">flowchart TD A[Apply diff] --> B{DiffApplyException?} B -->|No| C[Success] B -->|Yes| D[Fetch current file from repo] D --> E["Send to AI:\n• Current file content\n• Failed diff\n• 'Produce complete new file'"] E --> F[AI returns full file content] F --> G[Use directly — no diff needed] </div><p>This is far more robust than implementing ever-more-complex matching strategies in the <code>DiffApplyService</code>. The AI sees the actual current state of the file and can produce the intended result directly.</p> <p><strong>Lesson:</strong> When your deterministic code fails, don’t add more deterministic complexity — delegate back to the AI. It can reason about intent in ways that string matching never will.</p> <h2 id="postscript-%E2%80%94-tool-requests-beat-fragile-diff-protocols" tabindex="-1"><a class="header-anchor" href="#postscript-%E2%80%94-tool-requests-beat-fragile-diff-protocols">Postscript — Tool Requests Beat Fragile Diff Protocols</a></h2> <p>After publishing the original version of this architecture, I revisited a later iteration of the agent and checked how the design had evolved over time. The pattern was clear: there were several rounds of hardening around diff handling, but eventually the <code>fileChanges</code> mechanism was removed entirely and replaced with tool-based file operations.</p> <p>That refactoring switched the protocol from <em>describing</em> edits as a custom JSON diff format to <em>requesting concrete actions</em>:</p> <ul> <li><code>write-file</code></li> <li><code>patch-file</code></li> <li><code>mkdir</code></li> <li><code>delete-file</code></li> </ul> <p>All of these are executed through <code>runTools</code>, alongside validation commands. In other words, file mutation stopped being a special side channel and became just another tool request.</p> <p>This turned out to be much more robust for three reasons:</p> <ol> <li>The contract is simpler. The host executes explicit operations instead of parsing and heuristically applying a synthetic diff language.</li> <li>Failures are clearer. <code>patch-file</code> either finds the exact text once or it fails with a precise error; there is less “maybe this is close enough” behavior.</li> <li>The conversation model is cleaner. The agent can use <code>requestTools</code> to inspect files first, then issue <code>runTools</code> for exact changes and validation in the next round.</li> </ol> <p>The most important updated lesson is this: <strong>when the bridge between agent and host becomes fragile, prefer executable tool requests over clever output formats.</strong> A protocol that asks the model to say <em>what to do</em> is usually sturdier than one that asks it to emit a compact patch language the host must interpret.</p> <p>Diffs were still a valuable intermediate step. They reduced token usage and forced a more structured contract. But in practice, explicit tool requests turned out to be the more durable endpoint.</p> <h2 id="iteration-10-%E2%80%94-from-a-hand-wired-loop-to-a-generic-agent-loop-with-strategies" tabindex="-1"><a class="header-anchor" href="#iteration-10-%E2%80%94-from-a-hand-wired-loop-to-a-generic-agent-loop-with-strategies">Iteration 10 — From a Hand-Wired Loop to a Generic Agent Loop with Strategies</a></h2> <p>By the time we had a second agent in the system — one that turns vague user reports into well-formed issues, alongside the original code-generating one — the original implementation loop had been copy-pasted, renamed, and quietly diverged. Both loops did almost the same thing:</p> <ul> <li>bump a round counter and check budgets,</li> <li>mirror every message into the persisted session <em>and</em> into an in-memory list,</li> <li>call the AI with <code>(history, message, systemPrompt, modelOverride, maxTokens)</code>,</li> <li>parse the response into a plan,</li> <li>branch on “context request” vs. “tool run” vs. “final”,</li> <li>and call back into the agent-specific finisher.</li> </ul> <p>The divergences were the dangerous part. One loop carried an <code>attempt--</code> hack that decremented the validation counter whenever the AI asked for more context, “because context lookups shouldn’t burn an implementation attempt.” The other had a different off-by-one on the same idea. Neither could be reasoned about without reading the surrounding 200 lines.</p> <h3 id="the-refactor" tabindex="-1"><a class="header-anchor" href="#the-refactor">The refactor</a></h3> <p>Instead of designing the new loop top-down, we started bottom-up by <em>characterising</em> the existing one. Several new tests pinned three branch combinations that nobody dared touch: multi-round validation retry, the “ignore non-blocking tool failures after a successful build” policy, and the “file-only success without any validation tool” path. Only with those tests green did we extract:</p> <ul> <li>a generic loop class that owns rounds, history mirroring, and the AI call,</li> <li>a sealed <code>StepDecision</code> type with exactly two cases (<code>Continue(nextMessage)</code> / <code>Finish(outcome)</code>) that the strategy returns each round,</li> <li>a small <code>Strategy</code> interface with <code>systemPrompt()</code>, <code>step(ctx, aiResponse, round)</code>, and <code>onBudgetExhausted(ctx)</code>,</li> <li>an immutable <code>Budget(maxRounds, maxContextRounds, maxValidationRetries, maxTokensPerCall)</code> value object,</li> <li>and a <code>RunContext</code> carrying per-run mutable state (the workspace, the current base branch, the issue identifier — whatever the agent needs to know).</li> </ul> <p>The <code>attempt--</code> hack was deleted. Context-fetch rounds and implementation attempts now use <em>separate</em> counters in the strategy. The session/history double-bookkeeping disappeared into the loop, where it belongs.</p> <div class="mermaid">flowchart TD A["loop.run()"] --> B["ai.chat() / ai.chatWithTools()"] B --> C["strategy.step(ctx, response, round)"] C -->|Continue| A C -->|Finish| D[strategy-specific finisher] A -. budget exhausted .-> E[strategy.onBudgetExhausted] </div><h3 id="lesson" tabindex="-1"><a class="header-anchor" href="#lesson">Lesson</a></h3> <blockquote> <p>Don’t refactor what you cannot reproduce. Characterisation tests are the cheapest possible insurance policy when ripping apart a loop full of off-by-one tricks.</p> </blockquote> <p>The second, subtler lesson surfaced months later: a sealed <code>StepDecision</code> interface forces every new control-flow case to opt in. We later considered adding an <code>Abort(reason)</code> variant for Iteration 13’s critic step — and the compiler immediately pointed at every <code>switch</code> that would need updating. The exhaustiveness check turned the loop from a behavioural surprise generator into something <em>typed</em>. That alone justified the refactor.</p> <h2 id="iteration-11-%E2%80%94-schema-validation%3A-trust%2C-but-verify-(quietly)" tabindex="-1"><a class="header-anchor" href="#iteration-11-%E2%80%94-schema-validation%3A-trust%2C-but-verify-(quietly)">Iteration 11 — Schema Validation: Trust, but Verify (Quietly)</a></h2> <p>The output protocol from Iteration 5 onwards was JSON-in-prompt. We documented its shape in the system prompt, deserialised it with a standard JSON library, and patched up violations in a thicket of helpers — extract-the-JSON-from-the-fenced-block (four strategies), truncate-to-first-balanced-object, repair-truncated-JSON, sanitise-invalid-escapes, find-the-last-complete-tool-call. The pile grew with every new model we tested.</p> <p>The real cost was not the helpers themselves but the <em>invisibility</em> of their work. We had no idea how often the AI actually violated the contract — and which fields it got wrong most often. The agent might be quietly recovering from 30 % schema violations and we wouldn’t know.</p> <h3 id="what-we-built" tabindex="-1"><a class="header-anchor" href="#what-we-built">What we built</a></h3> <ul> <li><strong>JSON-Schema documents</strong> (Draft 2020-12) for every plan shape the AI is allowed to return. They accept the de-facto aliases the models had already invented (<code>requestedFiles</code> next to <code>requestFiles</code>, singular <code>runTool</code> next to plural <code>runTools[]</code>) via <code>oneOf</code>, because legacy responses must remain valid.</li> <li>A <strong>validator component</strong> that runs <em>after</em> the existing extract/repair pipeline and <em>before</em> deserialisation.</li> <li>A <strong>counter</strong> like <code>agent.plan.schema_violations_total{agent=...}</code> exposed on the metrics endpoint.</li> <li>A <strong>feature flag</strong> <code>agent.schema.enforce</code> (default <code>false</code>).</li> <li><strong>Snapshot tests</strong> against real captured AI responses, plus a couple of deliberately broken negatives.</li> </ul> <p>The key design decision: in the default mode, <strong>the validator does not change behaviour at all.</strong> It logs the violation, increments the counter, and lets the existing repair heuristics do their job. The whole layer is observational.</p> <h3 id="lesson-1" tabindex="-1"><a class="header-anchor" href="#lesson-1">Lesson</a></h3> <blockquote> <p>Adding a strict validator next to a forgiving parser is <em>not</em> a contradiction. Run them in parallel for a release or two, measure how often the strict layer would have rejected, and only then decide whether to flip enforcement on.</p> </blockquote> <p>This is the boring-but-correct path. We did not “delete the messy parser and replace it with the proper one” — that would have broken production the same day. We added a measurement and waited.</p> <p>A second lesson, hidden in the implementation: the schemas turned out to be reusable. In Iteration 12 we needed a JSON-Schema for each tool the agent can call. We already had two well-tested plan schemas to learn from.</p> <h2 id="iteration-12-%E2%80%94-provider-native-function-calling-(without-burning-the-bridges)" tabindex="-1"><a class="header-anchor" href="#iteration-12-%E2%80%94-provider-native-function-calling-(without-burning-the-bridges)">Iteration 12 — Provider-Native Function Calling (Without Burning the Bridges)</a></h2> <p>All four major LLM providers we support now expose first-class tool calling. The provider parses the model’s tool-call intent server-side and hands you a structured <code>{ name, arguments }</code> object. No JSON-in-prompt, no fenced code blocks, no fuzzy extraction.</p> <p>We had wanted to use this from the start. We could not, because:</p> <ol> <li>Not every model supports it (older fine-tunes, certain self-hosted setups, raw-completion endpoints).</li> <li>Switching providers mid-flight would have invalidated months of accumulated prompt engineering.</li> <li>Operators of an existing deployment may have a strong reason — model choice, cost, debuggability — to stay on the legacy path even on providers that <em>do</em> support tools.</li> </ol> <p>So the design constraint was firm: native function calling had to be <strong>opt-out per provider configuration</strong>, with the JSON-in-prompt path remaining fully functional and fully tested.</p> <h3 id="the-contract-changes" tabindex="-1"><a class="header-anchor" href="#the-contract-changes">The contract changes</a></h3> <pre><code class="language-java">record ChatTurn(String assistantText, List<ToolCall> toolCalls, StopReason stop) {} record ToolCall(String id, String name, JsonNode args) {} record ToolDescriptor(String name, String description, JsonNode jsonSchema) {} interface AiClient { String chat(List<Message> history, String msg, String sys, String model, int maxTokens); // default delegates to chat(), so non-native clients work unchanged default ChatTurn chatWithTools(List<Message> history, String msg, List<ToolDescriptor> tools, String sys, String model, int maxTokens) { return ChatTurn.text(chat(history, msg, sys, model, maxTokens)); } default boolean supportsNativeTools() { return false; } } </code></pre> <p>Three things make this work:</p> <ul> <li><strong>The default method shields old clients.</strong> Anything that hasn’t been migrated keeps its plain-text <code>chat()</code> semantics, even when the loop asks for tools. No big-bang client rewrite.</li> <li><strong>A shared helper</strong> wraps a <code>chat()</code> call into a <code>ChatTurn</code>. Every native client uses it as its fallback when the operator-level kill switch is set.</li> <li><strong>The loop triple-gates the native path:</strong> the strategy must prefer <code>NATIVE</code>, the client must report <code>supportsNativeTools() == true</code>, <em>and</em> the strategy must actually expose at least one <code>ToolDescriptor</code>. Any missing condition falls back to the legacy path, with the reason logged.</li> </ul> <h3 id="a-per-configuration-kill-switch" tabindex="-1"><a class="header-anchor" href="#a-per-configuration-kill-switch">A per-configuration kill switch</a></h3> <p>A <code>useLegacyToolCalling</code> flag was added to the AI-provider configuration record. The admin UI got a switch with a popover explaining both modes:</p> <blockquote> <p><em>“Native tool calls give the model a structured tools API and are the recommended default. Disable this if you observe parse failures with a specific model or want to keep using the prompt-engineered protocol.”</em></p> </blockquote> <p>When the operator flips the switch, the client is reconstructed with native mode disabled, so <code>chatWithTools(...)</code> quietly returns a <code>ChatTurn.text(...)</code> and the legacy text path runs end-to-end — without any change to the agent code above it.</p> <h3 id="telemetry-that-finally-tells-the-truth" tabindex="-1"><a class="header-anchor" href="#telemetry-that-finally-tells-the-truth">Telemetry that finally tells the truth</a></h3> <p>Three metrics were added:</p> <table> <thead> <tr> <th>Metric</th> <th>Tags</th> <th>What it measures</th> </tr> </thead> <tbody> <tr> <td><code>agent.tool_call.mode_total</code></td> <td><code>mode={native,legacy}</code>, <code>provider</code></td> <td>One increment per AI round. Lets us see migration progress per provider.</td> </tr> <tr> <td><code>agent.tool_call.parse_failures_total</code></td> <td><code>provider</code></td> <td>Where the model defied the protocol.</td> </tr> <tr> <td><code>agent.tool_call.latency_seconds</code></td> <td><code>mode</code>, <code>provider</code></td> <td>Wall-clock per AI round.</td> </tr> </tbody> </table> <p>The first metric matters more than it looks. The single hardest question in this kind of migration is “are we <em>actually</em> on the new path, or are we silently falling back?” A simple count per mode answers that without instrumenting the loop further.</p> <h3 id="lesson-2" tabindex="-1"><a class="header-anchor" href="#lesson-2">Lesson</a></h3> <blockquote> <p>When you change a contract that several providers must implement, give the interface a default method that preserves the old behaviour. Then add the new behaviour as an opt-in capability, with telemetry from day one. Migrating providers becomes a measurable, reversible operation instead of a big bang.</p> </blockquote> <p>A related, second-order lesson: the schemas from Iteration 11 were exactly the right artefact to hand to the providers’ tool-call APIs. The investment in a stricter protocol paid off precisely when we no longer needed the loose one.</p> <h2 id="iteration-13-%E2%80%94-persisted-state%2C-consolidated-budgets%2C-and-an-optional-critic" tabindex="-1"><a class="header-anchor" href="#iteration-13-%E2%80%94-persisted-state%2C-consolidated-budgets%2C-and-an-optional-critic">Iteration 13 — Persisted State, Consolidated Budgets, and an Optional Critic</a></h2> <p>The final iteration is really three small refactors that share a theme: the agent had grown subtle bugs from <em>implicit</em> state. Each fix replaces an inference-from-history with an explicit, persisted value.</p> <h3 id="13a.-persisting-the-last-plan-instead-of-replaying-history" tabindex="-1"><a class="header-anchor" href="#13a.-persisting-the-last-plan-instead-of-replaying-history">13a. Persisting the last plan instead of replaying history</a></h3> <p>The old “get the last plan” helper walked the conversation backwards, calling the JSON parser on every <code>assistant</code> message until one returned non-null. It was used in three places — the PR body, the follow-up comment, and the critic step we were about to add. It had three problems:</p> <ul> <li><strong>Performance.</strong> Long sessions re-parsed dozens of messages per call.</li> <li><strong>Inconsistency.</strong> If the plan format evolved between runs, an older message would parse differently than a fresh one would.</li> <li><strong>Brittleness in tests.</strong> “The last plan” depended on the exact mock of the history accessor, leading to several hours of debugging “why does this test see an empty plan?”.</li> </ul> <p>The fix is unglamorous: three new columns on the session table (<code>last_plan_summary</code>, <code>last_plan_json</code>, <code>last_plan_at</code>), a <code>recordPlan(...)</code> method on the session service, and a single call in the strategy right after the response is parsed successfully. Downstream readers now hit the row in O(1). The history walk survives as a fallback for sessions that pre-date the migration.</p> <blockquote> <p><strong>Lesson:</strong> If you find yourself re-deriving the same fact from history three times in three places, persist the fact. The history is for <em>audit</em>, not for <em>lookup</em>.</p> </blockquote> <h3 id="13b.-one-budget-config-to-rule-them-all" tabindex="-1"><a class="header-anchor" href="#13b.-one-budget-config-to-rule-them-all">13b. One budget config to rule them all</a></h3> <p>Before this iteration, the agent had at least seven overlapping limits scattered across three places: separate validation retries, max tool executions, a hard-coded <code>MAX_CONTEXT_TOOL_REQUESTS = 5</code> constant, separate writer-specific knobs, a hard-coded <code>fileRequestRounds < 3</code> literal in the loop, and a legacy <code>maxTokens</code> setting. Each one was set somewhere different and meant something subtly different.</p> <p>The new <code>BudgetConfig</code> collapses these into five named knobs with sensible defaults: <code>maxRounds = 10</code>, <code>maxContextRounds = 3</code>, <code>maxValidationRetries = 3</code>, <code>maxContextToolRequestsPerRound = 5</code>, <code>maxTokensPerCall = 16384</code>. The deprecated fields stay on the config object for backwards compatibility, but a constructor hook copies them into the new struct whenever its values are still at the defaults — so an operator who customised <code>agent.max-tokens=8192</code> in their YAML keeps that value, without ever touching the new property.</p> <blockquote> <p><strong>Lesson:</strong> When the operator-visible configuration drifts from the model the code actually uses, add a migration <em>inside</em> the config class. A post-construct hook (in your framework’s equivalent) is the cheapest possible adapter and keeps the documentation honest.</p> </blockquote> <h3 id="13c.-an-optional-critic-%2F-reflection-step" tabindex="-1"><a class="header-anchor" href="#13c.-an-optional-critic-%2F-reflection-step">13c. An optional Critic / Reflection step</a></h3> <p>The last addition is intentionally small, and intentionally off by default. It is inspired by the “Self-Refine” / “Reflexion” family of prompting techniques: after the implementation has passed validation, ask a <em>second</em> LLM call to read the diff and answer one question — <em>does this change actually address the issue?</em></p> <p>The contract is a three-valued return:</p> <pre><code class="language-json">{"outcome": "APPROVE" | "ITERATE" | "ABORT", "feedback": "short, actionable text"} </code></pre> <ul> <li><code>APPROVE</code> proceeds to commit and PR creation, unchanged.</li> <li><code>ITERATE</code> posts the critic’s feedback back to the issue and resets the session to “waiting for the user to ping the bot”. (We deliberately did <em>not</em> feed the feedback back into the same loop. That door is open, but it has subtle failure modes — a critic that keeps demanding “more tests” can lock the loop in an unproductive cycle.)</li> <li><code>ABORT</code> aborts the PR creation entirely, posts the reason as a comment, and marks the session as failed.</li> </ul> <p>There is also a hidden fourth outcome, <code>SKIPPED</code>, emitted whenever the feature is disabled (the default). It exists purely so the metric <code>agent.critic.outcome_total{outcome=…}</code> makes the off-state visible — <em>zero</em> additional LLM calls per implementation, but a non-zero count of <code>SKIPPED</code> increments.</p> <pre><code class="language-yaml">agent: critic: enabled: false # opt-in max-iterations: 1 require-approval-for: [LARGE_DIFF] # informational, future trigger </code></pre> <p>The most important property of this design is what it <em>does not do</em> in the default configuration: it makes no extra AI call, allocates no extra tokens, and adds no latency. A unit test asserts exactly that — “when disabled, the AI client is never called”.</p> <h3 id="lesson-3" tabindex="-1"><a class="header-anchor" href="#lesson-3">Lesson</a></h3> <blockquote> <p>A useful self-critique step is one that costs nothing when disabled, that always fails open in the face of unparseable AI output, and that publishes its outcome as a first-class metric. Reflection is powerful, but a critic that blocks PRs when the network blips is worse than no critic at all.</p> </blockquote> <p>The pragmatic policy we settled on: parse failure → APPROVE. AI exception → APPROVE. Empty response → APPROVE. The only paths that can stop a PR are an <em>explicit</em> <code>ITERATE</code> or <code>ABORT</code> with a well-formed JSON body. The metric tags (<code>approve</code>, <code>iterate</code>, <code>abort</code>, <code>skipped</code>, <code>error</code>, <code>parse_error</code>) make every fail-open visible to operators.</p> <hr> <h2 id="reflecting-on-the-architectural-iterations" tabindex="-1"><a class="header-anchor" href="#reflecting-on-the-architectural-iterations">Reflecting on the Architectural Iterations</a></h2> <p>Iterations 10–13 are different in flavour from 1–9. The early iterations were about <em>what the agent does</em>; the later ones are about <em>the shape of the host that contains it</em>. Three patterns kept recurring:</p> <ol> <li><strong>Sealed types beat strings.</strong> Replacing string-based control flow (<code>if (response.contains("requestFiles"))</code>) with a sealed <code>StepDecision</code> and a handful of records gave the compiler enough visibility to catch every new branch. Native function calling’s <code>StopReason</code> enum did the same job for provider responses.</li> <li><strong>Default methods are the migration tool nobody talks about.</strong> Both Iteration 12’s <code>chatWithTools</code> rollout and Iteration 13b’s budget migration leaned on the same idiom: an interface with a default implementation, or a config object with a post-construct hook that bridges old fields to new. Both were strictly additive. Neither broke a single existing call site.</li> <li><strong>A metric is the entry ticket for a feature flag.</strong> Every flag we added — schema enforcement, native tool calling, the critic — landed together with a counter that distinguishes “the flag is on” from “the flag is doing work”. Without that, “we have schema validation now” is a press release. With it, it’s an operations capability.</li> </ol> <h2 id="summary-of-key-takeaways" tabindex="-1"><a class="header-anchor" href="#summary-of-key-takeaways">Summary of Key Takeaways</a></h2> <p>After thirteen iterations — nine on the agent’s behaviour, four on the architecture around it — these are the principles I’d carry into any agentic system:</p> <table> <thead> <tr> <th>Principle</th> <th>Detail</th> </tr> </thead> <tbody> <tr> <td><strong>Inversion of control</strong></td> <td>The agent drives the logic; your app is the execution environment.</td> </tr> <tr> <td><strong>Choose the split wisely</strong></td> <td>Keep deterministic steps in the app; delegate reasoning to the LLM. Each side has clear trade-offs.</td> </tr> <tr> <td><strong>Context is everything</strong></td> <td>Invest heavily in smart, dynamic context gathering.</td> </tr> <tr> <td><strong>Cap your iteration loops</strong></td> <td>LLMs improve with feedback, but after 3 failed attempts, change your strategy — don’t just retry.</td> </tr> <tr> <td><strong>Let the agent ask for more</strong></td> <td>Never assume you’ve provided enough information upfront.</td> </tr> <tr> <td><strong>Define a protocol, but be resilient</strong></td> <td>Structured output formats are essential, but the AI will violate them. Build robust parsers.</td> </tr> <tr> <td><strong>Minimise token waste</strong></td> <td>Use summaries, targeted context, and compact tool requests to keep the context window focused.</td> </tr> <tr> <td><strong>Delegate validation to the agent</strong></td> <td>The AI knows the build system better than a hardcoded checker.</td> </tr> <tr> <td><strong>Minimise the toolset</strong></td> <td>Expose only the tools strictly needed. Every additional tool is a security risk and a source of complexity.</td> </tr> <tr> <td><strong>Prefer tool requests over patch protocols</strong></td> <td>If diff parsing keeps getting more complex, simplify the contract and let the host execute explicit file operations.</td> </tr> <tr> <td><strong>Use the AI to fix AI failures</strong></td> <td>When diff application or parsing fails, ask the AI to resolve it.</td> </tr> <tr> <td><strong>Don’t refactor what you can’t reproduce</strong></td> <td>Characterisation tests before invasive loop changes. Sealed <code>StepDecision</code> types beat string-based control flow.</td> </tr> <tr> <td><strong>Add the strict layer next to the lenient one</strong></td> <td>Run schema validation in parallel with the repair heuristics first. Flip enforcement only after measuring violation rates in production.</td> </tr> <tr> <td><strong>Migrate via default methods, not big bangs</strong></td> <td>Interface defaults (<code>chatWithTools</code> delegating to <code>chat</code>) and <code>@PostConstruct</code> config bridges let you ship the new contract without breaking the old one.</td> </tr> <tr> <td><strong>A flag without a metric is a press release</strong></td> <td>Every feature toggle ships with a Prometheus counter that distinguishes “on” from “doing work”.</td> </tr> <tr> <td><strong>Fail open on optional critique</strong></td> <td>A self-critic that blocks PRs on a parse error or network blip is worse than no critic. APPROVE on uncertainty; only block on explicit, well-formed <code>ITERATE</code> / <code>ABORT</code>.</td> </tr> </tbody> </table> <p>Building agentic systems is an exercise in designing for uncertainty. The AI is powerful but imprecise. Your surrounding infrastructure must be resilient, adaptive, and willing to hand control back to the agent when deterministic approaches fail. The result is a system that is more capable than either part alone.</p> Tom Seidel https://remus-software.org/articles/rest-explorer-1-0-released/ 2026-04-04T00:00:00.000Z 2026-04-04T00:00:00.000Z

Restic Explorer 1.0 is out — a lightweight, self-hosted web dashboard that monitors all restic backup repositories across S3, Azure, SFTP, REST, and Rclone from a single UI with automated scans, integrity checks, and retention policy tracking.

<p><strong>Backups are only as good as the confidence that they actually work.</strong> Restic Explorer 1.0 is now available — a focused, self-hosted web dashboard that provides exactly that confidence for all <a href="https://restic.net/">restic</a> repositories in one place.</p> <p><img src="https://raw.githubusercontent.com/tmseidel/restic-explorer/main/docs/screenshot_dashboard.png" alt="Restic Explorer Dashboard"></p> <h2 id="the-problem" tabindex="-1"><a class="header-anchor" href="#the-problem">The Problem</a></h2> <p>Restic is an outstanding backup tool. Fast, encrypted, deduplicated — it has become the go-to choice for backing up servers, NAS devices, and cloud workloads. But restic is a CLI tool by design. When running multiple repositories across different backends — S3 buckets, Azure Blob, SFTP servers — keeping track of <em>“is everything still running?”</em> becomes a chore. It often means writing shell scripts, parsing JSON output, wiring up cron jobs, and hoping someone notices when something breaks.</p> <p>Existing monitoring solutions are excellent pieces of software, but they tend to come with far more complexity than many use cases require: agent-based architectures, extensive plugin systems, or dashboards designed for hundreds of repositories across large teams. For operators who simply need a single pane of glass that answers <strong>are the backups running, are they healthy, and do they meet retention requirements?</strong> — a lighter approach is needed.</p> <h2 id="the-solution" tabindex="-1"><a class="header-anchor" href="#the-solution">The Solution</a></h2> <p>Restic Explorer is that single pane of glass. It connects directly to restic repositories — wherever they live — and provides:</p> <ul> <li><strong>Multi-Repository Dashboard</strong> — status of all repos at a glance with color-coded badges (green/red/amber)</li> <li><strong>Automated Scanning</strong> — scheduled <code>restic snapshots</code> calls cache metadata for fast browsing without CLI round-trips</li> <li><strong>Integrity Checks</strong> — scheduled <code>restic check --read-data</code> runs with configurable intervals per repository</li> <li><strong>Retention Policy Monitoring</strong> — daily/weekly/monthly/yearly rules with soft warnings when snapshots fall short</li> <li><strong>Health Endpoint</strong> — <code>/actuator/health</code> JSON endpoint reporting per-repo status, ready for Uptime Kuma, Prometheus, or any HTTP health checker</li> <li><strong>Snapshot Browser</strong> — paginated, sortable snapshot list with a dedicated detail page showing paths, tags, hostname, and size</li> <li><strong>Lock Detection</strong> — automatic stale lock detection with one-click unlock</li> <li><strong>Encrypted Credentials</strong> — AES-256-GCM encryption at rest for repository passwords and backend keys</li> </ul> <h3 id="five-backends%2C-one-ui" tabindex="-1"><a class="header-anchor" href="#five-backends%2C-one-ui">Five Backends, One UI</a></h3> <table> <thead> <tr> <th>Backend</th> <th>What it covers</th> </tr> </thead> <tbody> <tr> <td><strong>S3 / S3-Compatible</strong></td> <td>AWS S3, MinIO, Wasabi, Backblaze B2 (S3 API)</td> </tr> <tr> <td><strong>Azure Blob Storage</strong></td> <td>Native Azure integration</td> </tr> <tr> <td><strong>SFTP</strong></td> <td>Any SSH-accessible server, key-based auth</td> </tr> <tr> <td><strong>REST Server</strong></td> <td>Restic’s own REST backend with optional HTTP auth</td> </tr> <tr> <td><strong>Rclone</strong></td> <td>Google Drive, Dropbox, OneDrive, B2, and 40+ more via rclone</td> </tr> </tbody> </table> <h2 id="getting-started-in-60-seconds" tabindex="-1"><a class="header-anchor" href="#getting-started-in-60-seconds">Getting Started in 60 Seconds</a></h2> <p>The fastest way to get running is Docker Compose:</p> <pre><code class="language-yaml">services: app: image: tmseidel/restic-explorer:latest ports: - "8080:8080" environment: SPRING_PROFILES_ACTIVE: docker DB_HOST: db DB_PORT: 5432 DB_NAME: resticexplorer DB_USER: resticexplorer DB_PASSWORD: resticexplorer depends_on: db: condition: service_healthy restart: unless-stopped db: image: postgres:16-alpine environment: POSTGRES_DB: resticexplorer POSTGRES_USER: resticexplorer POSTGRES_PASSWORD: resticexplorer volumes: - db-data:/var/lib/postgresql/data healthcheck: test: ["CMD-SHELL", "pg_isready -U resticexplorer"] interval: 10s timeout: 5s retries: 5 restart: unless-stopped volumes: db-data: </code></pre> <pre><code class="language-bash">docker compose up -d </code></pre> <p>Open <code>http://localhost:8080</code>, create the admin account, and start adding repositories. That’s it.</p> <p>The image ships with restic, rclone, and openssh-client pre-installed — no additional setup required for any backend type.</p> <h2 id="why-restic-is-a-great-fit-for-cloud-%26-infrastructure-as-code" tabindex="-1"><a class="header-anchor" href="#why-restic-is-a-great-fit-for-cloud-%26-infrastructure-as-code">Why Restic is a Great Fit for Cloud & Infrastructure-as-Code</a></h2> <p>For teams managing cloud infrastructure through Terraform, Ansible, Pulumi, or similar tools, restic fits naturally into the workflow:</p> <h3 id="stateless-by-design" tabindex="-1"><a class="header-anchor" href="#stateless-by-design">Stateless by Design</a></h3> <p>Restic repositories are self-contained. There is no central server, no daemon, no database to maintain. A repository is just a structured set of encrypted blobs in any storage backend. This makes restic trivially reproducible — IaC can provision the storage bucket and the backup job in the same run.</p> <h3 id="backend-agnostic" tabindex="-1"><a class="header-anchor" href="#backend-agnostic">Backend Agnostic</a></h3> <p>Moving from AWS to Azure? Migrating from on-prem to cloud? Restic’s backend abstraction means the backup strategy isn’t tied to a vendor. A Terraform module provisions an S3 bucket today; tomorrow it provisions Azure Blob Storage. The restic commands stay the same.</p> <h3 id="encryption-without-infrastructure" tabindex="-1"><a class="header-anchor" href="#encryption-without-infrastructure">Encryption Without Infrastructure</a></h3> <p>Restic encrypts everything client-side. There is no need for a KMS, a Vault instance, or an HSM for backup encryption. One password, stored in the secrets manager of choice, and data is encrypted at rest regardless of the storage backend’s capabilities.</p> <h3 id="deduplication-saves-cloud-storage-costs" tabindex="-1"><a class="header-anchor" href="#deduplication-saves-cloud-storage-costs">Deduplication Saves Cloud Storage Costs</a></h3> <p>Restic’s content-defined chunking and deduplication means incremental backups are genuinely incremental — even across different source machines backing up to the same repository. In cloud environments where storage is metered, this translates directly to lower costs.</p> <h3 id="scriptable-and-composable" tabindex="-1"><a class="header-anchor" href="#scriptable-and-composable">Scriptable and Composable</a></h3> <p>Restic is a CLI tool that outputs JSON. It composes perfectly with cron, systemd timers, CI/CD pipelines, and container sidecars. No agents to install, no ports to open, no protocols to configure — just a binary and a repository URL.</p> <p>Restic Explorer adds the monitoring layer on top: existing restic workflows remain untouched, and Restic Explorer watches the repositories and surfaces issues when they need attention.</p> <h2 id="what%E2%80%99s-in-1.0" tabindex="-1"><a class="header-anchor" href="#what%E2%80%99s-in-1.0">What’s in 1.0</a></h2> <p>This release marks the point where the feature set is stable, tested, and production-ready:</p> <ul> <li><strong>Five backend types</strong> — S3, Azure, SFTP, REST, Rclone</li> <li><strong>Repository groups</strong> — organize repos by team, environment, or purpose</li> <li><strong>Configurable scan and check intervals</strong> per repository</li> <li><strong>Retention policy monitoring</strong> with violation warnings</li> <li><strong>Error log</strong> with date filtering and auto-cleanup</li> <li><strong>Dark mode</strong> with automatic theme detection</li> <li><strong>Health & info endpoints</strong> for external monitoring integration</li> <li><strong>Admin-only download</strong> of snapshots as <code>.tar</code> archives</li> <li><strong>Encrypted credential storage</strong> (AES-256-GCM)</li> <li><strong>Docker image</strong> running as non-root user with built-in healthcheck</li> </ul> <table> <thead> <tr> <th>Snapshots</th> <th>Snapshot Detail</th> </tr> </thead> <tbody> <tr> <td><img src="https://raw.githubusercontent.com/tmseidel/restic-explorer/main/docs/screenshot_snapshots.png" alt="Snapshots"></td> <td><img src="https://raw.githubusercontent.com/tmseidel/restic-explorer/main/docs/screenshot_snapshot.png" alt="Detail"></td> </tr> </tbody> </table> <h2 id="get-it" tabindex="-1"><a class="header-anchor" href="#get-it">Get It</a></h2> <ul> <li><strong>Docker Hub</strong>: <a href="https://hub.docker.com/r/tmseidel/restic-explorer"><code>tmseidel/restic-explorer:latest</code></a></li> <li><strong>GitHub</strong>: <a href="https://github.com/tmseidel/restic-explorer">tmseidel/restic-explorer</a></li> <li><strong>Documentation</strong>: <a href="https://github.com/tmseidel/restic-explorer/blob/main/docs/USER_GUIDE.md">User Guide</a> · <a href="https://github.com/tmseidel/restic-explorer/blob/main/docs/CONFIGURATION.md">Configuration</a> · <a href="https://github.com/tmseidel/restic-explorer/blob/main/docs/ARCHITECTURE.md">Architecture</a></li> </ul> <p>Licensed under MIT. Contributions, issues, and feedback welcome.</p> <hr> <p><em>Restic Explorer is built with Spring Boot 4, Thymeleaf, and Bootstrap 5. It runs as a single container alongside PostgreSQL and requires no additional infrastructure beyond what is already in place.</em></p> Tom Seidel https://remus-software.org/articles/replacing-veeam-with-restic/ 2026-03-29T00:00:00.000Z 2026-03-29T00:00:00.000Z

How we replaced a costly, complex backup system with a simple shell script and S3 storage — and the key questions to ask before you do the same.

<h1 id="" tabindex="-1"><a class="header-anchor" href="#"></a></h1> <p><strong>We ditched our expensive, bloated backup platform for a shell script and S3. Here’s how — and what to think about before you do the same.</strong></p> <hr> <h2 id="the-problem-nobody-wants-to-touch" tabindex="-1"><a class="header-anchor" href="#the-problem-nobody-wants-to-touch">The Problem Nobody Wants to Touch</a></h2> <p>Let’s be honest: most backup systems are set up once and then nobody looks at them again. They just… run. Hopefully.</p> <p>We were in that exact spot. A centralized commercial backup server on Windows, proprietary agents on every machine, enterprise licenses, the whole deal. It worked — until it didn’t:</p> <ul> <li><strong>The config kept breaking.</strong> More than once, the backup server’s internal state got corrupted. Trying to add a new backup job? Error dialog. Can’t configure anything until someone fixes it manually.</li> <li><strong>Way too much overhead.</strong> Each server needed a proprietary agent, a service user, SSH access, firewall rules — all for what’s basically “copy some files somewhere safe.”</li> <li><strong>We used 5% of the features.</strong> Bare-metal recovery? Granular restore? Application-aware snapshots? We never used any of that. Our servers are provisioned with automation — we can rebuild them from scratch. We just needed the <em>data</em>.</li> <li><strong>It cost real money.</strong> A Windows Server with commercial licenses, just to store backups. For a team that runs Linux everywhere else, that’s an expensive oddball.</li> </ul> <hr> <h2 id="before-you-migrate%3A-ask-yourself-these-questions" tabindex="-1"><a class="header-anchor" href="#before-you-migrate%3A-ask-yourself-these-questions">Before You Migrate: Ask Yourself These Questions</a></h2> <p>Don’t jump to a new tool just because the old one annoys you. Think it through first:</p> <ul> <li><strong>What are you actually backing up?</strong> If your servers can be rebuilt from code, you probably just need data-level backups (database dumps, config files), not full disk images.</li> <li><strong>Have you ever restored from backup?</strong> If the answer is “uh, I think so?” — that’s your real problem, regardless of the tool.</li> <li><strong>What’s the total cost?</strong> Licenses + the server it runs on + agent maintenance + engineer time spent debugging weird issues.</li> <li><strong>Do you get alerts when a backup fails?</strong> A backup that silently breaks is worse than no backup at all.</li> <li><strong>Is backup part of your provisioning?</strong> If setting up backup for a new server is a separate manual process, it <em>will</em> get skipped eventually.</li> </ul> <hr> <h2 id="what-we-switched-to" tabindex="-1"><a class="header-anchor" href="#what-we-switched-to">What We Switched To</a></h2> <p>We landed on <a href="https://restic.net/">Restic</a> — open-source, encrypts everything, deduplicates, compresses, and stores to any S3-compatible backend. It’s in the default Debian repos. Install is literally <code>apt install restic</code>.</p> <table> <thead> <tr> <th></th> <th>Old System</th> <th>Restic</th> </tr> </thead> <tbody> <tr> <td><strong>Install</strong></td> <td>Proprietary repo + agent + service user + firewall rules</td> <td><code>apt install restic</code></td> </tr> <tr> <td><strong>Storage</strong></td> <td>Dedicated Windows backup server</td> <td>Any S3-compatible object storage</td> </tr> <tr> <td><strong>Config</strong></td> <td>GUI on backup server</td> <td>Environment variables + shell script</td> </tr> <tr> <td><strong>Licensing</strong></td> <td>Per-server commercial license</td> <td>Free</td> </tr> <tr> <td><strong>Restore</strong></td> <td>Through backup server UI</td> <td><code>restic restore</code> from anywhere</td> </tr> </tbody> </table> <p>When picking any replacement tool, look for: simple deployment, storage flexibility (don’t get locked in), full CLI scriptability, client-side encryption, active community, and built-in retention management.</p> <hr> <h2 id="the-architecture" tabindex="-1"><a class="header-anchor" href="#the-architecture">The Architecture</a></h2> <p>Here’s what we ended up with — three layers:</p> <div class="mermaid">graph TB subgraph servers["Servers"] native["<b>Native App</b><br/>pg_dumpall → gzip<br/>→ restic backup"] docker["<b>Docker App</b><br/>docker exec → pg_dump<br/>→ gzip → restic backup"] legacy["<b>Legacy App</b><br/>mysqldump<br/>→ legacy agent"] end subgraph storage["Storage Layer"] s3["<b>S3-Compatible Object Store</b><br/>One bucket per project"] legacysrv["<b>Legacy Backup Server</b>"] end subgraph monitoring["Monitoring Layer"] explorer["<b>Backup Explorer</b><br/>Browse repos,<br/>check health"] heartbeat["<b>Heartbeat Monitor</b><br/>Push-based alerts on<br/>success / failure"] end native -- "Restic + S3" --> s3 docker -- "Restic + S3" --> s3 legacy -- "Legacy Agent" --> legacysrv s3 --> explorer s3 --> heartbeat </div><p>A few rules we learned the hard way:</p> <ul> <li><strong>One bucket per project.</strong> Never mix backups from different apps in the same bucket. Isolation, access control, cost tracking — all easier this way.</li> <li><strong>Every backup is individual.</strong> A Postgres DB needs <code>pg_dumpall</code>. A Docker service needs <code>docker compose exec</code>. A VPN server needs its config files. There’s no universal “back up everything” script. Write one per app.</li> <li><strong>Credentials go in a team vault.</strong> If the person who set up the backup leaves, you don’t want the passwords leaving with them.</li> </ul> <hr> <h2 id="the-script-pattern" tabindex="-1"><a class="header-anchor" href="#the-script-pattern">The Script Pattern</a></h2> <p>After iterating across a bunch of projects, we settled on a template every backup job follows:</p> <pre><code class="language-bash">#!/usr/bin/env bash set -euo pipefail source /opt/app/.restic-env # Error trap — always notify on failure trap 'notify_monitor "down" "Backup failed"; rm -f "${DUMP_FILE}"; exit 1' ERR # Init repo if first run restic snapshots > /dev/null 2>&1 || restic init # Create the dump (customize this per app) pg_dumpall | gzip > "${DUMP_FILE}" # Don't upload empty dumps [[ -s "${DUMP_FILE}" ]] || { notify_monitor "down" "Empty dump"; exit 1; } # Upload, clean up, prune old snapshots restic backup "${DUMP_FILE}" --tag app-name rm -f "${DUMP_FILE}" restic forget --keep-daily 30 --keep-weekly 8 --keep-monthly 12 --prune # All good notify_monitor "up" "OK" </code></pre> <p>The important bits: the <strong>error trap</strong> makes sure you hear about failures. The <strong>empty-dump check</strong> catches silent breakage (like a database dump that exits 0 but produces nothing). <strong>Retention runs on every backup</strong>, not as a separate task. And <strong>tags</strong> let you filter snapshots later.</p> <p>With default retention (30 daily, 8 weekly, 12 monthly) you end up with about 44 snapshots at any given time — good granularity without blowing up storage.</p> <hr> <h2 id="monitoring%3A-don%E2%80%99t-skip-this" tabindex="-1"><a class="header-anchor" href="#monitoring%3A-don%E2%80%99t-skip-this">Monitoring: Don’t Skip This</a></h2> <p>Two layers — you need both:</p> <p><strong>Heartbeat monitoring:</strong> Every backup script pings a monitor on success or failure (we use <a href="https://github.com/louislam/uptime-kuma">Uptime Kuma</a>, but anything push-based works). If no ping arrives within 26 hours → alert. This catches script failures, cron being broken, and servers being down.</p> <pre><code class="language-bash">curl -sf "${MONITOR_URL}?status=up&msg=OK" # on success curl -sf "${MONITOR_URL}?status=down&msg=Failed" # in error trap </code></pre> <p><strong>Repository browser:</strong> A heartbeat tells you <em>if</em> the backup ran. A browser tells you <em>what’s in it</em> — snapshot counts, sizes, retention compliance, integrity checks. This catches things like backups that “succeed” but are suspiciously small.</p> <hr> <h2 id="how-to-actually-migrate" tabindex="-1"><a class="header-anchor" href="#how-to-actually-migrate">How to Actually Migrate</a></h2> <p>Don’t flip the switch overnight. We did it in phases:</p> <ol> <li><strong>New servers get the new tool from day one.</strong> Zero risk, no migration needed.</li> <li><strong>Old servers run both systems in parallel.</strong> Set up the new backup alongside the legacy one.</li> <li><strong>Test restores from the new backup.</strong> Actually restore on a test environment. Verify the data.</li> <li><strong>Remove the legacy agent per server</strong> after the new backup has been solid for a couple of months.</li> <li><strong>Kill the legacy server last</strong> — only after every server is migrated and validated.</li> </ol> <p>Don’t rush step 4. Storage is cheap. Lost data is not.</p> <hr> <h2 id="tl%3Bdr" tabindex="-1"><a class="header-anchor" href="#tl%3Bdr">TL;DR</a></h2> <ul> <li>If your servers are provisioned from code, you don’t need image-level backups. Just back up the data.</li> <li>Write a backup script per application — there is no one-size-fits-all.</li> <li>Monitor everything. Heartbeats for “did it run?”, a browser for “what’s in it?”</li> <li>Bake backup into your provisioning. If it’s manual, it’ll get skipped.</li> <li>Test your restores. A backup you’ve never restored from is a hope, not a strategy.</li> <li>Migrate gradually. Parallel-run, validate, then decommission.</li> </ul> <p>A shell script, a cron job, encrypted uploads to S3, and a heartbeat ping. That’s the whole system. No servers, no GUI, no licenses.</p> <hr> <p><em>The best backup system is the one your team actually understands, maintains, and tests.</em></p> Tom Seidel https://remus-software.org/articles/self-hosted-ai-translation-service/ 2026-02-02T00:00:00.000Z 2026-02-02T00:00:00.000Z

A practical evaluation of replacing DeepL with a self-hosted translation service using open-source LLMs — comparing quality, performance, and cost.

<p>With freely available large language models now widely accessible, it has become straightforward to self-host software that was previously only available through commercial providers. The key question always comes down to the resulting costs and the effort involved.</p> <p>In this case study, I examined whether the translation service DeepL can be replaced by a self-hosted solution. The goal was to provide a DeepL-compatible REST API that:</p> <ul> <li>achieves comparable translation quality,</li> <li>offers similar performance, and</li> <li>implements the same REST API specification<sup class="footnote-ref"><a href="#fn1" id="fnref1">[1]</a></sup>,</li> </ul> <p>in order to then compare the one-time and ongoing costs. Using the DeepL API requires a paid subscription; while the pay-as-you-go model is transparent, it can become very expensive with heavy usage. Additionally, data leaves the corporate network, and the API’s behaviour under heavy load is not fully transparent.</p> <h2 id="choosing-a-suitable-local-model" tabindex="-1"><a class="header-anchor" href="#choosing-a-suitable-local-model">Choosing a Suitable Local Model</a></h2> <p>The first question is which freely available models are suitable for translation tasks. Hugging Face offers a large selection of models that can be easily integrated into custom software<sup class="footnote-ref"><a href="#fn2" id="fnref2">[2]</a></sup>. For this evaluation, Meta’s <strong>nllb-200-distilled</strong> model was chosen, as it is widely used, easy to deploy, and available in three sizes (600M, 1.3B, and 3.3B parameters).</p> <h2 id="implementing-the-deepl-compatible-rest-api" tabindex="-1"><a class="header-anchor" href="#implementing-the-deepl-compatible-rest-api">Implementing the DeepL-Compatible REST API</a></h2> <p>A pragmatic approach was taken for the implementation: a Spring Boot application serves as the API frontend and delegates the actual translation request to a Python Flask component that controls the LLM.</p> <p>For easy deployment, the system can be run either:</p> <ul> <li>in Docker containers, or</li> <li>natively on a Debian/Ubuntu server.</li> </ul> <p>The goal was a straightforward deployment on various cloud hardware platforms to test quality and performance there. The complete implementation is available on GitHub<sup class="footnote-ref"><a href="#fn3" id="fnref3">[3]</a></sup>. Ansible was used for automated native deployment.</p> <h2 id="test-%E2%80%94-translation-quality" tabindex="-1"><a class="header-anchor" href="#test-%E2%80%94-translation-quality">Test — Translation Quality</a></h2> <p>The following German reference sentence was used to evaluate translation quality:</p> <blockquote> <p><em>“Sobald der Glasfaser-Ausbau abgeschlossen ist, erhalten Sie eine Mitteilung zum Schaltungstermin und eine Schnell-Start-Anleitung für die Einrichtung des Glasfaser-Anschlusses.”</em></p> </blockquote> <p>DeepL produces the following translation:</p> <blockquote> <p>“Once the fiber optic expansion is complete, you will receive a notification of the activation date and a quick start guide for setting up your fiber optic connection.”</p> </blockquote> <p>This translation serves as the reference.</p> <h3 id="test-with-nllb-200-distilled-600m" tabindex="-1"><a class="header-anchor" href="#test-with-nllb-200-distilled-600m">Test with nllb-200-distilled-600M</a></h3> <p>The smallest model was first run on a development machine via Docker. Performance was not a concern at this stage. The generated translation was:</p> <blockquote> <p>“Once the glass-faser-Ausbau is closed, you receive a Mitteilung zum Schaltungstermin und eine Schnell-Start-Anleitung für die Einrichtung der Glasfaser-Anschlusses.”</p> </blockquote> <p><img src="nllb-200-distilled-600M.png" alt="Response of the small model"></p> <h3 id="test-with-nllb-200-distilled-1.3b" tabindex="-1"><a class="header-anchor" href="#test-with-nllb-200-distilled-1.3b">Test with nllb-200-distilled-1.3B</a></h3> <p>The medium model produced the following output:</p> <blockquote> <p>“The Commission shall inform the Member States of the date of the entry into force of this Regulation.”</p> </blockquote> <h3 id="test-with-nllb-200-distilled-3.3b" tabindex="-1"><a class="header-anchor" href="#test-with-nllb-200-distilled-3.3b">Test with nllb-200-distilled-3.3B</a></h3> <p>The largest model generated the following translation:</p> <blockquote> <p>“Once the glass fibre installation is completed, you will receive a notice on the date of installation and a quick start guide for the installation of the glass fibre connections.”</p> </blockquote> <p><img src="nllb-200-distilled-3.3B.png" alt="Response of the large model"></p> <h3 id="translation-quality-conclusion" tabindex="-1"><a class="header-anchor" href="#translation-quality-conclusion">Translation Quality Conclusion</a></h3> <p>A comprehensive assessment is difficult after just a few tests. Nevertheless, it became clear that only the largest model is viable for production use. It was also notable that the models performed significantly more reliably when the source language was English. If translation is exclusively from English, the medium model might therefore be sufficient.</p> <h2 id="test-%E2%80%94-performance" tabindex="-1"><a class="header-anchor" href="#test-%E2%80%94-performance">Test — Performance</a></h2> <p>Once the suitable model was identified, the next step was to determine under which hardware conditions productive operation is feasible. As a benchmark, it was assumed that translating the reference sentence should take no longer than two seconds. Additionally, the difference between a traditional CPU-based server and a GPU-based system was to be determined.</p> <h3 id="test%3A-traditional-server" tabindex="-1"><a class="header-anchor" href="#test%3A-traditional-server">Test: Traditional Server</a></h3> <p>A Hetzner CX53 with 16 vCPUs and 32 GB RAM was used as the CPU server (cost: €17 per month).</p> <p><strong>Response time: 12.93 seconds</strong></p> <h3 id="test%3A-gpu-server" tabindex="-1"><a class="header-anchor" href="#test%3A-gpu-server">Test: GPU Server</a></h3> <p>An Amazon g4dn.large with 16 GB GPU RAM (Nvidia) was used as the GPU server. The cost is €0.67 per hour, roughly €500 per month.</p> <p><strong>Response time: 1.31 seconds</strong> — GPU memory usage: approx. 13 GB</p> <h3 id="performance-conclusion" tabindex="-1"><a class="header-anchor" href="#performance-conclusion">Performance Conclusion</a></h3> <p>The difference between the two systems was significantly larger than expected. Even without deep knowledge of the internal workings of LLMs, it is clear that productive operation is practically only feasible with GPU-based hardware. Costs on AWS are currently high, but cheaper alternatives exist — for example at Hetzner<sup class="footnote-ref"><a href="#fn4" id="fnref4">[4]</a></sup>. The achieved response time is fundamentally suitable for production use. Parallel requests had no significant impact on latency in the tests.</p> <p><img src="nvidia-smi.png" alt="nvidia-smi output showing GPU memory usage"></p> <h2 id="overall-conclusion" tabindex="-1"><a class="header-anchor" href="#overall-conclusion">Overall Conclusion</a></h2> <p>This evaluation clearly demonstrates that it is possible to self-host AI-based services like machine translation using freely available models and modern hardware — with reasonable effort and competitive quality. While the ongoing costs for GPU-based systems are still relatively high, falling prices and increasing efficiency can be expected as adoption grows and technology advances. Moreover, more affordable hosting alternatives beyond the major cloud providers already exist today.</p> <p>Especially in heavily regulated industries — such as finance, healthcare, or the public sector — a self-hosted AI service can offer significant advantages:</p> <ul> <li><strong>Data sovereignty</strong> is fully preserved, as no sensitive information leaves external systems.</li> <li><strong>Compliance requirements</strong> are easier to meet, since infrastructure and data flows are fully controllable.</li> <li><strong>Performance and scalability</strong> can be precisely tailored to your own needs.</li> <li><strong>Competitive advantages</strong> emerge when you can offer services that are not only cheaper but also more secure and flexible than commercial alternatives.</li> </ul> <h2 id="references" tabindex="-1"><a class="header-anchor" href="#references">References</a></h2> <hr class="footnotes-sep"> <section class="footnotes"> <ol class="footnotes-list"> <li id="fn1" class="footnote-item"><p><a href="https://developers.deepl.com/docs/getting-started/intro">DeepL API Documentation</a> <a href="#fnref1" class="footnote-backref">↩︎</a></p> </li> <li id="fn2" class="footnote-item"><p><a href="https://huggingface.co/models?pipeline_tag=translation">Hugging Face Translation Models</a> <a href="#fnref2" class="footnote-backref">↩︎</a></p> </li> <li id="fn3" class="footnote-item"><p><a href="https://github.com/tmseidel/simple_ai_translation_service">simple_ai_translation_service on GitHub</a> <a href="#fnref3" class="footnote-backref">↩︎</a></p> </li> <li id="fn4" class="footnote-item"><p><a href="https://www.hetzner.com/dedicated-rootserver/matrix-gpu/">Hetzner GPU Dedicated Servers</a> <a href="#fnref4" class="footnote-backref">↩︎</a></p> </li> </ol> </section> Tom Seidel https://remus-software.org/articles/monolith-to-microservices/ 2024-03-15T00:00:00.000Z 2024-03-15T00:00:00.000Z

A hands-on walkthrough of the architectural decisions and patterns I use when migrating Java monoliths to cloud-native microservices.

<p>Migrating a monolithic Java application to microservices is one of the most impactful — and challenging — transformations you can undertake. This article shares the practical approach I’ve refined over multiple engagements.</p> <h2 id="why-migrate%3F" tabindex="-1"><a class="header-anchor" href="#why-migrate%3F">Why Migrate?</a></h2> <p>Before touching a single line of code, ask: <em>why are we doing this?</em> The most common drivers I encounter are:</p> <ul> <li><strong>Deployment bottlenecks</strong>: A single deployable artifact blocks independent team delivery.</li> <li><strong>Scalability constraints</strong>: You need to scale a specific module, not the entire application.</li> <li><strong>Technology modernisation</strong>: Teams want to adopt newer frameworks or languages for specific domains.</li> <li><strong>Organisational growth</strong>: Conway’s Law — architecture tends to mirror team structure.</li> </ul> <blockquote> <p>“Never migrate for migration’s sake. Identify the concrete pain point and validate that microservices solve it.”</p> </blockquote> <h2 id="the-strangler-fig-pattern" tabindex="-1"><a class="header-anchor" href="#the-strangler-fig-pattern">The Strangler Fig Pattern</a></h2> <p>My go-to approach is the <strong>Strangler Fig Pattern</strong>: incrementally replace monolith functionality behind a facade, leaving the monolith running until it’s fully strangled.</p> <div class="mermaid">graph LR Client -->|All traffic| Facade[API Gateway / Facade] Facade -->|Legacy routes| Monolith[(Monolith)] Facade -->|New routes| SvcA[User Service] Facade -->|New routes| SvcB[Order Service] Monolith -.->|Shared DB - phase 1| DB[(Database)] SvcA -->|Own DB - phase 2| DBA[(Users DB)] SvcB -->|Own DB - phase 2| DBB[(Orders DB)] </div><p>This lets you:</p> <ol> <li>Ship value incrementally</li> <li>Reduce risk by keeping the fallback running</li> <li>Validate each new service before extracting the next</li> </ol> <h2 id="identifying-service-boundaries" tabindex="-1"><a class="header-anchor" href="#identifying-service-boundaries">Identifying Service Boundaries</a></h2> <p>Domain-Driven Design (DDD) gives us the best tools for finding service boundaries. I use <strong>Event Storming</strong> workshops to:</p> <ol> <li>Map all domain events with the business team</li> <li>Identify <strong>bounded contexts</strong> — areas with consistent language and ownership</li> <li>Use bounded contexts as candidate service boundaries</li> </ol> <div class="mermaid">graph TD subgraph "Order Context" OE1[OrderPlaced] OE2[OrderConfirmed] OE3[OrderShipped] end subgraph "Inventory Context" IE1[StockReserved] IE2[StockReleased] end subgraph "Notification Context" NE1[EmailSent] NE2[SMSSent] end OE2 --> IE1 OE3 --> NE1 IE2 --> NE2 </div><h2 id="practical-steps" tabindex="-1"><a class="header-anchor" href="#practical-steps">Practical Steps</a></h2> <h3 id="1.-start-with-the-api-layer" tabindex="-1"><a class="header-anchor" href="#1.-start-with-the-api-layer">1. Start with the API Layer</a></h3> <p>Deploy an <strong>API Gateway</strong> (AWS API Gateway, Kong, or a simple Spring Cloud Gateway) in front of the monolith. This gives you:</p> <ul> <li>A single entry point for traffic</li> <li>The ability to route selectively to new services</li> <li>A foundation for cross-cutting concerns (auth, rate limiting, logging)</li> </ul> <h3 id="2.-extract-stateless-services-first" tabindex="-1"><a class="header-anchor" href="#2.-extract-stateless-services-first">2. Extract Stateless Services First</a></h3> <p>Pick a bounded context that:</p> <ul> <li>Has clear, stable APIs</li> <li>Is relatively self-contained</li> <li>Has low coupling to the rest of the monolith</li> </ul> <p>Notification services, reporting modules, and authentication are often good first targets.</p> <h3 id="3.-database-decomposition" tabindex="-1"><a class="header-anchor" href="#3.-database-decomposition">3. Database Decomposition</a></h3> <p>The hardest part. Never share a database between the monolith and a new service in the long run. The interim approach:</p> <div class="mermaid">sequenceDiagram participant New Service participant Monolith participant Shared DB participant New DB Note over New Service, Shared DB: Phase 1 – Dual Write New Service->>Shared DB: Write (compatibility) New Service->>New DB: Write (new schema) Monolith->>Shared DB: Read/Write Note over New Service, New DB: Phase 2 – Cutover New Service->>New DB: Write only Monolith->>Shared DB: Read/Write (deprecated path) </div><h3 id="4.-embrace-eventual-consistency" tabindex="-1"><a class="header-anchor" href="#4.-embrace-eventual-consistency">4. Embrace Eventual Consistency</a></h3> <p>With separate services comes eventual consistency. Use <strong>domain events</strong> over synchronous REST calls wherever possible:</p> <ul> <li>Publish events to a message broker (Kafka, RabbitMQ)</li> <li>Services subscribe to relevant events</li> <li>Saga pattern for distributed transactions</li> </ul> <h2 id="key-takeaways" tabindex="-1"><a class="header-anchor" href="#key-takeaways">Key Takeaways</a></h2> <ul> <li><strong>Migrate iteratively</strong> — the Strangler Fig pattern is your friend.</li> <li><strong>Define clear boundaries</strong> using DDD bounded contexts.</li> <li><strong>Decouple the database</strong> as a separate, explicit step.</li> <li><strong>Invest in observability</strong> early — distributed tracing (Jaeger, Zipkin) and centralised logging (ELK stack) become essential.</li> <li><strong>Automate everything</strong> — CI/CD per service, infrastructure as code, automated testing.</li> </ul> <p>The migration journey is long, but each extracted service pays dividends in team autonomy and deployment velocity. Start small, validate, and build momentum.</p> Tom Seidel