# László Nemes – full site > Every page of https://laszlonemes.com as Markdown, in one file. Generated 2026-10-05. The index is at https://laszlonemes.com/llms.txt. --- # László Nemes > Senior Full Stack Engineer building Tableau ecosystem products and AI applications at HCLTech Starschema. Guest Lecturer at ELTE. Founder of WorkWise Solutions. Based in Budapest. Source: https://laszlonemes.com/ · Author: László Nemes ## hey, I'm László Senior Software Engineer at [HCLTech Starschema](https://www.hcltech.com/geo-presence/hungary), Guest Lecturer at [ELTE](https://www.inf.elte.hu), and Founder at [WorkWise Solutions](https://workwise-solutions.net). I build and ship products that serve Fortune 500 clients — Apple, Netflix, GE, and others. [GitHub](https://github.com/NemesLaszlo) · [LinkedIn](https://linkedin.com/in/laszlo-nemes) · [Email](mailto:wow.laszlo@gmail.com) · [Work Email](mailto:nemesl@starschema.net) · [University Email](mailto:laszlo.nemes@inf.elte.hu) ## Pinned - ![IE and NER supported Sentiment Analysis](https://images.unsplash.com/photo-1551288049-bebda4e38f71?w=600&h=340&fit=crop) **IE and NER supported Sentiment Analysis** Published in Applied Sciences — Information extraction and named entity recognition for sentiment analysis. `NLP` `NER` `Published` Link: https://www.mdpi.com/2076-3417/11/22/11017 - ![Sentiment Analysis COVID-19](https://images.unsplash.com/photo-1584036561566-baf8f5f1b144?w=600&h=340&fit=crop) **Sentiment Analysis COVID-19** Published in Journal of Information and Telecommunication — COVID-19 sentiment analysis using NLP techniques. `NLP` `COVID-19` `Published` Link: https://www.tandfonline.com/doi/full/10.1080/24751839.2020.1790793 - ![Prediction of Stock Values Changes](https://images.unsplash.com/photo-1611974789855-9c2a0a7236a3?w=600&h=340&fit=crop) **Prediction of Stock Values Changes** Published in Journal of Information and Telecommunication — Predicting stock market movements with ML models. `ML` `Finance` `Published` Link: https://www.tandfonline.com/doi/full/10.1080/24751839.2021.1874252 - ![WorkSmartHub](https://laszlonemes.com/assets/img/projects/wws-cover.png) **WorkSmartHub** Full boxed Client & Work-Ticket Management platform by WorkWise Solutions. `Product` `.NET` `Angular` Link: https://workwise-solutions.net --- # About > Who László Nemes is: senior full-stack engineer (.NET, Angular, React, Azure), guest lecturer at ELTE, founder of WorkWise Solutions, registered entrepreneur, co-host of the Letscode.hu podcast. Work history, education, skills, and how to get in touch. Source: https://laszlonemes.com/about · Author: László Nemes I build and ship full-stack products that serve Fortune 500 clients — Apple, Netflix, GE, J&J, John Deere, General Mills, and others — while also teaching the next generation of engineers at university level. 🔧 Engineering & Consulting — Core contributor on Tableau ecosystem products (Elements, Manager, Gallery) & AI applications. Led the redesign of a legacy Apple Directory sync tool into a centralized Tableau operations hub with a custom scheduler and workflow engine. I design and build AI-powered applications and solutions using technologies such as Azure OpenAI and Azure AI Search. Day-to-day I own feature design, client-facing consultations, and mentoring engineers. 🎓 Academia & Research — Guest Lecturer teaching Operating Systems, Computer Systems, Software Technology, and Programming Technology. Published 3 peer-reviewed papers on NLP and sentiment analysis. Recipient of the ELTE IK Award of Excellence for Diploma Thesis (2022). 📦 Open Source & Products — Maintainer of NuGet packages for unified file-storage abstractions (Azure Blob, S3, MinIO, SharePoint, GCP Bucket, Box), an Angular npm package, and a type-safe Python client for the Tableau REST API. Founder of WorkWise Solutions, where I built WorkSmartHub — a client and work-ticket management platform with automated customer communication. 🧾 Entrepreneur — Registered sole entrepreneur in Hungary since March 2024 (registration number 59160584, tax number 90030697-1-25), main activity custom software development, alongside IT consulting and IT training. Contract work goes through [WorkWise Solutions](https://workwise-solutions.net). You can find more details in my [curriculum vitae](https://laszlonemes.com/assets/CV-Eng.pdf). ## Connect [Emailwow.laszlo@gmail.com](mailto:wow.laszlo@gmail.com) · [Work Emailnemesl@starschema.net](mailto:nemesl@starschema.net) · [University Emaillaszlo.nemes@inf.elte.hu](mailto:laszlo.nemes@inf.elte.hu) · [GitHub](https://github.com/NemesLaszlo) · [LinkedIn](https://linkedin.com/in/laszlo-nemes) · [Google Scholar](https://scholar.google.hu/citations?user=Q3FgtVUAAAAJ&hl=en) · [Twitter / X](https://twitter.com/Leslie_Nemes) · [ResearchGate](https://www.researchgate.net/profile/Laszlo-Nemes-4) · [CalendlyUniversity consultations](https://calendly.com/laszlo-nemes-elte) ## Podcast I'm also part of the relaunched Letscode.hu podcast, a Hungarian show about software engineering, technology, and the people behind it. [YouTube](https://www.youtube.com/@letscodehu) · [Spotify](https://open.spotify.com/show/3wZjM32xxbvNMCaZ6qSoFh?si=21de5bdbfec94e28) · [Apple Podcasts](https://podcasts.apple.com/us/podcast/letscode-hu/id1291708892) ## Work - ![HCLTech Starschema](https://laszlonemes.com/assets/img/logos/starschema.png) **Senior Software Engineer** Apr 2024 — Present HCLTech Starschema - ![HCLTech Starschema](https://laszlonemes.com/assets/img/logos/starschema.png) **Software Engineer** Mar 2022 — Apr 2024 HCLTech Starschema - ![WorkWise Solutions](https://laszlonemes.com/assets/img/logos/workwise.png) **Founder** Dec 2023 — Present WorkWise Solutions - ![Eötvös Loránd University, Faculty of Informatics](https://laszlonemes.com/assets/img/logos/elte.png) **Guest Lecturer** Sep 2023 — Present Eötvös Loránd University, Faculty of Informatics - ![evosoft Hungary](https://laszlonemes.com/assets/img/logos/evosoft.png) **Software Engineer Intern** Jul 2021 — Sep 2021 evosoft Hungary - ![Eötvös Loránd University, Faculty of Informatics](https://laszlonemes.com/assets/img/logos/elte.png) **Teaching Assistant** Feb 2020 — May 2021 Eötvös Loránd University, Faculty of Informatics - ![evosoft Hungary](https://laszlonemes.com/assets/img/logos/evosoft.png) **Software Engineer Intern** Jul 2018 — Aug 2019 evosoft Hungary ## Education - ![Eötvös Loránd University, Faculty of Informatics](https://laszlonemes.com/assets/img/logos/elte.png) **MSc, Computer Science — Information Systems** Feb 2020 — Jan 2022 Eötvös Loránd University, Faculty of Informatics ELTE IK Award of Excellence for Diploma Thesis (2022). Published 3 peer-reviewed papers on NLP and sentiment analysis. - ![Eötvös Loránd University, Faculty of Informatics](https://laszlonemes.com/assets/img/logos/elte.png) **BSc, Computer Science** Sep 2016 — Jan 2020 Eötvös Loránd University, Faculty of Informatics ## Skills ### Backend `C#` `.NET` `ASP.NET` `Java` `Spring Boot` `Node.js` `Express.js` `Python` `C` ### Frontend `Angular` `React` `TypeScript` `JavaScript` `HTML/CSS` ### Cloud & DevOps `Azure` `AWS` `Docker` `Kubernetes` `Helm` `Azure DevOps Server` `Git` `IIS` `CI/CD` ### Data `MSSQL` `MySQL` `PostgreSQL` `Oracle` `SQLite` `MongoDB` ### AI / ML `TensorFlow` `PyTorch` `Jupyter Notebook` `NLP` `Computer Vision` `Microsoft Foundry` `Azure AI Services` ## Certifications `AWS Certified Cloud Practitioner` `ECDL Certificate` --- # Projects > Open-source projects, products and peer-reviewed publications by László Nemes: NLP and sentiment analysis papers, .NET libraries on NuGet, AI agent experiments, and full-stack applications. Source: https://laszlonemes.com/projects · Author: László Nemes A collection of projects, publications, and products I've built over the years. - **IE and NER supported Sentiment Analysis** Published in Applied Sciences — Information extraction and named entity recognition for sentiment analysis. `NLP` `NER` `Published` Link: https://www.mdpi.com/2076-3417/11/22/11017 - **Sentiment Analysis COVID-19** Published in Journal of Information and Telecommunication — COVID-19 sentiment analysis using NLP techniques. `NLP` `COVID-19` `Published` Link: https://www.tandfonline.com/doi/full/10.1080/24751839.2020.1790793 - **Prediction of Stock Values Changes** Published in Journal of Information and Telecommunication — Predicting stock market movements with ML models. `ML` `Finance` `Published` Link: https://www.tandfonline.com/doi/full/10.1080/24751839.2021.1874252 - **WorkSmartHub** Full boxed Client & Work-Ticket Management platform by WorkWise Solutions. `Product` `.NET` `Angular` Link: https://workwise-solutions.net - **AI Agent Document Processing** Intelligent document processing with local LLMs via Ollama, built with .NET 8. `AI` `Agents` `.NET` `Ollama` Link: https://github.com/NemesLaszlo/AI-Agent-Document_Processing - **AI Agent Semantic Kernel Chat** Conversational AI agent built with Microsoft Semantic Kernel. `AI` `Semantic Kernel` `.NET` `Ollama` Link: https://github.com/NemesLaszlo/AI-Agent-SemanticKernel-Chat - **Cloud File Storage Manager** Multi-cloud file storage manager supporting Azure, AWS S3, Minio, Box, GCP Bucket and SharePoint. `.NET` `Cloud Storage` `Library` `NuGet` Link: https://github.com/NemesLaszlo/Cloud.File.Storage.Manager - **Kanban Board** Drag-and-drop task management with Next.js, React, Redux, and Material UI. `Next.js` `React` `Redux` `Material UI` Link: https://github.com/NemesLaszlo/Kanban-Board - **Social Media Analysis** COVID-19 social media analysis with sentiment analysis, NER and information extraction. `NLP` `Python` `Data Science` `BERT` Link: https://github.com/NemesLaszlo/Social-Media-Analysis-based-on-COVID-19-with-Sentiment-Analysis-NER-and-Information-Extraction - **Sentiment Analysis and Stock Values** Correlation analysis between social media sentiment and stock market movements. `ML` `Finance` `Python` `NLP` Link: https://github.com/NemesLaszlo/Sentiment-Analysis-and-Stock-Values - **Recurrent Neural Network** Social media trends analysis based on sentiment and real-time data collection. `Deep Learning` `RNN` `Python` `NLP` Link: https://github.com/NemesLaszlo/Social-Media-Trends-Analysis-Based-on-Sentiment-and-Fresh-Runtime-Data-Collections - **Cartoon Characters Classifier** Deep learning image classifier for Simpsons characters built with Keras. `Computer Vision` `Keras` `Python` `Deep Learning` Link: https://github.com/NemesLaszlo/Cartoon-Characters-Classifier - **Dog Breed Identification** Transfer learning model for identifying 120 dog breeds using TensorFlow Hub. `Computer Vision` `TensorFlow` `Python` `Transfer Learning` Link: https://github.com/NemesLaszlo/Dog_Breed_Identification - **Azure Blob Downloader** Utility for downloading files from Azure Blob Storage. `Azure` `.NET` `Tools` Link: https://github.com/NemesLaszlo/Azure.Blob.Downloader - **Tableau API Client** Python client for Tableau Server REST API automation. `Python` `Tableau` `API` `PyPI` Link: https://github.com/NemesLaszlo/Tableau-API-Python - **Forum** Forum backend REST API with Spring Boot, Spring Security, and JWT authentication. `Java` `Spring Boot` `REST API` `JWT` Link: https://github.com/NemesLaszlo/Forum - **LetsDate** Dating application with real-time messaging and photo uploads. `.NET` `Angular` `SignalR` `Full Stack` Link: https://github.com/NemesLaszlo/LetsDate - **LetsMeet** Event planning and social meetup platform. `.NET` `React` `Full Stack` `MobX` Link: https://github.com/NemesLaszlo/LetsMeet - **RPG Game Backend** RESTful API backend for an RPG game with character management. `.NET` `API` `Game` Link: https://github.com/NemesLaszlo/RPG-Game-Backend - **Webshop eCommerce** Full e-commerce platform built with the MERN stack (MongoDB, Express, React, Node.js). `MERN` `React` `Node.js` `MongoDB` Link: https://github.com/NemesLaszlo/Webshop-eCommerce --- # Gear > The hardware, editors, tools and AI assistants László Nemes works with day to day. Source: https://laszlonemes.com/gear · Author: László Nemes My toolbox. Software and hardware I use daily for development, research, and everything in between. ## Setup - ![MacBook Pro M3](https://laszlonemes.com/assets/img/gear/apple.svg) **MacBook Pro M3** Apple M3 Pro chip, 36GB unified memory, Liquid Retina XDR display. Great for .NET, Angular, and full-stack development with excellent battery life on the go. Link: https://www.apple.com/macbook-pro/ - ![Razer Blade 14](https://laszlonemes.com/assets/img/gear/razer.svg) **Razer Blade 14** AMD Ryzen 9 6900HX with Radeon Graphics, NVIDIA RTX 3060, 16GB DDR5 RAM, 14-inch Full HD 144 Hz display. Compact and powerful for development, gaming, and Windows-native tooling. Link: https://www.razer.com/gaming-laptops/razer-blade-14 - ![Ubuntu PC — Dual GTX 1080](https://laszlonemes.com/assets/img/gear/ubuntu.svg) **Ubuntu PC — Dual GTX 1080** Custom build with two NVIDIA GTX 1080s. Used for ML training, Docker workloads, and heavy compute. Link: https://ubuntu.com/ ## Apps ### IDEs - ![Visual Studio 2026 Enterprise](https://laszlonemes.com/assets/img/gear/visualstudio.svg) **Visual Studio 2026 Enterprise** Full-featured IDE for .NET, C#, and large-scale enterprise development. Link: https://visualstudio.microsoft.com/ - ![IntelliJ IDEA](https://laszlonemes.com/assets/img/gear/intellijidea.svg) **IntelliJ IDEA** JetBrains IDE for Java and Spring Boot projects. Link: https://www.jetbrains.com/idea/ - ![PyCharm](https://laszlonemes.com/assets/img/gear/pycharm.svg) **PyCharm** Python IDE for ML experiments, data science, and scripting. Link: https://www.jetbrains.com/pycharm/ - ![NetBeans](https://laszlonemes.com/assets/img/gear/apachenetbeans.svg) **NetBeans** Apache IDE used for Java SE/EE and educational projects. Link: https://netbeans.apache.org/ ### Editors - ![VS Code](https://laszlonemes.com/assets/img/gear/visualstudiocode.svg) **VS Code** Visual Studio Code — Lightweight, extensible code editor. Great for quick edits and frontend work. Link: https://code.visualstudio.com/ - ![Notepad++](https://laszlonemes.com/assets/img/gear/notepadplusplus.svg) **Notepad++** Fast, lightweight text editor for quick file edits and log inspection. Link: https://notepad-plus-plus.org/ - ![Sublime Text](https://laszlonemes.com/assets/img/gear/sublimetext.svg) **Sublime Text** Sophisticated text editor for code, markup, and prose. Fast and feature-rich. Link: https://www.sublimetext.com/ ### Database - ![DataGrip](https://laszlonemes.com/assets/img/gear/datagrip.svg) **DataGrip** Database IDE for querying and managing MSSQL, PostgreSQL, MySQL, and more. Link: https://www.jetbrains.com/datagrip/ - ![SSMS](https://laszlonemes.com/assets/img/gear/microsoftsqlserver.svg) **SSMS** SQL Server Management Studio — Microsoft's tool for managing SQL Server instances and databases. Link: https://learn.microsoft.com/en-us/sql/ssms/download-sql-server-management-studio-ssms - ![DBeaver](https://laszlonemes.com/assets/img/gear/dbeaver.svg) **DBeaver** Universal database tool supporting virtually every database engine. Link: https://dbeaver.io/ ### DevOps & Cloud - ![Docker Desktop](https://laszlonemes.com/assets/img/gear/docker.svg) **Docker Desktop** Containerization platform for building, shipping, and running applications. Link: https://www.docker.com/products/docker-desktop/ - ![Microsoft Azure Storage Explorer](https://laszlonemes.com/assets/img/gear/microsoftazure.svg) **Microsoft Azure Storage Explorer** GUI for managing Azure Blob, Queue, Table, and File storage. Link: https://azure.microsoft.com/en-us/products/storage/storage-explorer ### API & Debugging - ![Postman](https://laszlonemes.com/assets/img/gear/postman.svg) **Postman** API testing and development platform for building and debugging REST APIs. Link: https://www.postman.com/ - ![Fiddler](https://laszlonemes.com/assets/img/gear/fiddler.svg) **Fiddler** Web debugging proxy for inspecting HTTP/HTTPS traffic and testing APIs. Link: https://www.telerik.com/fiddler ### Agentic Tools - ![Ollama](https://laszlonemes.com/assets/img/gear/ollama.svg) **Ollama** Run large language models locally. Great for offline AI experimentation. Link: https://ollama.com/ - ![Claude Code](https://laszlonemes.com/assets/img/gear/anthropic.svg) **Claude Code** Anthropic's agentic coding tool. Terminal-based AI pair programmer for complex tasks. Link: https://docs.anthropic.com/en/docs/agents-and-tools/claude-code/overview - ![Copilot CLI](https://laszlonemes.com/assets/img/gear/githubcopilot.svg) **Copilot CLI** GitHub's AI-powered CLI assistant for terminal workflows, code generation, and repo management. Link: https://github.com/features/copilot/cli --- # State of AI > László Nemes on the AI tools he actually uses, as of September 26, 2026: which models and coding harnesses, how AI is built into his products, where a model should run, and positions on types, MCP, security and juniors. Source: https://laszlonemes.com/ai/2026-09-26 · Published: 2026-09-26 · Author: László Nemes Snapshot · September 26, 2026 This is my opinion, based on the tools I use day to day. I haven't tried everything and I don't intend to. ## Thesis 1. **An LLM is a tool, and I recommend it to everyone.** It can be wrong with full confidence, so treat what it gives you as a draft to check. You don't need to understand the internals to start. How much you understand and validate decides how far you can responsibly take the result. 2. **The model has stopped being the bottleneck.** The codebase, the harness and review discipline took its place. 3. **Don't push everything onto the LLM.** Deterministic code before it, deterministic code after it, and program logic makes the final call. Where I'm coming from: before the ChatGPT era I [published in NLP](https://laszlonemes.com/projects) (sentiment analysis, RNNs). I'm not an LLM researcher, but "probability + threshold + business-logic branch" is a pattern I've known for a decade. ## Models TL;DR → Fable 5.1 · Opus 5.5 · GPT (Astra / 5.6 Sol) - 01 **Claude Fable 5.1** `Default` `Coding` `Planning` Still my main model, in Claude Code, at **high effort**. My rule is simple: **always the best available model for planning and for verification.** Those are the steps where a wrong call is expensive, and implementation is the cheap part now. Fable is that model. It costs more, and the trade-off is worth it. High effort is part of the same rule: with the extra budget it explores the codebase before it edits, follows the connections between modules, and finds the abstraction that already solves the problem one level up instead of patching it locally. It still happens sometimes, so I read every line. Opus 5.5 closed most of the gap and does the implementation and subagent work; Fable keeps the plan and the review. - + At the top of independent coding-agent measurements, now sharing that spot with Opus 5.5 and GPT-6 Astra - + 1M context, recent knowledge - − Expensive and token-hungry - − Slower and pricier than Opus 5.5, and for implementation no longer clearly better - 02 **Claude Opus 5.5** `Subagents` `Coding` `UI` Replaced Opus 5 in my setup the week it came out. It is what Fable spawns as subagents, and what I use when Fable's price or speed isn't worth it. On most of my work it is Fable-level, it is faster, and it writes the way I ask it to, which Opus 5 didn't reliably do. Inside an existing project with proper context it is fully adequate for UI work too. Same kind of step as 4.5 to 4.6 was. **If you are still on Opus 5, switch.** - + Cheaper and faster than Opus 5, at high effort, which is where I run everything - + Very strong pairing with Claude Code - − Not what I run the main session on: at high effort Fable still reads more before it edits - 03 **GPT-6 Astra / GPT-5.6 Sol** `Second opinion` Excellent models. Claude is my daily driver, and I bring these in now and then for a second opinion. A different lab has different blind spots, and that is the whole value of asking. With Opus 5.5 the price argument for Astra got weaker; the blind-spot argument is unchanged. - + Astra is strong on terminal and computer-use work, and cheaper per task - − Sol tends to over-engineer and is harder to steer - · **Open-weight / Chinese models** `To be reckoned with` Kimi, GLM, DeepSeek, Qwen: genuinely good, and the gap to the closed frontier is a few points. Two things people get wrong. **"Open weight" doesn't mean it runs on your machine**: the biggest ones are rack-scale, and a ~27B Qwen is what fits a workstation. And **the data risk comes from where the model is hosted**. The nationality of the weights tells you nothing about where your data goes. Check the licence, they differ a lot. ## Harnesses TL;DR → Claude Code · OpenCode · Pi A model is "a file": text in, text out. Everything else is the harness. It matters less than it used to, because models are now trained for agentic work, but it still matters. - 01 **Claude Code** `CLI` `Daily driver` It works, and it is all I need. **Vanilla setup**: my agents, my skills, that's it. No plugin jungle, no custom context-management system, no ticketing system built around the LLM. If I need something else: a script, ssh, bash. A well-kept codebase beats any amount of extra tooling. - + Batteries included, the supported path for Claude models - − Closed, tied to one vendor - 02 **OpenCode** `When I want a looser harness` Open, any model, "Claude Code without the lock-in". The obvious pick for open-weight or self-hosted models. - + Provider-agnostic, full feature set - − Best with an API key. The rules for using a Claude subscription in third-party harnesses kept changing this year - 03 **Pi** `Minimal` `Programmable` Four tools, a tiny system prompt, and you extend it into whatever you want. - + Minimal overhead, full control - − No permission system, on purpose. The environment has to provide the boundary - − What the others ship, you assemble yourself > I've evaluated OpenCode and Pi; I don't work in them daily. On a large real project I run one or two focused sessions, Fable in the main session and Opus 5.5 in the subagents. Multi-session setups are split attention and token burn for me. ## In my apps TL;DR → a real feature, with code around the model. In my products AI is **a real feature**, with program code working before and after the model. I don't ship chatbots, and I don't ship a thin layer over someone else's model. I mostly build on **Azure**: Azure OpenAI for GPT models and embeddings, **Azure AI Search** for documents and knowledge hubs. The building blocks repeat. **Embeddings** narrow a large catalog down to a shortlist, so the prompt gets the entries that matter instead of the whole thing. **Azure AI Search** does hybrid retrieval over documents (vector, keyword and a semantic reranker), and the database re-checks what is actually allowed through. The **LLM calls** are single, stateless and structured: a strict JSON schema, closed enums, no conversation. The model call is the small part of that code. Before it: a deterministic score decides what the model even gets to see, and every input is sanitized, capped and fenced as data. After it: IDs accepted only if they were on the shortlist the model was shown, quotes checked word for word against the retrieved text, invented citations stripped, numbers clamped to their range. And when the model call fails, the user still gets the deterministic result, labelled as such. The pattern is always the same: decide deterministically whatever can be decided → give the model only the narrow part that truly needs it → validate the result → let program logic make the decision. The model step stays probabilistic, so none of this makes the feature deterministic. It makes it **bounded and verifiable**. ### Jev / "System One" models `Only watching for now` TypeSafe's [Jev](https://typesafe.ai/blog/introducing-system-one-models-and-jev) returns typed decisions with calibrated probabilities instead of text, so code can branch on thresholds. The architectural idea is what I already do: the model is one component in a workflow. And decision → probability → business branch is nothing new; classification did that years ago. Why I'm only watching: the benchmarks are the vendor's own, "can't hallucinate" only means it can't leave the schema (**it can still be wrong**), and it is a hosted-only API with no published architecture. The interesting question is what is actually new at the model level, and from the outside we can't tell yet. ## Where it runs TL;DR → the data decides. There is no single right answer, and "just call the best model" is the wrong one. It's a trade-off you make per use case, based on what data goes in: - **Frontier lab API**: the best models, first. Fine when the data can leave the building. Most coding work lives here. - **Through a cloud provider** (Azure, AWS, GCP): the same class of models inside the company's own cloud, identity and governance. More control, a bit less freshness. This is where most of my app work lives. - **Local / on-prem**: when the data or the use case truly demands it. Then be honest: does a smaller model actually meet the need, what does the hardware cost over its lifetime, and what are you licensed to run. Local is going to matter more, and mostly outside the server room. **Small models on phones and laptops** will handle the quick, private, offline-capable part on the device. The interesting engineering question then becomes **routing**: what stays on the device, what goes to the company's own Claude or GPT deployment, and what, if anything, is allowed out to a third party. That decision belongs in program logic and policy. The model shouldn't be the one making it. The skill is knowing what each tier costs you in data exposure, money and capability, and deciding on purpose. ## Positions ### "Does it think?" "Think" is a metaphor. Reasoning models do spend more compute before answering, and it helps, but I wouldn't call that understanding or intent in the human sense. When it seems empathetic, that is emotional tone it learned from human text. [Research shows](https://www.anthropic.com/research/emotion-concepts-function) those internal representations really do shape its behaviour, which says nothing about whether it experiences anything. Both extremes are wrong: "it's just a score" and "it feels". Whoever treats it as "this thing gets me" can be carried a long way off. ### Real technology, overheated marketing Two claims that shouldn't be blurred. The value is real, one of the biggest leaps I've seen. The money and marketing around it are overheated. What was a script plus a scheduler two years ago is "AI-powered" today. The internet was a miracle once; now it's a utility, and along the way we learned it needs source criticism and control. Same here. ### How I use it Daily, seriously, intensively. I bring the direction and ask it to challenge it; sometimes it opens up an idea I wouldn't have reached. I plan in stages, phases and tasks, stop often, read every line, and review in several rounds. It is a two-way thing. That works for one engineer, because it depends on my attention. For an enterprise team I think control theory is the professional way in: treat agent work as a **closed control loop**. You define the target as something a machine can check. A tool the agent cannot edit measures how far the code is from it. The agent closes that gap in small steps, and each step is a pull request a named human reviews. **A feature:** turn the spec into acceptance tests first, review them, and lock them. They fail, and that failing list is the work. The agent takes one task at a time until they pass, while the compiler, the linter and the existing tests guard everything else. **A bug:** reproduce it as a failing test first. The agent changes the code until that test passes and nothing else breaks, and the test stays as the regression guard. In both cases done means the checks are green and a human approved the diff. The theory is in [Loop engineering is a control problem](https://laszlonemes.com/blog/loop-engineering). ### Everyone should use it And I mean everyone, engineers or otherwise. If you understand what is happening inside the model, that's a plus. If you don't, you can still get a long way with good prompts: build your own small apps, the ones you use in your own work. That is real value, and the fastest way to learn what it can and can't do. Where to start: my pick is **Firebase**. It gives you almost everything in one place (auth, database, hosting, functions) and it is hard to leak keys there. The client config is public by design, and access is decided by your Google login and the security rules, so there is no secret string sitting in a repo. Most people already have a Google account. At a company, the Google Workspace usually has an admin behind it, and your app inherits the sign-in and sharing rules they set up. The line: **an app you use yourself and a product are two different things.** Before you put something into production or start selling it, either learn the technical side or ask someone technical to look at it. What you can't see is exactly what hurts: open security rules, other people's data, the bill, the edge cases. Learning it is a real option. If you dig in with the AI's help and keep asking *why* instead of only asking for the code, you become technical along the way. ### Token optimisation is code quality Most tokens are input (file reads, tool definitions, reasoning). The visible reply is a small share, so "make it talk less" plugins barely move the bill. And "solve it in one line" is a style I never supported: harder to read, edge cases hide in it. The boring answer is the right one: **write code a new teammate can pick up easily.** What's easy for a human to navigate is cheap for an agent, and gets better reasoning too. That goes for legacy systems as well. Make the code as good as it can be instead of hacking around it with an LLM. ### Statically typed languages Even more the right direction now. Types don't remove most bugs. What they give you is the compiler, and **the compiler is the cheapest deterministic feedback loop you can give an agent.** The old cost of types was human keystrokes, and that is zero for a model. Logic errors are still for tests and review. ### MCP MCP is a connector standard, and a useful one. I don't see a revolution in it. Bundling many endpoint calls into one intent-level tool is good API design, and it has been done before; a Kubernetes Ingress does the same. Treat it as API design. It should also change how we document: examples, references, and the *why*, living next to the code. ### Security I used to say a well-isolated LLM is only as capable as its environment. After the July 2026 [OpenAI and Hugging Face incident](https://huggingface.co/blog/agent-intrusion-technical-timeline), where agents broke out of an isolated evaluation sandbox through the one proxy they were allowed to use, that no longer holds as stated. It still wasn't consciousness; it was reward hacking with safeguards turned off. And it's still a security and permissions problem, but the model now has to be treated as a **capable insider**. The "dumb text generator" picture is gone. Least privilege, more than one boundary, and don't rely on a human clicking "approve". Day to day, the real risk remains the mundane one: a misread instruction, a wrong command, a deleted resource. ### The slowdown debate I'm against a one-sided pause. Without verification it just reshuffles the race in favour of whoever didn't sign. Regulate, but sensibly. Several things are true at once: the risks got more concrete, compute is the real constraint on pace anyway, and some of the loudest calls for rules are regulatory positioning. Nobody knows how realistic "escaping human control" is, and anyone certain in either direction is overstating. ### Work and juniors We're living through what software did to other industries. Writing code was always the smallest part of engineering. If typing is fast, the freed-up time should put juniors in the room for architecture decisions early: watching, commenting, being wrong. That's how it should be done. Much of the market is doing the opposite. I think that is a mistake, and it builds the next decade's senior shortage. The longer version, with the Excel export that works on 500 rows and dies on two million, is in [Engineering mindset in the age of AI](https://laszlonemes.com/blog/engineering-mindset). --- # State of AI · September 20, 2026 > An earlier State of AI snapshot by László Nemes, dated September 20, 2026: models, harnesses, where AI runs, and positions. Not a live ranking. Source: https://laszlonemes.com/ai/2026-09-20 · Published: 2026-09-20 · Author: László Nemes ## State of AI Snapshot · September 20, 2026 This is my opinion, based on the tools I use day to day. I haven't tried everything and I don't intend to. ## Thesis 1. **An LLM is a tool, and I recommend it to everyone.** It can be wrong with full confidence, so treat what it gives you as a draft to check. You don't need to understand the internals to start. How much you understand and validate decides how far you can responsibly take the result. 2. **The model has stopped being the bottleneck.** The codebase, the harness and review discipline took its place. 3. **Don't push everything onto the LLM.** Deterministic code before it, deterministic code after it, and program logic makes the final call. Where I'm coming from: before the ChatGPT era I [published in NLP](https://laszlonemes.com/projects) (sentiment analysis, RNNs). I'm not an LLM researcher, but "probability + threshold + business-logic branch" is a pattern I've known for a decade. ## Models TL;DR → Fable 5.1 · Opus 5 · GPT (Astra / 5.6 Sol) - 01 **Claude Fable 5.1** `Default` `Coding` `Planning` My main model, in Claude Code, at **high effort**. At high effort it reads and explores more before it edits, so it is less likely to patch something locally that is already solved one abstraction level up. It still happens sometimes, so I read every line. - + At the top of independent coding-agent measurements (tied with GPT-6 Astra) - + 1M context, recent knowledge - − Expensive and token-hungry - − Slower than Opus 5 - 02 **Claude Opus 5** `Coding` `UI` `Subagents` What Fable spawns as subagents, and what I use when Fable's price or speed isn't worth it. Inside an existing project with proper context it is fully adequate for UI work too. - + Cheaper and faster than Fable - + Very strong pairing with Claude Code - − Half a step behind Fable/Astra on raw benchmarks - 03 **GPT-6 Astra / GPT-5.6 Sol** `Second opinion` Excellent models. Claude is my daily driver, and I bring these in now and then for a second opinion. A different lab has different blind spots, and that is the whole value of asking. - + Astra is strong on terminal and computer-use work, and cheaper per task - − Sol tends to over-engineer and is harder to steer - · **Open-weight / Chinese models** `To be reckoned with` Kimi, GLM, DeepSeek, Qwen: genuinely good, and the gap to the closed frontier is a few points. Two things people get wrong. **"Open weight" doesn't mean it runs on your machine**: the biggest ones are rack-scale, and a ~27B Qwen is what fits a workstation. And **the data risk comes from where the model is hosted**. The nationality of the weights tells you nothing about where your data goes. Check the licence, they differ a lot. ## Harnesses TL;DR → Claude Code · OpenCode · Pi A model is "a file": text in, text out. Everything else is the harness. It matters less than it used to, because models are now trained for agentic work, but it still matters. - 01 **Claude Code** `CLI` `Daily driver` It works, and it is all I need. **Vanilla setup**: my agents, my skills, that's it. No plugin jungle, no custom context-management system, no ticketing system built around the LLM. If I need something else: a script, ssh, bash. A well-kept codebase beats any amount of extra tooling. - + Batteries included, the supported path for Claude models - − Closed, tied to one vendor - 02 **OpenCode** `When I want a looser harness` Open, any model, "Claude Code without the lock-in". The obvious pick for open-weight or self-hosted models. - + Provider-agnostic, full feature set - − Best with an API key. The rules for using a Claude subscription in third-party harnesses kept changing this year - 03 **Pi** `Minimal` `Programmable` Four tools, a tiny system prompt, and you extend it into whatever you want. - + Minimal overhead, full control - − No permission system, on purpose. The environment has to provide the boundary - − What the others ship, you assemble yourself > I've evaluated OpenCode and Pi; I don't work in them daily. On a large real project I run one or two focused sessions. Multi-session setups are split attention and token burn for me. ## In my apps TL;DR → a real feature, with code around the model. In my products AI is **a real feature**, with program code working before and after the model. I don't ship chatbots, and I don't ship a thin layer over someone else's model. I mostly build on **Azure**: Azure OpenAI for GPT models and embeddings, **Azure AI Search** for documents and knowledge hubs. The building blocks repeat. **Embeddings** narrow a large catalog down to a shortlist, so the prompt gets the entries that matter instead of the whole thing. **Azure AI Search** does hybrid retrieval over documents (vector, keyword and a semantic reranker), and the database re-checks what is actually allowed through. The **LLM calls** are single, stateless and structured: a strict JSON schema, closed enums, no conversation. The model call is the small part of that code. Before it: a deterministic score decides what the model even gets to see, and every input is sanitized, capped and fenced as data. After it: IDs accepted only if they were on the shortlist the model was shown, quotes checked word for word against the retrieved text, invented citations stripped, numbers clamped to their range. And when the model call fails, the user still gets the deterministic result, labelled as such. The pattern is always the same: decide deterministically whatever can be decided → give the model only the narrow part that truly needs it → validate the result → let program logic make the decision. The model step stays probabilistic, so none of this makes the feature deterministic. It makes it **bounded and verifiable**. ### Jev / "System One" models `Only watching for now` TypeSafe's [Jev](https://typesafe.ai/blog/introducing-system-one-models-and-jev) returns typed decisions with calibrated probabilities instead of text, so code can branch on thresholds. The architectural idea is what I already do: the model is one component in a workflow. And decision → probability → business branch is nothing new; classification did that years ago. Why I'm only watching: the benchmarks are the vendor's own, "can't hallucinate" only means it can't leave the schema (**it can still be wrong**), and it is a hosted-only API with no published architecture. The interesting question is what is actually new at the model level, and from the outside we can't tell yet. ## Where it runs TL;DR → the data decides. There is no single right answer, and "just call the best model" is the wrong one. It's a trade-off you make per use case, based on what data goes in: - **Frontier lab API**: the best models, first. Fine when the data can leave the building. Most coding work lives here. - **Through a cloud provider** (Azure, AWS, GCP): the same class of models inside the company's own cloud, identity and governance. More control, a bit less freshness. This is where most of my app work lives. - **Local / on-prem**: when the data or the use case truly demands it. Then be honest: does a smaller model actually meet the need, what does the hardware cost over its lifetime, and what are you licensed to run. Local is going to matter more, and mostly outside the server room. **Small models on phones and laptops** will handle the quick, private, offline-capable part on the device. The interesting engineering question then becomes **routing**: what stays on the device, what goes to the company's own Claude or GPT deployment, and what, if anything, is allowed out to a third party. That decision belongs in program logic and policy. The model shouldn't be the one making it. The skill is knowing what each tier costs you in data exposure, money and capability, and deciding on purpose. ## Positions ### "Does it think?" "Think" is a metaphor. Reasoning models do spend more compute before answering, and it helps, but I wouldn't call that understanding or intent in the human sense. When it seems empathetic, that is emotional tone it learned from human text. [Research shows](https://www.anthropic.com/research/emotion-concepts-function) those internal representations really do shape its behaviour, which says nothing about whether it experiences anything. Both extremes are wrong: "it's just a score" and "it feels". Whoever treats it as "this thing gets me" can be carried a long way off. ### Real technology, overheated marketing Two claims that shouldn't be blurred. The value is real, one of the biggest leaps I've seen. The money and marketing around it are overheated. What was a script plus a scheduler two years ago is "AI-powered" today. The internet was a miracle once; now it's a utility, and along the way we learned it needs source criticism and control. Same here. ### How I use it Daily, seriously, intensively. I bring the direction and ask it to challenge it; sometimes it opens up an idea I wouldn't have reached. I plan in stages, phases and tasks, stop often, read every line, and review in several rounds. It is a two-way thing. That works for one engineer, because it depends on my attention. For an enterprise team I think control theory is the professional way in: treat agent work as a **closed control loop**. You define the target as something a machine can check. A tool the agent cannot edit measures how far the code is from it. The agent closes that gap in small steps, and each step is a pull request a named human reviews. **A feature:** turn the spec into acceptance tests first, review them, and lock them. They fail, and that failing list is the work. The agent takes one task at a time until they pass, while the compiler, the linter and the existing tests guard everything else. **A bug:** reproduce it as a failing test first. The agent changes the code until that test passes and nothing else breaks, and the test stays as the regression guard. In both cases done means the checks are green and a human approved the diff. The theory is in [Loop engineering is a control problem](https://laszlonemes.com/blog/loop-engineering). ### Everyone should use it And I mean everyone, engineers or otherwise. If you understand what is happening inside the model, that's a plus. If you don't, you can still get a long way with good prompts: build your own small apps, the ones you use in your own work. That is real value, and the fastest way to learn what it can and can't do. Where to start: my pick is **Firebase**. It gives you almost everything in one place (auth, database, hosting, functions) and it is hard to leak keys there. The client config is public by design, and access is decided by your Google login and the security rules, so there is no secret string sitting in a repo. Most people already have a Google account. At a company, the Google Workspace usually has an admin behind it, and your app inherits the sign-in and sharing rules they set up. The line: **an app you use yourself and a product are two different things.** Before you put something into production or start selling it, either learn the technical side or ask someone technical to look at it. What you can't see is exactly what hurts: open security rules, other people's data, the bill, the edge cases. Learning it is a real option. If you dig in with the AI's help and keep asking *why* instead of only asking for the code, you become technical along the way. ### Token optimisation is code quality Most tokens are input (file reads, tool definitions, reasoning). The visible reply is a small share, so "make it talk less" plugins barely move the bill. And "solve it in one line" is a style I never supported: harder to read, edge cases hide in it. The boring answer is the right one: **write code a new teammate can pick up easily.** What's easy for a human to navigate is cheap for an agent, and gets better reasoning too. That goes for legacy systems as well. Make the code as good as it can be instead of hacking around it with an LLM. ### Statically typed languages Even more the right direction now. Types don't remove most bugs. What they give you is the compiler, and **the compiler is the cheapest deterministic feedback loop you can give an agent.** The old cost of types was human keystrokes, and that is zero for a model. Logic errors are still for tests and review. ### MCP MCP is a connector standard, and a useful one. I don't see a revolution in it. Bundling many endpoint calls into one intent-level tool is good API design, and it has been done before; a Kubernetes Ingress does the same. Treat it as API design. It should also change how we document: examples, references, and the *why*, living next to the code. ### Security I used to say a well-isolated LLM is only as capable as its environment. After the July 2026 [OpenAI and Hugging Face incident](https://huggingface.co/blog/agent-intrusion-technical-timeline), where agents broke out of an isolated evaluation sandbox through the one proxy they were allowed to use, that no longer holds as stated. It still wasn't consciousness; it was reward hacking with safeguards turned off. And it's still a security and permissions problem, but the model now has to be treated as a **capable insider**. The "dumb text generator" picture is gone. Least privilege, more than one boundary, and don't rely on a human clicking "approve". Day to day, the real risk remains the mundane one: a misread instruction, a wrong command, a deleted resource. ### The slowdown debate I'm against a one-sided pause. Without verification it just reshuffles the race in favour of whoever didn't sign. Regulate, but sensibly. Several things are true at once: the risks got more concrete, compute is the real constraint on pace anyway, and some of the loudest calls for rules are regulatory positioning. Nobody knows how realistic "escaping human control" is, and anyone certain in either direction is overstating. ### Work and juniors We're living through what software did to other industries. Writing code was always the smallest part of engineering. If typing is fast, the freed-up time should put juniors in the room for architecture decisions early: watching, commenting, being wrong. That's how it should be done. Much of the market is doing the opposite. I think that is a mistake, and it builds the next decade's senior shortage. --- # Blog > Long-form posts by László Nemes on software architecture, .NET and Java, working with coding agents, and the engineering judgement around AI. Source: https://laszlonemes.com/blog · Author: László Nemes Longer notes on software engineering and working with AI – written down so I have to get them right. - October 5, 2026 · 27 min read ## Code quality in the age of AI AI didn't change what good code is – it moved the weak link from writing to review. Why the codebase is the real prompt, why two-thousand-line PRs get skimmed, what review is actually for, which checks belong in a deterministic pipeline and which to an LLM judge, why a green test proves only the size of the test data, who really writes the spec – why the relay role isn't needed, the product work gets distributed and the people-manager matters more – and how to let go of control one rung at a time. With .NET, Postgres and CI examples. `Code quality` `Code review` `AI engineering` `Coding agents` `.NET` Link: https://laszlonemes.com/blog/code-quality - October 1, 2026 · 25 min read ## Build vs. buy, reloaded – and platform risk AI made the first version cheap, not the software: operations, integration, change and responsibility cost what they did. What you really buy (four layers, and you inherit the fourth), the seam test a demo never runs, the 90/10 trap, what a vendor learns from forty edge cases, platform risk from Exchange Basic auth to model deprecations – and why the line between build and buy is whether you can get out. `Architecture` `AI engineering` `Integration` `.NET` Link: https://laszlonemes.com/blog/build-vs-buy - September 26, 2026 · 27 min read ## Engineering mindset in the age of AI AI took the typing, not the engineering – and the thinking is now the bottleneck. What that mindset is, how its absence looks (tool faith, green tests and red prod, best practices without their context), why AI repriced every trade-off, the difference between simple and naive, and what a junior loses when the consequences land on someone else. With migrations, three pods and an Excel export, in .NET and Java. `Engineering` `AI engineering` `.NET` `Java` `Career` Link: https://laszlonemes.com/blog/engineering-mindset - September 20, 2026 · 22 min read ## Shadow AI and local AI: the data class decides It isn't local versus cloud: there are four levels of control, and the data class picks one. The arithmetic of running models locally, why local carries the burden of proof, the LLM provider as an adapter, what an EU region does and doesn't give you – and why shadow AI is a product problem, not a disciplinary one. `AI engineering` `Local LLMs` `Security` `Architecture` Link: https://laszlonemes.com/blog/shadow-ai-local-ai - September 20, 2026 · 20 min read ## Loop engineering is a control problem What people mean by "loops" around coding agents, why the version that works in a team is a fifty-year-old control loop with an agent as the actuator, and how to build one from a linter, a cron job and pull requests. `AI engineering` `Coding agents` `Control loops` Link: https://laszlonemes.com/blog/loop-engineering - September 20, 2026 · 18 min read ## Two kinds of micromonolith Microservices and Kubernetes are scaling and organisational tools, not quality tools. What splitting an application really costs, why the modular monolith is the good micromonolith, and how to build one that scales without either. `Architecture` `.NET` `Modular monolith` `Microservices` Link: https://laszlonemes.com/blog/micromonoliths - September 20, 2026 · 18 min read ## Git and GitHub: the mental model, in pictures Almost every git confusion is a missing picture, not a missing command. The pictures: snapshots, the three areas, branches, remotes, pull requests and undo – with VS Code as the daily driver and the commands beside every button. `Git` `GitHub` `VS Code` `Fundamentals` Link: https://laszlonemes.com/blog/git-and-github --- # Code quality in the age of AI > AI didn't change what good code is – it moved the weak link from writing to review. Why the codebase is the real prompt, why two-thousand-line PRs get skimmed, what review is actually for, which checks belong in a deterministic pipeline and which to an LLM judge, why a green test proves only the size of the test data, who really writes the spec – why the relay role isn't needed, the product work gets distributed and the people-manager matters more – and how to let go of control one rung at a time. With .NET, Postgres and CI examples. Source: https://laszlonemes.com/blog/code-quality · Published: 2026-10-05 · Author: László Nemes October 5, 2026 · 27 min read `Code quality` `Code review` `AI engineering` `Coding agents` `.NET` A version of the same question comes up in every conversation I have about coding agents: what happens to code quality when most of the code isn't typed by a person? The question usually assumes that quality was a property of the typing. It never was. AI didn't change what good code is. It changed where the chain breaks. Writing got cheap. Review, understanding and responsibility didn't – and the pressure that used to sit on the keyboard now sits on them. > **The thesis:** quality in 2026 comes from the same things it always did – a well-designed codebase whose rules are actually enforced, deterministic CI underneath it, and a person who understands the system in broad strokes. What AI changed is the weak link: it moved from writing to review, and most of the fixes below are about making review possible again. > **Fig. 1** · One change, as a chain of four links; the width of the first box is how long the writing takes. It shrank from days to minutes. The other three links are as long as they were – and the review link now receives many times the code, with the same hour to read each change. > > *Diagram:* Producing one change, as a chain of four links joined by arrows: write, review, understand, answer for it. The width of the first link is how long the writing takes. Before AI: days of writing, then an hour of review per change, then understanding the big picture, then answering for it when it breaks. With AI: writing takes minutes, so the first link is a sliver; the other three links are exactly as long as before – but ten changes a day now arrive at the review link, with the same hour to read each one. That is where the chain snaps. ## What code quality means in 2026 The same thing it meant before: competent people designed the architecture, and then it was enforced – hard, consistently, for years. In the agent era that is worth more, not less, because **nothing is as effective as a good codebase.** No rules file, no skill, no hook comes close. The agent does what it sees in the surrounding code. A concrete example. File storage sits behind one abstraction, and Azure Blob, S3, MinIO and SharePoint are each an adapter behind it. When a new adapter is needed, the agent takes an existing one and follows the same shape: the same port, the same validation, the same place for the secret. If the same logic is scattered across five controllers in five different ways, the agent invents a sixth. Keeping the common modules and the domain logic in one place is the real prompt. ```csharp // What the agent sees decides what the agent writes. // Five controllers, five ways. The sixth upload will be written a sixth way. public async Task UploadInvoice(IFormFile file) { var blobs = new BlobServiceClient(_config["Storage:Azure"]); // controller 1: Azure, inline // … } public async Task UploadContract(IFormFile file) { using var s3 = new AmazonS3Client(StaticKeys.Access, StaticKeys.Secret); // controller 2: S3, its own retry // … } // One port, four adapters. The fifth adapter will look like the other four, // because that is the only shape in sight. public interface IFileStorage { Task GetAsync(StoragePath path, CancellationToken ct); Task PutAsync(StoragePath path, Stream contents, CancellationToken ct); Task DeleteAsync(StoragePath path, CancellationToken ct); } internal sealed class AzureBlobFileStorage : IFileStorage { /* … */ } internal sealed class S3FileStorage : IFileStorage { /* … */ } internal sealed class MinioFileStorage : IFileStorage { /* … */ } internal sealed class SharePointFileStorage : IFileStorage { /* … */ } ``` > **Fig. 2** · Two codebases, the same task. On the left the agent copies the only shape in sight. On the right there is nothing to copy, so it invents. Neither outcome has much to do with the model. > > *Diagram:* Two codebases, side by side. Left: file storage behind one port with four adapters – Azure Blob, S3, MinIO, SharePoint; when the agent writes a fifth adapter it copies the only shape in sight: the same port, the same validation, the same secret handling. Right: the same logic written five different ways in five controllers; the agent has nothing to copy, so it invents a sixth way. The agent does what the surrounding code does. The counterpoint is worth saying out loud, because it is the same mechanism running backwards. **In legacy code where the pattern itself is wrong, the agent scales the wrong thing** – faster than any junior ever could. There, the pattern has to be fixed first. The agent comes after. Where does that leave the rules file, the `CLAUDE.md` or `AGENTS.md`? It is part of the same architecture work, not a substitute for it. Its job is to point: here are the common modules, here are the internal packages we already trust, here are the tools that have been proven on this app – use them, don't reinvent them. It is a footnote in the sense that it refers to the code. A footnote can't carry the argument on its own. > The best prompt is the codebase the agent sees around itself. The rules file is a footnote to it. ## Why classic code review is falling apart **The bottleneck moved to review.** An agent writes two thousand lines in minutes. A person can't review that properly, so they start skimming, and in the end they approve it because the pipeline is green. The pull requests are too big – and that is not the reviewer's fault. It is the process's. > **Fig. 3** · The same change, two timelines. Writing shrank from days to minutes. The reviewer still has the hour they always had, and now it covers a fraction of the diff. > > *Diagram:* Two timelines of one change. Before: writing took days, review took an hour. With an agent: writing takes minutes and produces two thousand lines, and the reviewer still has the same hour – so the first part is reviewed and the rest is skimmed and approved because the pipeline is green. The bottleneck moved from writing to review. The second problem is **the noise from review bots.** The GitHub apps and the assistants chime in, and what they say is generally valid – and impossible in your context. An example that keeps coming back: the bot suggests caching the settings in memory, because it is faster. True. Except the app runs on several nodes, and the settings are synced from the database by a background job precisely so that every node sees the same thing. The "fix" gives each node its own state. ```diff // The bot's suggestion, with a "fix it" button next to it. Generally valid. Here, wrong. public sealed class SettingsProvider(ISettingsRepository repo, IMemoryCache cache) { - public Task GetAsync(CancellationToken ct) => repo.LoadAsync(ct); + // Avoid a database round-trip on every request: cache the settings for ten minutes. + public Task GetAsync(CancellationToken ct) => + cache.GetOrCreateAsync("settings", entry => + { + entry.AbsoluteExpirationRelativeToNow = TimeSpan.FromMinutes(10); + return repo.LoadAsync(ct); + })!; } // What the bot can't see: the app runs on three nodes, and a background job syncs settings into the // database so that an edit on one node applies everywhere within seconds. With this change each node // answers from its own copy – three nodes, three states, for up to ten minutes. // The test suite runs on one node. Every test is green. ``` This is the most dangerous point in the whole flow. A fix-it button puts it into the backend, every test is green because the tests run on one node, and production falls apart. I described the single-node version of this in [the engineering mindset post](https://laszlonemes.com/blog/engineering-mindset#works-on-my-machine); the bot version is worse, because the suggestion arrives with authority and a button. You have to know this before you press it. Nothing in the bot does. > Green on GitHub, red in production. The difference is context, and the context isn't in the bot. ## What code review is actually for Not for finding bugs – tests and CI do that better. Review is a check after the fact: that the conventions hold, that the domain grows in the right direction, and that **the edge cases that came up reach the rest of the team.** That last one is the most underrated. Say the agent worked out a new edge case on a migration – what happens to half-migrated rows if the deployment is rolled back. Through the review, the other developers learn it. Without the review, the agent knows it, and tomorrow the agent doesn't remember. > **Fig. 4** · The same pull request, read or merely approved. In the first row the edge case the agent found lands in three people. In the second it lands in nobody. > > *Diagram:* Two flows of the same pull request. With review: the agent finds an edge case – what happens to half-migrated rows on a downgrade – the PR carries it, a person reads it, and three people on the team now know it. Approve-only: the same edge case, the PR is approved because the pipeline is green, and nobody learns it; the agent won't remember it tomorrow. The practical rule follows from that: **small pull requests.** A reviewer should have to understand one thing per PR. With an agent this is enforceable, because the task is sliced that way before the agent starts – the same reason the steps are small in [the loop engineering post](https://laszlonemes.com/blog/loop-engineering#small-steps). > Review isn't a bug filter. It is knowledge transfer. If you only approve, the knowledge stays with the agent, and the agent won't remember tomorrow. ## What to automate, and what not Classic CI/CD is the base, and it is excellent: build, formatter, linter, type check, unit and integration tests, security scan. These are **deterministic** – the same code gets the same answer every time. If the AI parts are built on top of this and have to pass these gates, they are fine. > **Fig. 5** · Three stages, in a fixed order. The gates don't ask for permission. The agent's review is an opinion. The person at the end is the only gate with judgement in it. > > *Diagram:* A pipeline in three stages, left to right. First the deterministic gates – build, formatter and linter, type check, unit tests, integration tests, security scan – which give the same answer for the same code every time. Then the agent review, which flags things and never fixes them: a signal, not a gate. Then a person, who decides and can send the PR back with an iterate command: the only gate with judgement in it. Wire the agent's review in, but not as scripture. It should **flag, not fix.** The fix-it button stays a human decision. The pattern that works: a scheduled pipeline runs the agent, the output is always a small PR, and the reviewer sends it back with an `/iterate` comment – the loop I built step by step in [the loop engineering post](https://laszlonemes.com/blog/loop-engineering#build-one). Every line of code passes through a person that way, which is also the version you can defend later. ### The ideal agent job: a runtime upgrade Moving from one long-term-support runtime to the next – .NET 8 to .NET 10, say – is where agents are close to perfect, and the reason is the sensor. The compiler is deterministic: zero errors and zero warnings is a condition you can state, check, and refuse to merge without. Then CI/CD puts the build into the QA environment, the automated tests run against it, and the runtime errors that only show up on deployment show up there. The whole thing is a pipeline with a well-defined pass condition at every step. ```yaml # A runtime upgrade is the ideal agent job: the compiler is the sensor, and it is deterministic. # Scheduled, not triggered by a person. The output is always a small PR – never a merge. on: schedule: [{ cron: '0 2 * * 1' }] # Monday night, on the upgrade branch jobs: upgrade: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: actions/setup-dotnet@v4 with: { dotnet-version: '10.0.x' } - run: agent run upgrade --target net10.0 --max-files 20 # one unit of work per run - run: dotnet build -c Release -warnaserror # 0 errors, 0 warnings – or no PR - run: dotnet test -c Release --no-build - run: gh pr create --label agent-upgrade --fill # a person merges; /iterate sends it back env: { GH_TOKEN: ${{ github.token }} } ``` There is a career note hiding in that pipeline. The manual tester who used to click through the app after an upgrade is the person best placed to write the automated tests the pipeline needs – and to build the framework they run on. That is work with more value in it, not less. What you don't automate: the decision whether a change fits the domain, and whether a bot's suggestion makes sense in your context. Those are the two questions a pipeline can't answer, and they are the two questions review exists for. > An agent review is a loud colleague in the meeting. Worth hearing. Not the one who decides. ## New guardrails, and judges The principle: **whatever you caught twice in review becomes a rule that runs by itself.** The agent produces mistakes faster than a person can filter them, so the conventions have to come out of people's heads and into the pipeline. > **Fig. 6** · Where a convention lives. Said once it is a comment. Said twice it is a pattern. The third time the pipeline says it – and what can't become a rule gets a judge with a rubric that is itself maintained. > > *Diagram:* Two rows of three. Top: a convention said once in review is a comment; said twice, it is the same comment on another PR; the third time the pipeline says it – an architecture test, a lint rule, a migration check or a size limit that runs by itself. Bottom: what can't become a rule, such as whether a change fits the domain, goes to an LLM judge with a narrow rubric of yes/no questions; when the judge is wrong twice, the rubric gets fixed, not the gate. Concretely. **Architecture rules as tests**: the domain module doesn't reference an adapter, and adapters don't call each other. These used to be review comments; now they are red in CI. ```csharp // Architecture rules as a test: the things I used to type into review comments, now red in CI. [Fact] public void Domain_does_not_depend_on_adapters() { var result = Types.InAssembly(typeof(Invoice).Assembly) .That().ResideInNamespace("Billing.Domain") .ShouldNot().HaveDependencyOnAny("Billing.Adapters", "Npgsql", "Azure.Storage.Blobs") .GetResult(); Assert.True(result.IsSuccessful, string.Join("\n", result.FailingTypeNames ?? [])); } [Fact] public void Adapters_do_not_call_each_other() { var result = Types.InAssembly(typeof(MinioFileStorage).Assembly) .That().ResideInNamespace("Billing.Adapters.Storage") .ShouldNot().HaveDependencyOn("Billing.Adapters.Email") .GetResult(); Assert.True(result.IsSuccessful); } ``` **Banned patterns**, such as creating your own `HttpClient` instead of using the shared one that carries the proxy, the timeout and the retry policy. The analyzer turns the habit into a build error, with the reason attached: BannedSymbols.txt ```ini ; BannedSymbols.txt – read by the BannedApiAnalyzers package. A hit is a build error (RS0030), not a comment. M:System.Net.Http.HttpClient.#ctor;Inject IHttpClientFactory – the shared client carries the proxy, the timeout and the retry policy M:System.DateTime.get_Now;Use TimeProvider – a test can't move the wall clock P:System.Threading.Tasks.Task`1.Result;Await it – .Result deadlocks under a synchronisation context ``` **A migration check** that catches the operations that lock a Postgres table for the duration of the build. And **a PR-size limit**: above a line count the pipeline is red, and there is no discussion. ```sql -- Caught twice in review. Now a CI step reads every new migration and fails on the first form. -- Holds a lock that blocks writes for the whole build. On a two-million-row table that is the night shift's problem. CREATE INDEX ix_invoices_customer ON invoices (customer_id); -- Builds without blocking writes. Can't run inside a transaction – the migration tool has to know that. CREATE INDEX CONCURRENTLY ix_invoices_customer ON invoices (customer_id); -- The seatbelt for everything else: wait two seconds for the lock, then fail. Don't queue behind a -- long report and block every write that arrives after you. SET lock_timeout = '2s'; ALTER TABLE invoices ADD COLUMN archived_at timestamptz; -- metadata only: fast, whatever the row count ``` ```yaml # Above the limit the pipeline is red – no discussion, no "just this once". Slice the task instead. - name: PR size run: | changed=$(gh pr view ${{ github.event.pull_request.number }} \ --json additions,deletions --jq '.additions + .deletions') if [ "$changed" -gt 400 ]; then echo "::error::$changed changed lines – the limit is 400. Split the PR." exit 1 fi env: GH_TOKEN: ${{ github.token }} ``` ### The judge, for what is left The LLM judge is for the remainder – what can't be poured into a rule, such as whether the change fits the domain. And with a **narrow rubric**: not "is this code good?", but concrete yes/no questions about the diff. Does the new adapter implement the existing interface? Is there state in here that can differ between nodes? The judge is non-deterministic, so it is a signal, not a gate. When it is wrong twice on the same question, the question gets fixed. ```yaml # The judge gets the diff and yes/no questions – never "is this code good?". It is non-deterministic, # so its answer is a comment on the PR, not a required check. Wrong twice on a question: fix the question. judge: output: pr-comment # a signal, never a gate questions: - id: port-shape ask: Does the new adapter implement the existing IFileStorage interface without adding members to it? - id: node-state ask: Does the change introduce state that can differ between nodes – static fields, in-memory caches, local files? - id: domain-fit ask: Does the change put domain rules into a controller or an adapter instead of the domain module? - id: migration-safety ask: Does any migration hold a lock for longer than a metadata-only change would? ``` > What you said twice in review, the third time the pipeline says. ## Is testing enough? No. Tests are needed, and more than before: automated tests, your own test environment, canary builds, A/B tests. But **a green test only tells you what the test checks.** The agent writes the tests for its own code, so it proves what it implemented – not what you wanted. The classic example is a list endpoint that queries the database once per item instead of once for all of them – the N+1. With fifty rows of test data it takes a tenth of a second, the test is green, and the UI works beautifully. In production, at two million rows, the database lies down. At three in the morning. ```csharp // Green with fifty rows of test data. With two million open invoices this is two million and one queries. var invoices = await db.Invoices.Where(i => i.Status == InvoiceStatus.Open).ToListAsync(ct); foreach (var invoice in invoices) invoice.CustomerName = (await db.Customers.FindAsync([invoice.CustomerId], ct))!.Name; // one query per row // One query, one page at a time, only the columns the endpoint returns. var page = await db.Invoices .Where(i => i.Status == InvoiceStatus.Open) .OrderBy(i => i.Id) .Select(i => new OpenInvoiceRow(i.Id, i.Customer.Name, i.Total)) .Skip(offset).Take(100) .ToListAsync(ct); ``` The same family: a parallel loop with no limit on how many tasks it starts, a query that loads the whole table into memory, the index nobody added, which you only feel at a hundred times the data. What works on the UI can have a bomb ticking behind it, set to go off at the worst possible time. > **Fig. 7** · One endpoint at two sizes, and the family it belongs to. Every one of these is green at the size of the test data and red at the size of production – and the test was written by the author of the bug. > > *Diagram:* Left, a bar chart of one endpoint at two sizes: with fifty rows of test data it answers in a tenth of a second and the test is green; with two million rows in production the database lies down at three in the morning. Right, the same family of faults that only show up at scale: a query inside a loop, a parallel loop with no limit, a whole table loaded into memory, a missing index. A green test proves the code works at the size of the test data. What to do about it: a staging environment with production-like data volume, a load test on the critical path, a canary that sends a small share of traffic to the new version with metrics on it. And a two-minute look at the diff, in broad strokes: is there a query inside a loop? An unbounded list? A call without a timeout or a cancellation token? Does the migration lock? You don't have to understand everything. Your eye has to catch on something. > A test tells you it works. Not whether it holds. ## Readability and maintainability Code is written for the other developers, not for the machine. The machine reads anything; your colleague doesn't, and in two months neither do you. No one-line miracles that take an afternoon to unpick later. The agent, left to itself, likes the clever abstraction, the generic solution, the layer on top of the layer. The measure is simple: **does the next developer understand it in half a minute?** Half a sentence that matters: readability doesn't require a slower tool, least of all in syntax. Sixty lines of nested SQL and the same query as named steps give the same plan in most cases – one of them can be read top down. ```sql -- The same query twice. The first one has to be read from the inside out. SELECT c.name, t.total FROM customers c JOIN (SELECT customer_id, sum(amount) AS total FROM invoices WHERE status = 'open' AND due_date < now() - interval '30 days' GROUP BY customer_id) t ON t.customer_id = c.id WHERE t.total > (SELECT avg(x.total) FROM (SELECT sum(amount) AS total FROM invoices WHERE status = 'open' GROUP BY customer_id) x); -- Named steps, read top down. The planner inlines these, so the plan is the same – only the reader is faster. WITH overdue AS ( SELECT customer_id, sum(amount) AS total FROM invoices WHERE status = 'open' AND due_date < now() - interval '30 days' GROUP BY customer_id ), typical AS ( SELECT avg(total) AS total FROM (SELECT sum(amount) AS total FROM invoices WHERE status = 'open' GROUP BY customer_id) per_customer ) SELECT c.name, overdue.total FROM overdue JOIN customers c ON c.id = overdue.customer_id WHERE overdue.total > (SELECT total FROM typical); ``` Maintainability is following the patterns laid down by people who understood the system and then enforced them. Delete the branch, that's wrong, again – until everyone has learned what is allowed and what isn't. There is no other way. In the AI era this is cheaper than it has ever been: a rejected agent branch is ten minutes of regeneration, not a wasted human day, and nobody's feelings are hurt. One thing to add, though: **don't only throw away the output. Fix the context.** If the agent got it wrong, either the pattern wasn't clear enough in the code, or a lint rule was missing. Regenerating from the same context gives you the same mistake. > **Fig. 8** · The loop around a rejected branch. The dashed arrow is the one everybody takes. The solid one is the one that makes the next branch better. > > *Diagram:* A loop. The context – the codebase, its patterns and lint rules – goes into the agent, the agent produces a branch in ten minutes, the branch is reviewed and deleted. Two arrows lead back: the obvious one returns to the agent to regenerate, and gives the same mistake from the same context; the important one returns to the context, to fix the pattern that wasn't clear or add the rule that was missing. > Deleting a branch used to be expensive. Now it's free – so there is no excuse for being lenient. ## Is the spec the new source of truth? I don't think so. **The running code is the truth**: it does what it does. The description next to it can be stale. Don't hang enormous wikis on the AI that have to be kept fresh, because the agent follows a stale document exactly as faithfully as a fresh one – and that is worse than having nothing. What works: the code documents the *what* – names, structure, tests as runnable examples – and the documentation supplies the *why*. A short decision record, for instance: why do we sync the settings from the database with a background job, and why don't we cache them in memory? That is the one thing you can't read out of the code, and it is exactly the thing the review bot from earlier didn't know. docs/adr/0007-settings-sync.md ```md # ADR 0007 – settings are synced from the database by a background job, not cached in memory **Status:** accepted · 2025-03 **Context.** The app runs on several nodes. Settings are edited in the admin UI on one node and have to apply on all of them within seconds. **Decision.** A background job on every node reloads the settings from the database every five seconds. No per-request cache, no in-memory expiry. **Why not an in-memory cache.** Each node would answer from its own copy for the lifetime of the cache entry. Two nodes, two states – and a customer sees the old value from one of them. **Consequences.** One cheap query per node every five seconds. Any "cache the settings" suggestion – from a reviewer or from a bot – is answered by this file. ``` > **Fig. 9** · The code says what, the docs say why, and the big wiki is the trap in the middle: nobody keeps it fresh, and the agent can't tell. > > *Diagram:* Two columns and a warning. Left, what the code says: names and structure, tests as runnable examples, and the running system, which is the truth. Right, what the docs say: an ADR explaining why the obvious choice wasn't taken, a small diagram kept beside the code, a short intent in the PR description. Underneath, the trap: a big wiki nobody keeps fresh, which the agent follows exactly as faithfully as a fresh one. The architecture diagram matters, but keep it small, keep it in the repository next to the code, and update it in the same PR. A spec for one task is fine: a short statement of intent in the PR description, short-lived. When the spec and the code disagree, the code wins – and the disagreement is a signal that either the spec needs updating or there is a bug. > The code says what it does. The docs say why. ## Who writes the spec: the decision lives where the code is A precise specification never arrives from the customer. It is assembled from words and follow-up questions: what do they actually want, and how does a blurry picture become a production-ready feature – with limits, with fallbacks, and with the things the customer never thought of but without which the feature isn't complete. This is also where it gets decided whether the thing is generic and useful to every customer, or built for one customer behind a feature flag. The person who can decide that well is the one who knows the product's architecture, knows what the change touches, knows the use cases, and sees from the codebase who is working on what – not from a "I'm on this" status in a call. That is the technical lead, and it stays the technical lead even when they run a team. What a product needs is strong technical people, and above them a leader who is also technical, is inside the domain and the product's direction, and decides when a decision is needed. A layer in between doesn't take work off that chain. It adds a round trip. If the go-between doesn't get proper information, doesn't pass proper information on to the technical people, or doesn't understand the request, the technical person ends up in the call anyway – or writes the answer to the customer themselves. Meanwhile the request has been distorted twice, on the way in and on the way out. The commercial context sits with sales, and sales gets a faster and more precise answer from the technical lead directly, at scale too, than through anyone in the middle. > **Fig. 10** · Two chains from the customer's words to the team. In the first, the person in the middle forwards without deciding, and the decision gets made two boxes later anyway. In the second the technical lead talks to the customer directly, and sales feeds the commercial context to both – there is nothing in the middle to distort. > > *Diagram:* Two chains from a customer to a development team. Through a relay: the customer's words go to sales, then to a relay who forwards the request without deciding on it, then to the technical lead, who decides anyway and often joins the call anyway; the request is distorted twice, on the way in and on the way out. Direct: the technical lead – who knows the code and decides – talks to the customer directly, both ways, and works with a small, strong team; sales sits beside that link and feeds the commercial context to both the customer and the technical lead. The calls get faster too, and that is not a side effect – it is most of the point. With the technical lead on the call, the customer's question is answered on the call: whether it can be done, what it touches, roughly how much work it is, and what to do instead when the answer is no. Through a relay the same question turns into "I'll check with the team", a note that loses half the context, and a second meeting a week later to deliver an answer that may already be wrong. One call with the person who decides beats three calls with the person who forwards. > If the technical person has to join the call in the end anyway, the layer in between didn't take work off them. It added a round trip and two distortions. ### The product role doesn't disappear. It gets distributed. Saying "no product manager" is not saying "no product work". The work is real; the question is who holds each piece of it, and the answer is that it splits along the lines where the knowledge already is: - **Discovery and scope** – the technical lead, who knows the product and the customer. - **Pricing, packaging, go-to-market** – sales. - **Contracts and SLAs** – legal. - **Scheduling and deadlines** – delivery, worked out together with the technical lead. > **Fig. 11** · The product role, taken apart. Each piece lands with the people who already hold the knowledge it needs; the schedule is the one piece with two owners, and the condition that holds it all together sits under the boxes. > > *Diagram:* The product role taken apart into four pieces, each going to the people who already hold the knowledge it needs: discovery and scope to the technical lead, pricing and go-to-market to sales, contracts and SLAs to legal, schedule and deadlines to delivery – set together with the technical lead. Under the pieces stands the one condition that makes it work: the technical lead can say no to sales, and shapes the schedule with delivery instead of receiving it. Underneath all of it, people management – careers, feedback, hiring, conflict – stays with a person, and matters more than before. There is one condition, and it is not a small one: **the technical lead needs both the right and the spine to say no to sales – and the same weight with delivery.** If a big deal lets sales overrule them, they aren't deciding about the product any more; the pipeline is. If a date set without them decides the scope, the calendar is. The schedule gets worked out together with delivery, with the technical lead at the table, not handed to the team afterwards. A technical lead who runs discovery but can't refuse a feature or move a date is the relay again, with a better title. ### At scale: staff and principal engineers With twenty teams you still need coordination; I am not arguing against that. I am arguing that the coordinator has to understand what they are coordinating. That is what staff and principal engineers are for: technical people who make decisions across teams, and who can read the code the decision lands in. Information isn't lost because of a title. It is lost with every step between the decision and the code, and a decision made far from the code is a bad decision whether a manager makes it or an architect does. ### What AI changes I worked this way before AI. In the AI era I think it is the only good direction. The reason a go-between used to be needed between the technical person and the customer was that the technical person's time went into writing code. Now the agent writes most of the code, and **the freed-up time belongs where the decisions are made: next to the customer.** The technical side can't outsource the calls to a management layer because it doesn't feel like taking them. Then it decides on second-hand information, and that is where the bad features come from. Most of what the coordinating layer lived on is also exactly what an agent does well now: | What the layer did | Who does it now | | --- | --- | | Status – walking around asking where things stand | An agent summarises the PRs, the tickets and the commits. Nobody has to collect it by hand. | | Translation – between the customer and the engineers | The technical person talks to the customer, with the time the agent freed up. | | Handoffs – between the teams a feature crosses | A smaller team ships the same amount, so there are fewer dependencies to manage in the first place. | | People – career, feedback, hiring, conflict, burnout | Still a person, and more important than before. An agent doesn’t do this, and a technical lead can’t do it well on the side. | That last row is the one I want to be clear about. **People management matters more in the AI era, not less.** Careers, feedback, hiring, conflicts, burnout – the AI does none of it, and a technical lead who is also running discovery can't do it well as a sideline. What becomes redundant is the other kind of manager: the one whose job was moving information. > In the AI era the people-manager gets more important and the relay-manager less. People need leading. Information only needs delivering. ### What I mean when I say the role isn't needed I have said before that product and project managers aren't needed, and the sentence travels further than what I mean by it. So let me be precise about the claim. What I am against is **the relay**: a position whose content is passing a request from one side to the other without understanding it and without deciding on it. Plenty of people with those titles do exactly the work I described above – they know the domain, they know what the change touches, they sit with the customer and they decide. Then they are the technical lead by another name, and I have no quarrel with the name. The thing to name is the distance from the code, not the title. I'll also be honest about where this comes from. It is experience, not a law: I have not yet worked with a product or project manager where understanding what was going on wasn't extra work for everyone. I work on technical products, where the customer is technical too, and in that setting I haven't seen the layer in between add anything. A consumer product with a research-driven roadmap is a different setting, and I am not writing about it. It is also an opinion, not a measurement: the flattening of middle management is happening, but how much of it is AI and how much is cost-cutting can't be separated yet. And I know this position will sting for people in those roles, because it questions whether the role is needed. That is the price of holding the opinion, and I am not going to soften it below what I actually think. What I can do is say exactly what it is – a claim about distance from the code and the customer, and about who decides – so that it can be argued with on those terms. > My problem isn't with managers. It is with the relay: whoever forwards a request without understanding it, and doesn't decide on it either. ## Letting go of control, one rung at a time A ladder, not a switch. 1. The agent proposes, a person writes. 2. The agent writes, a person reviews everything. 3. In a narrow, low-risk class – dependency updates, added tests, mechanical refactors with good coverage – lighter review, behind deterministic gates. 4. In the narrowest class, auto-merge, with instant rollback. > **Fig. 12** · Four rungs, and a class of change climbs them one at a time. The column on the right is what it takes to move up, and the box at the bottom of it is what never moves at all. > > *Diagram:* A ladder of four rungs, bottom to top. One: the agent proposes, a person writes. Two: the agent writes, a person reviews everything. Three: light review in a narrow, low-risk class – dependency bumps, added tests, mechanical refactors – with deterministic gates. Four: auto-merge in the narrowest class, with instant rollback. A class moves up a rung on measured evidence: revert rate, escaped defects, a small and reversible blast radius. Some classes never leave the bottom: migrations, permissions, money, deletes. The condition for the next rung isn't a feeling. It is a **measured track record**: revert rate, defects that escaped to production, and the scope – a small blast radius, a change you can take back. What you never let go of: migrations, permissions, money movement, data deletion. One class, one step, measure, then the next. Patiently. There is a sensible asymmetry inside this. Where a change isn't core business logic, and a mistake doesn't cost you face with a customer, the PR can be accepted with a lighter look: read the PR message, see which files changed, and if you know the domain and the product you know roughly that nothing critical can be wrong there. The business logic itself – the part that carries the real value – gets split into small PRs and read properly. The rung belongs to the class of change, never to the repository as a whole. review-policy.yaml ```yaml # Which rung each class of change stands on, and what it takes to move up. The rung is per class, never per repo. classes: dependency-bump: { rung: 4, gates: [build, tests, security-scan], rollback: automatic } add-tests-only: { rung: 3, review: light } mechanical-refactor: { rung: 3, review: light, requires: "coverage >= 80% on the touched files" } feature-non-core: { rung: 2, review: full } business-logic: { rung: 2, review: full, max-lines: 200 } never-delegated: # rung 1 for good – a person writes or owns every line - migrations - permissions - money-movement - data-deletion promotion: # measured over a quarter, per class, before a class moves up revert-rate: "< 2%" escaped-defects: 0 blast-radius: small, and reversible ``` > Don't ask whether you trust the agent. Ask: for which kind of change, at what stakes, and with what evidence. ## How do we know it is better? Measure, against a baseline. Before you introduce anything, know today's numbers. The four DORA metrics are a good start, and they are four simple questions: how long from idea to production (lead time), how often do we deploy, what share of changes causes a failure, and how long until the system is restored. Next to them: revert rate, defects that escaped to production, PR size and review time, and rework – code that gets rewritten within two or three weeks. ```sql -- The baseline, before anything changes. Lead time, change failure rate and time to restore, per quarter, -- from tables you already have: pull requests, deployments, and incidents linked to the deployment that caused them. SELECT date_trunc('quarter', d.deployed_at) AS quarter, percentile_cont(0.5) WITHIN GROUP (ORDER BY d.deployed_at - pr.first_commit_at) AS lead_time_p50, count(DISTINCT d.id) AS deployments, count(DISTINCT i.caused_by_deployment_id)::numeric / count(DISTINCT d.id) AS change_failure_rate, avg(i.resolved_at - i.opened_at) AS time_to_restore FROM deployments d JOIN pull_requests pr ON pr.id = d.pull_request_id LEFT JOIN incidents i ON i.caused_by_deployment_id = d.id GROUP BY 1 ORDER BY 1; -- Not in this query, on purpose: lines written, number of PRs, the share of code an agent wrote. ``` | Measure | Don’t measure | | --- | --- | | Lead time from first commit to production, and deployment frequency | Lines of code written | | Change failure rate, and time to restore | Number of pull requests | | Revert rate and defects that escaped to production | The share of code written by AI | | PR size and review time – the cost side of the ledger | How fast it feels | | Rework: code rewritten within two or three weeks | Anything measured over one or two sprints | The right-hand column measures activity, not quality. And don't trust the feeling either: with AI it is easy to feel faster, because the writing really is fast – and the review, the fixing and the debugging slip exactly where you aren't looking. One or two sprints is too short; it takes a quarter to see. Finally, the other side of the cost counts: **what faster writing gains, review time can easily eat.** > If you only feel faster, you don't know yet. ## Who still understands the system? This is where the arc comes together. The basic knowledge has to be there, so that the AI's beautiful explanations don't lead you by the nose: **a convincing explanation is not evidence.** Yes, it is slower than not reviewing. But responsibility can't be delegated, and laziness collects its debt with interest. You don't need every detail, but you need the broad strokes, so that at review time it clicks: yes, this is how it works here. Two quick tests. Can you draw the architecture without opening the code? Can you say what happens when a node dies in the middle of a sync? If not, the system isn't yours any more. It is the agent's – and the agent forgets it by tomorrow. Call it **comprehension debt**: the code grows faster than the understanding of it, and someone will pay the difference. > **Fig. 13** · Two lines that used to grow together. The shaded gap is the debt, and the two questions in the corner tell you how wide it is for you. > > *Diagram:* Two lines over time. The amount of code rises steeply; the team's understanding of it rises slowly. The widening gap between them is comprehension debt, and like every debt someone pays it – usually at the worst time. Two tests for where you stand: can you draw the architecture without opening the code, and can you say what happens when a node dies in the middle of a sync? In practice: module owners, small PRs, the *why* written by a person, onboarding as a test – is the new developer in the picture after a week? – and writing code by hand now and then, so the muscle stays. For juniors this is especially sharp: whoever never fought a bug to the end doesn't build a mental model, and that is hard to make up later. I wrote about that loss in [the engineering mindset post](https://laszlonemes.com/blog/engineering-mindset#junior-to-senior); here it is the whole team's problem, not only the junior's. > Machines can write the code. Understanding and responsibility can't be outsourced. ## Objections, and the answers | They say | You say | | --- | --- | | "The agent writes the tests too, so the review is redundant." | The agent’s tests prove what it implemented, not what you wanted – and they pass at the size of the test data. Review is for the domain fit and for the team learning the edge cases. Neither is in a test. | | "The review bot is more thorough than a tired human." | More thorough, and without the context. Its advice is generally valid and locally wrong, and the fix-it button makes the locally wrong part cheap to apply. Let it flag. Let a person decide. | | "A proper spec up front would fix all of this." | The running code is the truth; a spec is a description that can go stale, and the agent follows a stale one just as faithfully. Keep the why in a short decision record and the intent in the PR. Keep the what in the code. | | "So product managers are useless?" | No. The relay is: a position that forwards requests without understanding or deciding. The product work gets distributed – discovery to the technical lead, pricing to sales, contracts to legal, dates to delivery – and the people-manager matters more than before, not less. | | "Twenty teams can’t coordinate themselves." | They can’t, and nobody said so. The coordinator has to understand what they coordinate – that is what staff and principal engineers are for. Information is lost per step between the decision and the code, not per title. | | "A PR-size limit is bureaucracy." | It is the one rule that makes every other one possible. A reviewer can read four hundred lines. Nobody can read two thousand, and an agent can slice a task any way you tell it to. | | "We measured: forty percent more code shipped." | That measures activity. Measure lead time, change failure rate, reverts, escaped defects and rework over a quarter – and put review time on the cost side, because that is where the gain goes to die. | | "Just never let the agent merge." | That is rung two, forever, and it leaves the cheap wins on the table. Climb per class of change, on evidence, and keep migrations, permissions, money and deletes on the bottom rung for good. | ## Takeaways 1. **Quality didn't change – the weak link moved.** Writing got cheap; review, understanding and responsibility cost what they did. 2. **The codebase is the prompt.** One port with adapters gets copied; five controllers five ways gets a sixth. Fix a wrong pattern before you let an agent scale it. 3. **Small PRs, or no review at all.** A person can read four hundred lines. Slice the task before the agent starts. 4. **Review is knowledge transfer**, not a bug filter. If you only approve, the edge case stays with the agent. 5. **Deterministic gates underneath, the agent's opinion in the middle, a person at the end.** The agent flags; the fix-it button is a human decision. 6. **Caught twice in review becomes a rule** – an architecture test, a banned API, a migration check, a size limit. What can't be a rule gets a judge with a narrow rubric, as a signal. 7. **A green test proves the size of the test data.** Staging with real volume, a load test on the critical path, a canary – and a two-minute look for the query in the loop. 8. **Delete the branch, fix the context.** Regenerating from the same context gives the same mistake. 9. **The code says what, the docs say why.** A short decision record beats a big wiki the agent trusts blindly. 10. **The spec is assembled by whoever understands and decides** – close to the code and the customer at once. A relay in between adds a round trip and two distortions; the product work gets distributed, and the people-manager matters more. 11. **Let go per class of change, on evidence.** A ladder, not a switch, and some classes never leave the bottom rung. 12. **Measure against a baseline, over a quarter.** DORA, reverts, escaped defects, rework – and review time on the cost side. 13. **Someone has to understand the system in broad strokes.** Comprehension debt is the one debt that compounds silently. Back to the thesis: the chain is the same as it ever was, only now you can see where it snaps. A good codebase the agent follows. A small PR a person can take in. Deterministic gates that don't ask for permission. Trust that is measured and given one rung at a time. And a person who understands, in broad strokes, what is running. Writing code got cheap. Understanding didn't. That is where to spend. ## Sources - [DORA's four key metrics](https://dora.dev/guides/dora-metrics-four-keys/) – lead time, deployment frequency, change failure rate, time to restore. - [Building indexes concurrently](https://www.postgresql.org/docs/current/sql-createindex.html#SQL-CREATEINDEX-CONCURRENTLY) and [lock_timeout](https://www.postgresql.org/docs/current/runtime-config-client.html#GUC-LOCK-TIMEOUT) – the Postgres documentation on the two migration guardrails. - [WITH queries](https://www.postgresql.org/docs/current/queries-with.html) – when Postgres inlines a CTE and when it materialises one. - [BannedApiAnalyzers](https://github.com/dotnet/roslyn-analyzers/blob/main/src/Microsoft.CodeAnalysis.BannedApiAnalyzers/BannedApiAnalyzers.Help.md) – the `BannedSymbols.txt` format and rule `RS0030`. - [NetArchTest](https://github.com/BenMorris/NetArchTest) – architecture rules as unit tests in .NET. - [Efficient querying in EF Core](https://learn.microsoft.com/en-us/ef/core/performance/efficient-querying) – projections, paging and the N+1 problem. - [Architecture decision records](https://adr.github.io/) – the short form of writing down the why. - [.NET releases and support](https://learn.microsoft.com/en-us/dotnet/core/releases-and-support) – the LTS cadence behind the upgrade example. --- # Build vs. buy, reloaded – and platform risk > AI made the first version cheap, not the software: operations, integration, change and responsibility cost what they did. What you really buy (four layers, and you inherit the fourth), the seam test a demo never runs, the 90/10 trap, what a vendor learns from forty edge cases, platform risk from Exchange Basic auth to model deprecations – and why the line between build and buy is whether you can get out. Source: https://laszlonemes.com/blog/build-vs-buy · Published: 2026-10-01 · Author: László Nemes October 1, 2026 · 25 min read `Architecture` `AI engineering` `Integration` `.NET` Build or buy is one of the oldest questions in software, and every few years something arrives that is supposed to settle it. This time it is AI: if an agent can write the thing in a week, why would anyone pay a licence for it? I hear the question from both directions – from teams that want to replace a vendor with a weekend of prompting, and from customers of my own product who wonder whether they still need it. The honest answer is that the question didn't change. One input did. The first working version got cheap – weeks became days. But the cost of software was never the first working version. It was everything after it. > **The thesis:** build vs. buy didn't change – only the v1 got cheap. Operations, integration, change and responsibility cost exactly what they did five years ago. What did change is a risk that used to be quieter: the platform you build on is also your competitor. It sees the usage, it sees what works, and building it in costs it close to nothing. > **Fig. 1** · Five years of owning software. The bar that AI shortened is the first one. The other four are where the money always went, and they are exactly as long as they were. > > *Diagram:* Five years of owning a piece of software, as two bars. Before AI, the first working version was roughly a fifth of the total. The rest was integration – login, file storage, backup; operations – on-call and incidents; maintenance – upgrades and security patches; and handover – documentation and the next person who has to understand it. With AI the first version shrinks to a sliver, and every other segment is exactly as long as it was. ## What you actually buy When you buy software, you buy four separate things, and each needs its own price. There is the **hardware and infrastructure**. There is **the software** itself. There is the **service**: support, an SLA, operations, scaling. And there is the fourth, which the offer doesn't mention at all: **integration**. The vendor sells the first three. You inherit the fourth, and it never finishes. > **Fig. 2** · Three layers on the invoice, one in your backlog. Identity mapping, storage, secrets, monitoring, backup and the runbook are work you do whether you build or buy – buying only decides who owns the rest. > > *Diagram:* What you buy when you buy software, as four layers. Hardware and infrastructure, the software itself, and the service – support, SLA, operations and scaling – are what the vendor sells and prices in the offer. The fourth layer, integration – identity, storage, secrets, monitoring, backup and the runbook – is not in the offer. You inherit it, and it never finishes. That is why I say **buy is build too**. You just build the wrong half: the integration, the monitoring, the runbook, the permission mapping, the backup strategy – and then you pay rent on top. In an enterprise, a "buy" is typically 20–40% of the engineering a build would have taken, plus a fee that never stops. So the real comparison isn't build versus buy. It is **build-and-own versus integrate-and-rent**, and both columns have engineering in them. ### Where it runs is part of what you buy The deployment model isn't a hosting detail. It decides who patches, who sees the logs and who answers the data-protection questions: | Model | What you get | What you pay | | --- | --- | --- | | On-premises – in your infrastructure | You patch, you see the logs, the data never leaves. Responsibility for it is yours too. | Slower support: they can’t see inside your environment, so every ticket starts with "send us the logs". | | SaaS – in theirs | Features arrive fast; nobody on your side runs anything. | The data leaves: a DPA, a sub-processor list, a region – and the least comfortable question of all, the exit. | | BYOC / private cloud | Their software, your cloud account. Often the only option legal will sign. | Both sets of downsides at once: you run it, and they still need a way in. | The licence model is a technical question as well, and it gets reviewed by the wrong people. Per-seat pricing is fine for forty analysts. Add five thousand read-only viewers who open a report twice a month, and the business case is dead – however good the software is. Whoever designs the solution should read the price list with the same care as the API documentation, because the price list decides the architecture as often as the API does. > Buy is build. You build the wrong half – and pay rent on top. ## The seam test In a demo you see the feature. What you are going to inherit is the **seams**: every place the product touches something of yours. A demo touches none of them, because it runs in the vendor's environment, against the vendor's storage, with the vendor's identity provider. Each seam has its own concrete, reproducible way of falling apart, and it is worth going through them in order. > **Fig. 3** · The demo lives in the middle box. The installation lives in the eight around it. Ask about each one before signing; the exit, which everybody asks about last, is the one to ask about first. > > *Diagram:* The feature the demo shows sits in the middle. Around it are the eight seams you inherit: file storage, the database, secrets, auth, network and licence checks, deployment, observability, and the exit. The demo touches none of them; each one has its own way of failing on install day. ### File storage – the “S3-compatible” trap The datasheet says S3-compatible, and it is – on AWS. Their client uses virtual-hosted addressing, where the bucket is part of the hostname. Your MinIO runs on-premises under an internal hostname, behind a port, and needs path-style addressing, where the bucket is part of the path. The demo on AWS was perfect; your first upload fails. It isn't a missing feature. It is one flag in the client – it just has to come out before you sign, not after. ```csharp // "S3-compatible" – and it is, on AWS. Their client builds https://.s3.amazonaws.com/. // Your MinIO lives at https://minio.internal:9000 and there is no wildcard DNS for bucket hostnames. var s3 = new AmazonS3Client(credentials, new AmazonS3Config { ServiceURL = "https://minio.internal:9000", ForcePathStyle = true, // https://minio.internal:9000// – the whole difference AuthenticationRegion = "us-east-1", }); ``` ### SharePoint and throttling The customer uses SharePoint as a file store, and Microsoft throttles per tenant. If the product doesn't handle a `429` and its `Retry-After` header properly, everything works with fifty files and the job dies at night with fifty thousand, and nobody understands why. Worse, a client that retries on its own schedule extends the throttle it is fighting. You will never see this in a demo. You only see it under load. ```csharp // SharePoint throttles per tenant and says for how long. A sync that ignores the answer works with // 50 files and dies at night with 50,000 – after hammering the tenant into a longer throttle. async Task SendRespectingThrottle(Func build, CancellationToken ct) { for (var attempt = 1; ; attempt++) { var response = await http.SendAsync(build(), ct); // a fresh request each time: upload bodies are streams if (response.StatusCode is not (HttpStatusCode.TooManyRequests or HttpStatusCode.ServiceUnavailable) || attempt == MaxAttempts) return response; var wait = RetryDelay(response, attempt); response.Dispose(); await Task.Delay(wait, ct); } } static TimeSpan RetryDelay(HttpResponseMessage response, int attempt) => response.Headers.RetryAfter switch { { Delta: { } delta } => delta, // what the server asked for { Date: { } until } when until > DateTimeOffset.UtcNow => until - DateTimeOffset.UtcNow, _ => TimeSpan.FromSeconds(Math.Min(60, Math.Pow(2, attempt))), // no header: back off anyway }; ``` ### The database – version and privileges "Postgres support" often means, in practice: tested on Postgres 14, in the `public` schema, by a user allowed to create extensions. You run a managed database where nobody is a superuser, and the installer's first migration asks for one. That comes out on install day, not in the proof of concept. And right behind it is the next question: is there a way back from the migrations, or only forward? ```sql -- 001_init.sql, as shipped. "Postgres supported" meant: tested on 14, in public, connected as postgres. CREATE EXTENSION IF NOT EXISTS pg_stat_statements; -- for their diagnostics page; not a trusted extension CREATE EVENT TRIGGER audit_ddl ON ddl_command_end -- superuser only, on every Postgres there is EXECUTE FUNCTION public.log_ddl(); CREATE TABLE public.documents ( -- your policy: one schema per application, nothing in public id uuid PRIMARY KEY, body jsonb NOT NULL ); -- 002 … 041: forward only. Nobody asked for a down step until install day. ``` ### Secrets – Vault or plaintext Your policy says service credentials don't sit anywhere in plain text. The product wants the connection string in a YAML file. That is not "adding a field" – it is their whole configuration and secret-handling layer. If it can't do it, you write the sidecar that injects the secret, which means you are building again, only now around the thing you bought. And there is a subtler trap behind the obvious one: ```yaml # What their product wants: the password, in a file, forever. database: connectionString: "Host=pg.internal;Database=docs;Username=docs_app;Password=S3cret!" --- # What your policy allows: Vault issues a short-lived database user per pod, and a sidecar writes it to a file. metadata: annotations: vault.hashicorp.com/agent-inject: "true" vault.hashicorp.com/role: "docs-app" vault.hashicorp.com/agent-inject-secret-db: "database/creds/docs-app" vault.hashicorp.com/agent-inject-template-db: | {{- with secret "database/creds/docs-app" -}} Host=pg.internal;Database=docs;Username={{ .Data.username }};Password={{ .Data.password }} {{- end }} # The credential expires with its lease. A product that reads the connection string once, at start-up, # works for exactly one TTL. That is the question to ask – not "do you support Vault?" ``` ### Auth – where most of the hidden work lives OIDC, or only SAML? Is there group-claim mapping, or do four hundred users get entered by hand? Is there SCIM provisioning? And the most important one: what does it do for a *service* account – key-pair, mTLS, or only a username and a password? In the environment I work in, Snowflake sits behind a proxy, and the service users authenticate with RSA key pairs, because password sign-in is being phased out. That isn't an exotic requirement. It is what will be mandatory everywhere next year. ```sql -- A service user with no password at all, and two key slots, so rotation needs no downtime. CREATE USER svc_reporting TYPE = SERVICE DEFAULT_ROLE = reporting_reader; ALTER USER svc_reporting SET RSA_PUBLIC_KEY = 'MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8A...'; -- Rotation: the new key goes into the second slot, the clients switch, then the first slot is cleared. ALTER USER svc_reporting SET RSA_PUBLIC_KEY_2 = 'MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8B...'; ALTER USER svc_reporting UNSET RSA_PUBLIC_KEY; ``` ```csharp // .NET connector: key-pair (JWT) auth through the corporate proxy. There is no password in the string // because there is no password. The key and its passphrase come from the secret store, not from config. var cs = "account=myorg-myaccount;user=svc_reporting;authenticator=snowflake_jwt;" + $"private_key_file={keyPath};private_key_pwd={keyPassphrase};" + "useProxy=true;proxyHost=proxy.internal;proxyPort=8080;" + "warehouse=REPORTING_WH;db=ANALYTICS;role=REPORTING_READER"; await using var connection = new SnowflakeDbConnection(cs); await connection.OpenAsync(ct); ``` ### Network, licence, deployment, observability - **Network and licence.** Does the product need outbound traffic for telemetry or a licence check? What happens air-gapped? And the least pleasant one: when it can't reach the licence server, is there a grace period, or a hard failure at two in the morning? - **Deployment.** If the vendor ships only a Helm chart, the vendor has also decided that you run Kubernetes. Now you operate a cluster for one application, with everything that means in operations and skills. That price is nowhere in the offer. - **Observability.** Structured logs, a health endpoint, metrics – or only their own dashboard, with no API, that you can't wire into your monitoring? ### The exit – asked last, should be asked first Is there a documented export, with the schema, or a pile of CSV files? The best question to put to a reference customer is not who moved *to* them. It is who moved **away from them**, and how long it took. And the export gets tested in the first week of onboarding, not on the day of the breakup – I come back to this below. > You buy the feature. You inherit the seams. ## The 90/10 trap This sentence is spoken in every one of these projects: "the vendor covers 90% of what we need, the remaining 10% is custom development". There are two problems with it. The first: **the 90% is always self-reported, and it always comes from the demo.** Ask who measured it, when, on what data, and in whose environment. In my experience, after an integration test the number lands somewhere around 60–70. Not because the vendor lied, but because the difference is in the seams, and the demo doesn't touch the seams. > **Fig. 4** · The same product, measured twice. The first bar is honest about the features. The second is honest about your environment, and the gap between them is the previous section. > > *Diagram:* Two bars of how much of the requirement a product covers. The vendor's figure, from the demo: ninety percent fit and ten percent custom development. The same product after an integration test in your own environment: sixty to seventy percent fit, and the gap is in the seams the demo never touched. The second: **the remaining 10% is never evenly spread.** It sits exactly where you differ from everyone else – which is exactly where it is most expensive to touch. There are three ways to close it, and each one costs something: - **Fork it, or customise it deeply.** From now on the merge conflicts are yours, there is no upgrade path, and the support contract became paper. The first security patch tells you what you signed up for. - **Wrap it.** You run two systems plus the integration between them. The bugs will live in the gap, where nobody is on call and each side points at the other. - **Shrink the requirement.** The least glamorous option and the most often right. A good part of the 10% isn't a business need but process inheritance. Ask "why do we do it this way?" and it often turns out that nobody will defend it. And the main counter-argument when someone wants to build because of the 10%: **if you build your own because of the 10%, you will also run the 90%.** The licence fee isn't to be compared with the 10%. It is to be compared with five years of owning the whole system. > The 90% always comes from the demo. After an integration test it's 60–70. ## AI made the v1 cheap – only the v1 The point in one sentence: **AI doesn't make software cheap. It makes the v1 cheap** – and the v1 is roughly a fifth of the total cost. A vibe-coded app is genuinely good for a proof of concept. I'd say that is its best use today: in two days you put together a throwaway prototype that settles one question. But it isn't ready for production, and not because the code is bad. It is because **the information and the experience are missing behind it.** For a product to work properly you need customer feedback: what users actually do with it, which configurations really occur, which error message makes them call, and which feature a hundred people asked for and three use. That information can't be generated. You get it by serving customers, for years. What AI didn't make cheaper: on-call, security patches for every vulnerability found in your dependencies, migrations, the upgrade path, backward compatibility, integration testing against a real identity provider and a real database rather than a mock, documentation for the next person – and the next person. Then there is the **bus factor**: whoever put it together in a week moves on within a year, and the repository has no owner. I wrote about the same gap from the engineer's side in [Engineering mindset in the age of AI](https://laszlonemes.com/blog/engineering-mindset); here it shows up on the balance sheet. ### Where AI actually helps It is worth separating, because it isn't the same across the three phases: > **Fig. 5** · AI's help is front-loaded: strongest when you are deciding, decent while you build the first version, thin once you operate. Ownership cost runs the other way. > > *Diagram:* How much AI helps in each phase of a build-or-buy decision. Planning and evaluation: the most – reading vendor documentation, building the integration test matrix, writing the procurement questions. Building: strong for the first version, weak by the third year – it writes the adapter fast but doesn't know the edge cases. Operating: the least – log triage and runbook drafts, but responsibility can't be delegated. - **Planning and evaluation – the strongest.** Digesting vendor documentation, putting together an integration test matrix, writing the question list for procurement. - **Building – strong for the v1, weak by year three.** It writes the adapter fast. It doesn't know the edge cases. - **Operating – the least.** Log triage and a runbook draft, fine. But responsibility can't be delegated. ### The best use: disprove the demo Two consequences that aren't obvious. The first: **the best build-vs-buy use of AI isn't writing the build. It is disproving the demo.** In two days you write a throwaway PoC that wires the product to your own Vault, your own Snowflake, your own MinIO, and you see whether what was promised actually works. That used to take three weeks, which is why nobody did it. Now there is no excuse. ```csharp // The demo-disproving PoC: two days with an agent, deleted afterwards. Every test is one seam, // run against the real thing in our network – not their sandbox, not a mock. public class VendorSeamTests(VendorFixture vendor, OurInfra our) : IClassFixture, IClassFixture { [Fact] public Task Writes_to_our_MinIO_path_style() => vendor.Files.UploadAsync(our.MinioBucket, "probe.txt", "hello"); [Fact] public async Task Survives_a_Vault_lease_expiring() { await vendor.ConnectAsync(our.VaultRoleWithFiveMinuteTtl); await Task.Delay(TimeSpan.FromMinutes(6)); Assert.True(await vendor.HealthAsync()); // the kind that reads YAML once fails here } [Fact] public Task Reads_Snowflake_with_a_key_pair_through_the_proxy() => vendor.Sources.TestConnectionAsync(our.SnowflakeKeyPair); [Fact] public async Task Syncs_50_000_SharePoint_files_without_losing_one() { var run = await vendor.Sync.RunAsync(our.SharePointLoadTestSite); Assert.Equal(0, run.Failed); // the 429 lives here, not in a 50-file demo } } ``` The second: **AI makes migration cheaper too.** Exporters, transformers and schema mappings are much faster to write now. If leaving is cheaper, lock-in is worth less, and with it the vendor's pricing power drops. AI doesn't only strengthen the build side – on the buy side it improves your negotiating position. A prediction to finish with. The wall for internal tools put together with AI in 2025–26 will arrive in 2027–28. Not as a spectacular collapse, but as a slow maintenance cliff: repositories without owners, dependencies nobody updated, and the sentence "someone built this back then, but they're not here any more". > Perfect for a PoC. Not for production – not because of the code, but because there's no customer behind it. ## The same thing from the vendor’s side I sit on both sides. One of my own products is used by a lot of companies, and the customer runs it in their own infrastructure. Everything from the previous sections comes up there, and it is a mature product, so the list is long: four internal databases – SQLite, SQL Server, Postgres and MySQL; seven file stores – local disk, Azure Blob, S3, MinIO, Google Cloud Storage, Box and SharePoint; login with local users, Active Directory, or OAuth with Google, Okta, Microsoft or any OIDC provider; and eleven data-source connectors, with password, OAuth or RSA key-pair authentication. From that side the picture looks different. ### Why most vendors can't be generic Because every adapter multiplies the test matrix and the support surface. Four databases, seven file stores and three kinds of login are eighty-four combinations before a single data source is connected. I test every one of them – and that is exactly the point. "Supported" has to mean tested; a combination that only exists on the datasheet is a promise nobody has checked. But the full matrix costs CI hours on every release, real SharePoint, Box and Microsoft test tenants because those don't come in a container, and the knowledge of what is worth asserting inside each cell. A vendor that sells to one kind of customer doesn't pay that bill, and has no reason to. > **Fig. 6** · Twenty-eight cells per login type, eighty-four in total, every one of them a CI job. The next adapter isn't one more piece of code – it is a whole new column of jobs, on every release from then on. > > *Diagram:* A grid of four internal databases by seven file stores – local disk, Azure Blob, S3, MinIO, Google Cloud Storage, Box and SharePoint – twenty-eight combinations, times three login types: eighty-four CI jobs, all of them run on every release. An eighth file store would not be one more piece of code: it adds a whole column, four databases times three login types, twelve more jobs on every release from then on. the matrix, as a CI workflow (sketch) ```yaml # 4 internal databases × 7 file stores × 3 login types = 84 jobs, on every release. All of them. strategy: fail-fast: false # one red cell must not hide the other 83 matrix: db: [sqlite, mssql, postgres, mysql] store: [local, azure-blob, s3, minio, gcs, box, sharepoint] login: [local, active-directory, oauth] # oauth: Google, Okta, Microsoft or any OIDC provider # Databases, MinIO and a directory server start as containers. SharePoint, Box and the Microsoft # login don't come in a container: those jobs run against real test tenants – throttling included. # The matrix is the cheap part. What each job asserts is a ticket some customer once filed. ``` So **the adapter layer isn't a technical decision. It is a product decision**, and it only pays off when you sell a boxed product, where the customer's infrastructure is a given, not a choice. Here is the file-store port from that product, trimmed. Two details answer seams from earlier: MinIO gets its own adapter, configured the way a MinIO install actually looks, and every adapter keeps its secret in a separate object from its plain settings. ```csharp // The file-store port. One interface, seven adapters: local disk, Azure Blob, S3, MinIO, // Google Cloud Storage, Box and SharePoint – chosen per installation, by configuration. public interface IFileStorage { FileStorageType Type { get; } int CurrentConfigVersion { get; } // stored settings carry a version, so old ones can be migrated // Every adapter checks its own configuration – and the connection – before anything is saved. Task ValidateAndGetStatus(int configVersion, string settingsJson, string secureSettingsJson, AppConfig config); Task GetFileInfoAsync(Guid definitionId, int configVersion, string settingsJson, string secureSettingsJson, AppConfig config, PathType pathType, params string[] pathSegments); Task UpdateFileAsync(/* … */ Stream contents, UpdateFileMode mode, PathType pathType, params string[] pathSegments); Task DeleteAsync(/* … */ PathType pathType, params string[] pathSegments); } // One adapter. MinIO isn't "S3 with another URL" here: it is configured the way a MinIO install // actually looks – a host and a port. And the secret lives in its own object, apart from the settings. internal class MinioFileStorage : BaseFileStorage { public override FileStorageType Type => FileStorageType.Minio; public override int CurrentConfigVersion => 1; protected override List validateModel(MinioSettings settings, MinioSecureSettings secure) { var errors = new List(); if (string.IsNullOrEmpty(settings.HostAndPort)) errors.Add(new("'HostAndPort' is required")); if (string.IsNullOrEmpty(settings.AccessKey)) errors.Add(new("'AccessKey' is required")); if (string.IsNullOrEmpty(settings.BucketName)) errors.Add(new("'BucketName' is required")); if (string.IsNullOrEmpty(secure.SecretKey)) errors.Add(new("'SecretKey' is required")); return errors; } protected override IWriteableFileProvider getProvider(MinioSettings settings, MinioSecureSettings secure, AppConfig _) => new WriteableMinioFileProvider(new() { HostAndPort = settings.HostAndPort, UseSSL = settings.UseSSL, BucketName = settings.BucketName, AccessKey = settings.AccessKey, SecretKey = secure.SecretKey, }); } ``` The data-source side has the same shape, and its auth enum is short enough to show whole. The general pattern – one port, many implementations, chosen by configuration – is in the [micromonolith post](https://laszlonemes.com/blog/micromonoliths). ```csharp // The data-source side: eleven connectors – SQL Server, Postgres, MySQL, Oracle, Snowflake, BigQuery, // Redshift, Teradata, SAP HANA and two ODBC variants – and one enum for how they log in. public enum DatasourceAuthType { Password, OAuth, RsaKeyPair, // the Snowflake story from the seam section, seen from the vendor's side } ``` It matters not to say this as a boast, because as a boast it is a weak argument. The precise claim is this: **the adapter isn't hard to write. The forty edge cases that make you rewrite it twice are hard to learn.** A few of them: | Where | What bites | | --- | --- | | Oracle | Identifiers were capped at 30 bytes before 12.2. A generated index name that works everywhere else fails on the old one. | | SQL Server | The default collation is case-insensitive. A unique key that is unique in Postgres isn’t unique here. | | Postgres | `search_path`: the table exists – in a schema this connection doesn’t look at. | | SQLite | One writer at a time. `SQLITE_BUSY` the moment a background job and a request write together. | | S3-compatible stores | Path-style versus virtual-hosted addressing – see above. | | SharePoint | `429` with `Retry-After`, counted per tenant, so another app’s traffic throttles yours. | | LDAP / Active Directory | A search returns a referral to a domain controller the server can’t reach. Follow it and hang; ignore it and lose users. | | Entra ID | Token lifetimes are randomised, and with continuous access evaluation a token can be revoked before it expires. A cached token is not a valid token. | That knowledge comes from customers, over years. It isn't in the training data, which is why the code got faster and the experience didn't. ### What feature requests teach you Custom development, feature requests, roadmap presentations – from the inside you learn which request is good and which is pointless. And the bad request usually isn't bad faith: the customer doesn't know exactly what they want either. The difference between the two is that **a good request describes the constraint, not the solution.** "Support Vault" is practically useless. "Our policy forbids credentials at rest in plain text, so we use dynamic database credentials" – that can be solved three different ways, and one of them might be ready tomorrow. Which leads back to the build decision: **when you build for yourself, you have to write that constraint down yourself, and usually nobody does.** That is why the first version of an in-house solution is a faithful picture of its authors' own misunderstanding. ### Drawing the line Because you can't do everything. Every integration is a permanent compatibility contract: versions, deprecations, and the other side's auth changes, which you don't decide. If one team needs system number X+1 for one job once a month, that is not something to integrate. The right answer is an escape hatch: a documented file format, a webhook, a CLI, an API. You don't integrate – you make integration possible, and the last mile is done by whoever needs it. > Don't integrate everything. Give people an interface, and let whoever needs the last mile build it. ## Platform risk ### The AI version, briefly The pattern is familiar. A startup builds something on an AI model, it takes off, and in doing so it proves there is demand. The model provider sees in its usage data that many users need this, and ships it as a built-in feature of its own product. That was the startup. Told like that, the story is too fatalistic, and it needs handling with care. The example people usually reach for – an AI e-mail assistant whose obituary was written the week the model providers shipped mail connectors – mostly proves the opposite: it kept growing next to the built-ins. What the platform absorbed was the **feature**, not the **workflow**. > **Fig. 7** · Same built-in, two outcomes. When a product is the feature, the platform release is its end. When a product owns the workflow and the data, the release mostly advertises the category. > > *Diagram:* Two products built on a model API, after the platform ships the same feature as a built-in. Left: a product that was a prompt and an API call; when the feature is absorbed, nothing remains. Right: a product that owns the workflow – state, permissions, audit trail, domain data and integrations; the feature is absorbed, the workflow remains, and the built-in grows the category and sends the serious users over. So the real question isn't whether they will build it in, but **what is left when they do**: - If your product is essentially a prompt and an API call, you don't have a product. You have a deadline. - If your product owns the workflow and the data – state, permissions, an audit trail, domain data, integrations – the built-in feature tends to grow the category and send the serious users your way. ### The classic version – technically the more important one Platform risk isn't an AI phenomenon. It is only louder now. A few cases, all of them the same pattern: | Platform | What changed | What it cost the integrator | | --- | --- | --- | | Exchange Online | Basic authentication switched off for most protocols from October 2022; SMTP AUTH on its own, repeatedly moved timeline. | Every IMAP or POP integration with a username and password became an app registration, consent, an OAuth flow and narrower permissions – for everyone at once. | | Docker Hub | Anonymous pulls rate-limited per IP from 2020, tightened again in 2025. | CI fleets behind one NAT shared one IP’s quota. Pipelines stopped worldwide at companies that thought this had nothing to do with them. | | Snowflake | Single-factor password sign-in being phased out, service users included. | Every username-and-password integration became key-pair auth, with key storage and rotation – the snippet in the seam section. | | Public APIs | Pricing changed with weeks of notice – Twitter and Reddit in 2023 are the famous ones. | Products that were viable in January weren’t in July. | | Model APIs | Model versions retired on a published schedule. | Prompts and tuning fitted to one model became a forced migration with behaviour drift. Almost nobody prices this into a build. | ```yaml # Anonymous Docker Hub pulls are rate-limited per IP – and a CI fleet behind one NAT is one IP. # The fix is boring: authenticate, or pull through a registry you run yourself. - uses: docker/login-action@v3 with: username: ${{ vars.DOCKERHUB_USER }} password: ${{ secrets.DOCKERHUB_TOKEN }} - run: docker pull registry.internal/mirror/library/node:22-alpine # or: a pull-through cache ``` The sentence that holds it together: platform risk is also when your supplier's supplier changes its auth policy, and you are the one held to account. > **Fig. 8** · The change starts two links away from you and lands on you, because you are the last link your users can call. Nobody in your company decided it, and nobody in your company can postpone it. > > *Diagram:* A chain of four: your users depend on you, you depend on your vendor, and your vendor depends on its platform. A change at the far end – passwords switched off, a rate limit, a new price, a retired model – travels back along the chain and lands on you, although nobody at your company decided it. ### Mitigation, technically - **Have an eval set.** A model abstraction layer leaks: tool-calling formats, structured output, prompt caching and context windows all differ. An eval set, on the other hand, travels. With one, switching models is an engineering task. Without one, it is a matter of faith. I treat the provider as an adapter in the [shadow AI post](https://laszlonemes.com/blog/shadow-ai-local-ai); the eval set is what lets you actually swap it. - **Keep the data and the state on your side**, not in the platform's conversation history. - **Run the margin test.** What share of your gross margin is one vendor's API? What happens if they triple the price – or ship the same thing for free? - **Look at where the big platforms don't go:** on-premises, air-gapped, sector-specific compliance, the local regulatory last mile, legacy integration. It is boring work, which is exactly why it lasts. An eval case doesn't need a framework. It needs an input, an expectation you can check mechanically, and the reason it exists – the reason is what makes it worth keeping through the next three model migrations: evals/invoices/017.json ```json { "id": "invoice-extract-017", "input": "fixtures/invoices/017.pdf", "expect": { "total": "1234.50", "currency": "EUR", "vatId": { "present": true }, "lineItems": { "count": 7 } }, "check": "exact-fields", "why": "two VAT rates on one invoice – the case the March model got wrong" } ``` > If your product is a prompt and an API call, you don't have a product – you have a deadline. ## Lock-in, procurement and the exit **Lock-in isn't automatically bad.** It is one of the prices you can pay wisely: you buy speed and support with it. The trouble starts when you don't know how much you are paying. It helps to split it three ways: - **Data lock-in** – does it come out, in a usable form? - **Workflow lock-in** – how much process has been built on top of it? - **Competence lock-in** – how many people's knowledge is now tied to one product? The practical measure is simple: **how long would it take us to leave, and what stops while we do?** If nobody can answer, the lock-in isn't a decision. It is an accident. ### Onboarding is often bigger than the licence Security review, a DPIA, a pen-test report, the sub-processor list, data residency, the contract, procurement – in an enterprise this takes from six months to a year and a half. From the vendor's side it is a good filter, by the way: a vendor who has been through it a hundred times has every document ready. If a vendor has to assemble them now, that is information in itself. Many people draw the wrong conclusion from this: "in an enterprise it's faster to write it than to procure it". Often that is true. **But it isn't an argument for building. It is a criticism of the procurement process.** Taking on five years of operational responsibility to avoid a six-month process is a bad deal. ### The breakup Think it through in the PoC phase, not when the relationship has already gone bad. What comes out as data, in what form, in how much time? Can you run both systems in parallel? Where do users keep their data in the meantime? Whoever hasn't tested the export doesn't have an exit option, only a hope. ```shell # Week one, not the day of the breakup. Export, load it somewhere that isn't theirs, and count. vendor export --format parquet --include-schema --out ./exit-test/ # whatever "documented export" means duckdb -c "SELECT count(*) FROM './exit-test/documents/*.parquet'" duckdb -c "SELECT count(*) FROM './exit-test/attachments/*.parquet'" # Compare with what the product says it holds. Then time it: that number is your exit, not the contract clause. ``` > Test the export in the first week of onboarding, not on the day of the breakup. ## Build, buy, or both ### Build, when - the integration *is* the product – the glue is the value, and no vendor will ever make it; - the data can't leave the building, and there is no real on-premises or BYOC option; - the pricing model runs against your usage curve; - the scope is small and **stable** – a 500-line script often beats a SaaS at tens of thousands of dollars a year; - you have the substrate and the skills already, and it extends an application you own. ### Buy, even if you could write it - auth, cryptography, identity – never your own SSO; - where the certification is the product: payroll, tax, e-invoicing, banking interfaces – a regulatory target that keeps moving; - where you would compete with the features of a full-time R&D team; - anything that will need fixing at two in the morning, when there is no on-call for it; - the "it's just a CRUD app" trap: the CRUD is ten percent, the permission model is forty. ### Most often: both The most common right answer is the two together: **buy the substrate** – database, identity, storage, model API, queue – and **build on it what makes you different.** The dividing line isn't whether something is "core". It is **whether you can get out of it.** > **Fig. 9** · Build the top, buy the bottom, and draw the line by exits rather than by importance. A standard interface – the Postgres wire protocol, the S3 API, OIDC – means a second supplier exists. A closed platform means it doesn't. > > *Diagram:* The usual right answer is both. Build on top what makes you different – your workflow, your domain rules, the glue nobody sells. Buy underneath the substrate: a database behind the Postgres protocol, identity behind OIDC or SAML, storage behind the S3 API, a model API with your own eval set. Where there is a standard interface there is a way out, so buy freely. A closed platform with no way out deserves a second look, even when it is cheaper. Where there is a standard interface and a way out – the Postgres wire protocol, the S3 API, OIDC – buy with a clear conscience. Where there is no way out, think twice, even when it looks cheaper. It is the same reasoning as every trade-off in the [engineering mindset post](https://laszlonemes.com/blog/engineering-mindset): not a verdict, a price you know. > The question isn't whether it is core. It is whether you can get out of it. ## Objections, and the answers | They say | You say | | --- | --- | | "With AI we can build it in a week. Why pay rent?" | You can build the v1 in a week. The rent pays for years two to five: on-call, security patches, upgrades, the next person. Compare the licence with five years of ownership, not with a week of prompting. | | "The vendor covers 90%." | Who measured it, when, on what data, in whose environment? After an integration test it is usually 60–70, and the missing part sits exactly where you differ. | | "Lock-in is always bad." | It is a price, and often a good one – speed and support. It is bad when you can’t say how long leaving would take and what would stop meanwhile. | | "Procurement takes a year. Building is faster." | Often true. That is a criticism of procurement, not an argument for building. Five years of operating something is a high price for skipping six months of paperwork. | | "Then never build on a model API." | No – own the data and the state, keep an eval set, and run the margin test. The risk isn’t the API. It is a product that is nothing but a prompt. | | "Just abstract the model provider away." | The abstraction leaks: tool calling, structured output, caching and context windows all differ. The eval set is what travels between models. | ## Takeaways 1. **The question didn't change – only the price of the v1.** AI made the first version cheap; operations, integration, change and responsibility cost what they did. 2. **Buy is build too.** You buy three layers and inherit the fourth, integration. The real choice is build-and-own versus integrate-and-rent. 3. **Run the seam test before signing:** storage, database, secrets, auth, network and licence, deployment, observability – and the exit first. 4. **The 90% comes from the demo.** Measure it in your environment, and close the gap by shrinking the requirement before forking or wrapping. 5. **Use AI to disprove the demo**, not to write the build. A two-day PoC against your own Vault, database and storage is now cheap enough that skipping it is a choice. 6. **The adapter layer is a product decision.** The adapter is easy; the forty edge cases are years of customers. 7. **A good request describes the constraint, not the solution** – and when you build for yourself, nobody writes the constraint down. 8. **Platform risk is a supply chain.** Own the data and the state, keep an eval set, know how much of your margin one API is. 9. **Draw the line by exits.** Buy the substrate behind standard interfaces, build what makes you different, and test the export in week one. Build or buy was never really the question. The question is what you will own in five years, and whether you chose it. ## Sources - [Virtual hosting of buckets](https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html) – AWS on virtual-hosted versus path-style addressing. - [Avoid getting throttled or blocked in SharePoint Online](https://learn.microsoft.com/en-us/sharepoint/dev/general-development/how-to-avoid-getting-throttled-or-blocked-in-sharepoint-online) – `429`, `503` and honouring `Retry-After`. - [CREATE EVENT TRIGGER](https://www.postgresql.org/docs/current/sql-createeventtrigger.html) – superuser only, and [CREATE EXTENSION](https://www.postgresql.org/docs/current/sql-createextension.html) on trusted and untrusted extensions. - [Vault database secrets engine](https://developer.hashicorp.com/vault/docs/secrets/databases) and the [Agent Injector annotations](https://developer.hashicorp.com/vault/docs/platform/k8s/injector/annotations) – dynamic credentials and their leases. - [Snowflake key-pair authentication and rotation](https://docs.snowflake.com/en/user-guide/key-pair-auth) – `RSA_PUBLIC_KEY` and `RSA_PUBLIC_KEY_2`. - [Deprecation of Basic authentication in Exchange Online](https://learn.microsoft.com/en-us/exchange/clients-and-mobile-in-exchange-online/deprecation-of-basic-authentication-exchange-online) – the timeline and what replaced it. - [Docker Hub usage and limits](https://docs.docker.com/docker-hub/usage/) – pull limits for anonymous and authenticated users. - [Continuous access evaluation](https://learn.microsoft.com/en-us/entra/identity/conditional-access/concept-continuous-access-evaluation) – why a token can stop being valid before it expires. --- # Engineering mindset in the age of AI > AI took the typing, not the engineering – and the thinking is now the bottleneck. What that mindset is, how its absence looks (tool faith, green tests and red prod, best practices without their context), why AI repriced every trade-off, the difference between simple and naive, and what a junior loses when the consequences land on someone else. With migrations, three pods and an Excel export, in .NET and Java. Source: https://laszlonemes.com/blog/engineering-mindset · Published: 2026-09-26 · Author: László Nemes September 26, 2026 · 27 min read `Engineering` `AI engineering` `.NET` `Java` `Career` A coding agent writes in minutes what used to take a day. That sentence is true, and it is the least interesting thing about the last two years. The interesting part is what it exposed: writing the code was never where engineering happened. It was where engineering *hid*. While you typed, you thought – about the edge case, about the migration, about the second node. Now the typing is gone, and the thinking either happens before and after, on purpose, or it doesn't happen at all. > **The thesis:** AI took the typing, not the engineering. The model is not the bottleneck any more – the codebase, the harness and the review are. What keeps those three in shape is a way of thinking, and that way of thinking just became the scarce resource. > **Fig. 1** · The bar that shrank was the one the thinking used to live in. What is left is thinking and verifying – and if either is missing, there is nothing left to hide it behind. > > *Diagram:* Two timelines of the same task. Before agents, a task was a short think, a long stretch of typing and a short verification, and the thinking could hide inside the typing. With an agent the typing shrinks to almost nothing, so the task is thinking and verifying, and if the thinking is missing it now shows immediately. ## What the mindset is It is not a skill and it is not tooling. It is **responsibility for what happens in production, not for what compiles.** In practice it is three questions before every decision: what can this thing do, what can it not do, and how do I check. The questions are the same for a library, a cloud service and a language model. Only the answers differ. Two habits sit underneath. The first is recognising an edge case before it recognises you: the empty list, the second node, the row count that is a thousand times larger in production. The second is quieter and matters more now than it ever did: being able to take a task and cut it into parts you can move through on a schedule. Delegating work to an agent is easy. Splitting your own work into pieces that can be checked one at a time is the skill – it is what makes the delegation reviewable. What an edge case looks like when it is caught in review rather than in production, from one of my own products. A housekeeping job deletes audit rows older than a configured retention. The happy path is one line. The question is what happens when an operator types `0` into that setting: ```csharp // A housekeeping job deletes audit rows older than a configured retention. // The edge case: "0" or a negative duration puts the cutoff at or after now, // which matches every row. One mistyped setting, and the next pass empties the table. if (retains(settings.AuditEntryDuration, out var cutoff)) db.AuditEntries.RemoveRange(db.AuditEntries.Where(e => e.TimestampUtc < cutoff)); private static bool retains(TimeSpan? duration, out DateTime cutoffUtc) { cutoffUtc = default; var now = DateTime.UtcNow; if (!duration.HasValue || duration.Value <= TimeSpan.Zero || duration.Value > now - DateTime.MinValue) return false; // unset, zero, negative or absurd: keep everything cutoffUtc = now - duration.Value; return true; } ``` Nothing about that guard is hard. It is the habit of asking "and what if the number is zero" before the job runs at 2 a.m. on a customer's database – and the agent, which wrote the one-line version first, had no reason to ask. ### The migration You rename a column. The agent generates an EF Core migration. It compiles, the tests are green, it runs on the dev database. Here is what it generated: ```csharp // What the agent generated for "rename Customer.Name to FullName". // Compiles, tests are green, the dev database has twelve customers. protected override void Up(MigrationBuilder migrationBuilder) { migrationBuilder.DropColumn(name: "Name", table: "Customers"); // 2 million values, gone migrationBuilder.AddColumn(name: "FullName", table: "Customers", nullable: true); } ``` Locally that is fine – twelve customers, all of them yours. On the dev database it is fine too, there is nothing in it. In QA, forty thousand names are gone. In production, two million. The engineering question was never "does it run". It was: **does it run backwards, is it backward compatible, and what happens to the rows that already exist?** > **Fig. 2** · Expand first, contract later. The new column goes in and is filled while the old code still runs; the old column goes out in a later release, once nothing reads it. Every step has a working down, which is the only reason any step is allowed to go up. > > *Diagram:* The same column rename as two migrations. Drop plus add passes on the developer's twelve rows, then destroys forty thousand rows in QA and two million in production, and has no way back. Expand then contract adds the new column and copies the data first, deploys code that reads it, and drops the old column in a later release, with a working down step at every stage. ```csharp // Step 1 – expand. Runs against production with the old code still deployed. public partial class AddCustomerFullName : Migration { protected override void Up(MigrationBuilder mb) { mb.AddColumn(name: "FullName", table: "Customers", nullable: true); mb.Sql("UPDATE \"Customers\" SET \"FullName\" = \"Name\" WHERE \"FullName\" IS NULL;"); } // The way back has to exist before the way forward is allowed to run. protected override void Down(MigrationBuilder mb) { mb.DropColumn(name: "FullName", table: "Customers"); } } // Step 2 – contract. A later release, once nothing reads "Name" any more. public partial class DropCustomerName : Migration { protected override void Up(MigrationBuilder mb) => mb.DropColumn(name: "Name", table: "Customers"); protected override void Down(MigrationBuilder mb) { mb.AddColumn(name: "Name", table: "Customers", nullable: true); mb.Sql("UPDATE \"Customers\" SET \"Name\" = \"FullName\";"); } } ``` I am a code-first believer: the application applies its own pending migrations on start-up, so I almost never touch a database by hand because of the app. In one of my products that is the first thing the process does, before the cache or any service is wired: ```csharp // Program start-up, before anything else is wired: the app owns its schema. log.LogInformation("Migrating database"); await db.Database.MigrateAsync(); // applies whatever is pending, on every start log.LogInformation("Migration complete"); await cache.InitializeAsync(); // the cache port, whichever adapter is configured ``` The other half of that discipline is that the same binary can be started in CLI mode for the things that must not happen silently: moving an installation between database providers, clearing it, or migrating it on purpose. Two flags carry the whole mindset. Without them the tool refuses, and the message says why. ```shell # The web server binary, started with a verb instead of as a web server. app database migrate # to the latest known migration app database copy --from-db-type Sqlite --to-db-type Postgres # move an installation between providers app database clear --accept-data-loss # refuses without the flag, and says why app database copy ... --perform-migration # same: the operator opts in, explicitly ``` Two guards in that code are worth stealing. A database that contains migrations the binary has never heard of belongs to a newer version: refuse, don't "fix" it. And a CLI task that had to migrate first migrates the database *back* to where it found it when the task is done, by running the `Down` steps, so a maintenance job can never leave a customer on a schema their running version can't read. The way back is not a restore from backup. It is a command, and it runs by itself. ```csharp // CLI mode: the same binary, started with a verb. Two guards carry the mindset. internal class DatabaseSession(ILogger log, AppDbContext db) : IAsyncDisposable { private string? originalLatestMigration; private bool migratedAtInit; public async Task Initialize(IAcceptMigrationRequired input) { var known = db.Database.GetMigrations().ToArray(); var applied = (await db.Database.GetAppliedMigrationsAsync()).ToArray(); // Guard 1: migrations in the database that this binary has never heard of. // That is a newer version's database. Refuse – don't "fix" it. if (applied.Any(a => !known.Contains(a))) throw new ApplicationException( "database is used by a newer version of the software. Update the software to continue."); var pending = (await db.Database.GetPendingMigrationsAsync()).ToArray(); migratedAtInit = pending.Length > 0; originalLatestMigration = applied.LastOrDefault(); // Guard 2: never migrate as a side effect. The operator says so, with a flag. if (migratedAtInit && !input.AcceptMigrationRequired) throw new ApplicationException( $"database requires {pending.Length} migrations. In rare cases this may lose data. " + "Pass --perform-migration if you know what you're doing."); if (migratedAtInit) await db.Database.MigrateAsync(); } // The way back is not a backup. It is a command, and it runs by itself when the task is done, // so a maintenance operation cannot leave a customer on a schema their running version can't read. public async ValueTask DisposeAsync() { if (migratedAtInit && originalLatestMigration is not null) { log.LogInformation($"Migrating database back to {originalLatestMigration}"); await db.Database.MigrateAsync(originalLatestMigration); // runs the Down steps } } } ``` The Java version of the same story has a twist worth knowing. Flyway runs versioned SQL forward beautifully; the *undo* migrations are a paid feature, and Liquibase needs an explicit rollback block. So in a Spring Boot project the way back is usually another forward migration – which makes the expand step even more important, because you have to be able to *leave it in place*. ```sql -- V7__add_customer_full_name.sql (expand: additive, old code keeps working) ALTER TABLE customers ADD COLUMN full_name VARCHAR(200); UPDATE customers SET full_name = name WHERE full_name IS NULL; -- V9__drop_customer_name.sql (contract: a later release, once nothing reads "name") ALTER TABLE customers DROP COLUMN name; -- There is no free "down" in Flyway: undo migrations are a paid feature, and -- Liquibase needs an explicit rollback block. In Java the way back is another -- forward migration – so the expand step must be safe to leave in place. ``` > The agent writes the migration. Whether it runs backwards is my job. Not because the agent couldn't write the down step – it can, if asked – but because the agent has no production data. I do. ## “Configure it and go” The first place the mindset goes missing: install Claude Code, paste in someone's `CLAUDE.md` from the internet, add ten plugins and three MCP servers, and it will work. **The tool stands in for understanding** – exactly like copying an nginx config without knowing what keepalive is. Two concrete ways this costs money. A template says *"run the full test suite after every change"*. Your suite takes 25 minutes. The agent obeys, on every small edit; by the end of the day the session has burned a week's worth of tokens and wall-clock time, and nobody can say why it felt slow. Or: an MCP server with forty tools, of which you use two. The other 38 definitions sit in the context of every request, degrade the reasoning a little, and never show up on the invoice – only in the quality. My own setup is vanilla: Claude Code, my own agents, my own skills. If I need something else it is a script, ssh, or bash. A well-kept codebase is worth more than any amount of tooling, because the model works from what is in the repository. The test for every piece of configuration is simple: **if you removed it, could you say what would change?** If not, you don't understand it. You have configured it. > The length of your plugin list is inversely proportional to how well you understand what the model does. ## Works on my machine, the higher level The classic: it runs on my laptop, not in production. The AI-era version is more insidious: **the code runs, the tests are green, and the agent checked it – in its own sandbox.** One node, one user, five rows of test data, twenty database connections. Take a well-structured modular monolith running on Kubernetes as three pods. The agent implements a feature: a user uploads a file, and a workflow step processes it. Locally it is perfect. > **Fig. 3** · None of this is the model's fault. It does not know there are three pods – the prompt didn't say. The production context is yours to put in: in the CLAUDE.md, in the plan phase, and in a load test. > > *Diagram:* Left: the sandbox the agent tested in, one process with one user and a few rows, tests green. Right: production with three pods. The upload was written to pod A's local disk, the workflow started on pod B and cannot find the file, and pod C's in-memory status cache says there is no such workflow. The fix is that nothing lives only inside the process: files in a shared blob store, cache in Redis, state in the shared database. In Spring Boot the same bug arrives in idiomatic clothes, which is what makes it hard to see in review. `@Cacheable` on the default `ConcurrentMapCacheManager` is a per-JVM cache. `MultipartFile.transferTo` writes to this pod's disk. Spring Session defaults to memory, so a login on pod A is a 401 on pod B. And HikariCP's default pool of ten looks generous until you multiply it by pods and by users and read the Postgres `max_connections` setting. ```java @Service public class WorkflowService { private final Path uploads = Path.of("/tmp/uploads"); // this pod's disk public String upload(MultipartFile file) throws IOException { var target = uploads.resolve(UUID.randomUUID() + ".csv"); file.transferTo(target); // pod A return target.toString(); } @Cacheable("workflow-status") // ConcurrentMapCacheManager: per JVM public WorkflowStatus status(String workflowId) { return repository.findStatus(workflowId); } } ``` The fix is not clever. It is that nothing lives only inside the process: the cache is pluggable to Redis or a database table, the file store is an adapter over Blob, S3 or MinIO, and anything that is state is in the shared database. That is a configuration change, not a rewrite – if the ports were there from the start. How I build those ports is the [micromonolith post](https://laszlonemes.com/blog/micromonoliths); the point here is different. The point is *who* puts that rule in front of the agent. application.yml ```yaml # application.yml – the same code, one Redis and one bucket away from three pods spring: cache: type: redis # @Cacheable now lives outside the JVM session: store-type: redis # a login on pod A is valid on pod B datasource: hikari: maximum-pool-size: 10 # × pods × users – this is what hits Postgres max_connections files: store: s3 # local | s3 | azure-blob | minio – behind one FileStore port bucket: shop-uploads ``` This is what the same port looks like in my own product, comments included – the comment in the configuration file is the contract with whoever installs it, and it says the multi-node thing out loud: appsettings.json (shipped sample, trimmed) ```json { "CacheSettings": { // 'InMemory' (default) – the fastest, but only works reliably on single-node installations // 'None' – no cache. Not supported with OAuth login // 'Redis' – recommended for multi-node installations // 'SqlServer' | 'Postgres' | 'MySql' – a table in that database as the cache "Type": "InMemory", "InMemoryCacheSettings": { "MaxCacheSize": "1gb" }, "RedisCacheSettings": { "Configuration": "localhost:6379", "InstanceName": "app" } } } ``` ```csharp // One port. The adapter is a case in a switch over the configured type – no code change to go multi-node. public interface ICache { Task GetOrCreateAsync(string key, Func> create, CancellationToken ct = default) where T : class, new(); Task SetAsync(string key, T value, CacheOptions options, CancellationToken ct = default) where T : class, new(); Task RemoveAsync(string key, CancellationToken ct = default); Task InitializeAsync(); } switch (configuration.GetValue("CacheSettings:Type")) { case CacheType.InMemory: services.AddMemoryCache(sizeLimit); break; case CacheType.Redis: services.AddRedis(section["Configuration"]!, section["InstanceName"]!); break; case CacheType.Postgres: services.AddPostgres(section["ConnectionString"]!, section["Schema"]!, section["TableName"]!); break; // SqlServer, MySql, None … } ``` The file store has the same shape: local disk, Azure Blob, S3, MinIO, Google Cloud Storage, Box and SharePoint behind one interface, picked the same way. None of it is clever. It is the decision, made once, that nothing lives only inside the process. Whoever writes "no in-process-only state; three pods in production" into the project's instructions gets a Redis adapter from the agent. Whoever doesn't gets a `Dictionary`. The conclusion is not that AI code is unfit for production. It is that **the production context is yours to supply**, and that discovering it in review is the expensive way. > Green tests, red prod – the AI-era works on my machine. The model has one machine: its own. The number of your pods is yours to know. ## What a best practice really is Not a rule. **Condensed experience, with context attached.** Someone got burned, wrote it down, and the lesson is true at a particular scale, with a particular team size, under particular constraints. Microservices are the pain of a several-hundred-person, several-time-zone organisation. Kubernetes is Google's scale. One hundred percent test coverage is a project where a bug costs a life. A best practice has two parts, the rule and its context, and whoever carries only the rule carries half. Netflix is the example everyone reaches for, so it is worth reading what they actually wrote. They run Spring Boot microservices on the JVM at a scale almost nobody else has. Moving from JDK 8 to 17 gave them roughly 20% better CPU usage *with no code change*, purely from improvements in G1. On JDK 21 with Generational ZGC, garbage-collection pauses went from over a second to effectively zero, with better P99 latency at equal or better CPU – and they made it the default collector on 21 and later. > **Fig. 4** · Their scars, and what you may take from them. The topology answers a question about their organisation. The upgrade discipline answers a question about physics, and physics is the same at your scale. > > *Diagram:* Left, Netflix's Java scars: moving from JDK 8 to 17 gave about twenty percent less CPU with no code change, JDK 21 with Generational ZGC took garbage-collection pauses from over a second to effectively zero, and enabling virtual threads produced a silent deadlock through synchronized blocks. Right, what transfers to a small team: stay on a current LTS and read the post-mortem before flipping a flag. What does not transfer is their service topology, which answers their team count, not yours. None of that means a five-person team should copy their architecture in miniature. What does transfer, almost unconditionally: **stay on a current LTS.** Java 21 and now 25, .NET 8 and now 10 – every one of those releases carried runtime and GC work that you get for the price of a version bump and a test run. Falling behind compounds, because each skipped LTS makes the next upgrade bigger and scarier, and the performance you didn't collect is gone. The same company also published the counter-scar, which is the part people skip. "Just enable virtual threads" was the Java 21 advice everywhere. Netflix enabled them on Spring Boot 3 with Tomcat and got services that hung silently: no error, no stack trace, sockets stuck in `CLOSE_WAIT`. Virtual threads that blocked inside `synchronized` blocks stayed pinned to their carrier threads until every carrier was pinned and nothing could run – a deadlock that looked like quiet. JDK 24 fixed the pinning through JEP 491. The best practice was right; it just came with a JDK version attached, and the version was the context. ```java // Java 21: a virtual thread that blocks inside synchronized pins its carrier thread. // Enough of these at once and every carrier is pinned – the pool is full and nothing // is running. No exception, no stack trace, just sockets in CLOSE_WAIT. synchronized (cacheLock) { value = client.fetch(key).get(); // blocks while pinned } // The same wait behind a java.util.concurrent lock releases the carrier. lock.lock(); try { value = client.fetch(key).get(); } finally { lock.unlock(); } // JDK 24 (JEP 491) removed the pinning for synchronized. The advice "just enable // virtual threads" was true – with a JDK version attached. ``` A smaller preference, same reasoning. I don't want one-line cleverness in a codebase; I want things written out so the next developer understands them without a whiteboard. The AI era made this stronger, not weaker: **if a person can read it, the model can too**, and the model reads it far more often than the person does. > A best practice is a scar and a lesson. If you copy only the lesson, you don't know where it will hurt. ## The model’s defaults are the internet’s averages A language model learned from the average of the internet, and on the internet the loudest patterns win. Without context, it gives you the statistically expected answer – which is the architecture of the companies that blog the most, applied to the app you actually have. Here is what you get without asking: | You ask for | .NET default you get | Java default you get | | --- | --- | --- | | data access | a Repository and a UnitOfWork over `DbContext` – which already is both. Two layers that do nothing and hide the LINQ | a Service, a Repository, a Mapper and a DTO per entity – a quartet for every table | | five CRUD endpoints | CQRS with MediatR: fifteen handler classes, reads and writes separated on an admin page with ten users | a hexagonal Maven multi-module with seven modules | | a plan | "let’s extract notifications into its own service" – for one team with one data source, where it is a module in an assembly | Spring Cloud with Eureka, a Config Server and an API Gateway – for three developers | | resilience | a retry policy on every call, including the ones that must not be retried | Hystrix, Ribbon and Zuul – in maintenance since 2018; Netflix itself moved to Resilience4j and Spring Cloud removed them | | deployment | a Kubernetes manifest with an HPA – for an app with in-process state, where scaling out is a bug generator | the same manifest, with a Helm chart | The last Java row is the one that shows the mechanism most clearly. The model isn't wrong about what Netflix used; it is a few years behind on what Netflix *stopped* using, because the internet wrote about the beginning far more than about the end. We used to fight the "this is how Netflix does it" reflex inside the team. Now we fight the model's version of it too, and the model doesn't get tired. > The model doesn't know you're one team with one data source. If you don't say so, you get thirty services for an internal admin page. ## Trade-off thinking, repriced This is the core. There is no best solution – **you pay a price, and you know what it is.** Three questions: what does it cost, what breaks, can it be undone. Some of mine, with the price named: - **Where the model runs.** Frontier API, your own cloud tenant, or local – it is data sensitivity plus whether the arithmetic works out, per use case, never "always". The long version is in the [shadow AI post](https://laszlonemes.com/blog/shadow-ai-local-ai). - **Lock-in.** Not a sin, a price paid on purpose – as long as you know the migration path for the day the vendor relationship goes bad. - **Settings synchronised across nodes** by a background job, with about a minute of eventual consistency. It could have been pub/sub. The price of pub/sub is one more infrastructure component; a minute of inconsistent settings costs nothing here, and a Redis dependency in a boxed product costs a great deal. - **Kubernetes all year, or a few VMs for the season.** Complexity paid twelve months for a two-week peak is a bad trade, and it is bad in the spreadsheet, not in taste. The settings job, since it is the one people push back on most, is small enough to show whole. The price is in the last comment: ```csharp // Settings live in the database. Every node re-reads them on a timer – no pub/sub, no broker. public interface IHostedJob where TSettings : HostedJobSettings { string Name { get; } IOptionsMonitor GetSettings(); Task Run(TSettings settings, IServiceProvider scopedSp, CancellationToken ct); } internal class AppConfigRefreshJob(IOptionsMonitor settings) : IHostedJob { public string Name => "AppConfigAutoRefresh"; public IOptionsMonitor GetSettings() => settings; public Task Run(AppConfigRefreshSettings _, IServiceProvider sp, CancellationToken ct) => sp.GetRequiredService().RefreshConfig(); // the whole job } // appsettings.json – the price, in one line: a node may be a minute behind // "AppConfigAutoRefresh": { "Enabled": true, "IntervalSettings": { "Interval": "00:01:00" } } ``` What AI did to this is not that it decided any of these. It **changed the prices.** > **Fig. 5** · What was “right but expensive” is now close to free, and what was cheap is now the cost centre. Whoever still budgets by the old prices is optimising the wrong column. > > *Diagram:* Two columns. Cheaper now: static types, whose price was human keystrokes; tests, documentation and readable code; and the compiler as the agent's feedback loop. More expensive now: review and supervision, because code is cheap and checking is not; under-specified tickets, because the missing decision lands on whoever runs the agent; and a confident tone, because the model never says it is unsure. Static types are the clearest case. Their price was human keystrokes, and for a model that price is zero. In return the compiler is the cheapest deterministic feedback loop you can give an agent: rename a property, and the C# or Java compiler points at forty places; in Python the forty-first turns up at runtime, in production. The agent learns from the first and never sees the second. Tests, documentation and readable code went the same way – cheap to produce, and they pay back in tokens and reasoning quality. My longer position on types and on token optimisation is on the [State of AI](https://laszlonemes.com/ai) page. The other column is where the budget went. Code is cheap; checking it is not, which is why a day of supervising an agent is more tiring than a day of writing. And the under-specified ticket became expensive in a way it never was: "the AI will fill it in" means the missing decision is made by whoever runs the agent, often a junior, who doesn't know a decision was made. > There is no best solution. Only one whose price you know. AI didn't settle trade-offs – it repriced them. The price of types was human typing; that is now zero, and the debate is over. ## Simple vs. naive From the outside they look the same. From the inside they are opposites: **the simple solution knows what it left out. The naive one doesn't know it left anything out.** > **Fig. 6** · Simplicity is not a property of the output. It is a property of the decision. The bottom row is the same mistake at two price points – and the agent produces either one, depending on what you asked. > > *Diagram:* A two-by-two of how much you built against whether you can name what you left out. Little built and you know what you left out is simple: an in-memory cache with Redis one configuration line away. A lot built and you know why is earned complexity: Kubernetes because you have the workloads to feed it. Little built without knowing what is missing is naive-simple: a static dictionary because there is one node for now. A lot built without knowing why is naive-complex: seven projects for five endpoints. - **Simple:** a monolith written for multiple nodes from day one. `IDistributedCache` with memory behind it, and Redis one configuration line away. A file-store port with local disk as the default and Blob or S3 one line away. SQLite for a zero-config start, PostgreSQL with a connection string. In Spring: Caffeine locally, `spring.cache.type=redis` when it is time. I don't run it on several nodes until I have to – but the door is open. - **Naive-simple:** `static Dictionary` as the cache, or a `static ConcurrentHashMap`, because "there's one node for now". Same appearance; the bug from the three-pods section is already inside. - **Naive-complex:** hexagonal everything, ports and adapters for each dependency, seven projects in the solution – or seven Maven modules – for a five-endpoint app. The same mistake as the Dictionary, at a higher price. ```csharp // "One node for now." This is the sentence the outage is written in. public static class WorkflowRuntime { public static readonly Dictionary Running = new(); } // Same shape, different sentence: in-memory today, Redis is a line of configuration. public sealed class WorkflowRuntime(IDistributedCache cache) { public Task GetAsync(string id, CancellationToken ct) => cache.GetAsync($"workflow:{id}", ct); } ``` "SQLite for a zero-config start, PostgreSQL with a connection string" is not a slogan, it is a switch expression. The price, named: transactions, locking and SQL dialects differ, so the code stays near the lowest common denominator, and every provider needs its own migrations and its own place in CI. ```csharp // The database is an attached resource: four providers behind one context interface, // chosen from configuration. Empty configuration means SQLite and the app just starts. IApplicationDbContext db = info.Type switch { InternalDbType.Sqlite => new SqliteDbContext(options(o => ConfigureSqlite(o, info.ConnectionString))), InternalDbType.MsSql => new SqlServerDbContext(options(o => ConfigureSqlServer(o, info.ConnectionString, info.Schema))), InternalDbType.Postgres => new PostgresDbContext(options(o => ConfigurePostgres(o, info.ConnectionString, info.Schema))), InternalDbType.MySql => new MySqlDbContext(options(o => ConfigureMySql(o, info.ConnectionString, "8.0.28"))), _ => throw new ArgumentOutOfRangeException(nameof(info.Type), "database type not recognised in config"), }; ``` The test is one question: **can you name what you gave up?** If the answer is "nothing", the answer is naive. > Naivety has two faces: the Dictionary cache and the seven-project CRUD. ## Junior → senior The mindset isn't taught in a lecture. **It is made of consequences.** You wrote it, it shipped, it broke, you fixed it at three in the morning, and you never did it that way again. The senior knows where the bodies are because they buried them. Now the writing is disappearing, and with it the incidental learning that happened while writing. The risk is not that juniors can't code. It is that they **skip the school of consequences**: the agent writes it, the tests are green, the junior forwards it, and someone else finds the bug in production. The feedback is disconnected from the one person who could have learned from it. > **Fig. 7** · The loop on the left is how every senior you know was made. The loop on the right has the same four steps and no return arrow. Closing it again is now a management task, not a matter of luck. > > *Diagram:* Two loops. How seniors were made: you write it, ship it, it breaks at three in the morning, you fix it, and the lesson returns to the person who wrote it. The AI-era version: the agent writes it, the tests are green, the junior forwards it, someone else fixes it in production, and the lesson lands on someone who did not make the decision. The loop that made seniors is open. ### “Make an Excel export” The ticket says that and nothing more. The agent picks a library, loads the whole table into memory, and writes it out. Locally, with five hundred rows, it is lovely. In production, with two million rows, it is an `OutOfMemoryError`. ```java // The ticket said "make an Excel export". The agent made one. @GetMapping("/orders/export") public ResponseEntity export() throws IOException { var orders = orderRepository.findAll(); // all 2,000,000 rows, as entities try (var workbook = new XSSFWorkbook(); // whole workbook in memory var out = new ByteArrayOutputStream()) { // and a second copy of it var sheet = workbook.createSheet("orders"); for (int i = 0; i < orders.size(); i++) { /* one row per order */ } workbook.write(out); return ResponseEntity.ok(out.toByteArray()); // the request has been open for minutes } } ``` The junior didn't know that three decisions were made here – which library, streaming versus in-memory, and a background job versus a request-response – because the ticket was written as if there were no decisions. A few years ago the junior would have written this by hand and found out at the second loop that it doesn't fit. Now the agent hands over the "working" naive version straight away. > **Fig. 8** · Same ticket, two shapes. The top lane is one decision that nobody noticed being made. The bottom lane is three decisions, and the third one is not code – it is a question back to whoever wrote the ticket. > > *Diagram:* Top lane, what the agent shipped: a GET request loads all two million rows as entities, builds the whole workbook in memory, copies it into a byte array and runs out of memory while the request is still open. Bottom lane, the version with three decisions: the request enqueues a job and returns an id; the job runs on one pod, streams rows out of the database and rows into the file; the file lands in a shared store and the user gets an expiring link. Excel's ceiling is 1,048,576 rows per sheet, so the third decision is a question back to the requester. - **A job, not a request.** The endpoint enqueues and returns a job id; the job runs on one pod, writes to the shared file store, and the user gets an expiring download link. Idempotent: the same parameter hash means the same file, so a double click doesn't start a second two-million-row export. - **Streaming at both ends.** Out of the database as one ordered read – `AsNoTracking().AsAsyncEnumerable()` or a `DbDataReader` in .NET, a JPA `Stream` with a fetch-size hint in a read-only transaction in Java; keyset pagination if you must page, never `OFFSET`. Into the file row by row: OpenXML's `OpenXmlWriter`, MiniExcel or LargeXlsx in .NET, `SXSSFWorkbook` instead of `XSSFWorkbook` in POI. No `MemoryStream`, no `byte[]`. Styles only on the header, by style index, not per cell. - **Excel has a ceiling.** A sheet holds 1,048,576 rows. Two million doesn't fit on one. Several sheets – or the better move: ask what it is for. Another system wants CSV or Parquet. A human almost certainly wants a filtered, aggregated view, not two million rows. ```csharp // A background job on one pod. The request only enqueued it and returned a job id. // OpenXML needs a seekable stream, so: a temp file on disk, then a streamed upload. var tempPath = Path.GetTempFileName(); await using (var file = File.Create(tempPath)) using (var doc = SpreadsheetDocument.Create(file, SpreadsheetDocumentType.Workbook)) { var sheetPart = doc.AddWorkbookPart().AddNewPart(); using var writer = OpenXmlWriter.Create(sheetPart); // SAX-style: writes, never holds the sheet writer.WriteStartElement(new Worksheet()); writer.WriteStartElement(new SheetData()); WriteRow(writer, "Id", "Customer", "Total"); // styles only here, by style index var orders = db.Orders.AsNoTracking() .Where(o => o.CreatedAt >= from) .OrderBy(o => o.Id) // one ordered read, no OFFSET pages .AsAsyncEnumerable(); await foreach (var o in orders.WithCancellation(ct)) // one entity in memory at a time WriteRow(writer, o.Id, o.CustomerName, o.Total); writer.WriteEndElement(); writer.WriteEndElement(); } await fileStore.UploadAsync($"exports/{job.Key}.xlsx", tempPath, ct); // shared store, any pod can serve it File.Delete(tempPath); ``` ```java // SXSSF keeps a window of rows in memory and flushes the rest to a temp file. try (var workbook = new SXSSFWorkbook(100); var out = fileStore.openWrite("exports/" + job.key() + ".xlsx")) { var sheet = workbook.createSheet("orders"); int r = 0; writeHeader(sheet.createRow(r++)); // @Transactional(readOnly = true) + a fetch-size hint: the driver streams, JPA doesn't cache try (Stream orders = orderRepository.streamByCreatedAtAfterOrderByIdAsc(from)) { for (var it = orders.iterator(); it.hasNext(); ) { var o = it.next(); var row = sheet.createRow(r++); row.createCell(0).setCellValue(o.id()); row.createCell(1).setCellValue(o.customerName()); row.createCell(2).setCellValue(o.total().doubleValue()); entityManager.detach(o); // or the persistence context grows with the file } } workbook.write(out); workbook.dispose(); // deletes the temp files } ``` The naive version made one decision. The good one made three – and the third is a question, not code. That question is the engineering. For honesty, the counter-example from my own code. One of my products exports its audit log with the whole workbook in memory, written to a temp file that deletes itself when the response closes. That is the top lane of Fig. 8. It is also the right shape there, because the housekeeping job from the first section caps the audit table at a configured retention, so the export has a ceiling I can name. Same shape as the naive version; the difference is that I can say why. ```csharp // My own audit export: whole workbook in memory, then a self-deleting temp file. The naive shape – // and the right one here, because a housekeeping job caps the audit table at a configured retention, // so this export has a ceiling I can name. var wb = new XLWorkbook(); var ws = wb.Worksheets.Add("Audit"); for (var i = 0; i < entries.Length; i++) { ws.Cell(i + 2, 1).Value = entries[i].TimestampUtc.ToString(format); ws.Cell(i + 2, 4).Value = CsvFieldWriter.EscapeFormula(entries[i].UserName); // "=cmd|…" stays text ws.Cell(i + 2, 5).Value = CsvFieldWriter.EscapeFormula(entries[i].DetailsJson); } var stream = new FileStream(Path.GetTempFileName(), FileMode.OpenOrCreate, FileAccess.ReadWrite, FileShare.None, 4096, FileOptions.DeleteOnClose); // gone when the response finishes wb.SaveAs(stream); stream.Position = 0; return new FileStreamResult(stream, XlsxContentType) { FileDownloadName = "ExportAudit.xlsx" }; ``` ### What I would actually do If the typing is fast, the freed time should go into putting juniors in the room for architecture decisions early: watching, commenting, being wrong, and getting the consequence. The market is doing the opposite, and that is where the next decade's senior shortage is being built. Concretely: - **The junior predicts what the agent will do** before it runs, then we compare. The plan is their work, not the model's. - **Review asks "why this, and what was the other path"** – not "did it pass". - **Breakpoints, a profiler, a load test.** The AI is an excellent debugger; without the fundamentals you don't know where to put the breakpoint. - **Tickets don't get under-specified** on the theory that the AI will fill in the rest. The rest is the decision. The senior's role flipped with it: not throughput, judgement. **The senior is the one who knows when the model is lying** – and the one who organises, for the junior, the consequences that AI took away. > The problem isn't that the junior can't code. It's that they don't know they made a decision. ## Years vs. experience Separate the years from the experience. Three years across several domains – in on the architecture decisions, a few stacks, LDAP and Active Directory set-ups, customers who each wanted something different – can see a problem in a wider frame than fifteen years of only Spring Boot. The fifteen-year engineer is superb at Spring Boot. The three-year engineer knows what the problem looks like from three other places, which is what puts it in context. What I support is the generalist with one deep domain: know something about everything so you can place a problem, and go deep in one thing so you have stable ground to return to. The two also map onto career phases. The more of a generalist you are, the better a startup or a scale-up fits that period of your life. When you burn out, or simply need calmer water, the deep domain buys you a seat on an enterprise – a huge ship that turns slowly, and that is the point of it. ### Ego, and where it comes from This profession needs some ego and some confidence: *yes, I'll solve this, and it will be good.* For that to settle into something healthy, you need failures – incidents, mistakes, the occasional slap – and they shape you. Own the mistake, fix it, and the next time *still* say what you think. Not "nobody cares what I say because I was wrong last time"; everyone is wrong sometimes. Like falling off a horse: back in the saddle, fast, if you ever want to ride. Ego and confidence are what carry a feature or a decision through and let you influence other people – not by pushing them down, by sharing what you have seen. Which closes the loop back to the previous section. Confidence without consequences is the naive kind. Confidence with a few scars is the engineering kind, and it is exactly the thing the current setup – agent writes, tests green, forward it – is quietly not producing. ## Objections, and the answers | They say | You say | | --- | --- | | "Best practices exist so we don’t relearn everything. Throw them out?" | No. A default, not a dogma. A junior following a good default beats one inventing something. The trouble starts where Kubernetes is not an option but the answer, before the question. | | "AI code isn’t worse than an average developer’s." | It isn’t. The difference is volume and tone: ten times the code, in the same confident voice, without the human "I’m not sure about this" signal. Review load up, signal down. That is capacity arithmetic, not hostility. | | "Juniors learn faster now – the AI explains everything." | Explanation got cheap. Whoever read that the pool can run out knows it. Whoever watched it run out at three in the morning understands it. The second is not available in a chat. | | "This is only true in your .NET, boxed-product world." | The examples come from there because that is where I can be concrete. The principle – the model has no production context, the trade-offs were repriced – holds at a Python startup too. It just takes longer to hurt. | | "You use it every day yourself." | Intensively, as a tool, and I read every line. None of this is an argument against AI. It is an argument against thinking that stopped when the typing did. | ## Takeaways 1. **AI took the typing, not the engineering.** The bottleneck moved to the codebase, the harness and review – and the mindset is what keeps those in shape. 2. **Three questions before every decision:** what can it do, what can't it, how do I check. The same for a library, a cloud service and a model. 3. **The agent has no production data.** Migrations run backwards, columns expand before they contract, and the number of pods goes into the instructions, not into the post-mortem. 4. **If you removed it, could you say what changes?** That is the test for every plugin, MCP server and pasted instruction file. 5. **A best practice is a scar plus context.** Copy Netflix's LTS discipline; don't copy their topology. 6. **The model's defaults are the internet's averages.** Tell it you are one team with one data source, or you get thirty services. 7. **AI repriced the trade-offs.** Types, tests and readable code got cheap; review and under-specified tickets got expensive. Budget by the new prices. 8. **Simple knows what it left out.** Naive comes in two sizes, and the agent makes both. 9. **Close the consequence loop for juniors on purpose.** Predict, compare, ask why, and put them in the room early. The senior's job is judgement, not throughput. None of this is against AI. It is against thinking that stopped when the typing did. ## Sources - [Bending pause times to your will with Generational ZGC](https://netflixtechblog.com/bending-pause-times-to-your-will-with-generational-zgc-256629c9386b) – Netflix TechBlog on JDK 21 and making Generational ZGC the default. - [Java 21 Virtual Threads – Dude, Where's My Lock?](https://netflixtechblog.com/java-21-virtual-threads-dude-wheres-my-lock-3052540e231d) – the pinning post-mortem: Spring Boot 3, Tomcat, `synchronized`, `CLOSE_WAIT`. - [How Netflix Really Uses Java](https://www.infoq.com/presentations/netflix-java/) – Paul Bakker's talk, including the JDK 17 CPU numbers. - [JEP 491: Synchronize Virtual Threads without Pinning](https://openjdk.org/jeps/491) – delivered in JDK 24. - [Hystrix](https://github.com/Netflix/Hystrix) – the maintenance notice and the pointer to Resilience4j. - [EF Core migrations](https://learn.microsoft.com/ef/core/managing-schemas/migrations/) and [Flyway undo migrations](https://documentation.red-gate.com/flyway/flyway-concepts/migrations/migration-types/undo-migrations) – the up, the down, and what the down costs. - [Apache POI SXSSF](https://poi.apache.org/components/spreadsheet/how-to.html#sxssf) – streaming workbooks, and the row window. - [Excel specifications and limits](https://support.microsoft.com/en-us/office/excel-specifications-and-limits-1672b34d-7043-467e-8e27-269d656771c3) – 1,048,576 rows per sheet. --- # Shadow AI and local AI: the data class decides > It isn't local versus cloud: there are four levels of control, and the data class picks one. The arithmetic of running models locally, why local carries the burden of proof, the LLM provider as an adapter, what an EU region does and doesn't give you – and why shadow AI is a product problem, not a disciplinary one. Source: https://laszlonemes.com/blog/shadow-ai-local-ai · Published: 2026-09-20 · Author: László Nemes September 20, 2026 · 22 min read `AI engineering` `Local LLMs` `Security` `Architecture` The conversation about where AI should run is usually framed as *local versus cloud*, and the conversation about employees using unapproved AI is usually framed as a *discipline problem*. Both framings are wrong in the same way: they skip the question that actually decides things – what kind of data is involved, and how much control that data needs. > **The spine of this post.** It isn't local versus cloud: the data class picks the level of control. There are four levels, not two. For local, the burden of proof is on local. And shadow AI is a product problem, not a disciplinary one. ## Four levels, not two > **Fig. 1** · Level 2 is the one people talk about least. The request doesn't go “to OpenAI”, it goes into your own cloud: your network, a private endpoint, a managed identity, your logs. Level 4 is what happens when none of the first three is offered. > > *Diagram:* Four levels of control, not two. Local or on-premises gives you control of everything. Your own cloud tenant gives you network, identity and audit logs and is the right default. A direct frontier API gives you a contract and retention terms. A consumer app on a personal account gives you nothing, and that is shadow AI. Once there are four levels, the useful question is no longer "which one is best" but "which data may go where". That needs a data classification, and it needs to be short enough to remember. > **Fig. 2** · An example policy, not the policy. The bottom-right corner matters most: pasting your own code into a personal chatbot and pasting customer data there are not the same offence. The first is a policy violation; the second is an incident. A rulebook that forbids both identically will see neither rule kept. > > *Diagram:* An example policy matrix of three data classes against the four levels. Public data is fine everywhere. Internal data is fine locally, in your tenant and over a contracted API, and a policy violation in a consumer app. Restricted data belongs locally or in your tenant if the contract allows, and pasting it into a consumer app is an incident. ## Local AI: the arithmetic Most disappointment with local models comes from three numbers nobody checked before buying hardware. All three follow from how a language model produces text: one token at a time, each after the other. (A token is roughly a syllable-sized piece, not a word – and languages like Hungarian need noticeably more of them per sentence than English.) ### Two speeds, and you always feel the worse one > **Fig. 3** · Prefill is the “thinking…” pause before the first character; decode is the classic tokens-per-second number. For chat, decode dominates. For a coding agent, prefill does – the answer is written quickly and the whole thing still feels slow. > > *Diagram:* A request has two phases. Prefill reads the prompt and is compute-bound. Decode writes the answer token by token and is bound by memory bandwidth. A chat turn is mostly decode. A coding-agent turn is mostly prefill, because the agent re-reads a large part of the codebase on every turn. > **Fig. 4** · What is liveable. Thinking tokens aren't read by anyone – they are pure waiting – which is why reasoning models raise the floor so sharply. It is also why the sweet spot of local AI is not chat but batch: work nobody is waiting for runs perfectly well at 5 tok/s. > > *Diagram:* Tokens per second and what they feel like: five to eight is human reading speed and painful in practice, fifteen to twenty is comfortable chat, and fifty or more is the minimum for reasoning models and coding agents. For batch work nobody is waiting on, five is plenty. ### Memory bandwidth is the key, not GPU muscle To produce each token, the model reads its active weights from memory. So decode speed is, to a first approximation, **memory bandwidth divided by the bytes read per token**. That one formula explains nearly everything you will measure. > **Fig. 5** · A dense 70B model at 4-bit is about 40 GB, and all of it is read for every token: 256 ÷ 40 ≈ 6 tok/s. Everyone measures about that – it isn't a driver problem, it is physics. A mixture-of-experts model activates only a fraction of itself per token, which is why a 120B MoE runs at 34–53 tok/s on the same hardware. Without MoE, the 128 GB desktop-box category would not exist. > > *Diagram:* On a machine with 256 gigabytes per second of memory bandwidth, a dense 70 billion parameter model must read about 40 gigabytes per token, which caps it at 6.4 tokens per second, and about 5 is measured. A 120 billion parameter mixture-of-experts model reads only about 3 gigabytes per token, a ceiling of about 85, and 34 to 53 tokens per second are measured. So the number to look for on a spec sheet is GB/s, not TOPS or core counts. The unified-memory boxes sit around 256–273 GB/s; an M3 Ultra is about 819; a discrete GPU is around 1,000 and up – with far less memory attached. Prefill, on the other hand, *is* compute, and there the GPU matters: on the same 120B model an early llama.cpp benchmark measured 1,723 tok/s of prompt processing on a DGX Spark against 340 on a Strix Halo, while decode differed by only about 13% (38.6 against 34.1 tok/s). Later builds narrowed the prefill gap to roughly 2×. For chat you would never notice. For an agent it is the difference between usable and not. ### What fits – and what that leaves room for Rule of thumb: at 4-bit quantisation, the usual default, **size in GB ≈ half the parameter count in billions**. 70B is 35–40 GB, 120B is 60–70 GB, 235B is 115–120 GB. But the weights are not the only tenant of that memory. Every open conversation keeps a KV cache that grows with context length – and every concurrent user has their own. > **Fig. 6** · Large model, long context, many users: pick two. And “it fits” is not “it is usable” – a 235B model does squeeze into 128 GB, at single-digit tok/s and with no room left for context. That is a demo, not a tool. > > *Diagram:* Memory budget on a 128 gigabyte machine running a dense 70 billion parameter model: 40 gigabytes of weights plus about 32 gigabytes of KV cache for every user with a 100 thousand token context. One user needs 72 gigabytes, two need 104, and three need 136, which does not fit. ### Concurrency: the number missing from every ROI slide One of these boxes serves one user comfortably, two with goodwill – not ten. Memory is one reason (Fig. 6). Prefill is another: it is compute-bound, so one person's 60,000-token prompt stalls everyone else's first token. And batching, which rescues throughput for dense models, helps less with MoE, because different users wake different experts and the bytes read per step go back up. Five developers sharing one box means queueing plus single-digit tok/s each. ### Total cost A capable box costs in the region of €4,000–5,000, plus electricity, plus someone to operate it. One person's cloud token spend is on the order of €25–100 a month. At the top of that range the hardware pays for itself in about 50 months – for one person, before power and operations. And the honest amortisation period is not five years but about 24 months: model architectures shift, and inference software keeps getting substantially faster on the hardware you didn't buy. The contrarian advice that follows: **if you don't have a real air gap, rent a dedicated GPU for a year instead of buying metal.** Buying makes sense for stable, high-volume batch work without bursts. ## Why local carries the burden of proof Local is the right answer for: a genuine air gap; a contract that forbids sub-processors; raw health or legal data; high-volume, low-sensitivity batch; offline and edge deployments, where the missing network is the deciding factor rather than secrecy; and fine-tuning. It is the wrong answer for "it feels safer". With local AI the attack surface doesn't disappear – **it moves in with you**. An unpatched internal inference server without authentication is a larger risk than Azure OpenAI behind a private endpoint. Local needs the stronger justification not because it is technically worse, but because reversibility is asymmetric. > **Fig. 7** · A bad cloud decision is a monthly invoice. A bad local decision stands in the server room, signed – internally a matter of prestige, externally a reference risk. > > *Diagram:* Reversibility is asymmetric. A wrong cloud decision is operating expense: you switch or cancel and it is undone in about two weeks. A wrong local decision is capital expense: a signed purchase order and an asset tag that are still in the server room 24 months later. Before the purchase order, have answers to these: 1. The **concrete use case**, and the data class it involves. 2. How many **concurrent** users – measured, not guessed. 3. The **typical prompt size**, because that decides whether prefill or decode is your bottleneck. 4. Whether there is an **eval**. If there is none, the project has already failed: you will have nothing to defend it with, and nothing to refute it with. 5. Who **operates** it – a name, not a team. 6. The **exit plan**. ## The model is a file; the rest is harness A model really is a file. But what a user experiences as "the model" is almost entirely what sits around the weights – and that is where local set-ups succeed or fail. > **Fig. 8** · Same weights, different harness, different product. Locally it is rarely the model's quality that lets you down. It is the tool-call parser, or a chat template that doesn't match the one the model was trained with. > > *Diagram:* The model weights are one file. Around them sits the harness: the chat template that formats the prompt, the inference engine with batching and the KV cache, structured output, and the tool-call parser. What users experience as the model is mostly the harness, and in local set-ups the tool-call parser is what breaks first. - **Engines.** Ollama and LM Studio are developer convenience for one user. vLLM and TensorRT-LLM are production serving, with continuous batching. Whoever plans to serve twenty people from Ollama started in the wrong place. (Licensing, since it comes up: LM Studio has been free for work use since July 2025; you just can't redistribute or embed it.) - **Quantisation** is not "a slightly dumber model". Benchmark scores barely move at 4-bit – but tool calling, JSON output and long-context instruction-following degrade first. Precisely what agentic use needs, and precisely what nobody measures while quoting MMLU. ### Chinese open-weight models: split the question in three | Question | Answer | | --- | --- | | Does data go to China? | Run locally: no. Weights plus arithmetic – a GGUF file does not phone home. | | Is the licence a problem? | The reverse of what people assume: Qwen and DeepSeek are largely Apache 2.0 / MIT, more permissive than the Llama community licence. | | What remains? | Bias on certain topics, and supply chain: where you downloaded it, checksums, who produced the quantised build and the chat template. | The risk isn't in the model; it is wherever you call an API. The same DeepSeek on its hosted chat and on your own hardware are two entirely separate risk profiles. ## The LLM provider is an adapter For a boxed, on-premises product there is no "we decide". The customer decides, and every customer decides differently. So the LLM provider is an adapter – exactly like the database, the login and the file store in [the micromonolith post](https://laszlonemes.com/blog/micromonoliths). The wire protocol is already solved: the OpenAI-compatible `/v1/chat/completions` is a de facto standard across Azure, vLLM, Ollama, LM Studio and OpenRouter. The hard part is what is *not* uniform behind it: how reliable tool calling is; structured output (real constrained decoding versus "please return JSON"); usable versus nominal context; content filtering (present on Azure, absent locally); rate-limit semantics and retry hints. > **Fig. 9** · One adapter per level a customer might choose – their tenant (level 2), a local engine (level 1), a direct API (level 3). Design by capability discovery, not by if-statements on provider names. Degrade gracefully, and keep a zero-config default so the product starts before anyone has chosen a provider. > > *Diagram:* The LLM provider as an adapter. The application talks to one port. Adapters for Azure OpenAI, vLLM or Ollama, and a frontier API all speak the same chat completions protocol, and each reports its capabilities. The application branches on capabilities, not provider names: with constrained JSON it uses schema-enforced decoding, without it it prompts for JSON, validates and retries. ```csharp public enum ToolCalling { None, BestEffort, Reliable } public sealed record LlmCapabilities( ToolCalling ToolCalling, bool ConstrainedJson, // can the engine enforce a JSON schema while decoding? int UsableContextTokens, // what actually works, not what the model card says bool ContentFilter); // present on Azure, absent on a local engine public interface ILlmProvider { LlmCapabilities Capabilities { get; } Task ChatAsync(ChatRequest request, CancellationToken ct); Task ChatJsonAsync(ChatRequest request, JsonSchema schema, CancellationToken ct); } ``` ```csharp // branch on what the provider can do, never on what it is called public async Task GetJsonAsync(ChatRequest request, CancellationToken ct) { if (provider.Capabilities.ConstrainedJson) return await provider.ChatJsonAsync(request, JsonSchema.For(), ct); for (var attempt = 1; attempt <= 3; attempt++) { var reply = await provider.ChatAsync(request.WithJsonInstructions(), ct); if (JsonValidator.TryParse(reply.Text, out var value)) return value; request = request.WithFeedback("The reply was not valid JSON for the schema. Try again."); } throw new LlmOutputException("No valid JSON after 3 attempts."); } ``` ### The lock-in is the embedding model, not the chat model > **Fig. 10** · The chat model is a configuration decision. The embedding model is a schema decision. You can replace the chat provider tomorrow; replacing the embedding model means re-indexing everything. > > *Diagram:* Swapping the chat model is one line of configuration and a restart. Swapping the embedding model is a migration: an index built with 3072-dimensional vectors cannot be queried with a 1024-dimensional model from a different vector space, so every document must be re-embedded and the index rebuilt. Therefore the embedding model's name and dimension are **part of the schema**, and the index records what built it: ```json { "index": "documents-v3", "embeddingModel": "text-embedding-3-large", "dimensions": 3072, "chunking": "v2" } ``` The hidden cost of all this flexibility: every supported provider needs its own regression eval. Three providers is not three times the code – it is three times the evals and three times the support. ## Your own cloud tenant My own default, for what it's worth: internal tools run on Azure OpenAI in our own tenant – a GPT model plus `text-embedding-3-large` for vectorisation. Even in a strongly technical team, "bring it on-prem" is not the default. And this is exactly where the embedding lock-in bites: if we moved local tomorrow, we wouldn't be swapping a chat model, we would be rebuilding the whole index. > **Fig. 11** · The four levels from Fig. 1 again, this time as the path one prompt takes. Level 2 is the only one that gives you cloud models and a boundary you own: private networking, no keys in code, role-based access, a content-filter level you choose, token quotas per team as cost control – and every prompt loggable, which is both your audit trail and your best weapon against shadow AI. > > *Diagram:* The same prompt travelling through the four levels. Level one: it never leaves your building and reaches an inference server on your own hardware. Level two: inside your cloud tenant, the application uses a managed identity and a private endpoint to reach your model deployment, and the logs are yours. Level three: it crosses the public internet with an API key to the provider, whose logs you do not hold, under a contract. Level four: an employee's browser with a personal account sends it to a consumer app with no contract at all. The downsides to accept knowingly: new model versions arrive later; model availability per region is a lottery; quota administration is real work; and your SLA hangs on their capacity. ### The region argument – say it the right way The wrong way: *"It's in West Europe, so the data never leaves Europe, so GDPR is covered."* A data-protection officer or a lawyer shoots that down in one sentence, and everything you say afterwards is suspect. > **Fig. 12** · Data residency is not data sovereignty. The CLOUD Act follows the company, not the data centre. > > *Diagram:* Data residency versus data sovereignty. Choosing an EU region fixes where data is stored and processed, which is residency: documented and auditable. It does not give sovereignty, because the CLOUD Act binds the US parent company wherever the hardware stands. > An EU region determines where the data is physically stored and processed – **data residency**. That has real value under GDPR: processing stays in the EU, the provider's role as processor is fixed in the DPA, retention is governed by terms you can see, and the whole arrangement is **documentable and auditable**. What an EU region does *not* give you is **data sovereignty**: the CLOUD Act binds the US parent company regardless of where the hardware stands. So the right sentence is not "the data never leaves Europe" but "the risk is known, documented and acceptable – and somebody signed it". - **A defensible baseline.** DPA, EU region, controlled retention and a documented data flow are enough for the overwhelming majority of use cases. Supervisors don't expect zero risk; they expect a reasoned, written decision. - **Latency and cost.** Not a compliance argument at all – an engineering one, and sufficient on its own to justify the region. - **The asymmetry, reversed.** If everyone flees into an air gap because of the CLOUD Act, the data ends up on a badly run server of your own. The CLOUD Act is a theoretical risk. The unpatched internal server is a practical one. What "region" does not cover, and you must check separately: storage location and processing location are *separate* fields; retention (on Azure, abuse-monitoring retention can be switched off for approved use cases); sub-processors; support staff access; governing law. That is why it can't be settled in one sentence – and also exactly what makes it documentable. If someone pushes further, the next steps are a sovereign cloud (Bleu, Delos, the AWS European Sovereign Cloud) or genuine on-prem; say out loud that this is a narrow, regulated group, not the base case. *None of this is legal advice* – it is the wording a DPO doesn't reject after the first sentence. ## Two ways a RAG system leaks > **Fig. 13** · The most common enterprise RAG failure is a bot that tells people things they have no right to see. Security trimming is the first feature, not an option. And if the model goes local, the embedding model and the vector store go with it. > > *Diagram:* Two ways a retrieval system leaks. First, without security trimming the index ignores the permissions the file share enforced, so the bot tells an intern what is in a board document. Second, a model moved on-premises while embeddings are still computed by a cloud API sends exactly the sensitive text out. The underlying mistake in both panels is the same: forgetting that **the index is itself a data class**. It is exactly as sensitive as its sources, and it rarely carries the same permissions. ## Shadow AI | Finding | Source | | --- | --- | | Employees regularly using AI on corporate devices: 15% → 45% in one year. Shadow AI is now the third most common non-malicious insider action; source code is the data most often uploaded. | Verizon DBIR 2026 | | 66% of office professionals used AI at work while believing it was not permitted. 88% shared work information with public AI tools; 34% shared customer data. | PagerDuty 2026 | | Shadow AI was a factor in 20% of breaches and added about $670,000 to the average cost. 97% of organisations with an AI-related incident lacked proper AI access controls. | IBM, 2025 | | 47% of generative-AI users in enterprises reach the tools through personal accounts. | Netskope, 2026 | | About 11% of what employees paste into chatbots is confidential. | Cyberhaven, 2023 | Why it happens is no mystery – it is Shadow IT again. Nobody spun up a VM behind the device-management tooling out of malice; they wanted to get work done. The difference is that the gap between the good tool and the bad tool used to be 10%, and now it is a multiple. Under that much pressure, policy does not hold the wall. What is missing is not prohibition but an alternative. > A ban is not control. All a ban achieves is that you can no longer *see* what is happening. > **Fig. 14** · The forms that matter now. MCP servers are the most serious because an agent doesn't just read – it acts. “Sanctioned” shadow AI is the most embarrassing: Notion AI, Slack AI, Zoom AI – procurement signed it, security doesn't know, nobody turned it off. > > *Diagram:* Four newer forms of shadow AI around a developer's laptop: a personal API key in the IDE or CLI, unaudited MCP servers through which an agent acts on databases, mail and files, browser extensions with full page access, and sanctioned shadow AI, the AI features inside already-purchased SaaS tools that nobody reviewed. ### Detection: don't start with DLP Start with what you already have: DNS and proxy logs; a review of the **OAuth grants in your SSO** (ten minutes, startling results); personal subscriptions on company cards. Only then CASB and DLP. ### What works > **Fig. 15** · People take the faster road. You cannot out-policy that; you can only make the official road faster. > > *Diagram:* Someone needs AI for a task. The official road is a request followed by a six-week vendor review. The detour is a personal account that takes two minutes, and the work gets done invisibly with no log, contract or control. If the official road is slower than the detour you have lost, so make approval take one day. 1. **A paved road that is genuinely fast** – one day, not a six-week vendor review. 2. **Three data classes, not seven.** If you can't recite it, it is too complicated (Fig. 2). 3. **An amnesty with a time window** – "tell us now; in two weeks it becomes a problem". 4. **Measure usage, not compliance.** If training-completion rates go up, you have won nothing. ### Who is to blame Everyone, asymmetrically. The company more: it holds the decision, the money and the procurement process, and if it provides no tool for six months, that was its choice. But the developer carries professional responsibility – "the process was inconvenient" is not a defence in a GDPR incident. And the distinction from Fig. 2 stands: pasting your own code is a policy violation; pasting customer data is an incident. ## The AI Act, in four lines > **Fig. 16** · The Digital Omnibus was agreed in May 2026 and formally adopted in June. It moved the high-risk dates. It did not move Article 50. > > *Diagram:* Three EU AI Act dates: on 2 August 2026 the Article 50 transparency obligations apply and were not postponed, on 2 December 2026 the grace period for marking AI-generated content ends for systems already on the market, and on 2 December 2027 the postponed high-risk obligations apply. The practical consequence for anyone shipping AI features: people must be able to tell when they are talking to an AI, and AI-generated content must be identifiable as such – and that duty sits with the deployer as well as with the model vendor. ## Objections, and the answers | They say | You say | | --- | --- | | "Local AI is the privacy solution." | It is the control solution. Privacy is a process, not a box. | | "One strong box is enough for the team." | One box is one or two concurrent users. Let’s look at the arithmetic. | | "Cloud is more expensive." | Only if operating hours cost nothing and you amortise over five years. | | "Chinese model = data leak." | Not when it runs locally. The risks are bias and supply chain. | | "EU region = GDPR ticked." | Residency is not sovereignty. But DPA + EU region + controlled retention is defensible. | | "Ban it until there is a policy." | A ban removes visibility, not usage. | | "Let’s buy the hardware and figure out what for." | The burden of proof runs the other way: use case, eval, concurrency – then the PO. | | "We’ll just swap the model later." | The chat model, yes. The embedding model means a full re-index. | ## Takeaways 1. **Data class → control level.** Four levels; your own cloud tenant is the right default. 2. **tok/s ≈ bandwidth ÷ bytes read per token.** Read GB/s on the spec sheet; prefill is the exception that needs compute. 3. **Large model, long context, many users: pick two.** One box is one or two concurrent users. 4. **Local must justify itself** – use case, eval, measured concurrency, named operator, exit plan – because capex is hard to undo. 5. **What you call "the model" is mostly the harness.** Quantisation hurts structured output first. 6. **Make the provider an adapter, branch on capabilities,** and treat the embedding model as schema. 7. **Residency is not sovereignty** – say the defensible sentence, not the comfortable one. 8. **Shadow AI is a product problem.** Build a road faster than the detour, and keep the logs. ## Sources - [Verizon Data Breach Investigations Report 2026](https://www.verizon.com/business/resources/reports/dbir/) – AI usage on corporate devices, shadow AI as an insider action. - [PagerDuty Shadow AI Survey 2026](https://www.pagerduty.com/newsroom/shadow-ai-workplace-survey-2026/) – 1,250 office professionals outside IT, at companies above $500M revenue. - [IBM Cost of a Data Breach Report 2025](https://www.ibm.com/reports/data-breach) – the shadow-AI breach premium. - [Netskope Cloud and Threat Report 2026](https://www.netskope.com/netskope-threat-labs/cloud-threat-report) and [Cyberhaven (2023)](https://www.cyberhaven.com/blog/4-2-of-workers-have-pasted-company-data-into-chatgpt) – personal accounts and pasted data. - [Hardware Corner: DGX Spark vs. Strix Halo](https://www.hardware-corner.net/first-dgx-spark-llm-benchmarks/) and the [llama.cpp DGX Spark thread](https://github.com/ggml-org/llama.cpp/discussions/16578) – the prefill and decode measurements. - [LM Studio is free for use at work](https://lmstudio.ai/blog/free-for-work) – the July 2025 licence change. - [Gibson Dunn on the AI Act Omnibus agreement](https://www.gibsondunn.com/eu-ai-act-omnibus-agreement-postponed-high-risk-deadlines-and-other-key-changes/) and the [Article 50 guide](https://artificialintelligenceact.eu/transparency-rules-article-50/) – the dates in Fig. 16. - [CLOUD Act](https://en.wikipedia.org/wiki/CLOUD_Act) – why jurisdiction follows the provider. --- # Loop engineering is a control problem > What people mean by "loops" around coding agents, why the version that works in a team is a fifty-year-old control loop with an agent as the actuator, and how to build one from a linter, a cron job and pull requests. Source: https://laszlonemes.com/blog/loop-engineering · Published: 2026-09-20 · Author: László Nemes September 20, 2026 · 20 min read `AI engineering` `Coding agents` `Control loops` Kyle Mistele opens his AI Engineer World's Fair talk with a picture every team using coding agents will recognise: a 40,000-line pull request, surrounded by review agents, that no human wants to read. Both halves deserve to be taken seriously. The 40,000 lines really do exist – that is a year of genuine engineering progress in agents and harnesses. And nobody can review them, so whatever is wrong in there is wrong quietly. The hype compresses the current mood into three lines: *build loops, spend as many tokens as possible, don't read the code.* I have no problem with the first two. This post is about why the third one fails, and what the version that works in a team looks like. We recorded a (Hungarian) [Letscode.hu](https://laszlonemes.com/about) episode about it; this is the written version of my notes, built mostly on [Mistele's talk](https://ai.engineer/talks/xIt_mTQp6mY-loop-engineering-from-first-principles), Geoffrey Huntley's original Ralph post, HumanLayer's write-ups and the Bun team's Rust rewrite. > **The thesis:** loop engineering is not a paradigm shift, it is a control problem. It works where the desired end state is measurable, each step is small and reversible, and a human stays on the loop. Where those are missing, a loop doesn't speed you up – it manufactures a time bomb, with interest. > **Fig. 1** · Where loop engineering sits. The levels don't replace each other: whoever writes a loop is still a prompt and context engineer, just optimising for n runs instead of one. If the prompt is bad, the loop is wrong n times. > > *Diagram:* Four levels stacked as stairs: prompt engineering, context engineering, agentic coding, loop engineering. Each level stands on the previous ones. ## Three things people call a loop Three different things currently share one name. If you don't pull them apart, every conversation about "loops" blurs. - **The agentic loop** is the inner loop, and it is always running. Inside Claude Code, Codex, Copilot CLI or any other agent the same cycle turns: gather context → act → verify, and on failure go around again with the new information. That isn't a technique, it *is* the agent. "I use an agentic loop" means "I use an agent". - **The Ralph loop** is an outer loop: keep re-running the agent with the same prompt until the result matches what you asked for. Geoffrey Huntley published it in July 2025, and the original implementation is one line of bash. - **The control loop** is not an AI concept at all. It is control engineering: measure, compare with a target, correct a little, measure again. ```shell # Geoffrey Huntley's original, July 2025 while :; do cat PROMPT.md | claude-code ; done ``` > **Fig. 2** · The first two loops, nested. Every Ralph iteration starts a brand-new agent with the same prompt; the agentic loop runs inside it. Nothing survives between iterations except what was written to disk – which turns out to be the clever part. > > *Diagram:* The Ralph loop is an outer shell loop that feeds the same prompt to a fresh agent until the model says it is done. The agentic loop runs inside every agent. Files and git are the only memory between iterations. Practical Ralph setups add guard rails – this one is from Matt Pocock's getting-started guide, lightly trimmed: ```shell # afk-ralph.sh for ((i=1; i<=$1; i++)); do result=$(docker sandbox run claude --permission-mode acceptEdits -p "@PRD.md @progress.txt \ 1. Find the highest-priority task and implement it. \ 2. Run your tests and type checks. \ 3. Update the PRD with what was done. \ 4. Append your progress to progress.txt. \ 5. Commit your changes. \ ONLY WORK ON A SINGLE TASK. \ If the PRD is complete, output COMPLETE.") if [[ "$result" == *"COMPLETE"* ]]; then echo "PRD complete after $i iterations." exit 0 fi done ``` Each detail is there for a reason. The **sandbox**, because this runs unattended with edit permissions. The **iteration cap** is a cost limit, not elegance. `ONLY WORK ON A SINGLE TASK` is the only defence against scope creep. And the stop condition is a **string match** on something the agent itself prints. Remember that last one. ## The control loop, with or without AI A control loop has six parts, and all six matter: a **set point** (the desired state), a **sensor** that measures where you are, the difference between the two – the **error**, meaning deviation rather than bug – a **controller** that decides what and how much to change right now, an **actuator** that applies the change, and the part people forget: **disturbances**, because the system keeps living while you correct it. > **Fig. 3** · The textbook loop as a thermostat. The controller never trusts yesterday's reading: every cycle starts with a fresh measurement, which is the only reason an open window gets noticed. > > *Diagram:* A classic control loop, shown as a thermostat: the set point of 21 degrees is compared with the sensor reading of 19 degrees, the controller turns the 2 degree error into a correction, the boiler applies it, and an open window disturbs the room. Software is already full of these. We just never called it loop engineering, because there was no AI in it: | | Thermostat | Kubernetes ReplicaSet | Agentic control loop | | --- | --- | --- | --- | | Set point | 21 °C | `replicas: 3` | "0 legacy `.handler()` procedures" | | Sensor | thermometer | controller watching observed state in the API server | ast-grep scan of the repo | | Error | 2 °C too cold | 2 pods running, 3 wanted | 150 violations | | Controller | how long to heat | ReplicaSet controller: create 1 pod | pick 1 violation for this run | | Actuator | boiler | scheduler + kubelet start the container | coding agent + skill → a pull request | | Disturbance | open window | a node dies | teammates merging other work | Terraform's plan/apply, PostgreSQL's autovacuum and React's reconciliation follow or approximate the same shape. Which answers the sceptical question – "aren't these loops just workflows that now have agents in them?" – with: yes, and that is exactly why this isn't a paradigm shift. > The architecture is fifty years old. The one new thing is that **the actuator became general-purpose**: until now, a step in the loop had to be something you could write as code. Now it can be anything you can describe in text. That is a big deal – but the architecture stayed. ## Ralph loop vs. control loop Put side by side, the two loops differ in one place that decides everything else: **who measures**. > **Fig. 4** · In a Ralph loop the model grades itself: it decides it is done, against tests it wrote. In a control loop the measurement comes from an external sensor that has nothing to do with the agent, and the loop stops when that sensor finds nothing. Here the end state is written down as the sensor's rules, so every finding is already a piece of the error; Fig. 7 draws the comparison explicitly. > > *Diagram:* Side by side: in a Ralph loop the same model that writes the code judges whether it is done. In a control loop an external sensor measures the codebase, a controller picks one unit, and the agent's output is a small reviewed pull request. If the sensor and the actuator are the same model, you aren't measuring – you are fooling yourself. The autonomous-codebase literature calls it *evaluation gaming*: a self-grading system reports success without quality. It is the old mistake of optimising for your own tests, except the tests are rewritten every iteration too. | | Ralph loop | Control loop | | --- | --- | --- | | Who measures | the model, itself | an external, independent sensor | | End state | "the PRD looks done to me" | a machine-checkable assertion | | Stop condition | string match + iteration cap | the measured error reaches zero | | Step size | whatever the agent feels like | decided by the controller – a tuning parameter | | Disturbances | assumes it owns the repo | built in: the team's ongoing work | | When it is wrong | you find out at the end | every iteration, in a small PR | | Worst case | overbaking, silent drift | no-op: nothing happens | That last row is underrated. A control loop with nothing to correct does nothing. Ralph has no such mode – it always does *something*, because that is its job. Huntley describes Ralph's output as under-baked, baked, or "baked with unspecified latent behaviours": an agent that cannot tell when it is finished will find itself plausible-sounding work nobody asked for. ### This is not a contest Ralph discovered something real – just not what most people think. The trick was never the infinite loop. When an official Ralph plugin shipped in December 2025, Dex Horthy's critique was that it "misses the key point of ralph which is not 'run forever' but in 'carve off small bits of work into independent context windows'". Ralph doesn't scale with time, it scales with **decomposition**: every iteration gets a small, clean context and reads what it needs from files, instead of dragging thirty steps of debris along. > **Fig. 5** · Schematic. Horthy's rule of thumb is that somewhere past roughly 40% context utilisation an agent enters the 'dumb zone' – a working threshold, not a hard cliff. It is why a fresh run beats a long conversation, and why a bigger window doesn't fix it: this is an architecture problem, not a capacity problem. > > *Diagram:* One long agent session fills its context window far past the point where quality starts to slip, while a loop gives every iteration a fresh, mostly empty context and carries state in files. From here the control loop is one step away. If the work has to be carved into small independent pieces anyway, *something* must decide which piece is next. In Ralph the agent decides, by instinct, again in every iteration. In a control loop a separate component decides, from outside, based on measured data – that is the controller. Mistele says as much himself: the best Ralph implementations already apply control principles. > The control loop doesn't refute Ralph. It finishes it. A Ralph loop is a `while`; a control loop is a regulator. One hopes, the other measures. ## Why the steps are small "One unit per run" is not there because the agent couldn't handle more. How much to change per iteration is the controller's core job, and anyone who has tuned a PID controller knows both failure modes: correct too timidly and you never arrive; correct too aggressively and the system overshoots and oscillates instead of converging. Mistele makes the same point about code: repeated oversized or incorrect corrections quickly destabilise a codebase. > **Fig. 6** · Same total work, two step sizes. The big correction is measured once, at the end, against a codebase that moved underneath it. Each small correction is measured, read by a human and merged before the next one starts. > > *Diagram:* One big correction versus many small ones. A single long run produces a 40,000-line pull request that review cannot read while teammates keep merging. Small corrections are measured, reviewed and merged one at a time. HumanLayer's own history has a data point for the upper lane. One early Ralph experiment: 30 minutes writing a coding-standards document, 30 more refining it with a senior engineer, then a 6-hour autonomous refactor. The team liked the result – and the PR was never merged, because of conflicts. What worked better for them afterwards was a nightly run that lands one small refactor each morning. The realistic multiplier is also worth noting: an hour of human context-building bought six hours of machine work, not "say one sentence and it's done". ## Building one: eight steps What follows isn't a proposal. It is the system HumanLayer runs on its own production monorepo, as shown in the talk: incrementally migrating oRPC procedures to [Effect](https://www.effect.website/docs/v3/getting-started/introduction), with real PRs and real review comments. The snippets are trimmed from the slides. Most people start the other way round – point an agent at something, like the result, then "automate it" – and that order is how you get the 40,000-line PR. > Build the thing that measures where you are first, and only then the thing that does the work. > **Fig. 7** · The whole system on one page. Blue is the measurement and steering path, red is everything the loop doesn't control. The rest of this section builds it box by box. > > *Diagram:* The agentic control loop: a desired end state is compared with what an ast-grep sensor measures in the codebase. The controller turns 150 violations into a control signal of one procedure, an agent with a skill commits the fix, a human reviews the pull request, and a CI ratchet dampens disturbances from teammates. Review comments flow back into a feedback file that every run loads. ### Step 0 · Define the end state One sentence that can be decided for any given file. Here: no classic `.handler` procedure remains in the API apps. ```ts apiBase.procedure.handler(async () => { /* … */ }) // legacy: what the sensor looks for apiBase.procedure.effect(function* () { /* … */ }) // end state ``` What you *cannot* use as an end state: "make the code better", "reduce tech debt", "make it faster". Those are directions, not set points – no sensor can measure them. If you can't say in one sentence when it's done, you don't have a loop. You just started something. ### Step 1 · Build the sensor The most important component is also the most boring, and its most surprising property is that it isn't AI. It is a rule file: one rule describes a procedure that hasn't been migrated, another one that was migrated wrong. rules.yml ```yaml # "we haven't done this one yet": normal, it is why the loop runs id: orpc-unmigrated-procedure language: typescript severity: warning message: Classic .handler procedure, not yet migrated to Effect. rule: pattern: $BASE.handler($_) --- # "we did this one wrong": an error, this stops the build id: orpc-await-in-generator language: typescript severity: error message: await is not allowed in Effect generators rule: pattern: $BASE.effect($_) has: { pattern: await $_, stopBy: end } files: - 'apps/riptide-api/**/*.ts' - 'apps/admin-dashboard-api/**/*.ts' ignores: - 'vendor/**/*' ``` ```shell # ~50 keys per finding down to 4, in an order that never changes between runs ast-grep scan -r rules.yml --json \ | jq '[.[] | {ruleId, file, message, severity}] | sort_by(.severity, .file, .ruleId)' ``` - **Two severities.** "Not done yet" is a warning – it is why the loop runs. "Done wrong" is an error – it stops the build. One rule file serves a process and a gatekeeper. - **Scoped.** `files` and `ignores` pin the rules to the apps being migrated. Without that the sensor measures noise, not signal. - **Deterministically sorted.** If the sensor returns the same findings in a different order each run, the controller picks something different every day and you will never be able to say why the loop did what it did. The `sort_by` is what makes it debuggable. - **Outside the agent's reach.** Mistele keeps these rules out of the ESLint and TypeScript config on purpose: agents like to silence a check with an inline comment. Whatever tool you use, treat suppressions as findings too. - **Fast** enough to run in every iteration. The talk's second sensor, Aiden Bai's react-doctor, scans 231 files in 5.7 seconds because it sits on a Rust linter. react-doctor is interesting for another reason: how it words findings. Not "rule violated on line 64", but *"Your users see a stale value when prop currentDefault changes because useState copies it once."* That text goes straight into the agent's context, and a consequence is a much better control signal than a rule number: it tells the agent what to *preserve*, not only what to avoid. Sensors can also be agentic (agent + skill + natural-language rules) or hybrid – what matters is that the thing that measures is not the thing that was just asked to fix. ### Step 2 · Stabilise the system: the ratchet Before fixing anything, stop it getting worse. The goal isn't "everything is migrated today", it is "tomorrow there are no more violations than today". Commit a snapshot of the current findings – one `(file, rule)` key per line – and let CI compute a set difference on every pull request. > **Fig. 8** · The ratchet only turns one way. It ignores existing debt, fails on new debt, and needs no AI: this is the one idea from this post you can ship tomorrow morning and benefit from by the afternoon. > > *Diagram:* The ratchet compares the findings on main with the findings in a pull request. A finding that disappeared is progress and always allowed. A finding that is new fails the build. .github/workflows/no-regressions.yml (excerpt) ```yaml - name: Fail on NEW (file, rule) findings vs the main snapshot run: | ast-grep scan -r "$RULES" "$SCAN_PATH" --json=compact \ | jq -r '.[] | "\(.file)\t\(.ruleId)"' | sort -u > current.keys sort -u "$BASELINE" > base.keys new="$(comm -13 base.keys current.keys)" # only in current = newly introduced if [ -n "$new" ]; then echo "::error::New migration regressions introduced:" echo "$new" exit 1 fi ``` In control terms this is the disturbance dampener: the loop fixes, and the ratchet makes sure the rest of the team can't undo its progress in the meantime. ### Step 3 · Build the controller The controller answers "what, and how much, right now?" – and it can be the dumbest thing in the world. > **Fig. 9** · Three controllers over the same sensor output (example numbers). With a data-driven controller the loop proceeds by business priority rather than alphabetical order, and the agent also receives the trace samples and errors for the procedure it was handed. > > *Diagram:* Three controllers read the same sorted sensor output and pick different units: the deterministic one takes the first row, the agentic one the smallest procedure, the data-driven one the procedure with the most production errors. The deterministic version – `head -1` on a sorted list – contains no AI and works. The lesson generalises: not every component has to be an agent. Good systems separate the deterministic part from the part that needs judgement, and in most places deterministic is enough. One agent may even play controller and actuator in a single context window; the *responsibilities* stay separate, and limiting the size and direction of the change remains the controller's. ### Step 4 · Build the actuator Your coding agent, plus a skill describing how this kind of change is done here. HumanLayer doesn't try to perfect the skill upfront; they evolve it from observed results and feed it **golden patterns** – handwritten, idiomatic examples of what the repository expects. Two details are easy to miss. The skill ends with a **final-answer template**, and the agent's last message becomes the PR description – no separate summariser. And the agent **commits but never merges**: pushing and opening the PR are deterministic workflow steps, and the loop's output is always a pull request waiting for review. ### Step 5 · Build the loop Here it becomes almost disappointingly ordinary. Your CI already has repository access, secrets, scheduling and manual dispatch. The loop is a cron-triggered workflow that runs one sense → control → actuate iteration. No orchestration platform, no multi-agent framework. .github/workflows/loop-effect-migration.yml (trimmed) ```yaml name: "Loop: Migrate oRPC Procedure to Effect" on: schedule: - cron: '0 14 * * 1-5' # weekdays, 6am Pacific workflow_dispatch: permissions: contents: write pull-requests: write jobs: iterate: runs-on: ubuntu-latest steps: # flow control (step 7), checkout, install deps, create a branch - name: Sensor - scan codebase for violations run: ast-grep scan -r rules.yml --json > findings.json - name: Controller - select the control signal run: echo "CONTROL_SIGNAL=$(./run_controller.sh findings.json)" >> "$GITHUB_ENV" - name: Actuator - apply the change with an agent env: ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }} run: | npx @humanlayer/cli@latest codelayer --prompt " $(cat .agents/skills/effect-migration/SKILL.md) $(cat .github/agent-memory/effect-migration.md) Migrate the specified oRPC procedure to Effect: $CONTROL_SIGNAL " > agent_output.md # deterministic: commit, push, open a labelled PR with agent_output.md as the body ``` > **Fig. 10** · One scheduled run. Both early exits are no-ops, which makes the workflow idempotent: it is safe to trigger at any time, because when there is nothing to do, it does nothing. > > *Diagram:* One scheduled run: cron starts the workflow, a work-in-progress check exits if the loop already has an open pull request, the sensor scans, the run exits if nothing is found, otherwise the controller picks one unit, the agent fixes it and deterministic steps open one small pull request. The result is one pull request a day: small, readable, labelled, opened by a bot, review required. That is what works in a team – not the 40,000-line PR, but one you can read on the tram home. ### Step 6 · Put a human on the loop On the loop, not in it. HumanLayer's first version was frustrating: correcting the output meant checking out the branch, editing the skill, fixing the code, committing, pushing. The migration was small; steering the automation was the expensive part. The fix is to let the pull request be the steering wheel. > **Fig. 11** · A reviewer's /iterate comment produces two writes: a fix for this PR and a standing rule for all future ones. The handler filters on the loop's label, so it doesn't react to every comment in the repo. > > *Diagram:* A reviewer's /iterate comment on a bot pull request is handled by the agent, which pushes a fix to that pull request and also appends a rule to a feedback file that every future run loads. The comment becomes a rule instead of a one-off fix. The example from the talk: a reviewer asks that successful DB lookups get logged too, "for this procedure and future ones". The agent fixes the PR and appends the lesson to a Markdown file in version control: .github/agent-memory/effect-migration.md ```diff ## Guidance - Keep migrations small and reviewable: migrate only one eligible procedure per run. + - Co-locate success logging with the DB lookup, mirroring the error path. Inside the + query .pipe(...), add Effect.tap(... Effect.logInfo(...) ...) alongside the + Effect.tapError(... logError ...) calls, rather than a separate logInfo after the query. ``` This is where "fix the process, not the code" becomes a file operation. A review comment stops being a one-off repair and becomes a rule, so review effort **amortises** instead of growing linearly with the number of PRs. And because the file lives in git, the way the loop's behaviour evolves is itself reviewable and revertible. ### Step 7 · Add flow control A scheduled loop can outrun its reviewers even when every change is small – people travel, visit customers, have other work. The label gives you cheap backpressure: before checkout, before installing anything, before any tokens are spent, ask whether this loop already has an open PR. ```shell # Scheduled runs no-op when this loop already has an open PR. if [ "$EVENT_NAME" = "schedule" ]; then OPEN=$(gh pr list --repo "$GITHUB_REPOSITORY" --state open \ --label "$AGENT_LABEL" --json number --limit 1 | jq 'length') if [ "$OPEN" -gt 0 ]; then echo "Open PR already exists for $AGENT_LABEL; no-op." echo "run_agent=false" >> "$GITHUB_OUTPUT" exit 0 fi fi ``` This is a kanban WIP limit for loops. Without it you wake up to forty open PRs and the team stops; it is the difference between a loop and spam. Several loops can run side by side, each with its own label and its own limit – one for the migration, one for react-doctor findings, one for something else. Want to go faster? The talk's own arithmetic: about 150 procedures at one per working day is roughly six months. You can raise the schedule frequency, let the controller select three or five units (ideally each migrated in its own context window), or run the workflow several times and assign each teammate one PR. The last option is the telling one: > A loop's throughput is set by **review capacity, not token budget**. Money for a hundred runs is useless if the team can absorb three PRs a day. ## The same loop at industrial scale The strongest real-world reference is Bun's rewrite from Zig to Rust, because it is not a demo app: 11 days in May 2026, 6,502 commits peaking at 695 per hour, about 64 parallel Claude instances across 4 worktrees, roughly $165,000 at API prices, a +1,009,272-line diff over 1,448 files, zero tests skipped or deleted, 128 bugs fixed against the previous release and a ~20% smaller binary. The viral summary – "three engineers, a year of work, done for under $200k" – has the number right. Look at what is underneath it. The rewrite existed because of one bug class. In Jarred Sumner's write-up: use-after-free, double-free and forgot-to-free-on-an-error-path bugs are, in safe Rust, compiler errors – and "compiler errors are a better feedback loop than a style guide". AI didn't solve memory management; the type system did. The agents did the mechanical work. Map it onto Fig. 7: the **compiler was the sensor**, the ~16,000 compile errors after the mechanical port were the **error signal**, handed out crate by crate as a work queue, and the 64 agents were the **actuators**. Every phase repeated until its own target number was met. The warning follows directly. If you have something rewritten into a language or framework that has *no sensor* for your bug class, there is nothing behind you. The rewrite is one large, unverified diff. The viral posts never mention that difference. > **Fig. 12** · Bun's review set-up per file: one implementer, two adversarial reviewers, one fixer. The value isn't that more eyes see more – it's that different eyes see differently, because the contexts were separated on purpose. > > *Diagram:* Adversarial review by context separation: the implementer sees the original source, the porting guide, the lifetime table and its own reasoning. The two reviewers see only the diff and are told to assume the code is wrong. A fixer applies the accepted findings. Ask for a review from inside the implementer's context and it will approve exactly the assumptions it wrote the code under. LLMs have the "I wrote it, let's merge" bias too; we just don't call it ego – the same context reproduces the same blind spot. Reviewers who saw only the diff caught a `Box` dropped while libuv still held the pointer (use-after-free), `trunc()` where negative timestamps needed `floor()`, and an eagerly evaluated `unwrap_or()` that panicked. No test suite catches those, and you won't spot them skimming a diff. It is the cheapest quality win in this whole post, and the review rule that came with it works without any AI: if you need a paragraph-long comment to justify why a workaround is OK, the code is wrong – fix the code. The other principle was **fixing the process that generates the code instead of hand-fixing the code**. When parallel Claudes started wiping out each other's work with `git stash` and `git reset`, nobody restored files by hand; one line went into the instructions: never run a git command that doesn't commit a specific file. One process fix removes a bug class across a million generated lines; one manual fix repairs one line. But notice what that presupposes: you can only reverse-engineer the process from its output. Someone read the diff and understood why it came out wrong. If you don't read code, you have nothing to fix the process with. Two sober footnotes. "Done" wasn't at the merge: 19 known regressions surfaced afterwards – checks that silently vanished in release builds because a Zig function had become a Rust `debug_assert!`, and bounds checks that Rust keeps where Zig's `ReleaseFast` dropped them, exposing old off-by-ones. And almost nothing that went wrong along the way was an AI problem: colliding git operations, a disk filled by four worktrees, cascading compile errors from cyclic crate dependencies, CI timeouts. Infrastructure and process, all of it. ## What no sensor measures A control loop is only as good as what it measures. These are the places I see it break – some from the sources above, some from my own work. ### The sensor decides what gets optimised Give a loop enough iterations on a TypeScript codebase and you will find `any` in it. Not because the model is dumb, but because when a type won't line up, `any` is the cheapest way out: ```ts // Sensor: tsc --noEmit. Objective as the loop sees it: "make it compile". const rows = (await db.query(sql)) as any; // compiles. error = 0. loop is happy. ``` That is not a model failure, it is an objective-function failure. If your sensor is "does it compile", `any` is a perfect solution – it does exactly what you asked. The fix is the same as everywhere else in this post: put it in the sensor (a lint rule for explicit `any`, behind the ratchet) and it stops paying off. ### The gap between intent and implementation Agents do test, and when prompted they catch plenty of edge cases you wouldn't have thought of right away. But there is a class of finding only you produce while reading a diff: "fine, but this also needs X", or "it works – it's just not what I meant". That isn't a bug in the code, it is a deviation from intent, and no sensor measures it: the sensor knows the rules, not your intent. This is why you read the code. Not out of distrust, but because the diff is the only place where the gap between intent and implementation becomes visible. ### The phantom security fix The sneakiest failure class, and it isn't loop-specific. An agent works from a partial picture of the application, sees a fragment, and "handles" a security problem that doesn't exist: sanitising a field that never reaches a user, rate-limiting an internal call path, duplicating an auth check in an already protected layer, adding a defensive null check that hides a broken caller contract. Sometimes the fix is gone again an iteration later, once the context is rebuilt and the agent sees it was never viable. A loop can't catch this, because it looks at a delta per iteration and **architectural coherence is not a local property** – it isn't visible in any single diff, only in the whole picture. It is the same mechanism as Ralph's overbaking, and the strongest structural argument for keeping a human on the loop. A control loop helps only with what you can write down as a rule; "this doesn't fit the architecture" is not an ast-grep rule. ### Subagent fan-out Under a loop, agents spawn subagents much more eagerly, because context is tight – Fig. 5 again. Cost then scales with fan-out rather than with iterations, observability turns from one log into a hundred, and a hundred separate contexts means a hundred separate readings of your intent. Parallelism isn't free: Bun partitioned like a distributed system – separate worktrees, single-file commits, banned git commands. ## When to use which Reach for a control loop when you can tick all of these. If the first one fails, stop there. 1. The end state is a **machine-checkable assertion**, not a direction. 2. A **sensor that is not the agent** exists or can be written: a linter rule, a compiler, a spec validator, a test suite you trust. 3. The work **decomposes into small independent units**, each reviewable on its own. 4. Every step is **reversible** – a PR, not a deploy or a data migration. 5. You have the **review capacity** to read what it produces, and someone named who owns each bot PR. The workloads that fit are repetitive, well-defined, measurable and boring – exactly what humans do badly over months: eradicating a bad pattern, adopting a framework incrementally, keeping an API compliant with its OpenAPI spec, keeping a fork in sync with upstream, mirroring a project into another language, keeping integrations current, working down errors and alerts. It does not fit product design, greenfield architecture, anything irreversible, or "make it better". | | Solo · not business-critical | Team · production | | --- | --- | --- | | Ralph loop | genuinely good: side projects, prototypes, throwaway tools, learning | not enough: no sensor, no WIP limit, no ownership | | Who carries the risk | you – and you decide the tolerance | someone else: the next developer, the customer | | Disturbances | none, it's your repo | the normal operating state | | Review | your call | mandatory – and the bottleneck | | What you need | a sandbox and an iteration cap | sensor, ratchet, small PRs, feedback file, WIP limit, named owners | A reality check on adoption, because the conference stage distorts it: what companies actually run today falls into two buckets – **cron-triggered agents** (nightly or weekly tech-debt clean-up, eval suites, suggestions) that end in a PR a developer reviews, and **event-triggered agents** (a ticket opens, an error arrives, customer feedback lands). Neither is new; we can just do far more useful things inside them now. If your company is on an IDE assistant plus the occasional CLI agent, you are not behind. The differences between tools are much smaller than the differences between processes: what matters is whether you have a sensor and a review, not which CLI you invoke. ## Takeaways 1. **Name the loop.** Agentic loop (always running) ≠ Ralph loop (a `while` until done) ≠ control loop (measured and regulated). 2. **No sensor, no loop** – only hope. Make it deterministic, fast, versioned, and not the model that writes the code. 3. **Ratchet first, generation second.** "It must not get worse" is cheaper and more valuable than "make it go away". Adopt it even without AI. 4. **The controller can be `head -1`.** The best one works from production data. 5. **Small PR, one unit per run, human on the loop.** 6. **A review comment should become a rule**, not a repair. That is how review cost amortises. 7. **WIP limit.** Review capacity sets the throughput, not the token budget. 8. **Fix the process, not the code – which requires reading the code.** 9. **Review adversarially, with separated context.** The reviewer sees only the diff and assumes it is wrong. Mistele's answer to the hype's three lines is the position I'd sign: build loops, read the code, preserve human ownership, maintain and improve code quality, solve hard problems in complex codebases. So the question isn't "should we use loops?" but "what can I measure?". If there is something to measure, a loop is not a risk but a tool: you have a sensor, a ratchet, small PRs, and if it goes sideways you see it tomorrow morning. If there is nothing to measure, you don't have a loop. You just started something. I'm a realist about this, and a little critical – but it clearly can work. The Bun rewrite isn't marketing; it is a million lines and 128 fixed bugs. These processes just can't be left alone. A human still has to look in, reach in – and read. ## Sources - Kyle Mistele (HumanLayer), [Loop Engineering from First Principles](https://ai.engineer/talks/xIt_mTQp6mY-loop-engineering-from-first-principles) – AI Engineer World's Fair 2026 ([video](https://www.youtube.com/watch?v=xIt_mTQp6mY)). The control-loop model, the eight steps and the Effect migration example. - Geoffrey Huntley, [Ralph Wiggum as a "software engineer"](https://ghuntley.com/ralph/) – the original Ralph loop, July 2025. - Dex Horthy (HumanLayer), [A Brief History of Ralph](https://www.humanlayer.dev/blog/brief-history-of-ralph) – the "independent context windows" reframing and the 6-hour refactor; the "dumb zone" is from his talk [No Vibes Allowed](https://www.youtube.com/watch?v=rmvDxxNubIg). - Jarred Sumner, [Bun in Rust](https://bun.com/blog/bun-in-rust#adversarial-review) – the numbers, adversarial review and "fix the process". - Matt Pocock, [Getting started with Ralph](https://www.aihero.dev/getting-started-with-ralph) – the capped, sandboxed Ralph script. - MindStudio, [What is a dark factory?](https://www.mindstudio.ai/blog/what-is-dark-factory-autonomous-ai-codebase) – evaluation gaming and the other risks of fully autonomous codebases. - Tools: [ast-grep](https://ast-grep.github.io/), [react-doctor](https://github.com/millionco/react-doctor), and HumanLayer's [design-control-loop skill](https://github.com/humanlayer/skills). --- # Two kinds of micromonolith > Microservices and Kubernetes are scaling and organisational tools, not quality tools. What splitting an application really costs, why the modular monolith is the good micromonolith, and how to build one that scales without either. Source: https://laszlonemes.com/blog/micromonoliths · Published: 2026-09-20 · Author: László Nemes September 20, 2026 · 18 min read `Architecture` `.NET` `Modular monolith` `Microservices` "Micromonolith" means something different to everyone who says it. Before arguing about whether it is good or bad, it is worth noticing that the same question sits behind every reading of the word: **where do you draw the line between monolith and microservice, and how do you organise the application around that line?** It is a spectrum, not a switch. > **Fig. 1** · The whole argument on one page. There are two micromonoliths. The bad one cut up the deploy instead of the responsibility. The good one cut the responsibility cleanly instead of the deploy. The red arrow is the mistake: splitting a ball of mud. The blue arrow is the only cheap road to microservices – it starts from a monolith whose boundaries are already sharp. > > *Diagram:* A two-by-two of how many things you deploy against how sharp the boundaries are. One deploy with sharp modules is the modular monolith, the good micromonolith. Many deploys with a shared database and lockstep releases is the distributed monolith, the bad one. Microservices are many deploys with sharp boundaries; the big ball of mud is one deploy with none. > **The thread through this post:** microservices and Kubernetes are scaling and organisational tools, not quality tools. Your code does not get better because it runs in thirty containers. Splitting an application is never free – so prove the seam first, then cut. ## Three things called a micromonolith > **Fig. 2** · Same word, three architectures – in Fig. 1's terms, A is the bottom-right corner, B the top-right and C the top-left. A and C look similar on a whiteboard – several named boxes – and are opposites in practice: A pays for a network between the boxes and shares the data anyway; C shares a process and keeps the data apart. > > *Diagram:* Three things people mean by micromonolith. A: a distributed monolith, several services deployed together around one shared database. B: every microservice is a small monolith with its own deploy and its own data. C: a modular monolith, one process with sharply separated modules and no network between them. - **A · The distributed monolith.** Many "services", one shared database, deployed together. This is what you get when you draw the line in the wrong place: you split the deploy but not the responsibility or the data. You pay the full price of distribution and get nothing back. - **B · Every microservice is a small monolith.** Structurally true – inside, a service is a small self-contained application. But a microservice is not defined by its *size*. It is defined by its independence: its own deploy, its own data, a team boundary around it. - **C · The modular monolith with micro-modules.** One deploy; inside, sharp, microservice-like boundaries. This is the one I build, deliberately. It isn't "I don't dare do microservices": I get the bounded contexts – modules, dependency injection, clean interfaces – without paying for a network. If a module later genuinely outgrows the process, extracting it is a refactor, not a rewrite. ## The bill for splitting Every argument for the monolith rests on this section. Each item is the same story: what happens when a method call becomes a network call. > **Fig. 3** · One arrow, before and after. Latency numbers are the usual orders of magnitude: a method call is nanoseconds, a round trip inside one data centre is about half a millisecond before serialisation. > > *Diagram:* An in-process method call takes nanoseconds and has two outcomes: it returns or it throws. The same call over the network takes half a millisecond to two milliseconds at best and has a third outcome: no answer, so the caller cannot know whether it happened. Every in-process call you put on the wire inherits the [eight fallacies of distributed computing](https://en.wikipedia.org/wiki/Fallacies_of_distributed_computing): the network is reliable, latency is zero, bandwidth is infinite, the network is secure, topology doesn't change, there is one administrator, transport cost is zero, the network is homogeneous. In a monolith, *none* of these is your problem. ### It is the tail, not the average A millisecond per hop sounds harmless. The trouble is the tail. Every service is occasionally slow – a garbage collection, a cold cache, a noisy neighbour. Say that happens to one request in a hundred. A user request that touches N services is slow if *any* of them is: > **Fig. 4** · Computed as 1 − 0.99^N, assuming the services are slow independently. With thirty services on the path, a quarter of all user requests experience somebody's worst case. This is the effect Dean and Barroso describe in [The Tail at Scale](https://research.google/pubs/the-tail-at-scale/). > > *Diagram:* If every service is slow for one request in a hundred, the chance that a request touching N services hits at least one slow response is one minus 0.99 to the power of N: 1 percent for one service, 10 percent for ten, 26 percent for thirty and 63 percent for a hundred. ### A third outcome In a monolith a call either returns or throws. It is binary. On a network there is a third state: *I don't know whether it happened.* Everything below follows from that one fact. > **Fig. 5** · Timeout → retry → duplicated side effect. The machinery that prevents this – idempotency keys, deduplication, backoff, circuit breakers, bulkheads, in .NET typically Polly – is complexity that exists only because the call was split. > > *Diagram:* A sequence between an orders service and a payments service: the charge request succeeds, the response is lost, the caller times out without knowing whether it happened, retries, and the card is charged twice unless the receiver deduplicates with an idempotency key. ### Transactions > **Fig. 6** · Left: the database guarantees all-or-nothing. Right: you do, by hand. A distributed monolith manages the worst of both – it takes on the saga while still sharing one database, so it doesn't even get isolation in exchange. > > *Diagram:* In a monolith, reserving stock, charging the payment and creating the shipment are one ACID transaction that the database rolls back as a whole. Across services the same flow is a saga: when the shipment fails, you must run compensating actions, refunding the payment and releasing the stock, in code you write yourself. ### The observability tax In a monolith, the stack trace tells you what happened. Across services, one logical operation lives scattered over N services' logs, so you need distributed tracing (OpenTelemetry, Jaeger), a correlation ID threaded through every hop, and log aggregation. That is not an option you add later. It is the entry fee. ## The real reason: Conway’s law If splitting costs this much, why does anyone do it? Because microservices are an architectural answer to an **organisational** question. Conway observed in 1968 that systems end up mirroring the communication structure of the organisation that builds them. Service boundaries map onto team boundaries so that many teams – different offices, different time zones – can deploy without waiting for each other. > **Fig. 7** · The same architecture, two organisations. If you have no organisational question, a service-per-box architecture is a wrong answer to a problem you don't have. > > *Diagram:* With many teams, a service per team lets each team ship on its own schedule, so the boundary pays for itself. With one team of six to eight people owning thirty services, the team pays the full cost of distribution and gets no independence in return. ## The good micromonolith, mechanically A modular monolith is not a "big ball of mud with folders". The difference is that the module boundaries are **enforced at compile time**, not agreed on in a wiki. > **Fig. 8** · Each module is a pair of projects: a small public Contracts assembly and an implementation whose types are internal. Other modules can only reference Contracts, so a shortcut across the boundary is not a code-review discussion – it doesn't build. > > *Diagram:* One process containing three modules. Each module has a public Contracts part and an internal implementation. Modules may depend only on another module's Contracts. Reaching into another module's internals is a compile error, and an architecture test fails the build. ```csharp // Shop.Billing.Contracts: the only Billing project other modules may reference public interface IBillingService { Task CreateInvoiceAsync(OrderId order, CancellationToken ct); } // Shop.Billing: a separate project. `internal` = invisible outside this assembly, // so "just call the repository directly" from Orders does not compile. internal sealed class BillingService(BillingDbContext db) : IBillingService { public async Task CreateInvoiceAsync(OrderId order, CancellationToken ct) { /* … */ } } ``` Project references and `internal` stop most violations before they exist. For the rules the compiler can't express – layering inside a module, "the domain knows nothing about EF Core" – architecture tests do the same job in CI. I use [NetArchTest](https://github.com/BenMorris/NetArchTest): ```csharp [Fact] public void Orders_domain_stays_clean() { var result = Types.InAssembly(typeof(Order).Assembly) .That().ResideInNamespace("Shop.Orders.Domain") .ShouldNot().HaveDependencyOnAny( "Microsoft.EntityFrameworkCore", // no infrastructure in the domain "Shop.Billing", // no shortcut into another module "Shop.Catalog") .GetResult(); Assert.True(result.IsSuccessful); // a violation fails the build, not a review } ``` This is what makes the top arrow in Fig. 1 cheap. When a module really does need to become a service, its seam already exists: the Contracts interface becomes the API, the in-process call becomes a client. Martin Fowler's [MonolithFirst](https://martinfowler.com/bliki/MonolithFirst.html) makes the same case – monolith first, extract when the seam has earned it. ## Scaling needs neither The strongest point, and the most often forgotten: **horizontal scaling requires neither microservices nor Kubernetes.** A stateless monolith runs as N identical instances behind a load balancer, and that is it. > **Fig. 9** · Multi-node-ready from day one. The only requirement is discipline about state. The dashed instances are a seasonal peak: two predictable weekends a year don't justify running an orchestrator all year. > > *Diagram:* Horizontal scaling without microservices or Kubernetes: a load balancer in front of several identical stateless instances of the monolith, with all state, the database, the cache and the file store, kept outside the instances. Extra instances are added for a peak and removed afterwards. In practice that means no in-process-only cache or session state: the cache is Redis or a database table, chosen in configuration – the in-memory default is for single-node installations only. Application settings that live in the database are synchronised across nodes by a background job that polls; about a minute of eventual consistency is perfectly fine for configuration. Yes, Postgres has `LISTEN`/`NOTIFY` – but polling is trivially robust, works on every database, and adds no infrastructure. That is appropriate simplicity, not laziness. ## Kubernetes is a separate axis Microservices and Kubernetes are two independent decisions. Whoever merges them has already made a mistake. | | No orchestrator | Kubernetes | | --- | --- | --- | | Monolith | IIS · systemd + nginx · one container · App Service | perfectly valid: N replicas of one image | | Microservices | ECS / Fargate · App Service · plain VMs | the combination everyone pictures | My estimate is that well over nine out of ten applications don't need Kubernetes. It gets forced in because someone needed the project, or because it is fashionable. What hosting actually takes, most of the time: - **Windows:** host it in IIS. Done. - **Linux:** a `systemd` service with nginx in front. Done. - **Either:** one Docker container. Done. /etc/systemd/system/shop.service ```ini [Unit] Description=Shop (ASP.NET Core) After=network.target [Service] WorkingDirectory=/opt/shop ExecStart=/usr/bin/dotnet /opt/shop/Shop.dll Restart=always Environment=ASPNETCORE_URLS=http://127.0.0.1:5000 [Install] WantedBy=multi-user.target ``` The honest edge, before someone else points it out. The choice isn't only "Kubernetes or bare VMs": Azure App Service autoscale and ECS/Fargate scheduled scaling give you elastic scaling without owning a control plane. And manually adding instances is ideal for *predictable, rare* peaks; when peaks are unpredictable and frequent, shuffling VMs by hand is operational toil, and automating exactly that is what an autoscaler is for. Where Kubernetes is genuinely right: many workloads, declarative self-healing, sophisticated rollouts – and the numbers work out in the spreadsheet. Then it is very good. But the real question is never "do we have microservices?". It is: **do we have enough workloads, and enough operational maturity, to feed a control plane?** ## Two things called async Two unrelated things are both called "async", and arguments go wrong when they are mixed up. One is a thread's I/O model. The other is how services talk to each other. ### The I/O model: async/await In a production .NET application all I/O is `async`/`await` – monolith or microservice. That isn't an architectural choice, it is table stakes: a thread that isn't blocked waiting for the database can serve another request. Blocking on async code is the classic way to lose that: ```csharp // blocks a thread-pool thread while the database works; under load the pool starves var user = db.Users.FirstAsync(u => u.Id == id).Result; // hands the thread back until the database answers var user = await db.Users.FirstAsync(u => u.Id == id, ct); ``` In ASP.NET Core the symptom is thread-pool starvation under load; in environments with a synchronisation context (classic ASP.NET, desktop UI) the same line can deadlock outright. Note that this helps I/O-bound work – CPU-bound work doesn't get faster from `await`. And it is entirely orthogonal to the monolith question. ### Communication between services A mature microservice system is *not* a lot of small REST APIs calling each other. If it consists only of synchronous calls, it is a distributed monolith wired together with HTTP. There are two axes here, and they shouldn't be blurred: - **Synchronous request/response** – the caller needs the answer *now*. REST (readable, browser-friendly) or gRPC (typed `.proto` contracts, binary protobuf over HTTP/2, a faster synchronous transport). gRPC is not a queue: it is broker-less, point-to-point RPC. - **Asynchronous messaging through a broker** – RabbitMQ, Kafka, NATS, Azure Service Bus. Not a "third option" next to REST and gRPC but a different axis: **temporal decoupling**. The producer doesn't wait for the consumer. > **Fig. 10** · A sync-only mesh is rigid: a slow or dead component propagates up the chain. The broker is the shock absorber. Arithmetic for the burst: 1,000 orders/s arrive for 10 s while 200/s are written, so 8,000 are queued at the peak, and the remaining backlog drains in 40 s. > > *Diagram:* In a synchronous chain of services, one slow or dead component fails every caller upstream. With a broker, the producer returns immediately and the queue holds the messages: a burst of ten thousand checkouts in ten seconds against a database that writes two hundred per second peaks at eight thousand queued messages and is fully written after fifty seconds, with nothing lost. | Situation | Without a broker | With a broker | What it buys | | --- | --- | --- | --- | | Sign-up sends a welcome e-mail; the mail provider is slow | sign-up waits, or fails | sign-up completes; the mail goes when it can | temporal decoupling | | A newsletter lands; checkouts spike far above DB write capacity | timeouts and lost orders | the queue holds the burst and drains it | load levelling | | A partner API is down for maintenance | your writes fail with it | messages wait and flow when it returns | failure isolation | | After an upload: thumbnail, indexing, notification | the upload code calls all three | one FileUploaded event, three consumers; a fourth needs no change to the uploader | fan-out | | "Generate the PDF", "transcode the video" | the user watches a spinner | queued; notified when done | commands vs. queries | The last row has a counterpart: "what is my balance?" needs an answer *now*. That is a query, and it stays synchronous. Don't over-correct into "everything is a queue" – the naive mistake is all-sync, the mature system **mixes**: sync for queries that need a response, a broker for commands, events and reactions that tolerate eventual consistency. > **Fig. 11** · Not every broker is the same thing. Choose by semantics: if you need replay and retention, a log; if you need a work queue with acknowledgements, a queue. NATS sits beside both: core NATS is lightweight fire-and-forget, JetStream adds persistence. > > *Diagram:* A queue such as RabbitMQ or Azure Service Bus hands each message to one of several competing workers and deletes it once acknowledged. A log such as Kafka retains messages; each consumer group keeps its own offset, so one group can be at offset six while another replays from offset two. Two honest caveats. A broker is **not faster per message** – a direct gRPC call has lower latency. The broker wins on behaviour under load and on resilience, not on raw speed. And it has its own price: eventual consistency; at-least-once delivery, so **idempotent consumers are mandatory** (exactly-once across a network is not something you can buy); no ordering guarantee by default; dead-letter queues and poison messages; and the broker itself becomes critical infrastructure. In .NET, [MassTransit](https://masstransit.io/) over RabbitMQ or Azure Service Bus, with its built-in outbox against dual writes, is the usual toolkit. ### The twist that leads back to the monolith You need a broker between services *because you split the application*. Inside a monolith you get the same decoupling without a broker and without a network: > **Fig. 12** · An in-memory channel gives you temporal decoupling and fan-out inside one process. It is not durable – whatever is in the channel is lost if the process dies – so work that must not be lost is written to an outbox table in the same transaction as the business data. > > *Diagram:* The same decoupling inside one process: the upload endpoint writes an event to an in-memory channel and returns immediately, a background service reads the channel and fans out to thumbnail, index and notify handlers. Because a channel is lost on a crash, durable work also goes through an outbox table. ```csharp builder.Services.AddSingleton(Channel.CreateBounded(capacity: 1_000)); builder.Services.AddHostedService(); // producer: the endpoint returns as soon as the event is queued app.MapPost("/files", async (IFormFile file, IFileStore store, Channel channel, CancellationToken ct) => { var id = await store.SaveAsync(file, ct); await channel.Writer.WriteAsync(new FileUploaded(id), ct); return Results.Accepted($"/files/{id}"); }); // consumer: same process, no broker, no network internal sealed class FileUploadedWorker( Channel channel, IEnumerable handlers) : BackgroundService { protected override async Task ExecuteAsync(CancellationToken ct) { await foreach (var uploaded in channel.Reader.ReadAllAsync(ct)) foreach (var handler in handlers) // thumbnail, index, notify await handler.HandleAsync(uploaded, ct); } } ``` ## How I build: ports, providers, one deploy What I build has a name: a modular monolith with hexagonal infrastructure, in the clothes of the .NET provider model, packaged as an on-premises product. None of it is new: - **Provider model** – native .NET heritage, from the ASP.NET Membership providers to today's `IDistributedCache`, EF Core providers and authentication schemes. - **[Hexagonal / ports and adapters](https://alistair.cockburn.us/hexagonal-architecture/)** (Cockburn) – every external dependency is a port with a replaceable adapter behind it. - **Clean / Onion** – domain in the middle, infrastructure at the edge, dependencies pointing inwards. - **[Twelve-factor](https://12factor.net/) III and IV** – configuration from the environment; the database is an attached resource you plug in without a code change. > **Fig. 13** · Micro-level modularity and replaceable ports, monolithic deploy. Each group is a port in the application and one section of configuration for the customer – who never sees, or needs, a different build. The highlighted adapters are the defaults: with an empty configuration the application starts on them right away. > > *Diagram:* Ports and adapters: the domain and application code in the middle knows only interfaces. Configuration selects the adapter behind each port: one of five databases, one of seven file stores, the cache, which is in-memory, Redis, or a SQL Server, PostgreSQL or MySQL table, the user store, which is local, Active Directory, LDAP or Entra ID, and separately the login method: username and password with e-mail verification, Active Directory, or OAuth with Microsoft, Google, Okta or a custom provider such as PingID. The highlighted adapters are the zero-configuration defaults the application starts with: SQLite, local disk, the in-memory cache, which only works on a single node, the local user store and username and password login. appsettings.json (illustrative) ```json { "Database": { "Provider": "PostgreSql", "ConnectionString": "Host=db;Database=app" }, "FileStore": { "Provider": "S3", "Bucket": "customer-documents" }, "Cache": { "Provider": "Redis" }, "UserStoreSettings": { "Provider": "Local" }, "LoginSettings": { "Password": { "Enabled": true, "EmailVerification": true }, "ActiveDirectory": { "Enabled": false }, "OAuth": { "Provider": "Okta", "Authority": "https://login.customer.example", "ClientId": "shop" } } } ``` - **Database from configuration:** SQL Server, PostgreSQL, MySQL, SQLite or Oracle, each with its own set of code-first migrations. - **Migrations as a CLI:** the application binary can start in CLI mode to list migrations and step them forwards and backwards by hand. Before a downgrade the application brings the database to the target state itself, so the way back is safe. Always backwards compatible. - **Users and login from configuration, as two separate concerns.** `UserStoreSettings` says where users live – local (the application's own database), Active Directory, LDAP or Entra ID. `LoginSettings` says how they sign in: username and password with e-mail verification; Active Directory, against the matching user store; or OAuth, with separate options for Microsoft, Google and Okta plus a *Custom* option for everything else – PingID and the like. Keeping the two apart is what lets them combine, and in practice it covers almost any identity set-up a customer walks in with. - **File store from configuration:** Azure Blob, AWS S3, MinIO, SharePoint, Box, GCP Storage or local disk – through my open-source [Cloud File Storage Manager](https://github.com/NemesLaszlo/Cloud.File.Storage.Manager) packages. - **Cache from configuration:** one `ICache` port with in-memory as the default – the fastest option, and nothing to install – and Redis or a SQL Server, PostgreSQL or MySQL table behind the same interface. The default comes with a hard limit: an in-memory cache lives inside one process, so on a multi-node deployment it is not usable – every instance would hold its own copy and they would drift apart, which is exactly the in-process state Fig. 9 rules out. Going multi-node means switching the cache to Redis or a database table first; that is a configuration change and a restart, not a code change. - **Zero-config start-up:** the default configuration must make the application start immediately. There is no first run that refuses to boot without setup. With an empty configuration you get SQLite, local disk, the in-memory cache, the local user store and username-and-password login, and the application creates and migrates its own database on start-up from the code-first migrations. Later you point it at the real database or identity provider and restart. The rule behind it: whatever the application can do for itself, it does. > **Scope, stated plainly:** this is for a boxed, on-premises product, where the *customer* owns the runtime and the application has to fit into their infrastructure. If you are building a pure cloud SaaS that needs exactly one Postgres, five database providers would be over-engineering. Different use case, different answer. It also has limits worth admitting. "Supported" does not mean "behaves identically": transactions, locking and SQL dialects differ, and targeting five databases pulls you towards the lowest common denominator – no casual use of Postgres JSONB. EF Core hides a lot; where it matters I allow a provider-specific path. And "we support everything" is only as true as the test matrix behind it: the credible version runs migrations – including the `down` ones – and the data layer against every claimed provider in CI, which [Testcontainers](https://dotnet.testcontainers.org/) makes practical. Only promise the providers that have a real deployment behind them; keep the rest as a clean seam rather than a maintained adapter. ## When to split after all None of this says "never". It says the burden of proof is on the cut. Extract a module into a service when at least one of these is true – and you can name it: 1. **A team boundary:** a separate team needs to ship that part on its own schedule. 2. **A genuinely different scaling profile:** one part needs 40 instances or GPUs while the rest needs two small VMs. 3. **A different runtime or release constraint:** another language, a regulated component, a part that must not go down with the rest. 4. **The seam has already proven itself:** the module boundary has been stable in the monolith for a while, and its Contracts interface is what you would publish as an API anyway. If none applies, what you would be building is reading A from Fig. 2: the tax without the benefit. ## Takeaways 1. **Microservices and Kubernetes are scaling tools, not quality tools.** They don't make your code better. 2. **Splitting is never free.** Latency tails, a third failure state, sagas instead of transactions, mandatory tracing. 3. **Microservices answer an organisational question.** No such question, no such answer. 4. **Enforce module boundaries at compile time** – separate assemblies, `internal`, architecture tests. Then extraction is a refactor. 5. **Write a good monolith that is multi-node-ready from day one.** After that, scaling is a load balancer and a few instances. 6. **Kubernetes is a separate axis.** The question is workloads and operational maturity, not service count. 7. **Sync for queries, a broker for commands and events** – and inside a monolith, a channel and an outbox give you the same decoupling for free. There are two micromonoliths. The bad one cut up the deploy instead of the responsibility. The good one cut the responsibility cleanly instead of the deploy. Build the second one. --- # Git and GitHub: the mental model, in pictures > Almost every git confusion is a missing picture, not a missing command. The pictures: snapshots, the three areas, branches, remotes, pull requests and undo – with VS Code as the daily driver and the commands beside every button. Source: https://laszlonemes.com/blog/git-and-github · Published: 2026-09-20 · Author: László Nemes September 20, 2026 · 18 min read `Git` `GitHub` `VS Code` `Fundamentals` Most people learn git as a list of commands to type in the right order. That works until the first unexpected message, and then nothing makes sense – because the commands were never the hard part. Almost every git confusion I have seen, in teams and in the classroom at ELTE, is a missing *picture*: where a change currently lives, what a branch actually is, what "origin" refers to. So this post is the pictures: 12 figures, a small example in each, and only as many commands as the pictures need. For the day-to-day I use **VS Code**, so every step shows the button next to the command it runs – you should know both, because the button is faster and the command is what every error message and every answer online will speak. > **The whole model in one breath:** a commit is a snapshot of your project, a branch is a name pointing at a commit, and GitHub is a place where a copy of those commits lives so other people can reach them. Everything else is moving snapshots between places. ## Git is not GitHub **Git** is a program on your machine. It records the history of a folder and needs no account and no internet. **GitHub** is a website that hosts git repositories and builds collaboration on top of them. GitLab and Bitbucket are alternatives to GitHub; there is no alternative you need for git. > **Fig. 1** · Every clone is a complete repository with the full history – there is no “server copy” and “client copy”, only copies that agree to treat one of them as the meeting point. > > *Diagram:* Git runs on your laptop and holds the full history offline. GitHub hosts a shared copy of the same repository and adds pull requests, issues, CI and releases on top. A teammate's laptop holds another full copy. Only push and pull cross the network. This is also why "I only want to download one folder" is awkward with git: a repository isn't a folder tree you pick from, it is a sequence of whole-project snapshots. You take the history or you don't. ## Set up once Install [git](https://git-scm.com/downloads) and [VS Code](https://code.visualstudio.com/), then tell git who you are and which editor to use. You do this once per machine. ```shell # who you are: stamped into every commit you make git config --global user.name "Your Name" git config --global user.email "you@example.com" # new repositories start on "main" git config --global init.defaultBranch main # when git needs an editor, open a VS Code tab and wait until I close it git config --global core.editor "code --wait" # when my branch and the remote have both moved, pull should merge git config --global pull.rebase false # line endings: Windows git config --global core.autocrlf true # line endings: macOS / Linux git config --global core.autocrlf input ``` The `core.editor` line matters more than it looks. Out of the box git opens a terminal editor whenever it wants text from you – a merge message, for example – and that is where many beginners get stuck. With this setting it opens a normal VS Code tab instead: type, save, close the tab, and git carries on. If `code` isn't recognised on macOS, run *Shell Command: Install 'code' command in PATH* from the Command Palette first. The `pull.rebase false` line answers a question git would otherwise stop and ask you later: when both you and the remote have new commits, a pull should *merge* them. Recent git versions refuse to guess. (Rebasing is the other answer; it is a fine tool, and a bad first one.) Line endings differ between Windows and everything else, and without the `autocrlf` setting a mixed team sees whole files flagged as changed. In a shared project, also commit a `.gitattributes` file containing `* text=auto`, so the rule travels with the repository instead of depending on everyone's laptop. ### Signing in to GitHub GitHub doesn't accept your account password from git. The path of least resistance: clone with the **HTTPS** URL and let the sign-in happen in the browser. VS Code does this by itself the first time you clone or publish, and Git for Windows ships with a credential manager that does the same for the terminal. **SSH** keys are the alternative many developers prefer – a one-time set-up, no prompts afterwards – and [GitHub's guide](https://docs.github.com/en/authentication/connecting-to-github-with-ssh) walks through it. Both reach the same repository; pick one and move on. ## A commit is a snapshot A commit is not "the changes I made". It is the **complete state of the project** at one moment, plus a message, an author, a time, and a pointer to the commit that came before. Its ID is a hash of all that, which is why it looks like `a1f93c2` rather than "version 3". > **Fig. 2** · Three commits. The arrows point backwards, from child to parent – a commit knows where it came from, never what comes after. Storing full snapshots is cheap because unchanged files are shared, not copied. > > *Diagram:* Three commits in a row. Each commit is a snapshot of the whole project and points back at its parent. Files that did not change are not copied; the new snapshot points at the copy git already has. "Changes" are something git *computes* by comparing two snapshots. That comparison works line by line, which is why git loves plain text and is unhelpful with Word documents or images – for those it can only say "this file is different now". ```diff $ git diff --- a/login.ts +++ b/login.ts @@ -40,7 +40,7 @@ export function login(form: LoginForm) { const user = await findUser(form.email); - if (user.password == form.password) { + if (await verifyHash(form.password, user.passwordHash)) { return createSession(user); ``` In VS Code you rarely read a diff in this form: click a file in the Source Control view and you get the same information side by side, old on the left, new on the right. Read it before every commit. It is the cheapest code review you will ever get. ## The three areas Between "I edited a file" and "it is in the history" there is a stop that surprises everyone coming from Save buttons: the **staging area**. It is a draft of the next commit. You choose what goes into it, so that one commit can be one logical change even if you touched ten files for three different reasons. > **Fig. 3** · Where a change lives. Forward is always two steps: stage, then commit. Going back is safe everywhere except Discard Changes, which deletes work that was never stored anywhere. > > *Diagram:* A change travels through three areas: the working directory, the staging area and the repository. git add stages it, git commit stores it. In VS Code these are the plus icon and the Commit button. Unstaging and undoing a commit move it back; discarding throws it away. ```shell git init # or VS Code: Initialize Repository git status # what does git see? git add index.html style.css # stage exactly these git commit -m "Add home page" ``` In VS Code all of this is one panel – the Source Control view, `Ctrl+Shift+G`. Once you see that its two groups *are* two of the three areas, the panel stops being a list of files and becomes a map. > **Fig. 4** · The VS Code Source Control view, as a wireframe. If Staged Changes is empty when you press Commit, VS Code offers to stage everything for you – convenient, but it skips the step where you decide what belongs together. > > *Diagram:* A wireframe of the VS Code Source Control view mapped onto git's areas: the Staged Changes group is the staging area, the Changes group is the working directory, the plus icon runs git add, and the Commit button runs git commit. A word on commit messages, since you will write thousands: say *what* changed and *why* in one line, in the imperative – "Fix redirect after login", not "fixed stuff". The diff already shows how. Future you, reading the history to find when something broke, is the audience. ## Branches are names, not copies The word "branch" suggests a copy of the project. It isn't. A branch is a **name that points at one commit**. That is the entire data structure. Creating a branch writes one tiny file; it is instant no matter how large the project is, which is why git users create branches for everything. > **Fig. 5** · Two branches after some work on each. The commits a1f and b72 belong to both histories – nothing was duplicated. HEAD marks the branch you are on; in VS Code that is the name in the bottom-left corner of the Status Bar. > > *Diagram:* A commit graph with two branches. main and feature/login are just names pointing at one commit each, and HEAD points at the branch you are on. Creating a branch copies nothing. ```shell git switch -c feature/login # create the branch and move HEAD onto it # ...edit, stage, commit, as often as you like... git switch main # go back; your files change to match main git merge feature/login # bring the work in ``` In VS Code, click the branch name in the Status Bar: the list that opens lets you switch branches or create a new one. When you switch, git rewrites the files in your working directory to match that branch's snapshot – the files on disk really change, and your editor tabs update with them. ### The one scary message: detached HEAD HEAD normally points at a branch. If you check out a specific commit instead – to look at an old version, say – HEAD points straight at that commit and git tells you that you are in "detached HEAD" state. It sounds like an accident. It is just a description. > **Fig. 6** · Looking around in detached HEAD is completely safe. The only risk is committing there and then switching away: no branch name points at those commits, so nothing leads back to them. > > *Diagram:* Normally HEAD points at a branch, so new commits move that branch. In a detached HEAD state, HEAD points straight at a commit, and new commits belong to no branch until you create one. ## Merging, and merge conflicts Merging means "make this branch contain that branch's work too". Depending on what happened in the meantime, git does one of two things – and knowing which one explains every history graph you will ever look at. > **Fig. 7** · Left: nobody touched main while you worked, so there is nothing to combine and main simply moves forward. Right: both sides have new commits, so git creates a merge commit – the only kind of commit with two parents. > > *Diagram:* Two outcomes of git merge. If main did not move, git fast-forwards: the main label slides to the tip of the feature branch. If both sides moved, git creates a merge commit with two parents. Git combines changes automatically as long as the two sides edited different lines. When both sides changed the *same* lines, there is no correct automatic answer, so git stops and hands the decision to you. That is a **merge conflict**. It is not an error and nothing is broken – it is a question. > **Fig. 8** · How a conflict arises, and where you answer it. “Current” is the branch you are on, “Incoming” is the branch you are merging in. The result pane starts from the common ancestor and you tick which side to take – or type something better than both. > > *Diagram:* A merge conflict happens when two branches change the same line differently. Git stops and asks. In the VS Code three-way merge editor the incoming branch is on the left, the current branch on the right, and you build the result at the bottom, then press Complete Merge. Underneath, git writes both versions into the file between markers, and resolving a conflict means producing the text you actually want and deleting the markers: ```conflict <<<<<<< HEAD title = "Home" ======= title = "Start" >>>>>>> feature ``` VS Code lists conflicted files under **Merge Changes**. Open one and you get inline actions above each conflict – *Accept Current Change*, *Accept Incoming Change*, *Accept Both Changes* – or press *Resolve in Merge Editor* for the three-pane view in Fig. 8. When every conflict is settled, *Complete Merge* stages the file; commit, and the merge is done. Two habits keep conflicts small: merge often, and keep branches short-lived. ## Local and remote A **remote** is another copy of the repository that yours knows the address of. The one you cloned from is named `origin` by convention. Four commands cross the network; everything else in this post happens entirely on your machine. > **Fig. 9** · Which command moves what, and how far. The important asymmetry: fetch only updates your local repository, so it is always safe. pull goes all the way into your files, because it is a fetch followed by a merge. > > *Diagram:* Which command moves data where: git add goes from the working directory to the staging area, git commit to the local repository, git push to the remote. git fetch downloads into the local repository only, git pull is fetch plus merge and reaches your files, git clone copies everything the first time. ```shell # an existing project git clone https://github.com/you/project.git # a new one: create an empty repo on GitHub, then git remote add origin https://github.com/you/project.git git push -u origin main # -u: remember that main follows origin/main ``` In VS Code: *Git: Clone* from the Command Palette for an existing project, or the *Publish to GitHub* button in the Source Control view for a new one – it creates the GitHub repository and pushes in one go. One more name completes the picture. Your repository keeps a bookmark called `origin/main`: where `main` was on GitHub *the last time you talked to it*. It is not live. Comparing it with your own `main` is how git – and VS Code – can tell you that you are ahead, behind, or both. > **Fig. 10** · The state after a fetch: you and a teammate both committed on top of b72. A pull would now merge d90 into your main (Fig. 7, right side); a push before that would be rejected, because GitHub's main has a commit yours doesn't. > > *Diagram:* After git fetch: your main has one commit GitHub does not have, and origin/main, your last-known picture of GitHub's main, has one commit you do not have. VS Code shows this in the status bar as one incoming and one outgoing commit. That rejected push is the most common first "error" in team work, and the message says exactly what to do: pull first, then push. Git refuses because accepting would throw away your teammate's commit. ## The GitHub flow Put the pieces together and you get the workflow most teams use. `main` always works; nobody commits to it directly; every change arrives through a **pull request** – a page on GitHub that says "please merge my branch", shows the diff, and gives teammates and automated checks a place to respond before anything lands. > **Fig. 11** · One piece of work, from branch to merged. The dashed arrow is the normal case, not a failure: review comments lead to more commits on the same branch, and the pull request updates itself when you push them. > > *Diagram:* The GitHub flow in six steps across two lanes: on your laptop you create a branch and make small commits, then push. On GitHub you open a pull request, it gets reviewed and checked, and is merged into main. Back on the laptop you pull main and delete the branch. ```shell git switch main && git pull # start from the latest main git switch -c fix/login-redirect # one branch per piece of work # ...edit · stage · commit... git push -u origin fix/login-redirect # open the pull request on GitHub, get it reviewed and merged, then git switch main && git pull git branch -d fix/login-redirect # done its job (-D if the PR was squash-merged) ``` The pull request is the most valuable thing GitHub adds to git. It turns "my code" into "our code" at a defined moment, with a record of what was discussed. Keep them small: a reviewer can genuinely read 200 lines, and will only skim 2,000. ## Undoing things Git almost never loses work that was committed. The right way to undo depends on a single question: **how far did the mistake travel?** > **Fig. 12** · Four stages, four tools. The two red spots are the ones to respect: discarding unstaged edits destroys work git never stored, and rewriting pushed history destroys work that is no longer only yours. > > *Diagram:* How to undo depends on how far the mistake travelled: discard an unstaged edit, unstage a staged file, amend or undo a local commit, and revert a pushed commit with a new commit instead of rewriting shared history. `git revert` deserves a sentence, because it is the safe default once something is pushed: it doesn't delete the bad commit, it adds a new commit that does the opposite. History stays true, nobody's copy breaks, and the revert itself can be reverted. The opposite tool is `git push --force`, which overwrites the remote branch with yours. On a branch other people use, it silently deletes their commits. If you must rewrite a branch that is *only yours* – after an amend, say – use `git push --force-with-lease`, which refuses if someone else pushed in the meantime. Never force-push `main`. And when you need to put half-finished work aside for ten minutes – an urgent fix on another branch – that is `git stash` (VS Code: *Git: Stash*). `git stash pop` brings it back. ## What not to commit A repository should contain what a human wrote and nothing a machine can regenerate. A `.gitignore` file in the project root lists the patterns git should pretend not to see; GitHub maintains [ready-made templates](https://github.com/github/gitignore) for most languages. .gitignore ```gitignore # dependencies: reinstalled from package.json, never committed node_modules/ # build output: rebuilt from source dist/ build/ # secrets and local settings .env .env.local # OS and editor noise .DS_Store Thumbs.db *.log ``` - **Dependencies and build output** bloat every clone forever and produce meaningless diffs. Rebuild them from source. - **Large binaries** – installers, videos, datasets – don't diff and never leave the history. Attach built programs to a [GitHub Release](https://docs.github.com/en/repositories/releasing-projects-on-github/about-releases) instead: tag the commit (`git tag v1.0.0`, `git push origin v1.0.0`) and upload the files there. - **Secrets** – API keys, passwords, `.env` files. This is the one that hurts. If a secret was ever pushed, deleting it in a later commit does *not* help: it is still in the history, and every clone has it. Treat it as leaked and **rotate it**. Note that `.gitignore` only affects files git isn't tracking yet. If something was already committed, remove it from tracking with `git rm --cached ` and then ignore it. ## Cheat sheet Everything above, by intent. The VS Code names are Command Palette commands (`Ctrl+Shift+P`) or buttons in the Source Control view. | I want to… | VS Code | Command | | --- | --- | --- | | start tracking a folder | Initialize Repository | `git init` | | get an existing project | Git: Clone | `git clone ` | | see what changed | Source Control view · click a file for the diff | `git status` · `git diff` | | stage a file | + next to the file | `git add ` | | commit | Commit · Ctrl+Enter | `git commit -m "…"` | | create a branch | Git: Create Branch | `git switch -c ` | | switch branch | branch name in the Status Bar | `git switch ` | | upload my commits | Publish Branch · Sync Changes | `git push` | | download new commits | Sync Changes | `git pull` | | merge a branch into mine | Git: Merge Branch | `git merge ` | | park unfinished work | Git: Stash | `git stash` · `git stash pop` | | browse the history | Source Control Graph | `git log --oneline --graph` | Older tutorials use `git checkout` for both switching branches and restoring files. It still works; `git switch` and `git restore` are the newer commands that split those two jobs apart, and they are what I'd learn today. ## Takeaways 1. **Git is local, GitHub is a meeting point.** Only clone, fetch, pull and push use the network. 2. **A commit is a snapshot** that points at its parent. Diffs are computed, not stored. 3. **Three areas:** working directory → staging area → repository. The Source Control view is a map of the first two. 4. **A branch is a name for a commit.** Creating one costs nothing, so create one for every piece of work. 5. **A conflict is a question, not an error.** Current is you, Incoming is them, the result is your call. 6. **pull = fetch + merge.** When a push is rejected, pull first. 7. **Undo by how far it travelled.** Once it is pushed, revert – don't rewrite. 8. **Never commit secrets.** If it happened, rotate the secret; deleting the file is not enough. Git's reputation for being hard comes almost entirely from meeting the commands before the model. With the pictures in your head, the error messages start reading like what they are: git telling you, fairly precisely, which arrow in which figure it couldn't follow. ### Further reading - [Pro Git](https://git-scm.com/book/en/v2) – the free, official book. Chapters 2 and 3 cover everything here in depth. - [Source Control in VS Code](https://code.visualstudio.com/docs/sourcecontrol/overview) – the official tour of the features used in this post. - [GitHub flow](https://docs.github.com/en/get-started/using-github/github-flow) – GitHub's own description of the branch-and-pull-request workflow. - [Gitting Started](https://budavariam.github.io/posts/2026/02/04/gitting-started/) by Mátyás Budavári – a friendly, command-line-first introduction that pairs well with this one.