# Shadow AI and local AI: the data class decides

> It isn't local versus cloud: there are four levels of control, and the data class picks one. The arithmetic of running models locally, why local carries the burden of proof, the LLM provider as an adapter, what an EU region does and doesn't give you – and why shadow AI is a product problem, not a disciplinary one.

Source: https://laszlonemes.com/blog/shadow-ai-local-ai · Published: 2026-09-20 · Author: László Nemes

September 20, 2026 · 22 min read

`AI engineering` `Local LLMs` `Security` `Architecture`

The conversation about where AI should run is usually framed as *local versus cloud*, and the conversation about employees using unapproved AI is usually framed as a *discipline problem*. Both framings are wrong in the same way: they skip the question that actually decides things – what kind of data is involved, and how much control that data needs.

> **The spine of this post.** It isn't local versus cloud: the data class picks the level of control. There are four levels, not two. For local, the burden of proof is on local. And shadow AI is a product problem, not a disciplinary one.

## Four levels, not two

> **Fig. 1** · Level 2 is the one people talk about least. The request doesn't go “to OpenAI”, it goes into your own cloud: your network, a private endpoint, a managed identity, your logs. Level 4 is what happens when none of the first three is offered.
>
> *Diagram:* Four levels of control, not two. Local or on-premises gives you control of everything. Your own cloud tenant gives you network, identity and audit logs and is the right default. A direct frontier API gives you a contract and retention terms. A consumer app on a personal account gives you nothing, and that is shadow AI.

Once there are four levels, the useful question is no longer "which one is best" but "which data may go where". That needs a data classification, and it needs to be short enough to remember.

> **Fig. 2** · An example policy, not the policy. The bottom-right corner matters most: pasting your own code into a personal chatbot and pasting customer data there are not the same offence. The first is a policy violation; the second is an incident. A rulebook that forbids both identically will see neither rule kept.
>
> *Diagram:* An example policy matrix of three data classes against the four levels. Public data is fine everywhere. Internal data is fine locally, in your tenant and over a contracted API, and a policy violation in a consumer app. Restricted data belongs locally or in your tenant if the contract allows, and pasting it into a consumer app is an incident.

## Local AI: the arithmetic

Most disappointment with local models comes from three numbers nobody checked before buying hardware. All three follow from how a language model produces text: one token at a time, each after the other. (A token is roughly a syllable-sized piece, not a word – and languages like Hungarian need noticeably more of them per sentence than English.)

### Two speeds, and you always feel the worse one

> **Fig. 3** · Prefill is the “thinking…” pause before the first character; decode is the classic tokens-per-second number. For chat, decode dominates. For a coding agent, prefill does – the answer is written quickly and the whole thing still feels slow.
>
> *Diagram:* A request has two phases. Prefill reads the prompt and is compute-bound. Decode writes the answer token by token and is bound by memory bandwidth. A chat turn is mostly decode. A coding-agent turn is mostly prefill, because the agent re-reads a large part of the codebase on every turn.

> **Fig. 4** · What is liveable. Thinking tokens aren't read by anyone – they are pure waiting – which is why reasoning models raise the floor so sharply. It is also why the sweet spot of local AI is not chat but batch: work nobody is waiting for runs perfectly well at 5 tok/s.
>
> *Diagram:* Tokens per second and what they feel like: five to eight is human reading speed and painful in practice, fifteen to twenty is comfortable chat, and fifty or more is the minimum for reasoning models and coding agents. For batch work nobody is waiting on, five is plenty.

### Memory bandwidth is the key, not GPU muscle

To produce each token, the model reads its active weights from memory. So decode speed is, to a first approximation, **memory bandwidth divided by the bytes read per token**. That one formula explains nearly everything you will measure.

> **Fig. 5** · A dense 70B model at 4-bit is about 40 GB, and all of it is read for every token: 256 ÷ 40 ≈ 6 tok/s. Everyone measures about that – it isn't a driver problem, it is physics. A mixture-of-experts model activates only a fraction of itself per token, which is why a 120B MoE runs at 34–53 tok/s on the same hardware. Without MoE, the 128 GB desktop-box category would not exist.
>
> *Diagram:* On a machine with 256 gigabytes per second of memory bandwidth, a dense 70 billion parameter model must read about 40 gigabytes per token, which caps it at 6.4 tokens per second, and about 5 is measured. A 120 billion parameter mixture-of-experts model reads only about 3 gigabytes per token, a ceiling of about 85, and 34 to 53 tokens per second are measured.

So the number to look for on a spec sheet is GB/s, not TOPS or core counts. The unified-memory boxes sit around 256–273 GB/s; an M3 Ultra is about 819; a discrete GPU is around 1,000 and up – with far less memory attached. Prefill, on the other hand, *is* compute, and there the GPU matters: on the same 120B model an early llama.cpp benchmark measured 1,723 tok/s of prompt processing on a DGX Spark against 340 on a Strix Halo, while decode differed by only about 13% (38.6 against 34.1 tok/s). Later builds narrowed the prefill gap to roughly 2×. For chat you would never notice. For an agent it is the difference between usable and not.

### What fits – and what that leaves room for

Rule of thumb: at 4-bit quantisation, the usual default, **size in GB ≈ half the parameter count in billions**. 70B is 35–40 GB, 120B is 60–70 GB, 235B is 115–120 GB. But the weights are not the only tenant of that memory. Every open conversation keeps a KV cache that grows with context length – and every concurrent user has their own.

> **Fig. 6** · Large model, long context, many users: pick two. And “it fits” is not “it is usable” – a 235B model does squeeze into 128 GB, at single-digit tok/s and with no room left for context. That is a demo, not a tool.
>
> *Diagram:* Memory budget on a 128 gigabyte machine running a dense 70 billion parameter model: 40 gigabytes of weights plus about 32 gigabytes of KV cache for every user with a 100 thousand token context. One user needs 72 gigabytes, two need 104, and three need 136, which does not fit.

### Concurrency: the number missing from every ROI slide

One of these boxes serves one user comfortably, two with goodwill – not ten. Memory is one reason (Fig. 6). Prefill is another: it is compute-bound, so one person's 60,000-token prompt stalls everyone else's first token. And batching, which rescues throughput for dense models, helps less with MoE, because different users wake different experts and the bytes read per step go back up. Five developers sharing one box means queueing plus single-digit tok/s each.

### Total cost

A capable box costs in the region of €4,000–5,000, plus electricity, plus someone to operate it. One person's cloud token spend is on the order of €25–100 a month. At the top of that range the hardware pays for itself in about 50 months – for one person, before power and operations. And the honest amortisation period is not five years but about 24 months: model architectures shift, and inference software keeps getting substantially faster on the hardware you didn't buy. The contrarian advice that follows: **if you don't have a real air gap, rent a dedicated GPU for a year instead of buying metal.** Buying makes sense for stable, high-volume batch work without bursts.

## Why local carries the burden of proof

Local is the right answer for: a genuine air gap; a contract that forbids sub-processors; raw health or legal data; high-volume, low-sensitivity batch; offline and edge deployments, where the missing network is the deciding factor rather than secrecy; and fine-tuning. It is the wrong answer for "it feels safer". With local AI the attack surface doesn't disappear – **it moves in with you**. An unpatched internal inference server without authentication is a larger risk than Azure OpenAI behind a private endpoint.

Local needs the stronger justification not because it is technically worse, but because reversibility is asymmetric.

> **Fig. 7** · A bad cloud decision is a monthly invoice. A bad local decision stands in the server room, signed – internally a matter of prestige, externally a reference risk.
>
> *Diagram:* Reversibility is asymmetric. A wrong cloud decision is operating expense: you switch or cancel and it is undone in about two weeks. A wrong local decision is capital expense: a signed purchase order and an asset tag that are still in the server room 24 months later.

Before the purchase order, have answers to these:

1. The **concrete use case**, and the data class it involves.
2. How many **concurrent** users – measured, not guessed.
3. The **typical prompt size**, because that decides whether prefill or decode is your bottleneck.
4. Whether there is an **eval**. If there is none, the project has already failed: you will have nothing to defend it with, and nothing to refute it with.
5. Who **operates** it – a name, not a team.
6. The **exit plan**.

## The model is a file; the rest is harness

A model really is a file. But what a user experiences as "the model" is almost entirely what sits around the weights – and that is where local set-ups succeed or fail.

> **Fig. 8** · Same weights, different harness, different product. Locally it is rarely the model's quality that lets you down. It is the tool-call parser, or a chat template that doesn't match the one the model was trained with.
>
> *Diagram:* The model weights are one file. Around them sits the harness: the chat template that formats the prompt, the inference engine with batching and the KV cache, structured output, and the tool-call parser. What users experience as the model is mostly the harness, and in local set-ups the tool-call parser is what breaks first.

- **Engines.** Ollama and LM Studio are developer convenience for one user. vLLM and TensorRT-LLM are production serving, with continuous batching. Whoever plans to serve twenty people from Ollama started in the wrong place. (Licensing, since it comes up: LM Studio has been free for work use since July 2025; you just can't redistribute or embed it.)
- **Quantisation** is not "a slightly dumber model". Benchmark scores barely move at 4-bit – but tool calling, JSON output and long-context instruction-following degrade first. Precisely what agentic use needs, and precisely what nobody measures while quoting MMLU.

### Chinese open-weight models: split the question in three

| Question | Answer |
| --- | --- |
| Does data go to China? | Run locally: no. Weights plus arithmetic – a GGUF file does not phone home. |
| Is the licence a problem? | The reverse of what people assume: Qwen and DeepSeek are largely Apache 2.0 / MIT, more permissive than the Llama community licence. |
| What remains? | Bias on certain topics, and supply chain: where you downloaded it, checksums, who produced the quantised build and the chat template. |

The risk isn't in the model; it is wherever you call an API. The same DeepSeek on its hosted chat and on your own hardware are two entirely separate risk profiles.

## The LLM provider is an adapter

For a boxed, on-premises product there is no "we decide". The customer decides, and every customer decides differently. So the LLM provider is an adapter – exactly like the database, the login and the file store in [the micromonolith post](https://laszlonemes.com/blog/micromonoliths).

The wire protocol is already solved: the OpenAI-compatible `/v1/chat/completions` is a de facto standard across Azure, vLLM, Ollama, LM Studio and OpenRouter. The hard part is what is *not* uniform behind it: how reliable tool calling is; structured output (real constrained decoding versus "please return JSON"); usable versus nominal context; content filtering (present on Azure, absent locally); rate-limit semantics and retry hints.

> **Fig. 9** · One adapter per level a customer might choose – their tenant (level 2), a local engine (level 1), a direct API (level 3). Design by capability discovery, not by if-statements on provider names. Degrade gracefully, and keep a zero-config default so the product starts before anyone has chosen a provider.
>
> *Diagram:* The LLM provider as an adapter. The application talks to one port. Adapters for Azure OpenAI, vLLM or Ollama, and a frontier API all speak the same chat completions protocol, and each reports its capabilities. The application branches on capabilities, not provider names: with constrained JSON it uses schema-enforced decoding, without it it prompts for JSON, validates and retries.

```csharp
public enum ToolCalling { None, BestEffort, Reliable }

public sealed record LlmCapabilities(
    ToolCalling ToolCalling,
    bool ConstrainedJson,        // can the engine enforce a JSON schema while decoding?
    int UsableContextTokens,     // what actually works, not what the model card says
    bool ContentFilter);         // present on Azure, absent on a local engine

public interface ILlmProvider
{
    LlmCapabilities Capabilities { get; }
    Task<ChatResult> ChatAsync(ChatRequest request, CancellationToken ct);
    Task<T> ChatJsonAsync<T>(ChatRequest request, JsonSchema schema, CancellationToken ct);
}
```

```csharp
// branch on what the provider can do, never on what it is called
public async Task<T> GetJsonAsync<T>(ChatRequest request, CancellationToken ct)
{
    if (provider.Capabilities.ConstrainedJson)
        return await provider.ChatJsonAsync<T>(request, JsonSchema.For<T>(), ct);

    for (var attempt = 1; attempt <= 3; attempt++)
    {
        var reply = await provider.ChatAsync(request.WithJsonInstructions<T>(), ct);
        if (JsonValidator.TryParse<T>(reply.Text, out var value))
            return value;

        request = request.WithFeedback("The reply was not valid JSON for the schema. Try again.");
    }

    throw new LlmOutputException("No valid JSON after 3 attempts.");
}
```

### The lock-in is the embedding model, not the chat model

> **Fig. 10** · The chat model is a configuration decision. The embedding model is a schema decision. You can replace the chat provider tomorrow; replacing the embedding model means re-indexing everything.
>
> *Diagram:* Swapping the chat model is one line of configuration and a restart. Swapping the embedding model is a migration: an index built with 3072-dimensional vectors cannot be queried with a 1024-dimensional model from a different vector space, so every document must be re-embedded and the index rebuilt.

Therefore the embedding model's name and dimension are **part of the schema**, and the index records what built it:

```json
{
  "index": "documents-v3",
  "embeddingModel": "text-embedding-3-large",
  "dimensions": 3072,
  "chunking": "v2"
}
```

The hidden cost of all this flexibility: every supported provider needs its own regression eval. Three providers is not three times the code – it is three times the evals and three times the support.

## Your own cloud tenant

My own default, for what it's worth: internal tools run on Azure OpenAI in our own tenant – a GPT model plus `text-embedding-3-large` for vectorisation. Even in a strongly technical team, "bring it on-prem" is not the default. And this is exactly where the embedding lock-in bites: if we moved local tomorrow, we wouldn't be swapping a chat model, we would be rebuilding the whole index.

> **Fig. 11** · The four levels from Fig. 1 again, this time as the path one prompt takes. Level 2 is the only one that gives you cloud models and a boundary you own: private networking, no keys in code, role-based access, a content-filter level you choose, token quotas per team as cost control – and every prompt loggable, which is both your audit trail and your best weapon against shadow AI.
>
> *Diagram:* The same prompt travelling through the four levels. Level one: it never leaves your building and reaches an inference server on your own hardware. Level two: inside your cloud tenant, the application uses a managed identity and a private endpoint to reach your model deployment, and the logs are yours. Level three: it crosses the public internet with an API key to the provider, whose logs you do not hold, under a contract. Level four: an employee's browser with a personal account sends it to a consumer app with no contract at all.

The downsides to accept knowingly: new model versions arrive later; model availability per region is a lottery; quota administration is real work; and your SLA hangs on their capacity.

### The region argument – say it the right way

The wrong way: *"It's in West Europe, so the data never leaves Europe, so GDPR is covered."* A data-protection officer or a lawyer shoots that down in one sentence, and everything you say afterwards is suspect.

> **Fig. 12** · Data residency is not data sovereignty. The CLOUD Act follows the company, not the data centre.
>
> *Diagram:* Data residency versus data sovereignty. Choosing an EU region fixes where data is stored and processed, which is residency: documented and auditable. It does not give sovereignty, because the CLOUD Act binds the US parent company wherever the hardware stands.

> An EU region determines where the data is physically stored and processed – **data residency**. That has real value under GDPR: processing stays in the EU, the provider's role as processor is fixed in the DPA, retention is governed by terms you can see, and the whole arrangement is **documentable and auditable**. What an EU region does *not* give you is **data sovereignty**: the CLOUD Act binds the US parent company regardless of where the hardware stands. So the right sentence is not "the data never leaves Europe" but "the risk is known, documented and acceptable – and somebody signed it".

- **A defensible baseline.** DPA, EU region, controlled retention and a documented data flow are enough for the overwhelming majority of use cases. Supervisors don't expect zero risk; they expect a reasoned, written decision.
- **Latency and cost.** Not a compliance argument at all – an engineering one, and sufficient on its own to justify the region.
- **The asymmetry, reversed.** If everyone flees into an air gap because of the CLOUD Act, the data ends up on a badly run server of your own. The CLOUD Act is a theoretical risk. The unpatched internal server is a practical one.

What "region" does not cover, and you must check separately: storage location and processing location are *separate* fields; retention (on Azure, abuse-monitoring retention can be switched off for approved use cases); sub-processors; support staff access; governing law. That is why it can't be settled in one sentence – and also exactly what makes it documentable. If someone pushes further, the next steps are a sovereign cloud (Bleu, Delos, the AWS European Sovereign Cloud) or genuine on-prem; say out loud that this is a narrow, regulated group, not the base case. *None of this is legal advice* – it is the wording a DPO doesn't reject after the first sentence.

## Two ways a RAG system leaks

> **Fig. 13** · The most common enterprise RAG failure is a bot that tells people things they have no right to see. Security trimming is the first feature, not an option. And if the model goes local, the embedding model and the vector store go with it.
>
> *Diagram:* Two ways a retrieval system leaks. First, without security trimming the index ignores the permissions the file share enforced, so the bot tells an intern what is in a board document. Second, a model moved on-premises while embeddings are still computed by a cloud API sends exactly the sensitive text out.

The underlying mistake in both panels is the same: forgetting that **the index is itself a data class**. It is exactly as sensitive as its sources, and it rarely carries the same permissions.

## Shadow AI

| Finding | Source |
| --- | --- |
| Employees regularly using AI on corporate devices: 15% → 45% in one year. Shadow AI is now the third most common non-malicious insider action; source code is the data most often uploaded. | Verizon DBIR 2026 |
| 66% of office professionals used AI at work while believing it was not permitted. 88% shared work information with public AI tools; 34% shared customer data. | PagerDuty 2026 |
| Shadow AI was a factor in 20% of breaches and added about $670,000 to the average cost. 97% of organisations with an AI-related incident lacked proper AI access controls. | IBM, 2025 |
| 47% of generative-AI users in enterprises reach the tools through personal accounts. | Netskope, 2026 |
| About 11% of what employees paste into chatbots is confidential. | Cyberhaven, 2023 |

Why it happens is no mystery – it is Shadow IT again. Nobody spun up a VM behind the device-management tooling out of malice; they wanted to get work done. The difference is that the gap between the good tool and the bad tool used to be 10%, and now it is a multiple. Under that much pressure, policy does not hold the wall. What is missing is not prohibition but an alternative.

> A ban is not control. All a ban achieves is that you can no longer *see* what is happening.

> **Fig. 14** · The forms that matter now. MCP servers are the most serious because an agent doesn't just read – it acts. “Sanctioned” shadow AI is the most embarrassing: Notion AI, Slack AI, Zoom AI – procurement signed it, security doesn't know, nobody turned it off.
>
> *Diagram:* Four newer forms of shadow AI around a developer's laptop: a personal API key in the IDE or CLI, unaudited MCP servers through which an agent acts on databases, mail and files, browser extensions with full page access, and sanctioned shadow AI, the AI features inside already-purchased SaaS tools that nobody reviewed.

### Detection: don't start with DLP

Start with what you already have: DNS and proxy logs; a review of the **OAuth grants in your SSO** (ten minutes, startling results); personal subscriptions on company cards. Only then CASB and DLP.

### What works

> **Fig. 15** · People take the faster road. You cannot out-policy that; you can only make the official road faster.
>
> *Diagram:* Someone needs AI for a task. The official road is a request followed by a six-week vendor review. The detour is a personal account that takes two minutes, and the work gets done invisibly with no log, contract or control. If the official road is slower than the detour you have lost, so make approval take one day.

1. **A paved road that is genuinely fast** – one day, not a six-week vendor review.
2. **Three data classes, not seven.** If you can't recite it, it is too complicated (Fig. 2).
3. **An amnesty with a time window** – "tell us now; in two weeks it becomes a problem".
4. **Measure usage, not compliance.** If training-completion rates go up, you have won nothing.

### Who is to blame

Everyone, asymmetrically. The company more: it holds the decision, the money and the procurement process, and if it provides no tool for six months, that was its choice. But the developer carries professional responsibility – "the process was inconvenient" is not a defence in a GDPR incident. And the distinction from Fig. 2 stands: pasting your own code is a policy violation; pasting customer data is an incident.

## The AI Act, in four lines

> **Fig. 16** · The Digital Omnibus was agreed in May 2026 and formally adopted in June. It moved the high-risk dates. It did not move Article 50.
>
> *Diagram:* Three EU AI Act dates: on 2 August 2026 the Article 50 transparency obligations apply and were not postponed, on 2 December 2026 the grace period for marking AI-generated content ends for systems already on the market, and on 2 December 2027 the postponed high-risk obligations apply.

The practical consequence for anyone shipping AI features: people must be able to tell when they are talking to an AI, and AI-generated content must be identifiable as such – and that duty sits with the deployer as well as with the model vendor.

## Objections, and the answers

| They say | You say |
| --- | --- |
| "Local AI is the privacy solution." | It is the control solution. Privacy is a process, not a box. |
| "One strong box is enough for the team." | One box is one or two concurrent users. Let’s look at the arithmetic. |
| "Cloud is more expensive." | Only if operating hours cost nothing and you amortise over five years. |
| "Chinese model = data leak." | Not when it runs locally. The risks are bias and supply chain. |
| "EU region = GDPR ticked." | Residency is not sovereignty. But DPA + EU region + controlled retention is defensible. |
| "Ban it until there is a policy." | A ban removes visibility, not usage. |
| "Let’s buy the hardware and figure out what for." | The burden of proof runs the other way: use case, eval, concurrency – then the PO. |
| "We’ll just swap the model later." | The chat model, yes. The embedding model means a full re-index. |

## Takeaways

1. **Data class → control level.** Four levels; your own cloud tenant is the right default.
2. **tok/s ≈ bandwidth ÷ bytes read per token.** Read GB/s on the spec sheet; prefill is the exception that needs compute.
3. **Large model, long context, many users: pick two.** One box is one or two concurrent users.
4. **Local must justify itself** – use case, eval, measured concurrency, named operator, exit plan – because capex is hard to undo.
5. **What you call "the model" is mostly the harness.** Quantisation hurts structured output first.
6. **Make the provider an adapter, branch on capabilities,** and treat the embedding model as schema.
7. **Residency is not sovereignty** – say the defensible sentence, not the comfortable one.
8. **Shadow AI is a product problem.** Build a road faster than the detour, and keep the logs.

## Sources

- [Verizon Data Breach Investigations Report 2026](https://www.verizon.com/business/resources/reports/dbir/) – AI usage on corporate devices, shadow AI as an insider action.
- [PagerDuty Shadow AI Survey 2026](https://www.pagerduty.com/newsroom/shadow-ai-workplace-survey-2026/) – 1,250 office professionals outside IT, at companies above $500M revenue.
- [IBM Cost of a Data Breach Report 2025](https://www.ibm.com/reports/data-breach) – the shadow-AI breach premium.
- [Netskope Cloud and Threat Report 2026](https://www.netskope.com/netskope-threat-labs/cloud-threat-report) and [Cyberhaven (2023)](https://www.cyberhaven.com/blog/4-2-of-workers-have-pasted-company-data-into-chatgpt) – personal accounts and pasted data.
- [Hardware Corner: DGX Spark vs. Strix Halo](https://www.hardware-corner.net/first-dgx-spark-llm-benchmarks/) and the [llama.cpp DGX Spark thread](https://github.com/ggml-org/llama.cpp/discussions/16578) – the prefill and decode measurements.
- [LM Studio is free for use at work](https://lmstudio.ai/blog/free-for-work) – the July 2025 licence change.
- [Gibson Dunn on the AI Act Omnibus agreement](https://www.gibsondunn.com/eu-ai-act-omnibus-agreement-postponed-high-risk-deadlines-and-other-key-changes/) and the [Article 50 guide](https://artificialintelligenceact.eu/transparency-rules-article-50/) – the dates in Fig. 16.
- [CLOUD Act](https://en.wikipedia.org/wiki/CLOUD_Act) – why jurisdiction follows the provider.