Shadow AI and local AI: the data class decides

The conversation about where AI should run is usually framed as local versus cloud, and the conversation about employees using unapproved AI is usually framed as a discipline problem. Both framings are wrong in the same way: they skip the question that actually decides things – what kind of data is involved, and how much control that data needs.

The spine of this post. It isn't local versus cloud: the data class picks the level of control. There are four levels, not two. For local, the burden of proof is on local. And shadow AI is a product problem, not a disciplinary one.

Four levels, not two

Once there are four levels, the useful question is no longer "which one is best" but "which data may go where". That needs a data classification, and it needs to be short enough to remember.

Local AI: the arithmetic

Most disappointment with local models comes from three numbers nobody checked before buying hardware. All three follow from how a language model produces text: one token at a time, each after the other. (A token is roughly a syllable-sized piece, not a word – and languages like Hungarian need noticeably more of them per sentence than English.)

Two speeds, and you always feel the worse one

Memory bandwidth is the key, not GPU muscle

To produce each token, the model reads its active weights from memory. So decode speed is, to a first approximation, memory bandwidth divided by the bytes read per token. That one formula explains nearly everything you will measure.

So the number to look for on a spec sheet is GB/s, not TOPS or core counts. The unified-memory boxes sit around 256–273 GB/s; an M3 Ultra is about 819; a discrete GPU is around 1,000 and up – with far less memory attached. Prefill, on the other hand, is compute, and there the GPU matters: on the same 120B model an early llama.cpp benchmark measured 1,723 tok/s of prompt processing on a DGX Spark against 340 on a Strix Halo, while decode differed by only about 13% (38.6 against 34.1 tok/s). Later builds narrowed the prefill gap to roughly 2×. For chat you would never notice. For an agent it is the difference between usable and not.

What fits – and what that leaves room for

Rule of thumb: at 4-bit quantisation, the usual default, size in GB ≈ half the parameter count in billions. 70B is 35–40 GB, 120B is 60–70 GB, 235B is 115–120 GB. But the weights are not the only tenant of that memory. Every open conversation keeps a KV cache that grows with context length – and every concurrent user has their own.

Concurrency: the number missing from every ROI slide

One of these boxes serves one user comfortably, two with goodwill – not ten. Memory is one reason (Fig. 6). Prefill is another: it is compute-bound, so one person's 60,000-token prompt stalls everyone else's first token. And batching, which rescues throughput for dense models, helps less with MoE, because different users wake different experts and the bytes read per step go back up. Five developers sharing one box means queueing plus single-digit tok/s each.

Total cost

A capable box costs in the region of €4,000–5,000, plus electricity, plus someone to operate it. One person's cloud token spend is on the order of €25–100 a month. At the top of that range the hardware pays for itself in about 50 months – for one person, before power and operations. And the honest amortisation period is not five years but about 24 months: model architectures shift, and inference software keeps getting substantially faster on the hardware you didn't buy. The contrarian advice that follows: if you don't have a real air gap, rent a dedicated GPU for a year instead of buying metal. Buying makes sense for stable, high-volume batch work without bursts.

Why local carries the burden of proof

Local is the right answer for: a genuine air gap; a contract that forbids sub-processors; raw health or legal data; high-volume, low-sensitivity batch; offline and edge deployments, where the missing network is the deciding factor rather than secrecy; and fine-tuning. It is the wrong answer for "it feels safer". With local AI the attack surface doesn't disappear – it moves in with you. An unpatched internal inference server without authentication is a larger risk than Azure OpenAI behind a private endpoint.

Local needs the stronger justification not because it is technically worse, but because reversibility is asymmetric.

Before the purchase order, have answers to these:

1. 01The concrete use case, and the data class it involves.
2. 02How many concurrent users – measured, not guessed.
3. 03The typical prompt size, because that decides whether prefill or decode is your bottleneck.
4. 04Whether there is an eval. If there is none, the project has already failed: you will have nothing to defend it with, and nothing to refute it with.
5. 05Who operates it – a name, not a team.
6. 06The exit plan.

The model is a file; the rest is harness

A model really is a file. But what a user experiences as "the model" is almost entirely what sits around the weights – and that is where local set-ups succeed or fail.

• —Engines. Ollama and LM Studio are developer convenience for one user. vLLM and TensorRT-LLM are production serving, with continuous batching. Whoever plans to serve twenty people from Ollama started in the wrong place. (Licensing, since it comes up: LM Studio has been free for work use since July 2025; you just can't redistribute or embed it.)
• —Quantisation is not "a slightly dumber model". Benchmark scores barely move at 4-bit – but tool calling, JSON output and long-context instruction-following degrade first. Precisely what agentic use needs, and precisely what nobody measures while quoting MMLU.

Chinese open-weight models: split the question in three

The risk isn't in the model; it is wherever you call an API. The same DeepSeek on its hosted chat and on your own hardware are two entirely separate risk profiles.

The LLM provider is an adapter

For a boxed, on-premises product there is no "we decide". The customer decides, and every customer decides differently. So the LLM provider is an adapter – exactly like the database, the login and the file store in the micromonolith post.

The wire protocol is already solved: the OpenAI-compatible /v1/chat/completions is a de facto standard across Azure, vLLM, Ollama, LM Studio and OpenRouter. The hard part is what is not uniform behind it: how reliable tool calling is; structured output (real constrained decoding versus "please return JSON"); usable versus nominal context; content filtering (present on Azure, absent locally); rate-limit semantics and retry hints.

public enum ToolCalling { None, BestEffort, Reliable }

public sealed record LlmCapabilities(
    ToolCalling ToolCalling,
    bool ConstrainedJson,        // can the engine enforce a JSON schema while decoding?
    int UsableContextTokens,     // what actually works, not what the model card says
    bool ContentFilter);         // present on Azure, absent on a local engine

public interface ILlmProvider
{
    LlmCapabilities Capabilities { get; }
    Task<ChatResult> ChatAsync(ChatRequest request, CancellationToken ct);
    Task<T> ChatJsonAsync<T>(ChatRequest request, JsonSchema schema, CancellationToken ct);
}

// branch on what the provider can do, never on what it is called
public async Task<T> GetJsonAsync<T>(ChatRequest request, CancellationToken ct)
{
    if (provider.Capabilities.ConstrainedJson)
        return await provider.ChatJsonAsync<T>(request, JsonSchema.For<T>(), ct);

    for (var attempt = 1; attempt <= 3; attempt++)
    {
        var reply = await provider.ChatAsync(request.WithJsonInstructions<T>(), ct);
        if (JsonValidator.TryParse<T>(reply.Text, out var value))
            return value;

        request = request.WithFeedback("The reply was not valid JSON for the schema. Try again.");
    }

    throw new LlmOutputException("No valid JSON after 3 attempts.");
}

The lock-in is the embedding model, not the chat model

Therefore the embedding model's name and dimension are part of the schema, and the index records what built it:

{
  "index": "documents-v3",
  "embeddingModel": "text-embedding-3-large",
  "dimensions": 3072,
  "chunking": "v2"
}

The hidden cost of all this flexibility: every supported provider needs its own regression eval. Three providers is not three times the code – it is three times the evals and three times the support.

Your own cloud tenant

My own default, for what it's worth: internal tools run on Azure OpenAI in our own tenant – a GPT model plus text-embedding-3-large for vectorisation. Even in a strongly technical team, "bring it on-prem" is not the default. And this is exactly where the embedding lock-in bites: if we moved local tomorrow, we wouldn't be swapping a chat model, we would be rebuilding the whole index.

The downsides to accept knowingly: new model versions arrive later; model availability per region is a lottery; quota administration is real work; and your SLA hangs on their capacity.

The region argument – say it the right way

The wrong way: "It's in West Europe, so the data never leaves Europe, so GDPR is covered." A data-protection officer or a lawyer shoots that down in one sentence, and everything you say afterwards is suspect.

An EU region determines where the data is physically stored and processed – data residency. That has real value under GDPR: processing stays in the EU, the provider's role as processor is fixed in the DPA, retention is governed by terms you can see, and the whole arrangement is documentable and auditable. What an EU region does not give you is data sovereignty: the CLOUD Act binds the US parent company regardless of where the hardware stands. So the right sentence is not "the data never leaves Europe" but "the risk is known, documented and acceptable – and somebody signed it".

• —A defensible baseline. DPA, EU region, controlled retention and a documented data flow are enough for the overwhelming majority of use cases. Supervisors don't expect zero risk; they expect a reasoned, written decision.
• —Latency and cost. Not a compliance argument at all – an engineering one, and sufficient on its own to justify the region.
• —The asymmetry, reversed. If everyone flees into an air gap because of the CLOUD Act, the data ends up on a badly run server of your own. The CLOUD Act is a theoretical risk. The unpatched internal server is a practical one.

What "region" does not cover, and you must check separately: storage location and processing location are separate fields; retention (on Azure, abuse-monitoring retention can be switched off for approved use cases); sub-processors; support staff access; governing law. That is why it can't be settled in one sentence – and also exactly what makes it documentable. If someone pushes further, the next steps are a sovereign cloud (Bleu, Delos, the AWS European Sovereign Cloud) or genuine on-prem; say out loud that this is a narrow, regulated group, not the base case. None of this is legal advice – it is the wording a DPO doesn't reject after the first sentence.

Two ways a RAG system leaks

The underlying mistake in both panels is the same: forgetting that the index is itself a data class. It is exactly as sensitive as its sources, and it rarely carries the same permissions.

Shadow AI

Why it happens is no mystery – it is Shadow IT again. Nobody spun up a VM behind the device-management tooling out of malice; they wanted to get work done. The difference is that the gap between the good tool and the bad tool used to be 10%, and now it is a multiple. Under that much pressure, policy does not hold the wall. What is missing is not prohibition but an alternative.

A ban is not control. All a ban achieves is that you can no longer see what is happening.

Detection: don't start with DLP

Start with what you already have: DNS and proxy logs; a review of the OAuth grants in your SSO (ten minutes, startling results); personal subscriptions on company cards. Only then CASB and DLP.

What works

1. 01A paved road that is genuinely fast – one day, not a six-week vendor review.
2. 02Three data classes, not seven. If you can't recite it, it is too complicated (Fig. 2).
3. 03An amnesty with a time window – "tell us now; in two weeks it becomes a problem".
4. 04Measure usage, not compliance. If training-completion rates go up, you have won nothing.

Who is to blame

Everyone, asymmetrically. The company more: it holds the decision, the money and the procurement process, and if it provides no tool for six months, that was its choice. But the developer carries professional responsibility – "the process was inconvenient" is not a defence in a GDPR incident. And the distinction from Fig. 2 stands: pasting your own code is a policy violation; pasting customer data is an incident.

The AI Act, in four lines

The practical consequence for anyone shipping AI features: people must be able to tell when they are talking to an AI, and AI-generated content must be identifiable as such – and that duty sits with the deployer as well as with the model vendor.

Objections, and the answers

Takeaways

1. 01Data class → control level. Four levels; your own cloud tenant is the right default.
2. 02tok/s ≈ bandwidth ÷ bytes read per token. Read GB/s on the spec sheet; prefill is the exception that needs compute.
3. 03Large model, long context, many users: pick two. One box is one or two concurrent users.
4. 04Local must justify itself – use case, eval, measured concurrency, named operator, exit plan – because capex is hard to undo.
5. 05What you call "the model" is mostly the harness. Quantisation hurts structured output first.
6. 06Make the provider an adapter, branch on capabilities, and treat the embedding model as schema.
7. 07Residency is not sovereignty – say the defensible sentence, not the comfortable one.
8. 08Shadow AI is a product problem. Build a road faster than the detour, and keep the logs.

Sources

• —Verizon Data Breach Investigations Report 2026 – AI usage on corporate devices, shadow AI as an insider action.
• —PagerDuty Shadow AI Survey 2026 – 1,250 office professionals outside IT, at companies above $500M revenue.
• —IBM Cost of a Data Breach Report 2025 – the shadow-AI breach premium.
• —Netskope Cloud and Threat Report 2026 and Cyberhaven (2023) – personal accounts and pasted data.
• —Hardware Corner: DGX Spark vs. Strix Halo and the llama.cpp DGX Spark thread – the prefill and decode measurements.
• —LM Studio is free for use at work – the July 2025 licence change.
• —Gibson Dunn on the AI Act Omnibus agreement and the Article 50 guide – the dates in Fig. 16.
• —CLOUD Act – why jurisdiction follows the provider.