The problem
When you work with high-sensitivity data, three facts tend to hold at once:
- The intelligence is outside. The most capable models mostly run on the provider’s servers, and you reach them through an API.
- The data is inside. Your data sits in your own systems or a private cloud.
- The data cannot leave. Compliance and trade secrets mean that, by default, you do not send it to an external model.
Put these three together and they conflict. The smart brain can’t see the homework, and the brain that can see the homework isn’t very smart.
So the real question is this: how do you let the intelligence meet the data?
Either the data goes out, or the intelligence comes in
There are only two kinds of answer.
Send out only what is necessary, or bring the intelligence inside a trust boundary you define.
In practice you combine several of the paths below.
Data goes out
Path A: make the necessary data small and clean, then send it out. Use RAG to fetch what is relevant to the question, run access checks, minimize the data, and redact it in advance, so that only the necessary part reaches the external model.
Path B: let the data enter a hardware-protected, verifiable compute environment. The data stays encrypted in transit and at rest, and computation happens only after it enters a verified, hardware-isolated environment. That environment is called a TEE (Trusted Execution Environment), and the technology is also known as Confidential Computing.
Intelligence comes in
Path C: use local data to strengthen a local model. The data never crosses the trust boundary. You use your own data and fine-tuning to turn a local model into a specialist for one domain.
Path D: use intelligence to teach a local model. A stronger model acts as the teacher, generates training samples or supervision signals, and transfers part of its capability to a local student model. This is Knowledge Distillation.
What each path can do
Path A: RAG
RAG stands for Retrieval-Augmented Generation.
In RAG, documents are usually split into chunks and indexed first. The index can be a vector index, a BM25 or full-text index, a structured database, or some combination.
When the user asks a question, you retrieve the relevant chunks and send them to an internal or external model.
If you send them to an external model, one point matters: even as fragments, the data is still real. So it needs extra handling.
- Access Control: confirm the user is allowed to see the data that was retrieved.
- Data minimization: send only what the answer requires.
- Local embedding and reranking: do the vectorization and ranking locally.
- PII redaction and masking: use a local model or rules to replace or remove the real sensitive values, and fill them back in later if needed once the answer returns.
- Query rewriting: abstract away the specifics of the question.
A more recent proposal, SAG (SQL-Retrieval Augmented Generation), is one concrete approach to structured retrieval. It represents chunks as events and indexing entities, then uses SQL joins at query time to build local associations, which suits cross-document linking and multi-hop retrieval. It is still a fairly new and specific architecture.
Retrieval quality sets the ceiling for this path. If you cannot retrieve the key information, or too much irrelevant information comes back, no amount of model intelligence will save it. Beyond that, the raw data quality, access control, reranking, context assembly, and the generation model all affect the final result.
Path B: A hardware-protected compute environment
The data stays encrypted in transit and at rest. Once it enters the hardware-isolated secure area, only permitted code can use it. This area is a TEE (Trusted Execution Environment), and the technology is usually called Confidential Computing.
The point of a TEE is to protect data in use, so that programs and objects outside the TEE cannot easily read or tamper with the data inside.
It usually also supports Attestation. Before releasing keys and sending sensitive data, you can verify that the other side is running the agreed hardware, code version, and configuration.
But Attestation cannot prove that a program is free of bugs, and it cannot stop a badly designed program from leaking its results on purpose.
Cloud and hardware vendors already offer confidential VMs, containers, and GPUs. Whether a specific model service supports verifiable confidential inference still depends on the vendor and the deployment.
The downside is the extra cost, performance overhead, hardware constraints, and operational complexity it can bring.
Path C: Fine-tuning
Change the direction of the intelligence. Instead of making a small model bigger, make it a specialist.
Pick a self-hostable open-weight model, deploy it locally or in an approved private environment, and fine-tune it on the data you are allowed to use. To lower training cost, use a PEFT method such as LoRA. For implementation you can use frameworks like Hugging Face PEFT, Unsloth, or Axolotl, or a managed platform such as AWS SageMaker where compliance allows it.
A local model usually cannot match a larger frontier model on general capability. But on well-bounded tasks, such as ticket classification, field extraction, format conversion, and fixed-style rewriting, a small model can beat a general large one.
Fine-tuning is good for learning stable behavior, format, and task patterns. It is not good for storing facts that need frequent updates, exact citation, and timely deletion. That kind of knowledge is usually still better served through RAG.
The hard parts of this path are training compute, an evaluation system, and MLOps. The deeper trap is data quality, because garbage in, garbage out. After training on sensitive data, you also need to check whether the model memorizes or leaks its training content.
Path D: Knowledge Distillation
Classic Knowledge Distillation has the student learn the teacher’s output distribution or other supervision signals.
In current LLM engineering, a behavioral form is more common. You prepare synthetic data, public data, or redacted data, and a large teacher model generates candidate answers, structured outputs, or verifiable intermediate results. You then filter with rules, program execution, historical results, or domain experts to form high-quality input-output pairs, and train the local student with supervised fine-tuning.
It combines naturally with Path C. Distillation supplies the training target, and fine-tuning makes the student learn the behavior. This pairing suits high-frequency, stable, patterned tasks.
One caution: the teacher’s output is not a fully correct label. Without verification, the teacher’s mistakes get distilled into the student too.
The paths form a loop
In practice these paths form a loop, and you measure it with local handling rate.
Local handling rate is the share of requests you can complete without calling an external model, while still meeting your quality and risk requirements. The goal is to push this number as high as you can without breaking quality, security, and cost limits.
The front half is RAG. If the model only needs to restate or summarize a chunk, a local model will do. If you need a stronger model, redact first, send it out, and get a high-quality answer.
With data governance in place, these interactions can become candidate training data. To become an actual training set, they still need checks on data permissions, retention period, output quality, source tracking, and human feedback.
If you stop here, the system is only usable. To make it good, you need a quality-controlled feedback loop.
So the next step is to bring in Distillation and Fine-tuning. Take the filtered and verified candidate data from the previous step, combine it with governed internal data, and fine-tune the model internally so it becomes more specialized. This moves the system from usable to good.
For high-difficulty, high-sensitivity edge cases, if the model you need supports verifiable TEE, you can use Confidential Computing. Otherwise the system still has to hand off to a human or decline to handle the request automatically.
Once the loop is turning, the meeting point between intelligence and data moves closer and closer to the inside of your trust boundary. In the end, more of the requests suited to local handling can stay inside. For high-frequency, stable tasks, this can shrink how much sensitive data is exposed outside, and can lower the marginal cost of inference.
