On-Prem Clinical AI Assistant
Design AI-assisted diagnosis where no patient data may leave the hospital network - self-hosted serving, data residency, local observability, and a rule-based fallback for when the model is down.
Last updated:
A healthcare client wants AI-assisted diagnosis but no patient data may leave their network. That single requirement removes every hosted frontier model from the design, which means you own serving, upgrades, monitoring and the failure path. It also raises the question most candidates skip: a clinical workflow cannot stall waiting for a GPU, so the system has to degrade to something safe rather than to nothing. The interesting engineering is in the boundary, the fallback and the release process, not in the prompt.
“No data leaves the network is an infrastructure requirement, not a prompt instruction. It costs you model capability and buys you residency, and the trade has to be made explicitly: scope the task down to what a self-hosted open-weights model can actually do well.”
Clarifying Questions (Ask These First ~5 min)
| Question | Why it matters |
|---|---|
| Fully air-gapped, or egress-restricted through a proxy? | Air-gapped means model weights and updates arrive on reviewed media |
| Decision support, or documentation support? | Suggesting a diagnosis is usually a regulated medical device; summarising notes usually is not |
| What hardware exists on site, and who operates it? | GPU count sets the model size; hospital IT sets the operating model |
| Is a clinician always in the loop with sign-off? | Assistive with sign-off is a different risk class to autonomous |
| Which data is in scope - notes, labs, imaging? | Multimodal changes both the model and the hardware budget |
| Required uptime, and what happens when it is not met? | The fallback path is part of the clinical workflow, not an afterthought |
| Who audits this, and how often? | Drives the evidence you must retain from day one |
Architecture (Draw This)
Say this as you draw it
Key Components (30-second pitch each)
- 1Model selection is a hardware decision - Open weights only - a Llama, Qwen, Mistral or medical-domain model that fits the GPUs the hospital actually has. Pin the version, quantise deliberately rather than accidentally, and measure on your own clinical eval set instead of a public leaderboard.
- 2The de-identification boundary applies internally too - PHI stays in the clinical zone, but logs, traces, metrics and eval exports all leave it. Redact at the emitter, not at the dashboard, so a debugging session can never become a disclosure.
- 3Guardrails that abstain - A scope classifier rejects questions outside the intended use, deterministic contraindication rules run independently of the model, and low-confidence or ungrounded output is suppressed. In clinical software, no answer is a safe outcome and a plausible wrong one is not.
- 4Retrieval over local guidelines, with citation - Suggestions are grounded in the hospital's own protocols and formulary, with a link to the guideline section. This is what makes a suggestion reviewable in seconds and what keeps the system on the assistive side of the regulatory line.
- 5Observability with no vendor telemetry - You get no provider dashboard, so build the equivalent: latency, queue depth, GPU utilisation, abstain rate, acceptance rate, and output-length distribution, all inside the boundary. Without them, model degradation is invisible until a clinician complains.
- 6A tested degraded mode - When inference is unavailable, the workflow continues on a deterministic rule set and templates, with a clear offline state. The failure is not that the fallback is crude - it is that nobody ever exercised it, so it breaks on the day it matters.
What Separates a Strong Answer
- 1Implement de-identification - A strong candidate ensures that all logs and metrics are de-identified at the emitter, preventing any accidental disclosure of PHI during debugging or monitoring.
- 2Use open weights models - Choosing open weights models that fit the hospital's hardware ensures compliance and control over model updates, avoiding reliance on external vendors.
- 3Design robust guardrails - Implementing guardrails that abstain on low-confidence outputs prevents misleading suggestions, which is crucial in a clinical setting where safety is paramount.
- 4Test degraded mode regularly - A strong candidate schedules regular failover drills to ensure the degraded path functions correctly, preventing workflow disruption during GPU unavailability.
What Changes When You Cannot Use a Hosted API
| Dimension | Consequence on-prem |
|---|---|
| Model choice | Open weights only - no frontier hosted model, so the capability ceiling drops |
| Task scope | Scope down to what the available model does reliably, rather than promising parity |
| Operations | You own serving, GPU failure, upgrades and the on-call rota |
| Observability | No vendor dashboard; Prometheus and Grafana inside the boundary |
| Cost shape | Capital expenditure and idle GPUs instead of per-token billing - utilisation becomes the metric |
| Updates | Signed artefacts, reviewed and imported, rather than an API version bump |
Degraded Modes
| Failure | Behaviour |
|---|---|
| One GPU node down | Route to remaining nodes; surface queue depth to the UI |
| All inference down | Deterministic rules and templates, explicit offline state, workflow continues |
| Retrieval down | No suggestion at all rather than an ungrounded one |
| Output fails guardrails | Suppress, log for review, show the relevant guideline instead |
3 Biggest Risks
- 1Automation bias - Clinicians under time pressure accept plausible suggestions. Mitigated by showing evidence before conclusion, requiring an explicit accept, sampling accepted suggestions for independent review, and tracking acceptance rate as a safety signal rather than a success metric.
- 2Silent model degradation - With no vendor telemetry, a quantisation change, a driver upgrade or a corrupted index degrades quality invisibly. Mitigated by a local golden set run on a schedule and on every change, with results trended inside the boundary.
- 3An untested fallback - The degraded path is written once and never exercised, so it fails on the day the GPUs do. Mitigated by scheduled failover drills in production hours, treated like any other disaster-recovery test.
Google Stack: Google Distributed Cloud (air-gapped) or GKE on-prem → open-weights or medical-domain Gemma served with vLLM → local FHIR store → Cloud Logging replaced by in-boundary Prometheus and Loki → signed model artefacts imported through Artifact Registry mirrors
Azure Stack: Azure Local (Azure Stack HCI) with Arc-enabled Kubernetes → open weights on AKS with vLLM → Azure API for FHIR deployed in-boundary → local Prometheus and Grafana, with only aggregate metrics leaving through Azure Monitor over Private Link if permitted
AWS Stack: AWS Outposts or EKS Anywhere → Bedrock is unavailable in an air-gapped site, so vLLM on local GPU nodes → S3-compatible local storage for the guideline corpus → CloudWatch replaced by in-boundary Prometheus, Grafana and Loki
Other Options: Bare-metal Kubernetes + vLLM or TGI + Qwen or Llama + Qdrant or Weaviate + Prometheus, Grafana and Loki, all entirely inside the network. This is the most common real answer in hospitals, because the constraint rules out the managed services that make the cloud stacks attractive.
Frequently asked questions
Can you fine-tune on patient data?
Yes, inside the boundary and with ethics approval on a de-identified corpus. The trap is memorisation - a fine-tuned model can emit training text verbatim, so the evaluation must include extraction probes, and the resulting weights inherit exactly the same residency rules as the data they were trained on.
How do you evaluate without sending anything out?
The whole harness runs inside: a clinician-curated golden set, local scoring, and only aggregate metrics with no records ever leaving for a vendor conversation. This is one reason to prefer deterministic scorers - they need no second model to run.
Is this a regulated medical device?
If it suggests a diagnosis then in most jurisdictions yes, and raising that unprompted is a strong signal in an interview. Design for it: a fixed model version per release, documented intended use, change control, clinical evaluation, and post-market surveillance of the acceptance and override rates.
How do updates reach an air-gapped site?
As signed artefacts on reviewed media, validated by checksum, staged in a non-clinical environment, run against the local golden set, and promoted only on a documented approval. Nothing auto-updates, including the base images.
How much hardware do you actually need?
Size by concurrent clinicians and token budget per interaction, not by headcount. Most hospital deployments of this shape are a handful of GPUs with batching, and the real capacity question is peak-hour concurrency, since utilisation outside clinic hours is close to zero.
Would a smaller model everywhere be better?
Often yes. A smaller model that abstains reliably, cites a guideline and responds in under two seconds beats a larger one that saturates the GPUs and times out at ward rounds. Latency and abstention discipline matter more here than benchmark scores.