Healthcare data is among the most sensitive data there is. Sending it to a cloud AI service is often not an option because of privacy requirements, regulatory compliance, or both.

In this tutorial, you’ll build a healthcare AI assistant that runs entirely on your own machine using three open-source tools:

By the end, you’ll be able to chat with a medically tuned AI model, upload medical images such as chest X-rays for analysis, and do it all locally, without sending your data to the cloud.

Important disclaimer before we start: MedGemma is a developer model, not a medical device. Its outputs are not intended to directly inform clinical diagnosis, patient management, or treatment decisions.

Everything you build in this tutorial is for learning, prototyping, and research. Always consult qualified healthcare professionals for real medical questions.

What We'll Cover:

Who is This Tutorial For?

This tutorial is ideal if you’re:

What is MedGemma?

MedGemma is a collection of open models from Google, built on the Gemma 3 architecture and specifically trained for medical text and image comprehension. Think of it as Gemma after four years of medical school and a radiology residency.

Diagram showing MedGemma’s multimodal architecture, where medical images are processed by a SigLIP vision encoder and combined with a language model to understand medical text and images and generate responses.

Why MedGemma?

Unlike general-purpose models such as Llama or Mistral, MedGemma is designed specifically for healthcare applications.

MedGemma is intended as a foundation model for developers building healthcare applications, medical education tools, research assistants, report summarizers, and other AI-powered medical workflows.

Why Run Models Locally?

You could call a hosted medical model through an API. So why go local? In healthcare, the case is stronger than almost anywhere else.

First, there's the principle of privacy by architecture. When the model runs on your machine, medical text and images never leave your device. There's no API log, no third-party data processor, no data processing agreement to negotiate.

For anyone working near PHI (Protected Health Information), "the data never left the laptop" is the simplest compliance story that exists.

Next, you have zero per-token cost. Experimentation is free once the model is downloaded. You can iterate on prompts hundreds of times without watching a billing dashboard.

You also get offline access. Hospitals, labs, and field clinics often have restricted or air-gapped networks. A local model works without internet after the initial download.

And you have full control over the setup: you choose the model version, you pin it, and it never changes underneath you. No deprecation notices, no silent behavior changes.

Finally, it's a great way to learn. Running models locally demystifies them. You'll develop intuition for context windows, quantization, and memory constraints that you simply don't get from calling an API.

Prerequisites

Here's what you need before starting:

Hardware:

Software:

That's it. No API keys, no accounts, and no GPU cloud credits.

Architecture Diagram

Architecture diagram showing Open WebUI connected to Ollama, which runs the MedGemma model locally on the user’s computer. All medical text and image processing happens on the local machine without using cloud services.

Step 1: Install Ollama

Ollama is a lightweight runtime that handles downloading, quantizing, and serving open models through a simple CLI and a local REST API.

On macOS:

Download the app from ollama.com/download and drag it to Applications, or install via Homebrew:

brew install ollama

On Linux:

curl -fsSL https://ollama.com/install.sh | sh

On Windows:

Download the native Windows installer from ollama.com/download and run it. (Ollama now supports Windows natively, no WSL required.)

Once installed, verify it works:

ollama --version

You should see a version number printed. Ollama also starts a background service that listens on http://localhost:11434. This is the API that Open WebUI will talk to later. You can confirm the server is up with:

curl http://localhost:11434

which should return Ollama is running.

Step 2: Pull MedGemma

MedGemma is available directly in the official Ollama model library, so downloading it is one command:

ollama pull medgemma

This pulls the default 4B multimodal variant, about a 3.3 GB download.

If you want to be explicit about the size (useful when you later experiment with the 27B model):

ollama pull medgemma:4b     # 3.3 GB — multimodal, runs on most laptops
ollama pull medgemma:27b    # 17 GB — multimodal, needs serious hardware

When the download finishes, confirm the model is installed:

ollama list

You should see medgemma in the output along with its size.

Step 3: Test MedGemma from the Terminal

Before adding a UI, let's make sure the model actually works. Start an interactive session:

ollama run medgemma

You'll get a >>> prompt. Try a medical question:

>>> What are the classic radiographic signs of pneumonia on a chest X-ray?

MedGemma should respond with a structured answer covering findings like consolidation, air bronchograms, and silhouette signs — the kind of answer that shows its radiology training.

Try one more to see the clinical reasoning:

>>> Explain the difference between Type 1 and Type 2 diabetes to a first-year medical student.

A few useful commands inside the session:

You can also test image input directly from the terminal by passing a file path directly in the prompt:

>>> Describe the key findings in this image. ./chest_xray_sample.png

While this works, uploading images through Open WebUI is much more convenient.

Step 4: Install Open WebUI

Open WebUI gives you a clean, ChatGPT-style interface on top of Ollama: conversation history, model switching, image uploads, and multi-user support, all self-hosted.

Start by installing Docker.

Make sure Docker Desktop is running, then launch Open WebUI with:

docker run -d -p 3000:8080 \
  --add-host=host.docker.internal:host-gateway \
  -v open-webui:/app/backend/data \
  --name open-webui \
  --restart always \
  ghcr.io/open-webui/open-webui:main

Let's break down what this command does:

Option B: pip (no Docker)

If you'd rather skip Docker, you can instead install Open WebUI as a Python package (Python 3.11 is the supported version):

pip install open-webui
open-webui serve

This starts the interface at http://localhost:8080 instead of port 3000.

Step 5: Connect Open WebUI to Ollama

Open your browser and go to http://localhost:3000 (or :8080 if you used pip).

On first launch, you'll be asked to create an admin account. This account is stored locally on your machine (it's not a cloud signup).

In most setups, Open WebUI auto-detects Ollama at http://localhost:11434 and you're done.

If your models don't appear, wire up the connection manually:

  1. Click your profile icon and go to Admin Panel then Settings then Connections.

  2. Under Ollama API, set the URL:

    • Docker install: http://host.docker.internal:11434

    • pip install: http://localhost:11434

  3. Click the refresh icon to verify the connection, then save.

Head back to the main chat screen, and medgemma should now appear in the model dropdown at the top.

You can check the troubleshooting section below if you face any errors.

Step 6: Start Chatting with MedGemma

Select medgemma from the model selector and start a conversation. A good first test might look like this:

Summarize this radiology report in plain language a patient could understand:

"Impression: Mild cardiomegaly. Small right pleural effusion.
No focal consolidation. Degenerative changes of the thoracic spine."

You should get a clear, patient-friendly explanation of each finding. This "clinical language to plain language" translation is one of MedGemma's genuine strengths.

There are a few Open WebUI features worth knowing about:

Step 7: Upload Medical Images

This is where MedGemma really separates itself from general-purpose models. Because its vision encoder was pre-trained on medical imaging, it can meaningfully describe radiographs, skin lesions, fundus photos, and histopathology patches.

To try it:

  1. Start a new chat with medgemma selected.

  2. Click the + (or image) icon in the message box, or simply drag and drop an image file.

  3. Add a prompt alongside the image and hit send.

For sample images you can test with (without touching any real patient data), try public teaching datasets like the NIH ChestX-ray14 dataset, MedPix, or Radiopaedia's teaching cases.

Example workflow with a chest X-ray:

[Upload: chest_xray.png]

You are an expert radiology assistant. Describe this chest X-ray
systematically: technical quality, lungs, heart, mediastinum, bones,
and soft tissues. Then summarize the key findings.

MedGemma will typically walk through the image in the systematic order you asked for, which mirrors how radiologists are trained to read films.

Two important caveats:

Example Prompts to Try

Here are prompts that showcase different capabilities. Use them as starting points:

Medical education:

Create a comparison table of ACE inhibitors vs ARBs: mechanism, common examples, key side effects, and contraindications.

Clinical documentation:

Convert these shorthand clinic notes into a structured SOAP note:"45F, 3d cough + fever 101F, no SOB, lungs clear, likely viral URI, supportive care, return if worse"

Report translation for patients:

Explain this MRI impression to a worried patient in a reassuring but honest tone: "Small disc protrusion at L4-L5 without significant canal stenosis or nerve root compression."

Image analysis (with an uploaded dermatology photo):

Describe this skin lesion using the ABCDE criteria
(Asymmetry, Border, Color, Diameter, Evolution cannot be assessed from a single image — note that explicitly).

Differential reasoning:

A 60-year-old presents with sudden painless vision loss in one eye. List the top 5 differential diagnoses and the key distinguishing feature of each.

Notice a pattern: the best results come from prompts that give MedGemma a role, a structure to follow, and explicit constraints. That's true of all LLMs, but it matters even more in a domain where precision counts.

Running Larger Models

The 4B model is impressive for its size, but the 27B variant is noticeably stronger at complex clinical reasoning, longer differential diagnoses, and nuanced report interpretation.

The trade-off is hardware:

Model Download Realistic RAM/VRAM needed Best for
medgemma:4b 3.3 GB 8 GB+ RAM Laptops, quick iteration, image Q&A
medgemma:27b 17 GB 32 GB RAM or 24 GB VRAM Deep reasoning, complex cases

To try the 27B model:

ollama pull medgemma:27b
ollama run medgemma:27b

Practical tips for larger models:

You can keep both installed and switch between them in the Open WebUI dropdown — 4B for fast iteration, 27B when you need the deeper reasoning.

Troubleshooting Guide

Error: registry.ollama.ai/library/medgemma:latest does not support tools

This is the most common MedGemma-specific error, and it means Open WebUI is sending native tool/function definitions with your request. MedGemma (like base Gemma 3) doesn't support Ollama's tools API, so the request is rejected before the model even sees your message.

Hunt down whatever is attaching tools, in this order:

  1. Model capabilities (most likely culprit): Go to the Admin Panel, then Settings, then Models, then medgemma, then uncheck Builtin Tools, Web Search, Code Interpreter, and Terminal under Capabilities, and make sure every item in the Builtin Tools checklist is unticked. Keep Vision, File Upload, and File Context checked. Newer Open WebUI versions enable builtin tools by default, so a fresh install will hit this immediately.

  2. Task model: Go to Admin Panel, then Settings, then Interface, and make sure neither the local nor external Task Model is set to medgemma. Background jobs like title and follow-up generation use tool calls — route them to llama3.2 or similar.

  3. Function Calling mode: Set to Default (not Native) in the model's Advanced Params and in your user Settings, General, Advanced Parameters.

  4. Global functions/filters: Go to Admin Panel, then Functions, and disable the Global toggle on any active function, since global functions attach to every model.

  5. Per-chat toggles: In the message box, make sure web search and code interpreter toggles are off, and no Tools are attached via the + menu.

Then start a new chat (old chats can carry stale settings) and test. To confirm the model itself is fine, run ollama run medgemma "hello" in your terminal. If that works, the issue is purely Open WebUI configuration.

Open WebUI shows no models in the dropdown

The container can't reach Ollama. Check that:

ollama pull medgemma says model not found

Update Ollama, as MedGemma requires a recent version. Re-run the installer or, on macOS, click the menu bar icon and then Update. Then retry the pull.

Responses are extremely slow

Image upload doesn't work or the model ignores the image

Port 3000 is already in use

Map a different host port: change -p 3000:8080 to -p 3001:8080 and access the UI at http://localhost:3001.

"Out of memory" errors when loading the 27B model

Your machine doesn't have enough free RAM/VRAM. Stick with medgemma:4b, or free memory and try again. There is no shame in the 4B model — it punches well above its weight.

Conclusion

In this tutorial, you built a complete, private healthcare AI assistant from scratch — and it took three tools and a handful of terminal commands.

Let's recap what you accomplished:

You now have a fully private AI assistant running entirely on your own machine. From here, you can extend it with retrieval-augmented generation (RAG), integrate it with medical imaging pipelines, or connect it to de-identified clinical datasets to build more advanced healthcare AI applications.

Happy building!

Further reading: