Testing a Medical Image De-identification App with Passmark: How I Built and Tested Aegis

The bottleneck nobody automates

If you work with medical imaging data, you've probably heard this story or lived it. A researcher needs 500 ultrasound images for a study. Every single one has patient names, dates, and hospital IDs burned into the pixels. Not in DICOM metadata that you can strip programmatically. Literally rendered as text overlays on the image itself.

The standard approach? Open each image one by one in a tool like 3D Slicer or RadiAnt DICOM Viewer, manually locate the PHI regions, black them out, save, repeat. For 500 images, that's hours of mind-numbing work. For 10,000 images: a modest dataset by today's standards, it's simply not feasible.

This is why I built Aegis — an open-source pipeline that does this automatically, in batch, across entire datasets. For the Breaking Apps Hackathon, I deployed it as a web demo and wrote an AI-powered test suite for it using Passmark.

What is Aegis?

Aegis is an open-source medical image de-identification pipeline. It handles both DICOM files (the native format of medical scanners) and standard JPEG/PNG images. The core pipeline has three layers:

Layer 1 — Clinical Allowlist. Device manufacturer names like "Mindray" or "GE", imaging modes like "2D" or "Doppler", and measurement text like "D 13.0cm" are recognized and preserved. You don't want to black out the clinical information that makes the image diagnostically useful.

Layer 2 — PHI Heuristics. Regex patterns catch obvious PHI: date-ID combinations (20260117-091825), hospital names, doctor prefixes (Dr. Smith), and long numeric IDs. These are redacted immediately without needing AI inference.

Layer 3 — Stanford NER. Everything that wasn't caught by layers 1–2 goes through the Stanford De-identifier NER model (F1 ≥ 97.9%), which classifies text into 22 entity types — PATIENT, DOCTOR, HOSPITAL, DATE, IDNUM, and more. Any PHI entity gets redacted.

The key difference from manual tools: Aegis processes entire datasets in batch. No clicking through images one by one. Point it at a folder, walk away, and come back to a clean dataset.

Deploying to Hugging Face Spaces with Gradio

For the hackathon, I needed a publicly accessible web app — no bot checks, no login walls. Hugging Face Spaces with Gradio was the natural choice.

The demo lets you see the pipeline in action on a single image:

1. Upload a medical image (JPEG/PNG)

2. Click "De-identify Image"

3. See the redacted output, an annotated overlay showing what was detected, and a detailed report

Here's the core of the classification logic:

def classify_text(text): 

  # Layer 1 — clinical allowlist 
  if is_clinical_allowlisted(text): return {"is_phi": False,    "label": "CLINICAL"}

  # Layer 2 — clinical patterns (probe IDs, measurements)
  if matches_clinical_pattern(text):
    return {"is_phi": False, "label": "CLINICAL"}

  # Layer 3 — PHI heuristics
  if matches_phi_heuristic(text):
    return {"is_phi": True, "label": "PHI_HEURISTIC"}

  # Layer 4 — Stanford NER model
  if classify_with_ner(text):
    return {"is_phi": True, "label": "PHI_NER"}

return {"is_phi": False, "label": "SAFE"}

Each text region found by EasyOCR goes through this pipeline. The order matters — cheap checks first (allowlist, regex), expensive AI inference last. This keeps batch processing fast.

Deployment was three files pushed to a HF Spaces repo:

• app.py — the Gradio application

• requirements.txt — dependencies (gradio, easyocr, transformers, torch)

• README.md — HF Spaces metadata

Writing Passmark Tests

This is where it gets interesting. Passmark lets you write Playwright tests in plain English. The AI figures out how to interact with your app — no CSS selectors, no data-testid attributes, no brittle XPath queries.

This matters especially for Gradio apps. Gradio generates dynamic HTML where the DOM structure isn't designed for test automation. Traditional Playwright tests would mean reverse-engineering the class names and element hierarchy. Passmark skips that entirely.

Here's my test for the core de-identification flow:

import { test, expect } from "@playwright/test";
import { runSteps } from "passmark";

test("Upload image and verify de-identification runs", async ({ page }) => {
  test.setTimeout(180_000);

  await runSteps({
    page,
    userFlow: "Load Aegis app",
    steps: [
      { description: "Navigate to the Aegis app URL" },
      {
        description: "Wait for page to load",
        waitUntil: "The page shows text about Medical Image and has a large blue button",
      },
    ],
    assertions: [{ assertion: "You can see an image upload area on the page" }],
    test,
    expect,
  });

  // Native Playwright for file upload (Gradio's hidden input)
  const fileInput = page.locator('input[type="file"]');
  await fileInput.setInputFiles("tests/fixtures/sample_medical_image.png");
  await page.waitForTimeout(2000);

  await runSteps({
    page,
    userFlow: "Run de-identification",
    steps: [
      {
        description: "Click the large blue button below the image upload area",
        waitUntil: "A report appears with words like REDACTED or PRESERVED",
      },
    ],
    assertions: [
      { assertion: "You can see text mentioning 'Total text regions detected'" },
      { assertion: "You can see an output image on the right side of the page" },
    ],
    test,
    expect,
  });
});

One practical lesson: I used a hybrid approach — Passmark for navigation and assertions, native Playwright for the file upload. Gradio's file input is a hidden <input type="file"> element, and Passmark's AI agent struggled with the drag-and-drop widget. Native Playwright's setInputFiles works reliably. This hybrid pattern is worth knowing for anyone testing Gradio or Streamlit apps with Passmark.

The full test suite covers five flows:

1. App loads correctly — heading, upload area, and primary button exist

2. Upload and de-identify — the core happy path

3. Tab switching — Redacted ↔ Annotated views work

4. Empty state — clicking without an upload shows a warning

5. Report structure — output table has the expected columns (Text, Confidence, Classification, Action)

Setting up the test project

# Initialize
mkdir aegis-passmark-tests && cd aegis-passmark-tests
npm init -y
npm install @playwright/test passmark dotenv
npx playwright install chromium

# Add your API keys (.env)
OPENROUTER_API_KEY=sk-or-...  # free credits from hackathon registration

# Run tests
npx playwright test

What I learned

The classification problem is harder than detection. Finding text in medical images is straightforward — EasyOCR handles it well. The real challenge is deciding which text is PHI and which is clinically relevant. "D 13.0cm" is a measurement. "Jane Smith" is a patient name. "GE" is a device manufacturer. "MI 0.7" is a mechanical index. The three-layer pipeline exists because no single approach handles all these correctly.

Manual tools don't scale. Tools like 3D Slicer and RadiAnt are excellent for viewing and annotating individual images. But using them for de-identification means opening each image, manually identifying PHI regions, blacking them out, and saving — one at a time. For research datasets with thousands of images, this is the bottleneck that blocks entire projects. Batch automation changes the game.

Passmark is well-suited for AI-generated UIs. Gradio, Streamlit, and similar frameworks produce UIs where the DOM structure is an implementation detail, not a stable API. Natural language test steps abstract over that. One gotcha: be very specific in your descriptions. "Click the De-identify button" failed because the button had an emoji prefix. "Click the large blue button" worked immediately.

The hybrid testing pattern is practical. Pure Passmark tests are elegant, but some UI interactions (file uploads, drag-and-drop) work better with native Playwright. Mixing both in the same test gives you the best of both worlds — AI-driven navigation and assertions, deterministic file operations.

Links

• Aegis GitHub: github.com/lakshmi-mahabaleshwara/aegis

• Live Demo: Hugging Face Space

• Passmark Test Repo: https://github.com/lakshmi-mahabaleshwara/aegis-passmark-tests

• Passmark: passmark.dev

This project was created for the #BreakingAppsHackathon, where I used Passmark, Playwright, Gradio, and Hugging Face Spaces to test Aegis: Healthcare AI application.

Aegis is DICOM PS3.15 and HIPAA Safe Harbor compliant. The Stanford De-identifier NER model is from StanfordAIMI.