Even if a model performs well in testing, clinicians are unlikely to trust it if it behaves like a total black box.
With Kifua AI, we’re not just interested in what the model predicts, but also where it is “looking” on the image.
That’s where saliency maps and Grad-CAM come in.
The Risk of Black-Box AI in Medicine
Imagine an AI system that says:
“Abnormal CXR. High risk of consolidation.”
But offers no explanation. The opacity is subtle. The clinician doesn’t see anything obvious. Should they:
- Trust the AI and treat for pneumonia?
- Ignore it and follow their own judgment?
- Order more tests?
Over time, this lack of transparency erodes trust. Clinicians either over-rely (“the AI said so”) or under-use (“this thing is random”).
We need a middle ground where AI:
- Makes suggestions,
- Shows its work,
- And invites clinicians to question and verify.
What Is Grad-CAM in Simple Terms?
Grad-CAM (Gradient-weighted Class Activation Mapping) is a technique that helps us visualise which parts of an image contributed most to a model’s prediction.
In practical terms:
- We pass the chest X-ray through the model.
- We compute gradients that tell us how much each region influenced the output.
- We overlay a heatmap on the original image, highlighting “hot” areas the model considered important.
For Kifua AI, that means:
- If it predicts “consolidation,” we expect the heatmap to light up in the relevant lung zone.
- If it predicts “cardiomegaly,” we expect the cardiac silhouette to be highlighted.
How Overlays Help Clinicians
Saliency maps are not a perfect explanation. But they offer several benefits:
- Visual alignment with human reasoning
- Radiologists and clinicians can see whether the AI is focusing on plausible regions.
- If the model highlights a completely irrelevant area (e.g. corners, text labels), that’s a red flag.
- Educational value
- Junior clinicians can learn by comparing their own focus areas with the AI’s.
- Over time, this can sharpen pattern recognition skills.
- Triggering scrutiny, not blind trust
- If a heatmap highlights an unusual area, it prompts a closer look and potentially further imaging or tests.
Example: A Suspicious Right Lower Zone
Take a (hypothetical, anonymised) example:
- A middle-aged patient presents with fever and productive cough.
- The X-ray looks almost normal at first glance.
- Kifua AI flags “Abnormal – suspicious consolidation” and highlights the right lower lung zone.
The clinician might:
- Re-examine the film more carefully.
- Correlate with clinical findings (e.g. focal crackles).
- Decide to treat for pneumonia and plan follow-up imaging.
The AI hasn’t replaced clinical judgment—but it has changed where the clinician looks and how carefully.
Validation with Radiologist Panels
Of course, heatmaps can also be misleading or over-interpreted. That’s why part of Kifua AI’s development includes:
- Comparing AI heatmaps with radiologist-marked regions.
- Checking how often the model’s “focus” overlaps with expert-defined areas of interest.
- Iteratively improving training if the model repeatedly highlights irrelevant regions.
Our ultimate aim is to reach a place where clinicians say:
“I don’t always agree with Kifua AI, but I understand why it’s raising a flag.”
That level of informed skepticism is much healthier—and safer—than blind faith.