I have a dataset of 3D (medical) images / volumes, think CT/MRI recordings. In other words, each of my samples consists of a 3D array of grayscale values, typically in the range of e.g. 1000x1000x500. I am clustering these data (in a lower dimensional space obtained by some dimensionality reduction scheme, but that's irrelevant for my question), yielding a handful of clusters, each consisting of several dozen to hundreds of such 3D volume recordings. Are there any good techniques that I could use to visualize / understand / compare these clusters in the original 3D data space?


1 Answer 1


For each cluster, pick out a medoid, a representative element.

compare clusters in the original 3D data space?

Suppose you had facial images belonging to K clusters. How would you visualize?

We start by normalizing every face so their pupils are in the same location and chin and forehead fit in a uniform bounding box. You will want to find similar key points for normalizing organs. Consider running multiple analyses, each focused on a specific organ. Perhaps a given cluster suggests "the lung is weird" or "the liver is weird", and that's the criterion that explains why a CT scan was sent to that cluster.

Now when considering a particular image, compare it to its cluster's medoid. The simplest visualization would be per-pixel or per-voxel greyscale deltas. A more sophisticated visualization would borrow from optical flow, where we annotate the image with what it would take to move a clump of pixels from here to there in the source vs destination images. One image would be the relevant medoid. You may find the earth mover's metric to be of some use.

  • $\begingroup$ This is very helpful, thank you! The optical flow thing seems closely related to visual counterfactual examples, i.e., "What would this image look like if it belonged to that other cluster instead?" // It seems to me that the whole thing becomes particularly challenging for non-localized characteristics, such as "there is an annotation / tumor / whatever somewhere (but it can be in different locations per image)" - that would not really show up in a registered medoid image, right? $\endgroup$
    – Eike P.
    Apr 19 at 21:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.