Computer Vision: Handling dataset(3D data or scan) with different timesteps

Question

I'm planning on training a CNN on CT scans for classification. The problem is CT scans are taken slice by slice, and in a typical scan, there could be more than 200 slices. The number of slices in a scan isn't uniform and depend on the scanning machine and age of the person(for whom the scan is taken).

1)How should I make the number of slices uniform for feeding to a deep learning network?

This sorta problem is handled in NLP by padding a chosen vector( or something similar) to sentences which have lengths less than the predefined length and truncating sentences which have lengths greater than the preset length.

2)Can a similar approach be used to make slices(timesteps) uniform or is there a better way?

Answer 1

I would use an LSTM-RNN to encode sequences of arbitrary length (length meaning the number of slices) into fixed-size output. It should be very easy to implement it in Keras :)

Then feed the sequences in your CNN.

Check sequence-to-sequence encoder-decoder scheme, like this one.

The same concept is applied with state-of-the-art algorithms that require fixed size inputs, like this one.

Answer 2

First, while you could treat slices of a CT scans as a sequence (with some ordering) and one could even argue that this mimics the acquisition process (CT scan slices are acquired in a sequence, even though not necessarily in a plane you are thinking), this is not the common solution. Typically one uses one of two options:

• 3D model (e.g. 3D convolutions);
• 2D model with channels (different slices become channels, typically not all at once but some fixed number). In this case, you use regular 2D convolutions and then aggregate predictions. Sometimes the number of channels = 1.

Regarding your questions,

1. If you solve a classification task you could use Global (average/max) Pooling or Adaptive (average/max) Pooling. This ensures the same size of feature maps before passing them to the Dense layer. Alternatively, you could train on patches (of the same size), assign a label to every patch, and combine predictions from the patches of the same image. The results will depend on how you sample patches (e.g. in a grid or overlapping), and how you combine predictions (an average is the simplest way).

2. If you use a 3D model, and in particular with CT images, it is typically a good idea to resample your image into regular voxel spacing (e.g. 1 x 1 x 1 mm^3), so your 3D filters have the same receptive field (in millimeters). Otherwise, depending on image spacing (e.g., 1x1x1 versus 3x3x3) same convolutional filter will treat "1 mm^3" of volume and "27 mm^3" of volume in the same way. To resample the image you need to access the voxel's size information, typically from the associated DICOM file.