1
$\begingroup$

I have a large MRI dataset for an image segmentation task that cannot directly fit in memory in Colab, you can access the data with the link I put at the end. They are brain MRI images:

  • 484 training images, each has a shape of (240, 240, 155, 4), these 4 numbers are the height, width, number of layers and sequences respectively.
  • 484 labels, each has a shape of (240, 240, 155)

How are you going to preprocess those images before training? Below are the steps that I tried but it didn't work:

  1. Load and read the image. (I used nibabel)
  2. Convert the images' type from float64 to float32, labels' type to uint8.
  3. Remove the very first and last layers because they don't contain useful information .
  4. Stack/Add each of them into an array with a for loop.

What else do you think I can do do deal with this problem?

Datalink: https://drive.google.com/drive/folders/1HqEgzS8BV2c7xYNrZdEAnrHk7osJJ--2 (task 1 - Brain Tumour)

Please tell me if you need more information.

$\endgroup$

1 Answer 1

0
$\begingroup$

As you cannot read the whole dataset in a single time, you should read and preprocess the images batch-wise while training the model. You can write your preprocessing pipeline in a data loader and iterate through this data loader while training the model. During each iteration, your data loader will fetch a single batch of data. And you can write your custom pipeline in the data loader to get this single batch. Treat this as a generator and iterate through this generator in the training loop, and you will get batches on the run time (i.e. a single batch is read in the memory at a time).

The following links give a good example of creating a custom data loader in Pytorch -

https://pytorch.org/tutorials/beginner/basics/data_tutorial.html

https://stanford.edu/~shervine/blog/pytorch-how-to-generate-data-parallel

You have similar functionality in TensorFlow using input data pipelines -

https://www.tensorflow.org/guide/data

$\endgroup$
2
  • $\begingroup$ Thank you, I'll have a look at it. This is new for me. $\endgroup$ Nov 27, 2021 at 11:39
  • $\begingroup$ Your welcome! Please consider accepting the answer if it answers your question. $\endgroup$ Nov 28, 2021 at 3:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.