I use T4 GPU on Google Colab to train a model for multi regression. I use to training 30k RGB images 256x256, 5k to validate, 7k to evaluate. The model has 11 outputs [0-1] range (the sum of outputs [y1, y2 ... y11] is always equal 1, eg. 0 + 0 + 0 + 0.5 + 0.3 + 0.2 .... = 1). After the datasets preparing and the model compiling GPU VRAM utilization = 2.4/15 GB. During fitting VRAM utilization = 14.2/15 GB. It is near to max.

  • Could be there a problem with training performance? Why?
  • Are there any big mistakes in my architecture or hiperparams?
  • Is there a better way to load this data to training?

I'm a CNN newbie - any advice welcome.

Images are loaded by this piece of code from my Google Drive

def preproc_image_and_label(x, y):
  img = tf.io.read_file(x)
  img = tf.io.decode_jpeg(img, channels=3)
  img.set_shape((256, 256, 3))
  img = tf.cast(img, dtype=tf.float32)
  img = img / 255.
  return img, y

tf.data.Dataset.from_tensor_slices((tf.constant(glob("drive/MyDrive/path/to/img/train/")), tf.constant(y_train.values))).map(preproc_image_and_label).batch(32).prefetch(1)

The model is compiled with:

layers = [
    Conv2D(filters=128, kernel_size=5, activation="relu", input_shape=(256, 256, 3), padding="same"),
    Conv2D(filters=256, kernel_size=4, activation="relu", padding="same"),
    Conv2D(filters=256, kernel_size=4, activation="relu", padding="same"),
    Conv2D(filters=512, kernel_size=3, activation="relu", padding="same"),
    Conv2D(filters=512, kernel_size=3, activation="relu", padding="same"),
    Dense(256, activation="relu"),
    Dense(128, activation="relu"),
    Dense(11, activation="softmax"),



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Browse other questions tagged or ask your own question.