1
$\begingroup$

I want to train a convolutional neural network autoencoder on a csv file which contains values pixel neighborhood position of an original image of 1024x1024. When I try to train it, I have the following error that I don't manage to resolve. ValueError: Input 0 of layer max_pooling2d is incompatible with the layer: expected ndim=4, found ndim=5. Full shape received: (None, 1, 1024, 1024, 16) Do you have any idea what I am doing wrong ?

I managed to load my dataset, extract the x, y, and value columns as NumPy arrays and extract the relevant columns as NumPy arrays. My csv file contains 3 columns.

x = data[0].values
y = data[1].values
values = data[2].values

Then, I create an empty image with the correct dimensions and fill in the image with the pixel values

image = np.empty((1024, 1024))


for i, (xi, yi, value) in enumerate(zip(x, y, values)):
    image[xi.astype(int), yi.astype(int)] = value

To use this array as input to my convolutional autoencoder I reshaped it to a 4D array with dimensions

# Reshape the image array to a 4D tensor
image = image.reshape((1, image.shape[0], image.shape[1], 1))

Finally, I declare the convolutional autoencoder structure, at this stage I have the error incompatible with the layer: expected ndim=4, found ndim=5. Full shape received: (None, 1, 1024, 1024, 16)

import keras
from keras.layers import Input, Conv2D, MaxPooling2D, UpSampling2D
from keras.models import Model

# Define the input layer
input_layer = Input(shape=(1,image.shape[1], image.shape[2], 1))

# Define the encoder layers
x = Conv2D(16, (3, 3), activation='relu', padding='same')(input_layer)
x = MaxPooling2D((2, 2), padding='same')(x)
x = Conv2D(8, (3, 3), activation='relu', padding='same')(x)
x = MaxPooling2D((2, 2), padding='same')(x)
x = Conv2D(8, (3, 3), activation='relu', padding='same')(x)
encoded = MaxPooling2D((2, 2), padding='same')(x)

# Define the decoder layers
x = Conv2D(8, (3, 3), activation='relu', padding='same')(encoded)
x = UpSampling2D((2, 2))(x)
x = Conv2D(8, (3, 3), activation='relu', padding='same')(x)
x = UpSampling2D((2, 2))(x)
x = Conv2D(16, (3, 3), activation='relu')(x)
x = UpSampling2D((2, 2))(x)
decoded = Conv2D(1, (3, 3), activation='sigmoid', padding='same')(x)

# Define the autoencoder model
autoencoder = Model(input_layer, decoded)

# Compile the model
autoencoder.compile(optimizer='adam', loss='binary_crossentropy')

# Reshape the image array to a 4D tensor
image = image.reshape((1, image.shape[0], image.shape[1], 1))

# Train the model
autoencoder.fit(image, image, epochs=50, batch_size=1, shuffle=True)

I also tried also Reshape the image array to a 4D tensor image = image.reshape((1, image.shape[0], image.shape[1], 1)) but I have this error: ValueError: cannot reshape array of size 1048576 into shape (1,1,1024,1)

$\endgroup$
3
  • $\begingroup$ The first dimension is related to the batch size (number of examples in a batch) and is not included in the input tensor's shape. So, each example is a 3D tensor rather than a 4D one. Try image.reshape((image.shape[0], image.shape[1], 1)) and Input(shape=(image.shape[0], image.shape[1], 1)). $\endgroup$
    – learner
    Commented Jan 2, 2023 at 16:20
  • $\begingroup$ @learner, I drop second image.reshape since at that time image.shape == (1, 1024, 1024, 1). I also added padding='same' that I forgot on a Conv2D layer. It seems working now. Just I have a negative loss value during the training, I don't know if that good or bad. $\endgroup$
    – user979974
    Commented Jan 3, 2023 at 8:04
  • $\begingroup$ Depends on the nature of the loss. Sometimes you can drive it to negative infinity, other times you drive it to A constant value like zero, or negative one. $\endgroup$ Commented Nov 28, 2023 at 15:39

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.