Im trying to build an image classification model with multimodality, it takes SAR and optical images, both types of images have FITS format.

The optical images have shape (None, 512, 512, 3), while the SAR images have shape (None, 512, 512, 1), I have tried to reshape the channel dimension of the SAR images and change them into grayscale, but nothing has worked.

This is how I initialize the mode:

# Load pre-trained VGG16 model without top (fully connected) layers
base_model = VGG16(weights='imagenet', include_top=False, input_shape=(height, width, channels))


# Fine-Tune
for layer in base_model.layers[-1:]:
    layer.trainable = True

# Input layers for each modality
optical_input = Input(shape=(height, width, channels), name='optical_input')
sar_input = Input(shape=(height, width, channels), name='sar_input')

# Optical branch with pre-trained VGG16
optical_branch = base_model(optical_input)
optical_branch = spp_layer(optical_branch, optical_branch.shape[1:3], levels)
optical_branch = Flatten()(optical_branch)  # Flatten the output of SPP

# SAR branch
sar_branch = Conv2D(32, (3, 3), activation='relu')(sar_input)
sar_branch = MaxPooling2D((2, 2))(sar_branch)
sar_branch = Conv2D(64, (3, 3), activation='relu')(sar_branch)
sar_branch = MaxPooling2D((2, 2))(sar_branch)
sar_branch = spp_layer(sar_branch, sar_branch.shape[1:3], levels)
sar_branch = Flatten()(sar_branch)  # Flatten the output of SPP

# Feature fusion
fused_features = Concatenate()([optical_branch, sar_branch])

# Add fully connected layers
x = BatchNormalization(name='batch_normalization_top')(fused_features)
x = Dense(1024, activation='relu')(x)
x = Dropout(0.5)(x)  # Dropout layer
x = BatchNormalization(name='batch_normalization_bot')(x)
x = Dense(512, activation='relu',)(x)
x = Dropout(0.5)(x)  # Dropout layer
output = Dense(num_classes, activation='softmax')(x)

# Define the model
model = Model(inputs=[optical_input, sar_input], outputs=output)

# Compile the model
model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])

The when I try to train the model:

epochs = 120  # You can increase this for better performance
checkpoint = ModelCheckpoint("/content/drive/MyDrive/SPP_SAR_MODELS/SPP_SAR_1.keras", monitor="val_accuracy", save_best_only=True, mode="max")
early_stopping = EarlyStopping(monitor='val_accuracy', patience=15, restore_best_weights=True)

history = model.fit(
    callbacks=[checkpoint, early_stopping]

This error pops up:

Epoch 1/120
ValueError                                Traceback (most recent call last)
<ipython-input-163-07edd0bcee58> in <cell line: 6>()
----> 6 history = model.fit(
      7     train_dataset,
      8     epochs=epochs,

1 frames
/usr/local/lib/python3.10/dist-packages/keras/src/models/functional.py in _adjust_input_rank(self, flat_inputs)
    286                     adjusted.append(ops.expand_dims(x, axis=-1))
    287                     continue
--> 288             raise ValueError(
    289                 f"Invalid input shape for input {x}. Expected shape "
    290                 f"{ref_shape}, but input has incompatible shape {x.shape}"

ValueError: Exception encountered when calling Functional.call().

Invalid input shape for input Tensor("functional_15_1/Cast:0", shape=(None,), dtype=float32). Expected shape (None, 512, 512, 3), but input has incompatible shape (None,)

Arguments received by Functional.call():
  • inputs=('tf.Tensor(shape=(None, 512, 512, None), dtype=float32)', 'tf.Tensor(shape=(None,), dtype=int32)')
  • training=True
  • mask=('None', 'None')

I have tried to change the channels with this type of approach:

def preprocess_sar(image, label):
    # Check if the image is single-channel (grayscale) and convert to RGB if needed
    if image.shape[-1] == 1:
        image = tf.image.grayscale_to_rgb(image)
    return image, label

But I havent found a way to make it work.



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.