0
$\begingroup$

I have built a CNN to do image classification for images representing different weather conditions. I have 4 classes of images : Haze, Rainy, Snowy, Sunny.

I have built my CNN and evaluated the performances. N ow I have been given a blind test set, so images without a label, and I have to make a submission. So I have to buld a .csv file which contains should contain one line for each predicted class of images, so it should have the structure ,. Thus each line should be a string which identifies the image and its prediction.

Now the problem is that I don't understand how to do this. I am really confused because I have never done something similar.

My code is the following:

trainingset = '/content/drive/My Drive/Colab Notebooks/Train'
testset = '/content/drive/My Drive/Colab Notebooks/Test_HWI'


batch_size = 31
train_datagen = ImageDataGenerator(
    featurewise_center=True,
    featurewise_std_normalization=True,
    rescale = 1. / 255,\
    zoom_range=0.1,\
    rotation_range=10,\
    width_shift_range=0.1,\
    height_shift_range=0.1,\
    horizontal_flip=True,\
    vertical_flip=False)


train_generator = train_datagen.flow_from_directory(
    directory=trainingset,
    target_size=(256, 256),
    color_mode="rgb",
    batch_size=batch_size,
    class_mode="categorical",
    shuffle=True
)


test_datagen = ImageDataGenerator(
    featurewise_center=True,
    featurewise_std_normalization=True,
    rescale = 1. / 255
   )



test_generator = test_datagen.flow_from_directory(
    directory=testset,
    target_size=(256, 256),
    color_mode="rgb",
    batch_size=batch_size,
    class_mode="categorical",
    shuffle=False
)

num_samples = train_generator.n
num_classes = train_generator.num_classes
input_shape = train_generator.image_shape

classnames = [k for k,v in train_generator.class_indices.items()]

then I build the network:

def Network(input_shape, num_classes, regl2 = 0.0001, lr=0.0001):

model = Sequential()

# C1 Convolutional Layer 
model.add(Conv2D(filters=32, input_shape=input_shape, kernel_size=(3,3),\
                 strides=(1,1), padding='valid'))
model.add(Activation('relu'))
# Pooling
model.add(MaxPooling2D(pool_size=(2,2), strides=(2,2), padding='valid'))
# Batch Normalisation before passing it to the next layer
model.add(BatchNormalization())

# C2 Convolutional Layer
model.add(Conv2D(filters=64, kernel_size=(3,3), strides=(1,1), padding='valid'))
model.add(Activation('relu'))
# Batch Normalisation
model.add(BatchNormalization())

# C3 Convolutional Layer
model.add(Conv2D(filters=128, kernel_size=(3,3), strides=(1,1), padding='valid'))
model.add(Activation('relu'))
# Batch Normalisation
model.add(BatchNormalization())

# C4 Convolutional Layer
model.add(Conv2D(filters=256, kernel_size=(3,3), strides=(1,1), padding='valid'))
model.add(Activation('relu'))
#Pooling
model.add(MaxPooling2D(pool_size=(2,2), strides=(2,2), padding='valid'))
# Batch Normalisation
model.add(BatchNormalization())

# C5 Convolutional Layer
model.add(Conv2D(filters=400, kernel_size=(3,3), strides=(1,1), padding='valid'))
model.add(Activation('relu'))
# Pooling
model.add(MaxPooling2D(pool_size=(2,2), strides=(2,2), padding='valid'))
# Batch Normalisation
model.add(BatchNormalization())

 # C6 Convolutional Layer
model.add(Conv2D(filters=512, kernel_size=(3,3), strides=(1,1), padding='valid'))
model.add(Conv2D(filters=512, kernel_size=(3,3), strides=(1,1), padding='valid'))
model.add(Activation('relu'))
# Pooling
model.add(MaxPooling2D(pool_size=(2,2), strides=(2,2), padding='valid'))
# Batch Normalisation
model.add(BatchNormalization())

 # C7 Convolutional Layer
model.add(Conv2D(filters=800, kernel_size=(3,3), strides=(1,1), padding='valid'))
model.add(Conv2D(filters=800, kernel_size=(3,3), strides=(1,1), padding='valid'))
model.add(Activation('relu'))
# Pooling
model.add(MaxPooling2D(pool_size=(2,2), strides=(2,2), padding='valid'))
# Batch Normalisation
model.add(BatchNormalization())

# C8 Convolutional Layer
model.add(Conv2D(filters=1000, kernel_size=(3,3), strides=(1,1), padding='valid'))
model.add(Activation('relu'))
# Pooling
model.add(MaxPooling2D(pool_size=(2,2), strides=(2,2), padding='valid'))
# Batch Normalisation
model.add(BatchNormalization())

# Flatten
model.add(Flatten())

flatten_shape = (input_shape[0]*input_shape[1]*input_shape[2],)

# D1 Dense Layer
model.add(Dense(4096, input_shape=flatten_shape, kernel_regularizer=regularizers.l2(regl2)))
model.add(Activation('relu'))
# Dropout
model.add(Dropout(0.4))
# Batch Normalisation
model.add(BatchNormalization())

# D2 Dense Layer
model.add(Dense(4096, kernel_regularizer=regularizers.l2(regl2)))
model.add(Activation('relu'))
# Dropout
model.add(Dropout(0.4))
# Batch Normalisation
model.add(BatchNormalization())

# D3 Dense Layer
model.add(Dense(1000,kernel_regularizer=regularizers.l2(regl2)))
model.add(Activation('relu'))
# Dropout
model.add(Dropout(0.4))
# Batch Normalisation
model.add(BatchNormalization())

# Output Layer
model.add(Dense(num_classes))
model.add(Activation('softmax'))

# Compile

adam = optimizers.Adam(lr=lr)
model.compile(loss='categorical_crossentropy', optimizer=adam, metrics=['accuracy'])

return model

#create the model
model = Network(input_shape,num_classes)
model.summary()

I train the network:

steps_per_epoch=train_generator.n//train_generator.batch_size
val_steps=test_generator.n//test_generator.batch_size+1

try:
    history = model.fit_generator(train_generator, epochs=100, verbose=1,\
                    steps_per_epoch=steps_per_epoch,\
                    validation_data=test_generator,\
                    validation_steps=val_steps)
except KeyboardInterrupt:
    pass

now, I have the images without labels in the google drive, so I define the path to them:

blind_testSet = '/content/drive/My Drive/Colab Notebooks/blind_testset'

but now I don't know what shoul I do. I really don't know how to define the .csv file I mentioned above.

Can someone please help me? Thanks in advance.

[EDIT] Ok I am trying to make the predictions on the blind test set, but it is taking really a long time. What I have done is the following:

blind_testSet = '/content/drive/My Drive/Colab 
Notebooks/submission/blind_testset'

test_datagen_blind = ImageDataGenerator(
    featurewise_center=True,
    featurewise_std_normalization=True,
    rescale = 1. / 255
   )


test_generator_blind = test_datagen.flow_from_directory(
    directory=blind_testSet,
    target_size=(256, 256),
    color_mode="rgb",
    batch_size=batch_size,
    class_mode="categorical",
    shuffle=False
)

preds = model.predict_generator(test_generator_blind,verbose=1,steps=val_steps)

the images I have inside this blind test set are 1500, but is it normal that it takes so long? Thanks.

[EDIT 2] To try to make the submission I am trying to use a code similar to this:

def make_submission(model, filename="submission.csv"):
df = pd.read_csv("../input/test.csv")
X = df.values / 255
X = X.reshape(X.shape[0], 28, 28, 1)
preds = model.predict_classes(X)
subm = pd.DataFrame(data=list(zip(range(1, len(preds) + 1), preds)), columns=["ImageId", "Label"])
subm.to_csv(filename, index=False)

return subm

but it seems to not work in my case. I have also tried to keep only the last 2 lines and use them, so :

 subm = pd.DataFrame(data=list(zip(range(1, len(preds) + 1), preds)), columns=["ImageId", "Label"])
subm.to_csv(filename, index=False)

can someone help me creating this csv file? Thanks.

$\endgroup$
  • $\begingroup$ The edit should probably be a new question instead. $\endgroup$ – Ben Reiniger Dec 13 '19 at 22:27
1
$\begingroup$

You have to make predictions first. Than order these predictions to the ids in the blind_testSet. So something like this:

test_set=pd.read_csv(blind_testSet) test_set["predicted_labels"]=model.predict(quntified pictures from test set)

EDIT: on the question why is it taking so long to train: you have deep convolutional layers. Backpropagating is very expensive process. A lot can be said on this topic how to speed up the computations, but lets say that you should be looking at utilizing GPU power

$\endgroup$
  • $\begingroup$ sorry again, but I am not yet able to make a submission. I don't understand because I am using a generator to handle the images from a directory, and I can't find a way to build my desired .csv file. If you can, can you show me a way to do it? Thanks again. $\endgroup$ – J.D. Dec 13 '19 at 22:23
  • $\begingroup$ Predicting doesn't use backpropagation, and should be much much faster than training. But that part should probably be a new question anyway... $\endgroup$ – Ben Reiniger Dec 13 '19 at 22:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.