Keras cNN Transfer Model: Reduce Final Model Size?

Question

I'm working with multiple cNNs to be ran on mobile devices. If I create these cNNs from scratch (black n white, 256x256), I'm able to produce a binary classification model of about 10mb, which is a size that would work great on a model device.

However, using transfer learning such as VG16 Imagenet gives my much better classification accuracy. But, the model size is closer 100mb! That is 10x as big as from scratch. I have to take those grayscale images from scratch and convert them to 3 channel, which accounts for some of the size. Also, freezing less layers, like 10 instead of 14 reduces the size, but very slightly.

Any suggestion on a way I can still leverage Keras Transfer learning and produce a much smaller model (like a way there a transfer learning model that is B&W)?

Here is my Transfer baseline model:

def baseline_model_func():
vgg_model = VGG16(weights='imagenet', include_top=False, input_shape=(input_dim, input_dim, 3))

for layer in vgg_model.layers[:10]:
    layer.trainable = False

x = vgg_model.output
x = Flatten()(x) # use global pooling
x = Dense(1024, activation="relu")(x)
x = Dropout(0.5)(x)
x = Dense(1024, activation="relu")(x)
predictions = Dense(1, activation="sigmoid")(x)
model_final = Model(inputs = vgg_model.input, outputs = predictions)
model_final.compile(loss = "binary_crossentropy", optimizer = optimizers.SGD(lr=0.0001, momentum=0.9), 
                    metrics=["accuracy"])
return model_final

`

Here is my cNN from scratch baseline model:

def baseline_model_func():
model = Sequential()
model.add(Conv2D(filters=16, kernel_size=2, padding='same', activation='relu', 
                        input_shape=(input_dim, input_dim, channel_numbers)))
model.add(MaxPooling2D(pool_size=2))
model.add(Conv2D(filters=32, kernel_size=2, padding='same', activation='relu'))
model.add(MaxPooling2D(pool_size=2))
model.add(Conv2D(filters=64, kernel_size=2, padding='same', activation='relu'))
model.add(MaxPooling2D(pool_size=2))
model.add(Conv2D(filters=128, kernel_size=2, padding='same', activation='relu'))
model.add(MaxPooling2D(pool_size=2))
model.add(Conv2D(filters=256, kernel_size=2, padding='same', activation='relu'))
model.add(MaxPooling2D(pool_size=2))
model.add(Conv2D(filters=512, kernel_size=2, padding='same', activation='relu'))
model.add(MaxPooling2D(pool_size=2))
model.add(Dropout(0.3))
model.add(Flatten())
model.add(Dense(250, activation='relu'))
model.add(Dropout(0.4))
model.add(Dense(500, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(1, activation='sigmoid'))
model.compile(loss = "binary_crossentropy", optimizer = "adam", metrics=["accuracy"])
return model

```

Answer 1

There's actually some pretty interesting research on this topic. Key words would be model compression or CNN pruning (doing things like reducing the model size by removing filters with low activations).

Another interesting method uses a technique called teacher-student networks. Basically, this involves training the big network (the teacher) and using the output of the teacher as the labels when you train the small student network. This usually serves to improve the student over training with the ground truth.

For example, if you were training on MNIST, you'd have 10 classes and a softmax output of the teacher network that might look like [0, 0, 0, 0, 0.33, 0, 0, 0, 0, 0.67], in the case of a 9 that looks like a 4, when the original ground truth was [0, 0, 0, 0, 0, 0, 0, 0, 0, 1].

I'm not totally current on this research so give these a google first, there might be some better best practices out there. Hope this at least gets you going in the right direction.