Keras reuse trained weights on CNN with different number of channels

Question

Related to TrackNet, a CNN for tracking tennis balls on TV tennis matches, the Arxiv paper mentions it is scalable, ie. the input can be any number of frames concatenated rather than the three they used. So I tried to concatenate 11 frames and adjusted the input layer dimension:

#changed from 9 to 33 for 11 frames input
    imgs_input = Input(shape=(33,input_height,input_width))

But now when I try to load a weights file that comes with the open source code, I am getting an error:

Traceback (most recent call last):
  File "predict_video.py", line 55, in <module>
    m.load_weights(  save_weights_path  )
  File "/usr/local/lib/python2.7/dist-packages/keras/engine/network.py", line 1166, in load_weights
    f, self.layers, reshape=reshape)
  File "/usr/local/lib/python2.7/dist-packages/keras/engine/saving.py", line 1058, in load_weights_from_hdf5_group
    K.batch_set_value(weight_value_tuples)
  File "/usr/local/lib/python2.7/dist-packages/keras/backend/tensorflow_backend.py", line 2465, in batch_set_value
    assign_op = x.assign(assign_placeholder)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/variables.py", line 1952, in assign
    name=name)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/state_ops.py", line 227, in assign
    validate_shape=validate_shape)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_state_ops.py", line 66, in assign
    use_locking=use_locking, name=name)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper
    op_def=op_def)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 3616, in create_op
    op_def=op_def)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2027, in __init__
    control_input_ops)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1867, in _create_c_op
    raise ValueError(str(e))
  ValueError: Dimension 0 in both shapes must be equal, but are 3 and 64. Shapes are [3,3,33,64] and [64,9,3,3]. for 'Assign' (op: 'Assign') with input shapes: [3,3,33,64], [64,9,3,3].

The actual input for the original CNN is 3 video frames of height 360, width 640 and the code looks like this:

imgs_input = Input(shape=(9,input_height,input_width))

And the model is instantiated like this:

m = modelFN( n_classes , input_height=height, input_width=width   )

where n_classes is a command line argument with default value of 256

For 11 frames, I tried instantiating the 3 frames model, loading the weights and then instantiating the 11 frames model and tried to used old_model.get_weights() specified in this answer:
Stackoverflow answer

So the model and weights loading snippet looks like this:

#load TrackNet model
modelFN = Models.TrackNet.TrackNet
m = modelFN( n_classes , input_height=height, input_width=width   )
m.compile(loss='categorical_crossentropy', optimizer= 'adadelta' , metrics=['accuracy'])
#load and save from same path
m.set_weights(  save_weights_path  )

#load TrackNet 11 frames model and transfer weights
model11 = Models.TrackNet11.TrackNet11
m11 = model11(n_classes, input_height=height, input_width=width)
m11.compile(loss='categorical_crossentropy', optimizer='adadelta', metrics=['accuracy'])
m11.load_weights(m.get_weights)

The full code is available at the link below

TrackNet repo

I tried the Stackoverflow answer and tried to used None for the channel dimension because using 33 gave me an error saying dim2 is different ie. [3,3,33,64] vs [3,3,9,64] but now I am getting:

ValueError: The channel dimension of the inputs should be defined. Found `None`.

So the channel dimension has to be defined.

I am going to try this: datasciencestackexchange answer

But this means that the weights from inputs to first conv2d layer will not be the pretrained ones?

Anyways, I did try it but was unable to get any output, ie. it did not track the tennis ball at all and I am pretty sure there are no other errors in the code but will double check. If anyone has a easy solution that would be appreciated.

My attempt at converting from 3 frames concatenated input to 11 frames can be seen at the following link in files predict_video.py and predict_video11.py. In the Models folder you will see TrackNet.py for 3 frames and TrackNet11.py for 11. There is also a python 3 version that I converted to from the original python 2 version using py2to3 that works and comes with requirementspy3.txt assuming you have the correct version of tensorflow installed for your machine (cpu or gpu with cuda, cudnn).

TrackNet on Gitlab link

Arxiv paper link: Arxiv TrackNet

Answer 1

It's impossible to change the number of channels.

The weights of the model depend on the number of channels. Changing channels is changing weights. Changing weights is having a completely new model.

You can only change the image size (in purely convolutional networks - without Flatten - the image size does not affect the number of weights).

But: Frames are not channels.

Take care with this. Frames are entire images, not channels of images. But it's impossible to help further without knowing the code of the original CNN.

I don't know if the net is purely convolutional, if it uses the frames as samples, if it uses TimeDistributed frames, or if it uses recursive layers.