0
$\begingroup$

Related to TrackNet, a CNN for tracking tennis balls on TV tennis matches, the Arxiv paper mentions it is scalable, ie. the input can be any number of frames concatenated rather than the three they used. So I tried to concatenate 11 frames and adjusted the input layer dimension:

#changed from 9 to 33 for 11 frames input
    imgs_input = Input(shape=(33,input_height,input_width))

But now when I try to load a weights file that comes with the open source code, I am getting an error:

Traceback (most recent call last):
  File "predict_video.py", line 55, in <module>
    m.load_weights(  save_weights_path  )
  File "/usr/local/lib/python2.7/dist-packages/keras/engine/network.py", line 1166, in load_weights
    f, self.layers, reshape=reshape)
  File "/usr/local/lib/python2.7/dist-packages/keras/engine/saving.py", line 1058, in load_weights_from_hdf5_group
    K.batch_set_value(weight_value_tuples)
  File "/usr/local/lib/python2.7/dist-packages/keras/backend/tensorflow_backend.py", line 2465, in batch_set_value
    assign_op = x.assign(assign_placeholder)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/variables.py", line 1952, in assign
    name=name)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/state_ops.py", line 227, in assign
    validate_shape=validate_shape)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_state_ops.py", line 66, in assign
    use_locking=use_locking, name=name)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper
    op_def=op_def)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 3616, in create_op
    op_def=op_def)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2027, in __init__
    control_input_ops)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1867, in _create_c_op
    raise ValueError(str(e))
  ValueError: Dimension 0 in both shapes must be equal, but are 3 and 64. Shapes are [3,3,33,64] and [64,9,3,3]. for 'Assign' (op: 'Assign') with input shapes: [3,3,33,64], [64,9,3,3].

The actual input for the original CNN is 3 video frames of height 360, width 640 and the code looks like this:

imgs_input = Input(shape=(9,input_height,input_width))

And the model is instantiated like this:

m = modelFN( n_classes , input_height=height, input_width=width   )

where n_classes is a command line argument with default value of 256

For 11 frames, I tried instantiating the 3 frames model, loading the weights and then instantiating the 11 frames model and tried to used old_model.get_weights() specified in this answer:
Stackoverflow answer

So the model and weights loading snippet looks like this:

#load TrackNet model
modelFN = Models.TrackNet.TrackNet
m = modelFN( n_classes , input_height=height, input_width=width   )
m.compile(loss='categorical_crossentropy', optimizer= 'adadelta' , metrics=['accuracy'])
#load and save from same path
m.set_weights(  save_weights_path  )

#load TrackNet 11 frames model and transfer weights
model11 = Models.TrackNet11.TrackNet11
m11 = model11(n_classes, input_height=height, input_width=width)
m11.compile(loss='categorical_crossentropy', optimizer='adadelta', metrics=['accuracy'])
m11.load_weights(m.get_weights)

The full code is available at the link below

TrackNet repo

I tried the Stackoverflow answer and tried to used None for the channel dimension because using 33 gave me an error saying dim2 is different ie. [3,3,33,64] vs [3,3,9,64] but now I am getting:

ValueError: The channel dimension of the inputs should be defined. Found `None`.

So the channel dimension has to be defined.

I am going to try this: datasciencestackexchange answer

But this means that the weights from inputs to first conv2d layer will not be the pretrained ones?

Anyways, I did try it but was unable to get any output, ie. it did not track the tennis ball at all and I am pretty sure there are no other errors in the code but will double check. If anyone has a easy solution that would be appreciated.

My attempt at converting from 3 frames concatenated input to 11 frames can be seen at the following link in files predict_video.py and predict_video11.py. In the Models folder you will see TrackNet.py for 3 frames and TrackNet11.py for 11. There is also a python 3 version that I converted to from the original python 2 version using py2to3 that works and comes with requirementspy3.txt assuming you have the correct version of tensorflow installed for your machine (cpu or gpu with cuda, cudnn).

TrackNet on Gitlab link

Arxiv paper link: Arxiv TrackNet

$\endgroup$
1
$\begingroup$

It's impossible to change the number of channels.

The weights of the model depend on the number of channels. Changing channels is changing weights. Changing weights is having a completely new model.

You can only change the image size (in purely convolutional networks - without Flatten - the image size does not affect the number of weights).

But: Frames are not channels.

Take care with this. Frames are entire images, not channels of images. But it's impossible to help further without knowing the code of the original CNN.

I don't know if the net is purely convolutional, if it uses the frames as samples, if it uses TimeDistributed frames, or if it uses recursive layers.

$\endgroup$
  • $\begingroup$ thanks for dropping in from stackoverflow. That was my conclusion so I just created a new model and did new_model.set_weights(old.get_weights()) from layers[1:] and from layers[2:] but then I had some other error about a disconnected graph or something which I will have to rerun to jog my memory. There are no Flatten` layers and the Arxiv paper linked in the question says it can be scaled up in num frames to concatenate so I think I have to retrain the 11 frame network. Or could I condense the 11 frames concatenated image size down to fit the original 3 frames model? $\endgroup$ – mLstudent33 Jul 21 at 5:15
  • 1
    $\begingroup$ From the graph of the net, the author really mixed frames with channels and it will not be scalable as you think. I suggest you train your own net if possible and make the separation of these two concepts. That net presented will not be able to deal more frames unless you pass your images always in groups of 3. $\endgroup$ – Daniel Möller Jul 22 at 12:20
  • $\begingroup$ fyi, I think it's in fine print somewhere but the net is 'channels first' so 9 is 3 frames * RGB. Also in the concatenation line 107 and 108 here (axis=2): nol.cs.nctu.edu.tw:234/open-source/TrackNet/blob/master/Code/… I printed out shape and got (360, 640, 9) with a single img.shape printing (360,640,3). So I guess they concatenated by the channels dimension, ie. stacked the 3 frames RGB where as you are saying they should have concatenated side-by-side along axis=1 or 0 (can't remember off the top of my head. Is this correct? $\endgroup$ – mLstudent33 Jul 22 at 14:42
  • 1
    $\begingroup$ They should have created an extra dimension for frames, using a TimeDistributed layer, or a recurrent network. These are scalable options. But they simply merged channels and frames into a single dimension making the net stuck to this shape. $\endgroup$ – Daniel Möller Jul 22 at 14:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.