I have been attempting to implement a Convolutional Neural Network in python and have run into a bit of a roadblock. When backpropogating the error in a convolutional layer let us say that we receive the error from the next layer as a matrix dL/dH and that we have the input for this convolutional layer, X, and the filters used for this layer's convolution, W. Taking for example the layer in the illustration in this question where we have dL/dH of dimensions (5,96,96), X of dimensions (3,100,100), and W with 5 filters of dimensions (3,5,5): I understand how dL/dH is "tiled" so that the depth of dL/dH matches that of X - allowing one to convolve dL/dH over X and calculate the error for each of the filters in W (dL/dW). What I do not understand is how to then go about calculating dL/dX.
If we pad and dilate dL/dH as per this source then the convolution of the flipped filters over the padded, dilated error produce an output with the same width and height as X, however, in this case, the filters each have a depth of 3 and dL/dH has a depth of 5 - which, as far as I understand, means that we can not convolve W over dL/dH.
Is there a way to pad or otherwise manipulate the filters or dL/dH so that the depth of the resulting convolution is the same as the depth of X?
Thanks for reading, please let me know if I need to make the question clearer or if any other details are needed.