This is a major confusion for me - I've always thought filters (those small sliding windows size 3x3 or 5x5) are strictly 2D, this among other is specified in Caffe API:
kernel_size / kernel_h / kernel_w. The filter dimensions, given by kernel_size for square filters or kernel_h and kernel_w for rectangular filters
After searching it turned out that filters can be 3D?? How does it work then? I found a bit of information on CS 231n course, but it's not explained in detail anywhere.