I'd like to implement this symmetry in the network, and by my understanding of CNNs that means i'll generally need all the filters to be horizontally symmetric, and then the network would both be more robust and would take almost 40% less time to train assuming filter size = 5.
If two mirrored images are in the same class - e.g. they both show a dog or a cat - that is not the same as having all the components in the image (lines, textures, shapes) responding well to symmetric filters. In general this is not the case. Even for symmetric looking shapes such as faces, it is only true at a certain scale and specific pose.
what disadvantages does this architecture have that I can't find a single mention of such idea?
It will work poorly, because the lines, textures and shapes being detected as components of the image are rarely horizontally symmetrical.
CNNs cannot easily take advantage of mirroring or rotational invariance - where the class or detection does not vary under mirror or rotation transformations. One thing you can do to improve generalisation for transformations is data augmentation, i.e. performing the non-class-changing transformations of your training images, perhaps randomly on demand during the training process. This does work, but has the opposite impact that you were hoping for in terms of efficiency.
The recently-published CapsNet by Geoffrey Hinton's group may be able to take advantage of more types of transformation invariance. However, this is still in early stages of research, and not clear whether it offers a practical advantage in your case. There are implementations in various frameworks such as Keras, that you could try if you are interested.
