Batch Normalization when CNN with only 2 ConvLayer?

I wonder if it is a problem to use BatchNormalization when there are only 2 convolutional layers in a CNN. Can this have adverse effects on classification performance? Now I don't mean the training time, but really the accuracy? Is my network overloaded with unneccessary layers?

model=Sequential()