Bias operates per virtual neuron, so there is no value in having multiple bias inputs where there is a single output - that would equivalent to just adding up the different bias weights into a single bias.
In the feature maps that are the output of the first hidden layer, the colours are no longer kept separate*. Effectively each feature map is a "channel" in the next layer, although they are usually visualised separately where the input is visualised with channels combined. Another way of thinking about this is that the separate RGB channels in the original image are 3 "feature maps" in the input.
It doesn't matter how many channels or features are in a previous layer, the output to each feature map in the next layer is a single value in that map. One output value corresponds to a single virtual neuron, needing one bias weight.
In a CNN, as you explain in the question, the same weights (including bias weight) are shared at each point in the output feature map. So each feature map has its own bias weight as well as previous_layer_num_features x kernel_width x kernel_height
connection weights.
So yes, your example resulting in (3 x (5x5) + 1) x 32
weights total for the first layer is correct for a CNN with first hidden layer processing RGB input into 32 separate feature maps.
* You may be getting confused by seeing visualisation of CNN weights which can be separated into the colour channels that they operate on.