Check this video. Color channels in a CNN are only involved in the first layer, where the original image is given as the input. Let us assume we are taking a 3x3 filter. This actually means we have 3 3x3 filters for each color channels. Each filter is applied to its corresponding color channel: Red filter is applied to Red channel, Blue filter to Blue channel etc.., and at the end you simply add the result for each filter, for each pixel.
Assuming you use a padding that preserves the original size of the image, the output for these filters will be the same size as the original image, where each pixel is calculated by adding up the result of the filter. Of course we never use a single filter. The number of filters you use will determine the dimension of the output.
For example, check this model summary from TensorFlow documentation, and compare it to the model above. The last parameter in parentheses represents the number of filters, while the last parameter in the first row is the number color channels in the given image.