I think in the original paper they suggest using $\log_2(N +1$), but either way the idea is the following:
The number of randomly selected features can influence the generalization error in two ways: selecting many features increases the strength of the individual trees whereas reducing the number of features leads to a lower correlation among the trees increasing the strength of the forest as a whole.
What's interesting is that the authors of Random Forests (pdf) find an empirical difference between classification and regression:
An interesting difference between regression and classification is that the
correlation increases quite slowly as the number of features used increases.
Therefore, for regression often $N/3$ is recommended, which gives larger values than $\sqrt N$.
In general, there is no clear justification for $\sqrt N$ or $\log N$ for classification problems other than that it has shown that lower correlation among trees can decrease generalization error enough to more than offset the decrease in strength of individual trees. In particular, the authors note that the range where this trade-off can decrease the generalization error is quite large:
range is usually large. In this range, as the number of features goes up, the
correlation increases, but PE*(tree) compensates by decreasing.
(PE* being the generalization error)
As they say in Elements of Statistical Learning:
In practice the best values for these parameters will depend on the problem,
and they should be treated as tuning parameters.
One thing your problem can depend on is the number of categorical variables. If you have many categorical variables that are encoded as dummy-variables it usually makes sense to increase the parameter. Again, from the Random Forests paper:
When many of the variables are categorical, using a
low [number of features] results in low correlation, but also low strength. [The number of features] must be
increased to about two-three times $int(log_2M+1)$ to get enough strength to
provide good test set accuracy.