It's hard to put a general rule with this kind of methods, they depend heavily on the data at hand. You should know pros and cons of each, try all methods and see which performs best on the validation set. Remeber that:
- over-sampling the less frequent class can bring a lot of repetition in the data, because you just replicate the observations
- under-sampling can be bad because you throw away information in the most frequent class, and so you can lose performance
Hybrid methods tend to outperform both, see SMOTE or ROSE.
For example, SMOTE generates new information from the minority class (synthetic observation which are derived from the true samples), and also under-samples the majority class (you choose the final ratio of the two).
Again you need to test on the data the best solution.