What are the best ideas or approaches to trade off between bias and variance in Machine Learning models.
You want to decide this based on how well your model performs and generalizes. If your model is underfitting, you want to increase your model's complexity, increasing variance and decreasing bias. If your model is overfitting, you want to regularize the model and/or feed it more training data, decreasing variance and increasing bias.
In my opinion you should do the following:
1) Try to minimize the bias as much as possible. This can usually be achieved by choosing a more complex model. This step is done to ensure that your model has enough capacity to solve the problem. This will, however, cause your model to overfit.
2) Regularize the above model to reduce its variance. This can be done through norm penalties (L1, L2, etc.), early stopping, or other model specific techniques (dropout for neural networks, pruning for decision trees, etc.).