Your main question seems to be what about different algorithms?, and the answer to that is it makes no difference.
While hard to argue with the "test set" answer logic it is also easy to find a counter-example. If your training set has 100K examples, and you used, say 5-fold CV, and then you also have a test set with only 200 examples... the test set is 100x smaller and likely to be much less reliable.
Normally you would use cross-validation, or a test set. And if your training data is large, I would generally recommend just using a test set, because you probably have enough data and because CV takes longer to train. When training data is scarce, cross-validation makes more sense: training time probably matters less, you want to train the final model on everything you have, and also you will value the N different opinions that N-fold CV gives you, because your sample size is small.
If you have used both cross-validation and a test set for multiple models (whether same algorithm or different algorithms), then you should expect them both to agree on your choice of best model. If model A does better than model B on cross-validation, but model B does better on the test data, then you should go looking for a problem.
If that does happen, I'd first be suspicious that the test data is not representative. E.g. to exaggerate, your training data covers the period 1950 to 1970, and your test data is for the year 2020. At the other extreme, I'd check that the test data is not actually in your training data (implying that model B does a better job of memorizing).