The test set and cross validation set have different purposes. If you drop either one, you lose its benefits:
You cannot use the cross validation set to measure performance of your model accurately, because you will deliberately tune your results to get the best possible metric, over maybe hundreds of variations of your parameters. The cross validation result is therefore likely to be too optimistic.
For the same reason, you cannot drop the cross validation set and use the test set for selecting hyper parameters, because then you are pretty much guaranteed to be overestimating how good your model is. In the ideal world you use the test set just once, or use it in a "neutral" fashion to compare different experiments.
If you cross validate, find the best model, then add in the test data to train, it is possible (and in some situations perhaps quite likely) your model will be improved. However, you have no way to be sure whether that has actually happened, and even if it has, you do not have any unbiased estimate of what the new performance is.
From witnessing many Kaggle competitions, my experience is that tuning to the test set by over-using it is a real thing, and it impacts those competitions in a large way. There is often a group of competitors who have climbed the public leaderboard and selected their best model in test (the public leaderboard is effectively a test set), whilst not being quite so thorough on their cross validation . . . these competitors drop down the leaderboard when a new test set is introduced at the end.
One approach that is reasonable is to re-use (train + cv) data to re-train using the hyper-params you have found, before testing. That way you do get to train on more data, and you still get an independent measure of performance at the end.
If you want to get more out of cross validation, the usual approach is k-fold cross validation. A common trick in Kaggle competitions is to use k-fold cross validation, and instead of re-combining the data into a larger (train + cv) training set, to ensemble or stack the cv results into a meta-model.
Finally, always check that your splits for validation and test are robust against possible correlation within your data set.