3
$\begingroup$

As part of my master's thesis, I have made a prediction of data with approaches of machine learning in a topic where are no papers yet. The topic is a regression problem for which several machine learning approaches have been tested: SVR, RandomForestRegressor, GradientBoosting, DecisionTrees, Artificial Neural Networks, LSTMs, K-NearestNeighbors..

It has been shown that useful results can be achieved with all approaches. Which of the methods is the best depends heavily on the parameter configuration.

It would be too much work to test any parameter configuration for any approach. Besides, I think it would be uninteresting to show dozens of plots about the parameter tuning tests, because they also depend heavily on the respective data sets.

So how do you typically compare different types of machine learning approaches in scientific papers that are heavily dependent on the parameters? Just play a bit with grid search and randomized search until you are satisfied with the parameter configuration?

$\endgroup$
  • $\begingroup$ Good question. The specific accuracies typically depend on how much we are familiar with the method and how much time and passion we invest in parameter tuning. For a scientific paper, it would be optimal to provide details on these points, e.g. which parameters were tuned and how much time was spent. $\endgroup$ – Michael M May 6 '18 at 13:58
3
$\begingroup$

If I understand, you could evaluate your approach based on your efforts in using the right preprocessing, picking the right features, building the appropriate architecture (choosing the model that best suits the problem).

Each step will make the loss decrease until you reach a point, you will have to search over the hyperparameters to squeeze every percent of accuracy, which is heavily a computational problem.

$\endgroup$
1
$\begingroup$

Hello I don't know about scientific paper.

But I know that to compare quality of ML algorithms one against another you need to establish a metric : Make it a square mean error or anything else, a final number that can tell you if your algorithm is working as expected.

On a side note : Every algorithm, if used in their correct purpose, is highly parameters dependent. So I'm not sure what you're referring to with "machine learning approaches that are heavily dependent on the parameters"

$\endgroup$
  • $\begingroup$ Thanks for the answer, but unfortunately it does not help me any further. The quality of the approaches are tested with the mean squared error. The question is more about how to get a good mean squared error comparison without testing all the parameters and still be scientific. $\endgroup$ – MerklT Feb 5 '18 at 10:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.