0
$\begingroup$

I want to compare my proposed method with traditional machine learning classifiers like Multilayer Perceptron(MLP) and SVM to check the classification accuracy.How do I compare different classifiers?. I want to draw a comparison between these three terms

  • 1).Proposed method (Based on DNN)
  • 2).Shallow neural networks
  • 3).SVM
$\endgroup$
4
  • 3
    $\begingroup$ The usual approach if the key performance measure is a technical measure of classifier performance (e.g. accuracy, loss, AUROC) is to take a suitably challenging dataset, create a fixed holdout test set from it, then train classifiers on the remaining training data using the different approaches that you want to compare. That is a data science 101 issue really though, practically all tutorials and code examples use a train/test split - so could you clarify what your sticking point is? $\endgroup$ Jul 1, 2017 at 20:20
  • $\begingroup$ I did these steps. 1).Setting up dataset 2).Normalize & Split the data into train and test while using k-Fold Cross validation 3).Test the accuracies and loss using Confusion matrix.4).At the end i compare classification accuracies of different classifiers. I was wondering weather this is right/standard $\endgroup$ Jul 3, 2017 at 2:44
  • $\begingroup$ That would be OK if you had already decided on all models and hyper-parameters at the start that you want to compare. If you are using the k-fold cross validation also to search for best generalising model in each class, then you should have a separate hold-out test set - i.e. you would have train/cross-validation and test sets (you can implement this as a training set that you do k-fold cv on, but test is separate). Also, you should compare classification accuracies if that's the metric you are most interested in for the eventual use of the model. Other metrics are possible too. $\endgroup$ Jul 3, 2017 at 7:02
  • $\begingroup$ yes,i separated the test set.while using K fold Crossval as you can see here.(pastebin.com/rrWsTn2w) $\endgroup$ Jul 3, 2017 at 8:41

2 Answers 2

3
$\begingroup$

What about doing cross validation on your training set? Once you have the different train/test splits I would start by printing the accuracy (number of correct predictions / total predictions) and the confussion matrix for each method.

If you are using python sklearn.metrics offers a wide variety of useful functions.

$\endgroup$
1
  • $\begingroup$ yes,exactly i did that.I did these steps. 1).Setting up dataset 2).Normalize & Split the data into train and test while using k-Fold Cross validation 3).Test the accuracies and loss using Confusion matrix.4).At the end i compare classification accuracies of different classifiers. I was wondering weather this is right/standard approach. $\endgroup$ Jul 3, 2017 at 2:41
1
$\begingroup$

There are other metrics you can use to directly compare classifiers instead of accuracy, such as Precision and Recall to compare how your classifiers catch all possible true and false cases individually. This gives you a much better idea of how your classifier performs on different cases.

Or F-Measure(can be used for multiclass too!) and Matthews' Correlation Coefficient (binary classifications) to get a single metric that 'combines' the performance of your classifiers in several areas such as precision and recall to get a single comparable metric.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.