Feature Selection for K Nearest Neighbour and Decision Trees

Question

I have 2 digits numbers and 9 features.

I must pick 2 features, so decided to plot the features against each other to see whether I can get any insight on the best features to train my algorithm.

The plot colours indicates the two digits.

Algorithms I considered to use were, K-Nearest Neighbour and Decision Tree. I am very new to machine learning, I chose these two algorithms simply because I have come across them.

Feature matrix of f1 to f9 against f1 to f9

Decision Tree decision boundary

I have a few questions:

Will choosing the feature x against feature y with the least amount of overlap will help achieve the optimal decision boundary?

When I look at features should I initially consider linear data separation. Then work my way up to use an algorithm that can deal with non-linear separated feature points?

What important visual properties should I look out for when choosing optimal features for training?

How can I visualise the tree in sklearn python?

Thanks.

Answer 1

• Like others said Good visual split is a good starting point. (To me it seems the f8-f1 is a good starting point.
• However you could get better results by transforming the feature set via PCA and using top eigen factors(new combination features) to train.
• You've not clarified whether it is a supervised(you know the no. of classes and actual class for some data set), assuming you don't and it is unsupervised, I'd also try an algorithm like dbscan. It is generally fast.
• If you want to try the neural network approach try either sompy or neural gas

Answer 2

Visual properties to look out for when choosing optimal features for training: choose the two features that show the separate groups the best

Answer 3

In the basics a good visual split is a good starting point. And yes, it is smart to keep in mind how the algorithms divide the space.

A good strategy, I personally like to apply is to start of with simple learners to learn how your data is structered. Hoe well does NN work, are there hints of local behavior? How well does Naive Bayes work? Is the concept complex or do individual features hold information? Etc.

As for selecting the features: You could try ranking your features on methods that compare their use (such as information gain), or simply write a scheme that tries all combinations of two on your two methods (it's only 9 * 8 runs). If the space was a little bigger I would suggest a combination of the two. You might also want to try combining features (fi: PCA).