Can we select features by examining scatterplots?

Question

Suppose, I have a data set with eight features in my hand. I want to find features to predict the diamonds, hearts, clubs, spades.

------------------------------------------------------------------------------
|   f1   |   f2   |   f3    |   f4    |    f5    |   f6    |   f7    |  f8
+--------+--------+---------+---------+---------------------------------------
|        |        |         |         |          |         |         |

f1 column is for class-labels and the rest of them are features.

First, I have rendered scatter plots by taking the features f2 and f3 and it looks like the following,

Later, I have rendered scatter plots by taking the features f3 and f4 and it looks like the following,

If $Red$ = Diamonds, $Blue$ = Spades, $Magenta$ = Hearts, and $Black$ = Clubs,

I have the following questions,

(1) Why are some plots cohesive and others are separated?

(2) What do these two plots tell about those four types of cards?

(3) Which two features, among f2, f3, and f4, would you choose for further experiments and why?

Answer 1

Like one of the commenters, I also question the utility of these scatterplots in this situation.

What you've described is a standard multi-class classification problem. Your outcomes are labeled (clubs, spades, etc.), and you have seven features (f2-f8) with which to predict the outcomes.

So, simply try putting all of the features into a standard classification model (e.g., logistic regression), and see what comes out. If you believe that some features are unimportant, you can eliminate those features from the model, and examine the cross-validation score (accuracy, f1-score, etc.) to see if the feature was truly predictive or not. For a linear model, looking at the coefficients and their standard errors can also be useful in this regard. Another approach for feature selection for generalized linear models is to examine the deviance. Hill/Gelman write in their book (Data Analysis using Regression and Multilevel/Hierarchical Models) that "when an informative predictor is added to a model, we expect the deviance to decrease by more than 1. When $k$ predictions are added to a model, we expect the deviance to decrease by more than $k$".

What you should not do is to examine scatter plots to determine which features are important. The fundamental problem is that even though the data might be well-separated in seven dimensions (i.e., using all seven features), this separation might not show up well in a scatter plot since this plot is only two-dimensional.

Answer 2

To add to molig answer one way is to plot the relationships between all the pairs of attributes by looking at the distribution of their interactions. That way you can at least remove highly correlated features.

To at least scale the plotting of feature-feature plotting you would benefit from the automatic plotting in pandas if you are in python-land replacing "data" with your data and tweaking the plotting components:

from pandas.tools.plotting import scatter_matrix
scatter_matrix(data, alpha=0.3, diagonal='kde')

Other directions is to look at the feature-class relationships with a histogram per feature-class pair.

data.groupby('class').hist()

Check this post for other viz options for your initial data exploration: http://machinelearningmastery.com/quick-and-dirty-data-analysis-with-pandas/

hope that helps.

Answer 3

I guess that depends mainly on what you want to do with that data.

Since you have labels, I suppose that you want to perform some kind of supervised learning. In that case What you want to avoid are for example features which are correlated. That is something you could identify by scatterplots. To actually select a subset of features, I would say that for example principle component analysis is a better option.