5
$\begingroup$

Let's say I have a giant dataset (600+ columns) and I have no idea what insights I might get from it or what model I want to run.

What are some of the best ways to find the most influential columns/features? I know it's important to know what models to run first, but is there a general way to find columns that's more important than others?

So far I've already removed the columns that are 80%+ empty, but I still have over 600 columns and I have no idea where to start. Please let me know your thoughts! I'm kind of new so any pointers will help.

Improve this question
$\endgroup$
3
  • $\begingroup$ See en.wikipedia.org/wiki/Feature_selection $\endgroup$
    – DaL
    Feb 24, 2016 at 7:04
  • $\begingroup$ I assume you are looking to apply unsupervised learning, right? Do you have any labels in your data set? $\endgroup$
    – Juan Leni
    Feb 24, 2016 at 9:18
  • $\begingroup$ @mtk99, by labels you mean headers, correct? I do have headers but they are very vague headers. $\endgroup$
    – Hans Ziqiu Li
    Feb 28, 2016 at 21:11

4 Answers 4

Reset to default
5
$\begingroup$

Since you didn't mention labels of any kind, I presume you are not doing supervised learning. Note, if you were, there would be no need to throw out features. You could use something like lasso regression to build a model that sparsely selects the influential features while fitting a predictive model.

Unsupervised methods exist to reduce your feature set size but could potentially lose their original feature interpretability (if that's important to you), like PCA or auto encoders. But there are alternatives to these as well, so depends what you are looking for.

Improve this answer
$\endgroup$
5
$\begingroup$

Another way of looking at 600 columns is having a vector with 600 dimensions. So, the answer to reduce the 600 dimensional vectors to fewer dimensions without losing lot of information will be dimensionality reduction.

Principal component analysis(PCA) is a commonly used one. T-distributed stochastic neighbour embedding (TSNE) has been the state of the art for reducing higher dimensional vectors. Data visualisation using TSNE

To get the quick intuition about the vectors, the best thing is to reduce it to 2 or 3 dimensional vectors and plot them in a graph. Data visualisation always helps to get an overview of the data you are dealing with.

Improve this answer
$\endgroup$
3
  • $\begingroup$ I like the idea of using T-SNE to visualise the data set and draw conclusions. T-SNE main purpose is visualization and in particular with respect to the crowding problem. However, it is not good for general dimensionality reduction. Have a look at van der Maaten FAQ when he discusses embedding in more than 2D lvdmaaten.github.io/tsne $\endgroup$
    – Juan Leni
    Feb 24, 2016 at 17:28
  • $\begingroup$ @mtk99: Yes, that is true. Since the question is towards finding the important features, the best way is visualisation. when the dimensionality is reduced to more than 2D, it is hard to plot them. So, if you just want to reduce the dimensionality I cannot say TSNE is a good approach. But if you ask me to visualise the primary features, better to reduce it to 2D via TSNE and plot them. $\endgroup$
    – chmodsss
    Feb 24, 2016 at 17:37
  • $\begingroup$ @blueSerpent Thanks for the reply! Quick follow up question on the PCA part, so I understand the concept of using eigenvectors and eigenvalues to determine which dimension could be reduced with minimum information loss - but what is the next step after that? I'm reading up on a lot of PCA and I haven't found a source with clear steps to attribute selection. $\endgroup$
    – Hans Ziqiu Li
    Feb 29, 2016 at 7:30
4
$\begingroup$

Purging columns with >80% zeros might have been a mistake. Those non-zero values may be the most significant bits in your data. There's a whole field of study in text mining dedicated to discovering knowledge from bag of words models which are tables like yours except can be hundreds of thousands of features wide and >99% empty (i.e. sparse).

That said, an idiots approach to feature selection is to fit a linear curve (with linear or logistic regression; or another simple model) to the data, then extract the individual feature coefficients from the model. Those are your influence weights.

A simple tool to visualize feature importances is Orange. I find it's Rank widget works for me most of the time. enter image description here

Improve this answer
$\endgroup$
2
$\begingroup$

There are a lot of feature selection techniques, but depends if you have the "labels" of the samples. One simple technique involves the variance which

removes all features whose variance doesn’t meet some threshold

By default, it removes all zero-variance features"

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.