1
$\begingroup$

I have a pandas dataframe consisting of dimensions and features of a different fabric materials. I have several rows per product material type causing the dataset to seem very huge. From basic logic and domain knowledge I can judge that the columns that show dimensions of the fabric is not necessary to analyse the data I am looking for.

I tried to run a correlation and print a heatmap but it doesnt show any output for most of the columns. Only white color is printed out for most of the column correlation. This makes it difficult for me to judge.

Please advice if there is a better way to statistically prove the relationship between the columns and justify the reason to drop irrelevant columns

$\endgroup$
10
  • 2
    $\begingroup$ If you are talking about statistical significance then you could do a t-test with adjusted p-values for multiple comparisons. $\endgroup$ Sep 20, 2018 at 14:05
  • $\begingroup$ is there a option to run t-test even if the columns contain only string or mixture of numeric and non-numeric values? Or will I have to vectorize the columns or so? Please advice $\endgroup$
    – Sudhi
    Sep 20, 2018 at 14:17
  • $\begingroup$ No, t-test is only for numerical values and a categorical / numerical response variable. If you have different variable types you could build a model which will perform the feautre selection process for you. $\endgroup$ Sep 20, 2018 at 14:21
  • $\begingroup$ Thank you for your help. Do you have any model on top of your mind that is best for datasets containing only string data. I have been looking into Lasso-Ridge etc. Not sure if that will really help my scenario $\endgroup$
    – Sudhi
    Sep 20, 2018 at 14:46
  • 2
    $\begingroup$ The question is: what type of analysis will follow this preprocessing? $\endgroup$
    – Michael M
    Sep 20, 2018 at 16:56

6 Answers 6

1
$\begingroup$

With respect to the other answers i observed your point about having features which are strings, you must first find a way to encode them as numerical features which can then help you with PCA and the problem with the heat-map will be resolved too. Also if your problem is of regression then you can use methods such as L1-Regularization which help you in feature selection and you don't have to remove the features while pre-processing.

$\endgroup$
0
$\begingroup$

Maybe a principle component analysis (PCA) would be what you're looking for, identifying the components of your dataset that explain the most variance.

$\endgroup$
1
  • $\begingroup$ Since my data consists mostly of strings I dont think PCA helps. I did read into this. Do you think Multiple correspondence analysis will help? $\endgroup$
    – Sudhi
    Sep 20, 2018 at 16:26
0
$\begingroup$

You could use a feature selector, Random Forests can be used as one, but first I think that you should transform those columns to a usable variable (numerical or categorical), for example, since you have dimensions, maybe create a column for height and another for width, to prevent unwanted behaviours.

Other straightforward methods could be Forward Selection, Backward Elimination and some more resources here

$\endgroup$
0
$\begingroup$

I agree on Feature Selection. You can consult the Microsoft Learning Repository about ML with Python on GitHub , it provides you with some theoretical background as well as Python code to run. You could import and adapt that code for your own project. cheers

$\endgroup$
0
$\begingroup$

Lots of other good ideas already, but two come to my mind:

  1. Using a decision tree, you can demonstrate the importance of each field.

  2. Another approach could be a test for multicollinerarity which will show columns that have substantially the same relationship with the target variable. In this case, you can eliminate one of the collinear fields.

$\endgroup$
0
$\begingroup$

All above answers are on point, but just keep in mind that whatever analysis you will do, spurious correlations (seemingly significant) may occur, even though there is no causal link, as you claim (by knowing your domain, and basic logic, i.e.), or the link is purely technical (for example, if the variable would the rowid). Essentially, such knowledge should be above any statistical test. Whatever statistical test you're using is only suggesting that with certain amount of significance there may or may not be a certain relationship, but no test can know for sure.

While slightly out of topic, this discussion on statistical tests might we a worthwhile read.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.