New answers tagged feature-selection
0
Individual feature selection methods assign a numerical value to every feature so that features can be ranked according to this value. The calculated value is chosen to represent how much the feature contributes to knowing the label/response variable: common choices are conditional entropy, but also information gain or correlation.
The actual values assigned ...
2
Missing values doesn't necessarily mean missing information. Sometime missing value represent an information in itself. For example: we have a data set which have features such as pool area, no. Of rooms and area. Now pool area have 90% of its value missing. You can create a new column called is_pool, which tells if the house has pool or not, from pool area ...
1
I dont think there is one correct way, but what you can do is
Use PCA if you have many features. This will reduce some number of features based on the amount of variance in each feature. You may use other dimensionality reduction techniques.
You can use models like Lightgbm or random forest and know which feature are important. 3. You may use Lasso ...
2
data.loc[ data['age'].isin(range_1) & data['height'].isin(range_2) ]
0
From an NLP perspective, the main issue is to identify and extract the terms/relations of interest from the text.
It may be easy if the sentences always appear like these simple examples: one can use pattern matching "the <entity> is <property1> [and <property2>]" (no need for ML).
However with general text it's a quite complex ...
0
You should try FastText, which is open source library by Facebook research. https://fasttext.cc/docs/en/supervised-tutorial.html
You need to create a file format needed by Fasttext algorithm.
Also following suggestions for cleaning text
Change the case to lower
Remove hyper links
Try to remove typo words
Fasttext automatically converts words into n-grams. ...
0
Feature Engineering for Machine Learning: Principles and Techniques for Data Scientists, this book is the best book for feature Engineering. Once you even partially finish this you'll know what to look out for next!
1
There are several articles and tutorials online. For example,
PythonDataScienceHandbook or towardsdatascience.
Depend on your particular problem, you can also apply the Automated tools for feature engineering.
Most of these materials have python code and links for your learning.
3
When it comes to feature importance I always go with a model-agnostic measure, as you well mention if you have two different models, they will interpret importance in different terms (Linear models as the coefficient and Tree-based models as the information gain/impurity decrease on each feature.
So you already mention one measure that does not depends on ...
0
With the information you provide, I suggest you can use both and test based upon a metric like Siluhette Score or Davis Bouldin index to measure the quality of your clusters when using either of both preprocessing techniques. In this way, you will have an objective metric to compare between the two preprocessing techniques and keep the one that maximizes the ...
1
You can try this:
import pandas as pd
df_new = pd.get_dummies(df, columns=['column2'])
print(df_new)
Output:
column1 column2_Image column2_Video
0 Google 0 0
1 Google 0 0
2 Google 0 0
3 Google 0 0
4 Facebook 1 ...
3
Your intuition so far is correct. Feature importance does not extend across models. The feature score for an xgboost model might be irrelevant and a wrong assumption for trsining another model. There is no perfect way to define important features. It does require some prior knowledge about the data in general.
3
You could break the column 2 from your example into number of columns :
Image,Video....
So the new features will be like:
Column1 Image Video
Google 0 0
Google 0 0
Facebook 1 0
Facebook 0 1
Top 50 recent answers are included
