30
votes
Feature Scaling both training and test data
Generally speaking, best practice is to use only the training set to figure out how to scale / normalize, then blindly apply the same transform to the test set.
For example, say you're going to ...
- 1,264
24
votes
Accepted
Should one hot vectors be scaled with numerical attributes
Once converted to numerical form, models don't respond differently to columns of one-hot-encoded than they do to any other numerical data. So there is a clear precedent to normalise the {0,1} values ...
- 28.6k
14
votes
Accepted
What skills do I need to become a data scientist? And how to show them?
So you're still on the Basics and William's answer is pretty good, I will list here a bit of stuff to learn, and where to.
1 - You need the basics, that is already much more than you expected it to ...
- 1,626
13
votes
Feature Scaling both training and test data
To answer this question, let us take three scenarios.
...
- 271
12
votes
Accepted
How to Use Shap Kernal Explainer with Pipeline models?
The reason is kernel shap sends data as numpy array which has no column names.
so we need to fix it as follows:
...
- 426
11
votes
My data is highly overlapping, but when I apply logistic regression, it is giving an impressive accuracy of 79%. Why?
Decision Tree, KNN, & Random Forest (Methods that are suitable for overlapping data)
This statement is false. All those methods are good when the decision surface (separating surface) has a ...
- 4,643
10
votes
Accepted
When to remove correlated variables
You do not want to remove all correlated variables. It is only when the correlation is so strong that they do not convey extra information. This is both a function of the strength of correlation, how ...
- 363
9
votes
Accepted
May the training set and validation set overlap?
Definitions, so we are on the same page:
Training set: the data points used to train the model.
Validation set: the data points to keep checking the performance of your model in order to know when to ...
- 3,518
9
votes
Accepted
image_dataset_from_directory VS flow_from_directory
tf.keras.preprocessing.image_dataset_from_directory
Generates a tf.data.Dataset from image files in a directory.
ImageDataGenerator.flow_from_directory
Takes the path to a directory & generates ...
- 5,534
8
votes
What's the right input for gpt-2 in NLP
Updated answer
After reading @Jessica's answer, I carefully read the original GPT-2 paper and I confirm that the authors do not add special tokens, but simply the text ...
- 23.8k
8
votes
Accepted
How to Combat Data Drift
As you suggest, that situation could end up your monitoring system indicating a data drift. To evaluate this scenario, let's classify some types of data drift we could have:
features drift: given ...
- 2,696
7
votes
When to remove correlated variables
Weird that nobody else mentioned interpretability.
If all you are concerned with is performance, then it makes no sense to remove two correlated variables, unless correlation=1 or -1, in which case ...
- 3,410
7
votes
Accepted
Classification/Prediction based on Multivariate Time Series
Train an LSTM-RNN to perform direct sequence classification. This essentially means that it will have multiple inputs and 1 output, i.e. the label (0 or 1). In Keras/Python this is very easy to ...
- 3,920
7
votes
Accepted
will increasing threshold always increase precision?
here precision at threshold 0.85 > precision at threshold 0.90. shouldnt it be the other way round? increasing threshold will reduce False positive and precision will be greater than before?
...
- 28.6k
6
votes
Accepted
ValueError: not enough values to unpack (expected 4, got 2)
Its because you have not looked how the values are packed in plt.subplot function.
...
- 1,821
6
votes
Accepted
SHAP value can explain right?
I guess what you meant by correlation between SHAP values is "SHAP Interaction Value".
SHAP value is a measure how feature values are contributing a target variable in observation level. Likewise ...
- 866
6
votes
My data is highly overlapping, but when I apply logistic regression, it is giving an impressive accuracy of 79%. Why?
Your data is multidimensional, it is possible that any two dimensional projection overlaps while still existing an hyperplane on the original dimensionality that separates the two classes well
Say for ...
- 181
6
votes
Do model training pipeline should run on dev, staging and production environment?
Yes - Production data should be used. The highest quality, newest data should be used to train a machine learning model. Typically, new data is used to fine-tune existing models.
No - Training should ...
- 20.5k
6
votes
How to handle missing value if imputation doesnt make sense
I think this is a good solution. You could also try to set a unique negative value for non-married people, especially if you are using a tree-based model.
- 464
6
votes
How to handle missing value if imputation doesnt make sense
You could consider setting years_married to -1, then it is different from columns for the ones that are just married and could thus be understood by a decision tree....
- 126
5
votes
Accepted
How do I decide if I need to go for Normalization and not Standardization or vice-versa?
Before we start keep in mind that in most cases it doesn't play much of a difference which of the two you'll choose.
Now to answer your question, generally speaking the choice should be made based ...
- 7,908
5
votes
Accepted
Gradient Descent
Theoretically, it is possible to find a global minimum using gradient descent.
In reality, however, it rarely happens - it is also pretty much impossible to prove you have the global minimum!
...
- 14.7k
5
votes
Finding lookalike customers for Digital Media targeting
This seems to be a pretty common scenario in digital marketing, and a few companies have published their approach to lookalike modeling.
Here are a few links:
Lookalike at LinkedIn
Lookalike at ...
- 2,654
5
votes
Do model training pipeline should run on dev, staging and production environment?
Yes! You can take a dump of production data, merge with existing training data (with all processing steps) and retrain (as many number of experiments desired) your model. But before you do that, it ...
- 402
4
votes
Accepted
Is there any consensus on choosing an appropriate ML approach?
My data science studies started as a Masters in Applied Statistics. One of the courses was in machine learning and it had a similar approach to what you are describing. So, I can empathize a little ...
- 2,079
4
votes
Is there any consensus on choosing an appropriate ML approach?
Well, let's say in this way. Although there are numerous learning approaches, each is useful for a particular situation. It is possible that for a problem you have multiple choices. Each of learning ...
- 13.9k
4
votes
Accepted
Reg. Pandas factorize()
From the documnetation
Encode the object as an enumerated type or categorical variable.
This method is useful for obtaining a numeric representation of an
array when all that matters is ...
- 14.7k
4
votes
Support Vector Machine Errors
A SVM has 3 very important components: the support vectors, the separating hyperplane and the margin.
When a missclassification occurs, it is because a given point is on the
wrong side of the ...
- 81
4
votes
What are advantages or disadvantages of training deep learning model from scratch?
(Suggestions and edits will be appreciated)
let us discuss advantages of training a deep learning model from scratch:
Building and training NN from scratch is of a great use in the research field.
...
- 66
4
votes
Accepted
decision -tree regression to avoid multicollinearity for regression model?
To answer your questions directly, first:
Is there a decision tree regression model good when 10 features are
high correlated?
Yes, definitely. But even better than decision trees, is many decision ...
- 341
Only top scored, non community-wiki answers of a minimum length are eligible