Skip to main content
Share Your Experience: Take the 2024 Developer Survey
31 votes

Feature Scaling both training and test data

Generally speaking, best practice is to use only the training set to figure out how to scale / normalize, then blindly apply the same transform to the test set. For example, say you're going to ...
Matthew's user avatar
  • 1,284
25 votes
Accepted

Should one hot vectors be scaled with numerical attributes

Once converted to numerical form, models don't respond differently to columns of one-hot-encoded than they do to any other numerical data. So there is a clear precedent to normalise the {0,1} values ...
Neil Slater's user avatar
14 votes
Accepted

What skills do I need to become a data scientist? And how to show them?

So you're still on the Basics and William's answer is pretty good, I will list here a bit of stuff to learn, and where to. 1 - You need the basics, that is already much more than you expected it to ...
Pedro Henrique Monforte's user avatar
13 votes

Feature Scaling both training and test data

To answer this question, let us take three scenarios. ...
Saurabh Kansal's user avatar
13 votes
Accepted

How to Use Shap Kernal Explainer with Pipeline models?

The reason is kernel shap sends data as numpy array which has no column names. so we need to fix it as follows: ...
Nayana Madhu's user avatar
11 votes

My data is highly overlapping, but when I apply logistic regression, it is giving an impressive accuracy of 79%. Why?

Decision Tree, KNN, & Random Forest (Methods that are suitable for overlapping data) This statement is false. All those methods are good when the decision surface (separating surface) has a ...
rapaio's user avatar
  • 4,763
10 votes
Accepted

When to remove correlated variables

You do not want to remove all correlated variables. It is only when the correlation is so strong that they do not convey extra information. This is both a function of the strength of correlation, how ...
Björn's user avatar
  • 363
9 votes
Accepted

May the training set and validation set overlap?

Definitions, so we are on the same page: Training set: the data points used to train the model. Validation set: the data points to keep checking the performance of your model in order to know when to ...
Bruno Lubascher's user avatar
9 votes
Accepted

image_dataset_from_directory VS flow_from_directory

tf.keras.preprocessing.image_dataset_from_directory Generates a tf.data.Dataset from image files in a directory. ImageDataGenerator.flow_from_directory Takes the path to a directory & generates ...
10xAI's user avatar
  • 5,624
8 votes
Accepted

will increasing threshold always increase precision?

here precision at threshold 0.85 > precision at threshold 0.90. shouldnt it be the other way round? increasing threshold will reduce False positive and precision will be greater than before? ...
Neil Slater's user avatar
8 votes

What's the right input for gpt-2 in NLP

Updated answer After reading @Jessica's answer, I carefully read the original GPT-2 paper and I confirm that the authors do not add special tokens, but simply the text ...
noe's user avatar
  • 26.9k
8 votes
Accepted

How to Combat Data Drift

As you suggest, that situation could end up your monitoring system indicating a data drift. To evaluate this scenario, let's classify some types of data drift we could have: features drift: given ...
German C M's user avatar
  • 2,724
7 votes

When to remove correlated variables

Weird that nobody else mentioned interpretability. If all you are concerned with is performance, then it makes no sense to remove two correlated variables, unless correlation=1 or -1, in which case ...
Ricardo Cruz's user avatar
  • 3,420
7 votes
Accepted

Classification/Prediction based on Multivariate Time Series

Train an LSTM-RNN to perform direct sequence classification. This essentially means that it will have multiple inputs and 1 output, i.e. the label (0 or 1). In Keras/Python this is very easy to ...
pcko1's user avatar
  • 3,940
6 votes
Accepted

ValueError: not enough values to unpack (expected 4, got 2)

Its because you have not looked how the values are packed in plt.subplot function. ...
Ankit Seth's user avatar
  • 1,821
6 votes
Accepted

SHAP value can explain right?

I guess what you meant by correlation between SHAP values is "SHAP Interaction Value". SHAP value is a measure how feature values are contributing a target variable in observation level. Likewise ...
Ilker Kurtulus's user avatar
6 votes

My data is highly overlapping, but when I apply logistic regression, it is giving an impressive accuracy of 79%. Why?

Your data is multidimensional, it is possible that any two dimensional projection overlaps while still existing an hyperplane on the original dimensionality that separates the two classes well Say for ...
Khanis Rok's user avatar
6 votes

Do model training pipeline should run on dev, staging and production environment?

Yes - Production data should be used. The highest quality, newest data should be used to train a machine learning model. Typically, new data is used to fine-tune existing models. No - Training should ...
Brian Spiering's user avatar
6 votes

How to handle missing value if imputation doesnt make sense

I think this is a good solution. You could also try to set a unique negative value for non-married people, especially if you are using a tree-based model.
Just trying's user avatar
6 votes

How to handle missing value if imputation doesnt make sense

You could consider setting years_married to -1, then it is different from columns for the ones that are just married and could thus be understood by a decision tree....
justinlk's user avatar
  • 136
5 votes
Accepted

How do I decide if I need to go for Normalization and not Standardization or vice-versa?

Before we start keep in mind that in most cases it doesn't play much of a difference which of the two you'll choose. Now to answer your question, generally speaking the choice should be made based ...
Djib2011's user avatar
  • 7,998
5 votes
Accepted

Gradient Descent

Theoretically, it is possible to find a global minimum using gradient descent. In reality, however, it rarely happens - it is also pretty much impossible to prove you have the global minimum! ...
n1k31t4's user avatar
  • 14.9k
5 votes

Finding lookalike customers for Digital Media targeting

This seems to be a pretty common scenario in digital marketing, and a few companies have published their approach to lookalike modeling. Here are a few links: Lookalike at LinkedIn Lookalike at ...
zachdj's user avatar
  • 2,734
5 votes
Accepted

Calculating optimal number of topics for topic modeling (LDA)

LDA being a probabilistic model, the results depend on the type of data and problem statement. There is nothing like a valid range for coherence score but having more than 0.4 makes sense. By fixing ...
prashant0598's user avatar
  • 1,511
5 votes

Do model training pipeline should run on dev, staging and production environment?

Yes! You can take a dump of production data, merge with existing training data (with all processing steps) and retrain (as many number of experiments desired) your model. But before you do that, it ...
Random Nerd's user avatar
4 votes
Accepted

Is there any consensus on choosing an appropriate ML approach?

My data science studies started as a Masters in Applied Statistics. One of the courses was in machine learning and it had a similar approach to what you are describing. So, I can empathize a little ...
I_Play_With_Data's user avatar
4 votes

Is there any consensus on choosing an appropriate ML approach?

Well, let's say in this way. Although there are numerous learning approaches, each is useful for a particular situation. It is possible that for a problem you have multiple choices. Each of learning ...
Green Falcon's user avatar
  • 14.1k
4 votes
Accepted

Reg. Pandas factorize()

From the documnetation Encode the object as an enumerated type or categorical variable. This method is useful for obtaining a numeric representation of an array when all that matters is ...
n1k31t4's user avatar
  • 14.9k
4 votes

Support Vector Machine Errors

A SVM has 3 very important components: the support vectors, the separating hyperplane and the margin. When a missclassification occurs, it is because a given point is on the wrong side of the ...
Fábio Colaço's user avatar
4 votes
Accepted

PicklingError: Could not serialize object: TypeError: can't pickle fasttext_pybind.fasttext objects

In FastText Users FB page a certain Maksym Kysylov answered me " It's not a FastText problem. It's a Spark problem :) When you apply function to Dataframe (or RDD) Spark needs to serialize it and send ...
DS_Tn's user avatar
  • 81

Only top scored, non community-wiki answers of a minimum length are eligible