Last call to make your voice heard! Our 2022 Developer Survey closes in less than a week. Take survey.
25 votes

Feature Scaling both training and test data

Generally speaking, best practice is to use only the training set to figure out how to scale / normalize, then blindly apply the same transform to the test set. For example, say you're going to ...
user avatar
  • 1,154
19 votes
Accepted

Should one hot vectors be scaled with numerical attributes

Once converted to numerical form, models don't respond differently to columns of one-hot-encoded than they do to any other numerical data. So there is a clear precedent to normalise the {0,1} values ...
user avatar
  • 27.2k
14 votes
Accepted

What skills do I need to become a data scientist? And how to show them?

So you're still on the Basics and William's answer is pretty good, I will list here a bit of stuff to learn, and where to. 1 - You need the basics, that is already much more than you expected it to ...
user avatar
11 votes
Accepted

How to Use Shap Kernal Explainer with Pipeline models?

The reason is kernel shap sends data as numpy array which has no column names. so we need to fix it as follows: ...
user avatar
11 votes

My data is highly overlapping, but when I apply logistic regression, it is giving an impressive accuracy of 79%. Why?

Decision Tree, KNN, & Random Forest (Methods that are suitable for overlapping data) This statement is false. All those methods are good when the decision surface (separating surface) has a ...
user avatar
  • 4,408
10 votes
Accepted

When to remove correlated variables

You do not want to remove all correlated variables. It is only when the correlation is so strong that they do not convey extra information. This is both a function of the strength of correlation, how ...
user avatar
  • 363
9 votes
Accepted

May the training set and validation set overlap?

Definitions, so we are on the same page: Training set: the data points used to train the model. Validation set: the data points to keep checking the performance of your model in order to know when to ...
user avatar
9 votes
Accepted

image_dataset_from_directory VS flow_from_directory

tf.keras.preprocessing.image_dataset_from_directory Generates a tf.data.Dataset from image files in a directory. ImageDataGenerator.flow_from_directory Takes the path to a directory & generates ...
user avatar
  • 5,144
7 votes
Accepted

Classification/Prediction based on Multivariate Time Series

Train an LSTM-RNN to perform direct sequence classification. This essentially means that it will have multiple inputs and 1 output, i.e. the label (0 or 1). In Keras/Python this is very easy to ...
user avatar
  • 3,720
7 votes
Accepted

will increasing threshold always increase precision?

here precision at threshold 0.85 > precision at threshold 0.90. shouldnt it be the other way round? increasing threshold will reduce False positive and precision will be greater than before? ...
user avatar
  • 27.2k
6 votes

My data is highly overlapping, but when I apply logistic regression, it is giving an impressive accuracy of 79%. Why?

Your data is multidimensional, it is possible that any two dimensional projection overlaps while still existing an hyperplane on the original dimensionality that separates the two classes well Say for ...
user avatar
5 votes
Accepted

ValueError: not enough values to unpack (expected 4, got 2)

Its because you have not looked how the values are packed in plt.subplot function. ...
user avatar
  • 1,729
5 votes

Feature Scaling both training and test data

To answer this question, let us take three scenarios. ...
user avatar
5 votes
Accepted

How do I decide if I need to go for Normalization and not Standardization or vice-versa?

Before we start keep in mind that in most cases it doesn't play much of a difference which of the two you'll choose. Now to answer your question, generally speaking the choice should be made based ...
user avatar
  • 7,428
5 votes
Accepted

Gradient Descent

Theoretically, it is possible to find a global minimum using gradient descent. In reality, however, it rarely happens - it is also pretty much impossible to prove you have the global minimum! ...
user avatar
  • 13.8k
5 votes
Accepted

SHAP value can explain right?

I guess what you meant by correlation between SHAP values is "SHAP Interaction Value". SHAP value is a measure how feature values are contributing a target variable in observation level. Likewise ...
user avatar
5 votes

Finding lookalike customers for Digital Media targeting

This seems to be a pretty common scenario in digital marketing, and a few companies have published their approach to lookalike modeling. Here are a few links: Lookalike at LinkedIn Lookalike at ...
user avatar
  • 2,271
4 votes

When to remove correlated variables

Weird that nobody else mentioned interpretability. If all you are concerned with is performance, then it makes no sense to remove two correlated variables, unless correlation=1 or -1, in which case ...
user avatar
4 votes
Accepted

Is there any consensus on choosing an appropriate ML approach?

My data science studies started as a Masters in Applied Statistics. One of the courses was in machine learning and it had a similar approach to what you are describing. So, I can empathize a little ...
user avatar
4 votes

Is there any consensus on choosing an appropriate ML approach?

Well, let's say in this way. Although there are numerous learning approaches, each is useful for a particular situation. It is possible that for a problem you have multiple choices. Each of learning ...
user avatar
  • 13.3k
4 votes
Accepted

Reg. Pandas factorize()

From the documnetation Encode the object as an enumerated type or categorical variable. This method is useful for obtaining a numeric representation of an array when all that matters is ...
user avatar
  • 13.8k
4 votes

Support Vector Machine Errors

A SVM has 3 very important components: the support vectors, the separating hyperplane and the margin. When a missclassification occurs, it is because a given point is on the wrong side of the ...
user avatar
4 votes

What are advantages or disadvantages of training deep learning model from scratch?

(Suggestions and edits will be appreciated) let us discuss advantages of training a deep learning model from scratch: Building and training NN from scratch is of a great use in the research field. ...
user avatar
4 votes
Accepted

decision -tree regression to avoid multicollinearity for regression model?

To answer your questions directly, first: Is there a decision tree regression model good when 10 features are high correlated? Yes, definitely. But even better than decision trees, is many decision ...
user avatar
  • 331
4 votes
Accepted

Calculating optimal number of topics for topic modeling (LDA)

LDA being a probabilistic model, the results depend on the type of data and problem statement. There is nothing like a valid range for coherence score but having more than 0.4 makes sense. By fixing ...
user avatar
  • 1,246
4 votes
Accepted

How FLAIR NER algorithm detects entities with typo?

This is not specific to FLAIR, this is how NER models work in general. A NER model captures the clues in a sentence which are likely to correspond to an entity of a particular category, for example: ...
user avatar
  • 21k
3 votes
Accepted

Time-series decomposition to a base level and an effect of another feature

You are describing multivariate time series analysis, modeling the interactions and comovements among a group of time series variables. You can start with a vector autoregression (VAR) model, one of ...
user avatar
3 votes
Accepted

How do I determine if variables are correlated? Is it simply a mathematical calculation?

Yes, correlation is a mathematical concept, and it as well known as Pearson correlation. This is probably the one you are obtaining in your analysis. However, there are other correlation analysis you ...
user avatar
  • 208
3 votes

What the good general regression technqiue for a problem with 50 independent varaibles

In no way is this going to be an exhaustive answer, but it will definitely give you a starting point in Python - Data Exploration Start with ...
user avatar
3 votes
Accepted

how to implement a hierarchical clustering technique using parallel execution in R

The complexity of that algorithm is O(n³), and it needs O(n²) memory. So if your data grows "exponentially", you better settle for a sampling-based approach! Seriously: benchmark the run time and ...
user avatar

Only top scored, non community-wiki answers of a minimum length are eligible