We changed our privacy policy. Read more.
22

Generally speaking, best practice is to use only the training set to figure out how to scale / normalize, then blindly apply the same transform to the test set. For example, say you're going to normalize the data by removing the mean and dividing out the variance. If you use the whole dataset to figure out the feature mean and variance, you're using ...


19

Once converted to numerical form, models don't respond differently to columns of one-hot-encoded than they do to any other numerical data. So there is a clear precedent to normalise the {0,1} values if you are doing it for any reason to prepare other columns. The effect of doing so will depend on the model class, and type of normalisation you apply, but I ...


14

So you're still on the Basics and William's answer is pretty good, I will list here a bit of stuff to learn, and where to. 1 - You need the basics, that is already much more than you expected it to be: Linear Algebra: knowing the best way of inverting a matrix might be useful for a computer scientist, but you're not aiming for that. You need to understand ...


10

The reason is kernel shap sends data as numpy array which has no column names. so we need to fix it as follows: def model_predict(data_asarray): data_asframe = pd.DataFrame(data_asarray, columns=feature_names) return estimator.predict(data_asframe) Then, shap_kernel_explainer = shap.KernelExplainer(model_predict, x_train, link='logit') ...


10

Decision Tree, KNN, & Random Forest (Methods that are suitable for overlapping data) This statement is false. All those methods are good when the decision surface (separating surface) has a highly nonlinear form. They act as a non-parametric local approximation - all parameters are not in fact parameters of the decision function but are meta parameters ...


9

Definitions, so we are on the same page: Training set: the data points used to train the model. Validation set: the data points to keep checking the performance of your model in order to know when to stop training. Testing set: the data points used to check the performance once training is finished. May training and validation sets overlap? They should ...


9

You do not want to remove all correlated variables. It is only when the correlation is so strong that they do not convey extra information. This is both a function of the strength of correlation, how much data you have and whether any small difference between correlated variables tell you something about the outcome, after all. The first two you can tell ...


8

tf.keras.preprocessing.image_dataset_from_directory Generates a tf.data.Dataset from image files in a directory. ImageDataGenerator.flow_from_directory Takes the path to a directory & generates batches of augmented data. While their return type also differs but the key difference is that flow_from_directory is a method of ImageDataGenerator while ...


7

Train an LSTM-RNN to perform direct sequence classification. This essentially means that it will have multiple inputs and 1 output, i.e. the label (0 or 1). In Keras/Python this is very easy to implement, just make sure that you have a Dense layer in the end with sigmoid activation so that the output is between 0 and 1. You train the network based on your ...


6

here precision at threshold 0.85 > precision at threshold 0.90. shouldnt it be the other way round? increasing threshold will reduce False positive and precision will be greater than before? Precision is $\frac{\text{TP}}{\text{TP}+\text{FP}}$ Both $\text{TP}$ and $\text{FP}$ are reduced when you increase the threshold. If both decrease in proportion to ...


5

Its because you have not looked how the values are packed in plt.subplot function. >>> plt.subplots(2,2,figsize=(10,4)) (<matplotlib.figure.Figure at 0xa3918d0>, array([[<matplotlib.axes._subplots.AxesSubplot object at 0x000000000A389470>, <matplotlib.axes._subplots.AxesSubplot object at 0x000000000A41AD30>], [<...


5

Before we start keep in mind that in most cases it doesn't play much of a difference which of the two you'll choose. Now to answer your question, generally speaking the choice should be made based on what model you want to employ: If you use a distance-based estimator (e.g. k-NN, k-means) it's better to normalize your features so that they occupy the same ...


5

Theoretically, it is possible to find a global minimum using gradient descent. In reality, however, it rarely happens - it is also pretty much impossible to prove you have the global minimum! Imagine we have a 2d loss surface; a loss curve as in the figures below. In order to reach the global minimum (the lowest point on the curve), you would need to make ...


5

I guess what you meant by correlation between SHAP values is "SHAP Interaction Value". SHAP value is a measure how feature values are contributing a target variable in observation level. Likewise SHAP interaction value considers target values while correlation between features (Pearson, Spearman etc) does not involve target values therefore they might have ...


5

This seems to be a pretty common scenario in digital marketing, and a few companies have published their approach to lookalike modeling. Here are a few links: Lookalike at LinkedIn Lookalike at Pinterest Lookalike at Yahoo Another lookalike from Yahoo Academic paper on lookalike (not sure where the authors work) Google's lookalike patent (this one is a lot ...


5

Your data is multidimensional, it is possible that any two dimensional projection overlaps while still existing an hyperplane on the original dimensionality that separates the two classes well Say for instance you have 3 data points from 2 labels in 2d that are linearly separable X:(0,-1) O:(1,2) X:(4,3) X O X In the x axis they look ...


4

My data science studies started as a Masters in Applied Statistics. One of the courses was in machine learning and it had a similar approach to what you are describing. So, I can empathize a little with your current view. But, just like other things you might have learned in life, the way you do things in an academic setting and the way you do things in a ...


4

Well, let's say in this way. Although there are numerous learning approaches, each is useful for a particular situation. It is possible that for a problem you have multiple choices. Each of learning approaches has a special application domain and that is why people usually know where to use decision trees and where to choose neural networks, e.g. in ...


4

To answer this question, let us take three scenarios. Scenario 1: scaled_dataset = (dataset - dataset_mean) / dataset_std_deviation train, test = split(scaled_dataset) Scenario 2: train, test = split(dataset) scaled_train = (train - train_mean) / train_std_deviation scaled_test = (test - test_mean) / test_std_deviation Scenario 3: scaled_train = (train ...


4

From the documnetation Encode the object as an enumerated type or categorical variable. This method is useful for obtaining a numeric representation of an array when all that matters is identifying distinct values. factorize is available as both a top-level function pandas.factorize(), and as a method Series.factorize() and Index.factorize(). ...


4

A SVM has 3 very important components: the support vectors, the separating hyperplane and the margin. When a missclassification occurs, it is because a given point is on the wrong side of the separating hyperplane, and that's called a classification error. Whenever a point is inside the margin, that counts as a margin error. The total error of a SVM, is ...


4

(Suggestions and edits will be appreciated) let us discuss advantages of training a deep learning model from scratch: Building and training NN from scratch is of a great use in the research field. You will know your model to the most basics and can modify it in case needed as per the requirements. It will be more efficient in terms of size and training ...


4

To answer your questions directly, first: Is there a decision tree regression model good when 10 features are high correlated? Yes, definitely. But even better than decision trees, is many decision trees (RandomForest, Gradient Boosting (xGBoost is popular). I think you'd be well served by learning about how decision trees split, and how they naturally ...


4

LDA being a probabilistic model, the results depend on the type of data and problem statement. There is nothing like a valid range for coherence score but having more than 0.4 makes sense. By fixing the number of topics, you can experiment by tuning hyper parameters like alpha and beta which will give you better distribution of topics. The alpha controls ...


3

You are describing multivariate time series analysis, modeling the interactions and comovements among a group of time series variables. You can start with a vector autoregression (VAR) model, one of the most successful, flexible, and easy to use models for the analysis of multivariate time series. Then work through the decomposition of time series. You'll ...


3

Weird that nobody else mentioned interpretability. If all you are concerned with is performance, then it makes no sense to remove two correlated variables, unless correlation=1 or -1, in which case one of the variables is redundant. But if are concerned about interpretability then it might make sense to remove one of the variables, even if the correlation ...


3

Yes, correlation is a mathematical concept, and it as well known as Pearson correlation. This is probably the one you are obtaining in your analysis. However, there are other correlation analysis you can try in order to be totally sure about your result. The most famous ones out of Pearson are Kendall correlation and Spearman correlation. In addition, ...


3

In no way is this going to be an exhaustive answer, but it will definitely give you a starting point in Python - Data Exploration Start with Pandas Profiling. It will give you HTML reports of your variables. If the quality of the data is good, it will provide some insights into the fill rate, depending upon the variable type some statistics for each ...


3

The complexity of that algorithm is O(n³), and it needs O(n²) memory. So if your data grows "exponentially", you better settle for a sampling-based approach! Seriously: benchmark the run time and memory requirements for 5k, 10k, 20k, 40k, 80k instances. You should be able to observe something between O(n²) (for computing the distance matrix) and O(n³) for ...


3

It looks fine to me :) the only problem is that your plot (resulting from In [18]) is being displayed on your computer in a separate window somewhere - maybe you have to find it. Once you close that window, your iPython prompt woill return to In [19]. You could alternatively press Ctrl-C in the iPython session, but this will end the session. If the problem ...


Only top voted, non community-wiki answers of a minimum length are eligible