14

Once converted to numerical form, models don't respond differently to columns of one-hot-encoded than they do to any other numerical data. So there is a clear precedent to normalise the {0,1} values if you are doing it for any reason to prepare other columns. The effect of doing so will depend on the model class, and type of normalisation you apply, but I ...


13

So you're still on the Basics and William's answer is pretty good, I will list here a bit of stuff to learn, and where to. 1 - You need the basics, that is already much more than you expected it to be: Linear Algebra: knowing the best way of inverting a matrix might be useful for a computer scientist, but you're not aiming for that. You need to understand ...


9

Generally speaking, best practice is to use only the training set to figure out how to scale / normalize, then blindly apply the same transform to the test set. For example, say you're going to normalize the data by removing the mean and dividing out the variance. If you use the whole dataset to figure out the feature mean and variance, you're using ...


8

You do not want to remove all correlated variables. It is only when the correlation is so strong that they do not convey extra information. This is both a function of the strength of correlation, how much data you have and whether any small difference between correlated variables tell you something about the outcome, after all. The first two you can tell ...


7

Definitions, so we are on the same page: Training set: the data points used to train the model. Validation set: the data points to keep checking the performance of your model in order to know when to stop training. Testing set: the data points used to check the performance once training is finished. May training and validation sets overlap? They should ...


7

The reason is kernel shap sends data as numpy array which has no column names. so we need to fix it as follows: def model_predict(data_asarray): data_asframe = pd.DataFrame(data_asarray, columns=feature_names) return estimator.predict(data_asframe) Then, shap_kernel_explainer = shap.KernelExplainer(model_predict, x_train, link='logit') ...


5

Train an LSTM-RNN to perform direct sequence classification. This essentially means that it will have multiple inputs and 1 output, i.e. the label (0 or 1). In Keras/Python this is very easy to implement, just make sure that you have a Dense layer in the end with sigmoid activation so that the output is between 0 and 1. You train the network based on your ...


5

Theoretically, it is possible to find a global minimum using gradient descent. In reality, however, it rarely happens - it is also pretty much impossible to prove you have the global minimum! Imagine we have a 2d loss surface; a loss curve as in the figures below. In order to reach the global minimum (the lowest point on the curve), you would need to make ...


4

Its because you have not looked how the values are packed in plt.subplot function. >>> plt.subplots(2,2,figsize=(10,4)) (<matplotlib.figure.Figure at 0xa3918d0>, array([[<matplotlib.axes._subplots.AxesSubplot object at 0x000000000A389470>, <matplotlib.axes._subplots.AxesSubplot object at 0x000000000A41AD30>], [<...


4

My data science studies started as a Masters in Applied Statistics. One of the courses was in machine learning and it had a similar approach to what you are describing. So, I can empathize a little with your current view. But, just like other things you might have learned in life, the way you do things in an academic setting and the way you do things in a ...


4

Well, let's say in this way. Although there are numerous learning approaches, each is useful for a particular situation. It is possible that for a problem you have multiple choices. Each of learning approaches has a special application domain and that is why people usually know where to use decision trees and where to choose neural networks, e.g. in ...


4

Before we start keep in mind that in most cases it doesn't play much of a difference which of the two you'll choose. Now to answer your question, generally speaking the choice should be made based on what model you want to employ: If you use a distance-based estimator (e.g. k-NN, k-means) it's better to normalize your features so that they occupy the same ...


4

I guess what you meant by correlation between SHAP values is "SHAP Interaction Value". SHAP value is a measure how feature values are contributing a target variable in observation level. Likewise SHAP interaction value considers target values while correlation between features (Pearson, Spearman etc) does not involve target values therefore they might have ...


4

here precision at threshold 0.85 > precision at threshold 0.90. shouldnt it be the other way round? increasing threshold will reduce False positive and precision will be greater than before? Precision is $\frac{\text{TP}}{\text{TP}+\text{FP}}$ Both $\text{TP}$ and $\text{FP}$ are reduced when you increase the threshold. If both decrease in proportion to ...


4

(Suggestions and edits will be appreciated) let us discuss advantages of training a deep learning model from scratch: Building and training NN from scratch is of a great use in the research field. You will know your model to the most basics and can modify it in case needed as per the requirements. It will be more efficient in terms of size and training ...


4

To answer your questions directly, first: Is there a decision tree regression model good when 10 features are high correlated? Yes, definitely. But even better than decision trees, is many decision trees (RandomForest, Gradient Boosting (xGBoost is popular). I think you'd be well served by learning about how decision trees split, and how they naturally ...


3

You are describing multivariate time series analysis, modeling the interactions and comovements among a group of time series variables. You can start with a vector autoregression (VAR) model, one of the most successful, flexible, and easy to use models for the analysis of multivariate time series. Then work through the decomposition of time series. You'll ...


3

Yes, correlation is a mathematical concept, and it as well known as Pearson correlation. This is probably the one you are obtaining in your analysis. However, there are other correlation analysis you can try in order to be totally sure about your result. The most famous ones out of Pearson are Kendall correlation and Spearman correlation. In addition, ...


3

In no way is this going to be an exhaustive answer, but it will definitely give you a starting point in Python - Data Exploration Start with Pandas Profiling. It will give you HTML reports of your variables. If the quality of the data is good, it will provide some insights into the fill rate, depending upon the variable type some statistics for each ...


3

From the documnetation Encode the object as an enumerated type or categorical variable. This method is useful for obtaining a numeric representation of an array when all that matters is identifying distinct values. factorize is available as both a top-level function pandas.factorize(), and as a method Series.factorize() and Index.factorize(). ...


3

A SVM has 3 very important components: the support vectors, the separating hyperplane and the margin. When a missclassification occurs, it is because a given point is on the wrong side of the separating hyperplane, and that's called a classification error. Whenever a point is inside the margin, that counts as a margin error. The total error of a SVM, is ...


3

The complexity of that algorithm is O(n³), and it needs O(n²) memory. So if your data grows "exponentially", you better settle for a sampling-based approach! Seriously: benchmark the run time and memory requirements for 5k, 10k, 20k, 40k, 80k instances. You should be able to observe something between O(n²) (for computing the distance matrix) and O(n³) for ...


3

It looks fine to me :) the only problem is that your plot (resulting from In [18]) is being displayed on your computer in a separate window somewhere - maybe you have to find it. Once you close that window, your iPython prompt woill return to In [19]. You could alternatively press Ctrl-C in the iPython session, but this will end the session. If the problem ...


3

If your model is simple and you don't have a lot of training data, then you need a model with few parameters to compare to, or else you won't be able to train it. Using standard CNN architectures may not be a good option in this case, because even if the point of using a CNN rather than a full-connected network is to reduce the parameters, they still have ...


3

I don't use Python so I can't tell you exactly what is going on but I had a quick look at your data: A few remarks: it looks like the vast majority of the points are created artificially by interpolation. Why not, but that's unlikely to reflect the reality of the price changes: I would expect much more variation/noise in a real dataset about car prices. ...


3

Yes, you are right that when number of observations are very large, k fold cross validation (CV) are less useful. Let's look at why this is so: 1) Very high number of observations imply high training time for model and validation. Already the number of observations is large for the model to be trained and validated and now we are demanding it to be done k ...


3

It is not necessary to one-hot-encode your categorical features when using tree-based methods. Basically the idea is that the tree has to make many splits to figure out the category. Instead, you can use ordinal encoder (even if categories are not ordered). I also would not have set 'YEAR_BUILT' as a categorical variable, even though it is discrete. By one-...


3

The advantages of training a deep learning model from scratch and of transfer learning are subjective. It depends a lot on the problem you are trying to solve, the time constraints, the availability of data and the computational resources you have. Let's consider a scenario, you want to train a deep learning model for a task like sentiment classification ...


3

Your chart seems to show that light GBM models are very inconsistent in terms of F1 score. The other two types of model tend to have lower validation accuracy than training accuracy, suggesting overfitting is occurring to some extent (but this is ubiquitous in machine learning so it’s not a deal breaker by any means). The best median validation performance ...


3

You want to manually label some cases and then extend that "manual labeling" to the rest of the data. This is a supervised learning excercise with prior manual labeling by you. Let's suppose you have partitioned a random, suitably sized training data set. Now you need to model a classification algorithm via the classical modeling pipeline and use ...


Only top voted, non community-wiki answers of a minimum length are eligible