18

If you properly isolate your test set such that it doesn't affect training, you should only look at the test set accuracy. Here are some of my remarks: Having your model being really good on the train set is not a bad thing in itself. On the contrary, if the test accuracy is identical, you want to pick the model with the better train accuracy. You want to ...


10

Yes, using one-hot encoding on 24k features requires 24k input nodes. However this should not be a problem for Keras (or any other deep learning library). Natural language processing often uses one-hot encoding on words with a vocabulary size in the same ballpark. If you are using a "deep" model, one of your hidden layers should take care of reducing the ...


8

See the docs: You need to add an intercept to statsmodels manually, while it is added automatically in sklearn. import altair as alt import numpy as np import pandas as pd from sklearn.linear_model import LinearRegression import statsmodels.api as sm np.random.seed(0) data = pd.DataFrame({ 'Date': pd.date_range('1990-01-01', freq='D', periods=50), 'NDVI': ...


7

I think the easiest way to do this would be to have a decision tree where the final decision results in a linear formula. Setting aside whether this is actually easiest/best, this type of model does exist, usually called "model-based recursive partitioning". See e.g. https://stats.stackexchange.com/q/78563/232706 There are several packages in R for this: ...


6

Entity Embedding for Categorical Variables (original pager) would be a very suitable approach here. Read on here, or here. I have actually put pieces of codes from here and there and made an complete running implementation, see this git repo. This easily handles very high cardinal categorical variables using neural networks. I won't list pros and cons of OHE,...


6

Both curves show the training and validation scores of an estimator on the y-axis. A learning curve plots the score over varying numbers of training samples, while a validation curve plots the score over a varying hyper parameter. The learning curve is a tool for finding out if an estimator would benefit from more data, or if the model is too simple (...


6

GridSearchCV is built around cross validation, but if speed is your main concern, you may be able to get better performance using a smaller number of folds. From the docs: class sklearn.model_selection.GridSearchCV(estimator, param_grid, scoring=None, n_jobs=None, iid='deprecated', refit=True, cv=None, verbose=0, pre_dispatch='2*n_jobs', error_score=nan, ...


6

Test accuracy better reflects generalization error, so you want the one with higher test accuracy. In your first setup, the higher train accuracy indicates overfitting, as it's significantly higher than train accuracy. This is also kind of why it generalizes less well than the second one.


5

As for differences in OrdinalEncoder and LabelEncoder implementation, the accepted answer mentions the shape of the data: (OrdinalEncoder for 2D data; shape (n_samples, n_features), LabelEncoder is for 1D data: for shape (n_samples,)) That's why a OrdinalEncoder would get an error: ValueError: Expected 2D array, got 1D array instead: ...if trying to fit ...


5

In the general case this is impossible because standard learning algorithms require all the instances at once in order to compute the model. They usually work by making calculations based on the set of instances, then only the results of these calculations are stored as a model (for instance a linear regression classifier stores only the final weights which ...


5

A few things going on here: Your matrix is 100x100. So you have no degrees of freedom left in a linear model, which will cause $R^2=1$. See this post. You use random numbers. Thus, they should make little sense in terms of explaining your dependent variable (it's basically noise). Since Lasso "shrinks" parameters which are not useful (and none is useful ...


5

Generally, the scoring metrics you are looking at are defined as following (see for example Wikipedia): $$precision = \frac{TP}{TP+FP}$$ $$recall= \frac{TP}{TP+FN}$$ $$F1 = \frac{2 \times precision \times recall}{precision + recall}$$ For the multi-class case scikit learn offers the following parameterizations (see here for example): 'micro': ...


5

This has been a discussion for quite a while. For a more theoretical point of view you can find a good summary here. From a practical point of view I'd look at it as follows. With increasing $k$ two things happen: Your $k-1$ training folds increase in size Your validation folds decreases in size From the first point you can draw the conclusion that your $...


5

First I did a visualization of what your code is doing (see code at the bottom) The model seems completely fine. The coefficient of the linear regression is close to 0, where it should be given how you have created the data. You are misunderstanding regularized_logistic_regression_model.scores_ {1: array([[0.47058824, 0.47058824, 0.47058824, 0....


4

I am using svm.SVR() from scikit-learn to apply Logistic Regression on my training data to solve similarity problem. Wait a second, if you're using support-vector regression, then you're not using logistic regression. These two are very different algorithms. They aren't even applicable to the same type of problem. Support-vector regression is used when ...


4

It's a classic outlier. You could, for example, remove him or replace him with a new value(by interpolation). You have many ways to work around this problem. Links for you: https://towardsdatascience.com/ways-to-detect-and-remove-the-outliers-404d16608dba With some code: https://towardsdatascience.com/5-ways-to-detect-outliers-that-every-data-scientist-...


4

To quote from the documentation: The predicted class probabilities of an input sample are computed as the mean predicted class probabilities of the trees in the forest. The class probability of a single tree is the fraction of samples of the same class in a leaf. The predictions are then simply the class with the highest ...


4

Not every model is able to learn sample-by-sample or incrementally. However, in scikit-learn, there're some models which have partial_fit method: Incremental fit on a batch of samples. This method is expected to be called several times consecutively on different chunks of a dataset so as to implement out-of-core or online learning. This is ...


4

You can approximate a time series with a polynomial of degree n_degree by using ridge regression. You can try different degree numbers (e.g. [2,3,4,5,6]) and choose the best one. Keep in mind that higher degree models, always get lower error values. So you should somehow penalize higher degrees. from sklearn.linear_model import Ridge from sklearn....


4

f_val is the F Statistic value. Mathematically it is $ F = \frac{MS_{Between}}{MS_{Within}}$ The null hypothesis for your ANOVA is $H_0: \mu_{Entire home/apt} = \mu_{Private room} = \mu_{Shared room} $ which means all means ($\mu_i$ s) are equal and there is no need of grouping using explanatory variable vs $H_A:$ At least one $\mu_i$ is different. ...


4

Searching the source code of Sklearn for SimpleImputer (with strategy= "most_frequent"), the most frequent value is calculated within a loop in python, therefore that is the part of code that is so slow. In the source code of SimpleImputer there is also the comment that explains why they do not use the scipy.stats.mstats.mode, which is mush faster: scipy....


4

Sure, here is a skeleton in TF 2.0. import tensorflow as tf weights = tf.Variable(tf.random.normal(shape=(784, 10), dtype=tf.float64)) biases = tf.Variable(tf.random.normal(shape=(10,), dtype=tf.float64)) def logistic_regression(x): lr = tf.add(tf.matmul(x, weights), biases) return lr def cross_entropy(y_true, y_pred): y_true = tf.one_hot(...


4

If the "cost" for experimenting is not really that big I suggest you take the time to experiment and take this as a learning opportunity and just try if it could actually work. There are many approaches to address class imbalance and setting class weight is one of them and the easiest to implement. Change loss function (for example to focal loss for ...


4

A random forest model is an agglomeration of Decision Trees. tree.feature_importance_ defines the feature importance for each individual tree, but model.feature_importance_ is the feature importance for the forest as a whole. The docs give the explanation for calculation as: The relative rank (i.e. depth) of a feature used as a decision node in a tree can ...


4

_feature_importance of a random forest calculates the average feature importance across all trees in the forest. While tree.feature_importances_ is the feature importance for a single tree. Since feature importance is calculated as the contribution of a feature to maximize the split criterion (or equivalently: minimize impurity of child nodes) higher is ...


4

First, to directly answer your question, the easiest way to get Feature Importance using scikit learn is this, where model is the variable holding your classifier. print(model.feature_importances_) However, this method only exists on some of the Ensemble models, namely: AdaBoostClassifier AdaBoostRegressor ExtraTreesClassifier ExtraTreesRegressor ...


4

The most common approach is a combination of these two strategies: Domain expertise - Given knowledge about the dataset and the goal of the model, choose the order that best manipulates the data to accomplish the goal of the project. Empirical evidence - Permutation the order and benchmark results. Pick the permutation that has the highest performance on ...


4

Yes it would be possible that it happens. It means that this event happening has no importance for the target. Imagine a categorical feature with a lot of categories(high cardinality). Maybe only one of them does not have any influence in the target so if you do feature selection this feature might have a high chance of getting dropped. So this is normal ...


4

The minimum and maximum values are just known limits that are parts of the formula that reshapes the distribution of the data, so if a value is bigger than the previously known value the resulting feature scaling (Normalization) will be still appropriate. An alternative is z-scores if you don't feel like using minimum and maximum values. x'= (x-x̄) / σ ...


4

rfc.fit(X, y) should be rfc.fit(X_train, y_train) You are simply memorizing the entire dataset with RandomForestClassifier.


Only top voted, non community-wiki answers of a minimum length are eligible