139

There are some cases where LabelEncoder or DictVectorizor are useful, but these are quite limited in my opinion due to ordinality. LabelEncoder can turn [dog,cat,dog,mouse,cat] into [1,2,1,3,2], but then the imposed ordinality means that the average of dog and mouse is cat. Still there are algorithms like decision trees and random forests that can work with ...


137

To center the data (make it have zero mean and unit standard error), you subtract the mean and then divide the result by the standard deviation. $$x' = \frac{x-\mu}{\sigma}$$ You do that on the training set of data. But then you have to apply the same transformation to your testing set (e.g. in cross-validation), or to newly obtained examples before ...


93

You could just use sklearn.model_selection.train_test_split twice. First to split to train, test and then split train again into validation and train. Something like this: X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=1) X_train, X_val, y_train, y_val = train_test_split(X_train, y_train, test_size=0.2, ...


75

Kernelized SVMs require the computation of a distance function between each point in the dataset, which is the dominating cost of $\mathcal{O}(n_\text{features} \times n_\text{observations}^2)$. The storage of the distances is a burden on memory, so they're recomputed on the fly. Thankfully, only the points nearest the decision boundary are needed most of ...


60

In most of the well-established machine learning systems, categorical variables are handled naturally. For example in R you would use factors, in WEKA you would use nominal variables. This is not the case in scikit-learn. The decision trees implemented in scikit-learn uses only numerical features and these features are interpreted always as continuous ...


35

There is a great answer to this question over on SO that uses numpy and pandas. The command (see the answer for the discussion): train, validate, test = np.split(df.sample(frac=1), [int(.6*len(df)), int(.8*len(df))]) produces a 60%, 20%, 20% split for training, validation and test sets.


32

Assuming your target is (0,1), then the classifier would output a probability matrix of dimension (N,2). The first index refers to the probability that the data belong to class 0, and the second refers to the probability that the data belong to class 1. These two would sum to 1. You can then output the result by: probability_class_1 = model....


29

What you are looking for, is the Non-negative least square regression. It is a simple optimization problem in quadratic programming where your constraint is that all the coefficients(a.k.a weights) should be positive. Having said that, there is no standard implementation of Non-negative least squares in Scikit-Learn. The pull request is still open. But, ...


27

While AN6U5 has given a very good answer, I wanted to add a few points for future reference. When considering One Hot Encoding(OHE) and Label Encoding, we must try and understand what model you are trying to build. Namely the two categories of model we will be considering are: Tree Based Models: Gradient Boosted Decision Trees and Random Forests. Non-Tree ...


27

This is done when the variables span several orders of magnitude. Income is a typical example: its distribution is "power law", meaning that the vast majority of incomes are small and very few are big. This type of "fat tailed" distribution is studied in logarithmic scale because of the mathematical properties of the logarithm: $$log(x^n)= n log(x)$$ ...


26

Your problem can be solved with Word2vec as well as Doc2vec. Doc2vec would give better results because it takes sentences into account while training the model. Doc2vec solution You can train your doc2vec model following this link. You may want to perform some pre-processing steps like removing all stop words (words like "the", "an", etc. that don't add ...


25

Since you mention "numeric" features, I guess your features are not categorical and have a high arity (they can take a lot of different values, and thus there are a lot of possible split points). In such a case, growing trees is difficult since there are [a lot of features $\times$ a lot of split points] to evaluate. My guess is that the biggest effect ...


25

In the interest of preventing information about the distribution of the test set leaking into your model, you should go for option #2 and fit the scaler on your training data only, then standardise both training and test sets with that scaler. By fitting the scaler on the full dataset prior to splitting (option #1), information about the test set is used to ...


24

A regular SVM with default values uses a radial basis function as the SVM kernel. This is basically a Gaussian kernel aka bell-curve. Meaning that the no man's land between different classes is created with a Gaussian function. The linear-SVM uses a linear kernel for the basis function, so you can think of this as a ^ shaped function. It is much less ...


23

No, sklearn doesn't seem to have a forward selection algorithm. However, it does provide recursive feature elimination, which is a greedy feature elimination algorithm similar to sequential backward selection. See the documentation here: http://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.RFE.html


23

The comments about iteration number are spot on. The default SGDClassifier n_iter is 5 meaning you do 5 * num_rows steps in weight space. The sklearn rule of thumb is ~ 1 million steps for typical data. For your example, just set it to 1000 and it might reach tolerance first. Your accuracy is lower with SGDClassifier because it's hitting iteration limit ...


18

SVM solves an optimization problem of quadratic order. I do not have anything to add that has not been said here. I just want to post a link the sklearn page about SVC which clarifies what is going on: The implementation is based on libsvm. The fit time complexity is more than quadratic with the number of samples which makes it hard to scale to ...


18

First of all, sklearn.metrics.mutual_info_score implements mutual information for evaluating clustering results, not pure Kullback-Leibler divergence! This is equal to the Kullback-Leibler divergence of the joint distribution with the product distribution of the marginals. KL divergence (and any other such measure) expects the input data to have a sum of ...


18

You can use the dataframe's .values method to access raw data once you have manipulated the columns as you need them. E.g. train = pd.read_csv("train.csv") target = train['target'] train = train.drop(['ID','target'],axis=1) test = pd.read_csv("test.csv") test = test.drop(['ID'],axis=1) xgtrain = xgb.DMatrix(train.values, target.values) xgtest = xgb....


18

I use a workaround with Lasso on Scikit Learn (It is definitely not the best way to do things but it works well). Lasso has a parameter positive which can be set to True and force the coefficients to be positive. Further, setting the Regularization coefficient alpha to lie close to 0 makes the Lasso mimic Linear Regression with no regularization. Here's the ...


18

You are running into that error because your X and Y don't have the same length (which is what train_test_split requires), i.e., X.shape[0] != Y.shape[0]. Given your current code: >>> X.shape (1, 6, 29) >>> Y.shape (29,) To fix this error: Remove the extra list from inside of np.array() when defining X or remove the extra dimension ...


17

If you are using SKlearn, you can use their hyper-parameter optimization tools. For example, you can use: GridSearchCV RandomizedSearchCV If you use GridSearchCV, you can do the following: 1) Choose your classifier from sklearn.neural_network import MLPClassifier mlp = MLPClassifier(max_iter=100) 2) Define a hyper-parameter space to search. (All the ...


16

Scipy's entropy function will calculate KL divergence if feed two vectors p and q, each representing a probability distribution. If the two vectors aren't pdfs, it will normalize then first. Mutual information is related to, but not the same as KL Divergence. "This weighted mutual information is a form of weighted KL-Divergence, which is known to take ...


15

My strong opinion regarding automated tasks like imputation (but, here I can include also scaling, centering, feature selection, etc) is to avoid in any way do such things without carefully inspecting your data. Of course, after deciding what kind of imputation to apply it can be automated (under the assumption that the new data has the same shape/problems)...


15

The following explanation is based on fit_transform of Imputer class, but the idea is the same for fit_transform of other scikit_learn classes like MinMaxScaler. transform replaces the missing values with a number. By default this number is the means of columns of some data that you choose. Consider the following example: imp = Imputer() # calculating the ...


14

You can simply use the feature_importances_ attribute to select the features with the highest importance score. So for example you could use the following function to select the K best features according to importance. def selectKImportance(model, X, k=5): return X[:,model.feature_importances_.argsort()[::-1][:k]] Or if you're using a pipeline the ...


14

In general, the performance of classifiers are compared using accuracy, this is a measure of the number of correctly classified instances divided by the total number of instances. However, from the training data we can get a better approximation of the expected error from our classifier when we are using ensemble learning or bagging techniques. Out-of-bag ...


14

Sklearn's LabelEncoder module finds all classes and assigns each a numeric id starting from 0. This means that whatever your class representations are in the original data set, you now have a simple consistent way to represent each. It doesn't do one-hot encoding, although as you correctly identify, it is pretty close, and you can use those ids to quickly ...


13

Check the Stanford NLP Group's open source software (http://www-nlp.stanford.edu/software), in particular, Stanford Classifier (http://www-nlp.stanford.edu/software/classifier.shtml). The software is written in Java, which will likely delight you, but also has bindings for some other languages. Note, the licensing - if you plan to use their code in ...


13

You must look at this Multicore implementation of t-SNE. I actually tried it and can vouch for its superior performance.


Only top voted, non community-wiki answers of a minimum length are eligible