Questions tagged [scikit-learn]

Scikit-learn is a Python module comprising of simple and efficient tool for machine learning, data mining and data analysis. It is built on NumPy, SciPy, and matplotlib. It is distributed under the 3-Clause BSD license.

Filter by
Sorted by
Tagged with
137
votes
8answers
180k views

Difference between fit and fit_transform in scikit_learn models?

I am newbie to data science and I do not understand the difference between fit and fit_transform methods in scikit-learn. Can ...
107
votes
3answers
65k views

When to use One Hot Encoding vs LabelEncoder vs DictVectorizor?

I have been building models with categorical data for a while now and when in this situation I basically default to using scikit-learn's LabelEncoder function to transform this data prior to building ...
82
votes
11answers
82k views

SVM using scikit learn runs endlessly and never completes execution

I am trying to run SVR using scikit learn ( python ) on a training dataset having 595605 rows and 5 columns(features) and test dataset having 397070 rows. The data has been pre-processed and ...
68
votes
9answers
119k views

Train/Test/Validation Set Splitting in Sklearn

How could I split randomly a data matrix and the corresponding label vector into a X_train, X_test, X_val, y_train, y_test, y_val with Sklearn? As far as I know, ...
66
votes
6answers
90k views

strings as features in decision tree/random forest

I am doing some problems on an application of decision tree/random forest. I am trying to fit a problem which has numbers as well as strings (such as country name) as features. Now the library, scikit-...
39
votes
5answers
65k views

Does scikit-learn have forward selection/stepwise regression algorithm?

I'm working on the problem with too many features and training my models takes way too long. I implemented forward selection algorithm to choose features. However, I was wondering does scikit-learn ...
30
votes
3answers
21k views

How to force weights to be non-negative in Linear regression

I am using a standard linear regression using scikit-learn in python. However, I would like to force the weights to be all positive for every feature (not negative), is there any way I can accomplish ...
29
votes
3answers
81k views

Understanding predict_proba from MultiOutputClassifier

I'm following this example on the scikit-learn website to perform a multioutput classification with a Random Forest model. ...
29
votes
1answer
22k views

Why is xgboost so much faster than sklearn GradientBoostingClassifier?

I'm trying to train a gradient boosting model over 50k examples with 100 numeric features. XGBClassifier handles 500 trees within 43 seconds on my machine, while <...
25
votes
5answers
40k views

Calculating KL Divergence in Python

I am rather new to this and can't say I have a complete understanding of the theoretical concepts behind this. I am trying to calculate the KL Divergence between several lists of points in Python. I ...
24
votes
4answers
34k views

Scikit-learn: Getting SGDClassifier to predict as well as a Logistic Regression

A way to train a Logistic Regression is by using stochastic gradient descent, which scikit-learn offers an interface to. What I would like to do is take a scikit-learn's SGDClassifier and have it ...
24
votes
2answers
105k views

train_test_split() error: Found input variables with inconsistent numbers of samples

Fairly new to Python but building out my first RF model based on some classification data. I've converted all of the labels into int64 numerical data and loaded into X and Y as a numpy array, but I am ...
21
votes
2answers
35k views

Can you explain the difference between SVC and LinearSVC in scikit-learn?

I've recently started learning to work with sklearn and have just come across this peculiar result. I used the digits dataset ...
21
votes
3answers
15k views

StandardScaler before and after splitting data

When I was reading about using StandardScaler, most of the recommendations were saying that you should use StandardScaler before ...
20
votes
4answers
9k views

Improve the speed of t-sne implementation in python for huge data

I would like to do dimensionality reduction on nearly 1 million vectors each with 200 dimensions(doc2vec). I am using TSNE ...
19
votes
2answers
60k views

How to get p-value and confident interval in LogisticRegression with sklearn?

I am building a multinomial logistic regression with sklearn (LogisticRegression). But after it finishes, how can I get a p-value and confident interval of my model? It only appears that sklearn only ...
18
votes
3answers
4k views

Difference between OrdinalEncoder and LabelEncoder

I was going through the official documentation of scikit-learn learn after going through a book on ML and came across the following thing: In the Documentation it is given about ...
18
votes
1answer
23k views

How to deal with string labels in multi-class classification with keras?

I am newbie on machine learning and keras and now working a multi-class image classification problem using keras. The input is tagged image. After some pre-processing, the training data is represented ...
17
votes
1answer
12k views

Algorithms for text clustering

I have a problem of clustering huge amount of sentences into groups by their meanings. This is similar to a problem when you have lots of sentences and want to group them by their meanings. What ...
17
votes
5answers
4k views

Merging sparse and dense data in machine learning to improve the performance

I have sparse features which are predictive, also I have some dense features which are also predictive. I need to combine these features together to improve the overall performance of the classifier. ...
16
votes
6answers
7k views

What is the reason behind taking log transformation of few continuous variables?

I have been doing a classification problem and I have read many people's code and tutorials. One thing I've noticed is that many people take np.log or ...
16
votes
2answers
938 views

Where in the workflow should we deal with missing data?

I'm building a workflow for creating machine learning models (in my case, using Python's pandas and sklearn packages) from data ...
15
votes
5answers
51k views

Sentence similarity prediction

I'm looking to solve the following problem: I have a set of sentences as my dataset, and I want to be able to type a new sentence, and find the sentence that the new one is the most similar to in the ...
15
votes
1answer
10k views

RandomForestClassifier OOB scoring method

Is the random forest implementation in scikit-learn using mean accuracy as its scoring method to estimate generalization error with out-of-bag samples? This is not mentioned in the documentation, but ...
14
votes
2answers
48k views

How does SelectKBest work?

I am looking at this tutorial: https://www.dataquest.io/mission/75/improving-your-submission At section 8, finding the best features, it shows the following code. ...
12
votes
3answers
21k views

Pandas Dataframe to DMatrix

I am trying to run xgboost in scikit learn. And I only use Pandas to load data into dataframe. How am i supposed to use pandas df with xgboost. I am confused by the DMatrix routine required to run ...
12
votes
3answers
21k views

Mass convert categorical columns in Pandas (not one-hot encoding)

I have pandas dataframe with tons of categorical columns, which I am planning to use in decision tree with scikit-learn. I need to convert them to numerical values (not one hot vectors). I can do it ...
12
votes
3answers
39k views

How can I check the correlation between features and target variable?

I am trying to build a Regression model and I am looking for a way to check whether there's any correlation between features and target variables? This is my ...
12
votes
1answer
15k views

Feature selection using feature importances in random forests with scikit-learn

I have plotted the feature importances in random forests with scikit-learn. In order to improve the prediction using random forests, how can I use the plot information to remove features? I.e. how to ...
12
votes
1answer
3k views

Feature importance with high-cardinality categorical features for regression (numerical depdendent variable)

I was trying to use feature importances from Random Forests to perform some empirical feature selection for a regression problem where all the features are categorical and a lot of them have many ...
11
votes
2answers
14k views

What is the difference between a hashing vectorizer and a tfidf vectorizer

I'm converting a corpus of text documents into word vectors for each document. I've tried this using a TfidfVectorizer and a HashingVectorizer I understand that a ...
11
votes
2answers
13k views

Does scikit-learn use regularization by default?

I just fitted a logistic curve to some fake data. I made the data essentially a step function. data = -------------++++++++++++++ But when I look at the fitted ...
11
votes
2answers
7k views

When to use Standard Scaler and when Normalizer?

I understand what Standard Scalar does and what Normalizer does, per the scikit documentation: Normalizer, Standard Scaler. I know when Standard Scaler is applied. But in which scenario is Normalizer ...
11
votes
2answers
2k views

Efficient dimensionality reduction for large dataset

I have a dataset with ~1M rows and ~500K sparse features. I want to reduce the dimensionality to somewhere in the order of 1K-5K dense features. ...
10
votes
3answers
3k views

Predict the best time of call

I have a dataset including a set of customers in different cities of California, time of calling for each customer, and the status of call (True if customer answers the call and False if customer does ...
10
votes
1answer
669 views

How to use Scikit-Learn Label Propagation on graph structured data?

As part of my research, I am interested in performing label propagation on a graph. I am especially interested in those two methods: Xiaojin Zhu and Zoubin Ghahramani. Learning from labeled and ...
9
votes
3answers
2k views

How to encode a class with 24,000 categories?

I'm currently working on a logistic regression model for genomics. One of the input fields I want to include as a covariate is genes. There are around 24,000 known ...
9
votes
2answers
13k views

How to calculate the fold number (k-fold) in cross validation?

I am confused about how I choose the number of folds (in k-fold CV) when I apply cross validation to check the model. Is it dependent on data size or other parameters?
9
votes
2answers
8k views

How to use Cohen's Kappa as the evaluation metric in GridSearchCV in Scikit Learn?

I have class imbalance in the ratio 1:15 i.e. very low event rate. So to select tuning parameters of GBM in scikit learn I want to use Kappa instead of F1 score. My understanding is Kappa is a better ...
9
votes
4answers
18k views

Clustering for mixed numeric and nominal discrete data

My data includes survey responses that are binary (numeric) and nominal / categorical. All responses are discrete and at individuals level. Data is of shape (n=7219, p=105). Couple things: I am ...
9
votes
1answer
28k views

How to do stepwise regression using sklearn? [duplicate]

I could not find a way to stepwise regression in scikit learn. I have checked all other posts on Stack Exchange on this topic. Answers to all of them suggests using f_regression. But f_regression ...
9
votes
2answers
2k views

Is max_depth in scikit the equivalent of pruning in decision trees?

I was analyzing the classifier created using a decision tree. There is a tuning parameter called max_depth in scikit's decision tree. Is this equivalent of pruning a decision tree? If not, how could I ...
9
votes
2answers
6k views

How to train model to predict events 30 minutes prior, from multi-dimensionnal timeseries

Experts in my field are capable of predicting the likelyhood an event (binary spike in yellow) 30 minutes before it occurs. Frequency here is 1 sec, this view represents a few hours worth of data, i ...
9
votes
1answer
6k views

What is the difference between cross_validate and cross_val_score?

I understand cross_validate and how it works, but now I am confused about what cross_val_score actually does. Can anyone give me some example?
9
votes
4answers
13k views

Interpreting Decision Tree in context of feature importances

I'm trying to understand how to fully understand the decision process of a decision tree classification model built with sklearn. The 2 main aspect I'm looking at are a graphviz representation of the ...
9
votes
2answers
875 views

What is the most efficient method for hyperparameter optimization in scikit-learn?

An overview of the hyperparameter optimization process in scikit-learn is here. Exhaustive grid search will find the optimal set of hyperparameters for a model. The downside is that exhaustive grid ...
9
votes
1answer
1k views

Imbalanced data causing mis-classification on multiclass dataset

I am working on text classification where I have 39 categories/classes and 8.5 million records. (In future data and categories will increase). Structure or format of my data is as follows. ...
9
votes
2answers
849 views

Is there a method that is opposite of dimensionality reduction?

I am new to the field of machine learning, but have done my share of signal processing. Please let me know if this question has been mislabeled. I have two dimensional data which is defined by at ...
9
votes
1answer
2k views

Can training label confidence be used to improve prediction accuracy?

I have training data that is labelled with binary values. I also have collected the confidence of each of these labels i.e. 0.8 confidence would mean that 80% of the human labellers agree on that ...
9
votes
3answers
4k views

Building a machine learning model to predict crop yields based on environmental data

I have a dataset containing data on temperature, precipitation and soybean yields for a farm for 10 years (2005 - 2014). I would like to predict yields for 2015 based on this data. Please note that ...

1 2 3 4 5 28