People who code: we want your input. Take the Survey

Questions tagged [scikit-learn]

scikit-learn is a popular machine learning package for Python that has simple and efficient tools for predictive data analysis. Topics include classification, regression, clustering, dimensionality reduction, model selection, and preprocessing.

Filter by
Sorted by
Tagged with
25
votes
3answers
88k views

How to get p-value and confident interval in LogisticRegression with sklearn?

I am building a multinomial logistic regression with sklearn (LogisticRegression). But after it finishes, how can I get a p-value and confident interval of my model? It only appears that sklearn only ...
4
votes
2answers
6k views

NLTK Sklearn Genism Text to Topic

I aint no data scientist/machine learner. What Im Lookin for ...
2
votes
1answer
2k views

How can I run SVM on 500k rows with 81 columns?

I have about 500k rows of data. I'm new to data science and I'm trying to train a model utilizing support vector machines as part of my analysis. On my little macbook pro, it seems endless. Right now ...
0
votes
1answer
153 views

gold price detection using data mining

I have dataset of gold prices and after modifying and some preprocessing i ended up with dataframe below: There is 50,000 record in dataset and all columns expect ...
1
vote
0answers
200 views

Large/Medium scale text classification with scikit-learn and beyond

I'm trying to make text multilabel classification ~40 labels from products description. Labels are unbalanced. There are ~3 labels per sample. And I have ~250k samples. I digged Kaggle's text ...
1
vote
1answer
2k views

Image feature extraction Python skimage blob_dog

I am trying to extract features from images using: ...
124
votes
12answers
225k views

Train/Test/Validation Set Splitting in Sklearn

How could I split randomly a data matrix and the corresponding label vector into a X_train, X_test, X_val, y_train, y_test, y_val with Sklearn? As far as I know, ...
2
votes
0answers
497 views

How to evaluate the quality of representation for variables and individuals of a PCA in scikit-learn?

I just looked at the PCA in scikit-learn, but I didn't find a way to evaluate the quality of representation for variables and individuals like I usually do using the squared cosine. The squared ...
0
votes
1answer
3k views

Orange 3 - Feature selection / importance

I am using (and loving) Orange 3 for some projects at my school and have a question: When using Python and e.g. doing a RandomForest Classification, I can easily access the feature importances by ...
1
vote
0answers
732 views

GridSearchCV with SVM estimator AUC score not reproduced on SVM run

I've run GridSearchCV to determine 'best parameters' for a linear SVM, and then passed these [in a dictionary along with non-tuned parameters] into a new SVM. I've ...
4
votes
3answers
1k views

Feature Selection for K Nearest Neighbour and Decision Trees

I have 2 digits numbers and 9 features. I must pick 2 features, so decided to plot the features against each other to see whether I can get any insight on the best features to train my algorithm. ...
0
votes
2answers
504 views

Contrasting logistic regression vs decision tree performance in specific example

I have a set of 10,000 integers, and another set of 100. The integers in the first set are mapped to integers in the second set according to some rules (not mathematical rules, think of these values ...
11
votes
1answer
14k views

Multiple Categorical values for a single feature how to convert them to binary using python

I have a data set of movies which has 28 columns. One of them is genres. For each row in this data set, the value for column genres is of the form "Action|Animation|Comedy|Family|Fantasy". I want to ...
1
vote
1answer
3k views

Output data from scikit learn logistic regression

I have run a logistic regression using scikit learn in python. I know want to output the results to put into a csv and then load into Tableau. To do that I need to combine the y_test, y_actual, and ...
0
votes
1answer
456 views

Adaboost for 3D Input data

I want to do a text classification problem for which I want to train use Adaboost Classifier from sklearn using a Keras estimator. I do know how to use the Keras wrappers for using sklearn functions. ...
1
vote
1answer
16k views

Python TypeError: __init__() got an unexpected keyword argument 'decision_function_shape' [closed]

I tried creating a SVM Classifier, as: ...
3
votes
1answer
215 views

Ensemble Techniques for multilabel data

I observed that Adaboost or Bagging ensemble classifiers present in sklearn only work for single label training data. How do I use these for multilabel data?
0
votes
1answer
113 views

Sample selection through clustering

I have a biased set of samples going into a binary classification sklearn pipeline, white and black samples. It is easy for me to fetch as many black samples as required, while whites are a bit ...
1
vote
2answers
1k views

keras validation mean squared error always similar to 1

is there any reason why the validation mean squared error output from Keras is always very similar to 1? Thank you. All of my training results looks like: ...
1
vote
1answer
1k views

How to get the inertia at the begining when using sklearn.cluster.KMeans and MiniBatchKMeans

When I cluster a lot of data, it is hard to run KMeans and wait it stop until centers has not change, so I have to stop KMeans when it reach maximum number of iterations. Here come problem: how can I ...
3
votes
1answer
914 views

What preprocessing steps to be followed before image comparison?

1 down vote favorite For example I am trying to find the similarity between two images using skimage - SSIM. The code block will be as follows ...
1
vote
0answers
380 views

Sklearn Random Forest Prediction Correlation Issue

I'm having issues with fitting a Random Forest model to a completely new dataset. I'm trying to predict tenancy lengths for current tenants. I have a dataset with tenancy information since 2008, with ...
1
vote
1answer
3k views

Computing confidence interval of average output from random forest in scikit learn

I am computing confidence intervals for random forests using the package available here: http://contrib.scikit-learn.org/forest-confidence-interval/auto_examples/plot_mpg.html#sphx-glr-auto-examples-...
5
votes
2answers
12k views

Xgboost predict probabilities

When using the python / sklearn API of xgboost are the probabilities obtained via the predict_proba method "real probabilities" or do I have to use ...
-2
votes
1answer
4k views

clustering on multiple features and applying k-means

I am completely new to data science and this is a homework assignment so apologies beforehand. I have some raw data on restaurants that contain their categories (e.g. "Pizza", "Italian") and their ...
2
votes
0answers
3k views

Grid Search on Unsupervised Sklearn Clustering?

I am trying to use clustering algorithms in sklearn and am using Silhouette score with cosine similarity as a metric to compare different algorithms. My question is due to the varying hyperparameters ...
1
vote
1answer
246 views

Regression in Predicting Tenancy Lengths

I'm currently working on a project involving the prediction of tenancy lengths. I've so far managed to get to a point where I've processed the data and pruned my Random Forest model (via sklearn in ...
1
vote
2answers
240 views

How to interpret silouette coefficient?

I'm trying to determine number of clusters for k-means using sklearn.metrics.silhouette_score. I have computed it for range(2,50)...
4
votes
0answers
11k views

Tuning Gradient Boosted Classifier's hyperparametrs and balancing it

I am not sure if it is a correct stack. Maybe I should have put my question into crossvalidated. Nevertheless, I perform following steps to tune the hyperparameters for a gradient boosting model: ...
0
votes
1answer
811 views

Quick start using python and sklearn kmeans?

I started tinkering with sklearn kmeans last night out of curiosity with the goal of clustering users into groups to see what kind of user groups I can derive. I am lost when it comes to plotting the ...
1
vote
1answer
2k views

Silhouette calculation in k-means

I am trying to compute Silhouette with k-means. However I have the value really close to 0 and the clusters are very clearly separated. Do you know where can be the problem? This is the code: ...
-2
votes
2answers
1k views

Predictive Model with sort-of-non-binary labels

I'm working with government infrastructure public tenders data and want to build a predictive model. I'm want to train a model to predict if a company will or will not participate in a public tender. ...
1
vote
1answer
3k views

Feature selection with L1 regularization on sklearn's LogisticRegression

I'm using sklearn's LogisticRegression with penaly=l1 (lasso regularization, as opposed to ridge regularization l2). Lasso is causing the optimization function to ...
15
votes
3answers
4k views

Predict the best time of call

I have a dataset including a set of customers in different cities of California, time of calling for each customer, and the status of call (True if customer answers the call and False if customer does ...
6
votes
2answers
5k views

Naive Bayes: Divide by Zero error

OK this is my first time in ML and for starter I am implementing Naive Bayes. I have Cricket(sports) data in which I have to check whether the team will win or lost based on Toss Won|Lost and Bat ...
3
votes
5answers
2k views

Predictive modeling on big data set that can't fit into memory

I am trying to build a Decision-Tree model on top of a dataset that is about 10G in size on my local computer. However, I only have 8G memory. What I am doing now is just random sampling certain ...
12
votes
3answers
26k views

Mass convert categorical columns in Pandas (not one-hot encoding)

I have pandas dataframe with tons of categorical columns, which I am planning to use in decision tree with scikit-learn. I need to convert them to numerical values (not one hot vectors). I can do it ...
1
vote
2answers
6k views

sklearn.cross_validation.cross_val_score “cv” parameter question

I was working through a tutorial on the titanic disaster from Kaggle and I'm getting different results depending on the details of how I use ...
0
votes
3answers
6k views

How to deal with a skewed data-set having all the samples almost similar?

I have a very large skewed training set where every feature's data-points are very similar ? For example, following is some ...
-3
votes
1answer
1k views

I need to measure Performance : AUC for this code of NLTK and skLearn [closed]

The code below measures precision and recall and F-measure (source). How can I measure AUC? ...
0
votes
1answer
3k views

Using scikit-learn FeatureHasher

I have a huge data set with one of the columns named 'mail_id'. The mail_id is given in a very creepy format as shown below: ...
0
votes
2answers
2k views

Fitting error with Neural Network Grid Search in Keras

I try to build a NN classifier on the well-known MNIST image database with Sklearn's Grid Search according the following: ...
2
votes
1answer
159 views

Encoding features in sklearn

Suppose I have a dataset of size(10000, 45). One of the features in the dataset is activity_type in which the values vary from 1 to 15 as shown below: ...
7
votes
1answer
2k views

Naive Bayes Should generate prediction given missing features (scikit learn)

Seeing that Naive Bayes uses probability to make a prediction, and treats features as being conditionally independent of each other, then it makes sense that the model can still make a prediction ...
0
votes
2answers
343 views

Splitting data in scikit-learn

I know how to split the dataset into train and test sets using train_test_split but is there any way that I can split the dataset into three different sets, i.e., &...
2
votes
2answers
19k views

Problems with accuracy.score sklearn

I am learning python and trying myself out at mashine learning. I am reproducing a super simple example - based on the infamous iris dataset. Here it goes: ...
0
votes
1answer
397 views

Kmeans: Between class intertia

I'm using KMeans of scikit-learn for clustering of forums. The attribute inertia_ of model gives the within class inertia but how can i get the between class inertia? In other word how can I verify ...
2
votes
1answer
593 views

Can Gaussian Process be fit incrementally?

I am using Gaussian Process Regressor to fit data for a Bayesian Optimiser. This is a relevant part of my Python code. ...
3
votes
2answers
7k views

First steps with Python and scikit-learn

I believe I have a simple if not trivial question. I have a background in statistics and I tend to use Stata and R quite a bit. I am interested in learning Python. I used it for a while now and ...
1
vote
1answer
180 views

Python: how to handle categorial values in dataset to build models

I have a training dataframe dfTrain and the output of dfTrain.head() is shown below: ...