Questions tagged [scikit-learn]

Scikit-learn is a Python module comprising of simple and efficient tool for machine learning, data mining and data analysis. It is built on NumPy, SciPy, and matplotlib. It is distributed under the 3-Clause BSD license.

Filter by
Sorted by
Tagged with
0
votes
2answers
52 views
+50

Error in using sklearn's GridSearchCV on Word2Vec

I am using the sklearn_api of gensim to create an estimator for a Word2vec model to pass it to sklearn's gridsearch . My code is as follows : ...
0
votes
0answers
8 views

Dropping one category for regularized linear models

while reviewing the sklearn's OneHotEncoder documentation (Attached below) I noticed that when applying regularization (e.g., lasso, ridge, etc.) it is not recommended to drop the first category. ...
1
vote
1answer
57 views

Linear Regression not working due to wrong kind of array

I try to deal with my homework. The Job is to take this data and perform a linear regression on it. The code is published here. I am quite new to programming in Python and in data science. So I tried ...
1
vote
1answer
20 views

Got some troubles with using OneHotEncoder to multiple categories

I'm trying to get the final pipeline on the titanic dataset(Example was taken from the 'Hands-on ML' book). ...
0
votes
0answers
17 views

How to classify a new email as spam/not spam?

I am working on a small exercise for determining if an email is spam or not. My dataset is the following: ...
0
votes
0answers
19 views

DBSCAN - Best way to find the Eps and MinPts for geospatial data (coordinates)

Question: The best way to find out the Eps and MinPts parameters for DBSCAN algorithm? Problem: The goal is to find the locations (clusters) based on coordinates (input data). The algorithm calculates ...
0
votes
1answer
17 views

Solution for TF-IDF Vectorization in Angular project?

While making an Angular project to use my text-classification model on unseen data, i struggle in finding a way how to transform text to TFIDF features. Anyone faced same issue? Maybe a solution on ...
5
votes
3answers
4k views

What cost function and penalty are suitable for imbalanced datasets?

For an imbalanced data set, is it better to choose an L1 or L2 regularization? Is there a cost function more suitable for imbalanced datasets to improve the model score (...
0
votes
1answer
26 views

Cross-Validation in Anomaly Detection with Labelled Data

I am working on a project where I train anomaly detection algorithms Isolation Forest and Auto-Encoder. My data is labelled so I have the ground truth but the nature of the problem requires ...
0
votes
1answer
33 views

Sklearn SVM question classification

So, I have found that there are many ways to classify words with sklearn's SVM algorithm. But I want to classify questions by taxonomy, as shown in the following dataset: The goal of this task is to ...
1
vote
0answers
7 views

General practices for building an incremental learning model which never forgets?

I'm new to datascience and appreciate your sage advice! I need to build an incremental learning model, and I know there's a lot that goes into something like that, but I'd like to highlight the most ...
1
vote
2answers
35 views

How to extract true positives data (complete row with data) after training and testing from test dataset?

How do you extract true positive data from testing data after training and testing? For example, in the test data, I have two rows and one row is true positives and the other is false negatives. ...
0
votes
1answer
123 views

KDE on TF-IDF - sensitive bandwidth

I'm clustering texts based on TF-IDF features and DBSCAN (density based), and trying to rank points based on their 'belonging' to the cluster. Since my clustering is density based and my points can ...
0
votes
2answers
43 views

How can access to modify feature_importances of Random Forest Classifier model?

My goal is to extract the feature importances from already trained random forest classifier and transfer them to another classifier. How this can be done? and How can access to modify ...
1
vote
1answer
83 views

Interpreting Machine Learning Classification Metrics

I'm trying to understand the results of a classifier I used to predict two possible classes. Here is what I get: ...
182
votes
9answers
227k views

What's the difference between fit and fit_transform in scikit-learn models?

I'm a newbie to data science, and I do not understand the difference between the fit and fit_transform methods in scikit-learn. ...
0
votes
1answer
29 views

How fit_transform, transform and TfidfVectorizer works

I'm a machine learning beginner and I tried to use the cosine similarity on fuzzy matching purpose. In the following example I want to compare 'data_dirty' with 'data_clean' : When I have to ...
1
vote
1answer
2k views

scikit-learn classifier reset in loop

I'm trying to evaluate classifiers comparison by running the sample script that can be found here. What I noticed is that in some cases the classifier is not reset. In fact, duplicating some of those (...
0
votes
1answer
36 views

Split into test and train set before or after generating document-term matrix?

I'm working on simple machine learning problems and I trying to build a classifier that can differentiate between spam and non-spam SMS. I'm confused as to whether I need to generate the document-term ...
1
vote
2answers
243 views

How to interpret metrics of a model after scaling the data

I have a GradientBoostingRegressor from scikit-learn which I trained. Afterwards, I obviously would like to know how good the model is. So, on a non-scaled dataset ...
6
votes
1answer
1k views

Reproducing randomForest Proximity Matrix from R package in Python

I am trying to port this little piece of R code to python: ...
1
vote
1answer
28 views

ML Approach for Getting List of Observations with Similar Features (Discrete+Continuous)

I have a dataset with 19k observations. Each has approximately 448 features: - Text description turned into vectors of size 300 - 16 categorical variables represented numerically - The remainder ...
0
votes
0answers
17 views

Escaping from overfitting hell: introducing regularization vs increasing training data

I am trying to identify noisy intervals in geomagnetic data using logistic regression, working with scikit-learn. Here is a typical spectrum of the data that I am working with: In this example, the ...
0
votes
1answer
23 views

Problem with PCA [closed]

I am getting the message Input contains NaN, infinity or a value too large for dtype('float64') when I run the pca.fit(X_train) ...
2
votes
1answer
29 views

Multiple linear regression for multi-dimensional input and output?

Assume that I have $N$ points $x_i,i=1,...,N$ in some $A>1$-dimensional space $\mathbb{R}^A$ with pointwise evaluations of some function $f:\mathbb{R}^A \rightarrow \mathbb{R}^B$, i.e. $f(x_i),i=1,...
0
votes
3answers
6k views

Sklearn ValueError: X has 2 features per sample; expecting 11

I try to visualizing multiple logistic regression but I get the above error. I'm practicing on red wine quality data set from kaggle. Here is a full traceback: ...
0
votes
1answer
161 views

Guidance needed with dimension reduction for clustering - some numerical, lots of categorical data

I've my data in a Pandas df with 25.000 rows and 1.500 columns without any NaNs. Of the columns about 30 contain numerical data which I standardized with StandardScaler(). The rest are cols with ...
0
votes
0answers
10 views

How do you calculate the probability that a certain number of hyper-parameter combinations contains the optimum combination?

How do you calculate the probability that a certain number of hyper-parameter combinations contains the optimum combination? Summary I use SKLearn's RandomizedSearchCV module. It will test a certain ...
1
vote
1answer
25 views

Does mini-batch gradient descent nullify the effect of stratification on the training data set?

In data pre-processing, stratified shuffle is used to ensure that the distribution of the original dataset is reflected in the training, test and validation dataset. Mini-batch gradient descent uses ...
-2
votes
1answer
22 views

NotFittedError says this StandardScaler instance is not fitted yet while using inverse_transform() [closed]

I have a dataset and i have used Support Vector Regression.So i needed to use StandardScaler module from sklearn.preprocessing fro Feature Scaling. After training my model when i came to predict it ...
0
votes
1answer
17 views

How and where to set weights in case of imbalanced cost sensitive learning in machine learning?

I confront with a binary classification machine learning task which is both slightly imbalanced and cost sensitive. I wonder what (and where in the modeling pipeline, say, in sklearn) is the best way ...
2
votes
1answer
607 views

Online vs Batch Learning in Latent Dirichlet Allocation using Scikit Learn

Reference I'm looking at the LDA algorithm from Scikit Learn for topic modeling. Can someone tell me how the 'online' method of learning works vs the 'batch' method of learning? Also, what is learning ...
2
votes
4answers
466 views

Random Forests Feature Selection on Time Series Data

I have a dataset with N features, each one with 500 instances in time. For example, let's say that I have the following: Features ...
3
votes
1answer
594 views

Not able to interpret decision tree when using class_weights

I'm working with an imbalanced dataset. I'm using a decision tree (scikit-learn) to build a model. For explaining my problem I've taken iris dataset. When I'm setting ...
2
votes
2answers
362 views

Procedure for selecting optimal number of features with Python's Scikit-Learn

I have a dataset with 130 features (1000 rows) . I want to select the best features for my classifier. I started with RFE but Its taking too long, i done this: <...
0
votes
0answers
17 views

How to construct pipeline with different alternative transformations for different kind of features in Scikit-learn?

I try to construct a pipeline in sklearn where I do different (in some cases multiple) transformations on different kind (numeric/ordinal/binary nominal/non-binary non-ordinal nominal) features. An ...
2
votes
1answer
733 views

LightGBM vs Sklearn LightGBM- Mistake in Implementation- Exact same parameters giving different results

While passing the exact same parameters to LightGBM and sklearn's implementation of LightGBM, I am getting different results. Initially, I was getting the exact same results on doing this, however, I ...
1
vote
3answers
2k views

how to prepare data for cross validation in mnist dataset?

How to use k-fold cross validation for MNIST dataset? I read article documentation on sci-kit learn ,in that example they used the whole iris dataset for cross validation. ...
0
votes
1answer
15 views

Sklearn Random Feature Importances Identical for Predicting Different Response Variables

I have created four random forest models they have the same X data, but their y data are four different response variables. The sklearn random forest feature importance is identical for all four. All ...
4
votes
1answer
390 views

Can I do incremental learning with the sklearn implementation of Linear Discriminant Analysis

I have a large number of pictures that I would like to use LDA on. However, it requires too much memory, so I was wondering if it would be possible to make the learning incremental, using a sklearn ...
0
votes
1answer
477 views

How can I extract the residual array from the Scikit Learn PCA routine?

The Nipals PCA algorithm calculates the scores and loadings of a data array iteratively. If only (say) 3 scores and loadings are calculated from a data array with more than 3 variables, there is a ...
2
votes
1answer
59 views

Item-based recommender using K-NN

I'm trying to build an item-based recommender using k-nn. I have a list of items, all of which have some properties (features) in common. ...
14
votes
3answers
17k views

What is the difference between a hashing vectorizer and a tfidf vectorizer

I'm converting a corpus of text documents into word vectors for each document. I've tried this using a TfidfVectorizer and a HashingVectorizer I understand that a ...
1
vote
1answer
155 views

ValueError trying to use a pickled scikit-learn model

I am new to data science and trying to learn something. I was able to complete the prediction with 98% accuracy and i saved it as pickle model. Now while trying to predict using this model I am ...
23
votes
5answers
11k views

Improve the speed of t-sne implementation in python for huge data

I would like to do dimensionality reduction on nearly 1 million vectors each with 200 dimensions(doc2vec). I am using TSNE ...
2
votes
2answers
735 views

How can I find anomalies in each row of data?

I have some reported data I want to spot anomalies on. The columns are a facility name then monthly reports of that given facility. ...
0
votes
1answer
257 views
1
vote
1answer
19 views

Using sklearn knn imputation on a large dataset

I have a large dataset ~ 1 million rows by 400 features and I want to impute the missing values using sklearn KNNImputer. Trying this off the bat I hit memory problems, but I think I can solve this by ...
1
vote
0answers
29 views

Is it possible to do attribute, value extraction prediction model in Machine Learning?

Previously I asked a question at here, but it doesn't seems to be at the correct place. So, I moved it here with more details of what I've done. Here is the sample data image to be process. Start with ...
1
vote
1answer
82 views

Python sklearn PCA transform function output does not match

I am computing PCA on some data using 10 components and using 3 out of 10 as: ...

1
2 3 4 5
32