Questions tagged [scikit-learn]

scikit-learn is a popular machine learning package for Python that has simple and efficient tools for predictive data analysis. Topics include classification, regression, clustering, dimensionality reduction, model selection, and preprocessing.

Filter by
Sorted by
Tagged with
4
votes
2answers
256 views

Anomaly detection using clustering of highly correlated Categorical data

My data has two columns and both are highly correlated e.g. if column1 has value ABC, column2 should be XYZ i.e. ABC-->XYZ. If column2 has anything else it's Anomaly. Likewise, there are thousands ...
0
votes
0answers
14 views

Is there a quicker solution to Sklearn MAE?

I am attempting to run RandomForestRegressor on this fairly large dataset: ...
1
vote
1answer
7 views

In scikit-learn's LDA implementation, how can I sort the topics by frequency over the entire corpus?

I've used scikit-learn to perform LDA topic modeling, and I'd ultimately like to sort the topics by saliency/frequency over the entire corpus, but I'm unsure how to do as such. I've used pyldavis ...
0
votes
0answers
14 views

inbuilt python module for regression of multivariate

I am working on the following problem: In linear regression, I have used the python sklearn.linear_model LinearRegression by calling ...
0
votes
0answers
21 views

How to use Random Forest to predict probability?

I was using predict_proba() in Random Forest classifier of sklearn. It turn out that for all input that I fed to the model, the probability for every class are the same. Then I read about this in the ...
2
votes
1answer
9 views

I am curious about the interpretation of the elastic Net coefficient

I want to discover the importance of variables in data through sklearn's Elactic Net. But I don't understand the exact meaning of coefficient. When training, I used alpha: 0.01585598, l1_ratio: 1.000. ...
2
votes
1answer
19 views

Random forest regression model improvement

I am working with vehicle occupancy prediction and I am very much new to this, I have used random forest regression to predict the occupancy values. Random forest jupyter notebook have around 48 M ...
0
votes
2answers
2k views

What does the classification report interpret? Class 1 indicates abnormal data

How to interpret the report and How is precision, recall values are calculated for individual class labels. What is the significance of macro avg ? Does this report signify a good predictions by the ...
0
votes
0answers
10 views
4
votes
2answers
1k views

How to deal with missing data for Bernoulli Naive Bayes?

I am dealing with a dataset of categorical data that looks like this: ...
1
vote
1answer
2k views

Error with pandas dataframe (needs to be 1-dimensional)

I am trying to determine the conformal predictions for my model with my data. But it gives me following error that occurs at icp.calibrate(X_cal, y_cal) : ...
1
vote
1answer
19 views

Getting both results and probabilities running scikit learn random forest

I have a scikit learn RandomForestClassifier that returns 0s and 1s: ...
0
votes
1answer
680 views

Low memory error while performing degree 2 polynomial regression on (3000*1835) sized array

I am working on a problem to predict the revenue, a film will generate. Some of the features available in the data set are json collection for the crew, cast which worked in the film. I applied ...
0
votes
1answer
46 views

Is there a clustering algorithm that can cluster time series dataset based on variation ratio (or quantity)?

I am learning machine learning from scikit-learn and reading its docs. Clustering clusters groups based on the Euclidean distance and filters them by different ways ex: Gaussian distribution, or mean-...
1
vote
0answers
129 views

matching results with sklearn average_precision_score

I have written a (convenience) class for binary classification in python which takes several models and compares their ROCs, AvPrecisions, and creates several charts (distribution of scores, accuracy/...
0
votes
1answer
12 views

How to interprete the feature significance and the evaluation metrics in classification predictive model?

Consider a experiment to predict the Google-Play apps rating using a Random-Forest classifier with scikit-learn in Python. Three attributes 'Free', 'Size' and 'Category' are utilized to predict the ...
2
votes
1answer
33 views

Sensorfusion: Generate virtual sensor based on analysis of sensorsdata

I have a steam engine which is equipped with the following sensors: temperature sensor in the boiler room temperature sensor in the heating room pressure sensor in the boiler room rotations-per-...
0
votes
1answer
20 views

Scikit-learn estimator not changing predictions when random_state variable changes

I am trying to compute prediction intervals for a classifier I trained in scikit-learn. Even after setting a new random_state parameter in my pipeline, this does ...
0
votes
0answers
15 views

Using multiple TF-IDF to create a feature

I have around 200k comments and I extracted the top 200 words (without stop words) out of their content. Each comment is linked to a specific date. I would like to ask a very stupid question: Is it ...
2
votes
1answer
35 views

Possible to use predict_proba without normalizing to 1?

I'm using xgboost multi-class classifier to predict a collection of things likely to fail. I want to run that prediction, and report anything that the classifier ...
1
vote
1answer
263 views

Which machine learning technique to predict student passing based on standardized tests?

I have data on various standardized tests that certain students took and whether or not they ended up passing a specific class in college. Using the pass/fail as the dependent variable and the ...
2
votes
2answers
290 views

Techniques for Cluster Analysis of a Very Large (n=140000) Binary Dataset in Python?

In essence: what techniques in Python are possible to find clusters/trends in a very large categorical dataset? My very large dataset (140000 rows/observations, 80 variables) of categorical data has ...
0
votes
3answers
60 views

How to find the driver features towards a particular result in Classification problems

In a classifier model, we can predict the outcome class, but here I need to find out the features that drive towards a particular result in a classification problem, that are a strong indicator of a ...
1
vote
1answer
252 views
0
votes
1answer
98 views

How to save multi-output predicted masks into two different folders after using model.predict_generator

I have a multi output segmentation task, the training process went well, but when Im trying to get the prediction I found difficulties to separate the two output into two different folders, in my ...
2
votes
1answer
54 views

Get how similarity between the training data and the income data?

I'am trying to use Clustering and Classification methods as SVM using scikitlearn. I'm also studying some outliers/novelty detections I want something like a semi-supervised model. I want to predict ...
2
votes
3answers
62 views

Overfitting with sklearn pipeline - reasons why?

So.... I've been playing around with this for FAR TOOO LONG now and I really need some advice. Most people on kaggle concat training and testing set TOGETHER and then pre scale the data, this seems ...
2
votes
1answer
59 views

what does the standard deviation plot around my learning curve indicate?

I plotted a learning curve below. There is a thick red band around the top portion of my training score. Why is it so high at the beginning? Below is a snippet of the code used: ...
0
votes
3answers
3k views

GridSearchCV with Random Forest Classifier

I'm working with a supervised learning problem and trying to predict a binary label and using a Random Forest to do so. I'm trying to tune my hyper-parameters to give me a best model based on my data. ...
1
vote
0answers
14 views

Clustering dataset with and without estimating means (no EM algorithm)

Given a dataset $D$ of the form $$ D = \{ (x_0,y_0), (x_1,y_1),\ldots,(x_{n},y_n) $$ sampled from a Gaussian mixture model with identity covariance matrices, I want to understand what are my options ...
3
votes
2answers
5k views

Imputation of missing values and dealing with categorical values

I have a dataset (10 million rows, 55 columns) with many missing values. I need to predict those values somehow using other non-missing values, i.e. replace them with something that is not NaN. Mean ...
0
votes
2answers
752 views

Using a pipeline and transforming data with imputing and OneHotEncoding performs worse than get_dummies

I'm still in the process of learning, so I'm sorry if this doesn't make much sense. I'm doing Kaggle learns micro courses, and to work with missing tabular data we learned about using pipelines with ...
1
vote
0answers
17 views

How do the authors get this updating formula for all $\beta$ in $\beta$-divergence

I'm reading the paper Algorithms for nonnegative matrix factorization with the β-divergence by Cédric Févotte and Jérôme Idier. Package scikit-learn uses their algorithm for module sklearn....
0
votes
1answer
67 views

GridSearch on imbalanced datasets

Im trying to use gridsearch to find the best parameter for my model. Knowing that I have to implement nearmiss undersampling method while doing cross validation, should I fit my gridsearch on my ...
1
vote
1answer
550 views

Giving more weight to a particular feature in scikit-learn decision trees

I have a model that I train on same data, but i want a feature to have a stronger weight. Say I have three features: Car manufacturer's name Price Top speed ...
1
vote
1answer
65 views

Is it a good practice to evaluate model performance by comparing the metrics of rescaled (inverse transformed) predictions and true target values?

I am now working with a Linear Regression for a time-series regression problem (I am sorry that I cannot say too much about the problem and feature vector due to NDA). I scaled both the input values ...
1
vote
2answers
67 views

How can I use multiple features in basic sentiment analysis in scikit-learn?

I've tried to reduce the problem to it's absolute basics. Assume I have data (csv) as such: ...
1
vote
2answers
255 views

Sklearn LocalOutlierFactor contamination parameter

Can anyone provide an intuitive explanation of the choice of contamination parameter used in sklearn's LocalOutlierFactor implementation when ...
0
votes
1answer
198 views

How do feature selection on a sparse matrix?

Say I want to do features selection on a sparse matrix, i.e., 10,000 rows x 1500 features, but the matrix is mostly sparse. Let's say the features are all numeric and the target is binary and discrete....
0
votes
1answer
33 views

Can we calculate AUC for deep learning based regression task

There is a paper [ref.attaced], where they used a deep learning based regression and evaluated using mse and AUC. The targets are continuous values such as 1,2,3 up to 16 and has been normalized ...
0
votes
0answers
34 views

text classification - does number of features matters?

I'm working on a multi-class text classification project that aims to assign a "new bug" to his "final group assignee" To do that I was able to extract ~17000 samples and divided ...
0
votes
1answer
17 views

Visualise KMeans clusters in 2d, when number of input features is greater than 2

I am using KMeans clustering in Python (Scikit-learn) with around 70 input features per sample and a little over 1,000 samples. It is performing rather well, which is good. However, I would quite like ...
7
votes
1answer
6k views

How to remove columns in Transformer function in Pipeline?

I already used a custom transformation function in a scikit-learn pipeline. In this function I only added features to my data frame. It works great. Below is a working example: ...
0
votes
1answer
66 views

Python make_scorer giving incorrect outputs for Root Mean Square Logarithmic Error

I want to generate a grid search for which I need the scoring parameter based on which the search will take place. I have defined the following function to provide ...
0
votes
0answers
11 views

Forecast methodology for geographic variables that are somewhat related

I'm creating time series forecasts for different geographies and wanted an expert opinion on how I can take into account geographic relationship to improve my model. Is there an algorithm that's ...
0
votes
1answer
25 views

GridSearchCV is Giving me ValueError: number of labels does not match number of samples

I'm trying to run a grid CV parameter search using sklearn.model_selection.GridSearchCV. I keep getting a ValueError that is really confusing me. Below I've included the code for the pipeline I ...
2
votes
3answers
97 views

predictive modelling using Random Forest

I have created a random forest classification model in skicit-learn, but I am unsure how to finalize my forecast. I have built the model and it is showing good results on the testing data. I get a ...
0
votes
1answer
22 views

Is it reasonable to do train/test splitting based upon information/entropy?

I want to divide my time series dataset into training and test sets. The data is seasonal and very noisy. When I randomly split, the test and train samples do not resemble in their ...
0
votes
0answers
25 views

Novelty prediction Using DBSCAN on “unseen data”

I am trying to build an unsupervised learning model, which will be able to predict outliers on "unseen data." The algorithm I chose is DBSCAN (Density-based spatial clustering of ...

1
2 3 4 5
38