Questions tagged [scikit-learn]

Scikit-learn is a Python module comprising of simple and efficient tool for machine learning, data mining and data analysis. It is built on NumPy, SciPy, and matplotlib. It is distributed under the 3-Clause BSD license.

Filter by
Sorted by
Tagged with
3
votes
1answer
58 views

Machine Learning: Balanced training set but highly unbalanced prediction set? How to adjust?

I am trying to train a model to detect gender in a dataset of CEO speeches. Here are the datasets that I have: Final Dataset: 20K CEO voices analyzed (around 95% male) Testing dataset (?): 1K CEO ...
0
votes
1answer
32 views

Splitting large multi class dataset using leave one out scheme into train and test

I am doing some supervised learning using neural networks, and i have a Targets array containing 1906 samples, which contain 664 unique values. min. count of each unique value==2, by design. Is there ...
2
votes
1answer
512 views

memory error in matrix cosine_similarity

I have (20905040, 7) of a dataset to recommend 10 different product to the user it could be larger than that but anyway I got memory error when processing the ...
1
vote
2answers
188 views

How does Decision Tree with Gini Impurity Calculate Root Node?

I couldn't figure out how it selected the root node with with <=7.5 and it's gini impurity is 0.45 but I tried to manually ...
0
votes
1answer
30 views

Getting a ValueError from train_test_split

I'm working on this dataset. I'm trying to select features using Random Forest. This is the relevant code: ...
2
votes
1answer
28 views

How can I count the number of occurrences of a category in dataset as part of an Sklearn Pipeline

Let us say we have a dataset with a feature such as Surname. arr['Surname'] = ['Smith', 'Jones', 'Johnson', 'Smith'] And I want to encode this categorical info ...
1
vote
0answers
16 views

Scikit-learn randomforestclassifier error on fitting

So I am trying to use the random forest classifier from scikit-learn and I use tfidfvectorizer to create a feature set, then use test_train_split to create X_train and Y_train. I pass this into the ...
0
votes
0answers
49 views

How to detect anomalies (errors and exceptions) in log files?

Is this a good approach? So I'm working on a Root Cause Analysis system which should help find the cause/the root error of failed system builds (packaged in a tarball), through the analysis of log ...
1
vote
1answer
191 views

How to normalize complex-valued data?

I'm taking the abs of all elements, compute the mean, subtract it off from the original values. I just feel that this is not correct and can change the vectors. I'm also dividing by the standard ...
1
vote
1answer
406 views

Sklearn train_test_split() error: Found input variables with inconsistent numbers of samples

I am fitting a regression model on randomly generated X1,x2 and Y be the sum of x1, x2 but I am getting this error ValueError: Found input variables with inconsistent numbers of samples: [2, ...
0
votes
2answers
55 views

Can we use DecisionTreeClassifier of sklearn for continuous target variable?

I have a continuous target variable named "quality" which ranges from 0 to 10. Also I have 11 input variables in my dataset. When I'm building my model using DecisionTreeClassifier() of sklearn then ...
2
votes
1answer
40 views

Item-based recommender using K-NN

I'm trying to build an item-based recommender using k-nn. I have a list of items, all of which have some properties (features) in common. ...
1
vote
0answers
283 views

How to calculate pedigree function in diabetes prediction?

I am developing a model for diabetes prediction using this dataset using Logistic Regression. I have completed the model and my input variables are - Pregnancies, Glucose, blood pressure, BMI, ...
1
vote
1answer
20 views

ML Approach for Getting List of Observations with Similar Features (Discrete+Continuous)

I have a dataset with 19k observations. Each has approximately 448 features: - Text description turned into vectors of size 300 - 16 categorical variables represented numerically - The remainder ...
0
votes
0answers
34 views

do I need to upload all model files?

here is the link of the model inside the link there is a biobert model and i do not understand that how can i implement this model on a simple html page where there is a textfield for the model input? ...
0
votes
0answers
24 views

How to find coeffcients from pre-trained model

I have trained and pickled logistic model. Have also successfully loaded the pre-trained model and performed prediction on unseen/new data. However, would like to know how to find coefficients/...
1
vote
0answers
45 views

Feature Importance Scores from Gradient Boosting vs Random Forest

In sklearn, the feature_importances_ attribute exists for both RandomForestClassifier and GradientBoostingClassifier. Would like to know what are the fundamental differences in how this attribute is ...
1
vote
1answer
35 views

Python sklearn PCA transform function output does not match

I am computing PCA on some data using 10 components and using 3 out of 10 as: ...
2
votes
1answer
45 views

Is it possible to ensure that all classes are represented in the output of a scikit-learn decision tree?

I am working with an ordinal classification problem with six ordered classes and I want to compare a neural network classifier with a baseline classifier that is as simple and parameter-free as ...
0
votes
0answers
69 views

Cluster based on both positions and similarity scores

I have a dataframe position giving me the x and y positions of 87 points. I also have a 87 x 87 similarity matrix giving me the pairwise similarity scores between ...
0
votes
0answers
77 views

Regarding sklearn adjusted_rand_score in k means clustering

A dataset is given consisting of target class as categorical.K means was applied to it for clustering the data and got the corresponding cluster labels. But how to put the values in adjusted Rand ...
1
vote
1answer
32 views

Inputs required for Random Forest Regressor and ways to improve performance

I am using Random Forest Regressor to predict inventory needs. The data I am using to train the model lists the total quantity picked for each product per date, but does not include rows where total ...
0
votes
1answer
41 views

How to measure the correlation between categorical variables and a continuous variable

I have the following list of the names of the categorical variables in my dataset: ...
1
vote
2answers
31 views

Is this the way to obtain the same individuals for x_test and y_test?

x_train, x_test = train_test_split(x, test_size = 0.3,random_state=250) y_train, y_test = train_test_split(y, test_size = 0.3,random_state=250) Is this the way to ...
0
votes
0answers
10 views

How to find anomalies using NMF

I am new to NMF(Non negative matrix), I am trying to use it to detect anomalies in my credit card data (unsupervised data) However after generating the components that is nmf.components_ I am not ...
0
votes
1answer
366 views

Improve the accuracy for multi-label classification (Scikit-learn, Keras)

I am going to train machine learning models that assign certain tags to a paragraph describing an activity. In my database, for a give paragraph of description (X), there are several corresponding ...
1
vote
1answer
33 views

How does personalized machine learning work?

Many services (such as Netflix, Amazon, and Google Search, Apple's Siri) are said to get better by learning the 'habits' of their users. As I understand, they somehow create a customized machine ...
1
vote
0answers
29 views

Problem On Class Imbalanced Data

I am getting an F-Score of 0.99 on the train_test_split data, but only getting 0.40 for a competition's test data. I am oversampling with random forest (after ...
0
votes
2answers
45 views

How to train model to predict 1 value from multiple input samples

Very new in ML... Actual problem is more complex, but I'll give it shorter. Train data is a collection of samples, describing features of some items. There are many samples for each item. So ML ...
0
votes
1answer
30 views

metric accuracy for linear regression and non linear regression

Is there a one size fit all metric to measure accuracy / error rate for both linear or non linear regression models? For example adjust R2 is only for multi linear regression (or so they say). RMSE ...
0
votes
2answers
253 views

are OneHotEncoder and keras To_categorical same?

The length of human_vocab is 18377. The length of input X is 1000 I'm trying to run ...
0
votes
0answers
10 views

Classifying Short Texts with Spatial Features

I have a dataset of short texts (like tweets) in addition there's some geographical data attached to each tweet - coordinates, whether it was made on the road, street, outside or in the building, ...
0
votes
1answer
34 views

adding a supervizing process during knn process

I try to improve my knn regression process (I use sklearn / python, but it doesn't matter). Because I can have a scientific point of view on my data, I would like to improve my results. I give you an ...
1
vote
3answers
62 views

Is it possible to predict using fewer number of features than the number of features that was used in training the model?

I'm making a model using sklearn.svm.SVC that would predict machine performance (ErrorID). For the training of the model, I'm using 6 features, namely, ...
0
votes
1answer
95 views

Text PreProcessing not improving accuracy

I was wondering if TfidfVectorizer() from scikit-learn and its methods fit_transform/transform already do language preprocessing like lowercase/lemmatization/removing punctuation. I am using Imdbs ...
1
vote
2answers
238 views

convert predict_proba results using class_weight in training

As my dataset is unbalanced(class 1: 5%, class 0: 95%) I have used class_weight="balanced" parameter to train a random forest classification model. In this way I penalize the misclassification of a ...
0
votes
3answers
329 views

how to avoid tokenizing w/ sklearn feature extraction

I'm trying to analyze some machine log files and the column I'm looking at can have values like 'Part.C1.11.Reading Status'. I want to treat the complete string as one token and I don't want it to be ...
5
votes
1answer
543 views

Isolation forest sklearn contamination param

I'm working on an unsupervised anomaly detection task on time series using isolation forest algorithm. I'm developing in Python, more in detail using sklearn. I found out a lot of examples on this, ...
0
votes
0answers
8 views

Custom Imputation relative to targets in TRAIN and TEST sets

I have a methodology question for dealing with heaps of missing data in my project. My dataset is composed of parts A (~200 columns) and B (another ~200 columns). Together they are to be used for ...
0
votes
2answers
18 views

How do I fix mis-rendered matplotlib?

How do I correct my data or format it so that it is presentable, and fix my graphs? Dataset is 345551 rows × 7 columns. I am using numpy, pandas, seaborn and matplot lib. It seems that my pricing ...
1
vote
0answers
871 views

Getting 'ValueError: setting an array element with a sequence.' when attempting to fit mixed-type data

I have already seen this, this and this question, but none of the suggestions seemed to fix my problem (so I have reverted them). I have the following code: ...
0
votes
2answers
96 views

What is the best way to replace NaN values for ranked columns

I have a column named BsmntQual that gives a ranking on the height of the basement per each house. These are all of the unique values in this column: ...
1
vote
0answers
75 views

How to put KerasClassifier, Hyperopt and Sklearn cross-validation together

I am performing a hyperparameter tuning optimization (hyperopt) tasks with sklearn on a Keras models. I am trying to optimize KerasClassifiers using the Sklearn cross-validation, Some code follows: <...
1
vote
1answer
21 views

Why are there over 600 open pull-requests in Scikit-learn github repo?

I considered submitting a function that I deem missing to scikit-learn repo, but as of June 28th 2019 there are over 600 pending Pull Requests: https://github.com/scikit-learn/scikit-learn/pulls. Can ...
0
votes
0answers
34 views

Tag words of interest by using machine learning?

I have a set of documents (around 50k), each document is a sentence long. I would like to tag each document with the words that relate to risks. For example, "dangerous", "hazard", "fatal", etc... If ...
0
votes
3answers
23 views

How to find the driver features towards a particular result in Classification problems

In a classifier model, we can predict the outcome class, but here I need to find out the features that drive towards a particular result in a classification problem, that are a strong indicator of a ...
4
votes
3answers
158 views

Unexpected results from scikit learn regression decision tree

Apologies for this newbie question. I have a scikit learn DecisionTreeRegressor with muti-variable output. If the output is in the format [ output_var1, ...
2
votes
1answer
108 views

Number of Nodes in Isolation Forest

I am currently reading this paper on Isolation Forest. At page 3, there is a definition of Isolation Tree and there are a couple of sentences that I don't understand: Given a sample of data X = {...
2
votes
0answers
10 views

Confusion regarding prediction results of SVM and ANN on feature vectors

I am making a custom image classifier using Transfer Learning on Inception V3. I have 3 classes of images with ~6K images each. The input dimension of the network is 500X500 and the output of the ...
0
votes
0answers
26 views

Passing multiple rows as a single batch for prediction using sklearn

I am trying to train a standard sklearn ml model (random forest). However, my data is a collection of rows, each having a column of date and ...