Questions tagged [scikit-learn]

Scikit-learn is a Python module comprising of simple and efficient tool for machine learning, data mining and data analysis. It is built on NumPy, SciPy, and matplotlib. It is distributed under the 3-Clause BSD license.

Filter by
Sorted by
Tagged with
0
votes
0answers
38 views

SGDClassifier on Big Data

I am trying actually to train a SGDClassifier with over 4,000,000 samples of data without any positive results. X vector has 6 features and looks like : [ 2 , 4 , 56431555 , 1 , 0 , 33] Y vector has ...
0
votes
0answers
13 views

Supervised dimensionality reduction for multilabel data

re there algorithms for supervised dimensionality reduction like Linear Discriminant Analysis (LDA) for multilabel classification? If I understood it right, the implementation of LDA in scikit-learn ...
0
votes
0answers
15 views

Which statistical method to use for feature selection between numerical inputs and categorical input?

I have a classification problem where my inputs are all numerical and continuous, my outputs are categorical labels [1,0,-1]. My own domain knowledge and ...
0
votes
0answers
15 views

Should I create a tfidf on a subset of a data set or use the whole corpus?

My goal in this project is to see if businesses on a list are currently customers within my organization. One piece of this involves producing a similarity score using cosine similarity on the names ...
0
votes
0answers
10 views

HoG features taking longer time in Google CO-LAB

I have been doing this for 2 days now. I want to make a model using sklearn that uses the HoG (Histogram oriented Gradients) features to classify the image (MNIST dataset having 70,000 images both ...
0
votes
0answers
12 views
0
votes
0answers
13 views

upload model to S3

Hello I'm using AWS Sage Maker to build my model. I want to store the model in S3 for later use. How do you save your model in S3 with Amazon Sage Maker? I know this seem trivial but I didn't ...
0
votes
1answer
7 views

extract classifier properties from pickled file

I have *.clf file which I get from fit() of sklearn. I fit my data with ...
0
votes
0answers
14 views

Variable with extra small Pearson coefficient has bigger positive impact on ML model performance than variable with bigger Pearson

I made some machine learning models using Python sci-kit learn library and I found some strange situation for me regarding the real importance of some variables (features) to the ML model. I found ...
0
votes
0answers
34 views

T-SNE good clustering but SVM classification poor

I am trying to classify in 4 different classes, paragraph embedding vector computed with doc2vec using an non-linear svm over them. When I visualize the embeddings using tensorboard t-sne I can see ...
0
votes
0answers
23 views

Can someone provide me the code of the MiLoF(Memory Efficient Local Outlier Factor) algorithm?

I have to code the MiLoF algorithm for detecting outliers in an unsupervised manner using non-stationary data. I am attaching the paper which explains the algorithm here. However, there are many ...
0
votes
0answers
27 views

Custom Decision Function for Custom Outlier Detection Algorithm

I have built a custom algorithm for semi-supervised anomaly detection and here is my output example as following with probability threshold set to 0.05 and 1 = outlier, 0 = inlier: ...
0
votes
0answers
28 views

How do I improve the accuracy of this classifier?

I have a dataset of 20000 x 3072 images for a homework assignment. This is just the training set, and the images are 32x32 and can depict one of four labels/classes, namely cars, trucks, boats and ...
0
votes
0answers
38 views

How to correctly manage predictions when the inputs are outbound the original scaling range?

I have a neural network for a regression problem that was trained using MinMaxScaler(0,1) for features and I have two questions with this. I often find that scaling the output (or target variable) ...
0
votes
0answers
71 views

How to use Predefined Split for Randomized SearchCV

I'm trying to regularize my random forest regressor with RandomizedSearchCV. With RandomizedSearchCV the train and test are not ...
0
votes
0answers
17 views

How do I evaluate a K-Means unsupervised anomaly detection approach?

how do I evaluate K-means clustering anomaly detection method as there is no labelled data of anomaly class. To find the cluster (K), I have used the silhouette score from Scikit learn library. Scikit ...
0
votes
1answer
52 views

Split into test and train set before or after generating document-term matrix?

I'm working on simple machine learning problems and I trying to build a classifier that can differentiate between spam and non-spam SMS. I'm confused as to whether I need to generate the document-term ...
0
votes
1answer
21 views

Solution for TF-IDF Vectorization in Angular project?

While making an Angular project to use my text-classification model on unseen data, i struggle in finding a way how to transform text to TFIDF features. Anyone faced same issue? Maybe a solution on ...
0
votes
1answer
106 views

How fit_transform, transform and TfidfVectorizer works

I'm a machine learning beginner and I tried to use the cosine similarity on fuzzy matching purpose. In the following example I want to compare 'data_dirty' with 'data_clean' : When I have to ...
0
votes
1answer
470 views

Problem to display decision tree: InvocationException: GraphViz's executables not found in Python scikit-learn?

I try to display Decision Tree in Jupyter Notebook in Python. This is my code: ...
0
votes
0answers
59 views

Calculate the correlation coefficient without one-hot encoding the category variable

I just read about Cramers V statistic to calculate the correlation coefficient without one-hot encoding the category variable. I'm trying to use it for finding the correlation coefficient of my target ...
0
votes
0answers
20 views

Resul Logit, meaning of values Python scikit-learn?

What does the marked paces mean ?: Current function value, Df Model, Df Residuals, Pseudo R-squared, Log-Likehood, LL-Null, LLR p-value, Coef, Std Err., z, [0,025 0.975] ? Could you give mean a short ...
0
votes
2answers
99 views

How I can compute Matthews Correlation Coefficient?

I want to calculate MCC, I have two CSV files: result.csv and true.csv result.csv format: ...
0
votes
0answers
20 views

Index confusion after splitting, normalizing and oversampling the data set

I am working on a data set with 16,259 samples with class label 0 and 1,639 samples with class label 1. In order to balance the data set, I am applying oversampling (after normalization). The problem ...
0
votes
1answer
429 views

Low memory error while performing degree 2 polynomial regression on (3000*1835) sized array

I am working on a problem to predict the revenue, a film will generate. Some of the features available in the data set are json collection for the crew, cast which worked in the film. I applied ...
0
votes
0answers
9 views

Can i use different types of encodings for categorical variables in one dataset

Should I mix encodings. For example for features age and income i have one type of encoding and for features typeOfPerson i have another?
0
votes
0answers
101 views

Online courses for Anomaly Detection

As the title say I have been looking for some online courses that would teach me about anomaly detection using Unsupervised Machine Learning. I want to focus only on Scikit-Learn and not go deeper ...
0
votes
0answers
31 views

Multi-label classification yielding too much unlabeled rows

I am performing multi-label classification with xgboost + OneVsRestClassifier from sklearn. ...
0
votes
1answer
60 views

Potential speedup by applying PCA once on dataset with m rows vs. IncrementalPCA to x batches of size m/x?

I've been working on trying to perform dimensionality reduction on high-dimensional, high-volume datasets (with many rows and columns - around 100,000 - 1M rows and ~500 columns). While the size of ...
0
votes
0answers
20 views

Variable selection involving mixture of numerical, high cardinal,low cardinal features

Consider a dummy dataframe: A B C D …. Z 1 2 as we 2 2 4 qq rr 5 4 5 tz rc 9 This dataframe has 25 independent variables and one target variable ,the ...
0
votes
0answers
23 views

Is there a function that could find slope of a curve ignoring peaks?

Let's say I obtained a timeserie (in blue) (with some missing data) that is (as far as i understood) : - following a general trend between specific points - more or less cyclic I have drawn the red ...
0
votes
0answers
35 views

How can I test my trained model on a completely new dataset? [duplicate]

Preface I have an annotated text dataset on hate speech. Simply put, the dataset consists of a column called text which includes a piece of text, and a column ...
0
votes
1answer
55 views

Where does spacy, nltk, prodigy, sklearn fit in the AI project?

Where does tools like spacy, sklearn, prodigy, nltk fit in the below AI project architecture and what are some common alternatives to these:
0
votes
0answers
26 views

Code freezes and never returns when linear_kernel (sklearn.metrics.pairwise) is used on 20M Movielens dataset

I'm fairly new to ML/AI, i'm trying learn the content based recommendation - here is my source code - https://github.com/jaganlal/content-based-recommender I'm using MovieLens 20M dataset - tags.csv ...
0
votes
0answers
101 views

GridSearchCV and PyTorch with skorch shows error 'Invalid parameter lr' [duplicate]

I want to use sklearn's GridSearchCV in combination with PyTorch and use Skorch for compatibility. However, I receive an error telling me that ...
0
votes
0answers
85 views

why we use “class_weight='balanced” as a classifier parameter?

Please I have imbalanced data and at first, I used SMOTE to oversample it, and it gives a good result, then I see some code used sometimes class_weight='balanced" inside of the classifier ( RL, DT, ...
0
votes
0answers
49 views

EarlyStopping in GridSearch - how to get the mean epoch after which training stopped?

is there a way to get the mean number of epochs when training stopped by EarlyStopping in GridSearch? ...
0
votes
1answer
2k views

How to choose the best parameter values for TfidfVectorizer in sklearn library?

Recently, I used TfidfVectorizer in scikit-learn library to calculate a matrix of TF-IDF features. However, I do not know how to set some parameters such as ...
0
votes
0answers
29 views

Different extraction pipeline for train and test

I'm trying to create a production-ready ML model. The problem is as follows: Training data: Database A + Plus python aggregations. Testing data: Database B + Plus python aggregations. Both ...
0
votes
1answer
290 views

splitting into train test by train_test_split of float values?

How to split into train test by train_test_split of float values ? I used LabelEncoder but I have about 300K lines and when I used the cross_val I saw ...
0
votes
1answer
35 views

CV(Curriculum vitae) Recommendation System guidance

I am building a recommender system which matches people's CV with a vacancy. So far, I used TF-IDF & Cosine Similarity to get a matching score between a vacancy and a candidate's CV. I want to ...
0
votes
1answer
30 views

Unsupervised Learning::Satellite Images::Single Bands

Has anyone has success with building models using KMeans for classification? I have images that only have one band and it continues to fail. My guess is that the issue is with both size of the image ...
0
votes
1answer
3k views

bad input shape (5634, 2)

I tried everything and I am not sure how to resolve the following error: ValueError: bad input shape (5634, 2) This is my first machine learning example so please bear with me. This is the python ...
0
votes
1answer
139 views

Feature Selection algorithm/library for CRF

I am using the Conditional Random Fields CRF suite scikit-learn wrapper algorithm. I have read on the literature various approaches for feature selection, but I cannot find any on that package or, ...
0
votes
1answer
3k views

ValueError: Expected 2D array, got 1D array instead

I would like to extract the 20 most informative features of a very large set of features $X$ coming from a dataset containing clinical data by using the RFE function from scikit-learn in Python. $X$ ...
0
votes
0answers
60 views

Regression Task - Spark, PyTorch, TensorFlow or scikit

I know it's a broad question, sorry for that, but I'm still testing the waters with machine learning. I have a typical regression task (predict target numbers with the help of features x,y,z) and a ...
0
votes
1answer
80 views

Increase accuracy of classification problem [closed]

I am trying to build a classifier that predicts the compiler given some operations of assembly code. Here is the pandas dataframe: What I do is using a TfidfVectorizer and select the features that ...
0
votes
0answers
33 views

can i get weights per iteration of MLP?

im building an mlp with scikit learn. Is there a way I can access weights and biases of the output layer per iteration? There is an option mlp.coefs_ But it ...
0
votes
1answer
70 views

Building an efficient feature vector

I am building a classifier for malware analysis, which predicts if I have a malware by looking at the intructions of an assembly code, such as push, mov,... and predicting the optimization method. ...
0
votes
2answers
29 views

SVM, which range to use when normalizing

I am using the SVM classifier from Scikit Learn. I was wondering is there is a know-best-practice when it comes to normalization. I'm using different normalization tecniques, but all my normalized ...