Questions tagged [python]

Use for data science questions related to the programming language Python. Not intended for general coding questions (which should be asked on Stack Overflow).

Filter by
Sorted by
Tagged with
9 votes
1 answer
8k views

How to train data by batch from disk?

I am working on a convolutional neural network for image classification. The training dataset is too large to be loaded on my computer memory (4gb), on top of that I also need to try some augmentation ...
9 votes
1 answer
443 views

Theano in deep learning research

How widely is Theano used in deep learning research? Is Theano a good start to learn the implementation of machine learning algorithms? Will learning the implementation of something like a feed ...
user avatar
9 votes
2 answers
13k views

What does pandas describe() percentiles values tell about our data?

Let say this is my dataframe ...
  • 281
9 votes
3 answers
7k views

Splitting train/test sets by an identifier?

I know sklearn has train_test_split() to split a train and test set. But I read that, even with setting a random seed, if your actual dataset is updated regularly, ...
9 votes
7 answers
16k views

Python library that can compute the confusion matrix for multi-label classification

I'm looking for a Python library that can compute the confusion matrix for multi-label classification. FYI: scikit-learn doesn't support multi-label for confusion matrix) What is the difference ...
9 votes
3 answers
4k views

How to estimate the variance of regressors in scikit-learn?

Every classifier in scikit-learn has a method predict_proba(x) that predicts class probabilities for x. How to do the same thing ...
9 votes
1 answer
2k views

Imbalanced data causing mis-classification on multiclass dataset

I am working on text classification where I have 39 categories/classes and 8.5 million records. (In future data and categories will increase). Structure or format of my data is as follows. ...
  • 193
9 votes
3 answers
14k views

Clustering of documents using the topics derived from Latent Dirichlet Allocation

I want to use Latent Dirichlet Allocation for a project and I am using Python with the gensim library. After finding the topics I would like to cluster the documents using an algorithm such as k-means(...
  • 211
9 votes
3 answers
998 views

Binary (Unary) Recommendation System with Biased Views

I would like to create a content recommendation system based on binary click data that also takes views into account. What content a user has been exposed to, and therefore has the chance to click ...
  • 43
9 votes
1 answer
7k views

Which type auto encoder gives best results for text

I did I couple of examples for auto encoders for images and they worked fine. Now I want to do an auto encoder for text that takes as input a sentence and returns the same sentence. But when I try to ...
  • 109
9 votes
3 answers
9k views

How to run a pyspark application in windows 8 command prompt

I have a python script written with Spark Context and I want to run it. I tried to integrate IPython with Spark, but I could not do that. So, I tried to set the spark path [ Installation folder/bin ] ...
  • 1,045
9 votes
3 answers
20k views

Multivariate Time-Series Clustering

I have a streaming data along with timestamp dataset that looks like this: 1.png Timestamp can be inclusive of "seconds" too, but the data may or may not change every second. it depends on ...
9 votes
1 answer
8k views

How to customise cost function in Scikit learn's model?

For example, when I have a problem that false negative should be penalised more, how can I incorporate that requirement in the algorithm such as SVM?
9 votes
3 answers
8k views

Export weights (formula) from Random Forest Regressor in Scikit-Learn

I trained a prediction model with Scikit Learn in Python (Random Forest Regressor) and I want to extract somehow the weights of each feature to create an excel tool for manual prediction. The only ...
  • 3,830
9 votes
1 answer
1k views

sklearn - overfitting problem

I'm looking for recommendations as to the best way forward for my current machine learning problem The outline of the problem and what I've done is as follows: I have 900+ trials of EEG data, where ...
  • 1,071
9 votes
0 answers
1k views

Find missing object(s) in image with a priori knowledge about the missing object(s) (w.r.t base image)

Problem Statement: I am working on developing a method, or borrow/modify/combine existing ones, where given an golden image (reference or base with all expected objects to be present), it is able to ...
  • 4,126
9 votes
2 answers
307 views

Are there any graph embedding algorithms like this already?

I wrote an algorithm for generating node embeddings based on the graph's topology. Most of the explanation is done in the readme file and the examples. The question is: Am I reinventing the wheel? ...
9 votes
0 answers
2k views

Why is my Keras model not learning image segmentation?

Edit: as is turns out, not even the model's initial creator could successfully fine-tune it. This is most likely a problem of implementation, or possibly related to the non-intuitive way in which the ...
  • 199
8 votes
8 answers
50k views

Confusion Matrix - Get Items FP/FN/TP/TN - Python

After run my python code: print(confusion_matrix(x_test, x_pred)) I get this: [100 32 211 21] My question is how can I get the following list: True positive = ...
8 votes
5 answers
23k views

How to make LightGBM to suppress output?

I have tried for a while to figure out how to "shut up" LightGBM. Especially, I would like to suppress the output of LightGBM during training (i.e. feedback on the boosting steps). My model: ...
  • 7,124
8 votes
2 answers
69k views

How to delete entire row if values in a column are NaN [closed]

I'd like to drop all the rows containing a NaN values pertaining to a column. Lets assume I have a dataset like this: ...
  • 331
8 votes
3 answers
987 views

What to consider before learning a new language for data analysis

I'm currently in the very early stages of preparing a new research-project (still at the funding-application stage), and expect that data-analysis and especially visualisation tools will play a role ...
8 votes
3 answers
4k views

R vs. Python Decision Tree

From my experiences the R Decision tree returns more accurate results than the python decision tree. Can anymore confirm this assumption and maybe knows the reason?
  • 369
8 votes
3 answers
17k views

Algorithms and techniques for spell checking

Can anyone suggest any algorithm and technique for spell checking? After some googling, I found some interesting ones such as this one from Peter Norvig, http://norvig.com/spell-correct.html and few ...
  • 333
8 votes
3 answers
4k views

How to find out if two datasets are close to each other?

I have the following three datasets. ...
8 votes
5 answers
22k views

issue with oneHotEncoding

So i have a PandasDataFrame with categorical variables in a column which i want to one hot encode i've used the following code from an ML udemy course ...
  • 253
8 votes
1 answer
9k views

XGBoost: Quantifying Feature Importances

I need to quantify the importance of the features in my model. However, when I use XGBoost to do this, I get completely different results depending on whether I use the variable importance plot or the ...
  • 181
8 votes
1 answer
24k views

Resume Parsing - extracting skills from resume using Machine Learning

I am trying to extract a skill set of an employee from his/her resume. I have resumes stored as plain text in Database. I do not have predefined skills in this case. How should I approach this problem?...
  • 1,183
8 votes
2 answers
3k views

XGBoost and Random Forest: ntrees vs. number of boosting rounds vs. n_estimators

So I understand the main difference between Random Forests and GB Methods. Random Forests grow parallel trees and GB Methods grow one tree for each iteration. However, I am confused on the vocab used ...
8 votes
1 answer
4k views

What is GridSearchCV doing after it finishes evaluating the performance of parameter combinations that takes so long?

I'm running GridSearchCV to tune some parameters. For example: ...
  • 1,714
8 votes
3 answers
3k views

What is normalization for?

I am new in python and data science (and not great in math). I am learning machine learning. I got following normalize function. Can you please explain what does this normalize function do? ...
8 votes
1 answer
3k views

Why you shouldn't upsample before cross validation

I have an imbalanced dataset and I am trying different methods to address the data imbalance. I found this article that explains the correct way to cross-validate when oversampling data using SMOTE ...
  • 355
8 votes
2 answers
9k views

Date Extraction in Python

I would like to extract all date information from a given document. Essentially, I guess this can be done with a lot of regexes: 2019-02-20 20.02.2019 ("German format") 02/2019 ("February 2019") "...
  • 18.2k
8 votes
1 answer
10k views

Cosine Distance > 1 in scipy

I am working on a recommendation engine, and I have chosen to use SciPy's cosine distance as a way of comparing items. I have two vectors: ...
  • 183
8 votes
1 answer
11k views

Is it possible to have stratified train-test split of a set based on two columns?

Consider a dataframe that contains two columns, text and label. I can very easily create a stratified train-test split using ...
  • 203
8 votes
2 answers
3k views

Is There a Way to Re-Calibrate Predicted Probabilities After Using Class Weights?

I have classification data with far more negative instances than positive instances. I have used class weights in my models and have achieved the discrimination I want but the predicted probabilities ...
8 votes
2 answers
2k views

What are the disadvantages of having a left skewed distribution?

I'm currently working on a classification problem and I've a numerical column which is left skewed. i've read many posts where people are recommending to take log transformation or boxcox ...
  • 901
8 votes
2 answers
17k views

How to train ML model with multiple variables?

I am trying to learn Machine Learning concepts these days. I understand in a traditional ML data, we will have features and labels. I have following toy data in my mind where I have features like '...
  • 183
8 votes
1 answer
1k views

What is the best performance metric used in balancing dataset using SMOTE technique

I used smote technique to oversample my dataset and now I have a balanced dataset. The problem I faced is that the performance metrics; precision, recall, f1 measure, accuracy in the imbalanced ...
8 votes
3 answers
7k views

Clarification on the Keras Recurrent Unit Cell

I paste below the Keras documentation on Recurrent layer ...
8 votes
2 answers
10k views

Predicting probability from scikit-learn SVC decision_function with decision_function_shape='ovo'

I have a multiclass SVM classifier with labels 'A', 'B', 'C', 'D'. This is the code I'm running: ...
8 votes
2 answers
869 views

Pylearn2 vs TensorFlow

I am about to dive into a long NN research project and wanted a push in the direction of Pylearn2 or TensorFlow? As of Dec 2015 has the community started to lean one direction or another? This link ...
8 votes
1 answer
3k views

sklearn SimpleImputer too slow for categorical data represented as string values

I have a data set with categorical features represented as string values and I want to fill-in missing values in it. I’ve tried to use sklearn’s SimpleImputer but ...
8 votes
2 answers
14k views

Display Images (url) Inside Pandas Dataframe

I would like to display images (mostly jpg and png formats) directly from their url link inside a pandas dataframe. Imagine I already have the following dataframe: ...
  • 4,126
8 votes
3 answers
37k views

Extracting Features Using TensorFlow CNN

I'm trying to extract features of set of images. I'm using CNN from this site. Can anyone please tell me how to do feature extraction of images using CNN? I looked for various places. But nowhere it'...
  • 237
8 votes
1 answer
9k views

How to implement global contrast normalization in python?

I am trying to implement global contrast normalization in python from Yoshua Bengio's Deep Learning book (section 12.2.1.1 pg. 442). From the book, to get a normalized image using global contrast ...
8 votes
2 answers
5k views

Image clustering by similarity measurement (CW-SSIM)

I'm trying to use scikit-learn and pyssim for clustering a set of images - less than 100. The end goal is to place the images into several buckets (clusters) according to the calculated similarity ...
8 votes
2 answers
6k views

Applying dimensionality reduction on OneHotEncoded array

I have a really large data set with mixed variables. I have converted categorical variables to numerical using OneHotEncoding and it has resulted in more than a ...
  • 231
8 votes
1 answer
10k views

KL-divergence returns infinity

Given an original probability distribution P, I want to measure how much an approximation Q differs from the initial distribution. For that I calculate the KL-divergence via ...
user avatar
8 votes
4 answers
11k views

Exploratory Data Analysis with Image Datset

In Machine Learning Kernels on Kaggle I often see EDAs with structured data. So, I was wondering, if there are any recommended/standard procedures for EDA with image datasets. What kind of statistical ...

1
3 4
5
6 7
131