Questions tagged [python]

Use for data science questions related to the programming language Python. Not intended for general coding questions (which should be asked on Stack Overflow).

Filter by
Sorted by
Tagged with
10 votes
3 answers
22k views

Multivariate Time-Series Clustering

I have a streaming data along with timestamp dataset that looks like this: 1.png Timestamp can be inclusive of "seconds" too, but the data may or may not change every second. it depends on ...
Abhinaya Krishna's user avatar
10 votes
1 answer
719 views

Why does my Keras model learn to recognize the background?

I'm trying to train this Keras implementation of Deeplabv3+ on Pascal VOC2012, using the pretrained model (which was also trained on that dataset). I got weird results with the accuracy quickly ...
Matt's user avatar
  • 199
10 votes
2 answers
3k views

Find missing object(s) in image with a priori knowledge about the missing object(s) (w.r.t base image)

Problem Statement: I am working on developing a method, or borrow/modify/combine existing ones, where given an golden image (reference or base with all expected objects to be present), it is able to ...
TwinPenguins's user avatar
  • 4,259
9 votes
3 answers
31k views

How to setup and run Conda on Google Colab

I am interested in using Google Colab for data modeling. How do I install conda, create an environment and run python in a notebook? I did some searching and found some helpful hints, but had several ...
Donald S's user avatar
  • 1,939
9 votes
9 answers
58k views

Confusion Matrix - Get Items FP/FN/TP/TN - Python

After run my python code: print(confusion_matrix(x_test, x_pred)) I get this: [100 32 211 21] My question is how can I get the following list: True positive = ...
John_Rodgers's user avatar
9 votes
3 answers
11k views

Nested cross-validation and selecting the best regression model - is this the right SKLearn process?

If I understand correctly, nested-CV can help me evaluate what model and hyperparameter tuning process is best. The inner loop (GridSearchCV) finds the best ...
BobbyJohnsonOG's user avatar
9 votes
5 answers
3k views

Decision tree with final decision being a linear regression

Question: I want to implement a decision tree with each leaf being a linear regression, does such a model exist (preferable in sklearn)? Example case 1: Mockup data is generated using the formula: <...
Nathan's user avatar
  • 193
9 votes
2 answers
2k views

Transform an Autoencoder to a Variational Autoencoder?

I would like to compare the training by an Autoencoder and a variational autoencoder. I have already run the traing using AE. I would like to know if it's possible to transform this AE into a VAE and ...
Kahina's user avatar
  • 624
9 votes
1 answer
11k views

XGBoost: Quantifying Feature Importances

I need to quantify the importance of the features in my model. However, when I use XGBoost to do this, I get completely different results depending on whether I use the variable importance plot or the ...
NLR's user avatar
  • 191
9 votes
3 answers
31k views

Split a list of values into columns of a dataframe?

I am new to python and stuck at a particular problem involving dataframes. The image has a sample column, however the data is not consistent. There are also some floats and NAN. I need these to be ...
Drj's user avatar
  • 427
9 votes
4 answers
18k views

Algorithms and techniques for spell checking

Can anyone suggest any algorithm and technique for spell checking? After some googling, I found some interesting ones such as this one from Peter Norvig, http://norvig.com/spell-correct.html and few ...
edwin's user avatar
  • 343
9 votes
2 answers
4k views

XGBoost and Random Forest: ntrees vs. number of boosting rounds vs. n_estimators

So I understand the main difference between Random Forests and GB Methods. Random Forests grow parallel trees and GB Methods grow one tree for each iteration. However, I am confused on the vocab used ...
Jack Armstrong's user avatar
9 votes
3 answers
11k views

In elbow curve how to find the point from where the curve starts to rise?

I am computing a distance metric on my data. The result is then being sorted in ascending order. The samples having distance more than a specific threshold are to be marked as outliers and will be ...
Faiz Kidwai's user avatar
9 votes
4 answers
15k views

How to combine PCA and MCA on mixed data?

Suppose I have mixed data and (python) code which is capable of doing PCA (principal component analysis) on continuous predictors and MCA (multiple correspondence analysis) on nominal predictors. Is ...
Wojciech J. Migda's user avatar
9 votes
2 answers
29k views

predict gives the same output value for every image (Keras)

I am trying to classify images and assign them label 1 or 0. (Skin cancer or not). I am aware of the three main issues regarding having the same output in every input. I did not split the set and ...
Florian Laborde's user avatar
9 votes
1 answer
4k views

What is GridSearchCV doing after it finishes evaluating the performance of parameter combinations that takes so long?

I'm running GridSearchCV to tune some parameters. For example: ...
Dan Scally's user avatar
  • 1,754
9 votes
2 answers
41k views

How to determine input shape in keras?

I am having difficulty finding where my error is while building deep learning models, but I typically have issues when setting the input layer input shape. This is my model: ...
Josh Zwiebel's user avatar
9 votes
2 answers
41k views

Python - Converting 3D numpy array to 2D

I have a 3D matrix like this: ...
Tarlan Ahad's user avatar
9 votes
1 answer
19k views

How do i pass data into keras?

I am currently struggling to understand how i should train my regression network using keras. I am not sure how I should pass my input data to the network. Both the input data and the output data is ...
Loser's user avatar
  • 165
9 votes
1 answer
16k views

Is it possible to have stratified train-test split of a set based on two columns?

Consider a dataframe that contains two columns, text and label. I can very easily create a stratified train-test split using ...
Aventinus's user avatar
  • 213
9 votes
3 answers
4k views

Is There a Way to Re-Calibrate Predicted Probabilities After Using Class Weights?

I have classification data with far more negative instances than positive instances. I have used class weights in my models and have achieved the discrimination I want but the predicted probabilities ...
from keras import michael's user avatar
9 votes
1 answer
5k views

When does decision tree perform better than the neural network?

I was experimenting with different modelling methods including KNN, Decision Trees, Neural Networks and SVN and trying to fit my data to see which works the best. To my surprise, the decision tree ...
Suhail Gupta's user avatar
9 votes
2 answers
16k views

Display Images (url) Inside Pandas Dataframe

I would like to display images (mostly jpg and png formats) directly from their url link inside a pandas dataframe. Imagine I already have the following dataframe: ...
TwinPenguins's user avatar
  • 4,259
9 votes
1 answer
8k views

How to train data by batch from disk?

I am working on a convolutional neural network for image classification. The training dataset is too large to be loaded on my computer memory (4gb), on top of that I also need to try some augmentation ...
Learning is a mess's user avatar
9 votes
3 answers
9k views

How to train a xgboost model on data that is too big for the memory?

What are the best practices to train xgboost (eXtreme gradient boosting) models on data that is to big to hold it in memory at once? Splitting the data and train multiple models? Are there more ...
Soerendip's user avatar
  • 724
9 votes
1 answer
452 views

Theano in deep learning research

How widely is Theano used in deep learning research? Is Theano a good start to learn the implementation of machine learning algorithms? Will learning the implementation of something like a feed ...
user avatar
9 votes
2 answers
14k views

What does pandas describe() percentiles values tell about our data?

Let say this is my dataframe ...
Eka's user avatar
  • 301
9 votes
4 answers
25k views

Validation loss much higher than training loss

I am training a CNN on some text data. The sentences are padded and embedded and fed to a CNN. The model architecture is: ...
NoLand'sMan's user avatar
9 votes
7 answers
16k views

Python library that can compute the confusion matrix for multi-label classification

I'm looking for a Python library that can compute the confusion matrix for multi-label classification. FYI: scikit-learn doesn't support multi-label for confusion matrix) What is the difference ...
Franck Dernoncourt's user avatar
9 votes
3 answers
5k views

How to estimate the variance of regressors in scikit-learn?

Every classifier in scikit-learn has a method predict_proba(x) that predicts class probabilities for x. How to do the same thing ...
Vladislav Gladkikh's user avatar
9 votes
4 answers
14k views

Exploratory Data Analysis with Image Datset

In Machine Learning Kernels on Kaggle I often see EDAs with structured data. So, I was wondering, if there are any recommended/standard procedures for EDA with image datasets. What kind of statistical ...
sushipositive 's user avatar
9 votes
1 answer
2k views

Imbalanced data causing mis-classification on multiclass dataset

I am working on text classification where I have 39 categories/classes and 8.5 million records. (In future data and categories will increase). Structure or format of my data is as follows. ...
outlier's user avatar
  • 193
9 votes
5 answers
5k views

How can we extract fields from images?

I am making an document parser which extracts data fields from the documents and store them in a structured way. Each field in my dataset is horizontal which is easy to extract. But the model fails ...
hR 312's user avatar
  • 91
9 votes
3 answers
1k views

Binary (Unary) Recommendation System with Biased Views

I would like to create a content recommendation system based on binary click data that also takes views into account. What content a user has been exposed to, and therefore has the chance to click ...
elz's user avatar
  • 43
9 votes
1 answer
7k views

Which type auto encoder gives best results for text

I did I couple of examples for auto encoders for images and they worked fine. Now I want to do an auto encoder for text that takes as input a sentence and returns the same sentence. But when I try to ...
sspp's user avatar
  • 109
9 votes
3 answers
9k views

How to run a pyspark application in windows 8 command prompt

I have a python script written with Spark Context and I want to run it. I tried to integrate IPython with Spark, but I could not do that. So, I tried to set the spark path [ Installation folder/bin ] ...
SRS's user avatar
  • 1,065
9 votes
1 answer
13k views

Imbalanced dataset in MLP classifier in python

I am dealing with imbalanced dataset and I try to make a predictive model using MLP classifier. Unfortunately the algorithm classifies all the observations from test set to class "1" and hence the f1 ...
Blazej Kowalski's user avatar
9 votes
1 answer
9k views

How to customise cost function in Scikit learn's model?

For example, when I have a problem that false negative should be penalised more, how can I incorporate that requirement in the algorithm such as SVM?
Ghostintheshell's user avatar
9 votes
2 answers
5k views

LSTM: How to deal with nonstationarity when predicting a time series

I want to do one-step-ahead predictions for time series with LSTM. To understand the algorithm, I built myself a toy example: A simple autocorrelated process. ...
Elias Strehle's user avatar
9 votes
3 answers
10k views

Export weights (formula) from Random Forest Regressor in Scikit-Learn

I trained a prediction model with Scikit Learn in Python (Random Forest Regressor) and I want to extract somehow the weights of each feature to create an excel tool for manual prediction. The only ...
Tasos's user avatar
  • 3,920
9 votes
1 answer
1k views

sklearn - overfitting problem

I'm looking for recommendations as to the best way forward for my current machine learning problem The outline of the problem and what I've done is as follows: I have 900+ trials of EEG data, where ...
Simon's user avatar
  • 1,071
9 votes
2 answers
387 views

Are there any graph embedding algorithms like this already?

I wrote an algorithm for generating node embeddings based on the graph's topology. Most of the explanation is done in the readme file and the examples. The question is: Am I reinventing the wheel? ...
monomonedula's user avatar
9 votes
0 answers
2k views

Why is my Keras model not learning image segmentation?

Edit: as is turns out, not even the model's initial creator could successfully fine-tune it. This is most likely a problem of implementation, or possibly related to the non-intuitive way in which the ...
Matt's user avatar
  • 199
9 votes
0 answers
3k views

Python : Feature Matching + Homography to find Multiple Objects

I'm trying to use OpenCV via Python to find multiple objects in a train image and match it with the key points detected from a query image. For my case, I'm trying to detect the tennis courts in the ...
Reward's user avatar
  • 91
8 votes
4 answers
52k views

How to download a Jupyter Notebook from GitHub?

This is a fairly basic question. I am working on a data science project inside of a Pandas tutorial. I can access my Jupyter notebooks through my Anaconda installation. The only problem is that the ...
Ethan's user avatar
  • 1,633
8 votes
2 answers
70k views

How to delete entire row if values in a column are NaN [closed]

I'd like to drop all the rows containing a NaN values pertaining to a column. Lets assume I have a dataset like this: ...
uharsha33's user avatar
  • 331
8 votes
3 answers
994 views

What to consider before learning a new language for data analysis

I'm currently in the very early stages of preparing a new research-project (still at the funding-application stage), and expect that data-analysis and especially visualisation tools will play a role ...
Patrick Allo's user avatar
8 votes
3 answers
5k views

R vs. Python Decision Tree

From my experiences the R Decision tree returns more accurate results than the python decision tree. Can anymore confirm this assumption and maybe knows the reason?
Rene B.'s user avatar
  • 369
8 votes
5 answers
24k views

issue with oneHotEncoding

So i have a PandasDataFrame with categorical variables in a column which i want to one hot encode i've used the following code from an ML udemy course ...
Iltl's user avatar
  • 253
8 votes
2 answers
5k views

Why Scikit and statsmodel provide different Coefficient of determination?

First of all, I know there is a similar question, however, I didn't find it so much helpful. My issue is concerning simple Linear regression and the outcome of R-Squared. I founded that results can ...
Luckasino's user avatar
  • 183

1
3 4
5
6 7
134