Questions tagged [python]

Use for data science questions related to the programming language Python. Not intended for general coding questions (-> stackoverflow).

Filter by
Sorted by
Tagged with
72 votes
7 answers
82k views

Open source Anomaly Detection in Python

Problem Background: I am working on a project that involves log files similar to those found in the IT monitoring space (to my best understanding of IT space). These log files are time-series data, ...
user avatar
  • 933
7 votes
1 answer
2k views

Why you shouldn't upsample before cross validation

I have an imbalanced dataset and I am trying different methods to address the data imbalance. I found this article that explains the correct way to cross-validate when oversampling data using SMOTE ...
user avatar
  • 335
4 votes
3 answers
6k views

Obtaining consistent one-hot encoding of train / production data

I'm building an app that will require user input. Currently, on the training set, I run the following code, in which data is a pandas dataframe with a combination ...
user avatar
228 votes
10 answers
322k views

What's the difference between fit and fit_transform in scikit-learn models?

I do not understand the difference between the fit and fit_transform methods in scikit-learn. Can anybody explain simply why we ...
user avatar
  • 2,747
3 votes
4 answers
2k views

Is it possible to get worse model after optimization?

I am trying recently to optimize models but for some reason, whenever I try to run the optimization the model score in the end is worse than before, so I believe I do something wrong. in order to ...
user avatar
  • 349
90 votes
10 answers
382k views

ValueError: Input contains NaN, infinity or a value too large for dtype('float32')

I got ValueError when predicting test data using a RandomForest model. My code: ...
user avatar
  • 2,485
16 votes
1 answer
19k views

Multi-dimentional and multivariate Time-Series forecast (RNN/LSTM) Keras

I have been trying to understand how to represent and shape data to make a multidimentional and multivariate time series forecast using Keras (or TensorFlow) but I am still very unclear after reading ...
user avatar
  • 263
62 votes
9 answers
10k views

Tools and protocol for reproducible data science using Python

I am working on a data science project using Python. The project has several stages. Each stage comprises of taking a data set, using Python scripts, auxiliary data, configuration and parameters, and ...
user avatar
  • 751
111 votes
2 answers
102k views

Training an RNN with examples of different lengths in Keras

I am trying to get started learning about RNNs and I'm using Keras. I understand the basic premise of vanilla RNN and LSTM layers, but I'm having trouble understanding a certain technical point for ...
user avatar
  • 1,230
12 votes
1 answer
14k views

How to Predict the future values of time horizon with Keras?

I just built this LSTM neural network with Keras ...
user avatar
  • 263
4 votes
1 answer
381 views

How to preprocess with NLP a big dataset for text classification

TL;DR I've never done nlp before and I feel like I'm not doing it in the good way. I'd like to know if I'm really doing things in a bad way since the beginning or ...
user avatar
150 votes
17 answers
125k views

Best python library for neural networks

I'm using Neural Networks to solve different Machine learning problems. I'm using Python and pybrain but this library is almost discontinued. Are there other good alternatives in Python?
50 votes
2 answers
348k views

train_test_split() error: Found input variables with inconsistent numbers of samples

Fairly new to Python but building out my first RF model based on some classification data. I've converted all of the labels into int64 numerical data and loaded into X and Y as a numpy array, but I am ...
user avatar
  • 503
11 votes
1 answer
19k views

Keras LSTM with 1D time series

I'm learning how to use Keras and I've had reasonable success with my labelled dataset using the examples on Chollet's Deep Learning for Python. The data set is ~1000 Time Series with length 3125 with ...
user avatar
9 votes
2 answers
2k views

Transform an Autoencoder to a Variational Autoencoder?

I would like to compare the training by an Autoencoder and a variational autoencoder. I have already run the traing using AE. I would like to know if it's possible to transform this AE into a VAE and ...
user avatar
  • 574
134 votes
12 answers
86k views

Why do people prefer Pandas to SQL?

I've been using SQL since 1996, so I may be biased. I've used MySQL and SQLite 3 extensively, but have also used Microsoft SQL Server and Oracle. The vast majority of the operations I've seen done ...
34 votes
5 answers
55k views

Is it necessary to standardize your data before clustering?

Is it necessary to standardize your data before cluster? In the example from scikit learn about DBSCAN, here they do this in the line: ...
user avatar
  • 791
6 votes
2 answers
13k views

Python or R for implementing machine learning algorithms for fraud detection [closed]

I was wondering which language can I use: R or Python, for my internship in fraud detection in an online banking system: I have to build machine learning algorithms (NN, etc.) that predict transaction ...
user avatar
  • 61
20 votes
3 answers
44k views

Feature extraction of images in Python

In my class I have to create an application using two classifiers to decide whether an object in an image is an example of phylum porifera (seasponge) or some other object. However, I am completely ...
user avatar
14 votes
2 answers
8k views

How to train model to predict events 30 minutes prior, from multi-dimensionnal timeseries

Experts in my field are capable of predicting the likelyhood an event (binary spike in yellow) 30 minutes before it occurs. Frequency here is 1 sec, this view represents a few hours worth of data, i ...
user avatar
  • 143
8 votes
3 answers
40k views

Keras Sequential model returns loss 'nan'

I'm implementing a neural network with Keras, but the Sequential model returns nan as loss value. I have sigmoid activation ...
user avatar
  • 325
6 votes
2 answers
2k views

Is There a Way to Re-Calibrate Predicted Probabilities After Using Class Weights?

I have classification data with far more negative instances than positive instances. I have used class weights in my models and have achieved the discrimination I want but the predicted probabilities ...
user avatar
9 votes
1 answer
4k views

When does decision tree perform better than the neural network?

I was experimenting with different modelling methods including KNN, Decision Trees, Neural Networks and SVN and trying to fit my data to see which works the best. To my surprise, the decision tree ...
user avatar
1 vote
4 answers
8k views

Exceptionally high accuracy with Random Forest, is it possible?

I need your help to find a flaw in my model, since it's accuracy (95%) is not realistic. I'm working on a classification problem using Randomforest, with around 2500 positive case and 15000 negative ...
user avatar
6 votes
1 answer
3k views

Correlation between specific columns of a data set

I have a CSV file which has 150 columns belonging to 7 categories but I want a correlation between 2 categories. The categories are movies and music, 12 and 19 columns respectively. Is there a way ...
user avatar
2 votes
4 answers
1k views

Best methods to solve class imbalance problem and why?

I have a data set where I need to detect fraud. 99% are not fraud and 1% are. What methods can be used to solve problems where classes are imbalanced?
user avatar
2 votes
1 answer
1k views

String handling by OneHotEncoder

I am reading everywhere on new questions and blogs that since version 0.20, OneHotEncoder is able to handle string features. Moreover, the documentation is what looks more ambiguous. Here are the ...
user avatar
  • 161
82 votes
6 answers
131k views

strings as features in decision tree/random forest

I am doing some problems on an application of decision tree/random forest. I am trying to fit a problem which has numbers as well as strings (such as country name) as features. Now the library, scikit-...
user avatar
32 votes
3 answers
43k views

Hypertuning XGBoost parameters

XGBoost have been doing a great job, when it comes to dealing with both categorical and continuous dependant variables. But, how do I select the optimized parameters for an XGBoost problem? This is ...
user avatar
  • 8,016
43 votes
2 answers
68k views

Merging two different models in Keras

I am trying to merge two Keras models into a single model and I am unable to accomplish this. For example in the attached Figure, I would like to fetch the middle layer $A2$ of dimension 8, and use ...
user avatar
  • 973
22 votes
4 answers
22k views

Looking for a good package for anomaly detection in time series

Is there a comprehensive open source package (preferably in python or R) that can be used for anomaly detection in time series? There is a one class SVM package in scikit-learn but it is not for the ...
user avatar
  • 1,207
45 votes
3 answers
31k views

Difference between OrdinalEncoder and LabelEncoder

I was going through the official documentation of scikit-learn learn after going through a book on ML and came across the following thing: In the Documentation it is given about ...
user avatar
17 votes
5 answers
16k views

Do modern R and/or Python libraries make SQL obsolete?

I work in an office where SQL Server is the backbone of everything we do, from data processing to cleaning to munging. My colleague specializes in writing complex functions and stored procedures to ...
user avatar
18 votes
5 answers
16k views

Python library to implement Hidden Markov Models

What stable Python library can I use to implement Hidden Markov Models? I need it to be reasonably well documented, because I've never really used this model before. Alternatively, is there a more ...
user avatar
  • 1,593
16 votes
4 answers
64k views

Train, test split of unbalanced dataset classification

I have a model that does binary classification. My dataset is highly unbalanced, so I thought that I should balance it by undersampling before I train the model. So balance the dataset and then ...
user avatar
  • 378
17 votes
2 answers
8k views

Recommending movies with additional features using collaborative filtering

I am trying to build a recommendation system using collaborative filtering. I have the usual [user, movie, rating] information. I would like to incorporate an ...
user avatar
  • 397
9 votes
2 answers
15k views

Using Cross Validation technique for a CNN model

I am working on a CNN model. As always, I used batches with epochs to train my model. When it completed training and validation, finally I used a test set to measure the model performance and generate ...
user avatar
  • 1,067
44 votes
5 answers
40k views

How to force weights to be non-negative in Linear regression

I am using a standard linear regression using scikit-learn in python. However, I would like to force the weights to be all non-negative for every feature. is there any way I can accomplish that? I was ...
user avatar
  • 1,851
11 votes
6 answers
27k views

Python: Handling imbalance Classes in python Machine Learning

I have a dataset for which I am trying to predict target variables. ...
user avatar
  • 1,025
11 votes
1 answer
15k views

Multiple Categorical values for a single feature how to convert them to binary using python

I have a data set of movies which has 28 columns. One of them is genres. For each row in this data set, the value for column genres is of the form "Action|Animation|Comedy|Family|Fantasy". I want to ...
user avatar
  • 111
9 votes
3 answers
10k views

Nested cross-validation and selecting the best regression model - is this the right SKLearn process?

If I understand correctly, nested-CV can help me evaluate what model and hyperparameter tuning process is best. The inner loop (GridSearchCV) finds the best ...
user avatar
12 votes
4 answers
10k views

Neural networks - Find most similar images

I am working with Python, scikit-learn and keras. I have 3000 thousands images of front-faced watches like the following ones: Watch_1, Watch_2, Watch_3. I want to write a program which receives as ...
user avatar
  • 977
6 votes
6 answers
2k views

Is there any way to explicitly measure the complexity of a Machine Learning Model in Python

I'm interested in model debugging and one of the points that it mentions is to compare your model with a "less complex" one to check if the performance is substantially better on the most ...
user avatar
  • 2,191
12 votes
3 answers
6k views

Help regarding NER in NLTK

I have been working in NLTK for a while using Python. The problem I am facing is that their is no help available on training NER in NLTK with my custom data. They have used MaxEnt and trained it on ...
user avatar
  • 195
3 votes
5 answers
4k views

What are some nice algorithms/techniques for optimizing and predicting Click Through Rates (CTR)?

Definition of Click Through Rates: CTR is the number of clicks that your ad receives divided by the number of times your ad is shown expressed as a percentage (clicks ÷ impressions = CTR).
user avatar
  • 8,016
5 votes
2 answers
19k views

Get multiple output from Keras

I have a regression problem which I have to predict 3 numerical values from a provided data. For example let's say I have a data set containing ...
user avatar
  • 161
2 votes
2 answers
2k views

What is the formula to calculate the precision, recall, f-measure with macro, micro, none for multi-label classification in sklearn metrics?

I am working in the problem of multi-label classification tasks. But I would not able to understand the formula for calculating the precision, recall, and f-measure with macro, micro, and none. ...
user avatar
7 votes
2 answers
8k views

Date Extraction in Python

I would like to extract all date information from a given document. Essentially, I guess this can be done with a lot of regexes: 2019-02-20 20.02.2019 ("German format") 02/2019 ("February 2019") "...
user avatar
  • 17.5k
6 votes
2 answers
26k views

Keras - no prediction probability for multiple output models?

I have built the following model: ...
user avatar
  • 7,917
6 votes
1 answer
2k views

Pandas grouped data to Bokeh graph

I'm having trouble graphing Pandas grouped data in Bokeh. ...
user avatar

1
2 3 4 5 6