Questions tagged [python]
Use for data science questions related to the programming language Python. Not intended for general coding questions (which should be asked on Stack Overflow).
290
questions
73
votes
7
answers
83k
views
Open source Anomaly Detection in Python
Problem Background:
I am working on a project that involves log files similar to those found in the IT monitoring space (to my best understanding of IT space). These log files are time-series data, ...
- 933
240
votes
10
answers
347k
views
What's the difference between fit and fit_transform in scikit-learn models?
I do not understand the difference between the fit and fit_transform methods in scikit-learn. Can anybody explain simply why we ...
- 2,877
10
votes
1
answer
3k
views
Why you shouldn't upsample before cross validation
I have an imbalanced dataset and I am trying different methods to address the data imbalance. I found this article that explains the correct way to cross-validate when oversampling data using SMOTE ...
- 417
5
votes
3
answers
8k
views
Obtaining consistent one-hot encoding of train / production data
I'm building an app that will require user input. Currently, on the training set, I run the following code, in which data is a pandas dataframe with a combination ...
- 308
3
votes
4
answers
4k
views
Is it possible to get worse model after optimization?
I am trying recently to optimize models but for some reason, whenever I try to run the optimization the model score in the end is worse than before, so I believe I do something wrong.
in order to ...
- 349
95
votes
10
answers
436k
views
ValueError: Input contains NaN, infinity or a value too large for dtype('float32')
I got ValueError when predicting test data using a RandomForest model.
My code:
...
- 2,735
16
votes
1
answer
20k
views
Multi-dimentional and multivariate Time-Series forecast (RNN/LSTM) Keras
I have been trying to understand how to represent and shape data to make a multidimentional and multivariate time series forecast using Keras (or TensorFlow) but I am still very unclear after reading ...
- 263
63
votes
9
answers
10k
views
Tools and protocol for reproducible data science using Python
I am working on a data science project using Python.
The project has several stages.
Each stage comprises of taking a data set, using Python scripts, auxiliary data, configuration and parameters, and ...
- 761
4
votes
1
answer
789
views
How to preprocess with NLP a big dataset for text classification
TL;DR
I've never done nlp before and I feel like I'm not doing it in the good way. I'd like to know if I'm really doing things in a bad way since the beginning or ...
- 143
123
votes
2
answers
113k
views
Training an RNN with examples of different lengths in Keras
I am trying to get started learning about RNNs and I'm using Keras. I understand the basic premise of vanilla RNN and LSTM layers, but I'm having trouble understanding a certain technical point for ...
- 1,350
14
votes
1
answer
15k
views
How to Predict the future values of time horizon with Keras?
I just built this LSTM neural network with Keras
...
- 283
0
votes
2
answers
134
views
What is the state-of-the-art in prediction\classification missing labels in partially labeled data?
Overview
Let's say I have the following data:
...
- 364
150
votes
17
answers
126k
views
Best python library for neural networks
I'm using Neural Networks to solve different Machine learning problems. I'm using Python and pybrain but this library is almost discontinued. Are there other good alternatives in Python?
54
votes
2
answers
484k
views
train_test_split() error: Found input variables with inconsistent numbers of samples
Fairly new to Python but building out my first RF model based on some classification data. I've converted all of the labels into int64 numerical data and loaded into X and Y as a numpy array, but I am ...
- 643
11
votes
1
answer
21k
views
Keras LSTM with 1D time series
I'm learning how to use Keras and I've had reasonable success with my labelled dataset using the examples on Chollet's Deep Learning for Python. The data set is ~1000 Time Series with length 3125 with ...
- 165
9
votes
2
answers
2k
views
Transform an Autoencoder to a Variational Autoencoder?
I would like to compare the training by an Autoencoder and a variational autoencoder. I have already run the traing using AE. I would like to know if it's possible to transform this AE into a VAE and ...
- 614
4
votes
2
answers
5k
views
Imbalanced Dataset: Train/test split before and after SMOTE
This question is similar but different from my previous one. I have a binary classification task related to customer churn for a bank. The dataset contains 10,000 instances and 11 features. The target ...
- 67
145
votes
13
answers
97k
views
Why do people prefer Pandas to SQL?
I've been using SQL since 1996, so I may be biased. I've used MySQL and SQLite 3 extensively, but have also used Microsoft SQL Server and Oracle.
The vast majority of the operations I've seen done ...
87
votes
6
answers
148k
views
strings as features in decision tree/random forest
I am doing some problems on an application of decision tree/random forest. I am trying to fit a problem which has numbers as well as strings (such as country name) as features. Now the library, scikit-...
- 1,005
39
votes
5
answers
64k
views
Is it necessary to standardize your data before clustering?
Is it necessary to standardize your data before cluster? In the example from scikit learn about DBSCAN, here they do this in the line:
...
- 839
21
votes
3
answers
45k
views
Feature extraction of images in Python
In my class I have to create an application using two classifiers to decide whether an object in an image is an example of phylum porifera (seasponge) or some other object.
However, I am completely ...
- 315
15
votes
3
answers
61k
views
Keras Sequential model returns loss 'nan'
I'm implementing a neural network with Keras, but the Sequential model returns nan as loss value.
I have sigmoid activation ...
- 405
14
votes
2
answers
8k
views
How to train model to predict events 30 minutes prior, from multi-dimensionnal timeseries
Experts in my field are capable of predicting the likelyhood an event (binary spike in yellow) 30 minutes before it occurs. Frequency here is 1 sec, this view represents a few hours worth of data, i ...
- 143
11
votes
2
answers
4k
views
Is max_depth in scikit the equivalent of pruning in decision trees?
I was analyzing the classifier created using a decision tree. There is a tuning parameter called max_depth in scikit's decision tree. Is this equivalent of pruning a decision tree? If not, how could I ...
- 601
9
votes
1
answer
5k
views
When does decision tree perform better than the neural network?
I was experimenting with different modelling methods including KNN, Decision Trees, Neural Networks and SVN and trying to fit my data to see which works the best. To my surprise, the decision tree ...
- 601
9
votes
3
answers
4k
views
Is There a Way to Re-Calibrate Predicted Probabilities After Using Class Weights?
I have classification data with far more negative instances than positive instances. I have used class weights in my models and have achieved the discrimination I want but the predicted probabilities ...
6
votes
2
answers
14k
views
Python or R for implementing machine learning algorithms for fraud detection [closed]
I was wondering which language can I use: R or Python, for my internship in fraud detection in an online banking system: I have to build machine learning algorithms (NN, etc.) that predict transaction ...
- 61
6
votes
1
answer
3k
views
Correlation between specific columns of a data set
I have a CSV file which has 150 columns belonging to 7 categories but I want a correlation between 2 categories. The categories are movies and music, 12 and 19 columns respectively.
Is there a way ...
- 101
4
votes
1
answer
5k
views
F1_score(average='micro') is equal to calculating accuracy for multiclasification
Is f1_score(average='micro') always the same as calculating the accuracy. Or it is just in this case?
I have tried with different values and they gave the same answer but I don't have the analytical ...
- 6,082
2
votes
1
answer
2k
views
String handling by OneHotEncoder
I am reading everywhere on new questions and blogs that since version 0.20, OneHotEncoder is able to handle string features.
Moreover, the documentation is what looks more ambiguous. Here are the ...
- 171
2
votes
4
answers
1k
views
Best methods to solve class imbalance problem and why?
I have a data set where I need to detect fraud. 99% are not fraud and 1% are.
What methods can be used to solve problems where classes are imbalanced?
- 697
1
vote
4
answers
9k
views
Exceptionally high accuracy with Random Forest, is it possible?
I need your help to find a flaw in my model, since it's accuracy (95%) is not realistic.
I'm working on a classification problem using Randomforest, with around 2500 positive case and 15000 negative ...
1
vote
1
answer
110
views
what kind of algorithm should I use to classify the text data example given?
What kind of classification or learning algorithm that suits this kind of data example
If I have to build a model using the given key words then predict column B and then to column A?
what kind of ...
- 21
57
votes
4
answers
48k
views
Difference between OrdinalEncoder and LabelEncoder
I was going through the official documentation of scikit-learn learn after going through a book on ML and came across the following thing:
In the Documentation it is given about ...
- 703
46
votes
2
answers
74k
views
Merging two different models in Keras
I am trying to merge two Keras models into a single model and I am unable to accomplish this.
For example in the attached Figure, I would like to fetch the middle layer $A2$ of dimension 8, and use ...
- 1,033
46
votes
5
answers
47k
views
How to force weights to be non-negative in Linear regression
I am using a standard linear regression using scikit-learn in python.
However, I would like to force the weights to be all non-negative for every feature. is there any way I can accomplish that? I was ...
- 1,973
33
votes
3
answers
44k
views
Hypertuning XGBoost parameters
XGBoost have been doing a great job, when it comes to dealing with both categorical and continuous dependant variables. But, how do I select the optimized parameters for an XGBoost problem?
This is ...
- 8,256
24
votes
4
answers
24k
views
Looking for a good package for anomaly detection in time series
Is there a comprehensive open source package (preferably in python or R) that can be used for anomaly detection in time series?
There is a one class SVM package in scikit-learn but it is not for the ...
- 1,237
21
votes
4
answers
76k
views
Train, test split of unbalanced dataset classification
I have a model that does binary classification.
My dataset is highly unbalanced, so I thought that I should balance it by undersampling before I train the model. So balance the dataset and then ...
- 413
20
votes
5
answers
17k
views
Do modern R and/or Python libraries make SQL obsolete?
I work in an office where SQL Server is the backbone of everything we do, from data processing to cleaning to munging. My colleague specializes in writing complex functions and stored procedures to ...
- 363
20
votes
5
answers
16k
views
Python library to implement Hidden Markov Models
What stable Python library can I use to implement Hidden Markov Models? I need it to be reasonably well documented, because I've never really used this model before.
Alternatively, is there a more ...
- 1,783
17
votes
2
answers
9k
views
Recommending movies with additional features using collaborative filtering
I am trying to build a recommendation system using collaborative filtering. I have the usual [user, movie, rating] information. I would like to incorporate an ...
- 397
14
votes
2
answers
8k
views
Preprocessing for Text Classification in Transformer Models (BERT variants)
This might be silly to ask, but I am wondering if one should carry out the conventional text preprocessing steps for training one of the transformer models?
I remember for training a Word2Vec or Glove,...
- 4,207
13
votes
2
answers
26k
views
Activation function between LSTM layers
I'm aware the LSTM cell uses both sigmoid and tanh activation functions internally, however when creating a stacked LSTM architecture does it make sense to pass their outputs through an activation ...
- 242
13
votes
2
answers
86k
views
Perform k-means clustering over multiple columns
I am trying to perform k-means clustering on multiple columns. My data set is composed of 4 numerical columns and 1 categorical column. I already researched previous questions but the answers are not ...
- 141
12
votes
4
answers
10k
views
Neural networks - Find most similar images
I am working with Python, scikit-learn and keras. I have 3000 thousands images of front-faced watches like the following ones:
Watch_1, Watch_2, Watch_3.
I want to write a program which receives as ...
- 1,057
12
votes
3
answers
6k
views
Help regarding NER in NLTK
I have been working in NLTK for a while using Python. The problem I am facing is that their is no help available on training NER in NLTK with my custom data. They have used MaxEnt and trained it on ...
- 295
11
votes
1
answer
16k
views
Multiple Categorical values for a single feature how to convert them to binary using python
I have a data set of movies which has 28 columns. One of them is genres. For each row in this data set, the value for column genres is of the form "Action|Animation|Comedy|Family|Fantasy". I want to ...
- 111
11
votes
6
answers
28k
views
Python: Handling imbalance Classes in python Machine Learning
I have a dataset for which I am trying to predict target variables.
...
- 1,045
10
votes
2
answers
10k
views
Why `max_features=n_features` does not make the Random Forest independent of number of trees?
Consider the following simple classification problem (Python, scikit-learn)
...
- 253