Questions tagged [python]

Use for data science questions related to the programming language Python. Not intended for general coding questions (-> stackoverflow).

Filter by
Sorted by
Tagged with
62
votes
8answers
74k views

Open source Anomaly Detection in Python

Problem Background: I am working on a project that involves log files similar to those found in the IT monitoring space (to my best understanding of IT space). These log files are time-series data, ...
52
votes
9answers
8k views

Tools and protocol for reproducible data science using Python

I am working on a data science project using Python. The project has several stages. Each stage comprises of taking a data set, using Python scripts, auxiliary data, configuration and parameters, and ...
12
votes
1answer
15k views

Multi-dimentional and multivariate Time-Series forecast (RNN/LSTM) Keras

I have been trying to understand how to represent and shape data to make a multidimentional and multivariate time series forecast using Keras (or TensorFlow) but I am still very unclear after reading ...
11
votes
1answer
9k views

How to Predict the future values of time horizon with Keras?

I just built this LSTM neural network with Keras ...
68
votes
2answers
47k views

Training an RNN with examples of different lengths in Keras

I am trying to get started learning about RNNs and I'm using Keras. I understand the basic premise of vanilla RNN and LSTM layers, but I'm having trouble understanding a certain technical point for ...
43
votes
9answers
147k views

ValueError: Input contains NaN, infinity or a value too large for dtype('float32')

I got ValueError when predicting test data using a RandomForest model. My code: ...
10
votes
1answer
14k views

Keras LSTM with 1D time series

I'm learning how to use Keras and I've had reasonable success with my labelled dataset using the examples on Chollet's Deep Learning for Python. The data set is ~1000 Time Series with length 3125 with ...
132
votes
17answers
110k views

Best python library for neural networks

I'm using Neural Networks to solve different Machine learning problems. I'm using Python and pybrain but this library is almost discontinued. Are there other good alternatives in Python?
23
votes
4answers
21k views

Is it necessary to standardize your data before clustering?

Is it necessary to standardize your data before cluster? In the example from scikit learn about DBSCAN, here they do this in the line: ...
6
votes
2answers
13k views

Python or R for implementing machine learning algorithms for fraud detection [closed]

I was wondering which language can I use: R or Python, for my internship in fraud detection in an online banking system: I have to build machine learning algorithms (NN, etc.) that predict transaction ...
18
votes
3answers
37k views

Feature extraction of images in Python

In my class I have to create an application using two classifiers to decide whether an object in an image is an example of phylum porifera (seasponge) or some other object. However, I am completely ...
5
votes
2answers
434 views

Transform an Autoencoder to a Variational Autoencoder?

I would like to compare the training by an Autoencoder and a variational autoencoder. I have already run the traing using AE. I would like to know if it's possible to transform this AE into a VAE and ...
1
vote
4answers
2k views

Exceptionally high accuracy with Random Forest, is it possible?

I need your help to find a flaw in my model, since it's accuracy (95%) is not realistic. I'm working on a classification problem using Randomforest, with around 2500 positive case and 15000 negative ...
5
votes
1answer
1k views

When does decision tree perform better than the neural network?

I was experimenting with different modelling methods including KNN, Decision Trees, Neural Networks and SVN and trying to fit my data to see which works the best. To my surprise, the decision tree ...
4
votes
1answer
2k views

Correlation between specific columns of a data set

I have a CSV file which has 150 columns belonging to 7 categories but I want a correlation between 2 categories. The categories are movies and music, 12 and 19 columns respectively. Is there a way ...
2
votes
4answers
959 views

Best methods to solve class imbalance problem and why?

I have a data set where I need to detect fraud. 99% are not fraud and 1% are. What methods can be used to solve problems where classes are imbalanced?
27
votes
3answers
38k views

Hypertuning XGBoost parameters

XGBoost have been doing a great job, when it comes to dealing with both categorical and continuous dependant variables. But, how do I select the optimized parameters for an XGBoost problem? This is ...
15
votes
5answers
12k views

Python library to implement Hidden Markov Models

What stable Python library can I use to implement Hidden Markov Models? I need it to be reasonably well documented, because I've never really used this model before. Alternatively, is there a more ...
14
votes
3answers
12k views

Looking for good package for anomaly detection in time series

Is there a comprehensive open source package (preferably in python or R) that can be used for anomaly detection in time series? There is a one class SVM package in scikit-learn but it is not for time ...
14
votes
5answers
10k views

Do modern R and/or Python libraries make SQL obsolete?

I work in an office where SQL Server is the backbone of everything we do, from data processing to cleaning to munging. My colleague specializes in writing complex functions and stored procedures to ...
8
votes
3answers
7k views

Nested cross-validation and selecting the best regression model - is this the right SKLearn process?

If I understand correctly, nested-CV can help me evaluate what model and hyperparameter tuning process is best. The inner loop (GridSearchCV) finds the best ...
27
votes
3answers
19k views

How to force weights to be non-negative in Linear regression

I am using a standard linear regression using scikit-learn in python. However, I would like to force the weights to be all positive for every feature (not negative), is there any way I can accomplish ...
8
votes
6answers
21k views

Python: Handling imbalance Classes in python Machine Learning

I have a dataset for which I am trying to predict target variables. ...
12
votes
3answers
5k views

Help regarding NER in NLTK

I have been working in NLTK for a while using Python. The problem I am facing is that their is no help available on training NER in NLTK with my custom data. They have used MaxEnt and trained it on ...
11
votes
3answers
6k views

Neural networks - Find most similar images

I am working with Python, scikit-learn and keras. I have 3000 thousands images of front-faced watches like the following ones: Watch_1, Watch_2, Watch_3. I want to write a program which receives as ...
8
votes
2answers
6k views

How to train model to predict events 30 minutes prior, from multi-dimensionnal timeseries

Experts in my field are capable of predicting the likelyhood an event (binary spike in yellow) 30 minutes before it occurs. Frequency here is 1 sec, this view represents a few hours worth of data, i ...
2
votes
5answers
3k views

What are some nice algorithms/techniques for optimizing and predicting Click Through Rates (CTR)?

Definition of Click Through Rates: CTR is the number of clicks that your ad receives divided by the number of times your ad is shown expressed as a percentage (clicks ÷ impressions = CTR).
6
votes
2answers
1k views

Pandas grouped data to Bokeh graph

I'm having trouble graphing Pandas grouped data in Bokeh. ...
7
votes
2answers
5k views

Recommender system based on purchase history, not ratings

I'm exploring options for recommender systems optimized for the insurance industry, which would take into account i) product holdings ii) user characteristics (segment, age, affluence, etc.). I ...
1
vote
2answers
77 views

Regression Algorithms in Production

I am interested in predicting if a doctor would prescribe a specific drug and have chosen Logistic Regression as a starting point. I have a few questions: Is feature selection the first step to take ...
5
votes
1answer
3k views

Keras CNN with low/constant accuracies

I am dealing with the Street View House Number recognition problem. I am trying to train a CNN with Keras. Here is how I prepared the input: ...
3
votes
1answer
2k views

Validation loss increases and validation accuracy decreases

I have an issue with my model. I'm trying to use the most basic Conv1D model to analyze review data and output a rating of 1-5 class, therefore the loss is categorical_crossentropy. Model structure is ...
2
votes
3answers
510 views

Obtaining consistent one-hot encoding of train / production data

I'm building an app that will require user input. Currently, on the training set, I run the following code, in which data is a pandas dataframe with a combination ...
0
votes
2answers
119 views

Place a marker on a plot

I want to place a Call Marker on a plot. Call should be "buy" whenever the smaller moving average (21) crosses over longer moving average (34) AND the Call should be "sell" whenever smaller moving ...
0
votes
2answers
89 views

Problem importing CNTK in Azure jupyter notebook

Vineeth Sai indicated in this that with the following code: pip install cntk the problem is solved. However, I am getting the error shown in attached image:
120
votes
8answers
160k views

Difference between fit and fit_transform in scikit_learn models?

I am newbie to data science and I do not understand the difference between fit and fit_transform methods in scikit-learn. Can ...
50
votes
5answers
46k views

Neural networks: which cost function to use?

I am using TensorFlow for experiments mainly with neural networks. Although I have done quite some experiments (XOR-Problem, MNIST, some Regression stuff, ...) now, I struggle with choosing the "...
65
votes
6answers
86k views

strings as features in decision tree/random forest

I am doing some problems on an application of decision tree/random forest. I am trying to fit a problem which has numbers as well as strings (such as country name) as features. Now the library, scikit-...
53
votes
8answers
58k views

Clustering geo location coordinates (lat,long pairs)

What is the right approach and clustering algorithm for geolocation clustering? I'm using the following code to cluster geolocation coordinates: ...
35
votes
3answers
35k views

Multi GPU in keras

How we can program in the keras library (or tensorflow) to partition training on multiple GPUs? Let's say that you are in an Amazon ec2 instance that has 8 GPU's and you would like to use all of them ...
18
votes
1answer
12k views

Ways to deal with longitude/latitude feature [closed]

I am working on a fictional dataset with 25 features. Two of the features are latitude and longitude of a place and others are pH values, elevation, windSpeed etc with varying ranges. I can perform ...
26
votes
1answer
9k views

PyTorch vs. Tensorflow Fold

Both PyTorch and Tensorflow Fold are deep learning frameworks meant to deal with situations where the input data has non-uniform length or dimensions (that is, situations where dynamic graphs are ...
18
votes
4answers
14k views

Hyperparameter search for LSTM-RNN using Keras (Python)

From Keras RNN Tutorial: "RNNs are tricky. Choice of batch size is important, choice of loss and optimizer is critical, etc. Some configurations won't converge." So this is more a general question ...
16
votes
2answers
7k views

Recommending movies with additional features using collaborative filtering

I am trying to build a recommendation system using collaborative filtering. I have the usual [user, movie, rating] information. I would like to incorporate an ...
24
votes
5answers
3k views

VM image for data science projects

As there are numerous tools available for data science tasks, and it's cumbersome to install everything and build up a perfect system. Is there a Linux/Mac OS image with Python, R and other open-...
16
votes
3answers
2k views

One-Class discriminatory classification with imbalanced, heterogenous Negative background?

I'm working on improving an existing supervised classifier, for classifying {protein} sequences as belonging to a specific class (Neuropeptide hormone precursors), or not. There are about 1,150 known ...
8
votes
2answers
2k views

LSTM: How to deal with nonstationarity when predicting a time series

I want to do one-step-ahead predictions for time series with LSTM. To understand the algorithm, I built myself a toy example: A simple autocorrelated process. ...
11
votes
2answers
12k views

Validation loss and accuracy remain constant

I am trying to implement this paper on a set of medical images. I am doing it in Keras. The network essentially consists of 4 conv and max-pool layers followed by a fully connected layer and soft max ...
11
votes
2answers
22k views

Keras Callback example for saving a model after every epoch?

Can someone please post a straightforward example of Keras using a callback to save a model after every epoch? I can find examples of saving weights, but I want to be able to save a completely ...
9
votes
1answer
5k views

How to binary encode multi-valued categorical variable from Pandas dataframe?

Suppose we have the following dataframe with multiple values for a certain column: categories 0 - ["A", "B"] 1 - ["B", "C", "D"] 2 - ["B", "D"] How can we ...