Questions tagged [python]

Use for data science questions related to the programming language Python. Not intended for general coding questions (-> stackoverflow).

Filter by
Sorted by
Tagged with
132
votes
17answers
110k views

Best python library for neural networks

I'm using Neural Networks to solve different Machine learning problems. I'm using Python and pybrain but this library is almost discontinued. Are there other good alternatives in Python?
120
votes
8answers
160k views

Difference between fit and fit_transform in scikit_learn models?

I am newbie to data science and I do not understand the difference between fit and fit_transform methods in scikit-learn. Can ...
103
votes
15answers
108k views

Python vs R for machine learning

I'm just starting to develop a machine learning application for academic purposes. I'm currently using R and training myself in it. However, in a lot of places, I have seen people using Python. What ...
82
votes
1answer
103k views

Difference between isna() and isnull() in pandas

I have been using pandas for quite some time. But, I don't understood what's the difference between isna() and isnull() in ...
81
votes
11answers
77k views

SVM using scikit learn runs endlessly and never completes execution

I am trying to run SVR using scikit learn ( python ) on a training dataset having 595605 rows and 5 columns(features) and test dataset having 397070 rows. The data has been pre-processed and ...
68
votes
2answers
47k views

Training an RNN with examples of different lengths in Keras

I am trying to get started learning about RNNs and I'm using Keras. I understand the basic premise of vanilla RNN and LSTM layers, but I'm having trouble understanding a certain technical point for ...
65
votes
6answers
86k views

strings as features in decision tree/random forest

I am doing some problems on an application of decision tree/random forest. I am trying to fit a problem which has numbers as well as strings (such as country name) as features. Now the library, scikit-...
62
votes
8answers
74k views

Open source Anomaly Detection in Python

Problem Background: I am working on a project that involves log files similar to those found in the IT monitoring space (to my best understanding of IT space). These log files are time-series data, ...
53
votes
8answers
58k views

Clustering geo location coordinates (lat,long pairs)

What is the right approach and clustering algorithm for geolocation clustering? I'm using the following code to cluster geolocation coordinates: ...
52
votes
9answers
8k views

Tools and protocol for reproducible data science using Python

I am working on a data science project using Python. The project has several stages. Each stage comprises of taking a data set, using Python scripts, auxiliary data, configuration and parameters, and ...
50
votes
5answers
46k views

Neural networks: which cost function to use?

I am using TensorFlow for experiments mainly with neural networks. Although I have done quite some experiments (XOR-Problem, MNIST, some Regression stuff, ...) now, I struggle with choosing the "...
43
votes
9answers
147k views

ValueError: Input contains NaN, infinity or a value too large for dtype('float32')

I got ValueError when predicting test data using a RandomForest model. My code: ...
35
votes
3answers
161k views

Calculation and Visualization of Correlation Matrix with Pandas

I have a pandas data frame with several entries, and I want to calculate the correlation between the income of some type of stores. There are a number of stores with income data, classification of ...
35
votes
3answers
35k views

Multi GPU in keras

How we can program in the keras library (or tensorflow) to partition training on multiple GPUs? Let's say that you are in an Amazon ec2 instance that has 8 GPU's and you would like to use all of them ...
33
votes
5answers
60k views

Opening a 20GB file for analysis with pandas

I am currently trying to open a file with pandas and python for machine learning purposes it would be ideal for me to have them all in a DataFrame. Now The file is 18GB large and my RAM is 32 GB but I ...
30
votes
1answer
54k views

What is the best Keras model for multi-class classification?

I am working on research, where need to classify one of three event WINNER=(win, draw, lose) ...
27
votes
3answers
38k views

Hypertuning XGBoost parameters

XGBoost have been doing a great job, when it comes to dealing with both categorical and continuous dependant variables. But, how do I select the optimized parameters for an XGBoost problem? This is ...
27
votes
3answers
19k views

How to force weights to be non-negative in Linear regression

I am using a standard linear regression using scikit-learn in python. However, I would like to force the weights to be all positive for every feature (not negative), is there any way I can accomplish ...
26
votes
7answers
62k views

How to clone Python working environment on another machine?

I developed a machine learning model with Python (Anaconda + Flask) on my workstation and all goes well. Later, I tried to ship this program onto another machine where of course I tried to set up the ...
26
votes
1answer
9k views

PyTorch vs. Tensorflow Fold

Both PyTorch and Tensorflow Fold are deep learning frameworks meant to deal with situations where the input data has non-uniform length or dimensions (that is, situations where dynamic graphs are ...
25
votes
6answers
3k views

Machine learning techniques for estimating users' age based on Facebook sites they like

I have a database from my Facebook application and I am trying to use machine learning to estimate users' age based on what Facebook sites they like. There are three crucial characteristics of my ...
25
votes
2answers
28k views

Merging two different models in Keras

I am trying to merge two Keras models into a single model and I am unable to accomplish this. For example in the attached Figure, I would like to fetch the middle layer $A2$ of dimension 8, and use ...
25
votes
4answers
13k views

Is there a straightforward way to run pandas.DataFrame.isin in parallel?

I have a modeling and scoring program that makes heavy use of the DataFrame.isin function of pandas, searching through lists of facebook "like" records of ...
25
votes
4answers
11k views

Word2Vec for Named Entity Recognition

I'm looking to use google's word2vec implementation to build a named entity recognition system. I've heard that recursive neural nets with back propagation through structure are well suited for named ...
24
votes
5answers
3k views

VM image for data science projects

As there are numerous tools available for data science tasks, and it's cumbersome to install everything and build up a perfect system. Is there a Linux/Mac OS image with Python, R and other open-...
24
votes
4answers
32k views

Scikit-learn: Getting SGDClassifier to predict as well as a Logistic Regression

A way to train a Logistic Regression is by using stochastic gradient descent, which scikit-learn offers an interface to. What I would like to do is take a scikit-learn's SGDClassifier and have it ...
24
votes
3answers
9k views

Best practices to store Python machine learning models

What are the best practices to save, store, and share machine learning models? In Python, we generally store the binary representation of the model, using pickle or joblib. Models, in my case, can be ...
23
votes
4answers
21k views

Is it necessary to standardize your data before clustering?

Is it necessary to standardize your data before cluster? In the example from scikit learn about DBSCAN, here they do this in the line: ...
22
votes
5answers
36k views

Calculating KL Divergence in Python

I am rather new to this and can't say I have a complete understanding of the theoretical concepts behind this. I am trying to calculate the KL Divergence between several lists of points in Python. I ...
21
votes
3answers
76k views

Merging multiple data frames row-wise in PySpark

I have 10 data frames pyspark.sql.dataframe.DataFrame, obtained from randomSplit as ...
21
votes
2answers
69k views

How to sum values grouped by two columns in pandas

I have a Pandas DataFrame like this: ...
20
votes
2answers
5k views

Keras vs. tf.keras

I'm a bit confused in choosing between Keras (keras-team/keras) and tf.keras (tensorflow/tensorflow/python/keras/) for my new research project. There is a debate that Keras isn't owned by anyone, so ...
20
votes
2answers
88k views

train_test_split() error: Found input variables with inconsistent numbers of samples

Fairly new to Python but building out my first RF model based on some classification data. I've converted all of the labels into int64 numerical data and loaded into X and Y as a numpy array, but I am ...
18
votes
4answers
8k views

Improve the speed of t-sne implementation in python for huge data

I would like to do dimensionality reduction on nearly 1 million vectors each with 200 dimensions(doc2vec). I am using TSNE ...
18
votes
1answer
12k views

Ways to deal with longitude/latitude feature [closed]

I am working on a fictional dataset with 25 features. Two of the features are latitude and longitude of a place and others are pH values, elevation, windSpeed etc with varying ranges. I can perform ...
18
votes
3answers
37k views

Feature extraction of images in Python

In my class I have to create an application using two classifiers to decide whether an object in an image is an example of phylum porifera (seasponge) or some other object. However, I am completely ...
18
votes
3answers
19k views

Python implementation of cost function in logistic regression: why dot multiplication in one expression but element-wise multiplication in another

I have a very basic question which relates to Python, numpy and multiplication of matrices in the setting of logistic regression. First, let me apologise for not using math notation. I am confused ...
18
votes
4answers
14k views

Hyperparameter search for LSTM-RNN using Keras (Python)

From Keras RNN Tutorial: "RNNs are tricky. Choice of batch size is important, choice of loss and optimizer is critical, etc. Some configurations won't converge." So this is more a general question ...
18
votes
3answers
11k views

is there any data tidying tool for python/pandas similar to R tidyr tool?

I'm working on a Kaggle challenge where some variables are represented by rows instead of columns (Telstra Network Disruption). I am currently searching for the equivalent of gather(), separate() and ...
17
votes
4answers
9k views

What is the benefit of splitting tfrecord file into shards?

I'm working on speech recognition with Tensorflow and plan to train LSTM NN with massive waves dataset. Because of the performance gains, I plan to use tfrecords. There are several examples on ...
16
votes
6answers
5k views

What is the reason behind taking log transformation of few continuous variables?

I have been doing a classification problem and I have read many people's code and tutorials. One thing I've noticed is that many people take np.log or ...
16
votes
2answers
7k views

Recommending movies with additional features using collaborative filtering

I am trying to build a recommendation system using collaborative filtering. I have the usual [user, movie, rating] information. I would like to incorporate an ...
16
votes
2answers
905 views

Where in the workflow should we deal with missing data?

I'm building a workflow for creating machine learning models (in my case, using Python's pandas and sklearn packages) from data ...
16
votes
3answers
41k views

How to get predictions with predict_generator on streaming test data in Keras?

In the Keras blog on training convnets from scratch, the code shows only the network running on training and validation data. What about test data? Is the validation data the same as test data (I ...
16
votes
4answers
14k views

Python library for segmented regression (a.k.a. piecewise regression)

I am looking for a Python library that can perform segmented regression (a.k.a. piecewise regression). Example:
16
votes
3answers
2k views

One-Class discriminatory classification with imbalanced, heterogenous Negative background?

I'm working on improving an existing supervised classifier, for classifying {protein} sequences as belonging to a specific class (Neuropeptide hormone precursors), or not. There are about 1,150 known ...
15
votes
5answers
12k views

Python library to implement Hidden Markov Models

What stable Python library can I use to implement Hidden Markov Models? I need it to be reasonably well documented, because I've never really used this model before. Alternatively, is there a more ...
15
votes
3answers
3k views

Difference between OrdinalEncoder and LabelEncoder

I was going through the official documentation of scikit-learn learn after going through a book on ML and came across the following thing: In the Documentation it is given about ...
14
votes
3answers
12k views

Looking for good package for anomaly detection in time series

Is there a comprehensive open source package (preferably in python or R) that can be used for anomaly detection in time series? There is a one class SVM package in scikit-learn but it is not for time ...
14
votes
5answers
10k views

Do modern R and/or Python libraries make SQL obsolete?

I work in an office where SQL Server is the backbone of everything we do, from data processing to cleaning to munging. My colleague specializes in writing complex functions and stored procedures to ...