Questions tagged [python]

Use for data science questions related to the programming language Python. Not intended for general coding questions (which should be asked on Stack Overflow).

Filter by
Sorted by
Tagged with
240 votes
10 answers

What's the difference between fit and fit_transform in scikit-learn models?

I do not understand the difference between the fit and fit_transform methods in scikit-learn. Can anybody explain simply why we ...
Kaggle's user avatar
  • 2,877
192 votes
1 answer

Difference between isna() and isnull() in pandas

I have been using pandas for quite some time. But, I don't understand what's the difference between isna() and isnull(). And, ...
Vaibhav Thakur's user avatar
150 votes
17 answers

Best python library for neural networks

I'm using Neural Networks to solve different Machine learning problems. I'm using Python and pybrain but this library is almost discontinued. Are there other good alternatives in Python?
145 votes
13 answers

Why do people prefer Pandas to SQL?

I've been using SQL since 1996, so I may be biased. I've used MySQL and SQLite 3 extensively, but have also used Microsoft SQL Server and Oracle. The vast majority of the operations I've seen done ...
127 votes
14 answers

Python vs R for machine learning

I'm just starting to develop a machine learning application for academic purposes. I'm currently using R and training myself in it. However, in a lot of places, I have seen people using Python. What ...
124 votes
2 answers

Training an RNN with examples of different lengths in Keras

I am trying to get started learning about RNNs and I'm using Keras. I understand the basic premise of vanilla RNN and LSTM layers, but I'm having trouble understanding a certain technical point for ...
Tac-Tics's user avatar
  • 1,360
116 votes
12 answers

SVM using scikit learn runs endlessly and never completes execution

I am trying to run SVR using scikit-learn (python) on a training dataset that has 595605 rows and 5 columns (features) while the test dataset has 397070 rows. The data has been pre-processed and ...
tejaskhot's user avatar
  • 4,025
95 votes
10 answers

ValueError: Input contains NaN, infinity or a value too large for dtype('float32')

I got ValueError when predicting test data using a RandomForest model. My code: ...
Edamame's user avatar
  • 2,735
87 votes
6 answers

strings as features in decision tree/random forest

I am doing some problems on an application of decision tree/random forest. I am trying to fit a problem which has numbers as well as strings (such as country name) as features. Now the library, scikit-...
user3001408's user avatar
  • 1,005
74 votes
9 answers

How to clone Python working environment on another machine?

I developed a machine learning model with Python (Anaconda + Flask) on my workstation and all goes well. Later, I tried to ship this program onto another machine where of course I tried to set up the ...
Hendrik's user avatar
  • 8,477
73 votes
7 answers

Open source Anomaly Detection in Python

Problem Background: I am working on a project that involves log files similar to those found in the IT monitoring space (to my best understanding of IT space). These log files are time-series data, ...
ximiki's user avatar
  • 933
68 votes
4 answers

What is the use of torch.no_grad in pytorch?

I am new to pytorch and started with this github code. I do not understand the comment in line 60-61 in the code ...
mausamsion's user avatar
  • 1,237
67 votes
9 answers

Clustering geo location coordinates (lat,long pairs)

What is the right approach and clustering algorithm for geolocation clustering? I'm using the following code to cluster geolocation coordinates: ...
rok's user avatar
  • 813
63 votes
9 answers

Tools and protocol for reproducible data science using Python

I am working on a data science project using Python. The project has several stages. Each stage comprises of taking a data set, using Python scripts, auxiliary data, configuration and parameters, and ...
Yuval F's user avatar
  • 761
60 votes
5 answers

Neural networks: which cost function to use?

I am using TensorFlow for experiments mainly with neural networks. Although I have done quite some experiments (XOR-Problem, MNIST, some Regression stuff, ...) now, I struggle with choosing the "...
daniel451's user avatar
  • 723
58 votes
4 answers

Difference between OrdinalEncoder and LabelEncoder

I was going through the official documentation of scikit-learn learn after going through a book on ML and came across the following thing: In the Documentation it is given about ...
Saurabh Singh's user avatar
54 votes
2 answers

train_test_split() error: Found input variables with inconsistent numbers of samples

Fairly new to Python but building out my first RF model based on some classification data. I've converted all of the labels into int64 numerical data and loaded into X and Y as a numpy array, but I am ...
josh_gray's user avatar
  • 643
49 votes
5 answers

Opening a 20GB file for analysis with pandas

I am currently trying to open a file with pandas and python for machine learning purposes it would be ideal for me to have them all in a DataFrame. Now The file is 18GB large and my RAM is 32 GB but I ...
Hari Prasad's user avatar
46 votes
6 answers

Calculating KL Divergence in Python

I am rather new to this and can't say I have a complete understanding of the theoretical concepts behind this. I am trying to calculate the KL Divergence between several lists of points in Python. I ...
Nanda's user avatar
  • 773
46 votes
5 answers

How to force weights to be non-negative in Linear regression

I am using a standard linear regression using scikit-learn in python. However, I would like to force the weights to be all non-negative for every feature. is there any way I can accomplish that? I was ...
user's user avatar
  • 1,973
46 votes
2 answers

Merging two different models in Keras

I am trying to merge two Keras models into a single model and I am unable to accomplish this. For example in the attached Figure, I would like to fetch the middle layer $A2$ of dimension 8, and use ...
Rkz's user avatar
  • 1,033
45 votes
3 answers

What does from_logits=True do in SparseCategoricalcrossEntropy loss function?

In the documentation it has been mentioned that y_pred needs to be in the range of [-inf to inf] when from_logits=True. I truly ...
Nagendra Prasad's user avatar
44 votes
4 answers

Multi GPU in Keras

How we can program in the Keras library (or TensorFlow) to partition training on multiple GPUs? Let's say that you are in an Amazon ec2 instance that has 8 GPUs and you would like to use all of them ...
Hector Blandin's user avatar
39 votes
5 answers

Is it necessary to standardize your data before clustering?

Is it necessary to standardize your data before cluster? In the example from scikit learn about DBSCAN, here they do this in the line: ...
makansij's user avatar
  • 839
38 votes
3 answers

Calculation and Visualization of Correlation Matrix with Pandas

I have a pandas data frame with several entries, and I want to calculate the correlation between the income of some type of stores. There are a number of stores with income data, classification of ...
gdlm's user avatar
  • 535
38 votes
5 answers

Best practices to store Python machine learning models

What are the best practices to save, store, and share machine learning models? In Python, we generally store the binary representation of the model, using pickle or joblib. Models, in my case, can be ...
Antoine Dusséaux's user avatar
36 votes
6 answers

Sentence similarity prediction

I'm looking to solve the following problem: I have a set of sentences as my dataset, and I want to be able to type a new sentence, and find the sentence that the new one is the most similar to in the ...
lte__'s user avatar
  • 1,310
35 votes
6 answers

Merging multiple data frames row-wise in PySpark

I have 10 data frames pyspark.sql.dataframe.DataFrame, obtained from randomSplit as ...
krishna Prasad's user avatar
35 votes
1 answer

What is the best Keras model for multi-class classification?

I am working on research, where need to classify one of three event WINNER=(win, draw, lose) ...
SpanishBoy's user avatar
34 votes
4 answers

How to use LeakyRelu as activation function in sequence DNN in keras?When it perfoms better than Relu?

How do you use LeakyRelu as an activation function in sequence DNN in keras? If I want to write something similar to: ...
user10296606's user avatar
  • 1,824
33 votes
3 answers

Hypertuning XGBoost parameters

XGBoost have been doing a great job, when it comes to dealing with both categorical and continuous dependant variables. But, how do I select the optimized parameters for an XGBoost problem? This is ...
Dawny33's user avatar
  • 8,256
33 votes
1 answer

Ways to deal with longitude/latitude feature [closed]

I am working on a fictional dataset with 25 features. Two of the features are latitude and longitude of a place and others are pH values, elevation, windSpeed etc with varying ranges. I can perform ...
AllThingsScience's user avatar
32 votes
2 answers

How to calculate the fold number (k-fold) in cross validation?

I am confused about how I choose the number of folds (in k-fold CV) when I apply cross validation to check the model. Is it dependent on data size or other parameters?
Taimur Islam's user avatar
31 votes
8 answers

How to count the number of missing values in each row in Pandas dataframe?

How can I get the number of missing value in each row in Pandas dataframe. I would like to split dataframe to different dataframes which have same number of missing values in each row. Any suggestion?...
Kaggle's user avatar
  • 2,877
31 votes
6 answers

What is the reason behind taking log transformation of few continuous variables?

I have been doing a classification problem and I have read many people's code and tutorials. One thing I've noticed is that many people take np.log or ...
Sai Kumar's user avatar
  • 611
31 votes
4 answers

Is pandas now faster than data.table?

Here is the GitHub link to the most recent data.table benchmark. The data.table benchmarks has not been updated since 2014. I ...
xiaodai's user avatar
  • 630
31 votes
3 answers

Keras Callback example for saving a model after every epoch?

Can someone please post a straightforward example of Keras using a callback to save a model after every epoch? I can find examples of saving weights, but I want to be able to save a completely ...
I_Play_With_Data's user avatar
28 votes
5 answers

What is the benefit of splitting tfrecord file into shards?

I'm working on speech recognition with Tensorflow and plan to train LSTM NN with massive waves dataset. Because of the performance gains, I plan to use tfrecords. There are several examples on ...
striki70's user avatar
  • 281
28 votes
5 answers

VM image for data science projects

As there are numerous tools available for data science tasks, and it's cumbersome to install everything and build up a perfect system. Is there a Linux/Mac OS image with Python, R and other open-...
JeanVuda's user avatar
  • 421
28 votes
5 answers

Improve the speed of t-sne implementation in python for huge data

I would like to do dimensionality reduction on nearly 1 million vectors each with 200 dimensions(doc2vec). I am using TSNE ...
chmodsss's user avatar
  • 1,954
28 votes
6 answers

Machine learning techniques for estimating users' age based on Facebook sites they like

I have a database from my Facebook application and I am trying to use machine learning to estimate users' age based on what Facebook sites they like. There are three crucial characteristics of my ...
Wojciech Walczak's user avatar
28 votes
2 answers

Keras vs. tf.keras

I'm a bit confused in choosing between Keras (keras-team/keras) and tf.keras (tensorflow/tensorflow/python/keras/) for my new research project. There is a debate that Keras isn't owned by anyone, so ...
Mo-'s user avatar
  • 1,225
28 votes
4 answers

Scikit-learn: Getting SGDClassifier to predict as well as a Logistic Regression

A way to train a Logistic Regression is by using stochastic gradient descent, which scikit-learn offers an interface to. What I would like to do is take a scikit-learn's SGDClassifier and have it ...
hlin117's user avatar
  • 675
27 votes
3 answers

How to sum values grouped by two columns in pandas

I have a Pandas DataFrame like this: ...
Kevin's user avatar
  • 533
27 votes
4 answers

Is there a straightforward way to run pandas.DataFrame.isin in parallel?

I have a modeling and scoring program that makes heavy use of the DataFrame.isin function of pandas, searching through lists of facebook "like" records of ...
Therriault's user avatar
27 votes
4 answers

Word2Vec for Named Entity Recognition

I'm looking to use google's word2vec implementation to build a named entity recognition system. I've heard that recursive neural nets with back propagation through structure are well suited for named ...
Madison May's user avatar
  • 2,029
27 votes
1 answer

PyTorch vs. Tensorflow Fold

Both PyTorch and Tensorflow Fold are deep learning frameworks meant to deal with situations where the input data has non-uniform length or dimensions (that is, situations where dynamic graphs are ...
noe's user avatar
  • 23.8k
26 votes
7 answers

Is Python a viable language to do statistical analysis in?

I originally came from R, but Python seems to be the more common language these days. Ideally, I would do all my coding in Python as the syntax is easier and I've had more real life experience using ...
confused's user avatar
  • 488
26 votes
4 answers

Gradients for bias terms in backpropagation

I was trying to implement neural network from scratch to understand the maths behind it. My problem is completely related to backpropagation when we take derivative with respect to bias) and I ...
user34042's user avatar
  • 395
26 votes
10 answers

GraphViz not working when imported inside PydotPlus (`GraphViz's executables not found`)

I've been trying to make these packages work for quite some time now but with no success. Basically the error is: GraphViz's Executables not found EDIT: I had ...
Philippe Fanaro's user avatar

2 3 4 5