Questions tagged [pandas]

pandas is a python library for Panel Data manipulation and analysis, e.g. multidimensional time series and cross-sectional data sets commonly found in statistics, experimental science results, econometrics, or finance.

Filter by
Sorted by
Tagged with
82
votes
1answer
103k views

Difference between isna() and isnull() in pandas

I have been using pandas for quite some time. But, I don't understood what's the difference between isna() and isnull() in ...
75
votes
11answers
41k views

Why do people prefer Pandas to SQL?

I've been using SQL since 1996, so I may be biased. I've used MySQL and SQLite 3 extensively, but have also used Microsoft SQL Server and Oracle. The vast majority of the operations I've seen done ...
43
votes
9answers
146k views

ValueError: Input contains NaN, infinity or a value too large for dtype('float32')

I got ValueError when predicting test data using a RandomForest model. My code: ...
35
votes
3answers
161k views

Calculation and Visualization of Correlation Matrix with Pandas

I have a pandas data frame with several entries, and I want to calculate the correlation between the income of some type of stores. There are a number of stores with income data, classification of ...
33
votes
5answers
60k views

Opening a 20GB file for analysis with pandas

I am currently trying to open a file with pandas and python for machine learning purposes it would be ideal for me to have them all in a DataFrame. Now The file is 18GB large and my RAM is 32 GB but I ...
30
votes
5answers
90k views

Convert a list of lists into a Pandas Dataframe

I am trying to convert a list of lists which looks like the following into a Pandas Dataframe ...
25
votes
4answers
13k views

Is there a straightforward way to run pandas.DataFrame.isin in parallel?

I have a modeling and scoring program that makes heavy use of the DataFrame.isin function of pandas, searching through lists of facebook "like" records of ...
22
votes
5answers
141k views

How do I compare columns in different data frames?

I would like to compare one column of a df with other df's. The columns are names and last names. I'd like to check if a person in one data frame is in another one.
21
votes
2answers
69k views

How to sum values grouped by two columns in pandas

I have a Pandas DataFrame like this: ...
18
votes
3answers
11k views

is there any data tidying tool for python/pandas similar to R tidyr tool?

I'm working on a Kaggle challenge where some variables are represented by rows instead of columns (Telstra Network Disruption). I am currently searching for the equivalent of gather(), separate() and ...
16
votes
5answers
40k views

make seaborn heatmap bigger

I create a corr() df out of an original df. The corr() df came out 70 X 70 and it is impossible to visualize the heatmap... <...
16
votes
2answers
905 views

Where in the workflow should we deal with missing data?

I'm building a workflow for creating machine learning models (in my case, using Python's pandas and sklearn packages) from data ...
15
votes
4answers
58k views

How to fill missing value based on other columns in Pandas dataframe?

Suppose I have a 5*3 data frame in which third column contains missing value 1 2 3 4 5 NaN 7 8 9 3 2 NaN 5 6 NaN I hope to generate value for missing value based ...
13
votes
4answers
12k views

Is pandas now faster than data.table?

https://github.com/Rdatatable/data.table/wiki/Benchmarks-%3A-Grouping The data.table benchmarks hasn't been updated since 2014. I heard somewhere that Pandas is ...
13
votes
1answer
58k views

Convert a pandas column of int to timestamp datatype

I have a dataframe that among other things, contains a column of the number of milliseconds passed since 1970-1-1. I need to convert this column of ints to timestamp data, so I can then ultimately ...
12
votes
7answers
93k views

How to count the number of missing values in each row in Pandas dataframe?

How can I get the number of missing value in each row in Pandas dataframe. I would like to split dataframe to different dataframes which have same number of missing values in each row. Any suggestion?...
12
votes
2answers
53k views

How do I merge two data frames in Python Pandas?

I have two data frames df1 and df2 and I would like to merge them into a single data frame. It is as if df1 and df2 were created by splitting a single data frame down the center vertically, like ...
11
votes
3answers
20k views

Mass convert categorical columns in Pandas (not one-hot encoding)

I have pandas dataframe with tons of categorical columns, which I am planning to use in decision tree with scikit-learn. I need to convert them to numerical values (not one hot vectors). I can do it ...
11
votes
4answers
9k views

Advantages of pandas dataframe to regular relational database

In Data Science, many seem to be using pandas dataframes as the datastore. What are the features of pandas that make it a superior datastore compared to regular relational databases like MySQL, which ...
10
votes
2answers
75k views

Creating new columns by iterating over rows in pandas dataframe

I have a pandas data frame (X11) like this: In actual I have 99 columns up to dx99 ...
9
votes
3answers
19k views

Pandas Dataframe to DMatrix

I am trying to run xgboost in scikit learn. And I only use Pandas to load data into dataframe. How am i supposed to use pandas df with xgboost. I am confused by the DMatrix routine required to run ...
9
votes
2answers
64k views

How to group identical values and count their frequency in Python?

Newbie to analytics with Python so please be gentle :-) I couldn't find the answer to this question - apologies if it is already answered elsewhere in a different format. I have a dataset of ...
9
votes
3answers
10k views

Find the consecutive zeros in a DataFrame and do a conditional replacement

I have a dataset like this: Sample Dataframe ...
9
votes
1answer
24k views

Difference between interpolate() and fillna() in pandas

Since interpolate and fillna method does the same work of filling na values. What is the basic difference between the two. What is the significance of having these two different methods?? Can anyone ...
9
votes
1answer
68k views

How to plot two columns of single DataFrame on Y axis

i have two DataFrames(Action,Comedy). Action contains two columns(year,rating) ratings columns contains average rating with respect to year. Comedy Dataframe contains same two columns with different ...
9
votes
1answer
5k views

How to binary encode multi-valued categorical variable from Pandas dataframe?

Suppose we have the following dataframe with multiple values for a certain column: categories 0 - ["A", "B"] 1 - ["B", "C", "D"] 2 - ["B", "D"] How can we ...
9
votes
3answers
4k views

Building a machine learning model to predict crop yields based on environmental data

I have a dataset containing data on temperature, precipitation and soybean yields for a farm for 10 years (2005 - 2014). I would like to predict yields for 2015 based on this data. Please note that ...
8
votes
3answers
12k views

Improve Pandas dataframe filtering speed

I have a dataset with 19 columns and about 250k rows. I have worked with bigger datasets, but this time, Pandas decided to play with my nerves. I tried to split the original dataset into 3 sub-...
8
votes
6answers
21k views

Python: Handling imbalance Classes in python Machine Learning

I have a dataset for which I am trying to predict target variables. ...
8
votes
2answers
15k views

Counting indexes in pandas

I feel like this is a rudimentary question but I'm very new to this and just haven't been able to crack it / find the answer. Ultimately what I'm trying to do here is to count unique values on a ...
8
votes
1answer
28k views

Am I doing a log transformation of data correctly?

I'm doing some exploratory data analysis on some data and I get these histograms: That looks like a candidate for a log transformation on the data, so I run the following Python code to transform the ...
8
votes
4answers
17k views

Pandas: how can I create multi-level columns

I have a pandas DataFrame which has the following columns: n_0 n_1 p_0 p_1 e_0 e_1 I want to transform it to have columns and sub-columns: ...
7
votes
4answers
32k views

after grouping to minimum value in pandas, how to display the matching row result entirely along min() value

The dataframe contains ...
7
votes
4answers
798 views

How to replace NaN values for image data?

My data set has a total of 200 columns, where each column corresponds to the same pixel in all of my images. In total, I have 48,500 rows. The labels for the data range from 0-9. The data looks ...
7
votes
2answers
5k views

Struggling to integrate sklearn and pandas in simple Kaggle task

I'm trying to use the sklearn_pandas module to extend the work I do in pandas and dip a toe into machine learning but I'm struggling with an error I don't really understand how to fix. I was working ...
7
votes
2answers
9k views

Using TF-IDF with other features in SKLearn

What is the best/correct way to combine text analysis with other features? For example, I have a dataset with some text but also other features/categories. SKlearn's TF-IDF vectoriser transforms text ...
6
votes
4answers
27k views

How to plot multiple variables with Pandas and Bokeh

I'm new to Pandas and Bokeh; I'd to create a bar plot that shows two different variables next to each other for comparison. For instance, with the following Pandas data frame, I'd like to see how ...
6
votes
3answers
25k views

Export pandas to dictionary by combining multiple row values

I have a pandas dataframe df that looks like this ...
6
votes
1answer
7k views

Summary statistics by category using Python

I have a datset with Scores and Categories and I would like to calculate the summary statistics for each of these categories. The data look something like this: ...
6
votes
1answer
15k views

How does class_weights work in RandomForestClassifier

I'm facing a problem with unbalanced classes, and have tried out a couple of methods like over and under sampling. However, my cross validation mean comes out to be only 0.4 and my confusion matrix ...
6
votes
3answers
124 views

What is the best algorithm/solution for predicting the following?

I have a dataset that comprises 76 countries, and 6 columns of distinct quantitative variables, which are the mean values of that variable relative to each country: If I were to take a random sample ...
6
votes
1answer
20k views

How to scrape a table from a webpage?

I need to scrape a table off of a webpage and put it into a pandas data frame. But I am not being able to do it. Let me first give you a hint of how the table is encoded into html document. ...
6
votes
2answers
9k views

pandas count values for last 7 days from each date

There are two Dataframes. First is like this: ...
6
votes
1answer
993 views

Pandas v. SFrame in learning data science

I'm taking a machine learning course that introduces the related assignments as such that a student may use either Pandas or SFrame in solving them. As a beginner, it's hard for me to assess which ...
6
votes
2answers
1k views

Pandas grouped data to Bokeh graph

I'm having trouble graphing Pandas grouped data in Bokeh. ...
6
votes
3answers
117 views

How to use a one-hot encoded nominal feature in a classifier in Scikit Learn?

I'm working on a genre classification problem on a songs dataset. Since genre is a nominal feature, I used sklearn's LabelBinarizer to get the one-hot encoding for this feature for every row in the ...
5
votes
1answer
12k views

How to calculate Cumulative Sum with Groupby in Python?

I am trying to calculate cumulative sum with groupby using Pandas's DataFrame. However, I don't get expected output. My Source Code: ...
5
votes
1answer
761 views

Is shuffling training data beneficial for machine learning?

I was curious to know if shuffling ML training data is beneficial to better results? Sorry not a lot of wisdom here, but I have been reading a post from pythonprogramming.net for this topic. I ...
5
votes
1answer
27k views

How to deal with TypeError: ufunc 'isnan' not supported for the input types [closed]

I have dealt with all the Nan values in the features dataframe, then why I am still getting this error? ...
5
votes
1answer
3k views

How to transform raw data to fixed-frequency time series?

How to transform raw data to fixed-frequency time series? For example I have the following raw data in DataFrame ...