Questions tagged [pandas]

pandas is a python library for Panel Data manipulation and analysis, e.g. multidimensional time series and cross-sectional data sets commonly found in statistics, experimental science results, econometrics, or finance.

Filter by
Sorted by
Tagged with
122
votes
1answer
172k views

Difference between isna() and isnull() in pandas

I have been using pandas for quite some time. But, I don't understand what's the difference between isna() and isnull(). And, ...
101
votes
10answers
57k views

Why do people prefer Pandas to SQL?

I've been using SQL since 1996, so I may be biased. I've used MySQL and SQLite 3 extensively, but have also used Microsoft SQL Server and Oracle. The vast majority of the operations I've seen done ...
68
votes
9answers
229k views

ValueError: Input contains NaN, infinity or a value too large for dtype('float32')

I got ValueError when predicting test data using a RandomForest model. My code: ...
51
votes
5answers
132k views

Convert a list of lists into a Pandas Dataframe

I am trying to convert a list of lists which looks like the following into a Pandas Dataframe ...
39
votes
5answers
79k views

Opening a 20GB file for analysis with pandas

I am currently trying to open a file with pandas and python for machine learning purposes it would be ideal for me to have them all in a DataFrame. Now The file is 18GB large and my RAM is 32 GB but I ...
37
votes
3answers
180k views

Calculation and Visualization of Correlation Matrix with Pandas

I have a pandas data frame with several entries, and I want to calculate the correlation between the income of some type of stores. There are a number of stores with income data, classification of ...
34
votes
5answers
207k views

How do I compare columns in different data frames?

I would like to compare one column of a df with other df's. The columns are names and last names. I'd like to check if a person in one data frame is in another one.
26
votes
4answers
14k views

Is there a straightforward way to run pandas.DataFrame.isin in parallel?

I have a modeling and scoring program that makes heavy use of the DataFrame.isin function of pandas, searching through lists of facebook "like" records of ...
24
votes
3answers
80k views

How to sum values grouped by two columns in pandas

I have a Pandas DataFrame like this: ...
23
votes
8answers
119k views

How to count the number of missing values in each row in Pandas dataframe?

How can I get the number of missing value in each row in Pandas dataframe. I would like to split dataframe to different dataframes which have same number of missing values in each row. Any suggestion?...
23
votes
4answers
91k views

How to fill missing value based on other columns in Pandas dataframe?

Suppose I have a 5*3 data frame in which third column contains missing value 1 2 3 4 5 NaN 7 8 9 3 2 NaN 5 6 NaN I hope to generate value for missing value based ...
22
votes
4answers
16k views

Is pandas now faster than data.table?

https://github.com/Rdatatable/data.table/wiki/Benchmarks-%3A-Grouping The data.table benchmarks hasn't been updated since 2014. I heard somewhere that Pandas is ...
21
votes
3answers
14k views

is there any data tidying tool for python/pandas similar to R tidyr tool?

I'm working on a Kaggle challenge where some variables are represented by rows instead of columns (Telstra Network Disruption). I am currently searching for the equivalent of ...
19
votes
5answers
55k views

make seaborn heatmap bigger

I create a corr() df out of an original df. The corr() df came out 70 X 70 and it is impossible to visualize the heatmap... <...
16
votes
2answers
961 views

Where in the workflow should we deal with missing data?

I'm building a workflow for creating machine learning models (in my case, using Python's pandas and sklearn packages) from data ...
15
votes
3answers
25k views

Pandas Dataframe to DMatrix

I am trying to run xgboost in scikit learn. And I only use Pandas to load data into dataframe. How am i supposed to use pandas df with xgboost. I am confused by the DMatrix routine required to run ...
14
votes
1answer
71k views

Convert a pandas column of int to timestamp datatype

I have a dataframe that among other things, contains a column of the number of milliseconds passed since 1970-1-1. I need to convert this column of ints to timestamp data, so I can then ultimately ...
13
votes
5answers
12k views

Advantages of pandas dataframe to regular relational database

In Data Science, many seem to be using pandas dataframes as the datastore. What are the features of pandas that make it a superior datastore compared to regular relational databases like MySQL, which ...
13
votes
3answers
59k views

How do I merge two data frames in Python Pandas?

I have two data frames df1 and df2 and I would like to merge them into a single data frame. It is as if df1 and df2 were created by splitting a single data frame down the center vertically, like ...
12
votes
3answers
23k views

Mass convert categorical columns in Pandas (not one-hot encoding)

I have pandas dataframe with tons of categorical columns, which I am planning to use in decision tree with scikit-learn. I need to convert them to numerical values (not one hot vectors). I can do it ...
11
votes
3answers
2k views

What do Python's pandas/matplotlib/seaborn bring to the table that Tableau does not?

I spent the past year learning Python. As a person who thought coding was impossible to learn for those outside of the CS/IT sphere, I was obviously gobsmacked by the power of a few lines of Python ...
11
votes
3answers
14k views

Find the consecutive zeros in a DataFrame and do a conditional replacement

I have a dataset like this: Sample Dataframe ...
11
votes
1answer
88k views

How to plot two columns of single DataFrame on Y axis

i have two DataFrames(Action,Comedy). Action contains two columns(year,rating) ratings columns contains average rating with respect to year. Comedy Dataframe contains same two columns with different ...
11
votes
1answer
54k views

How to deal with TypeError: ufunc 'isnan' not supported for the input types [closed]

I have dealt with all the Nan values in the features dataframe, then why I am still getting this error? ...
10
votes
3answers
45k views

after grouping to minimum value in pandas, how to display the matching row result entirely along min() value

The dataframe contains ...
10
votes
3answers
18k views

Improve Pandas dataframe filtering speed

I have a dataset with 19 columns and about 250k rows. I have worked with bigger datasets, but this time, Pandas decided to play with my nerves. I tried to split the original dataset into 3 sub-...
10
votes
3answers
75k views

How to group identical values and count their frequency in Python?

Newbie to analytics with Python so please be gentle :-) I couldn't find the answer to this question - apologies if it is already answered elsewhere in a different format. I have a dataset of ...
10
votes
2answers
82k views

Creating new columns by iterating over rows in pandas dataframe

I have a pandas data frame (X11) like this: In actual I have 99 columns up to dx99 ...
10
votes
3answers
4k views

Building a machine learning model to predict crop yields based on environmental data

I have a dataset containing data on temperature, precipitation and soybean yields for a farm for 10 years (2005 - 2014). I would like to predict yields for 2015 based on this data. Please note that ...
9
votes
4answers
34k views

Export pandas to dictionary by combining multiple row values

I have a pandas dataframe df that looks like this ...
9
votes
1answer
28k views

Difference between interpolate() and fillna() in pandas

Since interpolate and fillna method does the same work of filling na values. What is the basic difference between the two. What is the significance of having these two different methods?? Can anyone ...
9
votes
6answers
24k views

Python: Handling imbalance Classes in python Machine Learning

I have a dataset for which I am trying to predict target variables. ...
9
votes
2answers
18k views

Counting indexes in pandas

I feel like this is a rudimentary question but I'm very new to this and just haven't been able to crack it / find the answer. Ultimately what I'm trying to do here is to count unique values on a ...
9
votes
1answer
28k views

How to scrape a table from a webpage?

I need to scrape a table off of a webpage and put it into a pandas data frame. But I am not being able to do it. Let me first give you a hint of how the table is encoded into html document. ...
9
votes
1answer
6k views

How to binary encode multi-valued categorical variable from Pandas dataframe?

Suppose we have the following dataframe with multiple values for a certain column: categories 0 - ["A", "B"] 1 - ["B", "C", "D"] 2 - ["B", "D"] How can we ...
8
votes
4answers
2k views

How to replace NaN values for image data?

My data set has a total of 200 columns, where each column corresponds to the same pixel in all of my images. In total, I have 48,500 rows. The labels for the data range from 0-9. The data looks ...
8
votes
2answers
58k views

Mapping column values of one DataFrame to another DataFrame using a key with different header names

I have two data frames df1 and df2 which look something like this. ...
8
votes
2answers
12k views

Using TF-IDF with other features in SKLearn

What is the best/correct way to combine text analysis with other features? For example, I have a dataset with some text but also other features/categories. SKlearn's TF-IDF vectoriser transforms text ...
8
votes
1answer
35k views

Am I doing a log transformation of data correctly?

I'm doing some exploratory data analysis on some data and I get these histograms: That looks like a candidate for a log transformation on the data, so I run the following Python code to transform the ...
8
votes
4answers
22k views

Pandas: how can I create multi-level columns

I have a pandas DataFrame which has the following columns: n_0 n_1 p_0 p_1 e_0 e_1 I want to transform it to have columns and sub-columns: ...
7
votes
3answers
3k views

Can we remove features that have zero-correlation with the target/label?

So I draw a pairplot/heatmap from the feature correlations of a dataset and see a set of features that bears Zero-correlations both with: every other feature and also with the target/label ....
7
votes
2answers
17k views

Pandas merge column duplicate and sum value [closed]

How to merge duplicate column and sum their value? What I have A 30 A 40 B 50 What I need A 70 B 50 DF for this ...
7
votes
3answers
23k views

Split a list of values into columns of a dataframe?

I am new to python and stuck at a particular problem involving dataframes. The image has a sample column, however the data is not consistent. There are also some floats and NAN. I need these to be ...
7
votes
2answers
5k views

Struggling to integrate sklearn and pandas in simple Kaggle task

I'm trying to use the sklearn_pandas module to extend the work I do in pandas and dip a toe into machine learning but I'm struggling with an error I don't really understand how to fix. I was working ...
7
votes
1answer
21k views

How does class_weights work in RandomForestClassifier

I'm facing a problem with unbalanced classes, and have tried out a couple of methods like over and under sampling. However, my cross validation mean comes out to be only 0.4 and my confusion matrix ...
7
votes
3answers
155 views

How to use a one-hot encoded nominal feature in a classifier in Scikit Learn?

I'm working on a genre classification problem on a songs dataset. Since genre is a nominal feature, I used sklearn's LabelBinarizer to get the one-hot encoding for this feature for every row in the ...
6
votes
4answers
32k views

How to plot multiple variables with Pandas and Bokeh

I'm new to Pandas and Bokeh; I'd to create a bar plot that shows two different variables next to each other for comparison. For instance, with the following Pandas data frame, I'd like to see how ...
6
votes
3answers
48k views

Pandas change value of a column based another column condition

I have values in column1, I have columns in column2. What I want to achieve: Condition: where column2 == 2 leave to be 2 if column1 < 30 elsif change to 3 if column1 > 90. Here is what i did so ...
6
votes
1answer
9k views

Summary statistics by category using Python

I have a datset with Scores and Categories and I would like to calculate the summary statistics for each of these categories. The data look something like this: ...
6
votes
3answers
95k views

Dataframe has no column names. How to add a header?

I am using a dataset to practice for building a decision tree classifier. Here is my code: ...

1
2 3 4 5
19