Questions tagged [pandas]

pandas is a python library for Panel Data manipulation and analysis, e.g. multidimensional time series and cross-sectional data sets commonly found in statistics, experimental science results, econometrics, or finance.

Filter by
Sorted by
Tagged with
43
votes
9answers
149k views

ValueError: Input contains NaN, infinity or a value too large for dtype('float32')

I got ValueError when predicting test data using a RandomForest model. My code: ...
4
votes
1answer
2k views

Correlation between specific columns of a data set

I have a CSV file which has 150 columns belonging to 7 categories but I want a correlation between 2 categories. The categories are movies and music, 12 and 19 columns respectively. Is there a way ...
4
votes
1answer
2k views

Cleaning input data with pd.get_dummies()

What is the advantage of converting a series like >>> df Color 0 Red 1 Blue 2 Green 3 Red To a multiple series like the below? ...
75
votes
11answers
41k views

Why do people prefer Pandas to SQL?

I've been using SQL since 1996, so I may be biased. I've used MySQL and SQLite 3 extensively, but have also used Microsoft SQL Server and Oracle. The vast majority of the operations I've seen done ...
8
votes
6answers
21k views

Python: Handling imbalance Classes in python Machine Learning

I have a dataset for which I am trying to predict target variables. ...
6
votes
2answers
1k views

Pandas grouped data to Bokeh graph

I'm having trouble graphing Pandas grouped data in Bokeh. ...
2
votes
3answers
569 views

Obtaining consistent one-hot encoding of train / production data

I'm building an app that will require user input. Currently, on the training set, I run the following code, in which data is a pandas dataframe with a combination ...
9
votes
1answer
5k views

How to binary encode multi-valued categorical variable from Pandas dataframe?

Suppose we have the following dataframe with multiple values for a certain column: categories 0 - ["A", "B"] 1 - ["B", "C", "D"] 2 - ["B", "D"] How can we ...
9
votes
3answers
10k views

Find the consecutive zeros in a DataFrame and do a conditional replacement

I have a dataset like this: Sample Dataframe ...
12
votes
2answers
53k views

How do I merge two data frames in Python Pandas?

I have two data frames df1 and df2 and I would like to merge them into a single data frame. It is as if df1 and df2 were created by splitting a single data frame down the center vertically, like ...
3
votes
2answers
12k views

Merging large CSV files in Pandas

I have two CSV files(each of the file size is in GBs). I am trying to merge the two CSV files but every time I try to it my computer hangs. Is there no way to merge the files in chunks in pandas ...
3
votes
2answers
5k views

Display Images (url) Inside Pandas Dataframe

I would like to display images (mostly jpg and png formats) directly from their url link inside a pandas dataframe. Imagine I already have the following dataframe: ...
3
votes
2answers
36k views

Could not convert string to float error on KDDCup99 dataset

I am trying to perform a comparison between 5 algorithms against the KDD Cup 99 dataset and the NSL-KDD datasets using Python and I am having an issue when trying to build and evaluate the models ...
3
votes
2answers
5k views

Perform k-means clustering over multiple columns

I am trying to perform k-means clustering on multiple columns. My data set is composed of 4 numerical columns and 1 categorical column. I already researched previous questions but the answers are not ...
1
vote
1answer
153 views

Any suitable way to describe the distributions of 2 Pandas Dataframes visually/graphically?

I am aware of the fact that the Pandas Dataframe's Statistical description can easily be obtained using df.describe(). I am having 2 dataframes of the same ...
0
votes
1answer
1k views

Groupby product, return tuple

I have found out the most profitable products in my dataframe by using: df.groupby('ProductName')['ProfitPerOrder'].sum().sort_values().tail() It gives me the ...
9
votes
3answers
4k views

Building a machine learning model to predict crop yields based on environmental data

I have a dataset containing data on temperature, precipitation and soybean yields for a farm for 10 years (2005 - 2014). I would like to predict yields for 2015 based on this data. Please note that ...
5
votes
2answers
624 views

Clustering Observations by String Sequences (Python/Pandas df)

I have a dataset consisting of approximately 2 million unique observations. It was initially a set of ID's and URLs. The goal is to cluster the ID's based on the URLs looked at. I transformed both ...
5
votes
2answers
936 views

Binary text classification with TfidfVectorizer gives ValueError: setting an array element with a sequence

I am using pandas and scikti-learn to do binary text classification using text features encoded using TfidfVectorizer on a DataFrame. Here is some dummy code that illustrates what I'm doing: ...
3
votes
1answer
380 views

Assigning values to missing target vector values in scikit-learn

I have a dataset containing data on temperature, precipitation, and soybean yields for a farm for 10 years (2005 - 2014). I would like to predict yields for 2015 based on this data. Please note that ...
3
votes
1answer
3k views

Performance issues when merging two dataframe columns into one on millions rows with Pandas

I am trying to merge two address columns into one and separate the resulting string with '--'. The dataset has 10 million rows and 33 columns - but the number of rows grows for a million or so a month....
3
votes
4answers
9k views

Check similarity between time series

I have time series of parameters A, B, C and D. All of ...
1
vote
3answers
117 views

creating a csv file from two csv files

I have two csv files: sub_compiler.to_csv('sub_compiler.csv') sub_compiler.head() and ...
1
vote
0answers
132 views

predict rank from physical measurements with various lengths

I have physical measurements with length 2*n, where the first vector represents a charge or a capacity (in Coulomb) $C$ and the second one is a voltage $V$. Let's call this measurement "forming". A ...
1
vote
0answers
833 views

pandas dataframes memory

I have a question about memory usage. I want to do 4 things: ...
1
vote
1answer
10k views

Remove Local Outliers from Dataframe using pandas

Could someone please suggest how to remove local outliers from the dataframe? I have the code to detect the local outliers, but I need help removing them(setting these values to zero) in the dataframe....