Questions tagged [pandas]

pandas is a python library for Panel Data manipulation and analysis, e.g. multidimensional time series and cross-sectional data sets commonly found in statistics, experimental science results, econometrics, or finance.

Filter by
Sorted by
Tagged with
4
votes
3answers
5k views

Obtaining consistent one-hot encoding of train / production data

I'm building an app that will require user input. Currently, on the training set, I run the following code, in which data is a pandas dataframe with a combination ...
85
votes
10answers
343k views

ValueError: Input contains NaN, infinity or a value too large for dtype('float32')

I got ValueError when predicting test data using a RandomForest model. My code: ...
128
votes
12answers
80k views

Why do people prefer Pandas to SQL?

I've been using SQL since 1996, so I may be biased. I've used MySQL and SQLite 3 extensively, but have also used Microsoft SQL Server and Oracle. The vast majority of the operations I've seen done ...
6
votes
1answer
2k views

Correlation between specific columns of a data set

I have a CSV file which has 150 columns belonging to 7 categories but I want a correlation between 2 categories. The categories are movies and music, 12 and 19 columns respectively. Is there a way ...
4
votes
1answer
3k views

Cleaning input data with pd.get_dummies()

What is the advantage of converting a series like >>> df Color 0 Red 1 Blue 2 Green 3 Red To a multiple series like the below? ...
2
votes
1answer
1k views

String handling by OneHotEncoder

I am reading everywhere on new questions and blogs that since version 0.20, OneHotEncoder is able to handle string features. Moreover, the documentation is what looks more ambiguous. Here are the ...
11
votes
6answers
27k views

Python: Handling imbalance Classes in python Machine Learning

I have a dataset for which I am trying to predict target variables. ...
6
votes
1answer
2k views

Pandas grouped data to Bokeh graph

I'm having trouble graphing Pandas grouped data in Bokeh. ...
5
votes
2answers
637 views

Is Label Encoding with arbitrary numbers ever useful at all?

From what I read online, there seems to be some confusion regarding the taxonomy and the terms used, so to avoid misunderstanding I'm going to define them here: Label Encoding - encoding a nominal ...
-1
votes
1answer
722 views

Recommend another product only on purchase history of users available

I have data in the following format. ...
44
votes
7answers
295k views

How do I compare columns in different data frames?

I would like to compare one column of a df with other df's. The columns are names and last names. I'd like to check if a person in one data frame is in another one.
10
votes
1answer
7k views

How to binary encode multi-valued categorical variable from Pandas dataframe?

Suppose we have the following dataframe with multiple values for a certain column: categories 0 - ["A", "B"] 1 - ["B", "C", "D"] 2 - ["B", "D"] How can we ...
12
votes
3answers
18k views

Find the consecutive zeros in a DataFrame and do a conditional replacement

I have a dataset like this: Sample Dataframe ...
11
votes
4answers
48k views

Export pandas to dictionary by combining multiple row values

I have a pandas dataframe df that looks like this ...
9
votes
2answers
47k views

Perform k-means clustering over multiple columns

I am trying to perform k-means clustering on multiple columns. My data set is composed of 4 numerical columns and 1 categorical column. I already researched previous questions but the answers are not ...
8
votes
6answers
20k views

Check similarity between time series

I have time series of parameters A, B, C and D. All of ...
7
votes
2answers
12k views

Display Images (url) Inside Pandas Dataframe

I would like to display images (mostly jpg and png formats) directly from their url link inside a pandas dataframe. Imagine I already have the following dataframe: ...
6
votes
1answer
5k views

Overfitting for minority class after SMOTE w/ random forests

I used SMOTE to make a predictive model, with class 1 having 1800 samples and 35000+ of class 0 samples. Hence, as per SMOTE, synthetic samples were created and the random forest was trained. However,...
6
votes
2answers
20k views

Merging large CSV files in pandas

I have two CSV files (each of the file size is in GBs) which I am trying to merge, but every time I do that, my computer hangs. Is there no way to merge them in chunks in pandas itself?
1
vote
1answer
17k views

Remove Local Outliers from Dataframe using pandas

Could someone please suggest how to remove local outliers from the dataframe? I have the code to detect the local outliers, but I need help removing them(setting these values to zero) in the dataframe....
14
votes
3answers
66k views

How do I merge two data frames in Python Pandas?

I have two data frames df1 and df2 and I would like to merge them into a single data frame. It is as if df1 and df2 were created by splitting a single data frame down the center vertically, like ...
3
votes
1answer
2k views

Pandas: How can I merge two dataframes? [closed]

I found (How do I merge two data frames in Python Pandas?), but do not get the expected result. I have these two CSV files: ...
3
votes
1answer
3k views

Using pandas get_dummies() on real world unseen data

I made a ML model, trained and tested it with my data containing categorical variables. To create dummy variables I used pd.get_dummies() before the split. I now ...
1
vote
2answers
1k views

Pros and cons of pandas or R for longitudinal data?

Note: I believe this question is not off-topic because it meets all of the criteria for subjective questions that are allowed. I would be happy to rephrase or clarify if others disagree I'm about to ...
0
votes
1answer
2k views

Groupby product, return tuple

I have found out the most profitable products in my dataframe by using: df.groupby('ProductName')['ProfitPerOrder'].sum().sort_values().tail() It gives me the ...
10
votes
3answers
4k views

Building a machine learning model to predict crop yields based on environmental data

I have a dataset containing data on temperature, precipitation and soybean yields for a farm for 10 years (2005 - 2014). I would like to predict yields for 2015 based on this data. Please note that ...
6
votes
2answers
1k views

Clustering Observations by String Sequences (Python/Pandas df)

I have a dataset consisting of approximately 2 million unique observations. It was initially a set of ID's and URLs. The goal is to cluster the ID's based on the URLs looked at. I transformed both ...
4
votes
5answers
12k views

Different number of features in train vs test

I'm doing the titanic exercise on kaggle and there is a categorical Cabin attribute that has a lot of different strings: C41, C11, B20 etc. (about 100). To be able to train my model I'm converting it ...
3
votes
1answer
4k views

Performance issues when merging two dataframe columns into one on millions rows with Pandas

I am trying to merge two address columns into one and separate the resulting string with '--'. The dataset has 10 million rows and 33 columns - but the number of rows grows for a million or so a month....
3
votes
1answer
422 views

Assigning values to missing target vector values in scikit-learn

I have a dataset containing data on temperature, precipitation, and soybean yields for a farm for 10 years (2005 - 2014). I would like to predict yields for 2015 based on this data. Please note that ...
3
votes
2answers
3k views

one-hot-encoding categorical data gives error

I am currently working on the Boston problem hosted on Kaggle. The dataset is nothing like the Titanic dataset. There are many categorical columns and I'm trying to one-hot-encode these columns. I'...
2
votes
0answers
861 views

pandas dataframes memory [closed]

I have a question about memory usage. I want to do 4 things: ...
1
vote
0answers
177 views

predict rank from physical measurements with various lengths

I have physical measurements with length 2*n, where the first vector represents a charge or a capacity (in Coulomb) $C$ and the second one is a voltage $V$. Let's call this measurement "forming". A ...
1
vote
1answer
74 views

Best practice - memory management when data wrangling

I was wondering what are the best resources to learn the best practice when it comes to memory management. For example, lets say I have the following code below: ...
1
vote
3answers
659 views

creating a csv file from two csv files

I have two csv files: sub_compiler.to_csv('sub_compiler.csv') sub_compiler.head() and ...
1
vote
1answer
262 views

General approach on time series for customer retention/churn in retail

I have a time series of data in the following form: ...
0
votes
1answer
2k views

Compare similarities between two data frames using more than one column in each data frame

This work started by comparing two columns in each data set in pandas. Previous research:here A lot of results online show how to compare 2 data frames with 1 column I'm trying to learn how to ...