Questions tagged [pandas]
pandas is a python library for Panel Data manipulation and analysis, e.g. multidimensional time series and cross-sectional data sets commonly found in statistics, experimental science results, econometrics, or finance.
1,041
questions
1
vote
2answers
42 views
How to combine two time dependent datasets?
I'm very new to data science so please be gentle.
I have a dataset that contains record of occurrence of fire for the past 35 years (+-700.000 rows). Each date and time can have more than one ...
0
votes
1answer
327 views
recognizing the correct word & “Set type is unordered”-error in python-pandas
My Data Set (CSV):
...
2
votes
2answers
194 views
What to replace outliers with? (supermarket transaction data)
I have a transaction dataset from a supermarket. Let's say the average spend is $50.
I want to get each customer's average spend and rank them based on where they fall based on this $50 average spend.
...
1
vote
2answers
50 views
Does binning a time series with pd.qcut (using quantiles) create data leakage?
Let's say I want to predict whether a company will default on it's debt at some point in time (so binary classification) and one of the time series variables I'm using is the "revenue" of ...
0
votes
2answers
40 views
Finding the duplicate values between all columns and sort in new column with Pandas?
I have this DataFrame:
...
0
votes
1answer
72 views
Extract data from json format and paste to column using python
In my column with json data, I have this list I want to extract to column:
...
0
votes
1answer
28 views
whats wrong with the graph [closed]
I wanted to plot a bar graph which shows covid cases reported in different regions in different regions of USA.
So here is my code:
...
1
vote
0answers
15 views
KNN with high-variance data [closed]
KNN doesn't work well with high-variance data, so how should I fit my data?
Here is an example of what the data looks like:
1
vote
2answers
14k views
pandas - under a column, count the total number of a specific value, instead of using value_counts()
value_counts()
function outputs the number of all unique values in a column, for example
apple 3
orange 2
banana 1
I ...
4
votes
1answer
104 views
Why removing rows with NA values from the majority class improves model performance
I have an imbalanced dataset like so:
df['y'].value_counts(normalize=True) * 100
...
2
votes
1answer
30 views
How to fill df with data from another df - Python [closed]
I am trying to fill df with values form a different one.
The first df is:
The second df is:
I want to add the column to first df having values from the second one to receive output:
number
trans
...
1
vote
0answers
23 views
How do I convert strings in a dataframe column to int or float [closed]
Encoding the column seperately works ,but when I try it on the dataset directly it throws an error.
The error is something like this.
...
3
votes
2answers
3k views
Mean across every several rows in pandas
I have a table of features and labels where each row has a time stamp. Labels are categorical. They go in a batch where one label repeats several times. Batches with the same label do not have a ...
9
votes
4answers
44k views
How to Write Multiple Data Frames in an Excel Sheet
I have multiple data frames with same column names. I want to write them together to an excel sheet stacked vertically on top of each other. And between each, there will be a text occupying a row. ...
1
vote
1answer
30 views
How to bin a distribution data reported with different frequencies ( salaries ), showing mixed linearity and non-linearity?
I am researching on pay-scales, and wish to receive advise to treat data of salaries.
Objective
My interest is to approximate the salary corresponding to different hierarchical levels in an ...
0
votes
0answers
34 views
How to assign date into pandas Dataframe?
Im trying to code correlation between exctracted stock prices and data frame with "count" and "date" rows. My dataframe (cvd) looks like this:
``
...
0
votes
0answers
277 views
ValueError: array length 13996092 does not match index length 214200 for Kaggle submission
My model is based on the model linked here on Kaggle.
Specifically, the following lines of code to generate an uploadable table to make my submission:
...
12
votes
3answers
151k views
Dataframe has no column names. How to add a header?
I am using a dataset to practice for building a decision tree classifier.
Here is my code:
...
1
vote
1answer
51 views
Merging 2 pandas dataframes on key with duplicates but do not want it to result in two lines
I am trying to merge a "roster" where there are supposed to be several positions with one person per office. However, I have come across two instances where there are two people in one ...
0
votes
1answer
83 views
How do you see the element of a csv table with many columns (>30) which the names of its columns is more than 10 character in pandas?
How do you see in pandas the element of a csv table with many columns (>25) which the names of its columns is more than 10 character?
I have 5000 rows and 32 columns and the label of some columns ...
1
vote
0answers
22 views
Are there decisive leaders in programming with tabular data?
What are the most effective bread-and-butter in-memory open source tabular data frameworks today? I have been working with tabular data for years with an in-house solution that integrates with Excel ...
0
votes
0answers
49 views
ValueError: Could not guess the value column! Please hand it to the function as an argument
I'm using the tsfresh library to extract features on my data but I keep getting an error:
...
0
votes
1answer
20 views
Fill the null values in the dataframe with condition
traindf[traindf['Gender'] == 'female']['Age'].fillna(value=femage,inplace=True)
I've tried to update the null values in the age column in the dataframe with the ...
0
votes
0answers
24 views
How to use pivot_table on a dataframe? How to handle exceptions.UFuncTypeError?
I tried to use the pivot_table function on a dataframe. This was working at the beginning now I get all the time errors, and don't really get why.
...
0
votes
1answer
21 views
Is it ok to fill a pandas dataframe with NaN values?
Is it correct to fill a pandas dataframe with NaN values? In specific: if I have a dataframe with a user name and his age is it ok to fill the age column with int and NaN values.
...
13
votes
3answers
2k views
What do Python's pandas/matplotlib/seaborn bring to the table that Tableau does not?
I spent the past year learning Python. As a person who thought coding was impossible to learn for those outside of the CS/IT sphere, I was obviously gobsmacked by the power of a few lines of Python ...
0
votes
0answers
13 views
Forecasting Weekly Average Usage
In python/pandas what would you suggest as a couple of good basic methodologies to use to forecast weekly average usage of user activity on a given platform?
0
votes
2answers
307 views
How to apply Normalisation using the MinMaxScaler() to all Columns, but Exclude the Categorical?
Below, I have the following datatset:
sample_df.head(2)
...
0
votes
0answers
12 views
Rating Sports Teams: How to perform least squares minimization with a constraint?
I am trying to rate NFL teams by minimizing the sum of squared errors subject to a constraint. My data looks like:
...
1
vote
1answer
3k views
AttributeError: 'numpy.ndarray' object has no attribute 'nan_to_num'
I'm trying to run a Random Forest model from sklearn but I keep getting an error: ValueError: Input contains NaN, infinity or a value too large for dtype('float32')....
0
votes
0answers
10 views
Surprise Library - RSME coming back as NaN
I'm using the Surprise library to train an SVD model based on some data I've collected from Steam. However, the RSME comes back as NaN when I try to train and test the model. For context, I'm using ...
5
votes
2answers
321 views
how to run bert's pretrained model word embeddings faster?
I'm trying to get word embeddings for clinical data using microsoft/pubmedbert.
I have 3.6 million text rows. Converting texts to vectors for 10k rows takes around 30 minutes. So for 3.6 million rows, ...
-1
votes
3answers
638 views
How do I use lagged independent variable in statsmodel OLS regression?
If there is good reason to believe that an independent variable (x) has a lagged effect on dependent ...
6
votes
1answer
2k views
How does Seaborn calculate error bars when using estimators other than the arithmetic mean?
If I create a barplot using Seaborn and specify the geometric mean or the median as the estimator, does Seaborn know to use the appropriate standard error formula to create error bars?
1
vote
2answers
738 views
Does the Koalas library allow to use all Pandas machine learning libraries like Scikit-Learn, XGBoost, and TensorFlow?
I would like to implement a model based on some cleaned and prepared data set. I already have a bit of experience with PySpark, but from a data scientist's perspective it can be cumbersome to work ...
0
votes
1answer
14 views
6
votes
2answers
19k views
Merging large CSV files in pandas
I have two CSV files (each of the file size is in GBs) which I am trying to merge, but every time I do that, my computer hangs. Is there no way to merge them in chunks in pandas itself?
1
vote
2answers
6k views
Replacing column values in pandas
I have a data frame which has three columns as shown below. There are about 10,000 entries in the data frame and there are duplicates as well.
...
2
votes
1answer
34 views
How to fill NaN based on the previous row that contains a specific value [closed]
I would like to fill the missing data using the value from the previous trading day of the same stock. In this example, the AAPL stock should be 100.
I've tried with ...
1
vote
4answers
30k views
String Values in a data frame in Pandas
Suppose I have a data frame like this :
...
0
votes
2answers
45 views
Building a summary string in a Pandas groupby (Possibly cross-tab or pivot-table question)
I'm a n00b in Python Pandas.
I want to do a particular aggregation / grouping / cross-tab but I know so little of the terminology that I don't even know how to look this up.
But here's what I want.
...
2
votes
2answers
291 views
Feature Engineering
I have a data frame with around 37,000 rows and 54 columns. Out of these 54 columns, two columns namely 'user_id' and 'mail_id' are provided in a very creepy format as shown below:
...
1
vote
1answer
156 views
Running scikit-learn with large volume
I need to run a Random Forest process with scikit-learn. To train the model, I have a database table with 10 million rows of features. The question is: what is the ...
2
votes
1answer
6k views
Plotting two pandas data frame columns against each other
I have a data frame like this:
...
6
votes
2answers
8k views
Plotting different values in pandas histogram with different colors
I am working on a dataset. The dataset consists of 16 different features each feature having values belonging to the set (0, 1, 2). In order to check the distribution of values in each column, I used <...
0
votes
2answers
67 views
Replace value of a column if the value of another column is a duplicate
I have a Pandas dataframe that contains three columns: ID, name and date. The name column is not unique and may contain duplicates. I want to change the date of the duplicated name to the earliest ...
0
votes
2answers
136 views
How to apply Geomertic mean in python
I am trying to built a function to calculate the Geometric mean for Growth column and Avg_growth.
Year
Company Name
FH_PBIDT
...
81
votes
10answers
305k views
ValueError: Input contains NaN, infinity or a value too large for dtype('float32')
I got ValueError when predicting test data using a RandomForest model.
My code:
...
0
votes
1answer
188 views
Pandas replace column values by condition with averages based on a value in another column
I have a dataframe with people's CV data.
Among others, there's a column with years of experience, and a column with age.
Some people stated their age and experience in a way that experience > age. ...
0
votes
2answers
8k views
How to measure correlation between several categorical features and a numerical label in Python?
I have for a few weeks measured the time it takes for a product to be released through a automated release pipeline. I have several different categorical features such as "Product Category", "Product ...
