People who code: we want your input. Take the Survey

Questions tagged [pandas]

pandas is a python library for Panel Data manipulation and analysis, e.g. multidimensional time series and cross-sectional data sets commonly found in statistics, experimental science results, econometrics, or finance.

Filter by
Sorted by
Tagged with
1
vote
2answers
42 views

How to combine two time dependent datasets?

I'm very new to data science so please be gentle. I have a dataset that contains record of occurrence of fire for the past 35 years (+-700.000 rows). Each date and time can have more than one ...
2
votes
2answers
194 views

What to replace outliers with? (supermarket transaction data)

I have a transaction dataset from a supermarket. Let's say the average spend is $50. I want to get each customer's average spend and rank them based on where they fall based on this $50 average spend. ...
1
vote
2answers
50 views

Does binning a time series with pd.qcut (using quantiles) create data leakage?

Let's say I want to predict whether a company will default on it's debt at some point in time (so binary classification) and one of the time series variables I'm using is the "revenue" of ...
0
votes
2answers
40 views

Finding the duplicate values between all columns and sort in new column with Pandas?

I have this DataFrame: ...
0
votes
1answer
72 views

Extract data from json format and paste to column using python

In my column with json data, I have this list I want to extract to column: ...
0
votes
1answer
28 views

whats wrong with the graph [closed]

I wanted to plot a bar graph which shows covid cases reported in different regions in different regions of USA. So here is my code: ...
1
vote
0answers
15 views

KNN with high-variance data [closed]

KNN doesn't work well with high-variance data, so how should I fit my data? Here is an example of what the data looks like:
1
vote
2answers
14k views

pandas - under a column, count the total number of a specific value, instead of using value_counts()

value_counts() function outputs the number of all unique values in a column, for example apple 3 orange 2 banana 1 I ...
4
votes
1answer
104 views

Why removing rows with NA values from the majority class improves model performance

I have an imbalanced dataset like so: df['y'].value_counts(normalize=True) * 100 ...
2
votes
1answer
30 views

How to fill df with data from another df - Python [closed]

I am trying to fill df with values form a different one. The first df is: The second df is: I want to add the column to first df having values from the second one to receive output: number trans ...
1
vote
0answers
23 views

How do I convert strings in a dataframe column to int or float [closed]

Encoding the column seperately works ,but when I try it on the dataset directly it throws an error. The error is something like this. ...
3
votes
2answers
3k views

Mean across every several rows in pandas

I have a table of features and labels where each row has a time stamp. Labels are categorical. They go in a batch where one label repeats several times. Batches with the same label do not have a ...
9
votes
4answers
44k views

How to Write Multiple Data Frames in an Excel Sheet

I have multiple data frames with same column names. I want to write them together to an excel sheet stacked vertically on top of each other. And between each, there will be a text occupying a row. ...
1
vote
1answer
30 views

How to bin a distribution data reported with different frequencies ( salaries ), showing mixed linearity and non-linearity?

I am researching on pay-scales, and wish to receive advise to treat data of salaries. Objective My interest is to approximate the salary corresponding to different hierarchical levels in an ...
0
votes
0answers
34 views

How to assign date into pandas Dataframe?

Im trying to code correlation between exctracted stock prices and data frame with "count" and "date" rows. My dataframe (cvd) looks like this: `` ...
0
votes
0answers
277 views

ValueError: array length 13996092 does not match index length 214200 for Kaggle submission

My model is based on the model linked here on Kaggle. Specifically, the following lines of code to generate an uploadable table to make my submission: ...
12
votes
3answers
151k views

Dataframe has no column names. How to add a header?

I am using a dataset to practice for building a decision tree classifier. Here is my code: ...
1
vote
1answer
51 views

Merging 2 pandas dataframes on key with duplicates but do not want it to result in two lines

I am trying to merge a "roster" where there are supposed to be several positions with one person per office. However, I have come across two instances where there are two people in one ...
0
votes
1answer
83 views

How do you see the element of a csv table with many columns (>30) which the names of its columns is more than 10 character in pandas?

How do you see in pandas the element of a csv table with many columns (>25) which the names of its columns is more than 10 character? I have 5000 rows and 32 columns and the label of some columns ...
1
vote
0answers
22 views

Are there decisive leaders in programming with tabular data?

What are the most effective bread-and-butter in-memory open source tabular data frameworks today? I have been working with tabular data for years with an in-house solution that integrates with Excel ...
0
votes
0answers
49 views

ValueError: Could not guess the value column! Please hand it to the function as an argument

I'm using the tsfresh library to extract features on my data but I keep getting an error: ...
0
votes
1answer
20 views

Fill the null values in the dataframe with condition

traindf[traindf['Gender'] == 'female']['Age'].fillna(value=femage,inplace=True) I've tried to update the null values in the age column in the dataframe with the ...
0
votes
0answers
24 views

How to use pivot_table on a dataframe? How to handle exceptions.UFuncTypeError?

I tried to use the pivot_table function on a dataframe. This was working at the beginning now I get all the time errors, and don't really get why. ...
0
votes
1answer
21 views

Is it ok to fill a pandas dataframe with NaN values?

Is it correct to fill a pandas dataframe with NaN values? In specific: if I have a dataframe with a user name and his age is it ok to fill the age column with int and NaN values. ...
13
votes
3answers
2k views

What do Python's pandas/matplotlib/seaborn bring to the table that Tableau does not?

I spent the past year learning Python. As a person who thought coding was impossible to learn for those outside of the CS/IT sphere, I was obviously gobsmacked by the power of a few lines of Python ...
0
votes
0answers
13 views

Forecasting Weekly Average Usage

In python/pandas what would you suggest as a couple of good basic methodologies to use to forecast weekly average usage of user activity on a given platform?
0
votes
0answers
12 views

Rating Sports Teams: How to perform least squares minimization with a constraint?

I am trying to rate NFL teams by minimizing the sum of squared errors subject to a constraint. My data looks like: ...
1
vote
1answer
3k views

AttributeError: 'numpy.ndarray' object has no attribute 'nan_to_num'

I'm trying to run a Random Forest model from sklearn but I keep getting an error: ValueError: Input contains NaN, infinity or a value too large for dtype('float32')....
0
votes
0answers
10 views

Surprise Library - RSME coming back as NaN

I'm using the Surprise library to train an SVD model based on some data I've collected from Steam. However, the RSME comes back as NaN when I try to train and test the model. For context, I'm using ...
5
votes
2answers
321 views

how to run bert's pretrained model word embeddings faster?

I'm trying to get word embeddings for clinical data using microsoft/pubmedbert. I have 3.6 million text rows. Converting texts to vectors for 10k rows takes around 30 minutes. So for 3.6 million rows, ...
-1
votes
3answers
638 views

How do I use lagged independent variable in statsmodel OLS regression?

If there is good reason to believe that an independent variable (x) has a lagged effect on dependent ...
6
votes
1answer
2k views

How does Seaborn calculate error bars when using estimators other than the arithmetic mean?

If I create a barplot using Seaborn and specify the geometric mean or the median as the estimator, does Seaborn know to use the appropriate standard error formula to create error bars?
1
vote
2answers
738 views

Does the Koalas library allow to use all Pandas machine learning libraries like Scikit-Learn, XGBoost, and TensorFlow?

I would like to implement a model based on some cleaned and prepared data set. I already have a bit of experience with PySpark, but from a data scientist's perspective it can be cumbersome to work ...
0
votes
1answer
14 views

How does for loops and elif work with python pandas?

the following code: ...
6
votes
2answers
19k views

Merging large CSV files in pandas

I have two CSV files (each of the file size is in GBs) which I am trying to merge, but every time I do that, my computer hangs. Is there no way to merge them in chunks in pandas itself?
1
vote
2answers
6k views

Replacing column values in pandas

I have a data frame which has three columns as shown below. There are about 10,000 entries in the data frame and there are duplicates as well. ...
2
votes
1answer
34 views

How to fill NaN based on the previous row that contains a specific value [closed]

I would like to fill the missing data using the value from the previous trading day of the same stock. In this example, the AAPL stock should be 100. I've tried with ...
0
votes
2answers
45 views

Building a summary string in a Pandas groupby (Possibly cross-tab or pivot-table question)

I'm a n00b in Python Pandas. I want to do a particular aggregation / grouping / cross-tab but I know so little of the terminology that I don't even know how to look this up. But here's what I want. ...
2
votes
2answers
291 views

Feature Engineering

I have a data frame with around 37,000 rows and 54 columns. Out of these 54 columns, two columns namely 'user_id' and 'mail_id' are provided in a very creepy format as shown below: ...
1
vote
1answer
156 views

Running scikit-learn with large volume

I need to run a Random Forest process with scikit-learn. To train the model, I have a database table with 10 million rows of features. The question is: what is the ...
6
votes
2answers
8k views

Plotting different values in pandas histogram with different colors

I am working on a dataset. The dataset consists of 16 different features each feature having values belonging to the set (0, 1, 2). In order to check the distribution of values in each column, I used <...
0
votes
2answers
67 views

Replace value of a column if the value of another column is a duplicate

I have a Pandas dataframe that contains three columns: ID, name and date. The name column is not unique and may contain duplicates. I want to change the date of the duplicated name to the earliest ...
0
votes
2answers
136 views

How to apply Geomertic mean in python

I am trying to built a function to calculate the Geometric mean for Growth column and Avg_growth. Year Company Name FH_PBIDT ...
81
votes
10answers
305k views
0
votes
1answer
188 views

Pandas replace column values by condition with averages based on a value in another column

I have a dataframe with people's CV data. Among others, there's a column with years of experience, and a column with age. Some people stated their age and experience in a way that experience > age. ...
0
votes
2answers
8k views

How to measure correlation between several categorical features and a numerical label in Python?

I have for a few weeks measured the time it takes for a product to be released through a automated release pipeline. I have several different categorical features such as "Product Category", "Product ...

1
3 4
5
6 7
21