Questions tagged [pandas]
pandas is a python library for Panel Data manipulation and analysis, e.g. multidimensional time series and cross-sectional data sets commonly found in statistics, experimental science results, econometrics, or finance.
1,004
questions
0
votes
1answer
20 views
Python: How to plot time interval from a Dataframe in Pandas
I have the a dataframe(df) which has the data of a Job being executed at different time intervals. It includes the following details about the execution of a job:
...
0
votes
2answers
16 views
I want to re frame my dataframe to find the total number of products for each id. Do i use Pivot table for that in Pandas? [closed]
So basically I had performed GroupBy to find Retailer ID wise, the product, number of products and total Sales Amount- for which I used the following:
...
1
vote
0answers
18 views
Python: How to convert Pandas Dataframe Row Values to individual columns? [migrated]
I have the following dataframe which has the data of different jobs start and end time at different intervals. A small part of dataframe is shown below.
Dataframe(df):
...
0
votes
0answers
12 views
Extracting structured data from semi structured data
I want to use machine learning and NLP to convert semi-structured data in text files to structured data by predicting the patterns in the files and splitting the fields for example if I have a text ...
1
vote
1answer
42 views
Plot Two Categorical Variables
How to plot two categorical variables in Python or using any library? I want to plot the Playing Role of a Cricketer (Batsman, Bowler, etc.) VS Bought_By (Franchise Names, e.g., CSK, DC, etc.). The ...
0
votes
0answers
18 views
What are the use-cases for heapsort option in numpy.sort()?
Could anybody share experience about use cases of the heapsort option in numpy.sort() (or pandas.DataFrame.sort_values()) in the data science field?
I mean if the <...
1
vote
1answer
19 views
txt file into pandas
I have a txt file where values are randomly separated, sometimes there is a single space, and sometimes there are many. I am importing the file into pandas with <...
0
votes
1answer
18 views
IterativeImputer Evaluation
I am having a hard time evaluating my model of imputation.
I used an iterative imputer model to fill in the missing values in all four columns.
For the model on the iterative imputer, I am using a ...
-1
votes
1answer
19 views
Multiplying a dataframe by a larger one [closed]
I have two dataframes df1 and df2 with the same columns but not the same row number.
I want to multiply them element-wise such ...
0
votes
1answer
27 views
0
votes
0answers
13 views
Accessing group elements in groupby
I had a doubt about groupby operations. Lets suppose I have grouped my data based on one column and got 5 groups as output. I know that we can iterate over these groups and apply functions to them as ...
-2
votes
1answer
35 views
ImportError: Pandas requires version '0.3.0' or newer of 's3fs' [closed]
I'm trying to read files from S3, using boto3, pandas, anaconda, but I have the following error:
...
-1
votes
1answer
15 views
.1 after column name while read csv file from pandas
I am getting .1 after column name while read csv file from pandas
0
votes
0answers
21 views
1
vote
1answer
36 views
Perform PCA on columns of different length
I have about 20-30 columns, all with different lengths. The first column has 25000 rows, the second column has 19000 rows, and it's different for all the columns.
All are the survey data with 0 (No),1(...
0
votes
2answers
24 views
Sort Pandas Dataframe per column
I have a dataset with age + another 14 variables. I have created 13 bins representing different age groups like so:
...
1
vote
1answer
19 views
How to Iterate over rows in a dataframe [closed]
So I've been trying to iterate over rows of my dataframe and my goal is to find matching rows based on two particular columns (say C and P). Then i need to do some manipulations as well in the data ...
0
votes
1answer
10 views
Dealing with dates in dataframe
I have a dataframe which has two date columns - Start Date and End Date,
Let's suppose start date is 4/10/2019 and end date is 4/10/2020 (MM/DD/YYYY) now I want to split these dates in a list which ...
0
votes
2answers
26 views
Pandas - Create a Integer Primary Key based in string columns - python [closed]
I need to create a primary key based in string columns in my dataframe
...
0
votes
1answer
13 views
How can I replace NaN values with other values in the CustomerID column if they have a same InvoiceNo?
InvoiceNo
StockCode
Quantity
InvoiceDate
UnitPrice
CustomerID
Country
573415
23434
20
2011-10-30 16:17:00
0.79
13607.0
UK
:--------
:---------
:---------
:-------------------
:---------
:----------
:...
0
votes
1answer
20 views
Why does storing the output of .count() result in all 1's or 0's? [closed]
I have a dataframe which I run loc on to find all values within the loc parameters. Whatever the loc returns gets stored in a variable. I try to use this variable to store it in another dataframe ...
1
vote
1answer
31 views
Replace values in dataframe with another dataframes values based on condition [closed]
I have 2 dataframes with the same columns names and rows number but the elements' values are different. I need to change the value of each element in the first dataframe to 1 if its value in the the ...
0
votes
1answer
15 views
Applying a robust scaler to column 0 to 6 of my data frame?
I have a dataframe X and would like to scale columns 0 to 6 using robust scaler, how do i achieve this please? the following code only returns the scaled section of the data frame without the ...
0
votes
1answer
21 views
Why is np.where not returning '1'? Only returns '0'
This code should return a new column called orc_4 with the value of 1 if the value of the row in ...
0
votes
1answer
25 views
Access keys of pandas dataframe when using groupby
I have the following database:
And I would like to know how many times a combination of BirthDate and Zipcode is repeated throughout the data table:
Now, my question is: How can I access the keys of ...
0
votes
1answer
50 views
recognizing the correct word & “Set type is unordered”-error in python-pandas
My Data Set (CSV):
...
1
vote
2answers
27 views
Does binning a time series with pd.qcut (using quantiles) create data leakage?
Let's say I want to predict whether a company will default on it's debt at some point in time (so binary classification) and one of the time series variables I'm using is the "revenue" of ...
0
votes
2answers
35 views
Finding the duplicate values between all columns and sort in new column with Pandas?
I have this DataFrame:
...
0
votes
1answer
35 views
Extract data from json format and paste to column using python
In my column with json data, I have this list I want to extract to column:
...
0
votes
1answer
17 views
copy data between two dataframes in python
I have two different dataframes, both containing a common element (identifier). I want to copy data from df1 for each identifier in df2 (each identifier can repeat maximum 5 times in df1)
df1 ---data ...
1
vote
0answers
11 views
KNN with high-variance data [closed]
KNN doesn't work well with high-variance data, so how should I fit my data?
Here is an example of what the data looks like:
0
votes
1answer
25 views
whats wrong with the graph [closed]
I wanted to plot a bar graph which shows covid cases reported in different regions in different regions of USA.
So here is my code:
...
2
votes
1answer
28 views
How to fill df with data from another df - Python [closed]
I am trying to fill df with values form a different one.
The first df is:
The second df is:
I want to add the column to first df having values from the second one to receive output:
number
trans
...
1
vote
0answers
22 views
How do I convert strings in a dataframe column to int or float [closed]
Encoding the column seperately works ,but when I try it on the dataset directly it throws an error.
The error is something like this.
...
4
votes
1answer
93 views
Why removing rows with NA values from the majority class improves model performance
I have an imbalanced dataset like so:
df['y'].value_counts(normalize=True) * 100
...
2
votes
2answers
39 views
What to replace outliers with? (supermarket transaction data)
I have a transaction dataset from a supermarket. Let's say the average spend is $50.
I want to get each customer's average spend and rank them based on where they fall based on this $50 average spend.
...
0
votes
0answers
24 views
How to assign date into pandas Dataframe?
Im trying to code correlation between exctracted stock prices and data frame with "count" and "date" rows. My dataframe (cvd) looks like this:
``
...
0
votes
0answers
56 views
ValueError: array length 13996092 does not match index length 214200 for Kaggle submission
My model is based on the model linked here on Kaggle.
Specifically, the following lines of code to generate an uploadable table to make my submission:
...
1
vote
1answer
27 views
How to bin a distribution data reported with different frequencies ( salaries ), showing mixed linearity and non-linearity?
I am researching on pay-scales, and wish to receive advise to treat data of salaries.
Objective
My interest is to approximate the salary corresponding to different hierarchical levels in an ...
0
votes
1answer
32 views
How do I get Multiple CSV files (csv file names will be column names) from a folder to a pandas dataframe?
I think the title is enough for what is my problem here. I have 100 tickers in a folder. I got all csv files to one list. but I wanna see the tickers name which these are csv file names.
what should ...
0
votes
0answers
23 views
ValueError: Could not guess the value column! Please hand it to the function as an argument
I'm using the tsfresh library to extract features on my data but I keep getting an error:
...
0
votes
0answers
15 views
How to use pivot_table on a dataframe? How to handle exceptions.UFuncTypeError?
I tried to use the pivot_table function on a dataframe. This was working at the beginning now I get all the time errors, and don't really get why.
...
0
votes
1answer
18 views
Is it ok to fill a pandas dataframe with NaN values?
Is it correct to fill a pandas dataframe with NaN values? In specific: if I have a dataframe with a user name and his age is it ok to fill the age column with int and NaN values.
...
0
votes
2answers
73 views
Error when trying .transform for OrdinalEncoder from Scikit Learn
I'm having a lot of issues using scikit learn recently and was hoping someone could help me with my problem. I can use other methods to ordinal encode but i want to figure this one out.
...
0
votes
2answers
34 views
How to apply Normalisation using the MinMaxScaler() to all Columns, but Exclude the Categorical?
Below, I have the following datatset:
sample_df.head(2)
...
0
votes
0answers
12 views
Forecasting Weekly Average Usage
In python/pandas what would you suggest as a couple of good basic methodologies to use to forecast weekly average usage of user activity on a given platform?
1
vote
1answer
33 views
Replace Missing Values with Most Frequent number under Condition
I'm trying to replace missing values of column "Age" but under condition of other columns on this data Titanic - Machine Learning from Disaster
...
0
votes
0answers
10 views
Rating Sports Teams: How to perform least squares minimization with a constraint?
I am trying to rate NFL teams by minimizing the sum of squared errors subject to a constraint. My data looks like:
...
1
vote
1answer
357 views
AttributeError: 'numpy.ndarray' object has no attribute 'nan_to_num'
I'm trying to run a Random Forest model from sklearn but I keep getting an error: ValueError: Input contains NaN, infinity or a value too large for dtype('float32')....
0
votes
0answers
13 views
How to find anomalies in (almost) constant stream of data?
I have a process that (simply put), starts every 5 minutes, collects data, and put that data into the database.
More detailed explanation would be that process starts, collects data (which takes some ...
