Questions tagged [pandas]
pandas is a python library for Panel Data manipulation and analysis, e.g. multidimensional time series and cross-sectional data sets commonly found in statistics, experimental science results, econometrics, or finance.
1,132
questions
0
votes
1answer
8 views
Call function from one class in the same class
I have 2 classes model and impute. I am defining a function mode_impute inside ...
0
votes
1answer
12 views
Data captured outside Pandas DataFrame
I'm new to Python for starters, when I got this DataFrame from Yahoo Finance, the Date shows up outside the DataFrame. This is leading me to get KeyError: 'Date' when I try to write this line of code
...
0
votes
1answer
18 views
Splitting string column into several columns
In Python, I want to split a string column into multiple columns. The problem is, the strings are not identical or the same length.
Example of data:
...
1
vote
1answer
27 views
Drop or impute the missing values?
I am working with a dataset having 45k rows and I was a bit confused on whether or not to drop the missing values OR impute the missing values.
Column wise missing value distribution :
As per this ...
0
votes
0answers
12 views
Keyword Imputation Using Machine Learning and NLP
I am fairly new to machine learning and data imputation so forgive me if I am using wrong words or not feasible ideas.
I have a table as follows, with text in the Headline column and a list of ...
0
votes
2answers
14 views
How do I remove float object in dataframe? Ex: "winter 2021" to "winter"
I have the premiered column where I want to remove the year and only keep the season. Example: "winter 2021" to become "winter" .
1
vote
1answer
18 views
Defining a function in Pandas [closed]
Hello Data Scientists,
Disclaimer: I am a Beginner learning Python and at the same time just trying out a lot of things.
I am doing some data extraction on a file with with weekly deaths in ...
0
votes
0answers
9 views
Making predictions based on the user's perspective
I have a dataframe containing Id's of tests and different features/characteristics of the tests (length, number of contents etc). The tests are either easy (=1) or hard (=0). Using these features, I ...
0
votes
0answers
18 views
Scaling columns pandas DataFrame
For a dataset having different numeric columns, they usually have different range and distributions. As an example, I have used the Iris dataset. The distributions of it's 4 columns are shown:
My ...
1
vote
1answer
28 views
How to find the distribution of a single variable based on population distribution
I am trying to find the age distribution of the subset of married people of a population. However, I realise that just visualising the married count by age (Figure 1) is highly dependent on the ...
0
votes
2answers
21 views
A strange decimal number on top of my plot of a pandas series
I am using the EU stock market dataset, and applied a box-cox transformation to each time series using stats.boxcox from the scipy module.
The resulting data frame ...
0
votes
1answer
28 views
How to convert duration column from 1 hr. 17 min to 77 min in pandas? [closed]
I am trying to convert all the hr., min, and sec into just mins. For example, 1 hr. 17 min to 77 min, and 34 sec to 0.56 min, not 0.34 min.
So I have used this code:
...
0
votes
1answer
14 views
Business days in Pandas: EU, USA, or another?
To create a pandas date range I usually do the following:
pd.date_range("1991-01-01","1998-01-01",freq="B")
where pd is the pandas ...
0
votes
1answer
29 views
Select Random Value from Pandas list column for each row ensuring that value don't get picked again [closed]
I have a pandas DataFrame below
...
1
vote
1answer
36 views
How to find interval for a series of values? [closed]
I have an example dataset ;
50
60
90
120
140
160
**200**
So I want to compute the interval for this series last element. (200)
I want to say that 200 is ok or not ...
0
votes
1answer
24 views
How can I Improve performance on datatframe calculations [closed]
I have a data frame which start with a few columns, some of them are:
Date
Amount
Text
Category
From this I create a few extra columns, at the moment everything works fine, but it is a bit slow ...
0
votes
1answer
37 views
Pandas series: how to compute daily cumulative sum of a column [closed]
Consider a pandas dataframe where each index is represented as a datetime object in the form like 2009-01-30 23:01:45 .
In order to calculate the total value of a ...
-1
votes
0answers
33 views
how to fill nan values based on conditions in pandas?
I am working on the Titanic dataset and there are null values in the Age column. I would like to fill them in based on there Pclass.
For example, if class is 1, I ...
2
votes
1answer
45 views
Converting YYYYQQ to YYYY-MM-DD [closed]
I have been trying to convert stuff like 2002Q3 to 2002-09-01, but if I use the following, it will convert 2002Q3 to 2002-07-01:
...
1
vote
2answers
20 views
how to convert pandas plot to OO Matplotlib bar chart
I am trying to convert pandas dataframe bar plot to matplotlib OO concept.
...
0
votes
1answer
32 views
Encoding with OrdinalEncoder: TypeError: unhashable type: 'numpy.ndarray'
I'm trying to do a Random Forest in a data-set with numerical and categorical variables in order to obtain a categorical result (two possible classes, column name "predicción"). I'm using ...
1
vote
2answers
36 views
Multiple Merges make the data frame in pandas to explode and causing Memory Issue in jupyter notebook
I have made a multiple merges using pandas data frame (refer the example script below).
It made the data frame to explode and consume more memory as it records reach to 18 Billion in ...
1
vote
1answer
33 views
How does pandas qcut decide the bin edges
I have pandas dataframe, and I want to bin the continuous values.
...
0
votes
1answer
28 views
Python Pandas - Compare Columns In Separate Dataframes, Then Delete Non-Matching Rows In 1 DataFrame
I would like to compare 2 columns in different dataframes, then delete entire non-matching rows.
The 2 columns are not of the same length.
Here is a mock-up of DataFrame #1:
Here is a mock-up of ...
0
votes
1answer
49 views
Different DataFrames need to compare and add new value
I've two Dataframes
DF1 = Contains (~30 Columns and 15000 entries) relevant are "TYP" (string) and "VERSION" (string)
DF2 = Contains "SOFTWARE_NAME" (string), "...
0
votes
1answer
24 views
Sampling a data based on average and variance of another data
I have a set of textual datasets that have the following average and variance tokens lengths:
...
1
vote
0answers
27 views
Recursively splitting on a dataset [closed]
I have a method for splitting a dataset via entropy discretization. My goal is to continue splitting the data sets such that the first set breaks down into two sets:
s -> s1, s2
This reaches a ...
0
votes
1answer
40 views
How to multiply a pandas dataframe with a numpy array
I have a simple program for performing PCA analysis on a dataset. The goal is to multiply the dataset by the feature vector at the end of the program.
...
0
votes
0answers
30 views
Data science/Dataframe question concerning an approach to a problem
I want to answer which kind of school type (mixed gender, only for boys or only for girls) is more efficient in training its students based on the result of a normalized exam score. And my model ...
0
votes
1answer
76 views
How to convert a nested dictionary's keys in a pandas data frame
I am new to Python and want to store a nested dictionary keys as rows in a pandas dataframe. Specifically, here is how the dictionary looks like:
...
0
votes
1answer
15 views
How to apply entropy discretization to a dataset
I have a simple dataset that I'd like to apply entropy discretization to. The program needs to discretize an attribute based on the following criteria
When either the condition “a” or condition “b” is ...
1
vote
1answer
44 views
Combine 2 dataframes of unequal length based on some condition [closed]
I have 2 dataframes e.g.:
df1 = [id, name, salary, address, address_if_present_in_US]
df2 = [id, name, salary,address]
I would like to add a new field to ...
2
votes
1answer
25 views
Change the ids of two linked pandas dataframes [closed]
Is it possible to change IDs columns into two linked pandas dataframes?
Example:
I have these two dataframes: Users and Ratings
I would like the Users IDs to become as follows:
This is my original ...
0
votes
1answer
30 views
Deleting a Column in a csv or Excel file using Pandas
I am trying to delete a column from my csv file (column 'A' called "Film Number") but have tried numerous variations of code and while it deletes the column in the dataFrame it doesn't do so ...
1
vote
1answer
24 views
How to find difference between two dates in different dataframes
I want to create a difference between two date columns from different dataframes based on customer ID from the first dataframe. For example: I find max date from the first dataframe for all customers:
...
0
votes
1answer
25 views
Test data missing values
I've been given a test and train data set, and I have null values in my test data set. What do I do with it?
1
vote
2answers
77 views
Do we need to pre-process both the test and train data set?
I've been given 2 datasets , and there are missing values in both the test and training data set. Do I need to pre-process test.csv also or is it only for train.csv?
-1
votes
2answers
21 views
0
votes
0answers
35 views
How to rename different level entries of deeply nested JSON data into a single name so the values populate the same renamed column
I am trying to flatten a deeply nested JSON data and converting it into a csv using Pandas dataframe. I am able to do this but I am getting column names as entry_0_resource_code_coding_0_code, ...
2
votes
2answers
62 views
Create a new column in a dataframe with pandas in python such that the new column should be True/False format based on existed column
I was trying to create a new column to a dataframe such that the new column should have the format as True/False based on some pattern.
My dataframe:
...
1
vote
0answers
27 views
What kind of data structure should I use for this data?
I've been using Pandas with the exhaust data from my job for about 2 years now, but I think I've somewhat outgrown strictly tabular data structures.
My problem: I work for a large logistics company (...
0
votes
0answers
11 views
Increase dimension of dataset for modelisation
I have a dataset like that :
| A header | Another header |
| -------- | -------------- |
| a | {player_1: stat_1,stat_2,stat_3} |
| b | {player_2: stat_1,stat_2,stat_3} |
| ...
0
votes
0answers
30 views
Pandas: Why is an inner join necessary prior to semi-join of Dataframes?
As described here and in other sources, the common approach to semi-join (filtering Dataframe1 with Dataframe 2 so that only the lines of Dataframe1 that appear at Dataframe2 are shown) two tables ...
1
vote
0answers
15 views
How to login a web site for take the pages tables with pandas? [closed]
I created a python code, using the pandas library for to take tables from web site, however appear to me a "forbidden error", I think that is why I need to login before to the take the ...
1
vote
1answer
21 views
When should we use jaccard score?
I am a newbie in Machine Learning, I trained a binary classifier for bank loan prediction through Logistic Regression.
I measured the accuracy of it with two methods: accuracy score and jaccard index.
...
1
vote
1answer
32 views
Replace nan with sequence of values
I have the dataset for covid tests in Ireland. In May the data just stops.
To continue the trend, I took the average mon-sun during the last stable period. So I have 7 different mean values.
Now I ...
1
vote
1answer
56 views
Numpy array vs Pandas DataFrame when training [closed]
https://towardsdatascience.com/speed-testing-pandas-vs-numpy-ffbf80070ee7
(You can open the link in incognito if its locked).
Numpy arrays are faster than DataFrame on normal mathematical operations.
...
0
votes
2answers
66 views
I am attempting to implement k-folds cross validation in python3. What is the best way to implement this? Is it preferable to use Pandas or Numpy? [closed]
I am attempting to create a script to implement cross validation in data. However, the splits cannot randomly take any records, so the training and testing can be done on equal data splits for each ...
0
votes
2answers
27 views
How to group by IDs and count the number of groups with occurrence of a variable after first point?
Language: Python 3.8
I have a dataframe that consists of a series of people (each appearing multiple times in the dataframe), dates, and binary variables. I am trying to figure out how many people ...
2
votes
1answer
18 views
How to calculate the difference based on matching criteria
Hello, I am trying to transfer over from excel to pandas.
I want to add new column called 'daily_volume' where if the 'project_name' is equal to the above row project_name then calculate the ...
