Questions tagged [pandas]
pandas is a python library for Panel Data manipulation and analysis, e.g. multidimensional time series and cross-sectional data sets commonly found in statistics, experimental science results, econometrics, or finance.
1,154
questions
0
votes
0answers
4 views
measuring variance of differences between two sets of corresponding means with significance
I have a dataframe id, w, x, y1, y2 (two categorical variables, two dependent variables)
id is the index which is not particularly informative
w, x are categorical variables - w in {0,1}, x in {0,1,2,...
0
votes
0answers
5 views
Clustering with multiindex DataFrame
I have a huge amount of multiindexed data that, very simplified, looks like this:
Code:
...
0
votes
0answers
17 views
Pyspark vs Pandas: When should we consider to change from Pandas to Pyspark
We trained a model in a single server using pandas, dataframe = 2.000.000 rows(I run it later in my own laptop), now we are migrating the code to the cloud and in that case to Databricks.
In ...
1
vote
1answer
22 views
Best way to optimize dataframe row by row sum of squared errors calculation?
I am seeing a significant slowdown with the following small snippet of code which computes the sum of squared errors between two dataframes - e.g. it takes approximately 2.5 seconds to run when ...
0
votes
0answers
23 views
Modeling a probability using scikit
I'm trying to create a model to predict the probability for an object to break in a specific year. I have information for different properties of different of these objects, such the year when they ...
3
votes
1answer
37 views
How to merge with smallest Euclidean distance?
Given a main left dataset, how can I merge with right dataset with smallest Euclidean distance (...
0
votes
2answers
36 views
Merge two dataframes on multiple columns, only if not NaN
Given two Pandas dataframes, how can I use the second dataframe to fill in missing values, given multiple key columns?
...
1
vote
1answer
18 views
I have two dataframes, df and df1, I want to check if df1 columns values is present in df columns and print the entire row of the df if present
first df:
df = pd.DataFrame({'col1':[1,2,3,4],
'col2':[5,6,7,8],
'col3':[9,10,11,12],
'col4':[13,14,15,16]})
second df:
<...
0
votes
0answers
11 views
Anomaly Detection with totally Categorical features
I am working on an anomaly detection project that aims to discover which merchant is the point of compromise. The data contains no numerical value and it looks like this;
date
account
merchant
fraud
...
0
votes
1answer
20 views
i need help in this python code please see below attached image for details, I have attached expected out put and what I done till now
the first picture is what we have to do
0
votes
0answers
11 views
How to find duplicate rows in a column then find out if two cells in another column sum up to a third cell in an Excel tab in Python?
I need to find all duplicate rows (string values) in "Name" column and then find out if two numerical values in "Amount" column sum up to a third value also in the "Amount&...
1
vote
0answers
26 views
What should be the target variable shape? should it be (n,1) or (n,)?
I tried logistic regression on the titanic dataset but used 2 different code:
1st code:
...
0
votes
1answer
10 views
0
votes
0answers
16 views
How to use YAML files as input for creating custom labels in Pandas?
I have a data frame that has a column called Price. This column contains continuous numeric values. The custom labels are created using the following code:
...
0
votes
2answers
19 views
How to convert String data to something more meaningful for regression model training
I have a bunch of data of employees and their salaries.
I would like to build a regression model that predicts
The columns in question are countries, employment_status, job_title, education
All of ...
0
votes
3answers
46 views
How to compare two columns of two dataframes and indicate the changes?
I have two dataframes, one is current week's information, one is of last week.
I want to create a new dataset that lists all the changes during the week. Please see the following example:
if there is ...
0
votes
0answers
18 views
Pandas On AWS Sagemaker
Hello Guys i have a question
We want to start working with AWS Sagemaker.
I understand that i can open Jupiter notebook and work
like it was in my computer.
but i know pandas working on single ...
1
vote
0answers
14 views
Handling imbalanced Feature (X) not lavbel (Y) in machine learning
I am very new to this field and have done a decent amount of research on this, but every time, I stumble upon handling the imbalanced label by using f1 score, recall, precision as metrics, and using ...
0
votes
1answer
12 views
Could I use some elements of my target variable to predict it?
I'm trying to predict if a company will bankrupt, I use a dataset of 2020 and I manually created my target variable with the status of the company the status date, status reason to create my target ...
3
votes
1answer
39 views
How do I normalize json data into pandas (Covid-19 data) [closed]
I am trying to import all up-to-date datasets in JSON format on the covid-19 pandemic into a pandas dataframe. I believe it should be possible by using ...
0
votes
1answer
20 views
Applied and view jobs ratio
I have the following data set where the column "kind" can be V(view) or A(apply), how can I do the following things given a particular job id how many applicant Apply (A) to that particular ...
0
votes
1answer
57 views
Error: `raise ValueError( ValueError: Missing column provided to 'parse_dates': 'Date' [closed]
I am using a .csv with two columns. The first has dates and the second has temperatures. I would like to plot it with dates on the x-axis and temperatures on the y-...
0
votes
1answer
22 views
How to strategize model training with new data coming in every day?
I have a mysql database in which new records are added every day to raw data. This raw data is cleaned and a ML model is trained with it once a week. What should be the best strategy to capture new ...
0
votes
1answer
31 views
Data Preparation for dates
I have two columns, a birth year and the first year of being a customer.
There is a large number of samples that have a birth year less than the first year of being a customer which does not make ...
0
votes
1answer
16 views
How Can I Process SageMaker Ground Truth NER JSON Output into DataFrame?
So, I've recently created a job using AWS SageMaker Ground Truth for NER purposes, and have received an output in the form a manifest file. I'm now trying to process the manifest file into a dataframe,...
0
votes
2answers
52 views
How to remove rows from a data frame that have special character (any character except alphabet and numbers)
How to remove rows from a data frame that have special character (any character except alphabet and numbers)
I have some unwanted labels which seems to be useless but I want to remove them, this are ...
2
votes
1answer
37 views
Correlating & Combining categorical and numerical features for classification problem
I just started with my first Machine Learning project (Jupyter Notebook, Python, Scikit-learn, pandas) and I am working on a Palmer Penguin Dataset. I have been doing some data analysis and I got ...
0
votes
1answer
26 views
How to find median/average values between data frames with slightly different columns?
I am trying to combat run-to-run variance of the data I collect by combining the data from different runs and finding the mean/average. The problem is that in each run there is a chance that some of ...
0
votes
1answer
20 views
panda grouping dates by variable with transposed value variables
I had read this post
panda grouping by month with transpose
and it gave me the nearest answer to my question but not the completely solution.
How would I get somewhat like the reverse output?
My ...
0
votes
0answers
14 views
Which plot should I use for plotting discrete variable with binary variable?
I am two columns Age(discrete), and Purchase(binary). I want to visualize it to understand which age group is most interested in purchasing things. I was thinking of plotting something like a line ...
1
vote
0answers
28 views
Understanding plotted ACF on stationary time-series data
I made time-series data stationary and plotter ACF & PCF, while PCF looks fine, I do not know how to interpretate ACF, as it looks like this - I could not say it geometrical. For PACF - it is ...
1
vote
1answer
31 views
Python - Remove Duplicates but only when other conditions are met
I'm trying to find a way in Python in which to drop rows where duplicates occur within specific columns, but only to drop those duplicates where they are not attributed to the latest date.
In the ...
1
vote
1answer
31 views
I want to do sub groups in pandas data frame [closed]
I have a dataframe as below I want to do first group by trip_number and then subgroup by mode.
...
0
votes
1answer
14 views
How to drop the previous rows of a database based on a matching value in a column?
So I am currently trying to sort through a data frame containing attribute classes and values of teams. However, my data has multiple rows of different classes and values of the same Team ID/Attribute ...
1
vote
0answers
13 views
Is there a way to create a continuous distribution graph from discrete percentiles?
I have an underlying distribution that I know is 1-tailed (Packet Response time), but the service that runs the benchmark is opaque to me. It only returns a percentile table up to some number of nines ...
0
votes
0answers
24 views
Create rows filling the sequence of numbers in a 'period column' and the 'value column' with 0 in python/pandas
I need to fill a dataframe with rows completing the values that the system doesn't show on the report (e. g. a week that the product had no sales):
The report:
Product
Week
Total_Sales
0
A
1
82
1
A
...
0
votes
1answer
29 views
0
votes
2answers
52 views
Call function from one class in the same class
I have 2 classes model and impute. I am defining a function mode_impute inside ...
0
votes
1answer
13 views
Data captured outside Pandas DataFrame
I'm new to Python for starters, when I got this DataFrame from Yahoo Finance, the Date shows up outside the DataFrame. This is leading me to get KeyError: 'Date' when I try to write this line of code
...
0
votes
1answer
21 views
Splitting string column into several columns
In Python, I want to split a string column into multiple columns. The problem is, the strings are not identical or the same length.
Example of data:
...
1
vote
1answer
37 views
Drop or impute the missing values?
I am working with a dataset having 45k rows and I was a bit confused on whether or not to drop the missing values OR impute the missing values.
Column wise missing value distribution :
As per this ...
0
votes
0answers
18 views
Keyword Imputation Using Machine Learning and NLP
I am fairly new to machine learning and data imputation so forgive me if I am using wrong words or not feasible ideas.
I have a table as follows, with text in the Headline column and a list of ...
0
votes
2answers
23 views
How do I remove float object in dataframe? Ex: "winter 2021" to "winter"
I have the premiered column where I want to remove the year and only keep the season. Example: "winter 2021" to become "winter" .
1
vote
1answer
18 views
Defining a function in Pandas [closed]
Hello Data Scientists,
Disclaimer: I am a Beginner learning Python and at the same time just trying out a lot of things.
I am doing some data extraction on a file with with weekly deaths in ...
0
votes
0answers
11 views
Making predictions based on the user's perspective
I have a dataframe containing Id's of tests and different features/characteristics of the tests (length, number of contents etc). The tests are either easy (=1) or hard (=0). Using these features, I ...
0
votes
0answers
25 views
Scaling columns pandas DataFrame
For a dataset having different numeric columns, they usually have different range and distributions. As an example, I have used the Iris dataset. The distributions of it's 4 columns are shown:
My ...
1
vote
1answer
28 views
How to find the distribution of a single variable based on population distribution
I am trying to find the age distribution of the subset of married people of a population. However, I realise that just visualising the married count by age (Figure 1) is highly dependent on the ...
0
votes
2answers
22 views
A strange decimal number on top of my plot of a pandas series
I am using the EU stock market dataset, and applied a box-cox transformation to each time series using stats.boxcox from the scipy module.
The resulting data frame ...
0
votes
1answer
31 views
How to convert duration column from 1 hr. 17 min to 77 min in pandas? [closed]
I am trying to convert all the hr., min, and sec into just mins. For example, 1 hr. 17 min to 77 min, and 34 sec to 0.56 min, not 0.34 min.
So I have used this code:
...