Questions tagged [pandas]
pandas is a python library for Panel Data manipulation and analysis, e.g. multidimensional time series and cross-sectional data sets commonly found in statistics, experimental science results, econometrics, or finance.
1,338
questions
0
votes
0
answers
5
views
I want to parse the data from the employee.json file
https://drive.google.com/file/d/1tqlKIm5leTS-ALj8w8fYtCrOqs62bNXL/view?usp=drive_link
This is the link to the json file i want to parse the data(resolve (a sentence) into its component parts and ...
0
votes
0
answers
16
views
Service Desk calls reports - Report Implementation assistance using Pandas
From the dataframe below I need to get the following information per USER using Pandas:
Total Number of calls per day (the csv report contains a month of calls per user)
Total number of RONA calls ...
- 1
0
votes
1
answer
17
views
Trying to write a code that detects a maximum value before a drop off in slope
I'm trying to write code for a force test that will output the maximum force before structural failure occurs. I'm a bit of a novice to python, so the issue here might be something simple that I'm ...
1
vote
2
answers
31
views
Pandas dataframe changing 2 columns values based on another columns value
I have a DF w' the following values:
id Count Exposure pct_affected
104 4 150,000 1.2
104 0 150,000 1.2
WHen I merged my 2 dataframes, the values from one carried onto the other. ...
- 11
0
votes
1
answer
16
views
calculate the predicted value based on coefficient and constant in python
i have the coefficients and the constant (alpha). i want to multiply and add the values together like this example. (it has to be done for 300000 rows)
Prediction = constant + (valOfRow1 * col1) + (-...
- 101
0
votes
1
answer
44
views
What is the use of using Vector databases over Pandas Dataframes?
My question is why can't we use a pandas dataframe to do all the data retrieval? Can someone give reasons apart from the fact that databases has the persistence of Data?
0
votes
1
answer
16
views
How to parse month string as Pandas Period
I have imported data as a Pandas DataFrame which has string representing months of various years such as: 'Nov-23', 'Mar-24' etc.
I would like to parse these strings into proper Pandas Periods. If I ...
- 131
0
votes
0
answers
15
views
pandas problem in calling rolling window and lambda function
I have a pandas dataframe with the following set of columns (from the df.columns function):
Index(['open', 'high', 'low', 'close'], dtype='object')
I am trying to ...
0
votes
0
answers
10
views
clustering droplet data along trajectories
I have regular t,x,y,z-datapoints of droplets, that move continuously from top to bottom on different but similar paths. The droplets also have a volume that is approximately measured, they can appear ...
0
votes
1
answer
35
views
Creating well-separated bins
I want to distribute my hospital data into 4 tiers (bins) based on the score. Tier 1 (good hospitals) has the highest score and Tier 4 (not so good hospitals) has the lowest score. The tiers can be of ...
- 101
0
votes
0
answers
10
views
Need help in Webscraping T20 World Cup 2022 data
I have a list of links of match scorecards and i have stored the respective html content of each link in a list. I need to get the csv files I am attaching as the output. Any help would be ...
0
votes
0
answers
9
views
Encode multiple label categorical variables with consideration of the frequency and standardization
I'm currently working on a dataset containing, among other features, multi-labeled categorical data per person for the last 3 years. I'm not sure how to handle this kind of categorical data where ...
- 1
0
votes
0
answers
28
views
Perform k-means clustering over multiple columns using python but without library sklearn.kmeans
I'm on a project with the topic "clustering" using the KMeans method.
so I have a data set with 10 columns and 1000 rows divided into 5 clusters (in this case there are 5 robot movements). I'...
0
votes
0
answers
28
views
Imputation of multi-peaked dataset
I am a beginner to model fitting, and I have been working on generating a model for a CO2 emissions data set. The distribution of the data points in a number of these columns are very markedly dual-...
0
votes
0
answers
24
views
time series analysis: lag features
I plan to include lag features in my multivariate time series data.
I have 17 input feature and one output feature. both input and output features are time series.
I will insert one more input feature ...
- 1
1
vote
2
answers
138
views
Why am I getting this TypeError: quantile() got an unexpected keyword argument 'numeric_only' for pandas quantile()?
I ran into a weird error with pandas quantile() (pandas.DataFrame.quantile()) method. I have a dataframe with both numeric and non-numeric columns. I wanted to ...
- 1,091
0
votes
1
answer
42
views
What data am I supposed to store into the "input_data" variable?
I'm supposed to pass preprocessed input to a TFLite model to make inferences. I have an excel file containing data and a tensorflowlite model file. I received instructions on how to do this; here is ...
- 1
0
votes
2
answers
47
views
Genuinely stumped. I can't divide using a dataframe
I loaded some state by state covid data from: https://www.cdc.gov/nchs/pressroom/sosmap/covid19_mortality_final/COVID19.htm
Then I loaded it into a data frame and try to do some basic analysis. But I ...
- 158
0
votes
0
answers
17
views
Formal terminology: metafeatures then groupby (GROUP BY) operation
In ML training and other analytics I often combine features to produce a 'metafeature' and then perform a 'groupby' (pandas) or 'GROUP BY' (SQL) query. What is the technical term for this operation?
...
- 208
0
votes
0
answers
21
views
How to check for correlations using ANOVA between the selected feature columns and the target feature column
Suppose I have a set of feature columns that I select to check for correlations with the target feature 'is_Star'; which is of binary values and the former consists of numeric values
Why do i end up ...
- 1
0
votes
1
answer
17
views
Best measure to inform how predicted value can differ from real one
I have trained a regression model and obtained a pandas series of the predicted values.
I am working on a "calculator" that will be able to return a predicted value after entering an input ...
- 5
0
votes
0
answers
27
views
Panda's SAS PROC transpose equivalent in case of only ID, multiple VARs
I would like to transform the following:
week_no
count
mean
std
min
max
24
762.0
774.002734
0.995213
771.0022
776.4584
25
2971.0
759.702417
4.241902
750.9586
767.6225
...
...
...
...
...
...
To:
...
- 1
0
votes
1
answer
22
views
How can I find what kind of args and kargs a pandas function takes?
I was wondering how do I find what are valid kargs and args for a given function? For example in pandas there is this function cumin DataFrame.cummin(axis=None, skipna=True, *args, **kwargs); how do I ...
- 111
0
votes
2
answers
66
views
Filter rows with multiple conditions using two dataframes
I have two dataframes that share an ID number column. I'd like to filter df1's rows based on two conditions: 1) it shares the ID with df2 and 2) it meets a condition in a column in df2.
I have this ...
- 103
0
votes
1
answer
116
views
Parsing Python List values from column in pandas data frame??? Data cleaning help me please
I am new to the community and data science. I have found this hub to be of extreme value and hope to be an additive contributor one day.
VS Code: Python: Pandas Question
I am working with a data set ...
1
vote
1
answer
29
views
How to proceed when correlation plot doesn't show as much multicorrelation as is seen by statsmodels variance_inflation_factor in a regression task?
I am working with the kaggle Blueberry Yield prediction dataset. There are 17 columns including the target variable. Below is the correlation heat map:
It can be seen that multiple features are ...
- 111
1
vote
1
answer
31
views
How to prepare data if each item has multiple categories (like tags)
I'm working on a recommender system that will recommend movies to users.
Movie ratings
Movie
User
Rating
100
201
5
105
256
8
...
...
...
Movie tags
Movie
Tag
100
1
100
2
100
8
105
2
105
5
....
- 131
0
votes
2
answers
21
views
How to filter out redundant id rows with deviant values
First post, so thank you in advance for any pointers! I have a dataframe with an issue similar to the following:
id value
10 25.03
11 26.00
11 0.95
12 32.95
13 34.43
13 1.02
One can see ...
0
votes
0
answers
24
views
what is the best practice to handle carriage returns when exporting data from pandas dataframe to CSV?
While trying to export data from a pandas dataframe to a CSV, we noticed that carriage returns (inserted at the middle of a line or at the end of a line) produce outputs that are different from ...
- 101
0
votes
2
answers
72
views
How to create a plot of specific row with every column using python and which package to choose matplotlib or seaborn?
I'm trying to create a plot for the ranking of each country from 2002 to 2023. I created this dataset by loading each csv file from the respective years, which contains the Countries and Ranking ...
2
votes
2
answers
69
views
Why does pandas.dataframe.info() not update when I delete the first row of a dataset?
I am working with a dataset that comes in with nonsense field names in the first row, with the actual field names in the second row.
Currently I'm using this script:
...
- 165
0
votes
1
answer
33
views
What is the minimum ratio to consider the data set as balanced for the classification algorithm?
Can I consider 20% / 80% as a balanced dataset?
My target variable ratio is 62% and 37%. I hope It is a balanced dataset. Please let me know if I am wrong.
However, I would like to know, what is the ...
0
votes
1
answer
455
views
Pandas: To find duplicate columns
I have a dataframe with different dtypes like int, float, object, datatime etc. I am performing data cleaning, to list or find ...
- 115
0
votes
2
answers
469
views
corr() is giving an error. please help out of this problem and tell me what is this error about
when I am trying to run sns.heatmap(df.corr(),annot=True) this code in my jupyter notebook. this error is occuring. I cannot understand this problem. please help me.
4
votes
1
answer
75
views
Working with time series data with several times stamps on a dates, and implementing machine learning
I'm trying to implement predictive analytics on a production data. my goal is to predict next downtime, it's reason and issues.
My data looks like below;
...
- 151
0
votes
1
answer
25
views
'Merge' two identically sized pandas dataframes to one equal sized dataframe [closed]
If I have a dataframe e.g.
| A | B
---------
| 1 | 21
---------
| 3 | -1
and
| C | D
---------
| 15 | 2
---------
| 7 | 9
...
- 91
0
votes
1
answer
26
views
how to prepare pandas dataframe from Solar Flare Data Set?
I am looking to download and use Solar Flare Data Set (https://archive.ics.uci.edu/ml/datasets/Solar+Flare) but unable to figure out how to get the pandas dataframe from the file that is being ...
- 101
0
votes
0
answers
5
views
How to identify best fit sub groups in time series data
I am working with a set of time series user activity data where i'm trying to figure out if the user performed an action daily for at least X days in Y. My use case is 15 days in 30, doesn't have to ...
0
votes
0
answers
13
views
I am creating a new column, income to debt ratio, but output is coming in integer, instead of float
I need the income to debt ratio in float, when i tried manually it is coming infloat, but when i do in pandas, it is coming as integer. how can i take output in float numbers.
0
votes
3
answers
55
views
1
vote
1
answer
152
views
Filling na values with condition from other column [closed]
I am working on famous titanic dataset and I want to replace na values in Age column but on such a condition that these people whose Pclass=3 receive 25, Pclass=2 29 and Pclass=1 38.
I was trying to ...
- 11
0
votes
1
answer
51
views
Understanding correlation - Machine Learning
I am experimenting a project on identifying cancer or not - Binary classification
The dataset has many columns. Here, I added correlation values between few input columns and the target column[cancer/...
- 21
0
votes
0
answers
176
views
How to filter rows using custom function in pyarrow
I've a parquet dataset that contains latitude and longitude values as separate columns. And I want to filter those rows that are inside a polygon, I'm able to do this in pandas dataframe but unable to ...
- 101
0
votes
0
answers
19
views
Running all cells again adding new rows instead of return new dataframe
i am creating a class just like lazypredict library but with k fold support and return a data frame withh the mean score of every model every thing works and i can ...
- 101
0
votes
1
answer
20
views
Force network to weigh specific variables during learning
I have a pandas data frame containing around 100000 observations of plant species and their age with additional numerical predictors (climate). I used tensorflow ...
- 53
0
votes
0
answers
15
views
What is the best way to whittle down rendundant categorical data?
I'm trying to build a linear regression model in Tensorflow (and preprocessing with pandas) that will help me categorize bank transactions. I'm trying to whittle down the vendor parameter, because the ...
0
votes
0
answers
16
views
Getting probability to finish in the last 3 ones after each game week
I'am working on a dataframe with five differents features :
team
game_week
season
cum_points : pre game cumulative number of points
final_position
Those datas are covering 10 seasons of Premier ...
0
votes
0
answers
23
views
passing pandas dataframe to function, but it appears as numpy array
In the function I call sklearn train_test_split
In Pycharm debugger everything looks ok until it gets to that call.
Then it backtracks and tells me
...
- 574
0
votes
1
answer
27
views
Counting values from diffrent columns
I'm working with a pandas dataframe that has some columns like these:
...
- 1
0
votes
0
answers
22
views
Which statistical test is suitable for analyze the correlation between non-independent categorical variables?
Im working on an EDA on chocolate bar ratings, trying to indentify relationships between variables.
In this case, im trying to test if the ingredients of a chocolate bar area associated with its ...
