Questions tagged [pandas]
pandas is a python library for Panel Data manipulation and analysis, e.g. multidimensional time series and cross-sectional data sets commonly found in statistics, experimental science results, econometrics, or finance.
1,105
questions
0
votes
0answers
17 views
Change the ids of two linked pandas dataframes
Is it possible to change IDs columns into two linked pandas dataframes?
Example:
I have these two dataframes: Users and Ratings
I would like the Users IDs to become as follows:
This is my original ...
0
votes
1answer
18 views
Deleting a Column in a csv or Excel file using Pandas
I am trying to delete a column from my csv file (column 'A' called "Film Number") but have tried numerous variations of code and while it deletes the column in the dataFrame it doesn't do so ...
1
vote
1answer
17 views
How to find difference between two dates in different dataframes
I want to create a difference between two date columns from different dataframes based on customer ID from the first dataframe. For example: I find max date from the first dataframe for all customers:
...
0
votes
1answer
18 views
Test data missing values
I've been given a test and train data set, and I have null values in my test data set. What do I do with it?
2
votes
2answers
42 views
Do we need to pre-process both the test and train data set?
I've been given 2 datasets , and there are missing values in both the test and training data set. Do I need to pre-process test.csv also or is it only for train.csv?
0
votes
2answers
18 views
0
votes
0answers
20 views
How to rename different level entries of deeply nested JSON data into a single name so the values populate the same renamed column
I am trying to flatten a deeply nested JSON data and converting it into a csv using Pandas dataframe. I am able to do this but I am getting column names as entry_0_resource_code_coding_0_code, ...
2
votes
1answer
18 views
Create a new column in a dataframe with pandas in python such that the new column should be True/False format based on existed column
Please help me!
I was trying to create a new column to a dataframe such that the new column should has format as True/False based on some pattern.
My dataframe:
df = pd.DataFrame([[1,16],[21,3],[0.9,...
0
votes
1answer
13 views
Loop Through DF and Add New Column with Values Based on Index Range [closed]
I have a dataframe similar to the following
Sales_Region1
Sales_Region2
10
60
30
10
80
20
20
40
40
70
90
80
I want to add a new columns for each region (Sales_Category1 & Sales_Category2 ...
1
vote
0answers
24 views
What kind of data structure should I use for this data?
I've been using Pandas with the exhaust data from my job for about 2 years now, but I think I've somewhat outgrown strictly tabular data structures.
My problem: I work for a large logistics company (...
0
votes
0answers
10 views
Increase dimension of dataset for modelisation
I have a dataset like that :
| A header | Another header |
| -------- | -------------- |
| a | {player_1: stat_1,stat_2,stat_3} |
| b | {player_2: stat_1,stat_2,stat_3} |
| ...
0
votes
0answers
18 views
Pandas: Why is an inner join necessary prior to semi-join of Dataframes?
As described here and in other sources, the common approach to semi-join (filtering Dataframe1 with Dataframe 2 so that only the lines of Dataframe1 that appear at Dataframe2 are shown) two tables ...
1
vote
0answers
12 views
How to login a web site for take the pages tables with pandas? [closed]
I created a python code, using the pandas library for to take tables from web site, however appear to me a "forbidden error", I think that is why I need to login before to the take the ...
1
vote
1answer
17 views
When should we use jaccard score?
I am a newbie in Machine Learning, I trained a binary classifier for bank loan prediction through Logistic Regression.
I measured the accuracy of it with two methods: accuracy score and jaccard index.
...
1
vote
1answer
31 views
Replace nan with sequence of values
I have the dataset for covid tests in Ireland. In May the data just stops.
To continue the trend, I took the average mon-sun during the last stable period. So I have 7 different mean values.
Now I ...
1
vote
1answer
35 views
Numpy array vs Pandas DataFrame when training [closed]
https://towardsdatascience.com/speed-testing-pandas-vs-numpy-ffbf80070ee7
(You can open the link in incognito if its locked).
Numpy arrays are faster than DataFrame on normal mathematical operations.
...
0
votes
2answers
57 views
I am attempting to implement k-folds cross validation in python3. What is the best way to implement this? Is it preferable to use Pandas or Numpy? [closed]
I am attempting to create a script to implement cross validation in data. However, the splits cannot randomly take any records, so the training and testing can be done on equal data splits for each ...
0
votes
2answers
23 views
How to group by IDs and count the number of groups with occurrence of a variable after first point?
Language: Python 3.8
I have a dataframe that consists of a series of people (each appearing multiple times in the dataframe), dates, and binary variables. I am trying to figure out how many people ...
1
vote
0answers
13 views
How to calculate the difference based on matching criteria
Hello, I am trying to transfer over from excel to pandas.
I want to add new column called 'daily_volume' where if the 'project_name' is equal to the above row project_name then calculate the ...
1
vote
3answers
21 views
Work with large number of features for machine learning with pandas and sklearn
I'm relatively new to data science and I'm working with a large dataset. It has lots of rows and around 270 features after removing features with a lot of nan valuew and encoding categorical features. ...
2
votes
1answer
20 views
how is validator created in python and what are most popular libraries / modules to learn first
I have a df which has a serial number generated with each new record. The serial number combines with some other part like state code, year of registration and ...
0
votes
1answer
15 views
Where are the predicted results located in this model?
I am new at this and am pretty sure this is a stupid question, but here it goes:
Where can I see the results of a model's prediction?
I did this course about deep learning, followed the tutorial, ran ...
1
vote
0answers
25 views
How to make the values in different columns in the correct order based on another data frame (mapping) in Python Pandas
I am pretty new to Python and Pandas and I struggle with combining a messy dataframe from excel with a mapping. I have tried to find some solutions on the Internet, however with no success.
My first ...
0
votes
0answers
11 views
Aggregate grouped data conditionally over many columns doing different operations in Python/Pandas [migrated]
I'm not sure this is a general coding question or not but I hope this is the correct forum. Consider the following reduced example data frame df:
...
0
votes
0answers
9 views
Calculating all near-tangential lines for inflection points of a time series
I have been thinking about this problem for a while and I'm curious if anyone knows of a good paper on this, has any ideas for algorithms or improvements to the framework.
The task is to store the ...
1
vote
0answers
46 views
CDF plot overlay histogram in python
I have a dataframe called df['ProgressStep']
I would like to get overlaid CDF plot in histogram. Have tried 2 methods, neither one meet my target perfectly. please help to fine tune the code, either ...
3
votes
1answer
46 views
Using output of ML algorithms as an input to different ML algorithm (Ensemble Learning)
I want to assign weights to multiple models and make an single ensemble model. I want to use my outputs as the input to a new machine learning algorithm and the algorithm will learn the correct ...
1
vote
0answers
28 views
Numpy-like reduction of particular dimensions in Pandas DataFrame?
With a numpy multidimensional arrays, it is easy to reduce particular axes with functions such as np.mean or np.sum. For example,...
0
votes
0answers
13 views
Best way to store large amounts of time series data? Relational database (SQL) or NoSQL route. Also additional python/pandas question inside
so I'm working on a project and I'm sort of stuck as to how to store my data. I have a concept I want to propose but am unsure whether it is possible, if it is not I would appreciate any help ...
0
votes
0answers
8 views
pandas reduce time for na checks (is_valid_na_for_dtype / is_na)
I have a small project that I iterate over a dataframe and do stuff with this data.. I know that if I can I should avoid iterate the rows but I'm afraid that I must so I use itertuples.
Now I want to ...
0
votes
1answer
19 views
How to get column names after One Hot Encoding when using Pipelines?
I am using Pipeline and ColumnTransformer to preprocess the data. Basically I am using them to impute null values, scale the ...
1
vote
0answers
10 views
How to compute similarity matrix for strings efficiently? [duplicate]
Here I'm trying to compute similarity between 1000 cross 10000 strings (using Levenshtein distance), I'm using a dataframe approach where you just need to compare n(n-1)/2 comparisons instead of n*n. ...
0
votes
0answers
13 views
How to merge one table to another by multiple columns?
I have a table that looks like this:
prodh1
prodh2
prodh3
prodh4
prodh5
prodh6
7
77XS
77XSBRC
77XSBRCBRC1
77XSBRCBRC17S0C
77XSBRCBRC17S0C70R
7
77XA
77XA7TR
77XA7TRPRC1
77XA7TRPRC17S0C
...
0
votes
2answers
24 views
Compare 2 Pandas Column, over different rows and Output the resultant in a 3rd column
I have a data frame like this.
EMPLOYEE_ID
NAME
MANAGER_EMPLOYEE_ID
0
42
S
40
1
40
G
NaN
2
T
M
40
3
0c
H
42
I want a separate column that outputs the Name of the Manager of every person.
So for ...
0
votes
2answers
53 views
Fastest way to replace a value in a pandas DataFrame?
I am loading in 1.5m images with 80,000 classes (or I will have to when I eventually train) into a Keras generator and am using a pandas dataframe to do so. The problem is, with so many images, my ...
1
vote
1answer
15 views
Why does isna()/isnull() function show different values compared to value_counts()?
I am quite new to Machine Learning/Data science so I started doing my first Kaggle competition problem from here: https://www.kaggle.com/c/house-prices-advanced-regression-techniques
I was cleaning ...
0
votes
0answers
59 views
Preserve column order after ColumTransformer
I'm using Pipeline and ColumnTransformer modules from sklearn library to perform feature ...
2
votes
2answers
117 views
How to convert mixed datetime formats into single one in pandas?
I am working with DataFrame which contains multiple datetime formats in one column. For example:
...
1
vote
1answer
14 views
Removing features that correlate with the target label
I know that it is better to remove correlated features, but what if the feature correlates with the target label? So, it gives the most information for prediction? Do I need to remove features that ...
0
votes
1answer
33 views
How to do EDA on large datasets
I have a table in Postgres with ~5million records. When I load the dataset using pandas to perform EDA, I run out of memory.
...
0
votes
2answers
17 views
What is '[0]'in boxcox transformation with scipy.stats.boxcox?
While performing a boxcox transformation on a series of data, I noticed a certain [0], which I don't quite understand
0
votes
0answers
15 views
formatting data from pandas into CNN-compatible array
I have a pandas dataframe of this form:
time
x
y
0
a
b
1
c
d
For my CNN model training, I want to slice the first 5 rows (from t=0 to t=4) and apply my custom function to label this sliced set, ...
0
votes
1answer
16 views
Select current column when iterating dataframe [closed]
I have a dataframe with 1k+ columns like this(i cut it).
I want to iterate the dataframe and check row by row if value is True then assign it the ...
0
votes
0answers
13 views
Is there row limit when saving pandas dataframe as hdf5?
I am making a dataframe with more than 100 million rows while preprocessing the data.
csv file knows that there is a limit on the number of rows, so I want to save it in a different format.
I'm trying ...
0
votes
1answer
101 views
Create a tooltip for each dot on a scatter plot created for a Pandas dataframe
I have a pandas based scatter plot that is created from a dataframe, with the following code:
...
1
vote
0answers
9 views
Using Quarterly Percent Change as Validation
this is a newbie question but it's very simple so I'm hoping it's okay to ask.
I would like to calculate the percent change from one quarter to the next using Pandas and use that as my Y-Test (...
0
votes
1answer
82 views
Error reproducing Eland Demo Notebook
I was trying to reproduce Python Elasticsearch client Eland notebook, available in Eland documentation.
Here are the first lines of code for producing an eland dataframe from my local Elasticsearch ...
0
votes
1answer
16 views
0
votes
0answers
12 views
Model on non-iid data performing badly
I am on this lecture about non-iid data where we generated a timeseries data using the function below:
...
-1
votes
1answer
47 views
Anomaly Detection and Removal/Interpolate [closed]
I am performing a machine learning regression task on time series data. I have a data frame filled with the close prices of various assets and economic data. I am looking to perform outlier detection ...
