Questions tagged [pandas]

pandas is a python library for Panel Data manipulation and analysis, e.g. multidimensional time series and cross-sectional data sets commonly found in statistics, experimental science results, econometrics, or finance.

Filter by
Sorted by
Tagged with
0
votes
1answer
244 views

Pandas: Compare two Dataframe and Groupby categorical [closed]

My Question is about pandas DataFrame, I have two DataFrame both follow the same structure. df_1: ...
0
votes
1answer
200 views

How do I transform a file to .txt file using pandas?

I have to submit a machine learning project, and it has to be node in a .txt file. I know that if I am using pandas, and I want to transform a file from another format to .csv format I can use .to_csv(...
0
votes
2answers
154 views

What is the best way to treat datetime in the preprocessing step of machine learning

I have two datetime columns in my dataset. What I have done so far I have extracted year, ...
0
votes
1answer
21 views

Looping 2 DataFrame Pandas [closed]

I am learning about pandas and find an interesting case i try to find total sales for every genre that exist in the data frame,I try looping a few time and get a dead end,maybe you can give me a few ...
0
votes
2answers
384 views

What pandas function is most similar to the transpose procedure in SAS?

I am trying to transpose Amt by Txn_ID across Item_Desc in Python using pandas. Which pandas ...
0
votes
2answers
30 views

Query relating to Pandas Rows manipulation

I have a query regarding Pandas data manipulation. Let's say I have a dataframe, df with following structure. A B C 1 1 7 5 3 3 3 3 2 7 5 2 5 NaN 2 We have 3 ...
0
votes
1answer
2k views

Choropleth Map with plotly.express doesn't show map, just legend

I copied the "Indexing by GeoJSON Properties" example as seen on the Choropleth Maps in Python page. The code was executed with no error but there was no map showing, only the legend. In ...
3
votes
3answers
140 views

In a pandas dataframe can I convert my column values into numbers?

So I am new to all this. I was wondering in pandas can I convert my column values into numbers? I'll try and give an example to explain what I mean So say for example I have a column called, 'animals',...
0
votes
2answers
426 views

Problem extracting words from dataframe

I have the following dataset which is a .json file: and I would like to get the first word for every string inside lista_asm, so I would like to get: jmp,push,...
0
votes
1answer
17 views

How to sample with guaranteed minimum spacing?

I have a timeseries stored as a pandas.Series, and have computed the autocorrelation to be negiligable for 12 hours lag. Therefore, I want to test the approach of ...
1
vote
0answers
133 views

Filtering redundant Association rules

After performing association rule mining, (using mlxtend library), I have gotten a pandas dataframe containing all the association rules. This contains too many rules, therefore we must reduce the ...
1
vote
1answer
255 views

Running scikit-learn with large volume

I need to run a Random Forest process with scikit-learn. To train the model, I have a database table with 10 million rows of features. The question is: what is the ...
0
votes
2answers
2k views

Skip NA in Mean function within Pandas agg function [closed]

Here's my working code: new_df = old_df.groupby('store', as_index = False).agg({'reviewScore': 'mean'}) I would like to add an argument to the mean function: ...
1
vote
1answer
114 views

Train and Test Data Split for a Tweet Classification Task

I am trying to train a few Machine Learning (ML) algorithms such as SVM, NB and Random Forest to do binary classification on disaster tweets. During this project, I want to train ML algorithms for a ...
3
votes
0answers
64 views

Detecting pushups based on pose data

I've been playing with Google's MLKit, and decided to detect push ups. As a quick test, I took the position of the left shoulder, and plotted the Y Axis. Here's how a variety of trials look: Five ...
2
votes
2answers
1k views

Pandas loads a csv file incorrectly, but without throwing an error [closed]

So I tried loading some data through pandas to practice manipulating it, but I ran into a slight problem. Basically, the pandas load the data improperly. And it does so consistently. Let me show you ...
1
vote
0answers
40 views

Compare ranking across trials with pandas

I am carrying out several runs of an experiment with different configurations on many different inputs. The experiment has a random element and I'm trying to investigate whether this randomness ...
-1
votes
2answers
7k views

How to deal with nan values after merging / joining two dataframes? [closed]

A lot of time after merging two pandas dataframes, I end up with NaNs in the new dataframe, that's just how the way it is, because one csv does not have all the ID's that the other has (Two dataframes ...
0
votes
1answer
2k views
-1
votes
1answer
38 views

Unable to successfully merge dataframes in pandas along labels

I have two different dataframes, they both share the same labels, "Country" and "Year", I am trying to merge these together as one by these two columns. This is my code: ...
3
votes
2answers
1k views

How to Set the Same Categorical Codes to Train and Test data? Python-Pandas

NOTE: If someone else it's wondering about this topic, I understand you're getting deeper in the Data Analysis world, so I did this question before to learn that: You encode categorical values as ...
0
votes
1answer
994 views

How to convert DataFrame column to Rows in Python?

I have the following dataset in df_1 which I want to convert into the format of df_2. In df_2...
5
votes
2answers
8k views

How to split train/test datasets having equal classes proportion

I would like to know how I can split in an equal number the following Target 0 1586 1 318 in order to have the same proportion of 0 and 1 classes in a ...
0
votes
1answer
125 views

How to add Error Bar to Matplotlib line plot?

I have the following dataset which I use to plot a line plot. The plot is obtained as the mean of values obtained from the data. I want to add error bars to this ...
1
vote
3answers
204 views

How to interprete percentile information from the describe function in Pandas?

I am a bit stumped on how to interpret the percentile information you see when you call the describe function on dataframes in Pandas. I believe I have a basic understanding of what percentile means. ...
2
votes
1answer
994 views

My corr() function in Python keeps resulting in an "ValueError: The truth value of a Series is ambiguous..." [closed]

I am a very inexperienced programmer, this is my first question on the Data Science StackExchange, I sorry if it is formatted poorly or comes across as basic. For some strange reason, in Python, ...
1
vote
0answers
272 views

PyTorch: Train without dataloader (loop trough dataframe instead)

I was wondering if it is bad practice to instead of using built in tools such as dataloader just loop trough each row in a pandas df. Lets say I am doing text classification and my training loop looks ...
1
vote
1answer
29 views

How do I control sensor data readings against (measured) outside influences?

I need some hints in this problem I have. I have a dataset of electro-dermal activity readings that are, by nature, influenced by the movements of the person that is being surveyed. In the same ...
3
votes
4answers
23k views

ValueError: Failed to convert a NumPy array to a Tensor (Unsupported object type int) in Python

I have written the following code for a neural network to perform regression on a dataset, but I am getting a ValueError. I have looked up to different answers and ...
-1
votes
3answers
2k views

How do I use lagged independent variable in statsmodel OLS regression?

If there is good reason to believe that an independent variable (x) has a lagged effect on dependent ...
2
votes
1answer
1k views

How to visualise a large correlation matrix?

I have a dataset with 24 variables, 21 of them numeric. As part of model building I decided to look into the correlation between features and so what I get is a large correlation matrix (21 * 21). Now ...
0
votes
2answers
703 views

including Zeros counts categories with pandas value_counts()

What i want is including zero counts categories while generating frequencies for categorical variables ...
0
votes
1answer
301 views

Replacing value not in list in Pandas [closed]

Say I have a data frame in Python. If I want to replace all values in the say, the Size column that are not 'M' or 'S' or 'L' with nan, how do I do so? Thanks in advance
0
votes
0answers
2k views

How to fix IndexError: Too many levels: Index has only 1 level, not 2 in Python

I have written the following code that has a function model_data to perform a particular set of tasks. I have to pass the list of ...
1
vote
1answer
105 views

Why we can't Remove features with missing values in Data Preprocessing [closed]

In a Real Time Dataset, There are many missing values available in the Dataset and also we need to deal with data preprocessing. And there are many ways to minimize the problem of missing values ...
1
vote
1answer
19 views

How we can Identify Specific Feature from a larger amount of Dataset?

In Machine Learning, we need to play with any kind of datasets. In the Dataset, There are too many records and features, Some datasets had lots of features (sometimes it's called ...
2
votes
3answers
8k views

How to combine and separate test and train data for data cleaning?

I am working on an ML model in which I have been provided the data in 2 files test.csv and train.csv. I want to perform data ...
1
vote
0answers
15 views

How to find average of Duplicate Column Headers in Python dataframe [closed]

I have data in this format ...
1
vote
1answer
58 views

How to apply multiple filter in Data Frame? [closed]

How to implement multiple filters for checking data cell in a range ? Suppose, I have a list of numbers like, ...
0
votes
1answer
36 views

How to obtain a count of the classes of a categorical var within a certain time interval for a time-stamped data?

I have a dataframe with several categorical variables. But for simplicity let's assume there is only 1 categorical variables with 3 classes. I want to obtain the counts of these classes within a ...
1
vote
1answer
29 views

Count occurences of rows based one cell content with pandas [closed]

I am currently struggling with the pandas framework. Let's imagine the following Data Frame: ...
0
votes
1answer
408 views

Time series plot using netcdf data in python

I am trying to plot a time series for a 3d plot. I know that the variable must be made 1d inorder to make it viable for the time series plot. Actually I want the data within a box of latitude and ...
2
votes
1answer
1k views

Group K-fold with target stratification

I have a pd.DataFrame ...
0
votes
1answer
1k views

Comparing 2 columns from separate dataframes and copy some row values from one df to another if column value matches in pandas

Im relatively new to pandas andpython . I have a simple problem but I can not find the solution . I have 2 dataframes DF1 and DF2 as shown in image Both dataframes have ID coluumns. I wannt to check ...
0
votes
1answer
593 views

Calculate all days for each years between two dates in a pandas dataframe

I'm new in python and coding. I'm doing a university project exercise. Last question is : " For each year, compute the total amount of loans. Each loan that has planned expiration time and ...
0
votes
1answer
226 views

Machine Learning: Predicting target based on a feature

I have a df looks as follow: -It is very likely that the same feature1Xfeature2Xfeature3 combination will appear multiple times....
0
votes
1answer
35 views

What is the best way to visualize range of data?

If I a have data like a bunch of classes where each class can have a different number of students, and each student has a grade (0-100), for example like this: ...
0
votes
2answers
340 views

Pandas get_dummies() rows dropping after joining back with X

I'm having an issue that I can't explain and am hoping I am missing something simple. I have a large dataset of shape(45Million+, 51) and am loading it in for some analyses (classifiers, deep learning,...
1
vote
1answer
8k views

ValueError: Found array with 0 sample(s) (shape=(0, 13)) while a minimum of 1 is required [closed]

I am new to this logistic regression and finding this error. Please help. ...
0
votes
1answer
70 views

How to handle Tokenized text content which is given in number?

i have one data set of customer review, but the text data is given is tokenized text number. I am unable to proceed thinking about how to proceed? As I am encountering such data set the first time, so ...

1
4 5
6
7 8
24