Questions tagged [pandas]

pandas is a python library for Panel Data manipulation and analysis, e.g. multidimensional time series and cross-sectional data sets commonly found in statistics, experimental science results, econometrics, or finance.

Filter by
Sorted by
Tagged with
0
votes
1answer
62 views

How to find anomalies in (almost) constant stream of data?

I have a process that (simply put), starts every 5 minutes, collects data, and put that data into the database. More detailed explanation would be that process starts, collects data (which takes some ...
0
votes
0answers
1k views

Excel error may be caused by pandas writing or big data? advise needed

I am reading multiple xml files extracting some data then forming a pandas Dataframe with my data. These are the main steps that I do: open an xml file extract some elements create a pandas dataframe ...
0
votes
0answers
24 views

Surprise Library - RSME coming back as NaN

I'm using the Surprise library to train an SVD model based on some data I've collected from Steam. However, the RSME comes back as NaN when I try to train and test the model. For context, I'm using ...
0
votes
0answers
1k views

Jupyter notebook kernel dies when reading CSV file on M1 Mac

Odd Jupyter notebook problem on M1 mac here. Using these steps I was able to get Jupyter Notebooks to run in the first place. It worked fine executing the following simple code: ...
5
votes
2answers
569 views

how to run bert's pretrained model word embeddings faster?

I'm trying to get word embeddings for clinical data using microsoft/pubmedbert. I have 3.6 million text rows. Converting texts to vectors for 10k rows takes around 30 minutes. So for 3.6 million rows, ...
0
votes
1answer
23 views

How does for loops and elif work with python pandas?

the following code: ...
2
votes
1answer
91 views

How to fill NaN based on the previous row that contains a specific value [closed]

I would like to fill the missing data using the value from the previous trading day of the same stock. In this example, the AAPL stock should be 100. I've tried with ...
0
votes
2answers
552 views

Replace value of a column if the value of another column is a duplicate

I have a Pandas dataframe that contains three columns: ID, name and date. The name column is not unique and may contain duplicates. I want to change the date of the duplicated name to the earliest ...
0
votes
2answers
54 views

Building a summary string in a Pandas groupby (Possibly cross-tab or pivot-table question)

I'm a n00b in Python Pandas. I want to do a particular aggregation / grouping / cross-tab but I know so little of the terminology that I don't even know how to look this up. But here's what I want. ...
0
votes
1answer
520 views

Pandas replace column values by condition with averages based on a value in another column

I have a dataframe with people's CV data. Among others, there's a column with years of experience, and a column with age. Some people stated their age and experience in a way that experience > age. ...
1
vote
1answer
297 views

Merging 2 pandas dataframes on key with duplicates but do not want it to result in two lines

I am trying to merge a "roster" where there are supposed to be several positions with one person per office. However, I have come across two instances where there are two people in one ...
0
votes
0answers
32 views

replacing last duplicate values in pandas

I have a dataframe ,that looks like this | **site** | **Active**| 0 |: deals| Active:| 1 |: deals | InActive:| 2 | : deals | Active:| 3 |:...
0
votes
2answers
1k views

How to find if any variable which is normally distributed in pandas dataframe [closed]

By plotting the df.hist(bins = n) as mentioned in the comments you can get a rough idea of what the distribution is, but there is a catch! The size of your bins ...
0
votes
1answer
138 views

replace values based on Number of duplicate rows are occured

I have a dataframe ,that looks like this site Active 0 deals Active 1 deals Active 2 deals Active 3 discount Active 4 discount Active i ...
0
votes
0answers
16 views

Demand Forecasting - Error in predictions

I've been trying to do daily demand forecasting using the H2O AutoML package for using 4 years of daily sales data as the training set. However, the daily estimates always seems to find a range around ...
0
votes
1answer
43 views

Ordering a material science dataset (properties names, properties scalars, formulas)

I'm dealing with a materials science dataset and I'm in the following situation, I have data organized like this: ...
0
votes
0answers
79 views

ValueError: Found input variables with inconsistent numbers of samples: [10000, 11200]

I am receiving the following error. I can't seem to figure out the exact change I need to make to my code to get solve this problem. I feel like it should be straightforward but I'm banging my head ...
0
votes
2answers
316 views

How to apply Geomertic mean in python

I am trying to built a function to calculate the Geometric mean for Growth column and Avg_growth. Year Company Name FH_PBIDT ...
0
votes
0answers
142 views

Using Downsampling with RFE using pandas and sklearn

I have dataset which consist of 2 class with heavily unbalance distribution (600 for class A and 3000 for class B). When i do RFE to select our feature and test them using Decission tree, it gives a ...
0
votes
0answers
260 views

Applying word embedding function on a large pandas dataset

We have a pandas dataframe with one column ('message') and 3.9 million rows and need to convert these messages to their word embeddings using Google's Universal Sentence encoder. ...
0
votes
0answers
11 views

How should I add reference key column to output after modeling is Complete?

0 I have created a csv of data with the following columns: (1) app_key (2) churn, (3)tenure https://i.stack.imgur.com/NAlFF.png I have performed the following code in order to drop app_key and churn <...
0
votes
1answer
76 views

Fill the null values in the dataframe with condition

traindf[traindf['Gender'] == 'female']['Age'].fillna(value=femage,inplace=True) I've tried to update the null values in the age column in the dataframe with the ...
-1
votes
1answer
26 views

Similar values cleaning [closed]

can someone know algorithm how to identify account names that are similar enough to be potentially merged and imported as one Duplicates with different values: Geico val1 NaN =====>>...
1
vote
0answers
130 views

Making a tagged Part-of-Speech corpus with the help of a lexicon

I have a part-of-speech lexicon that has two columns, words and part-of-speech tags, inside a Pandas's Dataframe. Also, I have a list of tokens (words) in another Dataframe. I want to take each ...
-1
votes
1answer
490 views

How can I get all column values from a panda dataframe and save them into one string [closed]

this question has been ask couple of times but none of the answer is working for that what I want. For example I have this excel Dokument: ...
0
votes
1answer
35 views

List value in Pandas DataFrame column makes analysis harder

Should I move to database? I have a list of courses data in JSON format that looks like this: ...
0
votes
1answer
212 views

Pandas: Compare two Dataframe and Groupby categorical [closed]

My Question is about pandas DataFrame, I have two DataFrame both follow the same structure. df_1: ...
0
votes
1answer
138 views

How do I transform a file to .txt file using pandas?

I have to submit a machine learning project, and it has to be node in a .txt file. I know that if I am using pandas, and I want to transform a file from another format to .csv format I can use .to_csv(...
0
votes
2answers
132 views

What is the best way to treat datetime in the preprocessing step of machine learning

I have two datetime columns in my dataset. What I have done so far I have extracted year, ...
0
votes
1answer
21 views

Looping 2 DataFrame Pandas [closed]

I am learning about pandas and find an interesting case i try to find total sales for every genre that exist in the data frame,I try looping a few time and get a dead end,maybe you can give me a few ...
0
votes
2answers
320 views

What pandas function is most similar to the transpose procedure in SAS?

I am trying to transpose Amt by Txn_ID across Item_Desc in Python using pandas. Which pandas ...
0
votes
2answers
30 views

Query relating to Pandas Rows manipulation

I have a query regarding Pandas data manipulation. Let's say I have a dataframe, df with following structure. A B C 1 1 7 5 3 3 3 3 2 7 5 2 5 NaN 2 We have 3 ...
0
votes
1answer
2k views

Choropleth Map with plotly.express doesn't show map, just legend

I copied the "Indexing by GeoJSON Properties" example as seen on the Choropleth Maps in Python page. The code was executed with no error but there was no map showing, only the legend. In ...
3
votes
3answers
138 views

In a pandas dataframe can I convert my column values into numbers?

So I am new to all this. I was wondering in pandas can I convert my column values into numbers? I'll try and give an example to explain what I mean So say for example I have a column called, 'animals',...
0
votes
2answers
392 views

Problem extracting words from dataframe

I have the following dataset which is a .json file: and I would like to get the first word for every string inside lista_asm, so I would like to get: jmp,push,...
0
votes
1answer
17 views

How to sample with guaranteed minimum spacing?

I have a timeseries stored as a pandas.Series, and have computed the autocorrelation to be negiligable for 12 hours lag. Therefore, I want to test the approach of ...
1
vote
0answers
108 views

Filtering redundant Association rules

After performing association rule mining, (using mlxtend library), I have gotten a pandas dataframe containing all the association rules. This contains too many rules, therefore we must reduce the ...
1
vote
1answer
244 views

Running scikit-learn with large volume

I need to run a Random Forest process with scikit-learn. To train the model, I have a database table with 10 million rows of features. The question is: what is the ...
0
votes
2answers
1k views

Skip NA in Mean function within Pandas agg function [closed]

Here's my working code: new_df = old_df.groupby('store', as_index = False).agg({'reviewScore': 'mean'}) I would like to add an argument to the mean function: ...
1
vote
1answer
103 views

Train and Test Data Split for a Tweet Classification Task

I am trying to train a few Machine Learning (ML) algorithms such as SVM, NB and Random Forest to do binary classification on disaster tweets. During this project, I want to train ML algorithms for a ...
3
votes
0answers
58 views

Detecting pushups based on pose data

I've been playing with Google's MLKit, and decided to detect push ups. As a quick test, I took the position of the left shoulder, and plotted the Y Axis. Here's how a variety of trials look: Five ...
2
votes
2answers
912 views

Pandas loads a csv file incorrectly, but without throwing an error [closed]

So I tried loading some data through pandas to practice manipulating it, but I ran into a slight problem. Basically, the pandas load the data improperly. And it does so consistently. Let me show you ...
1
vote
0answers
40 views

Compare ranking across trials with pandas

I am carrying out several runs of an experiment with different configurations on many different inputs. The experiment has a random element and I'm trying to investigate whether this randomness ...
-1
votes
2answers
6k views

How to deal with nan values after merging / joining two dataframes? [closed]

A lot of time after merging two pandas dataframes, I end up with NaNs in the new dataframe, that's just how the way it is, because one csv does not have all the ID's that the other has (Two dataframes ...
0
votes
1answer
1k views
-1
votes
1answer
36 views

Unable to successfully merge dataframes in pandas along labels

I have two different dataframes, they both share the same labels, "Country" and "Year", I am trying to merge these together as one by these two columns. This is my code: ...
3
votes
2answers
1k views

How to Set the Same Categorical Codes to Train and Test data? Python-Pandas

NOTE: If someone else it's wondering about this topic, I understand you're getting deeper in the Data Analysis world, so I did this question before to learn that: You encode categorical values as ...
0
votes
1answer
892 views

How to convert DataFrame column to Rows in Python?

I have the following dataset in df_1 which I want to convert into the format of df_2. In df_2...
5
votes
2answers
7k views

How to split train/test datasets having equal classes proportion

I would like to know how I can split in an equal number the following Target 0 1586 1 318 in order to have the same proportion of 0 and 1 classes in a ...
0
votes
1answer
123 views

How to add Error Bar to Matplotlib line plot?

I have the following dataset which I use to plot a line plot. The plot is obtained as the mean of values obtained from the data. I want to add error bars to this ...

1
3 4
5
6 7
23