Questions tagged [pandas]

pandas is a python library for Panel Data manipulation and analysis, e.g. multidimensional time series and cross-sectional data sets commonly found in statistics, experimental science results, econometrics, or finance.

Filter by
Sorted by
Tagged with
2
votes
1answer
74 views

KeyError: Selecting text from a dataframe based on values of another dataframe

I have the following two dataframes badges and comments. I have created a list of 'gold users' from ...
1
vote
1answer
55 views

pandas series match multiple keywords

Is there a direct python pandas method to match values of series and update different series with some string ? I couldn’t find any direct method of doing it. Here the match is to find a value in a ...
0
votes
1answer
1k views

Comparing dataframe object with string value in django

I'm implementing machine learning model and using training dataset from MySQL table and all this is built upon Django. So basically all the calculations are done by converting entire data from MySQL ...
1
vote
0answers
15 views

Making a tagged Part-of-Speech corpus with the help of a lexicon

I have a part-of-speech lexicon that has two columns, words and part-of-speech tags, inside a Pandas's Dataframe. Also, I have a list of tokens (words) in another Dataframe. I want to take each ...
0
votes
1answer
32 views

Train and Test Data Split for a Tweet Classification Task

I am trying to train a few Machine Learning (ML) algorithms such as SVM, NB and Random Forest to do binary classification on disaster tweets. During this project, I want to train ML algorithms for a ...
0
votes
1answer
26 views

Getting extra characters while loading data

While loading data using read_sas from sas dataset in Python getting b' along with every string. How can I fix this?
13
votes
2answers
95k views

How to plot two columns of single DataFrame on Y axis

i have two DataFrames(Action,Comedy). Action contains two columns(year,rating) ratings columns contains average rating with respect to year. Comedy Dataframe contains same two columns with different ...
1
vote
2answers
6k views

How to save Tensorflow predictions to data frame?

I am new to Tensorflow. I have trained Tensorflow model, but I need to take model predictions and add them to my original test set as a column. How can I do that? ...
2
votes
2answers
31 views

In feature selection, I came across a situation where NaN were filled by median of the column values

Why the median value is used for NaN? Why not something else like mean? What is the logic behind using the median value?
0
votes
1answer
27 views

List value in Pandas DataFrame column makes analysis harder

Should I move to database? I have a list of courses data in JSON format that looks like this: ...
0
votes
1answer
711 views

Error with pandas dataframe (needs to be 1-dimensional)

I am trying to determine the conformal predictions for my model with my data. But it gives me following error that occurs at icp.calibrate(X_cal, y_cal) : ...
0
votes
1answer
22 views

Adding and multiplying higher values based on different columns of a dataframe

I am trying to calculate scores based on different values in the dataframe. Since these scores are based on different conditions, I am having problems where in the final calculation, which I need in a ...
2
votes
2answers
691 views

pandas data frame doesn't show any thing ,when view as data frame in pycharm

import pandas as pd; dataSet = pd.read_csv("winequality-red.csv"); dataSet.describe(include = 'all'); When view data set as data frame ,it show empty table.But ...
0
votes
1answer
2k views

How to compare and find common values from different columns in same dataframe?

I would like to compare two columns and find common value sets in each column, then output the rows with the common values. Let's say I have a dataframe with: ...
0
votes
1answer
27 views

Pandas: Compare two Dataframe and Groupby categorical [closed]

My Question is about pandas DataFrame, I have two DataFrame both follow the same structure. df_1: ...
0
votes
1answer
32 views

How do I transform a file to .txt file using pandas?

I have to submit a machine learning project, and it has to be node in a .txt file. I know that if I am using pandas, and I want to transform a file from another format to .csv format I can use .to_csv(...
3
votes
1answer
99 views

Feature Engineering Lists\Vectors as values in dataframe

Let's say I have a dataframe where some of the columns have lists of strings as values. I would like to use ML Algorithms on this dataframe. In this case, I can: I could add many columns of 1's and ...
2
votes
2answers
811 views

Partitioning data into features/labels and train/test after reading from csv file

I need to read data from csv file and then first partition that data into features and labels and then into training and testing set. However, there are several issues cropping up again and again. ...
0
votes
2answers
31 views

What is the best way to treat datetime in the preprocessing step of machine learning

I have two datetime columns in my dataset. What I have done so far I have extracted year, ...
2
votes
1answer
75 views

Reading values from a column into a variable and then correlating using Python

I want to be able to correlate values from various IDs, where the date is the same with one another. The data looks something like this; ...
0
votes
1answer
23 views

How to add Error Bar to Matplotlib line plot?

I have the following dataset which I use to plot a line plot. The plot is obtained as the mean of values obtained from the data. I want to add error bars to this ...
0
votes
1answer
40 views

how to limit datetime in x axis

Hello folks, I am trying to plot three variables in a single XY plot. I am having a problem setting the x-axis limit which is for time. I tried setting limits using x_limit but getting ...
0
votes
2answers
23 views

Confusion between doing .copy() and using [[ ]]

I am reading "Hands-On Machine Learning with Scikit-Learn and Tensorflow" by Aurélien Géron. Sometimes he does xyz = DataFrame[["col1"]] Which ...
1
vote
2answers
56 views

What is the best way to store images in python for machine learning

I am currently working on a classification problem that requires me to classify whether an image contains cancerous tissue cells or not. Each image is 50x50x3 pixels, the 3 is for RGB values. So far ...
-1
votes
2answers
2k views

How to add a calculated column in a Pandas dataframe?

I am new to Python/Pandas so I'm struggling a bit here. I have a dataframe with air quality data from 2016 to 2020. I want to calculate the annual rate of change for each measured value to compare ...
0
votes
1answer
35 views

How to Manipulate data for multiple visits per person?

I have a query to solve. I have data regarding customers and number of visits done to them. These are in two tables. So I want to join two table and create different features so that I can find better/...
0
votes
1answer
19 views

Looping 2 DataFrame Pandas [closed]

I am learning about pandas and find an interesting case i try to find total sales for every genre that exist in the data frame,I try looping a few time and get a dead end,maybe you can give me a few ...
0
votes
2answers
28 views

Query relating to Pandas Rows manipulation

I have a query regarding Pandas data manipulation. Let's say I have a dataframe, df with following structure. A B C 1 1 7 5 3 3 3 3 2 7 5 2 5 NaN 2 We have 3 ...
0
votes
2answers
89 views

Problem extracting words from dataframe

I have the following dataset which is a .json file: and I would like to get the first word for every string inside lista_asm, so I would like to get: jmp,push,...
0
votes
0answers
12 views

Interpolate function isn't returning values

My input DataFrame is structured like this: I have a list of unique IDs (UIDs) which I would like to compare with each other, year on year (based on the "Age" column). The challenge is, the ...
1
vote
1answer
29 views

Normalize data from different groups

I have data that have been grouped in 27 groups by different criteria. The reason for these groupings is to show that each group has a different behavior. However, I would like to normalize everything ...
0
votes
1answer
29 views

Efficient Dataframe Transformation

So I'm trying to transform a pandas dataframe where I have multiple duplicates of a "user" value (my index being a user ID number), demographic info (which shouldn't change) and various purchase ...
3
votes
3answers
78 views

In a pandas dataframe can I convert my column values into numbers?

So I am new to all this. I was wondering in pandas can I convert my column values into numbers? I'll try and give an example to explain what I mean So say for example I have a column called, 'animals',...
0
votes
0answers
34 views

conversion of any date to datetime in pandas dataframe

I would like to define a function that takes in a pandas dateframe and iterates through its columns to find if there is any date field or a timestamp which can be an object. I would then like to ...
0
votes
1answer
16 views

How to sample with guaranteed minimum spacing?

I have a timeseries stored as a pandas.Series, and have computed the autocorrelation to be negiligable for 12 hours lag. Therefore, I want to test the approach of ...
1
vote
1answer
32 views

How to summarize multiple time series like dataset

I am running an experiment where I am sampling CPU utilization every second. A snippet of the data from each experiment run is something like this: ...
0
votes
2answers
251 views

The actual results and results from pickle files are not matching in pandas for DBSCAN clustering

I've built a DBSCAN clustering model. The output result and the result after using the pickle files are not matching. Based on HD and MC column, I am clustering WT column. ...
1
vote
1answer
611 views

Update a pandas data frame column using Apply,Lambda and Group by Functions

I have a data frame in the format mentioned in the screenshot below. Column 'Candidate Won' has only 'loss' as the column value for all the rows. I want to update the Column 'Candidate Won' to a value ...
1
vote
0answers
14 views

Filtering redundant Association rules

After performing association rule mining, (using mlxtend library), I have gotten a pandas dataframe containing all the association rules. This contains too many rules, therefore we must reduce the ...
3
votes
3answers
17k views

pandas datetime to Unix

I need to convert a datetime.date(2007, 4, 30) to a Unix timestamp in pandas. thank you. my datable: ...
4
votes
3answers
10k views

Multiple filtering pandas columns based on values in another column

I have a pandas dataframe df1: Now, I want to filter the rows in df1 based on unique combinations of ...
1
vote
2answers
6k views

How to turn a series that contains pandas.datetime.timedelta to floats e.g. hours or days?

I want to plot some data that contains timedeltas. However, the numpy plot() and errorbar() return: ...
2
votes
1answer
74 views

Plotting three lines on the same plot (with four hr frequency)

I want to plot the graph with datetime . I saw so many questions similiar to my questions. But that answers didn't work for me. Here is the code that I used: ...
2
votes
1answer
22 views

How to Set the Same Categorical Codes to Train and Test data? Python-Pandas

NOTE: If someone else it's wondering about this topic, I understand you're getting deeper in the Data Analysis world, so I did this question before to learn that: You encode categorical values as ...
0
votes
0answers
10 views

time syncronisation

Introduction Given is a vehicle which is moving past a station. If the vehicle passes the station (for simplicity determined by GPS) it sends RF-signals, to the station saying "Hello I am here ...
0
votes
2answers
70 views

Skip NA in Mean function within Pandas agg function [closed]

Here's my working code: new_df = old_df.groupby('store', as_index = False).agg({'reviewScore': 'mean'}) I would like to add an argument to the mean function: ...
0
votes
1answer
40 views

Look up a number inside a list within a pandas cell, and return corresponding string value from a second DF

(I've edited the first column name in the labels_df for clarity) I have two DataFrames, train_df and labels_df. train_df has integers that map to attribute names in the labels_df. I would like to ...
3
votes
0answers
42 views

Detecting pushups based on pose data

I've been playing with Google's MLKit, and decided to detect push ups. As a quick test, I took the position of the left shoulder, and plotted the Y Axis. Here's how a variety of trials look: Five ...

1 2
3
4 5
20