Questions tagged [pandas]

pandas is a python library for Panel Data manipulation and analysis, e.g. multidimensional time series and cross-sectional data sets commonly found in statistics, experimental science results, econometrics, or finance.

Filter by
Sorted by
Tagged with
0
votes
1answer
25 views

Getting extra characters while loading data

While loading data using read_sas from sas dataset in Python getting b' along with every string. How can I fix this?
-1
votes
0answers
9 views

how to convert a large xml file containing multiple xml pages into python pandas? [closed]

Every xml page is separated by a \n in the large file. So i have a large xml file sometimes about 0.5gb and in it there are multiple xml pages separated by \n. imagine that i need to take this file ...
0
votes
1answer
18 views

How can I get all column values from a panda dataframe and save them into one string

this question has been ask couple of times but none of the answer is working for that what I want. For example I have this excel Dokument: ...
12
votes
2answers
92k views

How to plot two columns of single DataFrame on Y axis

i have two DataFrames(Action,Comedy). Action contains two columns(year,rating) ratings columns contains average rating with respect to year. Comedy Dataframe contains same two columns with different ...
1
vote
2answers
5k views

How to save Tensorflow predictions to data frame?

I am new to Tensorflow. I have trained Tensorflow model, but I need to take model predictions and add them to my original test set as a column. How can I do that? ...
2
votes
2answers
28 views

In feature selection, I came across a situation where NaN were filled by median of the column values

Why the median value is used for NaN? Why not something else like mean? What is the logic behind using the median value?
0
votes
1answer
17 views

List value in Pandas DataFrame column makes analysis harder

Should I move to database? I have a list of courses data in JSON format that looks like this: ...
0
votes
2answers
24 views
0
votes
1answer
402 views

Error with pandas dataframe (needs to be 1-dimensional)

I am trying to determine the conformal predictions for my model with my data. But it gives me following error that occurs at icp.calibrate(X_cal, y_cal) : ...
-1
votes
0answers
14 views

What is the wrong in my histogram code?

The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all() ...
0
votes
1answer
20 views

Adding and multiplying higher values based on different columns of a dataframe

I am trying to calculate scores based on different values in the dataframe. Since these scores are based on different conditions, I am having problems where in the final calculation, which I need in a ...
-1
votes
2answers
61 views

How do I use lagged independent variable in statsmodel OLS regression?

If there is good reason to believe that an independent variable (x) has a lagged effect on dependent ...
2
votes
2answers
631 views

pandas data frame doesn't show any thing ,when view as data frame in pycharm

import pandas as pd; dataSet = pd.read_csv("winequality-red.csv"); dataSet.describe(include = 'all'); When view data set as data frame ,it show empty table.But ...
0
votes
1answer
16 views

Choropleth Map with plotly.express doesn't show map, just legend

I copied the "Indexing by GeoJSON Properties" example as seen on the Choropleth Maps in Python page. The code was executed with no error but there was no map showing, only the legend. In ...
0
votes
1answer
2k views

How to compare and find common values from different columns in same dataframe?

I would like to compare two columns and find common value sets in each column, then output the rows with the common values. Let's say I have a dataframe with: ...
0
votes
1answer
20 views

Pandas: Compare two Dataframe and Groupby categorical [closed]

My Question is about pandas DataFrame, I have two DataFrame both follow the same structure. df_1: ...
0
votes
1answer
28 views

How do I transform a file to .txt file using pandas?

I have to submit a machine learning project, and it has to be node in a .txt file. I know that if I am using pandas, and I want to transform a file from another format to .csv format I can use .to_csv(...
3
votes
1answer
89 views

Feature Engineering Lists\Vectors as values in dataframe

Let's say I have a dataframe where some of the columns have lists of strings as values. I would like to use ML Algorithms on this dataframe. In this case, I can: I could add many columns of 1's and ...
2
votes
2answers
749 views

Partitioning data into features/labels and train/test after reading from csv file

I need to read data from csv file and then first partition that data into features and labels and then into training and testing set. However, there are several issues cropping up again and again. ...
0
votes
2answers
25 views

What is the best way to treat datetime in the preprocessing step of machine learning

I have two datetime columns in my dataset. What I have done so far I have extracted year, ...
2
votes
1answer
75 views

Reading values from a column into a variable and then correlating using Python

I want to be able to correlate values from various IDs, where the date is the same with one another. The data looks something like this; ...
0
votes
0answers
24 views

Clean up a pandas column using regex

I have a Python Pandas dataframe with a column named "codes". This column is very messy and contains the following values: ...
0
votes
1answer
22 views

Better way to replace one type of data in a pandas DF than using for loop [closed]

I have a body of data on which I want to train my ANN on. Unfortunately, within this data, there are dates inserted into two columns (in error). I want to replace these dates with pn.nan, in order ...
0
votes
1answer
12 views

How to add Error Bar to Matplotlib line plot?

I have the following dataset which I use to plot a line plot. The plot is obtained as the mean of values obtained from the data. I want to add error bars to this ...
0
votes
1answer
29 views

how to limit datetime in x axis

Hello folks, I am trying to plot three variables in a single XY plot. I am having a problem setting the x-axis limit which is for time. I tried setting limits using x_limit but getting ...
0
votes
2answers
23 views

Confusion between doing .copy() and using [[ ]]

I am reading "Hands-On Machine Learning with Scikit-Learn and Tensorflow" by Aurélien Géron. Sometimes he does xyz = DataFrame[["col1"]] Which ...
1
vote
2answers
49 views

What is the best way to store images in python for machine learning

I am currently working on a classification problem that requires me to classify whether an image contains cancerous tissue cells or not. Each image is 50x50x3 pixels, the 3 is for RGB values. So far ...
-1
votes
2answers
1k views

How to add a calculated column in a Pandas dataframe?

I am new to Python/Pandas so I'm struggling a bit here. I have a dataframe with air quality data from 2016 to 2020. I want to calculate the annual rate of change for each measured value to compare ...
0
votes
1answer
35 views

How to Manipulate data for multiple visits per person?

I have a query to solve. I have data regarding customers and number of visits done to them. These are in two tables. So I want to join two table and create different features so that I can find better/...
0
votes
1answer
18 views

Looping 2 DataFrame Pandas [closed]

I am learning about pandas and find an interesting case i try to find total sales for every genre that exist in the data frame,I try looping a few time and get a dead end,maybe you can give me a few ...
0
votes
2answers
26 views

Query relating to Pandas Rows manipulation

I have a query regarding Pandas data manipulation. Let's say I have a dataframe, df with following structure. A B C 1 1 7 5 3 3 3 3 2 7 5 2 5 NaN 2 We have 3 ...
0
votes
2answers
83 views

Problem extracting words from dataframe

I have the following dataset which is a .json file: and I would like to get the first word for every string inside lista_asm, so I would like to get: jmp,push,...
0
votes
1answer
31 views

What pandas function is most similar to the transpose procedure in SAS?

I am trying to transpose Amt by Txn_ID across Item_Desc in Python using pandas. Which pandas ...
0
votes
0answers
12 views

Interpolate function isn't returning values

My input DataFrame is structured like this: I have a list of unique IDs (UIDs) which I would like to compare with each other, year on year (based on the "Age" column). The challenge is, the ...
1
vote
1answer
29 views

Normalize data from different groups

I have data that have been grouped in 27 groups by different criteria. The reason for these groupings is to show that each group has a different behavior. However, I would like to normalize everything ...
0
votes
1answer
28 views

Efficient Dataframe Transformation

So I'm trying to transform a pandas dataframe where I have multiple duplicates of a "user" value (my index being a user ID number), demographic info (which shouldn't change) and various purchase ...
3
votes
3answers
69 views

In a pandas dataframe can I convert my column values into numbers?

So I am new to all this. I was wondering in pandas can I convert my column values into numbers? I'll try and give an example to explain what I mean So say for example I have a column called, 'animals',...
0
votes
0answers
33 views

conversion of any date to datetime in pandas dataframe

I would like to define a function that takes in a pandas dateframe and iterates through its columns to find if there is any date field or a timestamp which can be an object. I would then like to ...
1
vote
1answer
24 views

Best practice - memory management when data wrangling

I was wondering what are the best resources to learn the best practice when it comes to memory management. For example, lets say I have the following code below: ...
0
votes
1answer
14 views

How to sample with guaranteed minimum spacing?

I have a timeseries stored as a pandas.Series, and have computed the autocorrelation to be negiligable for 12 hours lag. Therefore, I want to test the approach of ...
1
vote
1answer
31 views

How to summarize multiple time series like dataset

I am running an experiment where I am sampling CPU utilization every second. A snippet of the data from each experiment run is something like this: ...
0
votes
2answers
214 views

The actual results and results from pickle files are not matching in pandas for DBSCAN clustering

I've built a DBSCAN clustering model. The output result and the result after using the pickle files are not matching. Based on HD and MC column, I am clustering WT column. ...
0
votes
0answers
5 views

Help understanding ValueError: setting an array element with a sequence [migrated]

my journey to understand the subtleties of DL in organometallic chemistry continues so, I’m sorry if this question is, let’s say, “too basic”. I have finished the Feature Engineering part, optimizing ...
1
vote
1answer
343 views

Update a pandas data frame column using Apply,Lambda and Group by Functions

I have a data frame in the format mentioned in the screenshot below. Column 'Candidate Won' has only 'loss' as the column value for all the rows. I want to update the Column 'Candidate Won' to a value ...
0
votes
1answer
17 views

Running scikit-learn with large volume

I need to run a Random Forest process with scikit-learn. To train the model, I have a database table with 10 million rows of features. The question is: what is the ...
0
votes
0answers
11 views

Filtering redundant Association rules

After performing association rule mining, (using mlxtend library), I have gotten a pandas dataframe containing all the association rules. This contains too many rules, therefore we must reduce the ...
3
votes
3answers
16k views

pandas datetime to Unix

I need to convert a datetime.date(2007, 4, 30) to a Unix timestamp in pandas. thank you. my datable: ...
3
votes
3answers
10k views

Multiple filtering pandas columns based on values in another column

I have a pandas dataframe df1: Now, I want to filter the rows in df1 based on unique combinations of ...
0
votes
3answers
3k views

1
2 3 4 5
19