Questions tagged [dataframe]

A data frame is a tabular data structure. Usually, it contains data where rows are observations and columns are variables of various types. While "data frame" or "dataframe" is the term used for this concept in several languages (R, Apache Spark, deedle, Maple, the pandas library in Python and the DataFrames library in Julia), "table" is the term used in MATLAB and SQL.

Filter by
Sorted by
Tagged with
0
votes
1answer
37 views

RAM crashed for XML to DataFrame conversion function

I have created the following function which converts an XML File to a DataFrame. This function works good for files smaller than 1 GB, for anything greater than that the RAM(13GB Google Colab RAM) ...
0
votes
1answer
25 views

Time Series Plot for floating values

I have a Dataframe which looks as shown below I am trying to make a line plot for looking at the peaks for both columns (a,b), I have gotten as far as ...
0
votes
0answers
13 views

Does it make sense to train kalman filter on unscaled (Timeseries-)data, to clean it? [duplicate]

I have Timeseries data to clean and I got the tip to use a kalman Filter. My Question is, does it make sense to "fit" this kalman-filters parameters to an unscaled and unnormalized (...
1
vote
0answers
24 views

How to apply Kalman Filter for Cleaning Timeseries Data effectively without much optimization?

Someone gave me a tip to use kalman filter for my dataset. How time intensive is it to get a good kalman filter running, compared to simple interpolation methods like ...
0
votes
1answer
5k views

Pandas/Python - comparing two columns for matches not in the same row

I have this data: I wanted to compare A and B for matches not by row but rather search A0 if it is in column B and so on. Moreover, I wanted to ignore the .AX in column A because it would not find ...
1
vote
1answer
1k views

Pandas copy() different columns from different dataframes to a new dataframe

I have 2 dataframes that are coming from 2 different Excel files. I want to extract some columns from one file and other columns from the second file to print a new dataframe with the copied columns. ...
1
vote
0answers
52 views

cor() function giving unexpected result for numeric data frame(Updated) [closed]

I have a file named "specdata" which has a lot of CSV that each have the columns: ("Date", "sulfate", "nitrate", "ID"). I want to calculate the ...
1
vote
1answer
59 views

How to access the data of the column on which some groupby operation has been carried out? [closed]

Suppose there is a pandas dataframe which has one column consisting of names of something, and there are multiple entries respective to each entry in the first column. To count the number of entries ...
0
votes
1answer
4k views

ValueError: ('The truth value of a Series is ambiguous after applying if/else condition in Pandas data frames

I want to create a new variable for the dataframe details called lower after iterating through multiple ...
1
vote
2answers
5k views

AttributeError: 'DataFrame' object has no attribute 'ix' [closed]

I was learning Classification-based collaboration system and while running the code I faced the error AttributeError: 'DataFrame' object has no attribute 'ix'. Here is the code I write until now. <...
0
votes
1answer
23 views

How to Transpose dataframe with custom output? [closed]

I have a data like this: I want to transpose it like this in python:
1
vote
1answer
310 views

Pandas - Sum of multiple specific columns [closed]

I created this script: ...
0
votes
1answer
736 views

Unhashable type 'list' while looping through dataframe in Python [closed]

I have the following dataframe comments. I have segregated a list of users based on certain conditions. I want to get the count of words based on those users which ...
2
votes
1answer
108 views

KeyError: Selecting text from a dataframe based on values of another dataframe

I have the following two dataframes badges and comments. I have created a list of 'gold users' from ...
0
votes
1answer
1k views

Creating a new Dataframe with specific row numbers from another

I've found other posts that refer to creating a new dataframe using specific conditions from another (like ID = 27, etc.) but nothing that allows me to make a new dataframe from specific row numbers ...
0
votes
1answer
55 views

Load data from multiple dataframes containing both path and labels in keras

I am aware that there exists a function in keras.preprocessing.image.ImageDataGenerator called flow_from_dataframe. But this ...
-1
votes
1answer
27 views

How to Select Rows from Pandas DataFrame with condition [closed]

I have a dataframe (csv file) which consists of about 8000 songs (column name "Song")of e.g. 20 singers(column name "Artist"), so each singer has more than one song. I want to ...
0
votes
1answer
22 views

Transform large pd.Series into a DataFrame of n columns

I have a pd.Series with, for instance, n lines. I would like to transform this series in a ...
1
vote
0answers
29 views

Custom DataFrame format for exporting to excel sheets

I have the following DataFrame and I want it exported in an excel file with different sheets having different words as shown in the image ...
0
votes
2answers
335 views

Combine two DataFrames column wise in Pandas

I have 2 Dataframes as shown and I want a new DataFrame where the 1st column is the 1st column of the 1st DataFrame and 2nd column from the 1st column of the 2nd DataFrame. I have tried pivot table ...
1
vote
1answer
28 views

Plot three series on the same plot grouping data by day and month

I've a three years data set which I want to plot so to compare the data by data and month, but I'm having an hard time with the final result. I'm nearly there, I guess, but for some strange reason, ...
1
vote
1answer
1k views

Update a pandas data frame column using Apply,Lambda and Group by Functions

I have a data frame in the format mentioned in the screenshot below. Column 'Candidate Won' has only 'loss' as the column value for all the rows. I want to update the Column 'Candidate Won' to a value ...
0
votes
1answer
16 views

Generate dataframe series from current series which is a list of objects

I currently have a JSON object that looks like this ...
0
votes
0answers
36 views

error in pandas dataframe when trying to convert dtype object to float

I am trying to aggregate data converted from an xlsx file to a dataframe but it is all dtype object. I tried to convert it to strings and float so I can aggregate it but this is the error I get : <...
0
votes
1answer
34 views

Efficient Dataframe Transformation

So I'm trying to transform a pandas dataframe where I have multiple duplicates of a "user" value (my index being a user ID number), demographic info (which shouldn't change) and various purchase ...
0
votes
1answer
35 views

How do I show my pandas dataframe that I have in my jupyter notebook so that I can ask a question on stack exchange/stack overflow?

I've noticed that people produce such an amazing pandas dataframe which allows people to answer the questions. However, whenever I try to copy my table after doing ...
1
vote
1answer
56 views

Automatically Detect Valid/Interesting pivot/drill downs of a dataset

Imagine you are given a tabular data set with a limited set of columns and rows and you are asked to find the valid/interesting pivot configuration by exploring the data. The brute-force option is to ...
0
votes
2answers
67 views

Extract seperated value from columns

I am working on this CSV file which is a collection of movie details from IMDB. In this I have a genres column in the dataframe with all the genres of the movies seperated by a pipe (|) What I need ...
-1
votes
2answers
3k views

How to add a calculated column in a Pandas dataframe?

I am new to Python/Pandas so I'm struggling a bit here. I have a dataframe with air quality data from 2016 to 2020. I want to calculate the annual rate of change for each measured value to compare ...
0
votes
1answer
95 views

R: Rates of change from an initial value

I have a collection of csvs and must produce yearly rates of change per group within each csv, as well as a rate of change compared to the initial value. I am using the function below to calculate ...
0
votes
1answer
63 views

Better way of writing following Pandas Python code

Following code is given, we need to sum up the medals. ...
1
vote
2answers
1k views

Sort dataframe by date column stored as string

I have dataframe named df1. I want to sort data frame month column according to month (Jan, Feb, March..). For that I used code: ...
0
votes
2answers
1k views

Convert json to dataframe in python

I have a json array f below format ...
0
votes
2answers
4k views

Remove Outliers from Dataframe using pandas in Python

I would like to remove outliers from my dataset. It looks like this: ...
2
votes
1answer
25 views

Filling missing values for Embedded List in Python3

I searched for a similar question but I didn't come across. And I'm new in this area, I hope I explained my question well enough. I have a dataset consist of text data. I store them in a list and ...
0
votes
1answer
299 views

What is the efficient way to use apply method in column of pandas Dataframe for large dataset?

I have a dataset of approximate 1 hundred thousand records. I want to use apply method in each of the records for further data processing but it takes very long time to process (As apply method works ...
0
votes
1answer
1k views

find the difference between two columns in specific rows [closed]

I have the following columns from one data frame ...
1
vote
0answers
245 views

Dask error when reading data from a large zip file

I have a .zip folder with a lot of .csv files which I read into a Dask DataFrame like this: ...
0
votes
1answer
68 views

How to change random number of random elements in a column by group in R?

I have the following data frame: >df Name 1 Ed 2 Ed 3 Bill 4 Tom 5 Tom 6 Tom 7 Ed 8 Bill 9 Bill 10 Bill My goal is that from each group by "Name" ...
4
votes
2answers
148 views

Unformatted data entries

I have been working recently on an independent project using a database for Cybersecurity Attack classification. I imported the database using Pandas (Python) and before starting the processing step, ...
1
vote
3answers
46 views

Updating 1st dataframe columns from 2nd data frame coulmns

I have two data frames df1 and df2 which look something like this. ...
2
votes
1answer
80 views

Python dataframe iterate over rows

Dataset ...
-1
votes
1answer
281 views

Sampling randomly from pd.DataFrame, but ignoring NaN values

I am trying to sample random values from a dataframe where the NaN values should be ignored, without dropping the entire row or column. My sampling function at the moment looks like this: ...
-1
votes
1answer
190 views

Custom Pandas function for formatting

My data consists of a lot of Dataframes having the format as below: ...
0
votes
1answer
36 views

Reshaping Pandas DataFrame

I am trying to do the following as shown below. ...
2
votes
1answer
7k views

Tokenization of data in dataframe in python

I am performing tokenization to each row in my dataframe but the tokenization is being done for only the first row. Can someone please help me. thank you. Below are my codes: ...
0
votes
1answer
2k views

Plot dataframe with two columns on the x axis

how would I plot the data below with the x axis as Year & Month? Each Year-month combination has a unique monthly count (Y). I am unsure how to proceed, given than they are in different columns, ...
1
vote
1answer
253 views

Convert CSV from an api to a dataframe format in python

I am new to python, I have extracted some reviews from a website and I used the api of the webscrapping tool to import my data in python and the format is in csv. I want to convert this csv to a ...
1
vote
2answers
41 views

Can an entire data frame be used as a prediction variable?

I am attempting to use XGBoost in R to train a model that predicts a fixed number of target variables using all data from previous dates, as well as the two categorical variables (...
4
votes
2answers
5k views

Export pandas dataframe to a nested dictionary from multiple columns

Its a similar question to Export pandas to dictionary by combining multiple row values But in this case I want something different. ...