Questions tagged [dataframe]

A data frame is a tabular data structure. Usually, it contains data where rows are observations and columns are variables of various types. While "data frame" or "dataframe" is the term used for this concept in several languages (R, Apache Spark, deedle, Maple, the pandas library in Python and the DataFrames library in Julia), "table" is the term used in MATLAB and SQL.

Filter by
Sorted by
Tagged with
0
votes
1answer
15 views

How to Transpose dataframe with custom output? [closed]

I have a data like this: I want to transpose it like this in python:
1
vote
1answer
28 views

Pandas - Sum of multiple specific columns [closed]

I created this script: ...
0
votes
1answer
21 views

Unhashable type 'list' while looping through dataframe in Python [closed]

I have the following dataframe comments. I have segregated a list of users based on certain conditions. I want to get the count of words based on those users which ...
1
vote
1answer
40 views

KeyError: Selecting text from a dataframe based on values of another dataframe

I have the following two dataframes badges and comments. I have created a list of 'gold users' from ...
0
votes
1answer
13 views

Creating a new Dataframe with specific row numbers from another

I've found other posts that refer to creating a new dataframe using specific conditions from another (like ID = 27, etc.) but nothing that allows me to make a new dataframe from specific row numbers ...
0
votes
1answer
20 views

Load data from multiple dataframes containing both path and labels in keras

I am aware that there exists a function in keras.preprocessing.image.ImageDataGenerator called flow_from_dataframe. But this ...
-1
votes
1answer
19 views

How to Select Rows from Pandas DataFrame with condition [closed]

I have a dataframe (csv file) which consists of about 8000 songs (column name "Song")of e.g. 20 singers(column name "Artist"), so each singer has more than one song. I want to ...
0
votes
1answer
17 views

Transform large pd.Series into a DataFrame of n columns

I have a pd.Series with, for instance, n lines. I would like to transform this series in a ...
1
vote
0answers
17 views

Custom DataFrame format for exporting to excel sheets

I have the following DataFrame and I want it exported in an excel file with different sheets having different words as shown in the image ...
0
votes
2answers
43 views

Combine two DataFrames column wise in Pandas

I have 2 Dataframes as shown and I want a new DataFrame where the 1st column is the 1st column of the 1st DataFrame and 2nd column from the 1st column of the 2nd DataFrame. I have tried pivot table ...
0
votes
1answer
16 views

Plot three series on the same plot grouping data by day and month

I've a three years data set which I want to plot so to compare the data by data and month, but I'm having an hard time with the final result. I'm nearly there, I guess, but for some strange reason, ...
1
vote
1answer
24 views

Update a pandas data frame column using Apply,Lambda and Group by Functions

I have a data frame in the format mentioned in the screenshot below. Column 'Candidate Won' has only 'loss' as the column value for all the rows. I want to update the Column 'Candidate Won' to a value ...
0
votes
1answer
13 views

Generate dataframe series from current series which is a list of objects

I currently have a JSON object that looks like this ...
0
votes
1answer
58 views

pandas TypeError: 'range' object cannot be interpreted as an integer

I'm following this link for time-series forecasting. While splitting data set to create the data for the uni-variate model this ...
0
votes
0answers
13 views

error in pandas dataframe when trying to convert dtype object to float

I am trying to aggregate data converted from an xlsx file to a dataframe but it is all dtype object. I tried to convert it to strings and float so I can aggregate it but this is the error I get : <...
0
votes
1answer
19 views

Efficient Dataframe Transformation

So I'm trying to transform a pandas dataframe where I have multiple duplicates of a "user" value (my index being a user ID number), demographic info (which shouldn't change) and various purchase ...
0
votes
1answer
26 views

How do I show my pandas dataframe that I have in my jupyter notebook so that I can ask a question on stack exchange/stack overflow?

I've noticed that people produce such an amazing pandas dataframe which allows people to answer the questions. However, whenever I try to copy my table after doing ...
0
votes
1answer
23 views

Converting a pandas dataframe into a csv with multiple columns

When I convert my pandas dataframe into a csv it all puts it in one column, I would like it to be in multiple columns. this my code ...
1
vote
1answer
55 views

Automatically Detect Valid/Interesting pivot/drill downs of a dataset

Imagine you are given a tabular data set with a limited set of columns and rows and you are asked to find the valid/interesting pivot configuration by exploring the data. The brute-force option is to ...
0
votes
1answer
22 views

Extract seperated value from columns

I am working on this CSV file which is a collection of movie details from IMDB. In this I have a genres column in the dataframe with all the genres of the movies seperated by a pipe (|) What I need ...
0
votes
1answer
93 views

How to add a calculated column in a Pandas dataframe?

I am new to Python/Pandas so I'm struggling a bit here. I have a dataframe with air quality data from 2016 to 2020. I want to calculate the annual rate of change for each measured value to compare ...
0
votes
1answer
35 views

R: Rates of change from an initial value

I have a collection of csvs and must produce yearly rates of change per group within each csv, as well as a rate of change compared to the initial value. I am using the function below to calculate ...
0
votes
1answer
25 views

Better way of writing following Pandas Python code

Following code is given, we need to sum up the medals. ...
1
vote
2answers
34 views

Sort dataframe by date column stored as string

I have dataframe named df1. I want to sort data frame month column according to month (Jan, Feb, March..). For that I used code: ...
0
votes
1answer
34 views

Convert json to dataframe in python

I have a json array f below format ...
0
votes
1answer
72 views

Remove Outliers from Dataframe using pandas in Python

I would like to remove outliers from my dataset. It looks like this: ...
1
vote
1answer
20 views

Filling missing values for Embedded List in Python3

I searched for a similar question but I didn't come across. And I'm new in this area, I hope I explained my question well enough. I have a dataset consist of text data. I store them in a list and ...
0
votes
1answer
56 views

What is the efficient way to use apply method in column of pandas Dataframe for large dataset?

I have a dataset of approximate 1 hundred thousand records. I want to use apply method in each of the records for further data processing but it takes very long time to process (As apply method works ...
0
votes
1answer
114 views

find the difference between two columns in specific rows [closed]

I have the following columns from one data frame ...
0
votes
0answers
40 views

Dask error when reading data from a large zip file

I have a .zip folder with a lot of .csv files which I read into a Dask DataFrame like this: ...
0
votes
1answer
19 views

How to change random number of random elements in a column by group in R?

I have the following data frame: >df Name 1 Ed 2 Ed 3 Bill 4 Tom 5 Tom 6 Tom 7 Ed 8 Bill 9 Bill 10 Bill My goal is that from each group by "Name" ...
4
votes
2answers
119 views

Unformatted data entries

I have been working recently on an independent project using a database for Cybersecurity Attack classification. I imported the database using Pandas (Python) and before starting the processing step, ...
0
votes
0answers
21 views

conversion of any date to datetime in pandas dataframe

I want to define a function that takes in a pandas dateframe and iterates through its columns to find if there is any date field or a timestamp which can be an object. I want to convert those fields ...
0
votes
2answers
24 views

Updating 1st dataframe columns from 2nd data frame coulmns

I have two data frames df1 and df2 which look something like this. ...
1
vote
1answer
60 views

Python dataframe iterate over rows

Dataset ...
-1
votes
1answer
43 views

Sampling randomly from pd.DataFrame, but ignoring NaN values

I am trying to sample random values from a dataframe where the NaN values should be ignored, without dropping the entire row or column. My sampling function at the moment looks like this: ...
-1
votes
1answer
38 views

Custom Pandas function for formatting

My data consists of a lot of Dataframes having the format as below: ...
0
votes
1answer
33 views

Reshaping Pandas DataFrame

I am trying to do the following as shown below. ...
0
votes
0answers
6 views

Handling numpy data in highly frequently updated environment

Having the following environment: Where the input rows (150 million per day) are getting directed into 500 independent DataFrames (300k rows per DataFrame). In ...
0
votes
0answers
21 views

How to combine three lists of data.frames?

I have been reading over similar questions asked here but the solutions aren't working at all and I cannot work out how to make their examples applicable to me. Any help would be appreciated. I ...
0
votes
0answers
15 views

Same Output for every row when doing prediction. Thus I am having wrong predictions

I have implemented an emotion analysis using lstm method, I have trained my model in the first part of the codes and i am now am doing the prediction part. When I run my system, I am having same ...
2
votes
1answer
1k views

Tokenization of data in dataframe in python

I am performing tokenization to each row in my dataframe but the tokenization is being done for only the first row. Can someone please help me. thank you. Below are my codes: ...
0
votes
1answer
309 views

Plot dataframe with two columns on the x axis

how would I plot the data below with the x axis as Year & Month? Each Year-month combination has a unique monthly count (Y). I am unsure how to proceed, given than they are in different columns, ...
0
votes
0answers
12 views

count movements of values from 1 quarter to the other using pandas

i want to count all 10's that moved to 20 from 1 Quarter to the other and total of all 10's in each Quarter Below is the result i want ...
1
vote
1answer
58 views

Convert CSV from an api to a dataframe format in python

I am new to python, I have extracted some reviews from a website and I used the api of the webscrapping tool to import my data in python and the format is in csv. I want to convert this csv to a ...
0
votes
0answers
18 views

PyOD algorithms that support datasets with empty cells

The idea is to find outliers in large datasets. What am I going to use to detect these outliers? I am going to use a library called PyOD for the detection of anomalies which was developed by Yue ...
1
vote
2answers
34 views

Can an entire data frame be used as a prediction variable?

I am attempting to use XGBoost in R to train a model that predicts a fixed number of target variables using all data from previous dates, as well as the two categorical variables (...
1
vote
2answers
1k views

Export pandas dataframe to a nested dictionary from multiple columns

Its a similar question to Export pandas to dictionary by combining multiple row values But in this case I want something different. ...
0
votes
1answer
264 views

How do I read a dat file, for which I don't know its structure?

Is there any way to atleast read the text from the dat file. I have its corresponding mdf file hence I know what all data and columns are there in it. How do I figure out the contents in my dat file. ...
2
votes
2answers
803 views

Passing data to SMOTE after applying train/test split

I'm trying to resample my dataset after splitting it into train and test partitions using SMOTE. Here's my code: ...

1
2 3 4 5