Questions tagged [dataframe]

A data frame is a tabular data structure. Usually, it contains data where rows are observations and columns are variables of various types. While "data frame" or "dataframe" is the term used for this concept in several languages (R, Apache Spark, deedle, Maple, the pandas library in Python and the DataFrames library in Julia), "table" is the term used in MATLAB and SQL.

Filter by
Sorted by
Tagged with
0
votes
0answers
22 views

How to fix IndexError: Too many levels: Index has only 1 level, not 2 in Python

I have written the following code that has a function model_data to perform a particular set of tasks. I have to pass the list of ...
0
votes
3answers
21 views

How to combine and separate test and train data for data cleaning?

I am working on an ML model in which I have been provided the data in 2 files test.csv and train.csv. I want to perform data ...
0
votes
1answer
9 views

Comparing 2 columns from separate dataframes and copy some row values from one df to another if column value matches in pandas

Im relatively new to pandas andpython . I have a simple problem but I can not find the solution . I have 2 dataframes DF1 and DF2 as shown in image Both dataframes have ID coluumns. I wannt to check ...
0
votes
1answer
49 views

Calculate all days for each years between two dates in a pandas dataframe

I'm new in python and coding. I'm doing a university project exercise. Last question is : " For each year, compute the total amount of loans. Each loan that has planned expiration time and ...
0
votes
0answers
34 views

Numpy array value error

This is my code to train a sigmoid neuron for Iris dataset . ...
0
votes
2answers
21 views

sklearn KNN fit throws out error : value too large for dtype('float64')

I have cleaned the data from nan values and infinite values, the only feature which has a large float is the column 8 (it's a sum) I have no Idea how to fix this last error, I tried all previous ...
0
votes
1answer
48 views

Bootstrap clarification for datasets with multiple columns

How do I bootstrap a Dataset/DataFrame with multiple continuous and categorical columns? For eg: Say I'm trying to bootstrap the colour distribution of M&M's and I have 50 bags (samples) each with ...
0
votes
1answer
19 views

RAM crashed for XML to DataFrame conversion function

I have created the following function which converts an XML File to a DataFrame. This function works good for files smaller than 1 GB, for anything greater than that the RAM(13GB Google Colab RAM) ...
0
votes
1answer
19 views

Time Series Plot for floating values

I have a Dataframe which looks as shown below I am trying to make a line plot for looking at the peaks for both columns (a,b), I have gotten as far as ...
0
votes
0answers
10 views

Does it make sense to train kalman filter on unscaled (Timeseries-)data, to clean it?

I have Timeseries data to clean and I got the tip to use a kalman Filter. My Question is, does it make sense to "fit" this kalman-filters parameters to an unscaled and unnormalized (...
0
votes
0answers
10 views

How to apply Kalman Filter for Cleaning Timeseries Data effectively without much optimization?

Someone gave me a tip to use kalman filter for my dataset. How time intensive is it to get a good kalman filter running, compared to simple interpolation methods like ...
0
votes
1answer
132 views

Pandas/Python - comparing two columns for matches not in the same row

I have this data: I wanted to compare A and B for matches not by row but rather search A0 if it is in column B and so on. Moreover, I wanted to ignore the .AX in column A because it would not find ...
1
vote
1answer
54 views

Pandas copy() different columns from different dataframes to a new dataframe

I have 2 dataframes that are coming from 2 different Excel files. I want to extract some columns from one file and other columns from the second file to print a new dataframe with the copied columns. ...
1
vote
0answers
37 views

cor() function giving unexpected result for numeric data frame(Updated) [closed]

I have a file named "specdata" which has a lot of CSV that each have the columns: ("Date", "sulfate", "nitrate", "ID"). I want to calculate the ...
1
vote
1answer
51 views

How to access the data of the column on which some groupby operation has been carried out? [closed]

Suppose there is a pandas dataframe which has one column consisting of names of something, and there are multiple entries respective to each entry in the first column. To count the number of entries ...
0
votes
1answer
388 views

ValueError: ('The truth value of a Series is ambiguous after applying if/else condition in Pandas data frames

I want to create a new variable for the dataframe details called lower after iterating through multiple ...
1
vote
2answers
1k views

AttributeError: 'DataFrame' object has no attribute 'ix' [closed]

I was learning Classification-based collaboration system and while running the code I faced the error AttributeError: 'DataFrame' object has no attribute 'ix'. Here is the code I write until now. <...
0
votes
1answer
16 views

How to Transpose dataframe with custom output? [closed]

I have a data like this: I want to transpose it like this in python:
1
vote
1answer
36 views

Pandas - Sum of multiple specific columns [closed]

I created this script: ...
0
votes
1answer
32 views

Unhashable type 'list' while looping through dataframe in Python [closed]

I have the following dataframe comments. I have segregated a list of users based on certain conditions. I want to get the count of words based on those users which ...
2
votes
1answer
53 views

KeyError: Selecting text from a dataframe based on values of another dataframe

I have the following two dataframes badges and comments. I have created a list of 'gold users' from ...
0
votes
1answer
19 views

Creating a new Dataframe with specific row numbers from another

I've found other posts that refer to creating a new dataframe using specific conditions from another (like ID = 27, etc.) but nothing that allows me to make a new dataframe from specific row numbers ...
0
votes
1answer
28 views

Load data from multiple dataframes containing both path and labels in keras

I am aware that there exists a function in keras.preprocessing.image.ImageDataGenerator called flow_from_dataframe. But this ...
-1
votes
1answer
22 views

How to Select Rows from Pandas DataFrame with condition [closed]

I have a dataframe (csv file) which consists of about 8000 songs (column name "Song")of e.g. 20 singers(column name "Artist"), so each singer has more than one song. I want to ...
0
votes
1answer
19 views

Transform large pd.Series into a DataFrame of n columns

I have a pd.Series with, for instance, n lines. I would like to transform this series in a ...
1
vote
0answers
26 views

Custom DataFrame format for exporting to excel sheets

I have the following DataFrame and I want it exported in an excel file with different sheets having different words as shown in the image ...
0
votes
2answers
57 views

Combine two DataFrames column wise in Pandas

I have 2 Dataframes as shown and I want a new DataFrame where the 1st column is the 1st column of the 1st DataFrame and 2nd column from the 1st column of the 2nd DataFrame. I have tried pivot table ...
1
vote
1answer
21 views

Plot three series on the same plot grouping data by day and month

I've a three years data set which I want to plot so to compare the data by data and month, but I'm having an hard time with the final result. I'm nearly there, I guess, but for some strange reason, ...
1
vote
1answer
114 views

Update a pandas data frame column using Apply,Lambda and Group by Functions

I have a data frame in the format mentioned in the screenshot below. Column 'Candidate Won' has only 'loss' as the column value for all the rows. I want to update the Column 'Candidate Won' to a value ...
0
votes
1answer
15 views

Generate dataframe series from current series which is a list of objects

I currently have a JSON object that looks like this ...
0
votes
1answer
407 views

pandas TypeError: 'range' object cannot be interpreted as an integer

I'm following this link for time-series forecasting. While splitting data set to create the data for the uni-variate model this ...
0
votes
0answers
16 views

error in pandas dataframe when trying to convert dtype object to float

I am trying to aggregate data converted from an xlsx file to a dataframe but it is all dtype object. I tried to convert it to strings and float so I can aggregate it but this is the error I get : <...
0
votes
1answer
23 views

Efficient Dataframe Transformation

So I'm trying to transform a pandas dataframe where I have multiple duplicates of a "user" value (my index being a user ID number), demographic info (which shouldn't change) and various purchase ...
0
votes
1answer
32 views

How do I show my pandas dataframe that I have in my jupyter notebook so that I can ask a question on stack exchange/stack overflow?

I've noticed that people produce such an amazing pandas dataframe which allows people to answer the questions. However, whenever I try to copy my table after doing ...
0
votes
1answer
31 views

Converting a pandas dataframe into a csv with multiple columns

When I convert my pandas dataframe into a csv it all puts it in one column, I would like it to be in multiple columns. this my code ...
1
vote
1answer
55 views

Automatically Detect Valid/Interesting pivot/drill downs of a dataset

Imagine you are given a tabular data set with a limited set of columns and rows and you are asked to find the valid/interesting pivot configuration by exploring the data. The brute-force option is to ...
0
votes
1answer
22 views

Extract seperated value from columns

I am working on this CSV file which is a collection of movie details from IMDB. In this I have a genres column in the dataframe with all the genres of the movies seperated by a pipe (|) What I need ...
0
votes
1answer
637 views

How to add a calculated column in a Pandas dataframe?

I am new to Python/Pandas so I'm struggling a bit here. I have a dataframe with air quality data from 2016 to 2020. I want to calculate the annual rate of change for each measured value to compare ...
0
votes
1answer
35 views

R: Rates of change from an initial value

I have a collection of csvs and must produce yearly rates of change per group within each csv, as well as a rate of change compared to the initial value. I am using the function below to calculate ...
0
votes
1answer
28 views

Better way of writing following Pandas Python code

Following code is given, we need to sum up the medals. ...
1
vote
2answers
47 views

Sort dataframe by date column stored as string

I have dataframe named df1. I want to sort data frame month column according to month (Jan, Feb, March..). For that I used code: ...
0
votes
1answer
66 views

Convert json to dataframe in python

I have a json array f below format ...
0
votes
2answers
460 views

Remove Outliers from Dataframe using pandas in Python

I would like to remove outliers from my dataset. It looks like this: ...
2
votes
1answer
21 views

Filling missing values for Embedded List in Python3

I searched for a similar question but I didn't come across. And I'm new in this area, I hope I explained my question well enough. I have a dataset consist of text data. I store them in a list and ...
0
votes
1answer
94 views

What is the efficient way to use apply method in column of pandas Dataframe for large dataset?

I have a dataset of approximate 1 hundred thousand records. I want to use apply method in each of the records for further data processing but it takes very long time to process (As apply method works ...
0
votes
1answer
263 views

find the difference between two columns in specific rows [closed]

I have the following columns from one data frame ...
0
votes
0answers
81 views

Dask error when reading data from a large zip file

I have a .zip folder with a lot of .csv files which I read into a Dask DataFrame like this: ...
0
votes
1answer
25 views

How to change random number of random elements in a column by group in R?

I have the following data frame: >df Name 1 Ed 2 Ed 3 Bill 4 Tom 5 Tom 6 Tom 7 Ed 8 Bill 9 Bill 10 Bill My goal is that from each group by "Name" ...
4
votes
2answers
124 views

Unformatted data entries

I have been working recently on an independent project using a database for Cybersecurity Attack classification. I imported the database using Pandas (Python) and before starting the processing step, ...
0
votes
0answers
27 views

conversion of any date to datetime in pandas dataframe

I want to define a function that takes in a pandas dateframe and iterates through its columns to find if there is any date field or a timestamp which can be an object. I want to convert those fields ...

1
2 3 4 5