Questions tagged [dataframe]

A data frame is a tabular data structure. Usually, it contains data where rows are observations and columns are variables of various types. While "data frame" or "dataframe" is the term used for this concept in several languages (R, Apache Spark, deedle, Maple, the pandas library in Python and the DataFrames library in Julia), "table" is the term used in MATLAB and SQL.

Filter by
Sorted by
Tagged with
152
votes
1answer
195k views

Difference between isna() and isnull() in pandas

I have been using pandas for quite some time. But, I don't understand what's the difference between isna() and isnull(). And, ...
38
votes
6answers
254k views

How do I compare columns in different data frames?

I would like to compare one column of a df with other df's. The columns are names and last names. I'd like to check if a person in one data frame is in another one.
25
votes
3answers
87k views

How to sum values grouped by two columns in pandas

I have a Pandas DataFrame like this: ...
14
votes
2answers
103k views

How to plot two columns of single DataFrame on Y axis

I have two data frames (Action, Comedy). Action contains two columns (year, rating) ratings columns contains average rating with respect to year. The Comedy data frame contains the same two columns ...
14
votes
4answers
21k views

One hot encoding alternatives for large categorical values

I have a data frame with large categorical values over 1600 categories. Is there any way I can find alternatives so that I don't have over 1600 columns? I found this interesting link. But they are ...
13
votes
3answers
54k views

after grouping to minimum value in pandas, how to display the matching row result entirely along min() value

The dataframe contains ...
11
votes
3answers
17k views

Find the consecutive zeros in a DataFrame and do a conditional replacement

I have a dataset like this: Sample Dataframe ...
10
votes
2answers
86k views

Mapping column values of one DataFrame to another DataFrame using a key with different header names

I have two data frames df1 and df2 which look something like this. ...
9
votes
2answers
30k views

How to rename columns that have the same name? [closed]

I would like to rename the column names, but the Data Frame contains similar column names. How do I rename them? df.columns Output: ...
9
votes
2answers
31k views

How to remove rows from a data frame that are identical to other df? [closed]

I have two data frames. So how do I remove rows that have identical email addresses in df2 from df1? ...
8
votes
2answers
29k views

Pandas merge column duplicate and sum value [closed]

How to merge duplicate column and sum their value? What I have A 30 A 40 B 50 What I need A 70 B 50 DF for this ...
8
votes
4answers
41k views

How to Write Multiple Data Frames in an Excel Sheet

I have multiple data frames with same column names. I want to write them together to an excel sheet stacked vertically on top of each other. And between each, there will be a text occupying a row. ...
8
votes
6answers
205k views

I got the following error : 'DataFrame' object has no attribute 'data'

I am trying to get the 'data' and the 'target' of the iris setosa database, but I can't. For example, when I load the iris setosa directly from sklearn datasets I get a good result: Program: ...
7
votes
2answers
101k views

ValueError: could not convert string to float: '���'

I have a (2M, 23) dimensional numpy array X. It has a dtype of <U26, i.e. unicode string ...
6
votes
1answer
13k views

How to find the count of consecutive same string values in a pandas dataframe? [closed]

Assume that we have the following pandas dataframe: ...
5
votes
1answer
4k views

How to transform raw data to fixed-frequency time series?

How to transform raw data to fixed-frequency time series? For example I have the following raw data in DataFrame ...
4
votes
2answers
148 views

Unformatted data entries

I have been working recently on an independent project using a database for Cybersecurity Attack classification. I imported the database using Pandas (Python) and before starting the processing step, ...
4
votes
1answer
20k views

shifting the last column in the dataframe to the first place [closed]

While this question had relative answers, I couldn't find an efficient and robust way of doing it. I have to shift the last column in my data frame to the first without disturbing any of the other ...
4
votes
2answers
5k views

Export pandas dataframe to a nested dictionary from multiple columns

Its a similar question to Export pandas to dictionary by combining multiple row values But in this case I want something different. ...
4
votes
1answer
5k views

Python & Pandas : TypeError: to_sql() got an unexpected keyword argument 'flavor'

I want to store JSON Data into MySQL Database using Python. I used dataframe of pandas. I found that ...
4
votes
1answer
167 views

Pandas dataframe with multiple hierarchical indices

I have a data frame which looks like this ...
4
votes
1answer
12k views

What’s the best way to save many pandas dataframes together?

I’m looking for a way to save house prices data by city, for example a pandas panel with one dataframe per city. But I need the dataframes to be independent, meaning that if one dataframe is corrupted,...
4
votes
1answer
3k views

Pandas dataframe resample aggregation by mills too slow

Given this test data: ...
4
votes
1answer
3k views

How to properly store a list of pandas dataframes, so that I can import them in a new python script?

I'm trying to make a GUI in PyQt5 to present the results of my research. The results are around 800 pandas dataframes, stored in lists (so that I can call them individually according to the list index)...
3
votes
1answer
22k views

dataframe.columns.difference() use

I am trying to find the working of dataframe.columns.difference() but couldn't find a satisfactory explanation about it. Can anyone explain the working of this ...
3
votes
4answers
5k views

Is there a package for using SQL to manipulate Pandas dataframes in Python?

Rather than learn a new package/language, I'd like to use my existing SQL skills to manipulate pandas dataframes in Python. Does anyone know of a way to do this, or ...
3
votes
4answers
277 views

Clustering algorithm which does not require to tell the number of clusters

I have a dataframe with 2 columns of numerical values. I want to apply a clustering algorithm to put all the entries into the same group, which have a relatively small distance to the other entries. ...
3
votes
1answer
2k views

How can I combine two single-column datasets into a single Pandas data frame?

I'd like to import the Rotten Tomatoes Movie Review dataset into a single data frame. How can I combine two datasets that are 1-column strings into a text, label ...
3
votes
1answer
3k views

PicklingError: Could not serialize object: TypeError: can't pickle fasttext_pybind.fasttext objects

I built a fasttext classification model in order to do sentiment analysis for facebook comments (using pyspark 2.4.1 on windows). When I use the prediction model function to predict the class of a ...
3
votes
2answers
712 views

Inputting (a lot of )data into a dataframe one row at a time

I'm using python. Some 2D numpy arrays are stored in individual rows of a Series. They are 30x30 images. It looks something like this: ...
3
votes
1answer
2k views

Pandas DataFrame: Aggregating multi-level groups by matching keys

I have some data that looks like this; ...
3
votes
1answer
75k views

Create new data frames from existing data frame based on unique column values

I have a large data set (4.5 million rows, 35 columns). The columns of interest are company_id (string) and company_score (float)...
3
votes
2answers
66 views

Optimization of pandas row iteration and summation

i'm wondering if anyone can provide some input on improving the speed and calculations of a pandas result. What i am trying to obtain is a summation of IDs in one table (player table) based on each ...
3
votes
1answer
181 views

Processing csv file with more than 700K rows of data

I have a .csv file (around 400MB in size) which contains 700K rows of structured data. Table structure is: ...
3
votes
1answer
2k views

Handling Concat and Shift Feature in Pandas for Data Science [closed]

I am trying to USE Lag features AND Concat() and Shift() function, ...
3
votes
1answer
1k views

How to shuffle only a fraction of a column in a Pandas dataframe?

I would like to shuffle a fraction (for example 40%) of the values of a specific column in a Pandas dataframe. How would you do it? Is there a simple idiomatic way to do that, maybe using ...
3
votes
1answer
736 views

While using reindex method on any dataframe why do original values go missing?

This is the original Dataframe: What I wanted : I wanted to convert this above data-frame into this multi-indexed column data-frame : I managed to do it by this piece of code : ...
3
votes
1answer
266 views

Look for previous date in dataframe that has certain column category in R

I have the following data frame: ...
2
votes
5answers
15k views

How duplicated items can be deleted from dataframe in pandas

I have the following data: userid itemid 1 1 1 1 1 3 1 4 2 1 2 2 2 3 How can I drop duplicated items (not ...
2
votes
4answers
58k views

Using pandas, check a column for matching text and update new column if TRUE

My objective: Using pandas, check a column for matching text [not exact] and update new column if TRUE. From a csv file, a data frame was created and values of a particular column - COLUMN_to_Check, ...
2
votes
1answer
32k views

Pandas apply return: Must have equal len keys and value when setting with an iterable

I have my code where i want to apply a function on and overwirte the inputs based on the return data, here is my apply section ...
2
votes
2answers
40 views

Help with replacing values in a data frame

I'm a data science student at Mills and working on CoCalc. I am working with the 'wine quality' set that can be found on the UCI website for my final project. I'm just working with the white wine ...
2
votes
1answer
25 views

Filling missing values for Embedded List in Python3

I searched for a similar question but I didn't come across. And I'm new in this area, I hope I explained my question well enough. I have a dataset consist of text data. I store them in a list and ...
2
votes
1answer
77k views

ValueError: The truth value of a DataFrame is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all()

Firstly I have a pandas series of recommended product (recmd_prdt_list). In this series there is a possibility of presence of deleted products. So as to remove deleted products from the recommended ...
2
votes
1answer
2k views

Concatenate dataframes Pandas

I have three dataframes. Their shapes are (2656, 246), (2656, 2412) and (2656, 7025). I want to merge dataframes as above: So It will result a (2656, 9683) Dataframe. Thanks for any help. Typo on ...
2
votes
1answer
7k views

Tokenization of data in dataframe in python

I am performing tokenization to each row in my dataframe but the tokenization is being done for only the first row. Can someone please help me. thank you. Below are my codes: ...
2
votes
1answer
762 views

Change values of a particular column to value_count()

Suppose i have the following data set. I need to replace the value of a particular column by the value_count(). I saw few posts where it is done for the entire data set. I need to do it for a ...
2
votes
2answers
149 views

Pandas: Assign back to table from grouping by column and index

I am trying to implement Exponential Moving Average calculation on a DataFrame. The formula is An additional complication is that my table is grouped and there is ...
2
votes
1answer
11k views

Plotting in PySpark?

I have been searching for methods to plot in PySpark. I couldn't find any resource on plotting data residing in DataFrame in PySpark. The only methods which are listed are: through method ...
2
votes
2answers
1k views

how many rows have values from the same columns pandas

I have a df with many columns that represent the market cap of companies that compose an index. The index of the dataframe is dates. Before the company enters the index or after it leaves it, the ...

1
2 3 4 5 6