Questions tagged [dataframe]

A data frame is a tabular data structure. Usually, it contains data where rows are observations and columns are variables of various types. While "data frame" or "dataframe" is the term used for this concept in several languages (R, Apache Spark, deedle, Maple, the pandas library in Python and the DataFrames library in Julia), "table" is the term used in MATLAB and SQL.

Filter by
Sorted by
Tagged with
126
votes
1answer
175k views

Difference between isna() and isnull() in pandas

I have been using pandas for quite some time. But, I don't understand what's the difference between isna() and isnull(). And, ...
35
votes
5answers
214k views

How do I compare columns in different data frames?

I would like to compare one column of a df with other df's. The columns are names and last names. I'd like to check if a person in one data frame is in another one.
24
votes
3answers
81k views

How to sum values grouped by two columns in pandas

I have a Pandas DataFrame like this: ...
12
votes
1answer
89k views

How to plot two columns of single DataFrame on Y axis

i have two DataFrames(Action,Comedy). Action contains two columns(year,rating) ratings columns contains average rating with respect to year. Comedy Dataframe contains same two columns with different ...
11
votes
3answers
14k views

Find the consecutive zeros in a DataFrame and do a conditional replacement

I have a dataset like this: Sample Dataframe ...
10
votes
3answers
46k views

after grouping to minimum value in pandas, how to display the matching row result entirely along min() value

The dataframe contains ...
8
votes
2answers
62k views

Mapping column values of one DataFrame to another DataFrame using a key with different header names

I have two data frames df1 and df2 which look something like this. ...
7
votes
2answers
18k views

Pandas merge column duplicate and sum value [closed]

How to merge duplicate column and sum their value? What I have A 30 A 40 B 50 What I need A 70 B 50 DF for this ...
7
votes
2answers
72k views

ValueError: could not convert string to float: '���'

I have a (2M, 23) dimensional numpy array X. It has a dtype of <U26, i.e. unicode string ...
6
votes
2answers
20k views

How to rename columns that have the same name? [closed]

I would like to rename the column names, but the Data Frame contains similar column names. How do I rename them? df.columns Output: ...
6
votes
6answers
149k views

I got the following error : 'DataFrame' object has no attribute 'data'

I am trying to get the 'data' and the 'target' of the iris setosa database, but I can't. For example, when I load the iris setosa directly from sklearn datasets I get a good result: Program: ...
5
votes
4answers
28k views

How to Write Multiple Data Frames in an Excel Sheet

I have multiple data frames with same column names. I want to write them together to an excel sheet stacked vertically on top of each other. And between each, there will be a text occupying a row. ...
5
votes
2answers
19k views

How to remove rows from a data frame that are identical to other df? [closed]

I have two data frames. So how do I remove rows that have identical email addresses in df2 from df1? ...
5
votes
1answer
3k views

How to transform raw data to fixed-frequency time series?

How to transform raw data to fixed-frequency time series? For example I have the following raw data in DataFrame ...
4
votes
2answers
124 views

Unformatted data entries

I have been working recently on an independent project using a database for Cybersecurity Attack classification. I imported the database using Pandas (Python) and before starting the processing step, ...
4
votes
1answer
10k views

How to find the count of consecutive same string values in a pandas dataframe? [closed]

Assume that we have the following pandas dataframe: ...
4
votes
1answer
4k views

Python & Pandas : TypeError: to_sql() got an unexpected keyword argument 'flavor'

I want to store JSON Data into MySQL Database using Python. I used dataframe of pandas. I found that ...
4
votes
1answer
2k views

Pandas dataframe resample aggregation by mills too slow

Given this test data: ...
3
votes
4answers
275 views

Clustering algorithm which does not require to tell the number of clusters

I have a dataframe with 2 columns of numerical values. I want to apply a clustering algorithm to put all the entries into the same group, which have a relatively small distance to the other entries. ...
3
votes
4answers
4k views

Is there a package for using SQL to manipulate Pandas dataframes in Python?

Rather than learn a new package/language, I'd like to use my existing SQL skills to manipulate pandas dataframes in Python. Does anyone know of a way to do this, or ...
3
votes
1answer
15k views

shifting the last column in the dataframe to the first place [closed]

While this question had relative answers, I couldn't find an efficient and robust way of doing it. I have to shift the last column in my data frame to the first without disturbing any of the other ...
3
votes
1answer
2k views

How can I combine two single-column datasets into a single Pandas data frame?

I'd like to import the Rotten Tomatoes Movie Review dataset into a single data frame. How can I combine two datasets that are 1-column strings into a text, label ...
3
votes
2answers
2k views

Export pandas dataframe to a nested dictionary from multiple columns

Its a similar question to Export pandas to dictionary by combining multiple row values But in this case I want something different. ...
3
votes
1answer
130 views

Pandas dataframe with multiple hierarchical indices

I have a data frame which looks like this ...
3
votes
2answers
408 views

Inputting (a lot of )data into a dataframe one row at a time

I'm using python. Some 2D numpy arrays are stored in individual rows of a Series. They are 30x30 images. It looks something like this: ...
3
votes
1answer
1k views

Pandas DataFrame: Aggregating multi-level groups by matching keys

I have some data that looks like this; ...
3
votes
1answer
10k views

What’s the best way to save many pandas dataframes together?

I’m looking for a way to save house prices data by city, for example a pandas panel with one dataframe per city. But I need the dataframes to be independent, meaning that if one dataframe is corrupted,...
3
votes
1answer
71k views

Create new data frames from existing data frame based on unique column values

I have a large data set (4.5 million rows, 35 columns). The columns of interest are company_id (string) and company_score (float)...
3
votes
1answer
862 views

How to properly store a list of pandas dataframes, so that I can import them in a new python script?

I'm trying to make a GUI in PyQt5 to present the results of my research. The results are around 800 pandas dataframes, stored in lists (so that I can call them individually according to the list index)...
3
votes
2answers
47 views

Optimization of pandas row iteration and summation

i'm wondering if anyone can provide some input on improving the speed and calculations of a pandas result. What i am trying to obtain is a summation of IDs in one table (player table) based on each ...
3
votes
1answer
132 views

Processing csv file with more than 700K rows of data

I have a .csv file (around 400MB in size) which contains 700K rows of structured data. Table structure is: ...
3
votes
1answer
1k views

Handling Concat and Shift Feature in Pandas for Data Science [closed]

I am trying to USE Lag features AND Concat() and Shift() function, ...
3
votes
1answer
940 views

How to shuffle only a fraction of a column in a Pandas dataframe?

I would like to shuffle a fraction (for example 40%) of the values of a specific column in a Pandas dataframe. How would you do it? Is there a simple idiomatic way to do that, maybe using ...
3
votes
1answer
316 views

While using reindex method on any dataframe why do original values go missing?

This is the original Dataframe: What I wanted : I wanted to convert this above data-frame into this multi-indexed column data-frame : I managed to do it by this piece of code : ...
3
votes
1answer
192 views

Look for previous date in dataframe that has certain column category in R

I have the following data frame: ...
2
votes
5answers
13k views

How duplicated items can be deleted from dataframe in pandas

I have the following data: userid itemid 1 1 1 1 1 3 1 4 2 1 2 2 2 3 How can I drop duplicated items (not ...
2
votes
2answers
40 views

Help with replacing values in a data frame

I'm a data science student at Mills and working on CoCalc. I am working with the 'wine quality' set that can be found on the UCI website for my final project. I'm just working with the white wine ...
2
votes
1answer
21 views

Filling missing values for Embedded List in Python3

I searched for a similar question but I didn't come across. And I'm new in this area, I hope I explained my question well enough. I have a dataset consist of text data. I store them in a list and ...
2
votes
1answer
1k views

Concatenate dataframes Pandas

I have three dataframes. Their shapes are (2656, 246), (2656, 2412) and (2656, 7025). I want to merge dataframes as above: So It will result a (2656, 9683) Dataframe. Thanks for any help. Typo on ...
2
votes
1answer
484 views

Change values of a particular column to value_count()

Suppose i have the following data set. I need to replace the value of a particular column by the value_count(). I saw few posts where it is done for the entire data set. I need to do it for a ...
2
votes
2answers
119 views

Pandas: Assign back to table from grouping by column and index

I am trying to implement Exponential Moving Average calculation on a DataFrame. The formula is An additional complication is that my table is grouped and there is ...
2
votes
1answer
9k views

Plotting in PySpark?

I have been searching for methods to plot in PySpark. I couldn't find any resource on plotting data residing in DataFrame in PySpark. The only methods which are listed are: through method ...
2
votes
2answers
1k views

how many rows have values from the same columns pandas

I have a df with many columns that represent the market cap of companies that compose an index. The index of the dataframe is dates. Before the company enters the index or after it leaves it, the ...
2
votes
1answer
2k views

Tokenization of data in dataframe in python

I am performing tokenization to each row in my dataframe but the tokenization is being done for only the first row. Can someone please help me. thank you. Below are my codes: ...
2
votes
1answer
286 views

Get elements from lists in pandas dataframe [closed]

I have the following column of a data frame: I get it by doing dataFrame['opcodes]. and I would like to consider only the first 20 and the last 20 elements of ...
2
votes
1answer
76 views

Python dataframe iterate over rows

Dataset ...
2
votes
2answers
1k views

Passing data to SMOTE after applying train/test split

I'm trying to resample my dataset after splitting it into train and test partitions using SMOTE. Here's my code: ...
2
votes
2answers
93 views

How to group this dataframe in python?

I have this problem: ...
2
votes
1answer
314 views

how to improve searching index in dataframe

Given a pandas dataframe with a timestamp index, sorted. I have a label and I need to find the closest index to that label. Also, I need to find a smaller timestamp, so the search should be computed ...
2
votes
4answers
2k views

Using HashingVectorizer for text vectorization

Here is the sample data I have: Tag 1(Val: X), Tag 2(Val: Y), Tag 3(Val: Z), Label (Val: P) Tag 1(Val: A), Tag 2(Val: B), Tag 3(Val: C), Label (Val: Q) Tag 1(Val: D), Tag 2(Val: E), Tag 3(Val: F), ...

1
2 3 4 5