Questions tagged [dataframe]

A data frame is a tabular data structure. Usually, it contains data where rows are observations and columns are variables of various types. While "data frame" or "dataframe" is the term used for this concept in several languages (R, Apache Spark, deedle, Maple, the pandas library in Python and the DataFrames library in Julia), "table" is the term used in MATLAB and SQL.

Filter by
Sorted by
Tagged with
4
votes
1answer
20k views

shifting the last column in the dataframe to the first place [closed]

While this question had relative answers, I couldn't find an efficient and robust way of doing it. I have to shift the last column in my data frame to the first without disturbing any of the other ...
1
vote
1answer
5k views

Adding a new custom column in a python data frame [closed]

I need to add a new column in a python data frame that has the dates of January in a column. Each date is repeated 24 times in the column without any intervention. The total number of entries will ...
1
vote
3answers
9k views

Reliable way to verify Pyspark data frame column type

If I read data from a CSV, all the columns will be of "String" type by default. Generally, I inspect the data using the following functions which gives an overview of the data and its types ...
1
vote
2answers
5k views

how to I replace numeric values with a string in an R dataframe?

I want to replace all numeric values in a column in my data frame with a string value. The following doesn't seem to work. ...
1
vote
0answers
133 views

Converting time series data to MNIST like data? [closed]

There are some time series data like this: ...
1
vote
1answer
440 views

Value extraction from a python dataframe [ problem statement specific ]

This is an update on one of the older posts of a friend of mine (That post stopped getting responses as I got a partial solution to my problem) and I'm reworking that problem statement. Link to that ...
1
vote
2answers
5k views

LOOKUP using 2 dataframes in Python

I have 2 dataframes: df1 Id CategoryId 1 A 1 B 2 A 2 E 2 F df2: ...
0
votes
1answer
7k views

Replacing null with average in pyspark

I have a problem during upsampling operation in PySpark. My dataframe is: ...
1
vote
1answer
2k views

Coding Problem - Extracting values from a column and forming a new dataframe [edited]

The problem statement requires extracting certain weather parameters for every hour in a particular date as denoted in the dataframe. The column 'hourly' consists of 24 lists in each entry, denoting ...
2
votes
1answer
11k views

Plotting in PySpark?

I have been searching for methods to plot in PySpark. I couldn't find any resource on plotting data residing in DataFrame in PySpark. The only methods which are listed are: through method ...
152
votes
1answer
195k views

Difference between isna() and isnull() in pandas

I have been using pandas for quite some time. But, I don't understand what's the difference between isna() and isnull(). And, ...
8
votes
6answers
205k views

I got the following error : 'DataFrame' object has no attribute 'data'

I am trying to get the 'data' and the 'target' of the iris setosa database, but I can't. For example, when I load the iris setosa directly from sklearn datasets I get a good result: Program: ...
9
votes
2answers
31k views

How to remove rows from a data frame that are identical to other df? [closed]

I have two data frames. So how do I remove rows that have identical email addresses in df2 from df1? ...
2
votes
1answer
319 views

how to improve searching index in dataframe

Given a pandas dataframe with a timestamp index, sorted. I have a label and I need to find the closest index to that label. Also, I need to find a smaller timestamp, so the search should be computed ...
1
vote
1answer
168 views

Is there any way to collect categorical features quickly in Julia DataFrames?

I'm using Julia 0.6.3 with Dataframes.jl I was wondering if there was any way to get categorial features easily in Julia? For large datasets it can be impossible to enter everything by hand. My ...
2
votes
0answers
13k views

Adding multiple columns to spark dataframe [closed]

I would like to add several columns to a spark (actually pyspark) dataframe , these columns all being functions of several input columns in the df. There seems to be no 'add_columns' in spark, and ...
1
vote
1answer
16k views

Remove Local Outliers from Dataframe using pandas

Could someone please suggest how to remove local outliers from the dataframe? I have the code to detect the local outliers, but I need help removing them(setting these values to zero) in the dataframe....
1
vote
2answers
163 views

Predicting Customer Activity Absence

Could you please assist me with to following question? I have a customer activity dataframe that looks like this: It contains at least 500.000 customers and a "timeseries" of 42 months. The ones and ...
4
votes
1answer
12k views

What’s the best way to save many pandas dataframes together?

I’m looking for a way to save house prices data by city, for example a pandas panel with one dataframe per city. But I need the dataframes to be independent, meaning that if one dataframe is corrupted,...
38
votes
6answers
254k views

How do I compare columns in different data frames?

I would like to compare one column of a df with other df's. The columns are names and last names. I'd like to check if a person in one data frame is in another one.
0
votes
1answer
4k views

Implement Sliding Window Algorithm to Detect Spikes

How do I implement sliding window algorithm with a window size of 10 and visualize the data iteratively to see spikes/possible outliers in the dataframe, using python? Please help a beginner.
3
votes
1answer
266 views

Look for previous date in dataframe that has certain column category in R

I have the following data frame: ...
3
votes
1answer
2k views

Handling Concat and Shift Feature in Pandas for Data Science [closed]

I am trying to USE Lag features AND Concat() and Shift() function, ...
0
votes
1answer
1k views

Pandas column count mismatch after insert into dataframe [closed]

I am trying to insert a column of values from one dataframe to another. The total count of values of both the indiviual dataframes is 91744 but when I use ...
1
vote
1answer
3k views

Segmenting pandas dataframe with lists as elements

I am working with a pandas DataFrame that contains a combination of numerical and categorical data on customer demographics and preferences. Each row contains information about an individual customer. ...
1
vote
1answer
602 views

Formatting categories of data with pandas in Python

I have around 9500 cities that I want to put in a pandas dataframe, then store in a file for later use. For example, below, these are 3 cities that I have data for. Some range in size based on year. ...
2
votes
2answers
40 views

Help with replacing values in a data frame

I'm a data science student at Mills and working on CoCalc. I am working with the 'wine quality' set that can be found on the UCI website for my final project. I'm just working with the white wine ...
0
votes
1answer
837 views

Counting how many rows contain chinese in mixed value dataset using python [closed]

I'm wanting to count how often Chinese is used, I've tried langdetect but the issue is the values are a mix of numbers, text and even URLs which langdetect doesn't seem to support ...
3
votes
1answer
75k views

Create new data frames from existing data frame based on unique column values

I have a large data set (4.5 million rows, 35 columns). The columns of interest are company_id (string) and company_score (float)...
2
votes
1answer
10k views

Replacing words by numbers in multiple columns of a data frame in R

I want to replace the values in a data set (sample in the picture) using numbers instead of words, e.g., 1 instead of D, ...
0
votes
1answer
26k views

Counting the occurrence of each string in a pandas dataframe column [closed]

I'm working with a data set of movies which has various info on them. One of the columns contains the various genres a movie may belong to like so: What I would like to do is count how often a genre ...
2
votes
1answer
33k views

Pandas apply return: Must have equal len keys and value when setting with an iterable

I have my code where i want to apply a function on and overwirte the inputs based on the return data, here is my apply section ...
2
votes
4answers
58k views

Using pandas, check a column for matching text and update new column if TRUE

My objective: Using pandas, check a column for matching text [not exact] and update new column if TRUE. From a csv file, a data frame was created and values of a particular column - COLUMN_to_Check, ...
0
votes
1answer
402 views

Python - Feature Selection - Should I remove bad variables?

I've this code to print the importance of each variable on my model: ...
-3
votes
2answers
7k views

drop_duplicates() doesn't work in pandas [closed]

I am working on my pandas tutorial. Below is my dataframe I am trying to drop duplicated header row using ...
2
votes
4answers
2k views

Using HashingVectorizer for text vectorization

Here is the sample data I have: Tag 1(Val: X), Tag 2(Val: Y), Tag 3(Val: Z), Label (Val: P) Tag 1(Val: A), Tag 2(Val: B), Tag 3(Val: C), Label (Val: Q) Tag 1(Val: D), Tag 2(Val: E), Tag 3(Val: F), ...
1
vote
1answer
25k views

How to load this data from .dat into dataframe using python

1 1:31080.410200 2:2.871828 3:5.862267 4:7.100850 5:8.283706 6:-5.427875 7:-6.667087 8:-8.888233 9:28898.943400 Can someone please tell me how to load this data ...
2
votes
2answers
1k views

how many rows have values from the same columns pandas

I have a df with many columns that represent the market cap of companies that compose an index. The index of the dataframe is dates. Before the company enters the index or after it leaves it, the ...
3
votes
4answers
5k views

Is there a package for using SQL to manipulate Pandas dataframes in Python?

Rather than learn a new package/language, I'd like to use my existing SQL skills to manipulate pandas dataframes in Python. Does anyone know of a way to do this, or ...
0
votes
1answer
1k views

Comparing dataframe object with string value in django

I'm implementing machine learning model and using training dataset from MySQL table and all this is built upon Django. So basically all the calculations are done by converting entire data from MySQL ...
13
votes
3answers
54k views

after grouping to minimum value in pandas, how to display the matching row result entirely along min() value

The dataframe contains ...
14
votes
2answers
103k views

How to plot two columns of single DataFrame on Y axis

I have two data frames (Action, Comedy). Action contains two columns (year, rating) ratings columns contains average rating with respect to year. The Comedy data frame contains the same two columns ...
0
votes
3answers
56 views

Combining results via arithmetic mean

For combining the results of different measurements, I want to calculate their arithmetic mean. Imagine the following dataset: ...
14
votes
4answers
21k views

One hot encoding alternatives for large categorical values

I have a data frame with large categorical values over 1600 categories. Is there any way I can find alternatives so that I don't have over 1600 columns? I found this interesting link. But they are ...
4
votes
1answer
3k views

Pandas dataframe resample aggregation by mills too slow

Given this test data: ...
0
votes
1answer
16k views

Converting Json file to Dataframe Python

I have a json file which has multiple events, each event starts with EventVersion Key. The data looks similar to the following synthesized data. ...
2
votes
3answers
35k views

Convert List to DataFrame [closed]

Dictonary: ...
2
votes
5answers
15k views

How duplicated items can be deleted from dataframe in pandas

I have the following data: userid itemid 1 1 1 1 1 3 1 4 2 1 2 2 2 3 How can I drop duplicated items (not ...
11
votes
3answers
17k views

Find the consecutive zeros in a DataFrame and do a conditional replacement

I have a dataset like this: Sample Dataframe ...
2
votes
1answer
2k views

Concatenate dataframes Pandas

I have three dataframes. Their shapes are (2656, 246), (2656, 2412) and (2656, 7025). I want to merge dataframes as above: So It will result a (2656, 9683) Dataframe. Thanks for any help. Typo on ...