Questions tagged [dataframe]

A data frame is a tabular data structure. Usually, it contains data where rows are observations and columns are variables of various types. While "data frame" or "dataframe" is the term used for this concept in several languages (R, Apache Spark, deedle, Maple, the pandas library in Python and the DataFrames library in Julia), "table" is the term used in MATLAB and SQL.

Filter by
Sorted by
Tagged with
0 votes
1 answer
9 views

Plotly graph does not show x-axis values correctly

Consider the following code : ...
1 vote
2 answers
43 views

python compare 2 dataframes [closed]

in reference to this post, python-pandas-compare-columns-in-separate-dataframes-then-delete-non-matching, I can't get the isin function to work with my 2 example dataframes : df 1 : ...
3 votes
1 answer
429 views

Look for previous date in dataframe that has certain column category in R

I have the following data frame: ...
1 vote
1 answer
391 views

Replace Missing Values with Most Frequent number under Condition

I'm trying to replace missing values of column "Age" but under condition of other columns on this data Titanic - Machine Learning from Disaster ...
0 votes
0 answers
31 views

I'm iterating a dataframe with a for loop, does anybody know a faster way of doing this? [migrated]

I'm working on a dataframe where I need to add a new column which is based on other existing rows in the dataframe. Here is a simplified version of what I'm trying to do. Basically there is a ...
0 votes
1 answer
59 views

How to do encode this target vector containing strings

Consider a target vector like ...
1 vote
0 answers
65 views

How to apply Kalman Filter for Cleaning Timeseries Data effectively without much optimization?

Someone gave me a tip to use kalman filter for my dataset. How time intensive is it to get a good kalman filter running, compared to simple interpolation methods like ...
1 vote
3 answers
2k views

How to compare two columns of two dataframes and indicate the changes?

I have two dataframes, one is current week's information, one is of last week. I want to create a new dataset that lists all the changes during the week. Please see the following example: if there is ...
0 votes
1 answer
20 views

Seaborn Heatmap with month & hour of database entry

I have a dataframe with X rows. For each row, I have the information of the month (value from 1 to 12) and hour (value from 1 to 24) in separate columns I need to create a heatmap with seaborn in ...
1 vote
1 answer
199 views

Is there any way to collect categorical features quickly in Julia DataFrames?

I'm using Julia 0.6.3 with Dataframes.jl I was wondering if there was any way to get categorial features easily in Julia? For large datasets it can be impossible to enter everything by hand. My ...
1 vote
2 answers
981 views

How to create a df that gets sum of columns based on a groupby column?

...
0 votes
1 answer
51 views

Kernel dies or proses stuck when making LR prediction on dataframe using apply

I'm trying to making predictions with a simple model. model=LogisticRegression() model.fit(X_train,y_train) After fitting, i try to make predictions. A sample ...
1 vote
2 answers
210 views

Predicting Customer Activity Absence

Could you please assist me with the following question? I have a customer activity data frame that looks like this: It contains at least 500.000 customers and a "time series" of 42 months. ...
0 votes
0 answers
15 views

How to find location for stretch of values more than fixed number and their average in dataframe python3 [migrated]

I have a data-frame something like this: And I wish to find the the location (start and end) and the average of the stretch of stretch where values in column 2 are more than 0.2. Something like this: ...
0 votes
0 answers
8 views

Analysis of Variance of different groups

Im working with the following dataset: ...
0 votes
0 answers
8 views

Turning multiple binary columns into categorical (with less columns) with Python Pandas

I want to turn these categories into values of categorical columns. The values in each category are the current binary columns present in the data frame. We have : A11, A12.. is a detail of A1 so if ...
3 votes
1 answer
1k views

While using reindex method on any dataframe why do original values go missing?

This is the original Dataframe: What I wanted : I wanted to convert this above data-frame into this multi-indexed column data-frame : I managed to do it by this piece of code : ...
0 votes
1 answer
137 views

RAM crashed for XML to DataFrame conversion function

I have created the following function which converts an XML File to a DataFrame. This function works good for files smaller than 1 GB, for anything greater than that the RAM(13GB Google Colab RAM) ...
1 vote
1 answer
829 views

How to match a word from column and compare with other column in pandas dataframe

I have the below dataframe ...
2 votes
1 answer
86 views

Dataframe Python - Conditional Column based on multiple criteria

I want someone to correct my code in python. My goal is to create a code that will add a new column that is based on the conditional function of two columns in my dataframe. I want to add a fees ...
7 votes
1 answer
16k views

How to find the count of consecutive same string values in a pandas dataframe? [closed]

Assume that we have the following pandas dataframe: ...
0 votes
0 answers
6 views

ML methods for vector correlation

I am dealing with a timeseries consisting of input flow sampled every 5 minutes over 441 days. My aim is to find any possible correlation from data coming from: The same day of the week The same ...
0 votes
2 answers
21k views

How do I read a dat file, for which I don't know its structure?

Is there any way to at least read the text from the dat file. I have its corresponding mdf file hence I know what all data and columns are there in it. How do I figure out the contents in my dat file. ...
0 votes
1 answer
116 views

Is there a way to make the window in df.rolling dynamic depending on which row it is calculating for?

I have a dataset of stock prices, and I want to add a column of 52 week lows for each day, however for the rows which dont have 365 days above them I just want the column to have the rolling min using ...
0 votes
1 answer
9k views

ValueError: ('The truth value of a Series is ambiguous after applying if/else condition in Pandas data frames

I want to create a new variable for the dataframe details called lower after iterating through multiple ...
1 vote
1 answer
477 views

Add ID information from one dataframe to every row in another dataframe without a common key

I am trying to combine two dataframes into the second dataframe, but duplicating the first dataframe into every row of dataframe two. The first dataframe (df1) contains the identifing information, ...
0 votes
2 answers
31 views

Create new rows based on a value in a column

My dateset is generated like the example df = {'event':['A','B','C','D'], 'budget':['123','433','1000','1299'], 'duration_days':['6','3','4','2']} I need ...
0 votes
1 answer
763 views

Replacing rows of dataframe with rows of another dataframe that have the same index

I have a dataframe that has rows with indices 0 to 128 and a smaller dataframe with indices 4, 8, 105, and 107. I made edits to the rows in the smaller dataframe and am now trying to replace rows ...
2 votes
4 answers
11k views

How to combine and separate test and train data for data cleaning?

I am working on an ML model in which I have been provided the data in 2 files test.csv and train.csv. I want to perform data ...
2 votes
1 answer
3k views

lookup and fill some value from one dataframe to another

I have 2 dataframes, df1,and df2 as below. df1 and df2 I would like to lookup "result" from df1 and fill into df2 by "Mode" as below format. Note "Mode" has become my ...
0 votes
0 answers
23 views

How can i merge two datasets with similar words in python?

For instance i have a row value on the dataset_1 : "Entity" = Apple dataset_2 : "Entity" = iCloud Apple (Entity is the column) I need to merge one dataset to the other by the ...
0 votes
1 answer
52 views

Plot multiple time series from single dataframe

I have a dataframe with multiple time series and columns with labels. My goal is to plot all time series in a single plot, where the labels should be used in the legend of the plot. The important ...
0 votes
0 answers
13 views

Identify the existence of a wall like structure in a given int array

The idea is that I will query an API endpoint which will return me an array consisting of a price value and a quantity value [price, quantity]. In this dataset ...
2 votes
1 answer
29 views

Should hexadecimal addresses of a dataset be cleaned?

I am working on fraud detection on blockchains. To be more specific, I fetched a big number of transactions that took place on the blockchain, labeled them to spam / non spam using an appropriate API ...
1 vote
1 answer
49 views

Plot three series on the same plot grouping data by day and month

I have a dataset containing three years of data which I would like to plot and compare by date and month; but, I am having a hard time with the final result. I am nearly there, but for some strange ...
0 votes
2 answers
5k views

How to add a calculated column in a Pandas dataframe?

I am new to Python/Pandas so I'm struggling a bit here. I have a dataframe with air quality data from 2016 to 2020. I want to calculate the annual rate of change for each measured value to compare ...
0 votes
1 answer
22 views

How to set a new numeric index

I have read data from a json file, and the dataframe looks like: Col1 Col2 Col3 A: 1 Hi 7 B: 22 CC 1 I think it is because the file was a ...
1 vote
2 answers
4k views

Update a pandas data frame column using Apply,Lambda and Group by Functions

I have a data frame in the format mentioned in the screenshot below. Column 'Candidate Won' has only 'loss' as the column value for all the rows. I want to update the Column 'Candidate Won' to a value ...
1 vote
3 answers
307 views

Efficiently modify a large csv file in Pandas

I have a csv file and would like to do the following modification on it: ...
0 votes
2 answers
389 views

Extract seperated value from columns

I am working on this CSV file which is a collection of movie details from IMDB. In this I have a genres column in the dataframe with all the genres of the movies seperated by a pipe (|) What I need ...
1 vote
1 answer
567 views

Merging 2 pandas dataframes on key with duplicates but do not want it to result in two lines

I am trying to merge a "roster" where there are supposed to be several positions with one person per office. However, I have come across two instances where there are two people in one ...
0 votes
1 answer
117 views

Time Series Plot for floating values

I have a Dataframe which looks as shown below I am trying to make a line plot for looking at the peaks for both columns (a,b), I have gotten as far as ...
0 votes
1 answer
208 views

For each unique value in a column, count respective unique values in another column

I have a set of tabular data (e.g. csv) representing accesses to a server through a specific protocol . The data follows this format: ...
0 votes
1 answer
2k views

Python: How to plot time interval from a Dataframe in Pandas

I have the a dataframe(df) which has the data of a Job being executed at different time intervals. It includes the following details about the execution of a job: ...
2 votes
2 answers
1k views

pandas data frame doesn't show any thing ,when view as data frame in pycharm

import pandas as pd; dataSet = pd.read_csv("winequality-red.csv"); dataSet.describe(include = 'all'); When view data set as data frame ,it show empty table.But ...
1 vote
1 answer
214 views

Why is the difference of datetime = zero in the following dataframe?

This issue that I am facing is very simple yet weird and has troubled me to no end. I have a dataframe as follows : ...
1 vote
1 answer
173 views

Topic Modelling in an existing dataframe in python

I am trying to perform topic extraction in a panda dataframe. I am using LDA topic modeling in order to extract the topics in my dataframe. No problem. But, I would like to apply LDA topic modeling ...
0 votes
2 answers
294 views

How to group by IDs and count the number of groups with occurrence of a variable after first point?

Language: Python 3.8 I have a dataframe that consists of a series of people (each appearing multiple times in the dataframe), dates, and binary variables. I am trying to figure out how many people ...
2 votes
1 answer
159 views

KeyError: Selecting text from a dataframe based on values of another dataframe

I have the following two dataframes badges and comments. I have created a list of 'gold users' from ...
0 votes
1 answer
1k views

Comparing dataframe object with string value in django

I'm implementing machine learning model and using training dataset from MySQL table and all this is built upon Django. So basically all the calculations are done by converting entire data from MySQL ...

1
2 3 4 5
7