Questions tagged [dataframe]

A data frame is a tabular data structure. Usually, it contains data where rows are observations and columns are variables of various types. While "data frame" or "dataframe" is the term used for this concept in several languages (R, Apache Spark, deedle, Maple, the pandas library in Python and the DataFrames library in Julia), "table" is the term used in MATLAB and SQL.

Filter by
Sorted by
Tagged with
0 votes
0 answers
19 views

how to identify fraudulent behavior in water consumption data

how to identify fraudulent behavior or anomalys in a dataset with water consumption information measured in 12 months? let's assume i have the following dataframe with multiple lines: ...
0 votes
0 answers
23 views

Cumsum starting value not 0 in Pandas

I have the following dataframe df of athletes (indexed by Athlete_ID) and their Rank (indexed by ...
0 votes
1 answer
28 views

Set column names to column value in pandas forloop, add prefix to column names

I have a dataframe with repeat column names, and I am trying to add a prefix label to column names with an appropriate identifier that comes from a column within the dataframe. I have this data frame: ...
0 votes
0 answers
19 views

How to correctly create a PyTorch Tensor from a Pandas DataFrame?

I have loaded my data into a Pandas DataFrame, and performed some pre-processing, and then I need to convert it into a PyTorch Tensor for training as my features data. Obviously, This new tensor do ...
  • 101
0 votes
2 answers
34 views

How to Insert a Row before Dataframe in Pandas

I have a csv file that is used as a pandas dataframe, now I only need to insert a dummy row before the dataframe starts like in the screenshot denoted as "Label". How can I do that?
0 votes
1 answer
96 views

How to upload a Pandas DataFrame into a Postgres table in Airflow?

I need to create a task in Airflow which downloads data from one Postgres database and uploads it to another. My code looks like this: ...
1 vote
1 answer
41 views

do inner functions have a substantial impact when used in series.apply() in Pandas

In pandas, if I use series.apply() to apply a function with an inner function definition, for example: ...
  • 13
0 votes
1 answer
43 views

How to plot categorical variables with a pie chart

I am concerned with a single column (fruit) from my df: ...
1 vote
2 answers
54 views

nested xml to dataframe

I am trying to convert the below mentioned sample xml file to a pandas dataframe. I have multiple xml files which I will loop over to add all xml data into a single dataframe once i succeed with this ...
  • 11
1 vote
3 answers
34 views

Pandas Dataframe grouping and summarizing

I have the following pandas dataframe: df_1: ...
0 votes
1 answer
40 views

Str.contains and isin function do not return all correct rows of dataframe [closed]

Given a list of strings L1 L1 = ['a', 'b', 'c'], I need to extract the rows which contain the values given in list L1. I used the isin function: ...
  • 23
0 votes
0 answers
27 views

what is difference between greatest and max function in spark data frame?

What is the exact difference between greatest and max functions in Spark data frame ,Since it returns the max value in the specified range
-1 votes
1 answer
11 views

Removing rows with total number of ID occurred not equal to a specific number in Pandas Python [closed]

I have the following data frame in Pandas: ...
  • 101
0 votes
1 answer
185 views

How to fix ValueError: The truth value of a DataFrame is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all()

I am using pytest to test a function that reads a csv into a dictionary of lists. In the test function it writes a sample dictionary to a csv then calls the function to read it back out. To accomplish ...
0 votes
0 answers
13 views

Adding rows with changing variable values in R

I have the following extract of my dataset about the occupancy of a football match: ...
0 votes
1 answer
12 views

Conditional multiplicaiton of two dataFrames [closed]

I have a first dataFrame which looks like this ...
  • 101
1 vote
2 answers
87 views

python compare 2 dataframes [closed]

in reference to this post, python-pandas-compare-columns-in-separate-dataframes-then-delete-non-matching, I can't get the isin function to work with my 2 example dataframes : df 1 : ...
0 votes
1 answer
77 views

Seaborn Heatmap with month & hour of database entry

I have a dataframe with X rows. For each row, I have the information of the month (value from 1 to 12) and hour (value from 1 to 24) in separate columns I need to create a heatmap with seaborn in ...
  • 103
0 votes
0 answers
8 views

Analysis of Variance of different groups

Im working with the following dataset: ...
0 votes
1 answer
190 views

Turning multiple binary columns into categorical (with less columns) with Python Pandas

I want to turn these categories into values of categorical columns. The values in each category are the current binary columns present in the data frame. We have : A11, A12.. is a detail of A1 so if ...
  • 21
2 votes
1 answer
187 views

Dataframe Python - Conditional Column based on multiple criteria

I want someone to correct my code in python. My goal is to create a code that will add a new column that is based on the conditional function of two columns in my dataframe. I want to add a fees ...
0 votes
0 answers
6 views

ML methods for vector correlation

I am dealing with a timeseries consisting of input flow sampled every 5 minutes over 441 days. My aim is to find any possible correlation from data coming from: The same day of the week The same ...
0 votes
2 answers
138 views

Create new rows based on a value in a column

My dateset is generated like the example df = {'event':['A','B','C','D'], 'budget':['123','433','1000','1299'], 'duration_days':['6','3','4','2']} I need ...
0 votes
0 answers
57 views

How can i merge two datasets with similar words in python?

For instance i have a row value on the dataset_1 : "Entity" = Apple dataset_2 : "Entity" = iCloud Apple (Entity is the column) I need to merge one dataset to the other by the ...
0 votes
1 answer
331 views

Plot multiple time series from single dataframe

I have a dataframe with multiple time series and columns with labels. My goal is to plot all time series in a single plot, where the labels should be used in the legend of the plot. The important ...
0 votes
0 answers
13 views

Identify the existence of a wall like structure in a given int array

The idea is that I will query an API endpoint which will return me an array consisting of a price value and a quantity value [price, quantity]. In this dataset ...
  • 1
0 votes
1 answer
822 views

For each unique value in a column, count respective unique values in another column

I have a set of tabular data (e.g. csv) representing accesses to a server through a specific protocol . The data follows this format: ...
0 votes
1 answer
64 views

How to set a new numeric index

I have read data from a json file, and the dataframe looks like: Col1 Col2 Col3 A: 1 Hi 7 B: 22 CC 1 I think it is because the file was a ...
0 votes
0 answers
24 views

Pandas Dataframe Filtering based on rows

I have a dataset like this: ...
  • 1
1 vote
0 answers
32 views

How to solve this IndexError? [closed]

I have created a training dataframe Traindata as following: dataFile='/content/drive/Colab Notebooks/.../Normal_Anomalous_8Digits.csv' data8=pd.read_csv(dataFile) ...
  • 21
0 votes
1 answer
12 views

Subsetting dataframe by a subdataframe vs a series

I'm learning data science in datacamp and a doubt came to my mind: if we have a dataframe cars structured like ...
1 vote
1 answer
458 views

Topic Modelling in an existing dataframe in python

I am trying to perform topic extraction in a panda dataframe. I am using LDA topic modeling in order to extract the topics in my dataframe. No problem. But, I would like to apply LDA topic modeling ...
0 votes
1 answer
487 views

Using Pandas.groupby.agg with multiple columns and functions

I have a data frame which contains duplicates I'd like to combine based on 1 column (name). In half of the other columns I'd like to keep one value (as they should all be the same) whereas I'd like to ...
  • 3
0 votes
0 answers
47 views

Group a spark dataframe by a starting event to an ending event

Given a series of events (with datetime) such as: failed, failed, passed, failed, passed, passed I want to retrieve the time from when it first "failed" to when it first "passed," ...
  • 101
0 votes
0 answers
937 views

pandas fill NA based on merge with another dataframe

I already posted this here but since there is no response, I thought I will also post this here ...
  • 2,337
0 votes
1 answer
48 views

Python - comparing 2 data frames [closed]

I have a large dataset, for an idea please see below. Each row has a sender, receiver, and message. There can be multiple columns for messages. It has indexes. sdr rcr msg A B msg_a A B msg_b A B ...
  • 13
0 votes
1 answer
45 views

Storage of N-dimensional matrices (tensors) as part of machine learning pipelines

I'm an infra person working on a storage product. I've been googling quite a bit to find an answer to the following question but unable to do so. Hence, I am attemping to ask the question here. I am ...
  • 101
1 vote
1 answer
62 views

PowerTransformer Producing Unexpected Result for Just One Column

I'm doing some preprocessing on my training data before fitting it to a model. Upon checking the results, there is one column that is returning 0 rather than 1 for the standard deviation. (all columns ...
  • 11
0 votes
0 answers
77 views

atoti session.visualize() not displaying dashboard on jupyter lab

Dears I am starting with 'atoti' here and I can't see the dashboard displaying the result. Here are the steps I am using: Created the conda env exclusively for it. Using Python3 engine. Using ...
0 votes
1 answer
23 views

How can I define such a function that can name for same values?

I have such a dataframe, I want to group them both when they are consecutive and when they are the same. In fact, I would like to see every intervals in different names so that I can print them and ...
1 vote
1 answer
142 views

Best way to optimize dataframe row by row sum of squared errors calculation?

I am seeing a significant slowdown with the following small snippet of code which computes the sum of squared errors between two dataframes - e.g. it takes approximately 2.5 seconds to run when ...
  • 13
0 votes
0 answers
335 views

Separate discrete and continuous variables

I know how to separate numerical and categorical data as follows: ...
  • 1,293
1 vote
3 answers
4k views

How to compare two columns of two dataframes and indicate the changes?

I have two dataframes, one is current week's information, one is of last week. I want to create a new dataset that lists all the changes during the week. Please see the following example: if there is ...
0 votes
1 answer
556 views

Extracting the metadata form Json file making it columns

I have the following json data file which I have converted to pandas dataframe. The columns are as follows ...
  • 105
0 votes
1 answer
45 views

Print histogram for each of the columns in my table with one single command

I would like to draw a histogram for each of the columns in my data.frame without having to write the the names of all of them, similar to what I did for inspect their unique values with: ...
0 votes
0 answers
225 views

Split dataframe to a train and test sets with a cross validation of x%

I am working on a dataframe and need to split it into a training set and test set, with 90% for Cross-Validation training, and 10% for a final test set. The problem is that I do not know where to ...
0 votes
2 answers
2k views

How to remove rows from a data frame that have special character (any character except alphabet and numbers)

How to remove rows from a data frame that have special character (any character except alphabet and numbers) I have some unwanted labels which seems to be useless but I want to remove them, this are ...
  • 1,636
0 votes
1 answer
489 views

How to find median/average values between data frames with slightly different columns?

I am trying to combat run-to-run variance of the data I collect by combining the data from different runs and finding the mean/average. The problem is that in each run there is a chance that some of ...
1 vote
1 answer
90 views

I want to do sub groups in pandas data frame [closed]

I have a dataframe as below I want to do first group by trip_number and then subgroup by mode. ...
0 votes
1 answer
13 views

Dataframe group certain rows

I am wondering if there is a way or I just need to loop through my dataframe to group rows where my index is greater than, in other words based in the following dataframe: ...
  • 155

1
2 3 4 5
7