Questions tagged [dataframe]

A data frame is a tabular data structure. Usually, it contains data where rows are observations and columns are variables of various types. While "data frame" or "dataframe" is the term used for this concept in several languages (R, Apache Spark, deedle, Maple, the pandas library in Python and the DataFrames library in Julia), "table" is the term used in MATLAB and SQL.

Filter by
Sorted by
Tagged with
0 votes
0 answers
8 views

How to access dataframe created or upadted inside if statement outside?

A dataframe is created/updated isnide an if loop which itself is within a for loop. The DF need to be accessed outside the if statement. But outside the scope it is not updating. ...
user avatar
  • 1
0 votes
0 answers
9 views

How to count unique for each key pandas

data ...
user avatar
0 votes
0 answers
15 views

How to correctly identify the location and date column from different datasets?

I have csv files from different sources. Each source has at least two common columns which contain the date and the location. There are few issues: The names of these common columns is different in ...
user avatar
  • 133
0 votes
0 answers
5 views

ML methods for vector correlation

I am dealing with a timeseries consisting of input flow sampled every 5 minutes over 441 days. My aim is to find any possible correlation from data coming from: The same day of the week The same ...
user avatar
-1 votes
0 answers
28 views

How to constrain a dataframe to specific date range?

I have a dataframe that I would like to chunck- or rather run temporal sentiment analysis at different times. I am trying to measure how sentiment changes as part of user identity in extremist social ...
user avatar
0 votes
2 answers
22 views

Create new rows based on a value in a column

My dateset is generated like the example df = {'event':['A','B','C','D'], 'budget':['123','433','1000','1299'], 'duration_days':['6','3','4','2']} I need ...
user avatar
0 votes
0 answers
17 views

How can i merge two datasets with similar words in python?

For instance i have a row value on the dataset_1 : "Entity" = Apple dataset_2 : "Entity" = iCloud Apple (Entity is the column) I need to merge one dataset to the other by the ...
user avatar
0 votes
1 answer
32 views

Plot multiple time series from single dataframe

I have a dataframe with multiple time series and columns with labels. My goal is to plot all time series in a single plot, where the labels should be used in the legend of the plot. The important ...
user avatar
0 votes
0 answers
13 views

Identify the existence of a wall like structure in a given int array

The idea is that I will query an API endpoint which will return me an array consisting of a price value and a quantity value [price, quantity]. In this dataset ...
user avatar
  • 1
0 votes
1 answer
25 views

For each unique value in a column, count respective unique values in another column

I have a set of tabular data (e.g. csv) representing accesses to a server through a specific protocol . The data follows this format: ...
user avatar
0 votes
1 answer
16 views

How to set a new numeric index

I have read data from a json file, and the dataframe looks like: Col1 Col2 Col3 A: 1 Hi 7 B: 22 CC 1 I think it is because the file was a ...
user avatar
0 votes
0 answers
23 views

Pandas Dataframe Filtering based on rows

I have a dataset like this: ...
user avatar
  • 1
1 vote
0 answers
29 views

How to solve this IndexError? [closed]

I have created a training dataframe Traindata as following: dataFile='/content/drive/Colab Notebooks/.../Normal_Anomalous_8Digits.csv' data8=pd.read_csv(dataFile) ...
user avatar
  • 21
0 votes
1 answer
11 views

Subsetting dataframe by a subdataframe vs a series

I'm learning data science in datacamp and a doubt came to my mind: if we have a dataframe cars structured like ...
user avatar
1 vote
1 answer
53 views

Topic Modelling in an existing dataframe in python

I am trying to perform topic extraction in a panda dataframe. I am using LDA topic modeling in order to extract the topics in my dataframe. No problem. But, I would like to apply LDA topic modeling ...
user avatar
0 votes
1 answer
43 views

Using Pandas.groupby.agg with multiple columns and functions

I have a data frame which contains duplicates I'd like to combine based on 1 column (name). In half of the other columns I'd like to keep one value (as they should all be the same) whereas I'd like to ...
user avatar
  • 3
0 votes
0 answers
15 views

Group a spark dataframe by a starting event to an ending event

Given a series of events (with datetime) such as: failed, failed, passed, failed, passed, passed I want to retrieve the time from when it first "failed" to when it first "passed," ...
user avatar
  • 101
0 votes
0 answers
83 views

pandas fill NA based on merge with another dataframe

I already posted this here but since there is no response, I thought I will also post this here ...
user avatar
  • 2,129
0 votes
1 answer
44 views

Python - comparing 2 data frames [closed]

I have a large dataset, for an idea please see below. Each row has a sender, receiver, and message. There can be multiple columns for messages. It has indexes. sdr rcr msg A B msg_a A B msg_b A B ...
user avatar
  • 13
0 votes
1 answer
15 views

Storage of N-dimensional matrices (tensors) as part of machine learning pipelines

I'm an infra person working on a storage product. I've been googling quite a bit to find an answer to the following question but unable to do so. Hence, I am attemping to ask the question here. I am ...
user avatar
  • 101
1 vote
1 answer
24 views

PowerTransformer Producing Unexpected Result for Just One Column

I'm doing some preprocessing on my training data before fitting it to a model. Upon checking the results, there is one column that is returning 0 rather than 1 for the standard deviation. (all columns ...
user avatar
  • 11
0 votes
0 answers
40 views

atoti session.visualize() not displaying dashboard on jupyter lab

Dears I am starting with 'atoti' here and I can't see the dashboard displaying the result. Here are the steps I am using: Created the conda env exclusively for it. Using Python3 engine. Using ...
user avatar
0 votes
1 answer
18 views

How can I define such a function that can name for same values?

I have such a dataframe, I want to group them both when they are consecutive and when they are the same. In fact, I would like to see every intervals in different names so that I can print them and ...
user avatar
1 vote
1 answer
38 views

Best way to optimize dataframe row by row sum of squared errors calculation?

I am seeing a significant slowdown with the following small snippet of code which computes the sum of squared errors between two dataframes - e.g. it takes approximately 2.5 seconds to run when ...
user avatar
  • 13
0 votes
0 answers
54 views

Separate discrete and continuous variables

I know how to separate numerical and categorical data as follows: ...
user avatar
  • 1,241
1 vote
3 answers
874 views

How to compare two columns of two dataframes and indicate the changes?

I have two dataframes, one is current week's information, one is of last week. I want to create a new dataset that lists all the changes during the week. Please see the following example: if there is ...
user avatar
0 votes
1 answer
148 views

Extracting the metadata form Json file making it columns

I have the following json data file which I have converted to pandas dataframe. The columns are as follows ...
user avatar
  • 105
0 votes
1 answer
16 views

Print histogram for each of the columns in my table with one single command

I would like to draw a histogram for each of the columns in my data.frame without having to write the the names of all of them, similar to what I did for inspect their unique values with: ...
user avatar
0 votes
0 answers
112 views

Split dataframe to a train and test sets with a cross validation of x%

I am working on a dataframe and need to split it into a training set and test set, with 90% for Cross-Validation training, and 10% for a final test set. The problem is that I do not know where to ...
user avatar
0 votes
2 answers
628 views

How to remove rows from a data frame that have special character (any character except alphabet and numbers)

How to remove rows from a data frame that have special character (any character except alphabet and numbers) I have some unwanted labels which seems to be useless but I want to remove them, this are ...
user avatar
  • 1,526
0 votes
1 answer
40 views

How to find median/average values between data frames with slightly different columns?

I am trying to combat run-to-run variance of the data I collect by combining the data from different runs and finding the mean/average. The problem is that in each run there is a chance that some of ...
user avatar
1 vote
1 answer
43 views

I want to do sub groups in pandas data frame [closed]

I have a dataframe as below I want to do first group by trip_number and then subgroup by mode. ...
user avatar
0 votes
1 answer
13 views

Dataframe group certain rows

I am wondering if there is a way or I just need to loop through my dataframe to group rows where my index is greater than, in other words based in the following dataframe: ...
user avatar
  • 155
0 votes
0 answers
86 views

Create rows filling the sequence of numbers in a 'period column' and the 'value column' with 0 in python/pandas

I need to fill a dataframe with rows completing the values that the system doesn't show on the report (e. g. a week that the product had no sales): The report: Product Week Total_Sales 0 A 1 82 1 A ...
user avatar
0 votes
0 answers
14 views

How can I combine the information from these two dataframes?

If one of my dataframes gives me some info about items: ...
user avatar
0 votes
0 answers
13 views

An efficient way to encode & embed tabular data of a video into a transformer?

So a little bit of a background: I have a folder which contains video files of lets say humans doing a certain action (i.e. walking) where each .2 seconds is documented in a ...
user avatar
  • 101
2 votes
1 answer
29 views

Should hexadecimal addresses of a dataset be cleaned?

I am working on fraud detection on blockchains. To be more specific, I fetched a big number of transactions that took place on the blockchain, labeled them to spam / non spam using an appropriate API ...
user avatar
1 vote
1 answer
69 views

Date time conversion in a CSV column [closed]

I am new to data science. I am attempting to write a program using regression techniques, and all of my values are numerical, except for the date and time (UTC), which are written in this format: HH:...
user avatar
0 votes
1 answer
15 views

Data captured outside Pandas DataFrame

I'm new to Python for starters, when I got this DataFrame from Yahoo Finance, the Date shows up outside the DataFrame. This is leading me to get KeyError: 'Date' when I try to write this line of code ...
user avatar
0 votes
2 answers
151 views

How do I remove float object in dataframe? Ex: "winter 2021" to "winter"

I have the premiered column where I want to remove the year and only keep the season. Example: "winter 2021" to become "winter" .
user avatar
  • 3
0 votes
0 answers
34 views

Scaling columns pandas DataFrame

For a dataset having different numeric columns, they usually have different range and distributions. As an example, I have used the Iris dataset. The distributions of it's 4 columns are shown: My ...
user avatar
  • 121
0 votes
1 answer
109 views

How to convert duration column from 1 hr. 17 min to 77 min in pandas? [closed]

I am trying to convert all the hr., min, and sec into just mins. For example, 1 hr. 17 min to 77 min, and 34 sec to 0.56 min, not 0.34 min. So I have used this code: ...
user avatar
  • 3
1 vote
1 answer
729 views

Pandas series: how to compute daily cumulative sum of a column [closed]

Consider a pandas dataframe where each index is represented as a datetime object in the form like 2009-01-30 23:01:45 . In order to calculate the total value of a ...
user avatar
  • 111
-1 votes
1 answer
190 views

how to fill nan values based on conditions in pandas?

I am working on the Titanic dataset and there are null values in the Age column. I would like to fill them in based on there Pclass. For example, if class is 1, I ...
user avatar
  • 73
0 votes
1 answer
34 views

Kernel dies or proses stuck when making LR prediction on dataframe using apply

I'm trying to making predictions with a simple model. model=LogisticRegression() model.fit(X_train,y_train) After fitting, i try to make predictions. A sample ...
user avatar
1 vote
1 answer
95 views

How to detect anomalies in each feature - time series

I have a dataset with 5 features corresponding to 5 sensors that measure each three seconds the state of an accelerator. It is structured as well: ...
user avatar
  • 43
0 votes
0 answers
32 views

Data science/Dataframe question concerning an approach to a problem

I want to answer which kind of school type (mixed gender, only for boys or only for girls) is more efficient in training its students based on the result of a normalized exam score. And my model ...
user avatar
  • 23
1 vote
1 answer
28 views

Change the ids of two linked pandas dataframes [closed]

Is it possible to change IDs columns into two linked pandas dataframes? Example: I have these two dataframes: Users and Ratings I would like the Users IDs to become as follows: This is my original ...
user avatar
1 vote
1 answer
144 views

Numpy array vs Pandas DataFrame when training [closed]

https://towardsdatascience.com/speed-testing-pandas-vs-numpy-ffbf80070ee7 (You can open the link in incognito if its locked). Numpy arrays are faster than DataFrame on normal mathematical operations. ...
user avatar
0 votes
2 answers
126 views

How to group by IDs and count the number of groups with occurrence of a variable after first point?

Language: Python 3.8 I have a dataframe that consists of a series of people (each appearing multiple times in the dataframe), dates, and binary variables. I am trying to figure out how many people ...
user avatar
  • 21

1
2 3 4 5
7