Skip to main content
Share Your Experience: Take the 2024 Developer Survey

Questions tagged [aggregation]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
1 vote
1 answer
29 views

How to impute and aggregate data with ID variant variables for predictive modeling?

I have a dataset that looks like so ID var_id_invariant_1 ... var_id_invariant_p var_id_variant_1 ... var_id_variant_k target 315 25 ... a 2.4 ... A 1 246 31 ... nan 5.7 ... B 0 315 25 ... a 9.4 .....
Mateusz's user avatar
  • 125
0 votes
0 answers
11 views

Record / subject wise aggregate classification

In the project I am studying I have a set of records (each record is a 1D signal) with variable length. For normalizing length and data augmentation, I am segmenting them in fixed length samples so I ...
Throwaway123's user avatar
0 votes
0 answers
886 views

Aggregate a DataFrame using multiple conditions in groupby

I have a dataframe with many duplicates. I have determined rules for duplicate identification, for example : If street_name AND ...
Rusoiba's user avatar
  • 849
1 vote
2 answers
785 views

Learning from aggregated data

Online and in the literature there seems to be a general consensus that training a machine learning model using aggregated data is harder and/or fundamentally different from training on raw event data....
dendog's user avatar
  • 120
0 votes
1 answer
35 views

How to aggregate the metrics from two different regression problems?

I'm about to conduct some tests to compare two solutions to regression problems. And to make the results more robust, I want to apply both on a few different datasets (all problems will be a ...
Mehran's user avatar
  • 277
0 votes
0 answers
56 views

Aggregating variables by two other variables in R using pivot_wider

I have data on cancer case counts in R that I would like to pivot from long to wide, such that a new count variable is created for each five-year age group within race/ethnicity groups. I have tried ...
Jack Murphy's user avatar
1 vote
0 answers
34 views

Optimizing Rank Aggregation of Two Different Methods in Information Retrieval

As the title suggests, I would like to train a rank-aggregating model. My target problem is to rank text2s from a database as best as possible to a given query, <...
You_Donut's user avatar
0 votes
1 answer
29 views

How to aggregate qualitative results from a simulation

I am trying to look for methods to aggregate and find the average of qualitative data that are outputted from a simulation. There are 20 qualitative measures, each divided unevenly into 4 cycles ...
worldCurrencies's user avatar
2 votes
0 answers
11 views

Learning the Average of a 0/1 Dependent Variable

uppose I have a matrix 𝑋 and a dependent vector 𝑦 whose entries are each in {0,1} dependent on the corresponding row of 𝑋 Given this dataset, I'd like to learn a model, so that given some other ...
Ami Tavory's user avatar
  • 1,277
0 votes
1 answer
144 views

How can we predict a value after several rows of data?

I have a regression problem in which for each week I have several rows (variable between rows i.e 1 week might have 1800 rows and other might have 5000 rows). My target is to predict a value at end of ...
Aneeq's user avatar
  • 1
0 votes
1 answer
254 views

Tableau: keeping results independent of view / filter

I am using Tableau Desktop 2021.1.4 Suppose that my source sales data consists of 4 columns Region (dimension with values: N,E,W,S), Type (dimension with values: Furniture, Electronics, Appliances), ...
yurnero's user avatar
  • 101
2 votes
1 answer
76 views

Labeling and aggregating features issue

I am trying build a simple binary classifier (some tree based algorithm for now) and my training data will have features aggregated at the user level. So I'll have a unique records of each user. These ...
cryp's user avatar
  • 123
0 votes
1 answer
21 views

Concatenating Data in two years

I have to use a Machine Learning Model to predict the Electricity consumption and carbon emission based on some buildings' features. (Area, year of construction ...) Here is the link to the data. The ...
W.314's user avatar
  • 101
-1 votes
1 answer
91 views

How to aggregate data inserted by users to avoid outliers?

I'm developing a new application based on machine learning. In this application users can insert new data to improve the prediction system. As you may guess, users could insert data that doesn't make ...
zzzz's user avatar
  • 1
1 vote
0 answers
117 views

MongoDB Groupby Rank [closed]

Im Working With Mongodb And Wanted to do a query using Aggregate fucntion. Query Is ...
Noob's user avatar
  • 11
0 votes
1 answer
210 views

Using R to organize/rearrange CSV - group by multiple columns?

I have a CSV that I need to clean up / organize in a usable way using R. I need to group by the property ID and then want to take all the unique years for the defor year column and make each year into ...
SpookyDLX's user avatar
1 vote
1 answer
39 views

How to get a (descriptive) overview of a large database?

I'm facing a data framework with ~ 20 k observations and 151 variables across 2078 subjects At first I am primarily interested in how the data looks like related to a single parameter. But I cannot ...
Ben's user avatar
  • 560
1 vote
0 answers
1k views

Heatmap of large 2D array using datashader and plotly

I’m trying to show a heatmap of a large 2D array (160x250000 entries). This should go into a dash app so I'm using plotly to deal with graphics and my idea was to use datashader for performance but I’...
andins's user avatar
  • 56
1 vote
0 answers
21 views

Data system that manages aggregates over time intervals

I am looking to know if there is a data system that handles the following use case. To keep it simple, the data is a set of homogeneous enties E. E contains named numeric properties that the app code ...
naftalimich's user avatar
3 votes
0 answers
138 views

Difference Bagging and Bootstrap aggregating

Bootstrap belongs to Efron. Tibshirani wrote a book about that in reference to Efron. Bootstrap process for estimating the standard error of statistic s(x). B bootstrap sample are generatied from ...
martin's user avatar
  • 329
3 votes
0 answers
29 views

User defined aggregations on data of around 200GB where row order matters

I am working with "medium large" data of around 200GB. The data are long form log files, where there are several thousand logs for each "entity". The entities are actually flights ...
Placidia's user avatar
  • 226
1 vote
1 answer
63 views

How do I deal with data that has only limited target values?

I'm currently working on a small project using the D1NAMO dataset (1). I want to predict the glucose level (that is given in the dataset) based on several features: accelerometer data, heartbeat (ECG) ...
sander's user avatar
  • 93
0 votes
1 answer
14 views

Getting the earliest date (duplicates due to several call ids and agents)

I am trying to get the earliest call interaction, with only the agent that interacted at that call and the customer id. Using the following code, I still get duplicates: ...
ColRow's user avatar
  • 53
1 vote
2 answers
3k views

Python Pandas agg error

I am trying to generate descriptive statistics using agg function in Pandas. I am having trouble with one line with a lambda function. They work when I run them as separate lines of code, but when I ...
4Walk's user avatar
  • 11
2 votes
1 answer
217 views

Feature Selection on Aggregated Targetdata

I have a question about feature selection on a dataset where the target variable is aggregated by the sum of different data points. I want to predict the number of sales depending on a variety of ...
Alexander Fratzer's user avatar
3 votes
1 answer
216 views

Can I apply survival analysis to predict if a user will revisit the website?

I have one business problem in hand which is to predict if a user will revisit the website or not within 6 months. I need to majorly understand what are the factors which make the user return and also ...
asspsss's user avatar
  • 75
1 vote
1 answer
92 views

How do I deal with changing values in a categorical variable when I am aggregating customer records

My requirement is to build a model to predict if a new customer will return to their website. I need to analyze what drives customer repeat for both new and returning customers. The only information ...
asspsss's user avatar
  • 75
0 votes
1 answer
44 views

How can I get total sum of each group by using pandas

I have a dataframe shaped like below ...
JHJ's user avatar
  • 1
2 votes
1 answer
195 views

How to deal with a potencially multiple categorical variable

I'm build a model that has, as inputs, some categorical variables. I had already dealt with this sort of data before, and applied different techniques as creation of dummy variables and factor scoring....
Diogo Santos's user avatar
2 votes
1 answer
4k views

What are the approaches to aggregate categorical variables?

I am working on a clickstream dataset. I have come up with the following example dataset to explain my problem: ...
Amir's user avatar
  • 123
3 votes
1 answer
2k views

Pandas DataFrame: Aggregating multi-level groups by matching keys

I have some data that looks like this; ...
Steztric's user avatar
  • 181
1 vote
2 answers
518 views

I want to be able to collapse and sum values dependent on the gene name

I have a table that looks like this: I want add together all the values for each gene for each column. For example, for LINC01128, it should read: ConN1 ConN2 ConN3 StN1 StN2 ...
Mark Hickling's user avatar
1 vote
3 answers
363 views

Data aggregation and split train test samples

I'm working on a data science project where the goal is to predict daily electricity consumption of a building based on some of its characteristics (e.g., size, location, etc.) and weather features (e....
JaYo's user avatar
  • 15
1 vote
3 answers
2k views

Aggregating standard deviations

Imagine I have a collection of data, let's say the travel time for a road segment. On this collection I want to calculate the mean and the standard deviation. Nothing hard so far. Now imagine that ...
Nakeuh's user avatar
  • 238
0 votes
2 answers
297 views

R-GUI How do i aggregate survey data collected for multiple years and see if they contain a variable?

I am new to R and usually rely on stata. I have a considerably large data frame, it contains data from a broad range of surveys and years, each with their own types of classifications and answers. ...
Shivy b's user avatar
  • 11
0 votes
1 answer
258 views

Ideal aggregation function for Partially Connected Neural Network (PCNN)

I am building a Python library that creates Partially Connected Neural Networks based on input and output data (X,Y). The basic gist is that the network graph is arbitrarily updated with nodes and ...
abrahamrhoffman's user avatar
2 votes
2 answers
61 views

Privacy through moving averages?

I am considering the following hypothetical situation: I have a time series of data. In general, 'the public' should have access to features of this data. However, making the time series available ...
Elle Najt's user avatar
  • 131
1 vote
1 answer
66 views

Supervised learning on sources of information with different importance

I am trying to classify customer support sessions using supervised machine learning. In each customer support session I have 3 bags of information. 1. The title of the customer's complaint 2. ...
Toby's user avatar
  • 51
4 votes
2 answers
3k views

How can I calculate mean and variance incrementally?

Say I have a set S of values, and want to store in a database some summary information about that set, so that later when I acquire a new value v I can make a reasonable estimate of what the summary ...
dubiousjim's user avatar
0 votes
1 answer
2k views

R: Calculations based on frequencies / grouped / aggregate data

I am trying to do simple calculations in R when no raw data but grouped data with frequencies is available only. This is the case when I have a large amount of records in a database, say a large SQL ...
joffdd's user avatar
  • 11
4 votes
1 answer
3k views

Pandas dataframe resample aggregation by mills too slow

Given this test data: ...
hinzsa's user avatar
  • 41
3 votes
1 answer
327 views

Aggregation of Discount

I am trying to predict sales quantity of an item based on their attributes. Discount is one of those attributes. The problem is I am having different discounts in same period for same item .I need to ...
maddini madhuri's user avatar
2 votes
0 answers
112 views

Aggregating over sparse data

I am not sure if the title accurately reflects my problem but I essentially would like to aggregate a set of metrics of similar nature that comes from different data sources into a single metric. Say ...
chi's user avatar
  • 131
0 votes
2 answers
45 views

Analytics term for turning row values into column names and count its assigned values

Do we have a data mining/analysis term for turning row values into column names and count its assigned values?
SoakingHummer's user avatar
7 votes
4 answers
158k views

How to group by multiple columns in dataframe using R and do aggregate function

I have a dataframe with columns as defined below. I have provided one set of example, similar to this I have many countries with loan amount and gender variables ...
SRS's user avatar
  • 1,065
4 votes
2 answers
147 views

Approach to creating a user profile in music web application

I am working on a use case, and I'm unsure of the best way to proceed: in order to analyze the behavior of users of a web-based music application, we retain all songs each has played since 2009. We ...
user17241's user avatar
  • 151