Questions tagged [aggregation]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0
votes
0answers
11 views

Aggregate grouped data conditionally over many columns doing different operations in Python/Pandas [migrated]

I'm not sure this is a general coding question or not but I hope this is the correct forum. Consider the following reduced example data frame df: ...
2
votes
1answer
41 views

Labeling and aggregating features issue

I am trying build a simple binary classifier (some tree based algorithm for now) and my training data will have features aggregated at the user level. So I'll have a unique records of each user. These ...
0
votes
1answer
16 views

Concatenating Data in two years

I have to use a Machine Learning Model to predict the Electricity consumption and carbon emission based on some buildings' features. (Area, year of construction ...) Here is the link to the data. The ...
0
votes
0answers
11 views

How do you aggregate features of lists (pooling alternatives)?

Is it possible to reduce non-correlated multi-dimensional data over features to 1D data? A working option is pooling (mean/min/max) over an embedding vector (n samples of embeddings of m dimensions). ...
-1
votes
1answer
44 views

How to aggregate data inserted by users to avoid outliers?

I'm developing a new application based on machine learning. In this application users can insert new data to improve the prediction system. As you may guess, users could insert data that doesn't make ...
1
vote
0answers
53 views

MongoDB Groupby Rank [closed]

Im Working With Mongodb And Wanted to do a query using Aggregate fucntion. Query Is ...
0
votes
1answer
114 views

Using R to organize/rearrange CSV - group by multiple columns?

I have a CSV that I need to clean up / organize in a usable way using R. I need to group by the property ID and then want to take all the unique years for the defor year column and make each year into ...
1
vote
1answer
17 views

How to get a (descriptive) overview of a large database?

I'm facing a data framework with ~ 20 k observations and 151 variables across 2078 subjects At first I am primarily interested in how the data looks like related to a single parameter. But I cannot ...
1
vote
0answers
359 views

Heatmap of large 2D array using datashader and plotly

I’m trying to show a heatmap of a large 2D array (160x250000 entries). This should go into a dash app so I'm using plotly to deal with graphics and my idea was to use datashader for performance but I’...
1
vote
0answers
21 views

Data system that manages aggregates over time intervals

I am looking to know if there is a data system that handles the following use case. To keep it simple, the data is a set of homogeneous enties E. E contains named numeric properties that the app code ...
3
votes
0answers
75 views

Difference Bagging and Bootstrap aggregating

Bootstrap belongs to Efron. Tibshirani wrote a book about that in reference to Efron. Bootstrap process for estimating the standard error of statistic s(x). B bootstrap sample are generatied from ...
3
votes
0answers
23 views

User defined aggregations on data of around 200GB where row order matters

I am working with "medium large" data of around 200GB. The data are long form log files, where there are several thousand logs for each "entity". The entities are actually flights ...
1
vote
1answer
44 views

How do I deal with data that has only limited target values?

I'm currently working on a small project using the D1NAMO dataset (1). I want to predict the glucose level (that is given in the dataset) based on several features: accelerometer data, heartbeat (ECG) ...
0
votes
1answer
12 views

Getting the earliest date (duplicates due to several call ids and agents)

I am trying to get the earliest call interaction, with only the agent that interacted at that call and the customer id. Using the following code, I still get duplicates: ...
1
vote
2answers
1k views

Python Pandas agg error

I am trying to generate descriptive statistics using agg function in Pandas. I am having trouble with one line with a lambda function. They work when I run them as separate lines of code, but when I ...
2
votes
1answer
90 views

Feature Selection on Aggregated Targetdata

I have a question about feature selection on a dataset where the target variable is aggregated by the sum of different data points. I want to predict the number of sales depending on a variety of ...
3
votes
1answer
117 views

Can I apply survival analysis to predict if a user will revisit the website?

I have one business problem in hand which is to predict if a user will revisit the website or not within 6 months. I need to majorly understand what are the factors which make the user return and also ...
1
vote
1answer
52 views

How do I deal with changing values in a categorical variable when I am aggregating customer records

My requirement is to build a model to predict if a new customer will return to their website. I need to analyze what drives customer repeat for both new and returning customers. The only information ...
0
votes
1answer
26 views

How can I get total sum of each group by using pandas

I have a dataframe shaped like below ...
2
votes
0answers
112 views

How to deal with a potencially multiple categorical variable

I'm build a model that has, as inputs, some categorical variables. I had already dealt with this sort of data before, and applied different techniques as creation of dummy variables and factor scoring....
2
votes
1answer
2k views

What are the approaches to aggregate categorical variables?

I am working on a clickstream dataset. I have come up with the following example dataset to explain my problem: ...
3
votes
1answer
2k views

Pandas DataFrame: Aggregating multi-level groups by matching keys

I have some data that looks like this; ...
1
vote
2answers
168 views

I want to be able to collapse and sum values dependent on the gene name

I have a table that looks like this: I want add together all the values for each gene for each column. For example, for LINC01128, it should read: ConN1 ConN2 ConN3 StN1 StN2 ...
1
vote
3answers
184 views

Data aggregation and split train test samples

I'm working on a data science project where the goal is to predict daily electricity consumption of a building based on some of its characteristics (e.g., size, location, etc.) and weather features (e....
1
vote
2answers
269 views

Aggregating standard deviations

Imagine I have a collection of data, let's say the travel time for a road segment. On this collection I want to calculate the mean and the standard deviation. Nothing hard so far. Now imagine that ...
0
votes
1answer
260 views

R-GUI How do i aggregate survey data collected for multiple years and see if they contain a variable?

I am new to R and usually rely on stata. I have a considerably large data frame, it contains data from a broad range of surveys and years, each with their own types of classifications and answers. ...
0
votes
1answer
213 views

Ideal aggregation function for Partially Connected Neural Network (PCNN)

I am building a Python library that creates Partially Connected Neural Networks based on input and output data (X,Y). The basic gist is that the network graph is arbitrarily updated with nodes and ...
2
votes
2answers
52 views

Privacy through moving averages?

I am considering the following hypothetical situation: I have a time series of data. In general, 'the public' should have access to features of this data. However, making the time series available ...
1
vote
1answer
55 views

Supervised learning on sources of information with different importance

I am trying to classify customer support sessions using supervised machine learning. In each customer support session I have 3 bags of information. 1. The title of the customer's complaint 2. ...
2
votes
2answers
1k views

How can I calculate mean and variance incrementally?

Say I have a set S of values, and want to store in a database some summary information about that set, so that later when I acquire a new value v I can make a reasonable estimate of what the summary ...
0
votes
1answer
1k views

R: Calculations based on frequencies / grouped / aggregate data

I am trying to do simple calculations in R when no raw data but grouped data with frequencies is available only. This is the case when I have a large amount of records in a database, say a large SQL ...
4
votes
1answer
3k views

Pandas dataframe resample aggregation by mills too slow

Given this test data: ...
3
votes
1answer
197 views

Aggregation of Discount

I am trying to predict sales quantity of an item based on their attributes. Discount is one of those attributes. The problem is I am having different discounts in same period for same item .I need to ...
2
votes
0answers
57 views

Aggregating over sparse data

I am not sure if the title accurately reflects my problem but I essentially would like to aggregate a set of metrics of similar nature that comes from different data sources into a single metric. Say ...
0
votes
2answers
41 views

Analytics term for turning row values into column names and count its assigned values

Do we have a data mining/analysis term for turning row values into column names and count its assigned values?
4
votes
3answers
125k views

How to group by multiple columns in dataframe using R and do aggregate function

I have a dataframe with columns as defined below. I have provided one set of example, similar to this I have many countries with loan amount and gender variables ...
4
votes
2answers
138 views

Approach to creating a user profile in music web application

I am working on a use case, and I'm unsure of the best way to proceed: in order to analyze the behavior of users of a web-based music application, we retain all songs each has played since 2009. We ...