Questions tagged [aggregation]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
1
vote
1answer
15 views

How to get a (descriptive) overview of a large database?

I'm facing a data framework with ~ 20 k observations and 151 variables across 2078 subjects At first I am primarily interested in how the data looks like related to a single parameter. But I cannot ...
1
vote
0answers
42 views

Heatmap of large 2D array using datashader and plotly

I’m trying to show a heatmap of a large 2D array (160x250000 entries). This should go into a dash app so I'm using plotly to deal with graphics and my idea was to use datashader for performance but I’...
0
votes
0answers
17 views

Data system that manages aggregates over time intervals

I am looking to know if there is a data system that handles the following use case. To keep it simple, the data is a set of homogeneous enties E. E contains named numeric properties that the app code ...
3
votes
0answers
60 views

Difference Bagging and Bootstrap aggregating

Bootstrap belongs to Efron. Tibshirani wrote a book about that in reference to Efron. Bootstrap process for estimating the standard error of statistic s(x). B bootstrap sample are generatied from ...
2
votes
0answers
19 views

User defined aggregations on data of around 200GB where row order matters

I am working with "medium large" data of around 200GB. The data are long form log files, where there are several thousand logs for each "entity". The entities are actually flights ...
1
vote
1answer
16 views

How do I deal with data that has only limited target values?

I'm currently working on a small project using the D1NAMO dataset (1). I want to predict the glucose level (that is given in the dataset) based on several features: accelerometer data, heartbeat (ECG) ...
0
votes
0answers
15 views

Event modelling on aggregated data

I have a data that I want to use in event occurrence modeling, however, data was prepared in a way that I have exposure (can be fractional, eg. half of period) and event occurrence in some groups. It ...
0
votes
1answer
11 views

Getting the earliest date (duplicates due to several call ids and agents)

I am trying to get the earliest call interaction, with only the agent that interacted at that call and the customer id. Using the following code, I still get duplicates: ...
1
vote
1answer
544 views

Python Pandas agg error

I am trying to generate descriptive statistics using agg function in Pandas. I am having trouble with one line with a lambda function. They work when I run them as separate lines of code, but when I ...
2
votes
1answer
67 views

Feature Selection on Aggregated Targetdata

I have a question about feature selection on a dataset where the target variable is aggregated by the sum of different data points. I want to predict the number of sales depending on a variety of ...
3
votes
1answer
70 views

Can I apply survival analysis to predict if a user will revisit the website?

I have one business problem in hand which is to predict if a user will revisit the website or not within 6 months. I need to majorly understand what are the factors which make the user return and also ...
1
vote
1answer
42 views

How do I deal with changing values in a categorical variable when I am aggregating customer records

My requirement is to build a model to predict if a new customer will return to their website. I need to analyze what drives customer repeat for both new and returning customers. The only information ...
0
votes
1answer
24 views

How can I get total sum of each group by using pandas

I have a dataframe shaped like below ...
2
votes
0answers
92 views

How to deal with a potencially multiple categorical variable

I'm build a model that has, as inputs, some categorical variables. I had already dealt with this sort of data before, and applied different techniques as creation of dummy variables and factor scoring....
3
votes
1answer
1k views

What are the approaches to aggregate categorical variables?

I am working on a clickstream dataset. I have come up with the following example dataset to explain my problem: ...
3
votes
1answer
1k views

Pandas DataFrame: Aggregating multi-level groups by matching keys

I have some data that looks like this; ...
1
vote
2answers
99 views

I want to be able to collapse and sum values dependent on the gene name

I have a table that looks like this: I want add together all the values for each gene for each column. For example, for LINC01128, it should read: ConN1 ConN2 ConN3 StN1 StN2 ...
1
vote
3answers
139 views

Data aggregation and split train test samples

I'm working on a data science project where the goal is to predict daily electricity consumption of a building based on some of its characteristics (e.g., size, location, etc.) and weather features (e....
0
votes
2answers
174 views

Aggregating standard deviations

Imagine I have a collection of data, let's say the travel time for a road segment. On this collection I want to calculate the mean and the standard deviation. Nothing hard so far. Now imagine that ...
0
votes
1answer
232 views

R-GUI How do i aggregate survey data collected for multiple years and see if they contain a variable?

I am new to R and usually rely on stata. I have a considerably large data frame, it contains data from a broad range of surveys and years, each with their own types of classifications and answers. ...
0
votes
1answer
188 views

Ideal aggregation function for Partially Connected Neural Network (PCNN)

I am building a Python library that creates Partially Connected Neural Networks based on input and output data (X,Y). The basic gist is that the network graph is arbitrarily updated with nodes and ...
2
votes
2answers
50 views

Privacy through moving averages?

I am considering the following hypothetical situation: I have a time series of data. In general, 'the public' should have access to features of this data. However, making the time series available ...
1
vote
1answer
55 views

Supervised learning on sources of information with different importance

I am trying to classify customer support sessions using supervised machine learning. In each customer support session I have 3 bags of information. 1. The title of the customer's complaint 2. ...
1
vote
2answers
682 views

How can I calculate mean and variance incrementally?

Say I have a set S of values, and want to store in a database some summary information about that set, so that later when I acquire a new value v I can make a reasonable estimate of what the summary ...
-1
votes
1answer
1k views

R: Calculations based on frequencies / grouped / aggregate data

I am trying to do simple calculations in R when no raw data but grouped data with frequencies is available only. This is the case when I have a large amount of records in a database, say a large SQL ...
4
votes
1answer
3k views

Pandas dataframe resample aggregation by mills too slow

Given this test data: ...
3
votes
1answer
146 views

Aggregation of Discount

I am trying to predict sales quantity of an item based on their attributes. Discount is one of those attributes. The problem is I am having different discounts in same period for same item .I need to ...
2
votes
0answers
51 views

Aggregating over sparse data

I am not sure if the title accurately reflects my problem but I essentially would like to aggregate a set of metrics of similar nature that comes from different data sources into a single metric. Say ...
0
votes
2answers
40 views

Analytics term for turning row values into column names and count its assigned values

Do we have a data mining/analysis term for turning row values into column names and count its assigned values?
3
votes
3answers
101k views

How to group by multiple columns in dataframe using R and do aggregate function

I have a dataframe with columns as defined below. I have provided one set of example, similar to this I have many countries with loan amount and gender variables ...
4
votes
2answers
135 views

Approach to creating a user profile in music web application

I am working on a use case, and I'm unsure of the best way to proceed: in order to analyze the behavior of users of a web-based music application, we retain all songs each has played since 2009. We ...