# Tag Info

### How can I calculate mean and variance incrementally?

This problem was discussed, with proof and some alternate methods over on math.stackexchange.

### How can I calculate mean and variance incrementally?

Following that link about moving variance in my comment, I came upon this: Welford's online algorithm for calculating variance, which seems to supply what I'm looking for. Here's the algorithm: <...
• 181
Accepted

### Aggregation of Discount

Aggregate discount sequences are used to define the stacking logic for aggregate discounts. This logic can define: The order in which aggregate discounts are applied. Which aggregate discounts are ...

### Pandas dataframe resample aggregation by mills too slow

It appears you don't really want to use resampling. You are immediately throwing away the resampled data. I think what you actually need is to simply groupby records in the same millisecond. That ...
• 1,783

### How to group by multiple columns in dataframe using R and do aggregate function

I know this is an old post, but just thought I'd share my solution, which I think is a bit cleaner. ...
• 131

### Privacy through moving averages?

The key question is how we can allow the public to make useful queries to a dataset without revealing private information. The field of Differential Privacy deals with answering just that. The key ...
• 307
Accepted

### Pandas DataFrame: Aggregating multi-level groups by matching keys

Sorry, I realised all I needed to do was to add another level of grouping, but the aggregation is the same; ...
• 181

### Learning from aggregated data

I would disagree with the statement that "training a model on aggregated data is harder than training a model on raw data". It is different, but I wouldn't say fundametally. It ultimately ...
• 461

### How to impute and aggregate data with ID variant variables for predictive modeling?

This sounds like you're having issues grappling with relational theory. You have focused on the ID column as though it identifies an observed example. But your narrative ("multiple services")...
• 1,150
1 vote

### Learning from aggregated data

There are a few reasons why training a machine learning model on aggregated data might be different or harder than training on raw event data: Loss of Information: Aggregated data loses some of the ...
1 vote

### How to aggregate the metrics from two different regression problems?

Indeed, you cannot aggregate because two problems may predict the price of stocks in dollars and the squared meters of houses, respectively. Your solution is feasible but cannot adapt continually; the ...
• 669
1 vote

### How to aggregate qualitative results from a simulation

If the features are considered categorical, think of the values as A,B,C,D. There's no possible mean value in this case, the most common way to aggregate would be to pick the mode, i.e. the value ...
• 25.7k
1 vote

### How can we predict a value after several rows of data?

Try to make all the rows of the week one row. Considering the max number of rows/weeks: Week 1; $x_1, x_2, x_3...x_{90}, v_1, v_2, v_3...v_{90}...z_1, z_2, z_3...z_{90}; y_1$ Week 2; \$x_1, x_2, x_3......
1 vote
Accepted

### Labeling and aggregating features issue

The goal is to predict if a user will subscribe in the future. By definition you cannot have labelled data now about what people will do in the future. However you can phrase the problem like this: ...
• 25.7k
1 vote

### How to get a (descriptive) overview of a large database?

There's no way to have a complete summary of a large dataset like this, you have to analyze what can be relevant, decompose into more specific pieces of information and then find the best way to ...
• 25.7k
1 vote

### How do I deal with data that has only limited target values?

I would try two different approaches: interpolate the missing values on a user level. work with the sunset of rows for which we actually have the glucose level. Then, I would compare the test ...
1 vote

### Getting the earliest date (duplicates due to several call ids and agents)

This solved the problem: select distinct on (customer) customer, agent, min(call) from call_data; I am using PostgreSQL.
• 51
1 vote

### Python Pandas agg error

If I am not mistaken the dataset used is the Boston home values dataset from http://lib.stat.cmu.edu/datasets/boston The given code works fine on google colab if the dataset used is correct. Please ...
1 vote
Accepted

### Can I apply survival analysis to predict if a user will revisit the website?

If you want to use survival analysis (which can be more flexible and insightful), I'd recommend this package and this great tutorial. Speaking shortly, as a result, you'll get "probability of being ...
• 126
1 vote

### How can I get total sum of each group by using pandas

Assuming you have the above dataframe as df: ...
1 vote
Accepted

### What are the approaches to aggregate categorical variables?

You problem is essentially you have high cardinality in your features, right? This will be relative to your problem, but you can look for mean encodings. Essentially, you will replace categories by ...
1 vote

### R-GUI How do i aggregate survey data collected for multiple years and see if they contain a variable?

If I understand your question correctly you'd like to obtain the proportion of missing values of a specific column in your dataset. This can be obtained with the plyr library For example: ...
• 11
1 vote

### Privacy through moving averages?

Depending on what you're working with here, one approach that might work would be to "jitter" the data. In other words: add noise. Your concern seems to be about people getting recognized from ...
• 3,268
1 vote

### R: Calculations based on frequencies / grouped / aggregate data

I think working with "collapsed" or "summarized" data as opposed to the data itself directly inside the SQL database is counter-intuitive. Also, trying to download the original table for analysis in R ...
• 276

Only top scored, non community-wiki answers of a minimum length are eligible