4
votes
How can I calculate mean and variance incrementally?
This problem was discussed, with proof and some alternate methods over on math.stackexchange.
4
votes
How can I calculate mean and variance incrementally?
Following that link about moving variance in my comment, I came upon this: Welford's online algorithm for calculating variance, which seems to supply what I'm looking for.
Here's the algorithm:
<...
- 181
4
votes
Accepted
Aggregation of Discount
Aggregate discount sequences are used to define the stacking logic for aggregate discounts. This logic can define:
The order in which aggregate discounts are applied.
Which aggregate discounts are ...
- 373
3
votes
Pandas dataframe resample aggregation by mills too slow
It appears you don't really want to use resampling. You are immediately throwing away the resampled data. I think what you actually need is to simply groupby records in the same millisecond. That ...
- 1,783
3
votes
How to group by multiple columns in dataframe using R and do aggregate function
I know this is an old post, but just thought I'd share my solution, which I think is a bit cleaner.
...
- 131
2
votes
Privacy through moving averages?
The key question is how we can allow the public to make useful queries to a dataset without revealing private information.
The field of Differential Privacy deals with answering just that.
The key ...
- 307
2
votes
Accepted
Pandas DataFrame: Aggregating multi-level groups by matching keys
Sorry, I realised all I needed to do was to add another level of grouping, but the aggregation is the same;
...
- 181
2
votes
Learning from aggregated data
I would disagree with the statement that "training a model on aggregated data is harder than training a model on raw data". It is different, but I wouldn't say fundametally.
It ultimately ...
- 461
2
votes
How to impute and aggregate data with ID variant variables for predictive modeling?
This sounds like you're having issues grappling with
relational theory.
You have focused on the ID column as though it identifies an observed example.
But your narrative ("multiple services")...
- 1,150
1
vote
Learning from aggregated data
There are a few reasons why training a machine learning model on aggregated data might be different or harder than training on raw event data:
Loss of Information: Aggregated data loses some of the ...
1
vote
How to aggregate the metrics from two different regression problems?
Indeed, you cannot aggregate because two problems may predict the price of stocks in dollars and the squared meters of houses, respectively.
Your solution is feasible but cannot adapt continually; the ...
- 669
1
vote
How to aggregate qualitative results from a simulation
If the features are considered categorical, think of the values as A,B,C,D. There's no possible mean value in this case, the most common way to aggregate would be to pick the mode, i.e. the value ...
- 25.7k
1
vote
How can we predict a value after several rows of data?
Try to make all the rows of the week one row.
Considering the max number of rows/weeks:
Week 1; $x_1, x_2, x_3...x_{90}, v_1, v_2, v_3...v_{90}...z_1, z_2, z_3...z_{90}; y_1$
Week 2; $x_1, x_2, x_3......
- 11
1
vote
Accepted
Labeling and aggregating features issue
The goal is to predict if a user will subscribe in the future. By definition you cannot have labelled data now about what people will do in the future. However you can phrase the problem like this: ...
- 25.7k
1
vote
How to get a (descriptive) overview of a large database?
There's no way to have a complete summary of a large dataset like this, you have to analyze what can be relevant, decompose into more specific pieces of information and then find the best way to ...
- 25.7k
1
vote
How do I deal with data that has only limited target values?
I would try two different approaches:
interpolate the missing values on a user level.
work with the sunset of rows for which we actually have the glucose level.
Then, I would compare the test ...
- 335
1
vote
Getting the earliest date (duplicates due to several call ids and agents)
This solved the problem:
select distinct on (customer)
customer, agent, min(call)
from call_data;
I am using PostgreSQL.
- 51
1
vote
Python Pandas agg error
If I am not mistaken the dataset used is the Boston home values dataset from http://lib.stat.cmu.edu/datasets/boston
The given code works fine on google colab if the dataset used is correct.
Please ...
1
vote
Accepted
Can I apply survival analysis to predict if a user will revisit the website?
If you want to use survival analysis (which can be more flexible and insightful), I'd recommend this package and this great tutorial. Speaking shortly, as a result, you'll get "probability of being ...
- 126
1
vote
How can I get total sum of each group by using pandas
Assuming you have the above dataframe as df:
...
- 175
1
vote
Accepted
What are the approaches to aggregate categorical variables?
You problem is essentially you have high cardinality in your features, right? This will be relative to your problem, but you can look for mean encodings. Essentially, you will replace categories by ...
- 810
1
vote
R-GUI How do i aggregate survey data collected for multiple years and see if they contain a variable?
If I understand your question correctly you'd like to obtain the proportion of missing values of a specific column in your dataset.
This can be obtained with the plyr library
For example:
...
- 11
1
vote
Privacy through moving averages?
Depending on what you're working with here, one approach that might work would be to "jitter" the data. In other words: add noise. Your concern seems to be about people getting recognized from ...
- 3,268
1
vote
R: Calculations based on frequencies / grouped / aggregate data
I think working with "collapsed" or "summarized" data as opposed to the data itself directly inside the SQL database is counter-intuitive. Also, trying to download the original table for analysis in R ...
- 276
Only top scored, non community-wiki answers of a minimum length are eligible