2
$\begingroup$

I’m in a place where we’re unable to train models on data due to GDPR. What I want is to predict people getting a job (y) given (x,x,x,x…) their employment type working full time or part time, work sector, area and age.

Since I cannot train on real data, I created an aggregate from the real data. Then used numpy.random.choice to generate synthetic data frame with data from the aggregate. Then created a decision tree boost model that had a good f1 score and accuracy. The behaviour of the model is similar to the aggregate.

When I presented this to people, they informed me that what I did is not a real model and by recreating data from an aggregate, you loose important correlations from the data for individuals represented in the dataset.

I disagree on these points since the model is acting based on the aggregate.

After reading, I figured out that you do not need to have correlations to build models. There are many research papers based on non linear dataset with no/few correlations.

I don’t understand how a model from an aggregate looses its statistical traits.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

There are correlations regarding the creation dates or maybe the geographical zone, but if you keep the study in a similar time frame, you shouldn't lose something important.

There is a correlation loss as long as there are links between individuals and the environment (ex: time and place). If there are no links between individuals, I don't see why it is not meant to make an aggregate. In addition, most models need random data to find statistical straits, one exception is time series because the models learn from sequential data that couldn't be randomized for obvious reasons.

It would be very interesting to know the reason why an aggregate loses its statistical straits (apart from the time frame).

Improve this answer
$\endgroup$
2
  • $\begingroup$ Does it answer your question? If not, please let me know. $\endgroup$
    – Nicolas Martin
    Sep 22, 2022 at 7:33
  • 1
    $\begingroup$ Thank you very much! You are correct 👍. They did not fully understand my aggregate at first. When they saw the aggregate again, they mentioned something similar to what you mentioned here! I’m thrilled to say the idea worked in the end! $\endgroup$
    – SamTheGoodOne
    Oct 4, 2022 at 10:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.