1
$\begingroup$

I am trying to create several new datasets based on an existing dataset. The new datasets should follow the trends (i.e., the correlation within the data) of that existing dataset.

As an example, I would like to create five new datasets very likely to the following (existing) dataset. However, the new datasets do not contain the exact data/value from the following dataset, rather, they contain new data maintaining the similar trends/relationships that is present in the following dataset.

Column A Column B Column C Column D Column E Column B
20186 3132.94 2145.65 586.51 2189.64 740.06
20187 3132.94 2170.09 581.62 2174.98 750.56
20188 3137.83 2140.76 586.51 2194.53 751.31
20189 3137.83 2150.54 581.62 2184.75 748.69
20190 3137.83 2170.09 581.62 2174.98 746.25
20191 3137.83 2165.2 581.62 2155.43 743.25
20192 3137.83 2145.65 586.51 2189.64 741
20193 3137.83 2145.65 586.51 2184.75 747.94
20194 3132.94 2145.65 586.51 2189.64 741.56
20195 3137.83 2145.65 586.51 2189.64 735.94
20196 3132.94 2170.09 581.62 2165.2 738.56
20197 3137.83 2179.86 581.62 2150.54 734.44
20198 3137.83 2165.2 581.62 2179.86 734.06
20199 3132.94 2145.65 586.51 2194.53 734.63
20200 3137.83 2160.31 581.62 2174.98 742.13

I've tried creating dataset using pandas. There are several ways in pandas to create new dataset. Such as, creating a new dataset using multiple columns from the existing dataset, which is creating the subset of that existing dataset. I am not trying to create a subset of the data frame, rather I am trying to create new datasets that maintain the same relationship between the data as the existing one.

I guess I should use Pandas for this. However, I am not getting any lead to solve the problem. Any help is heartily appreciated.

$\endgroup$

2 Answers 2

2
$\begingroup$

You are looking for a generative models. Generative models are typically trained to create "similar" data without plainly copying it. These days, generative Models create astonishing results, including art, music, text. But back to your case:

There are multiple variants. You could look into Variational Autoencoders (VAEs) or Generative Adversarial Networks (GANs). Depending on your data, ctGAN might be a good start. It is designed to work with tabular data which you seem to have.

$\endgroup$
1
$\begingroup$

I am guessing you are hinting that you want new data where the relationship between columns is maintained (otherwise it would be easy just to create semi-random data using the values already seen).

If you have ground truth as to these relationships then it would be as simple as adding noise to the output of the calculations.

If you dont have ground truth I feel the best you can do is attempt to model it (regression) and then do the same as above. However this would not represent 'real ground truth data' in anyway but maybe that is not what you want anyway?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.