I am looking for an approach to generate synthetic data for anomaly detection. We have real data, but want to inject anomalies to battle-test the model (the real data is too limited for likely future anomalies).
I would like to mimic the statistical properties of the real data, such as mean, mode, standard deviation, etc., to create the synthetic data, then inject anomalies based off reasonable extreme values (if we know the statistical properties of each column in the real data then we can deduce what an extreme value might look like for that column).
Are there any Python packages that generate synthetic data based off known statistical properties in real data. I imagine this is similar to differential privacy, but we are not doing this to protect privacy and don't need an overkill method.
scikit-learn can generate synthetic data but it doesn't seem to have a method to base it off of existing real data statistical properties.
I can do something simple like this:
res = {}
for column in df:
nrows = len(df[column].index)
mean = df[column].mean()
std = df[column].std()
mu, sigma = mean, std # mean and standard deviation
synthetic_data = np.random.normal(mu, sigma, nrows)
res[column] = synthetic_data
...which just detects the means and standard deviation of each column then recreates it using a numpy draw from a normal distribution (big assumption), but obviously this doesn't mock the data well:
Real Data
Synthetic Data
