2
$\begingroup$

I am looking for an approach to generate synthetic data for anomaly detection. We have real data, but want to inject anomalies to battle-test the model (the real data is too limited for likely future anomalies).

I would like to mimic the statistical properties of the real data, such as mean, mode, standard deviation, etc., to create the synthetic data, then inject anomalies based off reasonable extreme values (if we know the statistical properties of each column in the real data then we can deduce what an extreme value might look like for that column).

Are there any Python packages that generate synthetic data based off known statistical properties in real data. I imagine this is similar to differential privacy, but we are not doing this to protect privacy and don't need an overkill method.

scikit-learn can generate synthetic data but it doesn't seem to have a method to base it off of existing real data statistical properties.

I can do something simple like this:

res = {}
for column in df: 
    nrows = len(df[column].index)
    mean = df[column].mean()
    std = df[column].std()
    mu, sigma = mean, std # mean and standard deviation
    synthetic_data = np.random.normal(mu, sigma, nrows)
    res[column] = synthetic_data

...which just detects the means and standard deviation of each column then recreates it using a numpy draw from a normal distribution (big assumption), but obviously this doesn't mock the data well:

Real Data

enter image description here

Synthetic Data

enter image description here

$\endgroup$
1
  • $\begingroup$ It is not 100% clear what you call an anomaly in your graph : the spike at 1400 ? the absence of value before it ? other spikes ? a signal that is not observed yet ? $\endgroup$ Oct 18, 2020 at 14:24

2 Answers 2

0
$\begingroup$

One option is the Python package imblearn which contains the SMOTE algorithm. SMOTE generates synthetic samples from a real dataset by interpolating plausible new datapoints based on observed data.

$\endgroup$
0
$\begingroup$

Why not work in the frequency domain? Take a Fourier transform of your real data and then independently perturb the resultant individual sine waves by increasing/decreasing the amplitudes by a small amount and, similarly, slightly increasing/decreasing the frequencies and then recombine to get similar looking synthetic data.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.