0
$\begingroup$

I have a very limited amount of data. I need to generate more synthetic data . My data shape is (x,21) . Right now , I have used KS (Kolmogorov-Smirnov) test to get the closest matching distribution and get its parameters. Can anyone please help me , how can I generate synthetic data from this approach ?

Here is my code snippet , which I tried.

   import scipy.stats as st
    def get_best_distribution(data):
        dist_names = ["norm", "exponweib", "weibull_max", "weibull_min", "pareto", "genextreme",\
                      "beta","invgauss","uniform","gamma","expon","lognorm","pearson3","triang"]
        dist_results = []
        params = {}
        for dist_name in dist_names:
            dist = getattr(st, dist_name)
            param = dist.fit(data)
    
            params[dist_name] = param
            # Applying the Kolmogorov-Smirnov test
            D, p = st.kstest(data, dist_name, args=param)
            #print("p value for "+dist_name+" = "+str(p))
            #print("D value for {} = {}".format(dist_name,str(D)))
            #print()
            dist_results.append((dist_name, p))
    
        # select the best fitted distribution
        best_dist, best_p = (max(dist_results, key=lambda item: item[1]))
        # store the name of the best fit and its p value
    
        print("Best fitting distribution: "+str(best_dist))
        print("Best p value: "+ str(best_p))
        print("Parameters for the best fit: "+ str(params[best_dist]))
        print()
    
        return best_dist, best_p, params[best_dist]

columns_list = list(synthetic_data.columns)

distribution_dict={}

for tmp_column in columns_list:
    
    sample_data=synthetic_data[tmp_column].astype(float)

    best_dist_sample, best_p_sample , params_sample = get_best_distribution(data=sample_data)    
    distribution_dict[tmp_column] = tuple((best_dist_sample,best_p_sample,params_sample))
    
    pass
    
$\endgroup$
6
  • $\begingroup$ I might be mistaken but I don't think the Kolmogorov distribution can be used for generation: as far as I'm aware the parameters you obtain are not those of the data distribution. In order to estimate the parameters of the data you would probably need to make an assumption about the distribution of the data, for instance assuming it's a normal distribution. $\endgroup$ – Erwan Mar 10 at 0:45
  • $\begingroup$ hi @Erwan , thanks for your reply. Do you recommend any way to generate data from the same distribution (apart from the Generative Adversarial Network) approach? $\endgroup$ – DukeLover Mar 10 at 13:10
  • 1
    $\begingroup$ sorry, I don't know much about this topic. The only thing I would know is to estimate the parameters of a specific kind of distribution (e.g. normal) and then generate from these parameters. $\endgroup$ – Erwan Mar 10 at 14:31
  • $\begingroup$ Hi @Erwan: Thanks once again. Could you please share some links on generating synthetic data from parameter estimation? It would be really helpful. $\endgroup$ – DukeLover Mar 11 at 5:59
  • 1
    $\begingroup$ I think SMOTE is one of the generic methods which can be used to generate synthetic data from a small sample, see for instance here. Otherwise the very basic method is to generate for each feature individually by calculating the parameters of this feature, for instance mean and standard deviation. Then you just generate points following these parameters. Be careful, synthetic data is rarely as good as real data. $\endgroup$ – Erwan Mar 11 at 10:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.