Gaussian models are often used (and maybe sometimes over-used) because of their mathematical convenience (many statistical models can be found as built-in functions, when based on the Gaussian distribution, in some libraries such as mixture models, hidden Markov models,...). Also, when one has no idea about what distribution could best model the data, given the relationship of the Gaussian distribution with the Central-Limit Theorem, it is often a reasonable assumption to make.
However, if say, your goal is to generate new data similar to some training data and that you know that the data you would like to generate should be strictly positive, then a Gaussian assumption may not be the best one. Indeed, generating data from a Gaussian-based model gives no guarantee that these data will be positive (the Gaussian has an infinite support). Then one could think of basing the model on some distribution that insure generated data to be positive such as the Beta or the Dirichlet distribution.
However, it is always a good start to take the Gaussian as an assumption, get some results and if they are not good enough, try other assumptions and compare.
This can enhance the accuracy but can also require a big amount of work as most classical machine learning algorithms are often not implemented in main libraries for assumptions other than the Gaussian.
To summarize:
- However, what if the data that I am dealing with is not normally distributed?
Then you should see an improvement of your results when using more appropriate distributions.
- Should I perform log/ exponential transformation on the data for the sake of getting normal distributed data?
It can indeed be worth it (and little work) to transform the data and see the impact on the results.
- Why is Gaussian data always best suited?
It is not. As a proof, you can look at several publications about mixture models that compare for different data sets how Dirichlet, Gaussian, Beta mixtures models (and other) behave.
