3
$\begingroup$

I'm playing around with deep neural networks for a regression problem. The dataset I have is skewed right and for a linear regression model, I would typically perform a log transform. Should I be applying the same practice to a DNN?

Specifically, I'm curious how skewed data affects regression with a DNN and, if it's negatively, are the same methods that would be applied to a linear regression model the right way to go about fixing it? I couldn't find any research articles about it but if you know of any feel free to link them in your answer!

Improve this question
$\endgroup$
2
  • $\begingroup$ When you say you have skewed data, what specifically do you mean? Skewed distributions of predictor variables? Skewed distribution of the response variable? Neither of those even matter to linear regression, where if there is a normality assumption (we get Gauss-Markov without such an assumption), it is about the error term, not about the data. I get the feeling that you're using the $\log$ transform when you don't need to in linear regression. $\endgroup$
    – Dave
    Commented Nov 9, 2020 at 17:36
  • $\begingroup$ The context I had in mind was one skewed predictor variable in multivariate regression with the rest of the predictors being normally distributed. $\endgroup$
    – shaye059
    Commented Nov 10, 2020 at 18:33

1 Answer 1

Reset to default
2
$\begingroup$

Strictly theoretically it makes no difference on DNN, I answered it today here and I said:

Here is why: We already know mathematically that NN can approximate any function. So lets say that we have Input X. X is highly correlated, than we can apply a decorrelation technique out there. Main Thing is, you get X` that has different numerical representation. Most likely more difficult for NN to learn to map to Outputs y. But still in Theory you can Change the architecure, Train for longer and you can still get the same Approximation, i.e. Accuracy.

Now, Theory and Praxis are same in Theory but different in Praxis, and I suspect that this Adjustments of Architecture etc will be much more costly in reality depending on the dataset.

BUT I want to add another point of view: Convergence speed. Strickly theoretically you dont even need [batch normalization] for performance (you can just adjust weights and bias and you should get same results) but we know that making this transformation has big benefits for NN

To conclude for you: Yeah, I had experience where it made difference, and where it didnt. You cant expect theoretical results that say skewed is bad

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.