ML regression for skewed data

Question

I'm trying to build a simple regression model to start with but my Y variable is very skewed to the right. My Y represents the number of views per day for a webpage and all the values are above 0. I have 4 big outliers with values over 1000 but there are still many with values over 50 that are skewing the distribution. I'm not quite sure how to approach this, should I remove all the outliers (a lot of data), do some normalization and use linear regression to start with? Should I not normalize but use some other kind of regression like Tobit for skewed data (I haven't seen it much used in machine learning but I'm a novice). I'm a bit reluctatnt to remove data with many views per day because it may carry some useful information... Any guidance will be appreciated, I'm attaching a histogram where I already removed a lot of data on the right sight because otherwise you can't really see the distribution. My predictor variable will be text (probably treated with BERT embeddings).

EDIT: I'm also attaching a histogram I got after removing a big chunk of data on the right and using the squaring transformation. It's still pretty skewed so I guess one of the non-linear models is the way to go:

no_outliers_df = data[data['views_per_day'] < 50]
plt.hist(np.sqrt(no_outliers_df['views_per_day']))
plt.show()

Answer 1

There are several ways to handle skewed data in regression analysis:

Log Transformation: This is a common method to handle right-skewed data. It can help to normalize the data and reduce the effect of outliers. In your case, you can apply log transformation to your Y variable (number of views per day). However, keep in mind that the interpretation of the coefficients will be in terms of percentage change rather than absolute change.

Square Root or Cube Root Transformation: These are other types of transformations that can help to reduce skewness. The square root transformation is particularly useful when dealing with counts, like your number of views per day.

Box-Cox Transformation: This is a more general form of transformation that includes log, square root, and reciprocal transformations as special cases. It finds the best power transformation of the data that reduces skewness to a target value, usually zero for normalization.

Non-linear Models: If the relationship between your predictors and the response variable is not linear, you might want to consider non-linear regression models or other machine learning models that can handle non-linearity.

Quantile Regression: This type of regression does not make any assumptions about the distribution of the error term, so it can be a good choice when dealing with skewed data. It predicts the median (or other quantiles) of the response variable, rather than the mean.

Robust Regression: This type of regression is less sensitive to outliers in the data. It uses a different loss function (like Huber loss or Tukey's bisquare loss), which gives less weight to outliers.

Tobit Model: This is a type of regression model that is used when the response variable is censored, i.e., it has a lower or upper limit. In your case, since the number of views per day cannot be less than zero, a Tobit model might be appropriate. However, Tobit models are more commonly used in econometrics than in machine learning.

In terms of dealing with outliers, it's important to understand why they are present in your data. If they are due to errors or anomalies, it might be appropriate to remove them. However, if they represent legitimate observations, they might contain valuable information and should not be removed. In this case, using a method that is robust to outliers, like robust regression or log transformation, might be a better approach.

Finally, when dealing with text data, you might want to consider using a method that can handle high-dimensional, sparse data, like regularized regression (Lasso or Ridge) or a tree-based method (Random Forest or Gradient Boosting). These methods can also handle non-linearity and interactions between predictors, which can be useful when dealing with text data.

Answer 2

A little bit of skewness is ok. In real world, the data is not always normally distributed. All you can do is check if you have any outliers by performing a quartile test or finding the z score. Choose a range of z scores and see what you can achieve. Additionally, model performance can depend not only on one feature but others too, so make sure to check for the distribution of other features from the dataset. Once you have an idea of the data distribution of entire features, you can then decide if you want to standardize or normalize your data.