we are designing a rules based engine to check the quality of the data before training our ML models. The data we have is time series data. We have about 3-4 features using which we have to make a prediction. We have about over 10000 securities for which we have to perform this data quality check. Many of these securities are missing one or more feature columns completely and for some of them we have many missing values. Currently I am generating a report of missing values, Shapiro-Wilk test for normality and a quartile based outlier analysis. Are there any other tests we should be performing along with these or it instead of these?



  • 3
    $\begingroup$ What is your goal in applying Shapiro-Wilk? There are a lot of misconceptions about normality requirements in regressions, forAmal normality testing is mostly worthless, and the reasons for wanting normality (when and where we do) often are not very important in machine learning. $\endgroup$ – Dave Jul 9 at 23:11
  • 1
    $\begingroup$ Please add more to your Features/response detail, end objective in the question $\endgroup$ – 10xAI Jul 10 at 6:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.