I am currently working on a project involving time series banking stock price data. I have around 3000 observations, some columns have a lot of missing values (null value); they can account for 5 to 50% of the total observations. I have no idea what is the proper order for handling missing values, outliers and take log transformation of the data. Should I impute the missing values first and take log transformation or vice versa. Thank you so much
$\begingroup$
$\endgroup$
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
4
In general, it is better to deal with missing values first because there could be data loss or additional noise applying operations like a log that could impact classification or prediction algorithms.
To deal with missing values, you can use regressors to have good results but it depends on the data quality.
It could be done using algorithms such as Random Forest, XGBoost or Deep Neural Networks.
Note: you can measure the model quality by hiding some known values and see if they are well predicted.
See also:
answered Aug 29, 2022 at 14:49
-
1
-
$\begingroup$ Thanks for your answer and the reference. It is really helpful. When I searched for ways to impute with missing values in time series, I found some approaches like LOCF or interpolation. Which methods do you think may be better for my stock data, LOCF, interpolation or machine learning methods like Random Forest, XGBoost. Thanks a lot $\endgroup$ Aug 30, 2022 at 12:18
-
$\begingroup$ You're welcome Minh. I don't know the data you are dealing with but stock data is generally complex, and used to have better results with non linear algorithm such as the ones I've mentioned. In all cases, the choice depends on the kind of correlations between features, but those correlations are often unknown. Furthermore, stock values have noise. Consequently, I would recommend a robust algorithm like Random Forest or XGBoost to start with, and then compare it with another algorithm to see which one is the best. $\endgroup$ Aug 30, 2022 at 14:46
-
$\begingroup$ Your answer is really informative, which indeed helps me a lot. Thanks for your help. $\endgroup$ Aug 31, 2022 at 2:43
$\begingroup$
$\endgroup$
I suggest imputing the missing values and converting any columns to numeric data types as your first steps. Then you can deal with outliers and make transformations.
Most machine learning packages (e.g. scikit-learn) will generally require you to have all numeric data before feeding the data to machine learning algorithms.
As you go, just make sure you document what you're doing so that other people can follow and share you code so you can reproduce your work.
