1
$\begingroup$

We have two time series columns - column A is the reference column ( source of truth) and column B is a ''cousin'' of column A, in the sense that it exhibits ( or should exhibit) the same patterns, evolution, rates of changes etc, as column A.

However, at certain periods column B values for wtv reason start exhibiting abnormal values which we are comfortable to classify as outliers. We are happy to remove these value as outliers.

Now given the unequal timeseries ( post outlier removal of column B) would like to explore machine learning techniques that would impute data in column B, using column A as the reference or source of truth, effectivly replicating the behavior of column A to impute missing data in column B.

I've researched many models to use, regression, LTSM etc. hoping for some expertise that would help shed light on what technique would best work for the problem statement at hand.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

If column B truly exhibits the same patterns, evolution, and rate-of-change as column A, then that means the values of column B are just a translation of A. That is, $B_i = A_i + c$, where $c$ is some constant.

If this is the case, then you don't need machine learning to impute values. Knowing $A$ is just as good as knowing $B$. You just need to estimate $c$. One good way to estimate $c$ would be to compute the mean difference between $B$ and $A$.

On the other hand, if B is related to A, but does not exhibit the exact same patterns and rate-of-change, then machine learning might be a little more appropriate. You'll want to train a regression model that predicts B as a function of A, then use the model to fill in missing values of B.

Given the degree of coupling you describe between A and B, linear regression is probably a good place to start. You can experiment with higher-order polynomials (quadratic, cubic, etc) if linear regression doesn't get the job done. An easy way to get started would be to experiment with polynomial regression in Excel or Python.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thank you zachdj. I actually tried the linear regression. I removed the periods of outliers then regressed the two TS and used the coefficeints to fill in the missing values however while it reduced the divergence beween the outlier data point and the column A point, it still had a wide diversion between the two. Which makes sense given that with outliers removed from column B, the two TS had similar although not exact patterns. Will attempt higher order polynomials are suggested. $\endgroup$
    – Tyler
    Sep 5, 2019 at 18:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.