0
$\begingroup$

I have a univariate dataset that is locally jagged (lots of ups and downs) that I need to smooth. I've also tried polynomial features with linear regression, but because the curve is extremely non-linear, I couldn't get the curve to fit properly no matter how I raise the degree. I'm basically looking to overfit the model, something like an exponential moving averages but without the lag.

Random forest regressors is a little too jagged; I've looked at splines from scipy but they just give back my original points since I'm not looking to interpolate. Is there anything else I could try?

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ Overfitting is kind of the opposite of smoothing: just play connect-the dots. Thus, what is your goal? Why doesn’t a low-degree polynomial model work? Why not a simple linear regression? // You can’t smooth it without losing some fit. Indeed, that is kind of the point of smoothing. $\endgroup$
    – Dave
    Oct 29, 2022 at 18:28
  • $\begingroup$ @Dave I'm not sure a polynomial would work -- the shape is highly irregular. imagine the contour of a person's face and neck lying down (so that it passes the vertical line test), but instead of a sequence of points creating straight lines, we have points such that if connected, resemble saw tooth with no discernible frequency for fourier transform $\endgroup$
    – ron burgundy
    Oct 30, 2022 at 4:24

1 Answer 1

Reset to default
2
$\begingroup$

Kalman filters are very interesting as it fits any kind of curve and you can fine-tune them to adjust the right noise reduction.

import numpy as np
import pylab as pl
from pykalman import KalmanFilter

#Some data with noise    
rnd = np.random.RandomState(0)
n_timesteps = 100
x = np.linspace(0, 3 * np.pi, n_timesteps)
y = 20 * (np.sin(x) + 0.2 * rnd.randn(n_timesteps))
pl.figure(figsize=(16, 6))
pl.scatter(x,y, marker='x', color='b', label='observations')
pl.xlabel('time')
pl.ylabel('Position')
pl.show()

enter image description here

The noise reduction is defined by the covariance and the transition matrix. A 1D matrix would be a linear simplification and would fit for simple shapes, whereas a 2D matrix would be for non linear simplifaction for more complex shapes.

#Here you define the smoothing strength
#High smoothing in green
kf_high = KalmanFilter(transition_matrices=np.array([1]),
                  transition_covariance=0.06)

#Low smoothing in blue
kf_low = KalmanFilter(transition_matrices=np.array([[1, 1], [0, 1]]),
                  transition_covariance=0.02 * np.eye(2))


states_pred_high = kf_high.em(y).smooth(y)[0]

states_pred_low = kf_low.em(y).smooth(y)[0]



pl.figure(figsize=(16, 6))
position_line = pl.plot(x, states_pred_high[:, 0],markersize=2,
                        linestyle='-', marker='o', color='b',
                        label='position est.')

obs_scatter = pl.scatter(x, y, marker='x', color='b',
                         label='observations')
position_line = pl.plot(x, states_pred_low[:, 0],markersize=2,
                        linestyle='-', marker='o', color='g',
                        label='position est.')



pl.legend(loc='lower left',fontsize=14)
pl.xlim(xmin=0, xmax=x.max())
pl.xlabel('time')
pl.show()

enter image description here

See: https://medium.com/dataman-in-ai/kalman-filter-explained-4d65b47916bf

Improve this answer
$\endgroup$
1
  • $\begingroup$ @ron-burgundy Does it answer your question? If not, please let me know. $\endgroup$
    – Nicolas Martin
    Nov 24, 2022 at 21:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.