# How to smooth noisy data set?

I have a univariate dataset that is locally jagged (lots of ups and downs) that I need to smooth. I've also tried polynomial features with linear regression, but because the curve is extremely non-linear, I couldn't get the curve to fit properly no matter how I raise the degree. I'm basically looking to overfit the model, something like an exponential moving averages but without the lag.

Random forest regressors is a little too jagged; I've looked at splines from scipy but they just give back my original points since I'm not looking to interpolate. Is there anything else I could try?

• Overfitting is kind of the opposite of smoothing: just play connect-the dots. Thus, what is your goal? Why doesn’t a low-degree polynomial model work? Why not a simple linear regression? // You can’t smooth it without losing some fit. Indeed, that is kind of the point of smoothing.
– Dave
Oct 29, 2022 at 18:28
• @Dave I'm not sure a polynomial would work -- the shape is highly irregular. imagine the contour of a person's face and neck lying down (so that it passes the vertical line test), but instead of a sequence of points creating straight lines, we have points such that if connected, resemble saw tooth with no discernible frequency for fourier transform Oct 30, 2022 at 4:24

Kalman filters are very interesting as it fits any kind of curve and you can fine-tune them to adjust the right noise reduction.

import numpy as np
import pylab as pl
from pykalman import KalmanFilter

#Some data with noise
rnd = np.random.RandomState(0)
n_timesteps = 100
x = np.linspace(0, 3 * np.pi, n_timesteps)
y = 20 * (np.sin(x) + 0.2 * rnd.randn(n_timesteps))
pl.figure(figsize=(16, 6))
pl.scatter(x,y, marker='x', color='b', label='observations')
pl.xlabel('time')
pl.ylabel('Position')
pl.show() The noise reduction is defined by the covariance and the transition matrix. A 1D matrix would be a linear simplification and would fit for simple shapes, whereas a 2D matrix would be for non linear simplifaction for more complex shapes.

#Here you define the smoothing strength
#High smoothing in green
kf_high = KalmanFilter(transition_matrices=np.array(),
transition_covariance=0.06)

#Low smoothing in blue
kf_low = KalmanFilter(transition_matrices=np.array([[1, 1], [0, 1]]),
transition_covariance=0.02 * np.eye(2))

states_pred_high = kf_high.em(y).smooth(y)

states_pred_low = kf_low.em(y).smooth(y)

pl.figure(figsize=(16, 6))
position_line = pl.plot(x, states_pred_high[:, 0],markersize=2,
linestyle='-', marker='o', color='b',
label='position est.')

obs_scatter = pl.scatter(x, y, marker='x', color='b',
label='observations')
position_line = pl.plot(x, states_pred_low[:, 0],markersize=2,
linestyle='-', marker='o', color='g',
label='position est.')

pl.legend(loc='lower left',fontsize=14)
pl.xlim(xmin=0, xmax=x.max())
pl.xlabel('time')
pl.show() • @ron-burgundy Does it answer your question? If not, please let me know. Nov 24, 2022 at 21:25