# Paramaeter estimation in noisy conditions with Machine Learning, possible?

Let's take two constants, $$\alpha$$ and $$\beta$$, both are given by two functions $$f_1(\vec{\theta})$$ and $$f_2(\vec\theta$$) (the model). These functions are known: we have an analytical closed expression, that for a given set of values $$\vec\theta$$, $$\alpha$$ and $$\beta$$ are retrieved.

PROBLEM:

The components of $$\vec\theta$$ are noisy measurements, so, the variance of $$\alpha$$ and $$\beta$$ is high: for different measurements, we have quite different values for them... though our model defines them as constants. Furthermore, the model is not actually realistic, and obviously it imposes some assumptions.

formulation:

The observed values are $$\vec\theta = [\beta, \dot\psi, \delta, A, B, C]$$

$$\begin{equation} \alpha = \frac{\dot\beta + \dot\psi}{-2A\beta + B\dot\psi A\delta} \end{equation}$$ $$\begin{equation} \beta = \frac{\alpha K\beta + \alpha C\dot\psi + \alpha D\delta}{\ddot\psi} \end{equation}$$

(The derivatives are computed using finite differences)

NOTE: the observed values of $$\vec\theta$$ are not independent since they come from the same sensing unit (which little do we know about), so I think assuming statistical independence won't work here...

QUESTIONS:

Are there common approaches in Machine Learning to mitigate the effect of the noise? Can Machine Learning help to get the true values? Can Machine Learning model the variance of the estimators? By Machine Learning I mean algorithms that are not as known as Kalman filtering.

MOTIVATION AND CONSIDERATIONS:

I was asked this question by a colleague at my job office, since I work on Machine Learning. I personally think that a better approach is to face it from the estimation theory (ref.). For instance, modeling the noise of the sensors and try to get Maximum Likelihood estimators, and if possible, derive a bound for their variance (CRLB) or a closed expression for the bias. For handling the noise, I suggested u-Kalman, Particle Filtering... But other terms that came to my mind were: "fuzzy", "GMM", "EM"...

EDIT:

I think what my colleague wants is a "data-driven/black-box" approach that can help him skip the math, since he has not the enough background...

• Can you further explain what you mean by "mitigate the effect of the noise"? Do you want to do some inference or something? – Romain Reboulleau Dec 11 '18 at 13:01
• For example, the simplest way that comes to my mind is to average measurements... but it is not very machine-learning-esque... – ignatius Dec 11 '18 at 13:19
• OK I think I see what you mean. I will try an answer, let me know what you think. – Romain Reboulleau Dec 11 '18 at 16:12

Statistical approach

This question is more related to statistics than data science, perharps you would have better answers on Cross Validated. As you suggested in the question, a purely statistical approach should work for sure (though it requires some assumptions).

As i understand your problem, your prior knowledge is that $$\alpha$$ (or $$\beta$$) should be always the same, but you get noise on the realizations of $$\alpha$$, and you suppose this noise comes from the input ($$\theta$$), not the process ($$f_1$$).

Let us assume that the true $$\alpha$$ corresponds to the mean value of $$\theta$$, i.e. we have $$\alpha_{real} = f_1(\bar{\theta})$$. Just taking $$\bar{\alpha}$$ is generally wrong as you are assimilating $$f_1(\bar{\theta})$$ with $$\overline{f_1(\theta)}$$: those are not the same if the problem is not linear.

I suppose you have no idea what the distribution of $$\theta$$ looks like. But the good news is that you know the $$f_1$$ and $$f_2$$ processes, so you can infer the distribution parameters from your data (by maximum likelihood estimation). For instance, you can assume $$\theta \sim \mathcal{N}(\mu, \sigma^2)$$ (or whatever distribution seems likely), and determine which $$\mu$$ and $$\sigma$$ explain the realizations of $$\alpha$$ and $$\beta$$ best. Having two different outputs is comfortable, this makes the approach more robust.

Then, your true $$\alpha$$ and $$\beta$$ would simply be $$f_1(\mu)$$ and $$f_2(\mu)$$.

An incomplete "data-driven" idea

If we make the assumption that $$\theta$$ is evenly distributed and you have enough samples, its median is the same as the mean. So basically if we find the median, we get a good estimator of the true $$\alpha$$ and $$\beta$$ values (again assuming that the true value is the one at the mean of $$\theta$$).

If either $$f_1$$ or $$f_2$$ process is monotonous, this is straightforward, because you just have to sort $$\alpha$$ or $$\beta$$ realizations. In the same way, if one process is easily invertible (analytically or numerically), the problem becomes almost trivial. If both are highly non-monotonous and non-invertible, you will need to find a way to sort them in a machine-learning fashion. In the following, I will assume that both processes are at least continuous.

First option: try to perform a binary clustering of your data ($$\alpha$$, $$\beta$$). You will need to use the kernel trick, and this will only work if:

• you have enough samples to map your processes in a continuous way
• there is no "crossing" zone if both processes, i.e. $$\forall \theta, \theta^*, [f_1(\theta^*) = f_1(\theta)\ and\ f_2(\theta^*) = f_2(\theta)] \iff \theta = \theta^*$$

This may work if your processes are not too "complex".

Second option (which should work better on complex processes): start with the samples in a random order, and compute the euclidian distance (or any other relevant distance) from one sample to the next one, then sum: this gives you the total distance of the pathway going through all samples. Then, use an optimizing procedure (simulated annealing for instance) interverting the index of two samples, until you find an order that minimizes this metric.

This second option is basically solving the travelling salesman problem.

On both cases, I cannot guarantee that it will work, but I think it's worth trying, just for the fun of using standard methods that are not at all suited to the initial problem!

• Thank you for your response. What you mention is the statistical approach, which I have worked with. I have worked with ML estimation, CRLB, fisher information matrices and so on... so, I suggested the same to my colleague: try to characterize the noise and go for ML estimation, or reconsider if the model does not fit your data. I was asking for another approaches... if they exist.. – ignatius Dec 11 '18 at 16:42
• OK, I think I get the idea now. Check out my second proposition in the edit. If I have more time I'll try to explain the idea further, later today or tomorrow. – Romain Reboulleau Dec 11 '18 at 17:08
• OK, I'll take my time to read your second approach. I think I can also give more details about the model and the data. Thank you so much! – ignatius Dec 11 '18 at 17:12
• I have added the mathematical formulation of the problem. I think that trying to get a ML estimator analytically will be quite difficult since all the observed values $\theta$ are not independent, they come from a sensing unit which is a black-box for us.... so trying to make assumptions about its pdf will be very challenging. For your "data-driven" approach, Iam sorry but i don't understand the idea of sorting the samples... – ignatius Dec 12 '18 at 9:24
• It's because I thought theta was just a scalar value! With a multi-dimensional output it is much more complicated. I'll think about it. – Romain Reboulleau Dec 12 '18 at 12:03