Paramaeter estimation in noisy conditions with Machine Learning, possible?

Question

Let's take two constants, $\alpha$ and $\beta$, both are given by two functions $f_1(\vec{\theta})$ and $f_2(\vec\theta$) (the model). These functions are known: we have an analytical closed expression, that for a given set of values $\vec\theta$, $\alpha$ and $\beta$ are retrieved.

PROBLEM:

The components of $\vec\theta$ are noisy measurements, so, the variance of $\alpha$ and $\beta$ is high: for different measurements, we have quite different values for them... though our model defines them as constants. Furthermore, the model is not actually realistic, and obviously it imposes some assumptions.

formulation:

The observed values are $\vec\theta = [\beta, \dot\psi, \delta, A, B, C]$

\begin{equation} \alpha = \frac{\dot\beta + \dot\psi}{-2A\beta + B\dot\psi A\delta} \end{equation} \begin{equation} \beta = \frac{\alpha K\beta + \alpha C\dot\psi + \alpha D\delta}{\ddot\psi} \end{equation}

(The derivatives are computed using finite differences)

NOTE: the observed values of $\vec\theta$ are not independent since they come from the same sensing unit (which little do we know about), so I think assuming statistical independence won't work here...

QUESTIONS:

Are there common approaches in Machine Learning to mitigate the effect of the noise? Can Machine Learning help to get the true values? Can Machine Learning model the variance of the estimators? By Machine Learning I mean algorithms that are not as known as Kalman filtering.

MOTIVATION AND CONSIDERATIONS:

I was asked this question by a colleague at my job office, since I work on Machine Learning. I personally think that a better approach is to face it from the estimation theory (ref.). For instance, modeling the noise of the sensors and try to get Maximum Likelihood estimators, and if possible, derive a bound for their variance (CRLB) or a closed expression for the bias. For handling the noise, I suggested u-Kalman, Particle Filtering... But other terms that came to my mind were: "fuzzy", "GMM", "EM"...

EDIT:

I think what my colleague wants is a "data-driven/black-box" approach that can help him skip the math, since he has not the enough background...

Answer 1

Statistical approach

This question is more related to statistics than data science, perharps you would have better answers on Cross Validated. As you suggested in the question, a purely statistical approach should work for sure (though it requires some assumptions).

As i understand your problem, your prior knowledge is that $\alpha$ (or $\beta$) should be always the same, but you get noise on the realizations of $\alpha$, and you suppose this noise comes from the input ($\theta$), not the process ($f_1$).

Let us assume that the true $\alpha$ corresponds to the mean value of $\theta$, i.e. we have $\alpha_{real} = f_1(\bar{\theta})$. Just taking $\bar{\alpha}$ is generally wrong as you are assimilating $f_1(\bar{\theta})$ with $\overline{f_1(\theta)}$: those are not the same if the problem is not linear.

I suppose you have no idea what the distribution of $\theta$ looks like. But the good news is that you know the $f_1$ and $f_2$ processes, so you can infer the distribution parameters from your data (by maximum likelihood estimation). For instance, you can assume $\theta \sim \mathcal{N}(\mu, \sigma^2)$ (or whatever distribution seems likely), and determine which $\mu$ and $\sigma$ explain the realizations of $\alpha$ and $\beta$ best. Having two different outputs is comfortable, this makes the approach more robust.

Then, your true $\alpha$ and $\beta$ would simply be $f_1(\mu)$ and $f_2(\mu)$.

---

An incomplete "data-driven" idea

If we make the assumption that $\theta$ is evenly distributed and you have enough samples, its median is the same as the mean. So basically if we find the median, we get a good estimator of the true $\alpha$ and $\beta$ values (again assuming that the true value is the one at the mean of $\theta$).

If either $f_1$ or $f_2$ process is monotonous, this is straightforward, because you just have to sort $\alpha$ or $\beta$ realizations. In the same way, if one process is easily invertible (analytically or numerically), the problem becomes almost trivial. If both are highly non-monotonous and non-invertible, you will need to find a way to sort them in a machine-learning fashion. In the following, I will assume that both processes are at least continuous.

First option: try to perform a binary clustering of your data ($\alpha$, $\beta$). You will need to use the kernel trick, and this will only work if:

• you have enough samples to map your processes in a continuous way
• there is no "crossing" zone if both processes, i.e. $\forall \theta, \theta^*, [f_1(\theta^*) = f_1(\theta)\ and\ f_2(\theta^*) = f_2(\theta)] \iff \theta = \theta^*$

This may work if your processes are not too "complex".

Second option (which should work better on complex processes): start with the samples in a random order, and compute the euclidian distance (or any other relevant distance) from one sample to the next one, then sum: this gives you the total distance of the pathway going through all samples. Then, use an optimizing procedure (simulated annealing for instance) interverting the index of two samples, until you find an order that minimizes this metric.

This second option is basically solving the travelling salesman problem.

On both cases, I cannot guarantee that it will work, but I think it's worth trying, just for the fun of using standard methods that are not at all suited to the initial problem!