0
$\begingroup$

There are many ML techniques to estimate latent variables such as the EM algorithm. Is there a technique that allows for thresholds for each of the latent variables?

I have a feature space with 10 variables $(X_1,\dots,X_{10})$ and the outcome $Y$. 7 of the $X$ features are known (I have their observations) and 3 are unknown. Each of the unknown can be within a range from 0 up to a positive constant number.

What ML technique would you recommend for estimating the above latent variables with the setup described above?

$\endgroup$
  • $\begingroup$ Please clarify "allows for thresholds". For example do you mean the latent variables are discrete values lying on a continuum or that they are disjoint categories. For the latter you'd usually encode as "one-hot" for the former you could relax to continuous range. $\endgroup$ – jayprich Jun 28 '18 at 14:02
  • $\begingroup$ My miss. I mean that a variable can take any real value between 0 and 10 for instance. Or between -3 and 20. $\endgroup$ – mrt Jun 28 '18 at 15:04
0
$\begingroup$

re. "estimate latent variables"

Quantities that are trained in order to fit a "best" model within a family of models are called hyper-parameters. To any instance of the model they are fixed. To the optimisation routine they are an index into search space. Adding constraints on the range of a hyper-parameter both reduces the search space of the optimisation and requires extra "feasibility" checks during typical gradient descent.

A variable is "latent" when it is purely internal to the model, i.e. not an observable. The meaning of its scale would depend on the context and on your interpretation, since it cannot be compared to anything observed. You rarely want to constrain that range inside the model.

I would suggest leaving the hyper-parameters and latent variables unconstrained and if you want to read an output train a "neuron"-like response to get what you want out : e.g. sigmoid / tanh / softmax

$\endgroup$
  • $\begingroup$ Do you see this as an application to NNs? My (limited) experience with NNs tells me that the latent variables and the parameters will be lost in the complexity of the network with the multiple layers and neurons. $\endgroup$ – mrt Jul 2 '18 at 15:01
  • $\begingroup$ The learned connection strengths, biases and what have you are the "parameter" .. the conventional train/predict scenario they do not change once found. Online learning that adapts to new data clearly blurs that line. A "latent variable" in a NN is any state information that's not an input or an output .. it changes all the time in response to neighbours etc .. nothing is "lost" it's just not clear to me how you think the word "latent" applies to your stated scenario .. seems you're really talking about constraining an input or an output .. but perhaps I misunderstand $\endgroup$ – jayprich Jul 3 '18 at 16:40
  • $\begingroup$ In RNN's i.e. $x_{t+1}= f(A x_t+B u_t)$ and $y_t=C x_t$ the $x_t$ is what I'd call a latent variable, not the A,B and C matrix. This is based on the State Space representation where the state vector $x_t$ may contain un-observable (latent) states. In the SS case when the latent states have some physical meaning - i.e. acceleration or temperature it would make sense to bound them. $\endgroup$ – David Waterworth 15 hours ago
0
$\begingroup$

Sure. Just treat the range as a prior on the latent variables. Typically we use a boring prior (e.g., a normal distribution, a uniform distribution), but in your case, if $X_7$ is unknown and in the range $[0, 7.3]$, then your prior for $X_7$ could be the uniform distribution on that range. Then apply the machinery of the EM algorithm, and it should all work.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.