Typical algorithms involve learning and applying a single mapping e.g. $ f: X \mapsto Y$

Are there any algorithms that learn multiple mappings given an extra variable e.g. $ f(Z): X \mapsto Y$ (This feels like an abuse of notation.).

Here we learn a mapping, given $Z$, that is then applied to $X$ to produce $Y$

One solution (although intractable) would be to train an algorithm (e.g. a neural network) whose output is then used as the weights of a second network.

Any suggestions would be much appreciated and feel free to ask for more details. Thanks.

  • 3
    $\begingroup$ Yes. This is the fundamental problem machine learning addresses. Welcome to the site. $\endgroup$ – Emre Mar 7 '18 at 0:27
  • $\begingroup$ @Emre Do typical models not just map input directly to output hence learning one mapping? $\endgroup$ – Daniel Mar 7 '18 at 0:32
  • $\begingroup$ NO, They Learn and generalize themselves, That's why we use CNN (for e.g.) $\endgroup$ – Aditya Mar 7 '18 at 0:35
  • $\begingroup$ @Aditya Sorry for being unclear. By 'map input directly to output' I mean when we train a CNN (for e.g.) we apply this network (or large complicated function) to every input the same way. $\endgroup$ – Daniel Mar 7 '18 at 0:38
  • $\begingroup$ That is not a problem; Z is just another variable. You will find many examples by searching for conditional or contextual models. $\endgroup$ – Emre Mar 7 '18 at 0:38

This is equivalent to thinking of this as learning a function of two inputs, i.e., $g : (X,Z) \mapsto Y$ (i.e., $g(X,Z)=Y$). Any machine learning method for learning functions can be used for that; you just treat the feature vector (the input) as the concatenation of the vector $X$ and the vector $Z$, and then use standard ML techniques on the resulting instances. So, you don't need anything special or fancy to handle this -- you can use any existing machine learning method that you are comfortable with.

| improve this answer | |

If $Z$ denotes a set of attributes such that for any $Z$, $f(Z)$ defines a family of functions, then you've basically stumbled on to hyperparameter search. As an example, consider a neural network with parameters $\theta$. We can reformulate your mapping $ f: X \mapsto Y$ as trying to learn the function $f(X;\theta) = Y$. But we have a lot of structural decisions to make here. What's our cost function? How many layers? What dimensions? What types of layers and activations? Do we want to have recurrence or convultions? Dropout? Normalization? Regularization? What learning rate? Should we use momentum? An annealing schedule? How many epochs? How much coffee should I drink before I start coding?

If we wrap all of those structural decisions into $Z$, then $f(Z)$ is a specification for a family of functions for whom we are trying to learn the parameters $\theta$ that minimizes error.

Hyperparameter search is a big problem and there's a lot of interesting research into the topic. I think the current SOTA is bayesian optimization with bandit algorithms, although simple approaches like grid search, random search, and even guess-and-check are probably the most common.

| improve this answer | |
  • 1
    $\begingroup$ Z is not necessarily a hyperparameter; that is true if Z can not be incorporated efficiently. Z can also be a regular parameter. $\endgroup$ – Emre Mar 7 '18 at 17:39
  • $\begingroup$ For the purposes of my response, I literally defined $Z$ to be the collection of hyperparameters, explicitly to distinguish it from the trivial responses OP was getting in comments from yourself and others. You're welcome to approach the question from a different angle yourself, but in the context of my answer: no. $Z$ is necessarily constrained to hyperparameters because that's how I defined it. $\endgroup$ – David Marx Mar 8 '18 at 18:15
  • $\begingroup$ It is misleading to suggest that OP's Z has to be treated as a hyperparameter. If that is not what you mean, I suggest you rephrase your answer because that's how it reads to me. $\endgroup$ – Emre Mar 8 '18 at 18:18
  • $\begingroup$ I never said OP's $Z$ did, I said mine did. OP's question was ambiguous, so I constrained attention to a particular interpretation for my answer. The first sentence of my response is an explicit reframing of the question to concretize it and permit my response. Your comment to OP wasn't technically wrong, but it's also not helpful. By narrowing the scope of the question, I was able to provide an answer that I felt was more in line with what OP was trying to ask. They accepted it, so I think they appreciated my approach. Sometimes you have to read between the lines around here. $\endgroup$ – David Marx Mar 8 '18 at 19:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.