1
$\begingroup$

Given A, B and C are matrices with dim(A) = m x n, dim(B) = n x p and dim (C) = m x p, the problem asks to evaluate

I need to learn $$\tilde{A}$$ such that $$\min_{\tilde{A}}||\tilde{A}^TB-C||$$

and $$\min_{\tilde{A}}||\tilde{A}-A||$$

$\endgroup$
1
$\begingroup$

In general, you can't. You can find a matrix that will minimize $||\tilde{A}^TB-C||$, and another that minimizes $||\tilde{A}-A||$, but, in general, the two matrices you find won't be the same. In fact, the matrix that solves the second problem is always $\tilde{A} = A$, and, if $||{A}^TB-C||$ is not the minimum of the first function, then your problem doesn't have a solution.

However, what would a machine learning practicioner do? Instead of finding a matrix that solves both of the problems (that might not exist), you can aim to solve a combination of the two problems. You can aim to find the matrix that minimizes $$||\tilde{A}^TB-C||^2 + ||\tilde{A}-A||^2 $$ or, in general, the matrix that minimizes $$||\tilde{A}^TB-C||^2 + \lambda ||\tilde{A}-A||^2 $$ for some $\lambda >0$. If $\lambda \approx 0$, then you are just solving the first problem, and if $\lambda >>0$, then you are just solving the second one.

And, this is just a quadratic optimization problem, that can be solved using gradient descent and will provide a unique global maximum.

$\endgroup$
  • $\begingroup$ one doubt, why is it required to make it a quadratic optimization problem.? ||A^~TB−C||+lambda||A^~−A|| is insufficient? $\endgroup$ – Prakhar Agarwal Jun 18 '18 at 13:49
  • 1
    $\begingroup$ It is sufficient, but it is harder. If you use the $L^2$ norm, then solving a quadratic problem with a gradient method is very easy. If you don't use the $L^2$ norm, or you use the formula that you have given, then it becomes hell. I don't even know if there are global minima, and optimizing square roots is always difficult. $\endgroup$ – David Masip Jun 18 '18 at 13:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.