Questions tagged [linear-algebra]
A field of mathematics concerned with the study of finite dimensional vector spaces, including matrices and their manipulation, which are important in statistics.
64
questions
13
votes
1answer
414 views
Finding linear transformation under which distance matrices are similar
I have $n$ sets of vectors, where each set $S_i$ contains $k$ vectors in $\mathbb{R}^d$. I know there is some unknown linear transformation $W$ under which the distance matrix $D_i$ (a $k\times k$ ...
12
votes
3answers
6k views
How does tensor product/multiplication work in TensorFlow?
In Tensorflow, I saw the following example:
...
8
votes
2answers
2k views
What is the use of additional column of 1s in normal equation?
Currently I am going through Normal Equation in Machine Learning.
$$
\hat\theta = (X^T \cdot X)^{-1} \cdot X^T \cdot y
$$
But when I see how they use this equation, I found they always add an ...
8
votes
1answer
1k views
Deriving backpropagation equations “natively” in tensor form
Image shows a typical layer somewhere in a feed forward network:
$a_i^{(k)}$ is the activation value of the $i^{th}$ neuron in the $k^{th}$ layer.
$W_{ij}^{(k)}$ is the weight connecting $i^{th}$ ...
6
votes
1answer
476 views
Closed form solution of linear regression via least squares using matrix derivatives
How is the closed form solution to linear regression derived using matrix derivatives as opposed to using the trace method as Andrew Ng does in his Machine learning lectures. Specifically, I am ...
4
votes
1answer
546 views
Mathematical formulation of Support Vector Machines?
I'm trying to learn maths behind SVM (hard margin) but due to different forms of mathematical formulations I'm bit confused.
Assume we have two sets of points $\text{(i.e. positives, negatives)}$ one ...
4
votes
2answers
59 views
Derivates with respect to a vector
Suppose I have an equation, $f = X^TY + \dots$ (a few more terms), where $X$ is a vector and $Y$ is a matrix of appropriate dimensions,
I want to know how can we take the derivative of $f \text{ w.r.t....
4
votes
2answers
625 views
How do we define a linearly separable problem?
When we talk about Perceptrons, we say that they are limited for approximating functions that are linearly separable, while Neural Networks that use non-linear transformations are not.
I am having ...
3
votes
1answer
150 views
RNN: why Wx + Uh instead of W[x,h]
Traditionally, a state for RNN is computed as
$$h_t = \sigma(W\cdot \vec x + U\cdot \vec h_{t-1} + \vec b)$$
For a RNN, why to add-up the terms $(Wx + Uh_{t-1})$ instead of just having a single ...
3
votes
1answer
130 views
PCA formulation - Deep Learning book by Ian Goodfellow
I am reading this deep learning book by Ian goodfellow. In the PCA formulation in the first chapter i.e Linear Algebra, he mentions the following:
we need to choose the encoding matrix D. To do so,...
3
votes
1answer
179 views
How can positional encodings including a sine operation be linearly transformable for any offset?
In the paper "Attention is all you need" the authors add a positional encoding to each token in the sequence (section 3.5). The following encoding is chosen:
$ PE(pos, 2dim) = sin(pos / 10000 ^ {2dim/...
3
votes
0answers
49 views
Intuition behind understanding eigenvectors and Machine Learning [closed]
I am struggling to understand linear algebra application in machine learning, and I am not able to answer the following question.
Is the model learned in Machine Learning the eigenvector of the ...
3
votes
0answers
21 views
Possible flaw in the MDS method for dimensionality reduction
The MDS (multidimensional scaling) method is used to solve the problem of dimensionality reduction. Basically, it does the following: given $n$ points $x_1,\cdots,x_n\in\mathbb R^d$, try to find a ...
3
votes
0answers
82 views
Best linear algebra library for C++?
I have been trying to find the substitute of numpy and perform some linear algebra using C++. Here's a list of the libraries I have encountered:
Eigen
Armadillo
Dlib
GNU Scientific library
Please ...
2
votes
4answers
131 views
How to solve Ax = b for A [closed]
Given two know vector x, and b (dimension 3*1 for example), what are the ways to approximate the matrix ...
2
votes
1answer
36 views
Need explanation of a matrix multiplication
I'm reading the Deep Learning book by MIT.
On the page 172, there's a part like this:
$$
f^{(1)}(x)=h=W^Tx \tag{1}
$$
$$
f^{(2)}(h)=h^Tw \tag{2}
$$
Substitute (1) into (2), they got:
$$
f(x)=w^TW^Tx
$...
2
votes
1answer
23 views
optimizing a linear optimization function with linear constarints and binary variables
I am new to optimizations and trying to solve a problem, which I feel falls in the umbrella of optimization.
I have an ojective function that needs to be maximized
...
2
votes
1answer
165 views
Why in this case are gradient steps not perpendicular to contour lines?
There is a theorem that gradient at point is perpendicular to tangent line to contour line at given point.
Why in this picture it seems that this rule is not respected?
source: http://www....
2
votes
1answer
466 views
Machine learning PhD Interview technical questions [closed]
I'm Software Engineer who applied to grad school for Machine Learning/Computer Vision PhD and currently waiting for interview calls. I'm brushing up Linear algebra/ ML topics. What kind of technical ...
2
votes
0answers
18 views
Deriving vectorized form of linear regression
We first have the weights of a D dimensional vector $w$ and a D dimensional predictor vector $x$, which are all indexed by $j$. There are $N$ observations, all D dimensional. $t$ is our targets, i.e, ...
2
votes
2answers
35 views
Backpropagation with a different sized training set?
I'm trying to create a NN whose input is a (length m) array of 3d vectors $$\vec{x}_i = [x_{i,1},x_{i,2},x_{i,3}], \hspace{5mm}i=1:m $$
and whose output is a similarly sized array:
$$\vec{h}_{\theta,...
2
votes
1answer
53 views
How to incorporate the uncertainty of the model coefficients in the prediction interval of a multiple linear regression
I'm dealing with the modeling of small experimental data sets. As most experimental work does not generate thousands of samples, but rather a handful, I need to be inventive in how to deal with this ...
2
votes
0answers
49 views
How does SVD actually provide the recommendations? I seem to get conflicting answers
I am reading a text book that basically says the following:
Given a matrix A where A is USERS x ITEMS we can use SVD to decompose the matrix into:
$$A = U \times \Sigma \times V^T$$
Then we can take ...
1
vote
1answer
429 views
Why does np.linalg.eig produce an opposite-signed eigenvector?
I am learning SVD by following this MIT course. In this video, the lecturer is finding the SVD for
$$
\begin{pmatrix} 5 & 5 \\ -1 & 7 \end{pmatrix},
$$
which involves finding the ...
1
vote
4answers
453 views
Statistics Before Linear Algebra?
I know this is an opinion-based question and will be closed but this is the only place I know that can answer it reasonably and it is a very important matter to me.
I am pursuing a machine ...
1
vote
2answers
188 views
How does “linear algebraic” weight training function work?
This answer shows that linear and polynomial function weights can be trained using this matrix operation:
$w = (X^TX)^{-1}X^Ty$
Therefore, algorithms such as gradient descent are not necessary for ...
1
vote
1answer
65 views
Confusion with Notation in the Book on Deep Learning by Ian Goodfellow et al
In chapter 6.1 on 'Example: Learning XOR', the bottom of page 168 mentions:
The activation function $g$ is typically chosen to be a function that
is applied element-wise, with $h_i = g(x^TW_{:,i}+c_i)...
1
vote
1answer
81 views
Removing constant from the regression model
I am trying to calibrate two variables $(X,Y)$ of different measuring techniques from two instruments, the result of the linear regression analysis appears as shown in the image.
The result shows the ...
1
vote
2answers
624 views
How can I implement tangent distance for k-nearest neighbor in python/scikit-learn?
My ultimate aim is to have a function which I can feed into scikit-learn's NearestNeighbor class as a custom metric parameter.
...
1
vote
1answer
274 views
What exactly is the “hyperbolic” tanh function used in the context of activation functions?
I know the plot of $\tanh$ activation function looks like. I also know that its output has a range of $[-1, 1]$. Furthermore, I also know the it is defined as follows
$$
\tanh(x) = \frac{\sinh(x)}{...
1
vote
1answer
508 views
Eigenvectors and eigenvalues for natural language processing
How are eigenvectors and eigenvalues can be applied/applicable to natural language processing problems ? Any examples ?
1
vote
1answer
32 views
Can all known ML algorithms be written as a sequence of matrix operations?
I keep hearing that machine learning is just linear algebra.
Does that mean all known (and all possible?) ML algos, from random forest, to support-vector machines, to recursive neural networks, can ...
1
vote
1answer
57 views
Eigen Decomposition of Data Matrix for PCA
In PCA we Eigen decompose the covariance matrix, not data matrix, Is it because most data matrices are non-square. If they were, isn't is correct to eigen decompose data matrix than the covariance ...
1
vote
1answer
5k views
How to “reshape” into square matrix for numpy.linalg.solve()?
I'm trying to find the intersection of lines $y=a_1x+b_1$ and $y=a_2x+b_2$ using numpy.linalg.solve(). What I can't get my head around is how to correctly make $A$ ...
1
vote
1answer
62 views
Optimizing vector values for maximum correlation
I'm new to ML, linear algebra, statistics, etc. so bear with me on the terminology...
Iām looking to find a vector that produces the maximum correlation for the relationship between 1) all ...
1
vote
1answer
56 views
On minimizing matrix norm (AB-C)
Given A, B and C are matrices with dim(A) = m x n, dim(B) = n x p and dim (C) = m x p, the problem asks to evaluate
I need to learn $$\tilde{A}$$ such that $$\min_{\tilde{A}}||\tilde{A}^TB-C||$$
and ...
1
vote
1answer
56 views
Least Squares Regression $Ax=b$ when $A$ is fixed and $b$ is varied
The typical setting for least squares regression (or over-determined linear system) for $Ax=b$ is to solve $x$ given $A$ and $b$.
In other words, $A$ and $b$ are fixed when we solve the problem.
My ...
1
vote
1answer
378 views
Is there a quick way to speed up ICP in python using a cached KD-tree
I am currently using ICP to match 2 point clouds. These point clouds evolve in time, so I have to repeat this process many times. I am using a standard KD tree from scipy for my nearest neighbor ...
1
vote
1answer
40 views
Hypothesis vs Hyperplane in Machine Learning
I am finding it hard to understand the clear difference between Hypothesis and Hyperplane.
I know that Hypothesis is a candidate model that maps inputs to outputs after training . And , Hyperplane is ...
1
vote
0answers
11 views
Predicting a variably-placed value in a vector
I have $m$ vectors in $\mathbb{R}^n$, where $m >> n$, and I want to train a model to impute a value $x_i$ in $\mathbf{x}$, where $1 \leq i \leq n$ (and can vary by vector).
For instance, I may ...
1
vote
1answer
37 views
Why transpose of independent feature matrix is necessary in case of linear regression?
I can follow classical linear regression steps:
$Xw=y$
$X^{-1}Xw=X^{-1}y$
$Iw=X^{-1}y$
$w=X^{-1}y$
However, on implementing in Python, I see that instead of simply using
...
1
vote
2answers
55 views
Linear regression with a fixed intercept and everything is in log
I have a set of values for a surface (in pixels) that becomes bigger over time (exponentially). The surface consists of cells that divide over time. After doing some modelling, I came up with the ...
1
vote
0answers
24 views
What's wrong with my backpropagation through time (BTT) calculation or how to multiple a scaled vector and a matrix without matching dimensions?
I am trying to make a pretty simple RNN from scracth, using only Numpy library of Python.
At this moment I am having troubles with BTT as I do not know how to proceed with situation when a ...
1
vote
0answers
17 views
How effective is Moore Penrose for solving regression problems with overdetermined system of equations?
For regression problems with #Predictors > #observations, I recently read about Moore Penrose (pseudo inverse method) which solves the problem of non invertible matrix in OLS for regression problems.
...
1
vote
0answers
19 views
How to add extra constraints to an equationļ¼
Backgroundļ¼
I have an equation which looks like as follows:
$W \times P = R$
$\left[\begin{array} &{1}&{0}&{0}&-\frac{w_{1}}{w_{o1}} &\dots &{0} &-\frac{w_{1}}{w_{0} } \\...
1
vote
0answers
76 views
Can we think of neurons as maps between matrices?
Usually when we think about neurons, we imagine that they enact some kind of map between real numbers. For example, a neuron might take in real numbers $x_{i}$ and weight them with parameters $W_{ij}$,...
1
vote
0answers
13 views
Structures for incorporating linear functions into a nonlinear optimization problem
I'm working on a problem which naturally involves both linear and nonlinear operations, and I'd like some help understanding the best way to combine these into a neural network framework. To be more ...
1
vote
2answers
43 views
Are euclidian vectors and unit vectors same thing? [closed]
Consider this statement : Let the field K be the set R of real numbers, and let the vector space V be the Euclidean space R3. Consider the vectors e1 = (1,0,0), e2 = (0,1,0) and e3 = (0,0,1). Then any ...
0
votes
2answers
116 views
Dose finding slope/intercept using the formula of m,b gives best fit line always In linear regression?
In liner regression We have to fit different lines and chose one with minimum error so What is the motive of having a formula for m,b that can give slope and intercept value in the regression line ,...
0
votes
1answer
27 views
Understanding the algebra behind a specific partial derivative equation
I am following this article about neural networks.
Given:
Until here I understand everything, but then he continues to:
I don't understand how he got to that conclusion. I think he skipped some ...
