Questions tagged [linear-algebra]
A field of mathematics concerned with the study of finite dimensional vector spaces, including matrices and their manipulation, which are important in statistics.
42
questions
3
votes
1answer
23 views
optimizing a linear optimization function with linear constarints and binary variables
I am new to optimizations and trying to solve a problem, which I feel falls in the umbrella of optimization.
I have an ojective function that needs to be maximized
...
2
votes
1answer
31 views
Need explanation of a matrix multiplication
I'm reading the Deep Learning book by MIT.
On the page 172, there's a part like this:
$$
f^{(1)}(x)=h=W^Tx \tag{1}
$$
$$
f^{(2)}(h)=h^Tw \tag{2}
$$
Substitute (1) into (2), they got:
$$
f(x)=w^TW^Tx
$...
2
votes
1answer
32 views
Why in this case are gradient steps not perpendicular to contour lines?
There is a theorem that gradient at point is perpendicular to tangent line to contour line at given point.
Why in this picture it seems that this rule is not respected?
source: http://www....
0
votes
0answers
24 views
How do I get confidence intervals for an ElasticNet in sklearn?
I need to produce a row for the confidence interval for every field that I am calculating coefficients and scores off of. So here is my code so far-
...
0
votes
1answer
154 views
Gradient descent formula implementation in python
So I recently started with Andrew Ng's ML Course and this is the formula that Andrew lays out for calculating gradient descent on a linear model.
$$ \theta_j = \theta_j - \alpha \frac{1}{m} \sum_{i=1}...
0
votes
0answers
12 views
What does sparsely compute mean?
I heard someone say a neural network needs to sparsely compute the output.
I get what compute means, I get what a sparse matrix is, but what does sparsely compute mean?
6
votes
0answers
229 views
Finding linear transformation under which distance matrices are similar
I have n sets of vectors, where each set S_i contains k vectors in ...
0
votes
0answers
14 views
Chaining bias terms in backprob
let's say I have a few linear layers $l_1 \dots l_n$:
$y=I(\dots I(IX + b_1) + b_2) \dots +b_n)$
where $n$ is sufficiently large and $I$ is the (nonparametric) identity matrix. The gradient for $b_n$...
-1
votes
1answer
33 views
I can't understand polynomial in the book
I'm reading a book called Bishop - Pattern Recognition and Machine learning.
I came across the following equation, in which I don't understand what $W$ stands for.
So, what is $W$?
4
votes
1answer
289 views
Mathematical formulation of Support Vector Machines?
I'm trying to learn maths behind SVM (hard margin) but due to different forms of mathematical formulations I'm bit confused.
Assume we have two sets of points $\text{(i.e. positives, negatives)}$ one ...
1
vote
1answer
39 views
Eigen Decomposition of Data Matrix for PCA
In PCA we Eigen decompose the covariance matrix, not data matrix, Is it because most data matrices are non-square. If they were, isn't is correct to eigen decompose data matrix than the covariance ...
3
votes
1answer
57 views
PCA formulation - Deep Learning book by Ian Goodfellow
I am reading this deep learning book by Ian goodfellow. In the PCA formulation in the first chapter i.e Linear Algebra, he mentions the following:
we need to choose the encoding matrix D. To do so,...
1
vote
1answer
187 views
Why does np.linalg.eig produce an opposite-signed eigenvector?
I am learning SVD by following this MIT course. In this video, the lecturer is finding the SVD for
$$
\begin{pmatrix} 5 & 5 \\ -1 & 7 \end{pmatrix},
$$
which involves finding the ...
0
votes
0answers
10 views
Performant alternatives to Matrix package in R that requires minimal effort by end users to install
Background
I'm going to be releasing a package for R that does calculations involving extremely large sparse matrices (1,000,000 x 1,000,000 is the minimum for what we consider useful). For this ...
0
votes
1answer
158 views
Linear regression with white Gaussian noise
I am new to machine learning , so this question may sound fundamental. My task is to estimate the parameter vector of the equation with the least squares method:
$y = \theta_0 + \theta_1x + \theta_2x^...
1
vote
0answers
19 views
How to add extra constraints to an equation？
Background：
I have an equation which looks like as follows:
$W \times P = R$
$\left[\begin{array} &{1}&{0}&{0}&-\frac{w_{1}}{w_{o1}} &\dots &{0} &-\frac{w_{1}}{w_{0} } \\...
1
vote
1answer
2k views
How to “reshape” into square matrix for numpy.linalg.solve()?
I'm trying to find the intersection of lines $y=a_1x+b_1$ and $y=a_2x+b_2$ using numpy.linalg.solve(). What I can't get my head around is how to correctly make $A$ ...
1
vote
0answers
145 views
Connection between piecewise linear basis functions and RELU activation function
ReLU activation is defined as follows
$$\sigma(x)=\max(0, x).$$
Let's assume that I have deep network of 1 hidden layer, than output from my layer has form
$$ f(x)= \sigma(Wx +b), $$
where matrix W ...
1
vote
1answer
53 views
Optimizing vector values for maximum correlation
I'm new to ML, linear algebra, statistics, etc. so bear with me on the terminology...
I’m looking to find a vector that produces the maximum correlation for the relationship between 1) all ...
1
vote
1answer
55 views
How can positional encodings including a sine operation be linearly transformable for any offset?
In the paper "Attention is all you need" the authors add a positional encoding to each token in the sequence (section 3.5). The following encoding is chosen:
$ PE(pos, 2dim) = sin(pos / 10000 ^ {2dim/...
1
vote
0answers
74 views
Can we think of neurons as maps between matrices?
Usually when we think about neurons, we imagine that they enact some kind of map between real numbers. For example, a neuron might take in real numbers $x_{i}$ and weight them with parameters $W_{ij}$,...
1
vote
0answers
12 views
Structures for incorporating linear functions into a nonlinear optimization problem
I'm working on a problem which naturally involves both linear and nonlinear operations, and I'd like some help understanding the best way to combine these into a neural network framework. To be more ...
0
votes
0answers
28 views
Dimensionality reduction categories
According to what I found, dimensionality reduction has two types feature selection and feature extraction . In feature extraction, we find PCA, LDA ,LLE , ISOMAP, etc.. In other works i find random ...
2
votes
1answer
248 views
Machine learning PhD Interview technical questions [closed]
I'm Software Engineer who applied to grad school for Machine Learning/Computer Vision PhD and currently waiting for interview calls. I'm brushing up Linear algebra/ ML topics. What kind of technical ...
1
vote
2answers
396 views
How can I implement tangent distance for k-nearest neighbor in python/scikit-learn?
My ultimate aim is to have a function which I can feed into scikit-learn's NearestNeighbor class as a custom metric parameter.
...
3
votes
0answers
54 views
Linear algebra library for c++
I have been trying to find the substitute of numpy and perform some linear algebra using c++ and here's a list of libraries I have encountered:
Eigen
Armadillo
Dlib
GNU Scientific library
Please ...
1
vote
4answers
120 views
Statistics Before Linear Algebra?
I know this is an opinion-based question and will be closed but this is the only place I know that can answer it reasonably and it is a very important matter to me.
I am pursuing a machine ...
8
votes
3answers
4k views
0
votes
1answer
37 views
Can I use regression to solve a multiple equation problem
I'm working on a problem which is a multiple equation. I have a group of people and each person in the group is working on different tasks (e.g. n tasks in total). Each person in this group is working ...
1
vote
1answer
34 views
On minimizing matrix norm (AB-C)
Given A, B and C are matrices with dim(A) = m x n, dim(B) = n x p and dim (C) = m x p, the problem asks to evaluate
I need to learn $$\tilde{A}$$ such that $$\min_{\tilde{A}}||\tilde{A}^TB-C||$$
and ...
1
vote
1answer
51 views
Least Squares Regression $Ax=b$ when $A$ is fixed and $b$ is varied
The typical setting for least squares regression (or over-determined linear system) for $Ax=b$ is to solve $x$ given $A$ and $b$.
In other words, $A$ and $b$ are fixed when we solve the problem.
My ...
1
vote
2answers
27 views
Are euclidian vectors and unit vectors same thing? [closed]
Consider this statement : Let the field K be the set R of real numbers, and let the vector space V be the Euclidean space R3. Consider the vectors e1 = (1,0,0), e2 = (0,1,0) and e3 = (0,0,1). Then any ...
3
votes
1answer
126 views
RNN: why Wx + Uh instead of W[x,h]
Traditionally, a state for RNN is computed as
$$h_t = \sigma(W\cdot \vec x + U\cdot \vec h_{t-1} + \vec b)$$
For a RNN, why to add-up the terms $(Wx + Uh_{t-1})$ instead of just having a single ...
3
votes
2answers
47 views
Derivates with respect to a vector
Suppose I have an equation, $f = X^TY + \dots$ (a few more terms), where $X$ is a vector and $Y$ is a matrix of appropriate dimensions,
I want to know how can we take the derivative of $f \text{ w.r.t....
1
vote
2answers
124 views
How does “linear algebraic” weight training function work?
This answer shows that linear and polynomial function weights can be trained using this matrix operation:
$w = (X^TX)^{-1}X^Ty$
Therefore, algorithms such as gradient descent are not necessary for ...
1
vote
1answer
244 views
What exactly is the “hyperbolic” tanh function used in the context of activation functions?
I know the plot of $\tanh$ activation function looks like. I also know that its output has a range of $[-1, 1]$. Furthermore, I also know the it is defined as follows
$$
\tanh(x) = \frac{\sinh(x)}{...
5
votes
1answer
304 views
Closed form solution of linear regression via least squares using matrix derivatives
How is the closed form solution to linear regression derived using matrix derivatives as opposed to using the trace method as Andrew Ng does in his Machine learning lectures. Specifically, I am ...
4
votes
2answers
802 views
What is the use of additional column of 1s in normal equation?
Currently I am going through Normal Equation in Machine Learning.
$$
\hat\theta = (X^T \cdot X)^{-1} \cdot X^T \cdot y
$$
But when I see how they use this equation, I found they always add an ...
2
votes
0answers
49 views
How does SVD actually provide the recommendations? I seem to get conflicting answers
I am reading a text book that basically says the following:
Given a matrix A where A is USERS x ITEMS we can use SVD to decompose the matrix into:
$$A = U \times \Sigma \times V^T$$
Then we can take ...
1
vote
1answer
297 views
Is there a quick way to speed up ICP in python using a cached KD-tree
I am currently using ICP to match 2 point clouds. These point clouds evolve in time, so I have to repeat this process many times. I am using a standard KD tree from scipy for my nearest neighbor ...
8
votes
1answer
1k views
Deriving backpropagation equations “natively” in tensor form
Image shows a typical layer somewhere in a feed forward network:
$a_i^{(k)}$ is the activation value of the $i^{th}$ neuron in the $k^{th}$ layer.
$W_{ij}^{(k)}$ is the weight connecting $i^{th}$ ...
1
vote
1answer
400 views
Eigenvectors and eigenvalues for natural language processing
How are eigenvectors and eigenvalues can be applied/applicable to natural language processing problems ? Any examples ?
