Skip to main content

Questions tagged [linear-algebra]

A field of mathematics concerned with the study of finite dimensional vector spaces, including matrices and their manipulation, which are important in statistics.

Filter by
Sorted by
Tagged with
14 votes
3 answers
9k views

How does tensor product/multiplication work in TensorFlow?

In Tensorflow, I saw the following example: ...
frt132's user avatar
  • 159
12 votes
1 answer
587 views

Finding linear transformation under which distance matrices are similar

I have $n$ sets of vectors, where each set $S_i$ contains $k$ vectors in $\mathbb{R}^d$. I know there is some unknown linear transformation $W$ under which the distance matrix $D_i$ (a $k\times k$ ...
user1767774's user avatar
11 votes
2 answers
5k views

What is the use of additional column of 1s in normal equation?

Currently I am going through Normal Equation in Machine Learning. $$ \hat\theta = (X^T \cdot X)^{-1} \cdot X^T \cdot y $$ But when I see how they use this equation, I found they always add an ...
Sreeraj Chundayil's user avatar
9 votes
1 answer
2k views

Deriving backpropagation equations "natively" in tensor form

Image shows a typical layer somewhere in a feed forward network: $a_i^{(k)}$ is the activation value of the $i^{th}$ neuron in the $k^{th}$ layer. $W_{ij}^{(k)}$ is the weight connecting $i^{th}$ ...
Neil Slater's user avatar
  • 28.9k
5 votes
1 answer
989 views

Closed form solution of linear regression via least squares using matrix derivatives

How is the closed form solution to linear regression derived using matrix derivatives as opposed to using the trace method as Andrew Ng does in his Machine learning lectures. Specifically, I am ...
user8919's user avatar
5 votes
0 answers
85 views

How is image convolution actually implemented in deep learning libraries using simple linear algebra?

As a clarifier, I want to implement cross-correlation, but the machine learning literature keeps referring to it as convolution so I will stick with it. I am trying to implement image convolution ...
Jozef Nagy's user avatar
4 votes
1 answer
2k views

Mathematical formulation of Support Vector Machines?

I'm trying to learn maths behind SVM (hard margin) but due to different forms of mathematical formulations I'm bit confused. Assume we have two sets of points $\text{(i.e. positives, negatives)}$ one ...
Jeeth's user avatar
  • 931
4 votes
2 answers
71 views
+50

What type of technique can be used to solve this question?

Apology for the ambiguous title, I do not know the term. I have data of some products which a few variables: origin, weight, brand. For example: Product A = "China, 100g, Brand X" Product ...
lpounng's user avatar
  • 1,018
4 votes
1 answer
369 views

How can positional encodings including a sine operation be linearly transformable for any offset?

In the paper "Attention is all you need" the authors add a positional encoding to each token in the sequence (section 3.5). The following encoding is chosen: $ PE(pos, 2dim) = sin(pos / 10000 ^ {2dim/...
Stephan Heijl's user avatar
4 votes
2 answers
1k views

How do we define a linearly separable problem?

When we talk about Perceptrons, we say that they are limited for approximating functions that are linearly separable, while Neural Networks that use non-linear transformations are not. I am having ...
Stefan Radonjic's user avatar
3 votes
1 answer
724 views

Machine learning PhD Interview technical questions [closed]

I'm Software Engineer who applied to grad school for Machine Learning/Computer Vision PhD and currently waiting for interview calls. I'm brushing up Linear algebra/ ML topics. What kind of technical ...
Aniruddha's user avatar
  • 131
3 votes
1 answer
264 views

RNN: why Wx + Uh instead of W[x,h]

Traditionally, a state for RNN is computed as $$h_t = \sigma(W\cdot \vec x + U\cdot \vec h_{t-1} + \vec b)$$ For a RNN, why to add-up the terms $(Wx + Uh_{t-1})$ instead of just having a single ...
Kari's user avatar
  • 2,726
3 votes
1 answer
765 views

Why in this case are gradient steps not perpendicular to contour lines?

There is a theorem that gradient at point is perpendicular to tangent line to contour line at given point. Why in this picture it seems that this rule is not respected? source: http://www....
MajorTom's user avatar
3 votes
1 answer
338 views

PCA formulation - Deep Learning book by Ian Goodfellow

I am reading this deep learning book by Ian goodfellow. In the PCA formulation in the first chapter i.e Linear Algebra, he mentions the following: we need to choose the encoding matrix D. To do so,...
Kunal Kishore Singh's user avatar
3 votes
0 answers
241 views

Multi-dimensional Euclidian R^2 squared - reasonable?

I have a high-dimensional space, say $\mathbb{R}^{1000}$, and I have samples $y_1, \ldots , y_n \in \mathbb{R}^{1000}$ and $\hat{y}_1, \ldots , \hat{y}_n \in \mathbb{R}^{1000}$. Would $$ R^2 = 1 - \...
AspiringToAspire's user avatar
3 votes
2 answers
59 views

Backpropagation with a different sized training set?

I'm trying to create a NN whose input is a (length m) array of 3d vectors $$\vec{x}_i = [x_{i,1},x_{i,2},x_{i,3}], \hspace{5mm}i=1:m $$ and whose output is a similarly sized array: $$\vec{h}_{\theta,...
Spinach's user avatar
  • 31
3 votes
1 answer
120 views

How to incorporate the uncertainty of the model coefficients in the prediction interval of a multiple linear regression

I'm dealing with modeling small experimental data sets. As most experimental work does not generate thousands of samples, but rather a handful, I need to be inventive about how to deal with this small ...
DannyVanpoucke's user avatar
3 votes
0 answers
28 views

Possible flaw in the MDS method for dimensionality reduction

The MDS (multidimensional scaling) method is used to solve the problem of dimensionality reduction. Basically, it does the following: given $n$ points $x_1,\cdots,x_n\in\mathbb R^d$, try to find a ...
trisct's user avatar
  • 131
3 votes
0 answers
245 views

Best linear algebra library for C++?

I have been trying to find the substitute of numpy and perform some linear algebra using C++. Here's a list of the libraries I have encountered: Eigen Armadillo Dlib GNU Scientific library Please ...
thanatoz's user avatar
  • 2,415
3 votes
2 answers
97 views

Derivates with respect to a vector

Suppose I have an equation, $f = X^TY + \dots$ (a few more terms), where $X$ is a vector and $Y$ is a matrix of appropriate dimensions, I want to know how can we take the derivative of $f \text{ w.r.t....
Ayush's user avatar
  • 31
2 votes
4 answers
488 views

How to solve Ax = b for A [closed]

Given two know vector x, and b (dimension 3*1 for example), what are the ways to approximate the matrix ...
silkAdmin's user avatar
  • 143
2 votes
1 answer
2k views

Why does np.linalg.eig produce an opposite-signed eigenvector?

I am learning SVD by following this MIT course. In this video, the lecturer is finding the SVD for $$ \begin{pmatrix} 5 & 5 \\ -1 & 7 \end{pmatrix}, $$ which involves finding the ...
Jay's user avatar
  • 425
2 votes
1 answer
4k views

3d input for Dense Layer Keras

Is there any example of how Keras Dense layer handles 3D input. The documentation explains the following: ...
data_person's user avatar
2 votes
1 answer
2k views

Difference between FDA and LDA

I have asked this question in Mathematics Stackexchange, thought however that it might be more fit for here: I am currently taking a Data-Analysis course and I learned about both the terms LDA (Linear ...
Nestroy's user avatar
  • 21
2 votes
1 answer
63 views

Need explanation of a matrix multiplication

I'm reading the Deep Learning book by MIT. On the page 172, there's a part like this: $$ f^{(1)}(x)=h=W^Tx \tag{1} $$ $$ f^{(2)}(h)=h^Tw \tag{2} $$ Substitute (1) into (2), they got: $$ f(x)=w^TW^Tx $...
Kyouko's user avatar
  • 23
2 votes
1 answer
105 views

optimizing a linear optimization function with linear constarints and binary variables

I am new to optimizations and trying to solve a problem, which I feel falls in the umbrella of optimization. I have an ojective function that needs to be maximized ...
chink's user avatar
  • 555
2 votes
0 answers
152 views

Geometric classification models

In class we have been presented with a Geometric classification model such that the goal is to construct a linear decision boundary $\bf{w} \cdot \bf{x} = t$; where $\bf{w}$ is the vector from the ...
Zafir Stojanovski's user avatar
2 votes
0 answers
118 views

Deriving vectorized form of linear regression

We first have the weights of a D dimensional vector $w$ and a D dimensional predictor vector $x$, which are all indexed by $j$. There are $N$ observations, all D dimensional. $t$ is our targets, i.e, ...
user2793618's user avatar
2 votes
0 answers
66 views

Intuition behind understanding eigenvectors and Machine Learning [closed]

I am struggling to understand linear algebra application in machine learning, and I am not able to answer the following question. Is the model learned in Machine Learning the eigenvector of the ...
Hermes Morales's user avatar
2 votes
0 answers
57 views

How does SVD actually provide the recommendations? I seem to get conflicting answers

I am reading a text book that basically says the following: Given a matrix A where A is USERS x ITEMS we can use SVD to decompose the matrix into: $$A = U \times \Sigma \times V^T$$ Then we can take ...
theGuy05's user avatar
  • 121
1 vote
2 answers
5k views

Dose finding slope/intercept using the formula of m,b gives best fit line always In linear regression?

In liner regression We have to fit different lines and chose one with minimum error so What is the motive of having a formula for m,b that can give slope and intercept value in the regression line ,...
star's user avatar
  • 1,471
1 vote
2 answers
3k views

Removing constant from the regression model

I am trying to calibrate two variables $(X,Y)$ of different measuring techniques from two instruments, the result of the linear regression analysis appears as shown in the image. The result shows the ...
Yazan Alatoom's user avatar
1 vote
4 answers
2k views

Statistics Before Linear Algebra?

I know this is an opinion-based question and will be closed but this is the only place I know that can answer it reasonably and it is a very important matter to me. I am pursuing a machine ...
boinka's user avatar
  • 53
1 vote
2 answers
335 views

How does "linear algebraic" weight training function work?

This answer shows that linear and polynomial function weights can be trained using this matrix operation: $w = (X^TX)^{-1}X^Ty$ Therefore, algorithms such as gradient descent are not necessary for ...
ShellRox's user avatar
  • 409
1 vote
2 answers
65 views

Affine 2D mapping in python

I have two sets of 2D data $A$ and $B$ (representing 2D positions on a 2D plane $x,y$) which are related (the first pair of $x,y$ of $A$ is related to the first pair of $x,y$ of $B$ for instance). I ...
patjol's user avatar
  • 11
1 vote
1 answer
176 views

Why linear model cannot understand the interaction between any two input features?

The book Deep Learning by Ian Goodfellow states that: ...
Gull Noor's user avatar
1 vote
1 answer
125 views

What are some application of Google's PageRank Algorithm in Data Science

I came across a topic on computational linear algebra that talks about iterative algorithms to compute eigenvalues. I've worked with power method which is an iterative algorithm that converges a ...
SPARSE's user avatar
  • 115
1 vote
1 answer
176 views

Confusion with Notation in the Book on Deep Learning by Ian Goodfellow et al

In chapter 6.1 on 'Example: Learning XOR', the bottom of page 168 mentions: The activation function $g$ is typically chosen to be a function that is applied element-wise, with $h_i = g(x^TW_{:,i}+c_i)...
KGhatak's user avatar
  • 123
1 vote
1 answer
454 views

When is it useful to measure the Frobenius norm of a matrix?

In Deep Learning section 2.5 the author review some measures for the size of vectors and matrices. When in general is it useful for someone to know these? For instance they give the example of the ...
nid's user avatar
  • 131
1 vote
2 answers
10k views

How to "reshape" into square matrix for numpy.linalg.solve()?

I'm trying to find the intersection of lines $y=a_1x+b_1$ and $y=a_2x+b_2$ using numpy.linalg.solve(). What I can't get my head around is how to correctly make $A$ ...
basse's user avatar
  • 297
1 vote
2 answers
984 views

How can I implement tangent distance for k-nearest neighbor in python/scikit-learn?

My ultimate aim is to have a function which I can feed into scikit-learn's NearestNeighbor class as a custom metric parameter. ...
nocibambi's user avatar
  • 175
1 vote
1 answer
898 views

Eigenvectors and eigenvalues for natural language processing

How are eigenvectors and eigenvalues can be applied/applicable to natural language processing problems ? Any examples ?
Sreejithc321's user avatar
  • 1,920
1 vote
1 answer
56 views

Plot a matrix as a single point in space

I have a dataset of drugs represented as a graph, each of which is described by three non-square matrices: edge index (A), an 2xe matrix, where e are the bonds of the molecule, the first line ...
Gianmarco Luchetti 's user avatar
1 vote
1 answer
730 views

How does the equation "dW = - (2 * (X^T ).dot(Y - Y_hat)) / m" comes in Linear Regression (using Matrix + Gradient Descent)?

I was trying to code the Linear Regression in Python using Matrix Multiplication method using Gradient Descent and followed a code where there was no mention what is the loss but just a code as Per ...
Deshwal's user avatar
  • 323
1 vote
1 answer
1k views

Dot product and linear regression

I'm studying PCA and my professor said something about finding the linear regression by doing the dot product of both axis. Could someone explain to me why? The dot product returns a number. What's ...
leoperassoli's user avatar
1 vote
1 answer
44 views

Can all known ML algorithms be written as a sequence of matrix operations?

I keep hearing that machine learning is just linear algebra. Does that mean all known (and all possible?) ML algos, from random forest, to support-vector machines, to recursive neural networks, can ...
SubstantialRange's user avatar
1 vote
3 answers
589 views

Why transpose of independent feature matrix is necessary in case of linear regression?

I can follow classical linear regression steps: $Xw=y$ $X^{-1}Xw=X^{-1}y$ $Iw=X^{-1}y$ $w=X^{-1}y$ However, on implementing in Python, I see that instead of simply using ...
Fredrik's user avatar
  • 1,007
1 vote
1 answer
109 views

Eigen Decomposition of Data Matrix for PCA

In PCA we Eigen decompose the covariance matrix, not data matrix, Is it because most data matrices are non-square. If they were, isn't is correct to eigen decompose data matrix than the covariance ...
tam's user avatar
  • 151
1 vote
1 answer
137 views

Optimizing vector values for maximum correlation

I'm new to ML, linear algebra, statistics, etc. so bear with me on the terminology... I’m looking to find a vector that produces the maximum correlation for the relationship between 1) all ...
SuperCodeBrah's user avatar
1 vote
1 answer
142 views

On minimizing matrix norm (AB-C)

Given A, B and C are matrices with dim(A) = m x n, dim(B) = n x p and dim (C) = m x p, the problem asks to evaluate I need to learn $$\tilde{A}$$ such that $$\min_{\tilde{A}}||\tilde{A}^TB-C||$$ and ...
Prakhar Agarwal's user avatar