Questions tagged [linear-algebra]
A field of mathematics concerned with the study of finite dimensional vector spaces, including matrices and their manipulation, which are important in statistics.
75
questions
1
vote
0answers
10 views
Pseudo inverse of the covariance matrix?
I've been looking for methods to compute a pseudo inverse of a covariance matrix. And found that one way is to construct a regularized inverse matrix. By constructing the eigen system, and removing ...
2
votes
0answers
45 views
How is image convolution actually implemented in deep learning libraries using simple linear algebra?
As a clarifier, I want to implement cross-correlation, but the machine learning literature keeps referring to it as convolution so I will stick with it.
I am trying to implement image convolution ...
0
votes
0answers
19 views
Is Regression Line an 1-D affine subspace of 2-D vector space?
Background
I currently read a book called "Mathematics for Machine Learning" and I read chapter 2 which is
about Linear Algebra, especially on subchapter 2.8 which is about Affine Space.
The ...
1
vote
1answer
27 views
Understanding Lagrangian equation for SVM
I was trying to understand Lagrangian from SVM section of Andrew Ng's Stanford CS229 course notes. On page 17 and 18, he says:
Given the problem
$$\begin{align}
min_w & \quad f(w) \\
s.t. &...
0
votes
1answer
54 views
Understanding SVM mathematics
I was referring SVM section of Andrew Ng's course notes for Stanford CS229 Machine Learning course. On pages 14 and 15, he says:
Consider the picture below:
How can we find the value of $\gamma^{(i)}...
1
vote
1answer
31 views
What are some application of Google's PageRank Algorithm in Data Science
I came across a topic on computational linear algebra that talks about iterative algorithms to compute eigenvalues. I've worked with power method which is an iterative algorithm that converges a ...
0
votes
0answers
23 views
Inverting a matrix using a convolutional neural network
Just for a fun exercise, I am trying to invert a matrix, say size 28x28 (or even 5x5) with a neural network. The way I approached this (quite naively) is as follows:
I built a fully convolutional ...
-1
votes
1answer
24 views
Why do we use 'T' when we are to say matrix-vector product? [closed]
On the first picture author uses $T$ meaning matrix-vector product
But other website do not use $T$, but says that $x$ is a vector, I do not understand if it is important or not
0
votes
0answers
25 views
Why does it produce an eigenvector when all of the direction has changed?
Suppose we have a matrix transformation that corresponds to:
$$A = \begin{bmatrix}2&0\\0&2\\\end{bmatrix}$$
Matrix transformation is transformation from the basis vectors, right?
$$ij = \begin{...
1
vote
1answer
1k views
3d input for Dense Layer Keras
Is there any example of how Keras Dense layer handles 3D input.
The documentation explains the following:
...
1
vote
1answer
241 views
Difference between FDA and LDA
I have asked this question in Mathematics Stackexchange, thought however that it might be more fit for here:
I am currently taking a Data-Analysis course and I learned about both the terms LDA (Linear ...
0
votes
0answers
31 views
What is the Intuition behind weight vector W which is normal to the plane? Is the weight vector W same as the W which is normal to the plane π?
In an interview, I was asked the intuition behind the weight vector. I told the weight vector is a vector which we try to minimize to a local minima with the help of regulariser so we don't overfit. ...
0
votes
1answer
66 views
How does the equation "dW = - (2 * (X^T ).dot(Y - Y_hat)) / m" comes in Linear Regression (using Matrix + Gradient Descent)?
I was trying to code the Linear Regression in Python using Matrix Multiplication method using Gradient Descent and followed a code where there was no mention what is the loss but just a code as Per ...
0
votes
0answers
28 views
How do I find the eigenvectors corresponding to the largest eigenvalue of a matrix in scikit?
Im trying to determine the principal component 1 and 2 of a symmetric matrix using sklearn. Id appreciate any help. Thank you.
0
votes
1answer
25 views
How does tree-based algorithms handle linearly combined features?
While I am aware that tree-based algorithms (e.g., DT, RF, XGBoost) are 'immune' to multi-collinearity, how do they handle linearly combined features?
For example, is there is any additional value or ...
2
votes
0answers
34 views
Geometric classification models
In class we have been presented with a Geometric classification model such that the goal is to construct a linear decision boundary
$\bf{w} \cdot \bf{x} = t$; where $\bf{w}$ is the vector from the ...
0
votes
1answer
233 views
Dot product and linear regression
I'm studying PCA and my professor said something about finding the linear regression by doing the dot product of both axis. Could someone explain to me why? The dot product returns a number. What's ...
0
votes
0answers
59 views
Why do we usually have fully connected layers of same sizes in CNNs?
Is there any specific reason that we observe in CNNs, the fully connected layers usually have the same sizes? You can verify this for many CNNs.
I'm aware that if, for instance, we have a vector of ...
1
vote
1answer
88 views
Hypothesis vs Hyperplane in Machine Learning
I am finding it hard to understand the clear difference between Hypothesis and Hyperplane.
I know that Hypothesis is a candidate model that maps inputs to outputs after training. And, Hyperplane is ...
1
vote
1answer
87 views
Confusion with Notation in the Book on Deep Learning by Ian Goodfellow et al
In chapter 6.1 on 'Example: Learning XOR', the bottom of page 168 mentions:
The activation function $g$ is typically chosen to be a function that
is applied element-wise, with $h_i = g(x^TW_{:,i}+c_i)...
2
votes
0answers
22 views
Deriving vectorized form of linear regression
We first have the weights of a D dimensional vector $w$ and a D dimensional predictor vector $x$, which are all indexed by $j$. There are $N$ observations, all D dimensional. $t$ is our targets, i.e, ...
1
vote
1answer
483 views
Removing constant from the regression model
I am trying to calibrate two variables $(X,Y)$ of different measuring techniques from two instruments, the result of the linear regression analysis appears as shown in the image.
The result shows the ...
0
votes
1answer
32 views
Understanding the algebra behind a specific partial derivative equation
I am following this article about neural networks.
Given:
Until here I understand everything, but then he continues to:
I don't understand how he got to that conclusion. I think he skipped some ...
0
votes
2answers
1k views
Dose finding slope/intercept using the formula of m,b gives best fit line always In linear regression?
In liner regression We have to fit different lines and chose one with minimum error so What is the motive of having a formula for m,b that can give slope and intercept value in the regression line ,...
0
votes
1answer
62 views
Normal equation for linear regression is illogical
Currently I'm taking Andrew Ng's course. He gives a following formula to find solution for linear regression analytically:
$θ = (X^T * X)^{-1} * X^T * у$
He doesn't explain it so I searched for it and ...
0
votes
1answer
94 views
When is it useful to measure the Frobenius norm of a matrix?
In Deep Learning section 2.5 the author review some measures for the size of vectors and matrices.
When in general is it useful for someone to know these?
For instance they give the example of the ...
2
votes
0answers
58 views
Intuition behind understanding eigenvectors and Machine Learning [closed]
I am struggling to understand linear algebra application in machine learning, and I am not able to answer the following question.
Is the model learned in Machine Learning the eigenvector of the ...
1
vote
1answer
37 views
Can all known ML algorithms be written as a sequence of matrix operations?
I keep hearing that machine learning is just linear algebra.
Does that mean all known (and all possible?) ML algos, from random forest, to support-vector machines, to recursive neural networks, can ...
2
votes
4answers
219 views
How to solve Ax = b for A [closed]
Given two know vector x, and b (dimension 3*1 for example), what are the ways to approximate the matrix ...
1
vote
3answers
131 views
Why transpose of independent feature matrix is necessary in case of linear regression?
I can follow classical linear regression steps:
$Xw=y$
$X^{-1}Xw=X^{-1}y$
$Iw=X^{-1}y$
$w=X^{-1}y$
However, on implementing in Python, I see that instead of simply using
...
1
vote
2answers
387 views
Linear regression with a fixed intercept and everything is in log
I have a set of values for a surface (in pixels) that becomes bigger over time (exponentially). The surface consists of cells that divide over time. After doing some modelling, I came up with the ...
3
votes
2answers
44 views
Backpropagation with a different sized training set?
I'm trying to create a NN whose input is a (length m) array of 3d vectors $$\vec{x}_i = [x_{i,1},x_{i,2},x_{i,3}], \hspace{5mm}i=1:m $$
and whose output is a similarly sized array:
$$\vec{h}_{\theta,...
1
vote
0answers
36 views
What's wrong with my backpropagation through time (BTT) calculation or how to multiple a scaled vector and a matrix without matching dimensions?
I am trying to make a pretty simple RNN from scracth, using only Numpy library of Python.
At this moment I am having troubles with BTT as I do not know how to proceed with situation when a ...
3
votes
1answer
90 views
How to incorporate the uncertainty of the model coefficients in the prediction interval of a multiple linear regression
I'm dealing with modeling small experimental data sets. As most experimental work does not generate thousands of samples, but rather a handful, I need to be inventive about how to deal with this small ...
1
vote
0answers
30 views
How effective is Moore Penrose for solving regression problems with overdetermined system of equations?
For regression problems with #Predictors > #observations, I recently read about Moore Penrose (pseudo inverse method) which solves the problem of non invertible matrix in OLS for regression problems.
...
0
votes
0answers
42 views
Workng of LME model used for a set of category variable(s) and a continuous variable?
LME models are being used to analyze the effect of continuos data and category data. Is
this model appropriate for checking the effect of two independent variables - one with continuous values and ...
3
votes
0answers
22 views
Possible flaw in the MDS method for dimensionality reduction
The MDS (multidimensional scaling) method is used to solve the problem of dimensionality reduction. Basically, it does the following: given $n$ points $x_1,\cdots,x_n\in\mathbb R^d$, try to find a ...
4
votes
2answers
1k views
How do we define a linearly separable problem?
When we talk about Perceptrons, we say that they are limited for approximating functions that are linearly separable, while Neural Networks that use non-linear transformations are not.
I am having ...
2
votes
1answer
25 views
optimizing a linear optimization function with linear constarints and binary variables
I am new to optimizations and trying to solve a problem, which I feel falls in the umbrella of optimization.
I have an ojective function that needs to be maximized
...
2
votes
1answer
38 views
Need explanation of a matrix multiplication
I'm reading the Deep Learning book by MIT.
On the page 172, there's a part like this:
$$
f^{(1)}(x)=h=W^Tx \tag{1}
$$
$$
f^{(2)}(h)=h^Tw \tag{2}
$$
Substitute (1) into (2), they got:
$$
f(x)=w^TW^Tx
$...
2
votes
1answer
346 views
Why in this case are gradient steps not perpendicular to contour lines?
There is a theorem that gradient at point is perpendicular to tangent line to contour line at given point.
Why in this picture it seems that this rule is not respected?
source: http://www....
1
vote
2answers
762 views
Gradient descent formula implementation in python
So I recently started with Andrew Ng's ML Course and this is the formula that Andrew lays out for calculating gradient descent on a linear model.
$$ \theta_j = \theta_j - \alpha \frac{1}{m} \sum_{i=1}...
0
votes
0answers
17 views
What does sparsely compute mean?
I heard someone say a neural network needs to sparsely compute the output.
I get what compute means, I get what a sparse matrix is, but what does sparsely compute mean?
12
votes
1answer
502 views
Finding linear transformation under which distance matrices are similar
I have $n$ sets of vectors, where each set $S_i$ contains $k$ vectors in $\mathbb{R}^d$. I know there is some unknown linear transformation $W$ under which the distance matrix $D_i$ (a $k\times k$ ...
-1
votes
1answer
38 views
I can't understand polynomial in the book
I'm reading a book called Bishop - Pattern Recognition and Machine learning.
I came across the following equation, in which I don't understand what $W$ stands for.
So, what is $W$?
4
votes
1answer
924 views
Mathematical formulation of Support Vector Machines?
I'm trying to learn maths behind SVM (hard margin) but due to different forms of mathematical formulations I'm bit confused.
Assume we have two sets of points $\text{(i.e. positives, negatives)}$ one ...
1
vote
1answer
80 views
Eigen Decomposition of Data Matrix for PCA
In PCA we Eigen decompose the covariance matrix, not data matrix, Is it because most data matrices are non-square. If they were, isn't is correct to eigen decompose data matrix than the covariance ...
3
votes
1answer
190 views
PCA formulation - Deep Learning book by Ian Goodfellow
I am reading this deep learning book by Ian goodfellow. In the PCA formulation in the first chapter i.e Linear Algebra, he mentions the following:
we need to choose the encoding matrix D. To do so,...
1
vote
1answer
898 views
Why does np.linalg.eig produce an opposite-signed eigenvector?
I am learning SVD by following this MIT course. In this video, the lecturer is finding the SVD for
$$
\begin{pmatrix} 5 & 5 \\ -1 & 7 \end{pmatrix},
$$
which involves finding the ...
0
votes
1answer
896 views
Linear regression with white Gaussian noise
I am new to machine learning , so this question may sound fundamental. My task is to estimate the parameter vector of the equation with the least squares method:
$y = \theta_0 + \theta_1x + \theta_2x^...
