# Questions tagged [linear-algebra]

A field of mathematics concerned with the study of finite dimensional vector spaces, including matrices and their manipulation, which are important in statistics.

57 questions
Filter by
Sorted by
Tagged with
48 views

### Normal equation for linear regression is illogical

Currently I'm taking Andrew Ng's course. He gives a following formula to find solution for linear regression analytically: $θ = (X^T * X)^{-1} * X^T * у$ He doesn't explain it so I searched for it and ...
17 views

### When is it useful to measure the Frobenius norm of a matrix?

In Deep Learning section 2.5 the author review some measures for the size of vectors and matrices. When in general is it useful for someone to know these? For instance they give the example of the ...
10 views

### Predicting a variably-placed value in a vector

I have $m$ vectors in $\mathbb{R}^n$, where $m >> n$, and I want to train a model to impute a value $x_i$ in $\mathbf{x}$, where $1 \leq i \leq n$ (and can vary by vector). For instance, I may ...
33 views

### Intuition behind understanding eigenvectors and Machine Learning

I am struggling to understand linear algebra application in machine learning, and I am not able to answer the following question. Is the model learned in Machine Learning the eigenvector of the ...
19 views

### What are some theoretical approches to study Machine Learning/Deep Learning?

I've recently studied "basics of machine learning" (chapter 5) in Deep Learning by Ian Goodfellow. They're explaining ML with most of statistical and probabilitical methods. But I wish to ...
7 views

### Projecting 3D Chemical Data Onto a 2D Plane in Motion

I'm trying to model the rotation of two hydrogen atoms about a carbon atom. Say I have a conceptual wheel on an axle that is attached to my car. The axle is described by two points in 3D space, as ...
39 views

### How to calculate latent vector for a user in ALS based on some new input?

So I have an ALS trained in pyspark but then I get some interactions from a new user that wasn't in the training set. I want to give recommendations to that new user without retraining the ALS based ...
29 views

### Can all known ML algorithms be written as a sequence of matrix operations?

I keep hearing that machine learning is just linear algebra. Does that mean all known (and all possible?) ML algos, from random forest, to support-vector machines, to recursive neural networks, can ...
119 views

### How to solve Ax = b for A [closed]

Given two know vector x, and b (dimension 3*1 for example), what are the ways to approximate the matrix ...
27 views

### Why transpose of independent feature matrix is necessary in case of linear regression?

I can follow classical linear regression steps: $Xw=y$ $X^{-1}Xw=X^{-1}y$ $Iw=X^{-1}y$ $w=X^{-1}y$ However, on implementing in Python, I see that instead of simply using ...
20 views

### Linear regression with a fixed intercept and everything is in log

I have a set of values for a surface (in pixels) that becomes bigger over time (exponentially). The surface consists of cells that divide over time. After doing some modelling, I came up with the ...
I'm trying to create a NN whose input is a (length m) array of 3d vectors $$\vec{x}_i = [x_{i,1},x_{i,2},x_{i,3}], \hspace{5mm}i=1:m$$ and whose output is a similarly sized array: $$\vec{h}_{\theta,... 0answers 17 views ### What's wrong with my backpropagation through time (BTT) calculation or how to multiple a scaled vector and a matrix without matching dimensions? I am trying to make a pretty simple RNN from scracth, using only Numpy library of Python. At this moment I am having troubles with BTT as I do not know how to proceed with situation when a ... 0answers 45 views ### How to incorporate the uncertainty of the model coefficients in the prediction interval of a multiple linear regression I'm dealing with the modeling of small experimental data sets. As most experimental work does not generate thousands of samples, but rather a handful, I need to be inventive in how to deal with this ... 0answers 13 views ### How effective is Moore Penrose for solving regression problems with overdetermined system of equations? For regression problems with #Predictors > #observations, I recently read about Moore Penrose (pseudo inverse method) which solves the problem of non invertible matrix in OLS for regression problems. ... 0answers 38 views ### Workng of LME model used for a set of category variable(s) and a continuous variable? LME models are being used to analyze the effect of continuos data and category data. Is this model appropriate for checking the effect of two independent variables - one with continuous values and ... 0answers 21 views ### Possible flaw in the MDS method for dimensionality reduction The MDS (multidimensional scaling) method is used to solve the problem of dimensionality reduction. Basically, it does the following: given n points x_1,\cdots,x_n\in\mathbb R^d, try to find a ... 1answer 238 views ### How do we define a linearly separable problem? When we talk about Perceptrons, we say that they are limited for approximating functions that are linearly separable, while Neural Networks that use non-linear transformations are not. I am having ... 1answer 23 views ### optimizing a linear optimization function with linear constarints and binary variables I am new to optimizations and trying to solve a problem, which I feel falls in the umbrella of optimization. I have an ojective function that needs to be maximized ... 1answer 34 views ### Need explanation of a matrix multiplication I'm reading the Deep Learning book by MIT. On the page 172, there's a part like this:$$ f^{(1)}(x)=h=W^Tx \tag{1}  f^{(2)}(h)=h^Tw \tag{2} $$Substitute (1) into (2), they got:$$ f(x)=w^TW^Tx $... 1answer 99 views ### Why in this case are gradient steps not perpendicular to contour lines? There is a theorem that gradient at point is perpendicular to tangent line to contour line at given point. Why in this picture it seems that this rule is not respected? source: http://www.... 0answers 39 views ### How do I get confidence intervals for an ElasticNet in sklearn? I need to produce a row for the confidence interval for every field that I am calculating coefficients and scores off of. So here is my code so far- ... 1answer 344 views ### Gradient descent formula implementation in python So I recently started with Andrew Ng's ML Course and this is the formula that Andrew lays out for calculating gradient descent on a linear model. $$\theta_j = \theta_j - \alpha \frac{1}{m} \sum_{i=1}... 0answers 16 views ### What does sparsely compute mean? I heard someone say a neural network needs to sparsely compute the output. I get what compute means, I get what a sparse matrix is, but what does sparsely compute mean? 1answer 367 views ### Finding linear transformation under which distance matrices are similar I have n sets of vectors, where each set S_i contains k vectors in \mathbb{R}^d. I know there is some unknown linear transformation W under which the distance matrix D_i (a k\times k ... 1answer 35 views ### I can't understand polynomial in the book I'm reading a book called Bishop - Pattern Recognition and Machine learning. I came across the following equation, in which I don't understand what W stands for. So, what is W? 1answer 444 views ### Mathematical formulation of Support Vector Machines? I'm trying to learn maths behind SVM (hard margin) but due to different forms of mathematical formulations I'm bit confused. Assume we have two sets of points \text{(i.e. positives, negatives)} one ... 1answer 45 views ### Eigen Decomposition of Data Matrix for PCA In PCA we Eigen decompose the covariance matrix, not data matrix, Is it because most data matrices are non-square. If they were, isn't is correct to eigen decompose data matrix than the covariance ... 1answer 96 views ### PCA formulation - Deep Learning book by Ian Goodfellow I am reading this deep learning book by Ian goodfellow. In the PCA formulation in the first chapter i.e Linear Algebra, he mentions the following: we need to choose the encoding matrix D. To do so,... 1answer 330 views ### Why does np.linalg.eig produce an opposite-signed eigenvector? I am learning SVD by following this MIT course. In this video, the lecturer is finding the SVD for$$ \begin{pmatrix} 5 & 5 \\ -1 & 7 \end{pmatrix}, $$which involves finding the ... 1answer 319 views ### Linear regression with white Gaussian noise I am new to machine learning , so this question may sound fundamental. My task is to estimate the parameter vector of the equation with the least squares method: y = \theta_0 + \theta_1x + \theta_2x^... 0answers 19 views ### How to add extra constraints to an equation？ Background： I have an equation which looks like as follows: W \times P = R \left[\begin{array} &{1}&{0}&{0}&-\frac{w_{1}}{w_{o1}} &\dots &{0} &-\frac{w_{1}}{w_{0} } \\... 1answer 3k views ### How to “reshape” into square matrix for numpy.linalg.solve()? I'm trying to find the intersection of lines y=a_1x+b_1 and y=a_2x+b_2 using numpy.linalg.solve(). What I can't get my head around is how to correctly make A ... 0answers 173 views ### Connection between piecewise linear basis functions and RELU activation function ReLU activation is defined as follows$$\sigma(x)=\max(0, x).$$Let's assume that I have deep network of 1 hidden layer, than output from my layer has form$$ f(x)= \sigma(Wx +b), $$where matrix W ... 1answer 58 views ### Optimizing vector values for maximum correlation I'm new to ML, linear algebra, statistics, etc. so bear with me on the terminology... I’m looking to find a vector that produces the maximum correlation for the relationship between 1) all ... 1answer 140 views ### How can positional encodings including a sine operation be linearly transformable for any offset? In the paper "Attention is all you need" the authors add a positional encoding to each token in the sequence (section 3.5). The following encoding is chosen: PE(pos, 2dim) = sin(pos / 10000 ^ {2dim/... 0answers 76 views ### Can we think of neurons as maps between matrices? Usually when we think about neurons, we imagine that they enact some kind of map between real numbers. For example, a neuron might take in real numbers x_{i} and weight them with parameters W_{ij},... 0answers 13 views ### Structures for incorporating linear functions into a nonlinear optimization problem I'm working on a problem which naturally involves both linear and nonlinear operations, and I'd like some help understanding the best way to combine these into a neural network framework. To be more ... 1answer 373 views ### Machine learning PhD Interview technical questions [closed] I'm Software Engineer who applied to grad school for Machine Learning/Computer Vision PhD and currently waiting for interview calls. I'm brushing up Linear algebra/ ML topics. What kind of technical ... 2answers 534 views ### How can I implement tangent distance for k-nearest neighbor in python/scikit-learn? My ultimate aim is to have a function which I can feed into scikit-learn's NearestNeighbor class as a custom metric parameter. ... 0answers 61 views ### Linear algebra library for c++ I have been trying to find the substitute of numpy and perform some linear algebra using c++ and here's a list of libraries I have encountered: Eigen Armadillo Dlib GNU Scientific library Please ... 4answers 309 views ### Statistics Before Linear Algebra? I know this is an opinion-based question and will be closed but this is the only place I know that can answer it reasonably and it is a very important matter to me. I am pursuing a machine ... 3answers 5k views ### How does tensor product/multiplication work in TensorFlow? In Tensorflow, I saw the following example: ... 1answer 45 views ### Can I use regression to solve a multiple equation problem I'm working on a problem which is a multiple equation. I have a group of people and each person in the group is working on different tasks (e.g. n tasks in total). Each person in this group is working ... 1answer 44 views ### On minimizing matrix norm (AB-C) Given A, B and C are matrices with dim(A) = m x n, dim(B) = n x p and dim (C) = m x p, the problem asks to evaluate I need to learn$$\tilde{A}$$such that$$\min_{\tilde{A}}||\tilde{A}^TB-C||$$and ... 1answer 55 views ### Least Squares Regression Ax=b when A is fixed and b is varied The typical setting for least squares regression (or over-determined linear system) for Ax=b is to solve x given A and b. In other words, A and b are fixed when we solve the problem. My ... 2answers 33 views ### Are euclidian vectors and unit vectors same thing? [closed] Consider this statement : Let the field K be the set R of real numbers, and let the vector space V be the Euclidean space R3. Consider the vectors e1 = (1,0,0), e2 = (0,1,0) and e3 = (0,0,1). Then any ... 1answer 141 views ### RNN: why Wx + Uh instead of W[x,h] Traditionally, a state for RNN is computed as$$h_t = \sigma(W\cdot \vec x + U\cdot \vec h_{t-1} + \vec b)$$For a RNN, why to add-up the terms$(Wx + Uh_{t-1})$instead of just having a single ... 2answers 51 views ### Derivates with respect to a vector Suppose I have an equation,$f = X^TY + \dots$(a few more terms), where$X$is a vector and$Y$is a matrix of appropriate dimensions, I want to know how can we take the derivative of$f \text{ w.r.t....
This answer shows that linear and polynomial function weights can be trained using this matrix operation: $w = (X^TX)^{-1}X^Ty$ Therefore, algorithms such as gradient descent are not necessary for ...