Skip to main content

Questions tagged [linear-algebra]

A field of mathematics concerned with the study of finite dimensional vector spaces, including matrices and their manipulation, which are important in statistics.

Filter by
Sorted by
Tagged with
2 votes
1 answer
48 views
+50

What type of technique can be used to solve this question?

Apology for the ambiguous title, I do not know the term. I have data of some products which a few variables: origin, weight, brand. For example: Product A = "China, 100g, Brand X" Product ...
lpounng's user avatar
  • 998
1 vote
0 answers
17 views

Outlier detection with elliptic envelope - unexpected error

I am trying to detect outliers with sklearn.covariance.EllipticEnvelope for a single variable, but it throws an unexpected error. Here is an example the reproduces ...
Maya's user avatar
  • 11
1 vote
2 answers
65 views

Affine 2D mapping in python

I have two sets of 2D data $A$ and $B$ (representing 2D positions on a 2D plane $x,y$) which are related (the first pair of $x,y$ of $A$ is related to the first pair of $x,y$ of $B$ for instance). I ...
patjol's user avatar
  • 11
0 votes
0 answers
39 views

Multiplying by Diagonal Matrix On Top of Standard Linear Regression

In minimizing $$min_x||Ax-b||$$ where $A$ is overdetermined, one could use least squares method. However, if there is another diagonal matrix $d$ which has $k$ unique entries along the diagonal with $...
Trevor Arashiro's user avatar
0 votes
0 answers
45 views

How can I calculate the data energy loss after PCA?

Recently in a slide in about PCA (Principal Component Analysis) I saw a question: "How much is the data energy loss in PCA?&...
hasanghaforian's user avatar
0 votes
0 answers
7 views

Dual of the SCM square hinge loss

Let $x_1,\dots,x_n\in \mathbb{R}^n$, $y_1,\dots,y_n\in \{-1,1\}$, $\lambda \ge 0$ and $K$ be the invertible Gram matrix $K=(x_i\cdot x_j)_{ij}$. Consider $$ (P) \qquad \qquad \min_{a\in \mathbb{R}^n} \...
Smilia's user avatar
  • 101
0 votes
0 answers
16 views

Can reducing information improve regression prediction?

Variable A is either 0 or 1. It is 0 if the sum of variables a + b + c + d … is less than some constant threshold, and is 1 if the sum of variables a + b + c + d … is greater than some constant ...
BigMistake's user avatar
0 votes
0 answers
23 views

Calculating the solution of OLS efficiently when adding one feature at a time

We know that the analytical solution for an OLS problem is $𝛽̂ =(𝐗^T𝐗)^{-1}𝐗^𝑇𝐲$. I am looking for an efficient algorithm to solve for $𝛽̂$ when I add one feature at a time. More specifically, ...
Ali s.k's user avatar
  • 101
0 votes
1 answer
151 views

About the last decoder layer in transformer architecture

So, in the decoder layer of transfomer, suppose I have predicted 3 words till now, including the start token then the last decoder layer will produce 3 vectors of size d-model, and only the last ...
Nishan Poudel's user avatar
0 votes
1 answer
950 views

How is weight matrix calculated in a neural network?

Context: I am a pure mathematician trying to understand machine learning. I am studying it from various sources, now focusing on NLP and word embeddings. My question: What is the weight matrix for a ...
Tereza Tizkova's user avatar
0 votes
1 answer
30 views

What (in the world) is well-conditioned vs. low rank fat-tail singular profile?

Scikit learn has a make_regression data generator. Can someone explain it to me like I'm 5 what is meant in the help docs by "The input set can either be well ...
Snehal Patel's user avatar
0 votes
1 answer
32 views

Proof of perpendicular distance of an observation from the Maximal Margin Hyperplane

I was reading about Maximal Margin Classifiers in "Introduction to Statistical Learning" and could not understand how is the perpendicular distance of an observation (which is a vector) from ...
Circuit_Breaker0.7's user avatar
1 vote
1 answer
56 views

Plot a matrix as a single point in space

I have a dataset of drugs represented as a graph, each of which is described by three non-square matrices: edge index (A), an 2xe matrix, where e are the bonds of the molecule, the first line ...
Gianmarco Luchetti 's user avatar
0 votes
1 answer
63 views

Beginner Question on Understanding Linear Classifier

I have been trying to understand the math behind Linear classifier for images and I'm hitting a roadblock to understanding this image below: I can to some extent agree that we stretch the pixels into ...
joesan's user avatar
  • 137
3 votes
0 answers
239 views

Multi-dimensional Euclidian R^2 squared - reasonable?

I have a high-dimensional space, say $\mathbb{R}^{1000}$, and I have samples $y_1, \ldots , y_n \in \mathbb{R}^{1000}$ and $\hat{y}_1, \ldots , \hat{y}_n \in \mathbb{R}^{1000}$. Would $$ R^2 = 1 - \...
AspiringToAspire's user avatar
0 votes
1 answer
36 views

What what will happen if all the layers of a MLP or any DL architecture are set as same in the beginning?

Setting the initial weights as all zeros will have the output dependent on the bias and setting the weights of all the neurons of a layer as same, will update the gradients in same way thus removing ...
Deshwal's user avatar
  • 323
1 vote
0 answers
19 views

Nearest neighbor face recognition in eigenspace when using dot product of test set with eigenvectors does not match the performance when using sklearn

I am trying to perform Face recognition using PCA (eigenfaces). I have a set of N training images (of dimensions M=wxh), which I have pre-processed into a vertical ...
zr0gravity7's user avatar
0 votes
1 answer
94 views

If an SVM decision boundary is the perpendicular bisector of the line connecting the support vectors, why iterate for it using a loss function?

Would it not make more sense to do some linear algebra to find the vector of the decision boundary? Is that more computationally expensive?
user132380's user avatar
1 vote
1 answer
175 views

Why linear model cannot understand the interaction between any two input features?

The book Deep Learning by Ian Goodfellow states that: ...
Gull Noor's user avatar
0 votes
0 answers
22 views

NCHW input matrix to Dm conversion logic for convolution in cuDNN

I have been trying to understand the convolution lowering operation shown in the cuDNN paper. I was able to understand most of it by reading through and mapping various parameters to the image below. ...
Rajesh Shashi Kumar's user avatar
1 vote
0 answers
54 views

Pseudo inverse of the covariance matrix?

I've been looking for methods to compute a pseudo inverse of a covariance matrix. And found that one way is to construct a regularized inverse matrix. By constructing the eigen system, and removing ...
nimo96's user avatar
  • 11
5 votes
0 answers
85 views

How is image convolution actually implemented in deep learning libraries using simple linear algebra?

As a clarifier, I want to implement cross-correlation, but the machine learning literature keeps referring to it as convolution so I will stick with it. I am trying to implement image convolution ...
Jozef Nagy's user avatar
1 vote
1 answer
226 views

Understanding Lagrangian equation for SVM

I was trying to understand Lagrangian from SVM section of Andrew Ng's Stanford CS229 course notes. On page 17 and 18, he says: Given the problem $$\begin{align} min_w & \quad f(w) \\ s.t. &...
Mahesha999's user avatar
0 votes
1 answer
87 views

Understanding SVM mathematics

I was referring SVM section of Andrew Ng's course notes for Stanford CS229 Machine Learning course. On pages 14 and 15, he says: Consider the picture below: How can we find the value of $\gamma^{(i)}...
Rnj's user avatar
  • 225
1 vote
1 answer
125 views

What are some application of Google's PageRank Algorithm in Data Science

I came across a topic on computational linear algebra that talks about iterative algorithms to compute eigenvalues. I've worked with power method which is an iterative algorithm that converges a ...
SPARSE's user avatar
  • 115
0 votes
0 answers
180 views

Inverting a matrix using a convolutional neural network

Just for a fun exercise, I am trying to invert a matrix, say size 28x28 (or even 5x5) with a neural network. The way I approached this (quite naively) is as follows: I built a fully convolutional ...
Ashesh's user avatar
  • 90
-1 votes
1 answer
68 views

Why do we use 'T' when we are to say matrix-vector product? [closed]

On the first picture author uses $T$ meaning matrix-vector product But other website do not use $T$, but says that $x$ is a vector, I do not understand if it is important or not
Vladislav Kruglikov's user avatar
2 votes
1 answer
4k views

3d input for Dense Layer Keras

Is there any example of how Keras Dense layer handles 3D input. The documentation explains the following: ...
data_person's user avatar
2 votes
1 answer
2k views

Difference between FDA and LDA

I have asked this question in Mathematics Stackexchange, thought however that it might be more fit for here: I am currently taking a Data-Analysis course and I learned about both the terms LDA (Linear ...
Nestroy's user avatar
  • 21
0 votes
0 answers
88 views

What is the Intuition behind weight vector W which is normal to the plane? Is the weight vector W same as the W which is normal to the plane Ο€?

In an interview, I was asked the intuition behind the weight vector. I told the weight vector is a vector which we try to minimize to a local minima with the help of regulariser so we don't overfit. ...
Vinit Sutar's user avatar
1 vote
1 answer
730 views

How does the equation "dW = - (2 * (X^T ).dot(Y - Y_hat)) / m" comes in Linear Regression (using Matrix + Gradient Descent)?

I was trying to code the Linear Regression in Python using Matrix Multiplication method using Gradient Descent and followed a code where there was no mention what is the loss but just a code as Per ...
Deshwal's user avatar
  • 323
0 votes
1 answer
38 views

How does tree-based algorithms handle linearly combined features?

While I am aware that tree-based algorithms (e.g., DT, RF, XGBoost) are 'immune' to multi-collinearity, how do they handle linearly combined features? For example, is there is any additional value or ...
thereandhere1's user avatar
2 votes
0 answers
151 views

Geometric classification models

In class we have been presented with a Geometric classification model such that the goal is to construct a linear decision boundary $\bf{w} \cdot \bf{x} = t$; where $\bf{w}$ is the vector from the ...
Zafir Stojanovski's user avatar
1 vote
1 answer
1k views

Dot product and linear regression

I'm studying PCA and my professor said something about finding the linear regression by doing the dot product of both axis. Could someone explain to me why? The dot product returns a number. What's ...
leoperassoli's user avatar
0 votes
0 answers
68 views

Why do we usually have fully connected layers of same sizes in CNNs?

Is there any specific reason that we observe in CNNs, the fully connected layers usually have the same sizes? You can verify this for many CNNs. I'm aware that if, for instance, we have a vector of ...
Green Falcon's user avatar
  • 14.1k
1 vote
1 answer
240 views

Hypothesis vs Hyperplane in Machine Learning

I am finding it hard to understand the clear difference between Hypothesis and Hyperplane. I know that Hypothesis is a candidate model that maps inputs to outputs after training. And, Hyperplane is ...
AnonymousMe's user avatar
1 vote
1 answer
175 views

Confusion with Notation in the Book on Deep Learning by Ian Goodfellow et al

In chapter 6.1 on 'Example: Learning XOR', the bottom of page 168 mentions: The activation function $g$ is typically chosen to be a function that is applied element-wise, with $h_i = g(x^TW_{:,i}+c_i)...
KGhatak's user avatar
  • 123
2 votes
0 answers
118 views

Deriving vectorized form of linear regression

We first have the weights of a D dimensional vector $w$ and a D dimensional predictor vector $x$, which are all indexed by $j$. There are $N$ observations, all D dimensional. $t$ is our targets, i.e, ...
user2793618's user avatar
1 vote
2 answers
3k views

Removing constant from the regression model

I am trying to calibrate two variables $(X,Y)$ of different measuring techniques from two instruments, the result of the linear regression analysis appears as shown in the image. The result shows the ...
Yazan Alatoom's user avatar
0 votes
1 answer
35 views

Understanding the algebra behind a specific partial derivative equation

I am following this article about neural networks. Given: Until here I understand everything, but then he continues to: I don't understand how he got to that conclusion. I think he skipped some ...
Alon's user avatar
  • 23
1 vote
2 answers
5k views

Dose finding slope/intercept using the formula of m,b gives best fit line always In linear regression?

In liner regression We have to fit different lines and chose one with minimum error so What is the motive of having a formula for m,b that can give slope and intercept value in the regression line ,...
star's user avatar
  • 1,471
0 votes
1 answer
83 views

Normal equation for linear regression is illogical

Currently I'm taking Andrew Ng's course. He gives a following formula to find solution for linear regression analytically: $ΞΈ = (X^T * X)^{-1} * X^T * Ρƒ$ He doesn't explain it so I searched for it and ...
Martian's user avatar
  • 123
1 vote
1 answer
452 views

When is it useful to measure the Frobenius norm of a matrix?

In Deep Learning section 2.5 the author review some measures for the size of vectors and matrices. When in general is it useful for someone to know these? For instance they give the example of the ...
nid's user avatar
  • 131
2 votes
0 answers
66 views

Intuition behind understanding eigenvectors and Machine Learning [closed]

I am struggling to understand linear algebra application in machine learning, and I am not able to answer the following question. Is the model learned in Machine Learning the eigenvector of the ...
Hermes Morales's user avatar
1 vote
1 answer
44 views

Can all known ML algorithms be written as a sequence of matrix operations?

I keep hearing that machine learning is just linear algebra. Does that mean all known (and all possible?) ML algos, from random forest, to support-vector machines, to recursive neural networks, can ...
SubstantialRange's user avatar
2 votes
4 answers
487 views

How to solve Ax = b for A [closed]

Given two know vector x, and b (dimension 3*1 for example), what are the ways to approximate the matrix ...
silkAdmin's user avatar
  • 143
1 vote
3 answers
589 views

Why transpose of independent feature matrix is necessary in case of linear regression?

I can follow classical linear regression steps: $Xw=y$ $X^{-1}Xw=X^{-1}y$ $Iw=X^{-1}y$ $w=X^{-1}y$ However, on implementing in Python, I see that instead of simply using ...
Fredrik's user avatar
  • 1,007
1 vote
2 answers
964 views

Linear regression with a fixed intercept and everything is in log

I have a set of values for a surface (in pixels) that becomes bigger over time (exponentially). The surface consists of cells that divide over time. After doing some modelling, I came up with the ...
a0142204's user avatar
3 votes
2 answers
59 views

Backpropagation with a different sized training set?

I'm trying to create a NN whose input is a (length m) array of 3d vectors $$\vec{x}_i = [x_{i,1},x_{i,2},x_{i,3}], \hspace{5mm}i=1:m $$ and whose output is a similarly sized array: $$\vec{h}_{\theta,...
Spinach's user avatar
  • 31
1 vote
0 answers
57 views

What's wrong with my backpropagation through time (BTT) calculation or how to multiple a scaled vector and a matrix without matching dimensions?

I am trying to make a pretty simple RNN from scracth, using only Numpy library of Python. At this moment I am having troubles with BTT as I do not know how to proceed with situation when a ...
été's user avatar
  • 29