Skip to main content
Share Your Experience: Take the 2024 Developer Survey
42 votes
Accepted

What is GELU activation?

GELU function We can expand the cumulative distribution of $\mathcal{N}(0, 1)$, i.e. $\Phi(x)$, as follows: $$\text{GELU}(x):=x{\Bbb P}(X \le x)=x\Phi(x)=0.5x\left(1+\text{erf}\left(\frac{x}{\sqrt{2}}...
Esmailian's user avatar
  • 9,322
33 votes
Accepted

What does it mean when we say most of the points in a hypercube are at the boundary?

Speaking of '$99\%$ of the points in a hypercube' is a bit misleading since a hypercube contains infinitely many points. Let's talk about volume instead. The volume of a hypercube is the product of ...
Elias Strehle's user avatar
16 votes

Do you actually need math for your data science job?

Having a solid mathematical background is crucial for data science. Someone without solid mathematical background will always use the algorithms as black box models. Mathematical reasoning is needed ...
MachineLearner's user avatar
10 votes
Accepted

Beginner math books for Machine Learning

Although you need book, I recommend the following courses respectively for understanding statistics which are used for machine learning and other tasks in data science. They are free. Learn ...
10 votes

What does it mean when we say most of the points in a hypercube are at the boundary?

You can see the pattern clearly even in lower dimensions. 1st dimension. Take a line of length 10 and a boundary of 1. The length of the boundary is 2 and the interior 8, 1:4 ratio. 2nd dimension. ...
HP Williams's user avatar
10 votes

What is GELU activation?

First note that $$\Phi(x) = \frac12 \mathrm{erfc}\left(-\frac{x}{\sqrt{2}}\right) = \frac12 \left(1 + \mathrm{erf}\left(\frac{x}{\sqrt2}\right)\right)$$ by parity of $\mathrm{erf}$. We need to show ...
BookYourLuck's user avatar
6 votes
Accepted

Estimating the value of $\pi$ with a Monte Carlo dartboard: $<$ or $\leq$?

Short answer: Both formulations lead to the same answer. Mathematical explanation: In order to understand that let us look at two similar problems. Imagine we want to integrate a function $f(x)=x^2$ ...
MachineLearner's user avatar
6 votes
Accepted

Why both ChatGPT and Bard can't get a simple matrix calculation right?

In my understanding, LLMs are, very simplified speaking, probabilistic solvers. Math problems such as matrix multiplications are, on the other hand, deterministic in nature. Thus, using a an LLM for ...
Eulenfuchswiesel's user avatar
5 votes
Accepted

Help in understanding the maths behind Logistic Regression

This feels like a bit of a convoluted way to introduce the concept, but alright :D Let me start at a slightly different point. Maybe in Machine Learning or in other places you have encountered the $...
matthiaw91's user avatar
  • 1,545
5 votes
Accepted

Which is meant by +/-9.2e18 years in timespan?

From the documentation you referred: "The length of the span is the range of a 64-bit integer times the length of the date or unit." 64 bit integer has values from ...
Kirill Fedyanin's user avatar
4 votes
Accepted

How are the channels handled in CNN? Is it independently processed or fused?

Let $n$ be a convolutional layer with dimensions $w' \times h' \times c'$. Then each of its $c'$ filters is connected to all $c$ filters (or channels*) of the previous layer. I find it helpful to ...
Jonathan's user avatar
  • 5,420
4 votes
Accepted

Is there such thing as linear and non-linear data?

Naming linear data or non-linear data is a bit misleading and wrong I would say. Instead, there is linear relation and non-linear relation between variables would be better and correct naming. It can ...
Shahriyar Mammadli's user avatar
4 votes

Do you actually need math for your data science job?

Statistical knowledge or statistical thinking is useful or necessary to: Understand, evaluate and pick appropriate metrics to use to evaluate the performance of models. You need to understand the ...
NotThatGuy's user avatar
4 votes

Do you actually need math for your data science job?

No, you don't need mathematics for data science in the same way that you need it for physics. As a data scientist, you won't be integrating a stress-energy tensor, or even solving a differential ...
Alexander Hanysz's user avatar
4 votes
Accepted

AutoDiff on different operations?

I will address your question in a roundabout manner, but you will see why. We can compute a gradient on any convolutional layer, no matter the dimension, because convolution is defined in a similar ...
AndrewJaeyoung's user avatar
3 votes
Accepted

How can positional encodings including a sine operation be linearly transformable for any offset?

I elected to ask this question on the Mathematics Stack Exchange and I thought it prudent to add the answer here: https://math.stackexchange.com/q/3119882 From what I have learned from @Servaes, who ...
Stephan Heijl's user avatar
3 votes

What does the term "proportional to" mean in Bayes Equation?

It means that $P(\theta | y ) = kP(\theta) P(y | \theta)$, where $k$ is a constant that does not depend on $\theta$. In fact, the Bayes Theorem states $P(\theta | y ) = \frac{P(\theta) P(y | \theta)}...
David Masip's user avatar
  • 6,081
3 votes
Accepted

Knn and euclidean distance

You can think of examples as vectors in $\mathbb{R}^p$, where $p$ is the number of features. Two examples will be very similar if the distance between them is close to $0$ (in the extreme case, if two ...
David Masip's user avatar
  • 6,081
3 votes
Accepted

Recreating the sum symbol using python

In most cases, I would go for NumPy. Implement a Python function f(t) that calculates the $t$-th summand. Then run ...
Elias Strehle's user avatar
3 votes

Beginner math books for Machine Learning

Introduction to Linear Algebra is a good starting point. Make sure you are good with probability theory, linear algebra, and statistics. A very in depth knowledge may not be necessary, but having a ...
3 votes
Accepted

Neural networks, optimization math intuition

Note that $\frac{\partial L}{\partial \theta}$ is different from $\frac{\partial \theta}{\partial L}$. What you tried to describe seems to be $\frac{\partial L}{\partial \theta}$ where $\theta$ is a ...
Siong Thye Goh's user avatar
3 votes
Accepted

SVM hyperplane margin

After we have $$w^Tx + b = \pm \delta$$ We can always divide everything by $\delta$, $$\left( \frac{w}{\delta}\right)^Tx + \left( \frac{b}{\delta}\right)=\pm1$$ Now, we can set $\tilde{w}=\frac{w}...
Siong Thye Goh's user avatar
3 votes
Accepted

Understanding the algebra behind a specific partial derivative equation

We know that: (1) $\frac{\partial}{\partial x}\big (f(x) + g(x) \big) = \frac{\partial}{\partial x}f(x) + \frac{\partial}{\partial x}g(x)$ (2) $\frac{\partial}{\partial x}a = 0$ Now, \begin{align*} &...
mujjiga's user avatar
  • 166
3 votes
Accepted

Derivative of MSE Cost Function

Any term $f$ that is not a function of $\theta_j$ in any equation will have a partial derivative $\frac{\partial}{\partial\theta_j}(f) = 0$. Importantly, no $x_i$, $y$ or $\theta_{i \ne j}$ depend in ...
Neil Slater's user avatar
3 votes
Accepted

Does performing k-NN on the centroids of clusters obtained from k-means make sense mathematically?

It depends on what you mean by "viable". What you are doing is reducing the resolution of your embedding space. You have n embeddings that you map to 4 ...
Karl's user avatar
  • 746
2 votes

Beginner math books for Machine Learning

Before doing my master in Analytics, I was suggested by my seniors to go through these couple of books to know more about Machine Learning and Statistics. Namely: Discovering statistics with SPSS/R -...
2 votes
Accepted

word2vec - log in the objective softmax function

No, the logartihm doesn't disappear. From the equation , When you want to calculate , it essentially means calculating , Now , So , as .
Gyan Ranjan's user avatar
2 votes

Comparison between addition and multiplication function in deep neural network?

The observation is very interesting you report, since concatenation and addition are practically the same. A nice explanation can be found in https://distill.pub/2018/feature-wise-transformations/ .
Andreas Look's user avatar
2 votes
Accepted

Meaning of this notion in 0-1 loss?

Your understanding is correct. This is known as the indicator function. The indicator function of a subset $A$ of a set $X$ is a function $$1_A(x)= \begin{cases}1, & x \in A \\ 0, & x \...
Siong Thye Goh's user avatar
2 votes
Accepted

Can a single-layer ANN get XOR wrong?

Assuming the implementation is as simple as possible, with no advanced concepts, is it likely for something like this to happen or is it definitely an error in the implementation? In my experience, ...
Neil Slater's user avatar

Only top scored, non community-wiki answers of a minimum length are eligible