42
votes
Accepted
What is GELU activation?
GELU function
We can expand the cumulative distribution of $\mathcal{N}(0, 1)$, i.e. $\Phi(x)$, as follows:
$$\text{GELU}(x):=x{\Bbb P}(X \le x)=x\Phi(x)=0.5x\left(1+\text{erf}\left(\frac{x}{\sqrt{2}}...
- 9,382
33
votes
Accepted
What does it mean when we say most of the points in a hypercube are at the boundary?
Speaking of '$99\%$ of the points in a hypercube' is a bit misleading since a hypercube contains infinitely many points. Let's talk about volume instead.
The volume of a hypercube is the product of ...
- 1,646
16
votes
Do you actually need math for your data science job?
Having a solid mathematical background is crucial for data science. Someone without solid mathematical background will always use the algorithms as black box models.
Mathematical reasoning is needed ...
- 1,938
10
votes
Accepted
Beginner math books for Machine Learning
Although you need book, I recommend the following courses respectively for understanding statistics which are used for machine learning and other tasks in data science. They are free.
Learn ...
10
votes
What does it mean when we say most of the points in a hypercube are at the boundary?
You can see the pattern clearly even in lower dimensions.
1st dimension. Take a line of length 10 and a boundary of 1. The length of the boundary is 2 and the interior 8, 1:4 ratio.
2nd dimension. ...
- 201
10
votes
What is GELU activation?
First note that $$\Phi(x) = \frac12 \mathrm{erfc}\left(-\frac{x}{\sqrt{2}}\right) = \frac12 \left(1 + \mathrm{erf}\left(\frac{x}{\sqrt2}\right)\right)$$ by parity of $\mathrm{erf}$. We need to show ...
- 456
6
votes
Accepted
Estimating the value of $\pi$ with a Monte Carlo dartboard: $<$ or $\leq$?
Short answer: Both formulations lead to the same answer.
Mathematical explanation:
In order to understand that let us look at two similar problems. Imagine we want to integrate a function $f(x)=x^2$ ...
- 1,938
6
votes
Accepted
Why both ChatGPT and Bard can't get a simple matrix calculation right?
In my understanding, LLMs are, very simplified speaking, probabilistic solvers. Math problems such as matrix multiplications are, on the other hand, deterministic in nature. Thus, using a an LLM for ...
- 276
5
votes
Accepted
Help in understanding the maths behind Logistic Regression
This feels like a bit of a convoluted way to introduce the concept, but alright :D Let me start at a slightly different point.
Maybe in Machine Learning or in other places you have encountered the $...
- 1,555
5
votes
Accepted
Which is meant by +/-9.2e18 years in timespan?
From the documentation you referred: "The length of the span is the range of a 64-bit integer times the length of the date or unit."
64 bit integer has values from ...
- 353
4
votes
Accepted
How are the channels handled in CNN? Is it independently processed or fused?
Let $n$ be a convolutional layer with dimensions $w' \times h' \times c'$. Then each of its $c'$ filters is connected to all $c$ filters (or channels*) of the previous layer.
I find it helpful to ...
- 5,450
4
votes
Accepted
Is there such thing as linear and non-linear data?
Naming linear data or non-linear data is a bit misleading and wrong I would say. Instead, there is linear relation and non-linear relation between variables would be better and correct naming. It can ...
- 1,168
4
votes
Do you actually need math for your data science job?
Statistical knowledge or statistical thinking is useful or necessary to:
Understand, evaluate and pick appropriate metrics to use to evaluate the performance of models.
You need to understand the ...
- 191
4
votes
Do you actually need math for your data science job?
No, you don't need mathematics for data science in the same way that you need it for physics. As a data scientist, you won't be integrating a stress-energy tensor, or even solving a differential ...
- 141
4
votes
Accepted
AutoDiff on different operations?
I will address your question in a roundabout manner, but you will see why.
We can compute a gradient on any convolutional layer, no matter the dimension, because convolution is defined in a similar ...
- 281
3
votes
Accepted
How can positional encodings including a sine operation be linearly transformable for any offset?
I elected to ask this question on the Mathematics Stack Exchange and I thought it prudent to add the answer here:
https://math.stackexchange.com/q/3119882
From what I have learned from @Servaes, who ...
- 181
3
votes
What does the term "proportional to" mean in Bayes Equation?
It means that $P(\theta | y ) = kP(\theta) P(y | \theta)$, where $k$ is a constant that does not depend on $\theta$.
In fact, the Bayes Theorem states $P(\theta | y ) = \frac{P(\theta) P(y | \theta)}...
- 6,111
3
votes
Accepted
Knn and euclidean distance
You can think of examples as vectors in $\mathbb{R}^p$, where $p$ is the number of features. Two examples will be very similar if the distance between them is close to $0$ (in the extreme case, if two ...
- 6,111
3
votes
Accepted
Recreating the sum symbol using python
In most cases, I would go for NumPy.
Implement a Python function f(t) that calculates the $t$-th summand.
Then run
...
- 1,646
3
votes
Beginner math books for Machine Learning
Introduction to Linear Algebra is a good starting point. Make sure you are good with probability theory, linear algebra, and statistics. A very in depth knowledge may not be necessary, but having a ...
3
votes
Accepted
Neural networks, optimization math intuition
Note that $\frac{\partial L}{\partial \theta}$ is different from $\frac{\partial \theta}{\partial L}$. What you tried to describe seems to be $\frac{\partial L}{\partial \theta}$ where $\theta$ is a ...
- 3,128
3
votes
Accepted
SVM hyperplane margin
After we have
$$w^Tx + b = \pm \delta$$
We can always divide everything by $\delta$,
$$\left( \frac{w}{\delta}\right)^Tx + \left( \frac{b}{\delta}\right)=\pm1$$
Now, we can set $\tilde{w}=\frac{w}...
- 3,128
3
votes
Accepted
Understanding the algebra behind a specific partial derivative equation
We know that:
(1) $\frac{\partial}{\partial x}\big (f(x) + g(x) \big) = \frac{\partial}{\partial x}f(x) + \frac{\partial}{\partial x}g(x)$
(2) $\frac{\partial}{\partial x}a = 0$
Now,
\begin{align*}
&...
- 166
3
votes
Accepted
Derivative of MSE Cost Function
Any term $f$ that is not a function of $\theta_j$ in any equation will have a partial derivative $\frac{\partial}{\partial\theta_j}(f) = 0$. Importantly, no $x_i$, $y$ or $\theta_{i \ne j}$ depend in ...
- 29.1k
3
votes
Accepted
Does performing k-NN on the centroids of clusters obtained from k-means make sense mathematically?
It depends on what you mean by "viable".
What you are doing is reducing the resolution of your embedding space. You have n embeddings that you map to 4 ...
- 826
2
votes
Beginner math books for Machine Learning
Before doing my master in Analytics, I was suggested by my seniors to go through these couple of books to know more about Machine Learning and Statistics.
Namely:
Discovering statistics with SPSS/R -...
2
votes
Accepted
word2vec - log in the objective softmax function
No, the logartihm doesn't disappear. From the equation
,
When you want to calculate
,
it essentially means calculating ,
Now ,
So ,
as .
- 811
2
votes
Comparison between addition and multiplication function in deep neural network?
The observation is very interesting you report, since concatenation and addition are practically the same. A nice explanation can be found in https://distill.pub/2018/feature-wise-transformations/ .
- 931
2
votes
Accepted
Meaning of this notion in 0-1 loss?
Your understanding is correct.
This is known as the indicator function.
The indicator function of a subset $A$ of a set $X$ is a function
$$1_A(x)= \begin{cases}1, & x \in A \\ 0, & x \...
- 3,128
2
votes
Accepted
Can a single-layer ANN get XOR wrong?
Assuming the implementation is as simple as possible, with no advanced concepts, is it likely for something like this to happen or is it definitely an error in the implementation?
In my experience, ...
- 29.1k
Only top scored, non community-wiki answers of a minimum length are eligible