34

GELU function We can expand the cumulative distribution of $\mathcal{N}(0, 1)$, i.e. $\Phi(x)$, as follows: $$\text{GELU}(x):=x{\Bbb P}(X \le x)=x\Phi(x)=0.5x\left(1+\text{erf}\left(\frac{x}{\sqrt{2}}\right)\right)$$ Note that this is a definition, not an equation (or a relation). Authors have provided some justifications for this proposal, e.g. a ...


32

Speaking of '$99\%$ of the points in a hypercube' is a bit misleading since a hypercube contains infinitely many points. Let's talk about volume instead. The volume of a hypercube is the product of its side lengths. For the 50-dimensional unit hypercube we get $$\text{Total volume} = \underbrace{1 \times 1 \times \dots \times 1}_{50 \text{ times}} = 1^{50} =...


10

Although you need book, I recommend the following courses respectively for understanding statistics which are used for machine learning and other tasks in data science. They are free. Learn Statistics - Intro to Statistics Course Intro to Descriptive Statistics Inferential Statistics: Learn Statistical Analysis If I want to recommend a book, I would ...


10

You can see the pattern clearly even in lower dimensions. 1st dimension. Take a line of length 10 and a boundary of 1. The length of the boundary is 2 and the interior 8, 1:4 ratio. 2nd dimension. Take a square of side 10, and boundary 1 again. The area of the boundary is 36, the interior 64, 9:16 ratio. 3rd dimension. Same length and boundary. The volume ...


10

First note that $$\Phi(x) = \frac12 \mathrm{erfc}\left(-\frac{x}{\sqrt{2}}\right) = \frac12 \left(1 + \mathrm{erf}\left(\frac{x}{\sqrt2}\right)\right)$$ by parity of $\mathrm{erf}$. We need to show that $$\mathrm{erf}\left(\frac x {\sqrt2}\right) \approx \tanh\left(\sqrt{\frac2\pi} \left(x + a x^3\right)\right)$$ for $a \approx 0.044715$. For large values ...


6

Short answer: Both formulations lead to the same answer. Mathematical explanation: In order to understand that let us look at two similar problems. Imagine we want to integrate a function $f(x)=x^2$ on two intervals $I_1=[0,1]$ (including both bounds) and $I_2=(0,1)$ (excluding both bounds). For $I_1$ we have $$\int_0^1 x^2~dx=1/3.$$ For the second ...


5

This feels like a bit of a convoluted way to introduce the concept, but alright :D Let me start at a slightly different point. Maybe in Machine Learning or in other places you have encountered the $sigmoid$ function: $$ sigmoid(S) = \frac{e^S}{1+e^S} = \frac{1}{1+e^{-S}} $$ The sigmoid has the nice property to map any real number $S$ to a number between 0 ...


3

Note that $\frac{\partial L}{\partial \theta}$ is different from $\frac{\partial \theta}{\partial L}$. What you tried to describe seems to be $\frac{\partial L}{\partial \theta}$ where $\theta$ is a variable. If $\theta$ is high dimensional, sometimes we just use the $\nabla$ notation. Gradient descent is $$\theta_{n+1}=\theta_n-\gamma \nabla L(\theta_n)...


3

It means that $P(\theta | y ) = kP(\theta) P(y | \theta)$, where $k$ is a constant that does not depend on $\theta$. In fact, the Bayes Theorem states $P(\theta | y ) = \frac{P(\theta) P(y | \theta)}{P(y)}$. This proportion has to be added in order that $\sum_{\theta}P(\theta | y) = 1$ is satisfied (that is, $P(\theta | y)$ has to be a probability).


3

You can think of examples as vectors in $\mathbb{R}^p$, where $p$ is the number of features. Two examples will be very similar if the distance between them is close to $0$ (in the extreme case, if two examples are equal their euclidean distance is $0$). One way to measure the distance is using euclidean distance, but other distances can be used, as cosine ...


3

In most cases, I would go for NumPy. Implement a Python function f(t) that calculates the $t$-th summand. Then run import numpy as np result = np.array([f(t) for t in range(1,m+1)]).sum() This will be very fast, unless $m$ is so large that [f(t) for t in range(1,m+1)] does not fit into memory. In this case, I would follow your approach and use a for-loop: ...


3

Introduction to Linear Algebra is a good starting point. Make sure you are good with probability theory, linear algebra, and statistics. A very in depth knowledge may not be necessary, but having a good knowledge is required.


3

After we have $$w^Tx + b = \pm \delta$$ We can always divide everything by $\delta$, $$\left( \frac{w}{\delta}\right)^Tx + \left( \frac{b}{\delta}\right)=\pm1$$ Now, we can set $\tilde{w}=\frac{w}{\delta}$ and $\tilde{b}=\frac{b}{\delta}$. $$\tilde{w}^Tx+\tilde{b}=\pm1$$ This is as if we have set $\delta=1$ from the beginning. The derivation of the ...


3

We know that: (1) $\frac{\partial}{\partial x}\big (f(x) + g(x) \big) = \frac{\partial}{\partial x}f(x) + \frac{\partial}{\partial x}g(x)$ (2) $\frac{\partial}{\partial x}a = 0$ Now, \begin{align*} &\frac{\partial}{\partial w_{12}^{1}} (w_{11}^{1}h_1^{2} + w_{12}^{1}h_2^{2} + w_{13}^{1}h_3^{2} + b_1^{1}) = & \text{[using (1)]}\\ &\frac{\partial}{\...


3

Naming linear data or non-linear data is a bit misleading and wrong I would say. Instead, there is linear relation and non-linear relation between variables would be better and correct naming. It can easily be explained by example from real life. Non-linear relation: Any kind of relationship that is not linear can be put forward as an example (e.g. ...


2

Before doing my master in Analytics, I was suggested by my seniors to go through these couple of books to know more about Machine Learning and Statistics. Namely: Discovering statistics with SPSS/R - Andy Field R Beginner and R for Everyone Predictive Analytics - The Power to Predict Who Will Click, Buy, Lie or Die Data Science for Business and many more ...


2

Your understanding is correct. This is known as the indicator function. The indicator function of a subset $A$ of a set $X$ is a function $$1_A(x)= \begin{cases}1, & x \in A \\ 0, & x \notin A \end{cases}$$


2

No, the logartihm doesn't disappear. From the equation , When you want to calculate , it essentially means calculating , Now , So , as .


2

Assuming the implementation is as simple as possible, with no advanced concepts, is it likely for something like this to happen or is it definitely an error in the implementation? In my experience, using the simplest possible network, and simplest gradient descent algorithm, then yes this happens relatively frequently. It is an accident of the starting ...


2

What is $t$? It means observed $r_{ui}$ in the one-week test set (page 6-left). Is my understanding of the metric correct? First two examples are correct. Assuming user-item relation $r_{ui}^t$ is constant $a$ for all items in the test set, and predicted ranks are uniform across $[0, 1]$, then, the third one would be: $$\overline{\text{rank}} = \frac{\...


2

As Elias mentioned, the expectations are related to random variables and you would be good to go if you know about conditional probability, multivariate probability, joint and marginal distributions. I would suggest you take a course that has a syllabus on the lines of https://secure.oregonstate.edu/ap/cps/documents/view/134169.


2

A simple and direct answer is that skewness and kurtosis are both defined in terms of the $Z-$values, $Z = (X-\mu)/\sigma$ : Skewness = $E(Z^3)$ and Kurtosis = $E(Z^4)$. When talking about a data set, you can just replace the expectation operator "$E$" with the ordinary average. Since raising a number to either the third or fourth power amplifies values ...


2

I think the intuition here is that you want to push the negative loglikelihood (NLL) * the reward to be as negative as possible. Since our reward is often not differentiable because it is obtained through sampling, we can only change the NLL. For actions with high reward, we have greater "pressure" / gradient to push the NLL down, than if the ...


2

Yes, the line indicates that KNN is weighted and that the weight is the inverse of the distance. All of this can easily be found in scikit-learn's documentation Also, pro-tip, you can find an object's documentation using the help function. In this case: from sklearn.neighbors import KNeighborsClassifier print(help(KNeighborsClassifier)) As to how ...


2

You're using the Mean Square Error $\Sigma\frac{1}{N}(y-(Wx+b))^2$as the loss function, if you take the derivative, you will have the $2$. In some materials, we will use $\frac{1}{2}\Sigma\frac{1}{N}(y-(Wx+b))^2$ as the loss function to cancel out the $2$. In fact, this doesn't matter at all and it has no impacts on params optimization.


2

There can be no objective answer to this question. Obviously the more one understands the better, but the field of ML is vast, quite specialized and ranges from very theoretical to very applied research, so it's perfectly reasonable to publish in ML without a strong background in maths. A better way to estimate your own ability to publish papers in a ...


2

Let $n$ be a convolutional layer with dimensions $w' \times h' \times c'$. Then each of its $c'$ filters is connected to all $c$ filters (or channels*) of the previous layer. I find it helpful to look at the number of weights here: A single filter of that convolutional layer $n$ with kernel size $k'\times k'$ will have $c \times k' \times k'$ weights. And ...


2

Yes and no The YES Let's consider at two distributions $F_X(x)$ and $G_Y(y)$ whose joint distribution is simply the product of the the two distributions: $H_{X,Y}(x,y) = F_X(x)G_Y(y)$. This means that the distributions are 100% independent: not just linearly (no correlation), but that there is absolutely no dependence between the two distributions. (Copula,...


2

At a higher level, using multiple activation functions in a NN may not work as these activation functions operate differently for the same inputs. For an input of -1.56, ReLU will a 0, sigmoid will give a 0.174 and Tanh will give -0.91. These differences will not allow the gradients to flow uniformly during backpropagation. You may try using two different ...


2

Reading the article they state that The two vertical lines represent the $L^2$ norm of the error, or what is known as the sum-of-squares error (SSE) Otherwise known as the Euclidean norm, Euclidean length, $L^2$ distance, $ℓ^2$ distance, $L^2$ norm, $ℓ^2$ norm, $2$-norm, or square norm. Also speaking from my experience, most of the time when a norm is ...


Only top voted, non community-wiki answers of a minimum length are eligible