143

A "dead" ReLU always outputs the same value (zero as it happens, but that is not important) for any input. Probably this is arrived at by learning a large negative bias term for its weights. In turn, that means that it takes no role in discriminating between inputs. For classification, you could visualise this as a decision plane outside of all possible ...


122

UPDATE: the landscape has changed quite a bit since I answered this question in July '14, and some new players have entered the space. In particular, I would recommend checking out: TensorFlow Blocks Lasagne Keras Deepy Nolearn NeuPy They each have their strengths and weaknesses, so give them all a go and see which best suits your use case. Although I ...


114

Let's review how the ReLU (Rectified Linear Unit) looks like : The input to the rectifier for some input $x_n$ is $$z_n=\sum_{i=0}^k w_i a^n_i$$ for weights $w_i$, and activations from the previous layer $a^n_i$ for that particular input $x_n$. The rectifier neuron function is $ReLU = max(0,z_n)$ Assuming a very simple error measure $$error = ReLU - y$$ ...


108

One way to interpret cross-entropy is to see it as a (minus) log-likelihood for the data $y_i'$, under a model $y_i$. Namely, suppose that you have some fixed model (a.k.a. "hypothesis"), which predicts for $n$ classes $\{1,2,\dots, n\}$ their hypothetical occurrence probabilities $y_1, y_2,\dots, y_n$. Suppose that you now observe (in reality) $k_1$ ...


106

You could just use sklearn.model_selection.train_test_split twice. First to split to train, test and then split train again into validation and train. Something like this: X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=1) X_train, X_val, y_train, y_val = train_test_split(X_train, y_train, test_size=0.25, ...


97

Some real important differences to consider when you are choosing R or Python over one another: Machine Learning has 2 phases. Model Building and Prediction phase. Typically, model building is performed as a batch process and predictions are done realtime. The model building process is a compute intensive process while the prediction happens in a jiffy. ...


75

Is the learning rate related to the shape of the error gradient, as it dictates the rate of descent? In plain SGD, the answer is no. A global learning rate is used which is indifferent to the error gradient. However, the intuition you are getting at has inspired various modifications of the SGD update rule. If so, how do you use this information to inform ...


72

I recently found this online tool that produces publication-ready NN-architecture schematics. It is called NN-SVG and made by Alex Lenail. You can easily export these to use in, say, LaTeX for example. Here are a few examples: AlexNet style LeNet style and the good old Fully Connected style


70

Really great question, and one that I find that most people don't really understand on an intuitive level. AUC is in fact often preferred over accuracy for binary classification for a number of different reasons. First though, let's talk about exactly what AUC is. Honestly, for being one of the most widely used efficacy metrics, it's surprisingly obtuse to ...


69

The cross entropy formula takes in two distributions, $p(x)$, the true distribution, and $q(x)$, the estimated distribution, defined over the discrete variable $x$ and is given by $$H(p,q) = -\sum_{\forall x} p(x) \log(q(x))$$ For a neural network, the calculation is independent of the following: What kind of layer was used. What kind of activation was ...


63

In most of the well-established machine learning systems, categorical variables are handled naturally. For example in R you would use factors, in WEKA you would use nominal variables. This is not the case in scikit-learn. The decision trees implemented in scikit-learn uses only numerical features and these features are interpreted always as continuous ...


61

Good question. Actually there is a lot of confusion on this subject, mainly because both are quite new jobs. But if we focus on the semantics, the real meaning of the jobs become clear. Beforehand is better to compare apples with apples, talking about a single subject, the Data. Machine Learning and its sub-genre (Deep Learning, etc.) are just one aspect of ...


59

K-means is not the most appropriate algorithm here. The reason is that k-means is designed to minimize variance. This is, of course, appearling from a statistical and signal procssing point of view, but your data is not "linear". Since your data is in latitude, longitude format, you should use an algorithm that can handle arbitrary distance functions, in ...


58

Gini impurity and Information Gain Entropy are pretty much the same. And people do use the values interchangeably. Below are the formulae of both: $\textit{Gini}: \mathit{Gini}(E) = 1 - \sum_{j=1}^{c}p_j^2$ $\textit{Entropy}: H(E) = -\sum_{j=1}^{c}p_j\log p_j$ Given a choice, I would use the Gini impurity, as it doesn't require me to compute logarithmic ...


53

Your loss function would not work because it incentivizes setting $\theta_1$ to any finite value and $\theta_0$ to $-\infty$. Let's call $r(x,y)=\frac{1}{m}\sum_{i=1}^m {h_\theta\left(x^{(i)}\right)} -y$ the residual for $h$. Your goal is to make $r$ as close to zero as possible, not just minimize it. A high negative value is just as bad as a high positive ...


53

The black box thing has nothing to do with the level of expertise of the audience (as long as the audience is human), but with the explainability of the function modelled by the machine learning algorithm. In logistic regression, there is a very simple relationship between inputs and outputs. You can sometimes understand why a certain sample was incorrectly ...


51

The key advantage of using minibatch as opposed to the full dataset goes back to the fundamental idea of stochastic gradient descent1. In batch gradient descent, you compute the gradient over the entire dataset, averaging over potentially a vast amount of information. It takes lots of memory to do that. But the real handicap is the batch gradient ...


50

I would start by graphing the time variable vs other variables and looking for trends. For example In this case there is a periodic weekly trend and a long term upwards trend. So you would want to encode two time variables: day_of_week absolute_time In general There are several common time frames that trends occur over: absolute_time day_of_year ...


49

There's a number of different ways of going about this depending on exactly how much semantic information you want to retain and how easy your documents are to tokenize (html documents would probably be pretty difficult to tokenize, but you could conceivably do something with tags and context.) Some of them have been mentioned by ffriend, and the paragraph ...


46

For RNNs (e.g., LSTMs and GRUs), the layer input is a list of timesteps, and each timestep is a feature tensor. That means that you could have a input tensor like this (in Pythonic notation): # Input tensor to RNN [ # Timestep 1 [ temperature_in_paris, value_of_nasdaq, unemployment_rate ], # Timestep 2 [ temperature_in_paris, value_of_nasdaq,...


45

Overfitting is empirically bad. Suppose you have a data set which you split in two, test and training. An overfitted model is one that performs much worse on the test dataset than on training dataset. It is often observed that models like that also in general perform worse on additional (new) test datasets than models which are not overfitted. One way ...


45

Let's say you have time-series data with $N$ rows and $700$ columns which you want to feed to a SimpleRNN(200, return_sequence=True) layer in Keras. Before you feed that to the RNN, you need to reshape the previous data to a 3D tensor. So it becomes a $N \times 700 \times 1$. $ $ The image is taken from https://colah.github.io/posts/2015-08-Understanding-...


44

From your question, I'm assuming that you're using xgboost to fit boosted trees for binary classification. The importance matrix is actually a data.table object with the first column listing the names of all the features actually used in the boosted trees. The meaning of the importance data table is as follows: The Gain implies the relative contribution of ...


43

A CNN will learn to recognize patterns across space. So, as you say, a CNN will learn to recognize components of an image (e.g., lines, curves, etc.) and then learn to combine these components to recognize larger structures (e.g., faces, objects, etc.). You could say, in a very general way, that a RNN will similarly learn to recognize patterns across time....


43

There is a great answer to this question over on SO that uses numpy and pandas. The command (see the answer for the discussion): train, validate, test = np.split(df.sample(frac=1), [int(.6*len(df)), int(.8*len(df))]) produces a 60%, 20%, 20% split for training, validation and test sets.


43

It is more than just numerical. A quick reminder of the softmax: $$ P(y=j | x) = \frac{e^{x_j}}{\sum_{k=1}^K e^{x_k}} $$ Where $x$ is an input vector with length equal to the number of classes $K$. The softmax function has 3 very nice properties: 1. it normalizes your data (outputs a proper probability distribution), 2. is differentiable, and 3. it uses the ...


42

This link contains an amazing amount of deep learning literature. Summarizing it here(going in the order a beginner ideally should)- NOTE: All these resources mainly use python. 1) First of all, a basic knowledge of machine learning is required. I found Caltech's Learning from data to be ideal of all the machine learning courses available on the net. ...


42

Quote from the author of xgboost: Both xgboost and gbm follows the principle of gradient boosting. There are however, the difference in modeling details. Specifically, xgboost used a more regularized model formalization to control over-fitting, which gives it better performance. We have updated a comprehensive tutorial on introduction to the model, ...


41

For a quick simple explanation: In both gradient descent (GD) and stochastic gradient descent (SGD), you update a set of parameters in an iterative manner to minimize an error function. While in GD, you have to run through ALL the samples in your training set to do a single update for a parameter in a particular iteration, in SGD, on the other hand, you ...


41

Q: How many machine learning specialists does it take to change a light bulb? A: Just one, but they require a million light bulbs to train properly. Q: How many machine learning specialists does it take to change a fluorescent light bulb? A: That wasn't in the training data!


Only top voted, non community-wiki answers of a minimum length are eligible