125 votes

The cross-entropy error function in neural networks

One way to interpret cross-entropy is to see it as a (minus) log-likelihood for the data $y_i'$, under a model $y_i$. Namely, suppose that you have some fixed model (a.k.a. "hypothesis"), which ...
  • 2,101
61 votes
Accepted

How to disable GPU with TensorFlow?

I've seen some suggestions elsewhere, but they are old and do not apply very well to newer TF versions. What worked for me was this: ...
58 votes
Accepted

What does from_logits=True do in SparseCategoricalcrossEntropy loss function?

The from_logits=True attribute inform the loss function that the output values generated by the model are not normalized, a.k.a. logits. In other words, the softmax ...
  • 774
51 votes
Accepted

Multi GPU in Keras

From the Keras FAQs, below is copy-pasted code to enable 'data parallelism'. I.e. having each of your GPUs process a different subset of your data independently. ...
  • 636
51 votes

Why does Keras need TensorFlow as backend?

This makes more sense when understood in its historical context. These were the chronological events: April 2009 Theano 0.1 is released. It would dominate the deep ...
  • 16.7k
42 votes

What is the relationship between the accuracy and the loss in deep learning?

There is no relationship between these two metrics. Loss can be seen as a distance between the true values of the problem and the values predicted by the model. Greater the loss is, more huge is the ...
41 votes
Accepted

Merging two different models in Keras

I figured out the answer to my question and here is the code that builds on the above answer. ...
  • 1,013
40 votes
Accepted

Neural networks: which cost function to use?

This answer is on the general side of cost functions, not related to TensorFlow, and will mostly address the "some explanation about this topic" part of your question. In most examples/tutorial I ...
  • 1,336
40 votes
Accepted

Intuitive explanation of Noise Contrastive Estimation (NCE) loss?

Taken from this post:https://stats.stackexchange.com/a/245452/154812 The issue There are some issues with learning the word vectors using an "standard" neural network. In this way, the word vectors ...
29 votes

The cross-entropy error function in neural networks

The first logloss formula you are using is for multiclass log loss, where the $i$ subscript enumerates the different classes in an example. The formula assumes that a single $y_i'$ in each example is ...
  • 27.7k
27 votes
Accepted

In CNN, why do we increase the number of filters in deeper Convolution layers for complex images?

For this you need to understand what filters actually do. Every layer of filters is there to capture patterns. For example, the first layer of filters captures patterns like edges, corners, dots etc. ...
  • 837
25 votes
Accepted

Difference between RMSProp with momentum and Adam Optimizers

(My answer is based mostly on Adam: A Method for Stochastic Optimization (the original Adam paper) and on the implementation of rmsprop with momentum in Tensorflow (which is operator() of struct ...
24 votes
Accepted

How to use LeakyRelu as activation function in sequence DNN in keras?When it perfoms better than Relu?

You can use the LeakyRelu layer, as in the python class, instead of just specifying the string name like in your example. It works similarly to a normal layer. Import the LeakyReLU and instantiate a ...
  • 14.2k
22 votes

How to define a custom performance metric in Keras?

You have to use Keras backend functions. Unfortunately they do not support the &-operator, so that you have to build a workaround: We generate matrices of the ...
  • 381
22 votes

What is weight and bias in deep learning?

Mathematically speaking, Imagine you are a model (No! not that kind, figure 8 ones). Bias is simply how biased you are. If you are a Nigerian, and you are asked "Which nationality have the most ...
  • 321
22 votes

Keras vs. tf.keras

From Keras repo.: Keras is a high-level neural networks API, written in Python and capable of running on top of TensorFlow, CNTK, or Theano. And Before installing Keras, please install one of ...
  • 8,767
20 votes

What is the meaning of "The number of units in the LSTM cell"?

Most LSTM/RNN diagrams just show the hidden cells but never the units of those cells. Hence, the confusion. Each hidden layer has hidden cells, as many as the number of time steps. And further, each ...
20 votes
Accepted

How to calculate the output shape of conv2d_transpose?

Here is the correct formula for computing the size of the output with tf.layers.conv2d_transpose(): ...
  • 316
20 votes
Accepted

What is one hot encoding in tensorflow?

Suppose you have a categorical feature in your dataset (e.g. color). And your samples can be either red, yellow or blue. In order to pass this argument to a ML algorithm, you first need to encode it ...
  • 7,658
19 votes
Accepted

How to deal with string labels in multi-class classification with keras?

Sklearn's LabelEncoder module finds all classes and assigns each a numeric id starting from 0. This means that whatever your class representations are in the ...
  • 27.7k
19 votes
Accepted

Using TensorFlow with Intel GPU

At this moment, the answer is no. Tensorflow uses CUDA which means only NVIDIA GPUs are supported. For OpenCL support, you can track the progress here. BTW, Intel/AMD CPUs are supported. The default ...
  • 4,156
19 votes

Custom loss function with additional parameter in Keras

You can write a function that returns another function, as is done here on GitHub ...
17 votes
Accepted

Tensorflow neural network TypeError: Fetch argument has invalid type

The problem lay in using the name 'cost' on two occasions, the problem was solved by changing this: ...
17 votes
Accepted

Why TensorFlow can't fit simple linear model if I am minimizing absolute mean error instead of the mean squared error?

I tried this and got same result. It is because the gradient of .abs is harder for a simple optimiser to follow to the minima, unlike squared difference where ...
  • 27.7k
17 votes

Neural Network for Multiple Output Regression

What you are describing is a normal multidimensional linear regression. This type of problem is normally addressed with a feed-forward network, either MLP or any other architecture that suits the ...
  • 16.7k
16 votes

Keyword/phrase extraction from Text using Deep Learning libraries

This is an open area of research and it certainly depends on the way you frame the problem. If you're talking about multi-document summarization then the problem is slightly different than if you were ...
16 votes
Accepted

Keyword/phrase extraction from Text using Deep Learning libraries

The Google Research Blog should be helpful in the context of TensorFlow. In the above article, there is a reference to the Annotated English Gigaword dataset which is routinely used for text ...
16 votes
Accepted

What is the meaning of "The number of units in the LSTM cell"?

As the helpful comments in that function say, The definition of cell in this package differs from the definition used in the literature. In the literature, cell refers to an object with a ...
  • 355
16 votes

Using TensorFlow with Intel GPU

You might want to check out https://github.com/benoitsteiner/tensorflow-opencl/ which is a fork of Tensorflow with OpenCL support. If your OS is supported by the fork and you are able to properly ...
16 votes
Accepted

What more does TensorFlow offer to keras?

Deep Learning frameworks operate at 2 levels of abstraction: Lower Level: This is where frameworks like Tensorflow, MXNet, Theano, and PyTorch sit. This is the level where mathematical operations ...
  • 345

Only top scored, non community-wiki answers of a minimum length are eligible