Hot answers tagged

32

If you are using SKlearn, you can use their hyper-parameter optimization tools. For example, you can use: GridSearchCV RandomizedSearchCV If you use GridSearchCV, you can do the following: 1) Choose your classifier from sklearn.neural_network import MLPClassifier mlp = MLPClassifier(max_iter=100) 2) Define a hyper-parameter space to search. (All the ...


9

I personally think that the general idea of optimising your model with different random seeds is not a good idea. There are many other, more important, aspects of the modelling process that you can worry about, tweak and compare before spending time on the effects of random initialisation. That being said, if you just want to test the effect of random ...


6

The short answer is that there is not a method in scikit-learn to obtain MLP feature importance - you're coming up against the classic problem of interpreting how model weights contribute towards classification decisions. However, there are a couple of great python libraries out there that aim to address this problem - LIME, ELI5 and Yellowbrick: LIME (or ...


5

There can be other reasons related to the model but the most simple explanation is that the data contains contradicting patterns: if the same features correspond to different target values, there is no way for any model to achieve perfect performance. Let me illustrate with a toy example: x y 0 4 0 4 1 3 1 4 1 3 1 3 2 8 2 7 2 7 2 6 ...


4

When the data is linearly inseparable, we use MLP. Here what is meant by "data"--is it the response or the input feature that is linearly inseparable? This means that a linear function of the input features is unable to separate the response. To answer your question a bit more directly: Given only a linear function of the inputs, the response is the thing ...


3

Well you missed the diagram they provided for the GFNN. Here is the diagram from their page: Clearly you can see what the GFNN does, unlike MLP the inputs are applied to the hidden layers also. While in MLP the only way information can travel to hidden layers is through previous layers, in GFNN the input information is directly available to the hidden ...


3

As a complement to the very practical answer of @BrunoGL, I'd like to give a more theoretical answer. I'd like to suggest everyone trying to adjust hyperparameters of a simple Neural Network to read Efficient Backprop, by Lecun and others (http://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf). Yes it's from 1998, no it's not outdated. It covers the impact ...


3

First try a simple model: The input layer and the output layers dimension are defined by your data / your problem definition. Then train a model without any hidden layer. See how good it performs. Is it good enough? If yes, you're done. If no, continue Add a hidden layer of reasonable size or adjust a hidden layers size. Go to step (2). The "reasonable" ...


3

That's because the square loss is defined as 0.5*MSE. See definition here:


3

I don't use Python so I can't tell you exactly what is going on but I had a quick look at your data: A few remarks: it looks like the vast majority of the points are created artificially by interpolation. Why not, but that's unlikely to reflect the reality of the price changes: I would expect much more variation/noise in a real dataset about car prices. ...


3

The two pictures you show illustrate how to interprete one perceptron and a MLP consisting of 3 layers. Let us discuss the geometry behind one perceptron first, before explaining the image. We consider a perceptron with $n$ inputs. Thus let $\mathbf{x} \in \mathbb{R}^{n}$ be the input vector, $\mathbf{w} \in \mathbb{R}^{n}$ be the weights, and let $b \in \...


3

An epoch is not a standalone training process, so no, the weights are not reset after an epoch is complete. Epochs are merely used to keep track of how much data has been used to train the network. It's a way to represent how much "work" has been done. Epochs are used to compare how "long" it would take to train a certain network regardless of hardware. ...


2

... each layer of a neural network is responsible for recognizing one feature of the input data. For example, if we build a neural network that classifies cars, buses, vans and bicycles, a layer will be responsible to recognize the tires, another one will responsible for recognizing the size of the vehicle. There are numerous kinds of neural networks and ...


2

Maybe you are making a mistake, put your code here. But without seeing your code, these are possible points: Vanishing problem, I don't think you this problem due to having a very shallow network. You can change your activation function to relu for avoiding that. Covariat shift, What it means is that similar to input features which have to be normalized, ...


2

You could say every type of neural network gets 1d input data. It's just more convenient to think about 2d-CNNs taking 2d data because the convolution operation is best illustrated by moving squares across a grid, and similarly for max-pooling. But you could easily write out all the multiplications, additions, and max operations you performed in one line ...


2

The reason is that by adding more layers, you've added more trainable parameter to your model. You have to train it more. You should consider that MNIST data set is a very easy-to-learn dataset. You can have to layers with much less number of neurons in each layer. Try $10$ neurons for each to facilitate the learning process. You can reach to $100%$ accuracy....


2

Since one could see there is no difference in the minmal loss value, this could well be an issue of differnce in floating point precision with respect to CPU and GPU. You should try to cross check unit operations in the intermediate layers(Considering its an ANN) and look for possible discrepencies to make sure it is a floating point precision variance.


2

Consistently underestimating target could be due to the distribution of the target variable. If the target distribution has a negative-skew (i.e., a long tail towards lower values), then the neural network is just pattern matching to minimize those errors. For example, if the network has a squared loss function then large estimation errors in the tail are ...


2

Would my code be faster if I rewrite it with matrices? Without seeing the code it's impossible to know, but very likely. Also, I would never model single neurons. Too much overhead without any use. Model layers instead. how can I get my performance to be comaparable to that of sklearn Sklearn is open source. Read the code: https://github.com/scikit-...


2

This can be accomplished by a modification to multi-class cross-entropy. We are faced with two types of supervision. First type is "data $i$ belongs to class $k$" denoted by $y_{ik}=1$, and second type is "data $i$ does not belong to class $k$" denoted by $\bar{y}_{ik}=1$. For example, for 3 classes, $y_i=(1, 0, 0)$ denotes that point $i$ belongs to class $...


2

The problem here is the input_shape argument you are using, firstly that is the wrong shape and you should only provide an input shape for your first layer. For example Let's import the CIFAR 10 data from Keras from __future__ import print_function import keras from keras.datasets import cifar10 from keras.preprocessing.image import ImageDataGenerator ...


2

Your max iteration values are strings. max_iter': ['200', '1000', '5000', '10000'] Try max_iter': [200, 1000, 5000, 10000] }


2

I'm guessing you want something like this: model.layers[-1].get_weights()


2

The callback function can be used with model.layers[-1].get_weights() to get weights per iteration. weights=[] getweights = LambdaCallback(on_epoch_end=lambda batch, logs: weights.append(model.layers[-1].get_weights()[1])) model.fit(x, y, batch_size=5,epochs=10, callbacks=[getweights]) print(weights) In the given code weights is a list which contains ...


2

There are several things I would like to mention : I do not think that much change on your architecture will impact a lot. Try comparing 10, 20, 50, 100 or more depth. Difference will be most likely to be slightly more noticeable. You are comparing accuracy which is based on labels instead of the probabilities. Try comparing logloss or auc score which is ...


2

If one permutes the connections of the hidden layer ($d!$ ways to do that), and move and rename connections appropriately, then one effectively has the same MLP with the exact same minima, yet the configuration has changed (in a trivial sense). Thus there are (at least) $d!$ configurations only trivialy different with the exact same minima. To see it in your ...


1

This looks like a case of the model outputting the probability of being in category 1. It then is up to you to decide on the cutoff. You give an example of an output of $(0.43, 0.56, 0.1, 0.8)$. If your cutoff is $0.5$, you’d get classifications of $(0, 1, 0, 1)$. If you set your cutoff at $0.2$, which you’re allowed to do, you get classifications of $(1,1,...


1

If the feature doesn't make sense in a subset of the samples, doesn't this mean that this is (or should be) a separate dataset, that needs a second model? That's one approach I'd think about. The second would be to work with the data (feature) itself. It's probably best to use neutral value. In case of numerical value: try using a mean or median value, ...


1

Does the network recognize A as an A as good as before? Yes, it can. The reason is that if you try to illustrate the confusion matrix you can see that the FP, FN, TP and TN can be kept at a good level. What you have done is actually changing the distribution of your input data. Suppose that you train a network and it should recognize cats and dogs. You can ...


1

The drawbacks of one-hot encoding is that it generates very sparse matrices. Neural Networks hate sparse data: gradient descent optimizers give their best on continuous variables, while weight updates do not work very good when there is so little variability. I would never recommend one-hot encoding for ANNs. I use it only when it's absolutely necessary, ...


Only top voted, non community-wiki answers of a minimum length are eligible