Hot answers tagged

85

Kernelized SVMs require the computation of a distance function between each point in the dataset, which is the dominating cost of $\mathcal{O}(n_\text{features} \times n_\text{observations}^2)$. The storage of the distances is a burden on memory, so they're recomputed on the fly. Thankfully, only the points nearest the decision boundary are needed most of ...


81

I recently found this online tool that produces publication-ready NN-architecture schematics. It is called NN-SVG and made by Alex Lenail. You can easily export these to use in, say, LaTeX for example. Here are a few examples: AlexNet style LeNet style and the good old Fully Connected style


51

SVM is a powerful classifier. It has some nice advantages (which I guess were responsible for its popularity)... These are: Efficiency: Only the support vectors play a role in determining the classification boundary. All other points from the training set needn't be stored in memory. The so-called power of kernels: With appropriate kernels you can transform ...


50

I wrote some latex code to draw Deep networks for one of my reports. You can find it here: https://github.com/HarisIqbal88/PlotNeuralNet With this, you can draw networks like these:


37

I would say, the choice depends very much on what data you have and what is your purpose. A few "rules of thumb". Random Forest is intrinsically suited for multiclass problems, while SVM is intrinsically two-class. For multiclass problem you will need to reduce it into multiple binary classification problems. Random Forest works well with a mixture of ...


27

A regular SVM with default values uses a radial basis function as the SVM kernel. This is basically a Gaussian kernel aka bell-curve. Meaning that the no man's land between different classes is created with a Gaussian function. The linear-SVM uses a linear kernel for the basis function, so you can think of this as a ^ shaped function. It is much less ...


20

SVM solves an optimization problem of quadratic order. I do not have anything to add that has not been said here. I just want to post a link the sklearn page about SVC which clarifies what is going on: The implementation is based on libsvm. The fit time complexity is more than quadratic with the number of samples which makes it hard to scale to ...


18

The regularization parameter (lambda) serves as a degree of importance that is given to miss-classifications. SVM pose a quadratic optimization problem that looks for maximizing the margin between both classes and minimizing the amount of miss-classifications. However, for non-separable problems, in order to find a solution, the miss-classification ...


17

To answer your questions: Yes your approach is right Of A, B and C the right answer is B. The explanation is the following: In order to calculate Mean Average Precision (mAP) in the context of Object Detection you must compute the Average Precision (AP) for each class, and then compute the mean across all classes. The key here is to compute the AP for each ...


15

It's hard to say without knowing a little more about your dataset, and how separable your dataset is based on your feature vector, but I would probably suggest using extreme random forest over standard random forests because of your relatively small sample set. Extreme random forests are pretty similar to standard random forests with the one exception that ...


13

The kernel trick is based on some concepts: you have a dataset, e.g. two classes of 2D data, represented on a cartesian plane. It is not linearly separable, so for example a SVM could not find a line that separates the two classes. Now, what you can do it project this data into an higher dimension space, for example 3D, where it could be divided linearly by ...


12

For low parameters, pretty limited sample size, and a binary classifier logistic regression should be plenty powerful enough. You can use a more advanced algorithm but it's probably overkill.


12

For automated drawing, see How do you visualize neural network architectures?, https://softwarerecs.stackexchange.com/q/28169/903 and https://softwarerecs.stackexchange.com/q/47841/903 For manual drawing, see https://redd.it/574usi


12

There is a nice and detailed explanation with an easy to use code on my Github. Certainly it will help you guys.


11

I think basic support vector machine means hard-margin SVM. So, let's review: What is a Hard-Margin SVM In short, we want to find a hyperplane with the largest margin which be able to separate all observations correctly in our training sample space. The optimisation problem in hard-margin SVM Given the above definition, what is the optimisation problem which ...


11

Geometrically, the vector w is directed orthogonal to the line defined by $w^{T} x = b$. This can be understood as follows: First take $b = 0$. Now it is clear that all vectors, $x$, with vanishing inner product with $w$ satisfy this equation, i.e. all vectors orthogonal to w satisfy this equation. Now translate the hyperplane away from the origin over a ...


11

First problem: Minimizing $\|w\|$ or $\|w\|^2$: It is correct that one wants to maximize the margin. This is actually done by maximizing $\frac{2}{\|w\|}$. This would be the "correct" way of doing it, but it is rather inconvenient. Let's first drop the $2$, as it is just a constant. Now if $\frac{1}{\|w\|}$ is maximal, $\|w\|$ will have to be as small as ...


11

If you use logistic regression and the cross-entropy cost function, it's shape is convex and there will be a single minimum. But during optimization, you may find weights that are near to optimal point and not exactly on the optimal point. This means that you can have multiple classifies that reduce the error and maybe set it to zero for the training data ...


9

The term consensus, as far as I'm concerned, is used rather for cases when you have more a than one source of metric/measure/choice from which to make a decision. And, in order to choose a possible result, you perform some average evaluation/consensus over the values available. This is not the case for SVM. The algorithm is based on a quadratic optimization,...


9

In answering this question one significant distinction to make is whether we are talking about linear Support Vector Machines or non-linear, that is, kernelized Support Vector Machines. Linear SVMs Linear SVMs are both in theory and practice very good models when your data can be explained by linear relations of your features. They are superior over ...


9

Did you include scaling in your pre-processing step? I had this issue when running my SVM. My dataset is ~780,000 samples (row) with 20 features (col). My training set is ~235k samples. It turns out that I just forgot to scale my data! If this is the case, try adding this bit to your code: scale data to [-1,1] ; increase SVM speed: from sklearn.preprocessing ...


9

Implementing a custom transformer is simple. You have to implement the fit and transform methods like below. Since your ANN is already trained (right?) the fit method has to do nothing, just return self. And the transform method has to pass the incoming data to the ANN and return its output. from sklearn.base import BaseEstimator, TransformerMixin class ...


8

With such a huge dataset I think you'd be better off using a neural network, deep learning, random forest (they are surprisingly good), etc. As mentioned in earlier replies, the time taken is proportional to the third power of the number of training samples. Even the prediction time is polynomial in terms of number of test vectors. If you really must use ...


8

SVM can be used for classification (distinguishing between several groups or classes) and regression (obtaining a mathematical model to predict something). They can be applied to both linear and non linear problems. Until 2006 they were the best general purpose algorithm for machine learning. I was trying to find a paper that compared many implementations ...


8

First of all, start with a subset until you know what you are doing. There is no use in waiting for hours for a result that doesn't work, or to run out of memory, or to optimize, just to find out it does not work. Secondly, makes sure your preprocessing is very very well done. Bad preprocessing will hurt your algorithms. From my experience, one-class SVM ...


8

Your link has sufficient resources, so let's go through: When you call decision_function(), you get the output from each of the pairwise classifiers (n*(n-1)/2 numbers total). See pages 127 and 128 of "Support Vector Machines for Pattern Classification". Click on the "page 127 and 128" link (not shown here, but in the Stackoverflow answer). You should ...


8

Even though I am more familiar with the use of RBF kernel with Gaussian Processes, I think your intuition is correct since, generally speaking, a larger lengthscale means that the learnt function varies less in that direction, which is another way of saying that that feature is irrelevant for the learnt function. So if you have to choose which feature is ...


7

This was meant as a comment but it is too long. The fact that your test set has a different range might be a sign that the training set is not a good representation of the test set. However, if the difference is really small as in your example, it is likely that it won't affect your predictions. Unfortunately, I don't think I have a good reason to think it ...


7

SVM models perform better on sparse data than does trees in general. For example in document classification you may have thousands, even tens of thousands of features and in any given document vector only a small fraction of these features may have a value greater than zero. There are probably other differences between them, but this is what I found for my ...


Only top voted, non community-wiki answers of a minimum length are eligible