# What does the “dual” parameter in sklearn.svm.LinearSVC and sklearn.svm.LinearSVR do?

While I am more or less familiar with the idea of the SVM, I do not understand the meaning of the dual parameter, which is described in the documentation as:

dual : bool, (default=True)

Select the algorithm to either solve the dual or primal optimization problem.
Prefer dual=False when n_samples > n_features.


Can someone explain it, preferably in ELI5 style?

ELI5-style, here we go.

Kidwarts is a world-famous magical kindergarten for talented 5-year old wizards. Most of the day the kids spend playing in a large hall, which is full of fluffy puppies, magical seesaws, jumping chairs, tiny flying carpets (remote-controlled using kid-approved magic-wands), and all the other cool toys one could even imagine. Each kid has their favourite spot in the hall, where they would typically spend most of the day, petting their favourite puppy or jumping on their favourite jumping chair.

One sunny Tuesday morning miss Fluffypuff (the kindergarten's supervisor) brought a huge magic mirror to the center of the playhall. The mirror has two faces. If one looks into the mirror from one side, they would see meadows full of beautiful rainbow ponies hopping around. From the other face the mirror shows heroic fables of brave young wizards riding dragons and fighting evil witches. Obviously, it would be best if the mirror could be oriented in the room in such a way, that the "pony" face would be turned towards where most of the girls play, while the "dragon" one could be observed mostly by the boys. Everyone knows that 5 year old girls love ponies while boys adore dragons!

It is not an easy task, however. Although many boys indeed mostly hang around the corner of the hall with the flying carpets while a lot of girls prefer the corner with puppies, a bunch of kids are spread all around and could become unhappy and start crying using magical tears if the mirror is positioned unfavourably to them. Miss Fluffypuff has been strugging to find the best spot for the mirror the whole day.

Here is how she did it. First she hovered the mirror right in the middle of the room, turned in a somewhat random direction. She then waited and observed who of the kids started crying because they did not like what the mirror showed them. Based on that information, she nudged the mirror slightly towards the most unhappy kids, so that they would get to see more of their preferred side (or less of the "bad" side). After the whole day of such nudging the mirror seems to have converged to a more or less optimal orientation, with just a few kids left unhappy. To them miss Fluffypuff gave pieces delicious magical candy, so they were still quite happy after all!

In the evening Miss Fluffypuff was excited to tell the mirror story to her friend, Mister Puppypaw. "Ah, I would have done things differently" said Mister Puppypaw, twirling his gorgeous mustache with his fingers. My method would be the following: I would first pick two kids in the room, and ask them to decide on their favourite orientation for the mirror. Then I would see who of the other kids complains about the mirror, add them to the team and have the new team decide together again. If at this point there were still unhappy kids, I would continue adding them to the "council" and let them find a compromise which suited everyone. We would thus end up with the situation, where the configuration of the mirror is decided to satisfy precisely the group of kids who care about it, while those without opinion would not have to be bothered. This would be super fair, wouldn't it?

Miss Fluffypuff agreed, but noted that her method also seemed totally fair in the end. They kissed and lived happily ever after. The End.

In this story, Miss Fluffypuff was solving the primal problem. That is, she was searching for the orientation $$\mathbf{w}$$ of the mirror directly. Mister Puppypaw, instead, proceeded in a roundabout, dual manner. He assumed that we can always compute the orientation $$\mathbf{w}$$ by having each kid vote on their preference, and aggregating these votes:

$$\mathbf{w} = \sum_{i} \alpha_i \mathbf{x}_i,$$

where $$\mathbf{x}_i$$ is the favourite position of kid $$i$$ and $$\alpha_i$$ is the vote (which will be $$0$$ if the kid does not care, positive if the kid prefers one face of the mirror, and negative otherwise). Consequently, the problem of Mister Puppypaw is not in finding the best orientation $$\mathbf{w}$$ directly, but rather in figuring out the vector of "votes" $$\mathbf{\alpha}$$ (from which the orientation $$\mathbf{w}$$ can be derived in a straightforward manner). YOu can also say that his main problem would be to find a subset of kids whose vote would be nonzero - these would be the "support vectors".

When $$\mathbf{w}$$ is just a two-dimensional vector and the number of kids is large, as is in the original story, the method of miss Fluffypuff is probably the best. Keeping track of a two-dimensional orientation of the mirror is simpler than managing a list of hundred or so "votes" or figuring out a proper subset of "kids who care".

If there were only one or two kids, however, it would instead be faster to just ask them rather than play this "mirror-nudging" game. Similarly, if the mirror had a million different knobs which needed to be configured, it would be easier to let even a hundred kids simply vote on their preferences (assuming each kid has a clear idea what million-knob configuration fits him best), rather than carefully tune these knobs in tiny increments, checking who of the kids started crying after every change.

In the end, either of the two methods would produce the same result, though.