I'm studying a binary classification task with an objective function, derived from SVM, defined so:

$$\vec{\xi}' S \vec{\xi}$$

with:

$$y_i (f(\vec{x}_i)) >= 1 - \xi_i, i=1..l$$

and:

$$\xi_i >=0, i=1..l$$

$$f(\vec{x}_i) = \sum_{p=1}^{QP} a_p K(\vec{x}_i, \vec{x}_p) + b$$

where $$QP << l$$.

The point is that I don't want to use the whole kernel matrix. Instead, I use submatrices of K. With this method I was able to train a data-set with 26.000 patterns (pictures converted to 32x32 RGB values and divided by 65536), so with 1024 dimensions each (F1 score close to 1.0). But when I test the model against the validation data-set I'm not able to reach an F1 score of 0.75. The prototype project is available here.