10
$\begingroup$

I am new to the field of machine learning, but have done my share of signal processing. Please let me know if this question has been mislabeled.

I have two dimensional data which is defined by at least three variables, with a highly non-linear model way too complicated to simulate.

I have had varying level of success at extracting the two main components from the data using methods like PCA and ICA (from the python library Scikit-Learn), but it seems these method (or at least, these implementation of the methods) are limited to extracting as many components as there are dimensions in the data, for example, 2 components from a 2D point cloud.

When plotting the data, it is clear to the trained eye that there are three different linear trends, the three color lines show the directions.

data plot

When using PCA, the main component is aligned to one of the color lines, and the other is at 90°, as expected. When using ICA, the first component is aligned with the blue line, and the second is somewhere in between the red and green ones. I am looking for a tool which could reproduce all three components in my signal.

EDIT, Additional info: I am here working in a small subset of a bigger phase plane. In this small subset, each input variables produce a linear change on the plane, but the direction and amplitude of this change is non-linear and depends on where exactly on the bigger plane I am working. At some places, two of the variables can be degenerate: they produce change in the same direction. for example, say the model depends on X, Y, and Z. A change in the variable X will produce a variation along the blue line; Y causes a variation along the green line; Z, along the red one.

Improve this question
$\endgroup$
6
  • 3
    $\begingroup$ If I understand correctly, the concept you are looking for is embedding. Look up kernel methods, and kernel PCA in particular. $\endgroup$
    – Emre
    Commented Jun 26, 2015 at 0:22
  • 1
    $\begingroup$ I am not sure about this, so I am not posting it as an answer. In a neural network type of model, you can keep the hidden layer dimensionality > input layer dimensionality. Then you can use the hidden layer as input to another network/model. But doing so requires lots of data. $\endgroup$
    – Azrael
    Commented Jun 26, 2015 at 9:01
  • $\begingroup$ When you say 2 dimensional data, defined by at least three variables, in what sense do you use the term 'variable'? Would classes be a suitable substitution ? It's worth noting that PCA extracts maximally variant dimensions from data, this is not necessarily the most discriminative transform to apply. Have you looked at clustering ? $\endgroup$
    – image_doctor
    Commented Jun 28, 2015 at 11:20
  • $\begingroup$ Do you know anything about the nonlinearity of the model? Though it may be too complex to simulate, knowing that it is at most made up of degree 3 polynomials restricts the feature engineering significantly e.g. you could add all 3rd degree polys and then PCA it back down to 3D. $\endgroup$
    – AN6U5
    Commented Jul 13, 2015 at 16:20
  • $\begingroup$ I have discussed with a statistician friend of mine who suggested using kernel PCA on the derivative of my data, since I'm looking for slopes. Would taking the derivative count as "feature engineering"? $\endgroup$
    – PhilMacKay
    Commented Jul 13, 2015 at 18:49

2 Answers 2

Reset to default
8
$\begingroup$

The short answer is yes.

Essentially you will be performing some sort of feature engineering. This means constructing a series of functions of your data, often:

$\phi_j(x): \mathbb R^p \rightarrow \mathbb R \ ,\ \ j=1,...,K$

Which, strung together, define a transformed data vector $\boldsymbol \phi(x)$ of length $K$.

There are a number of ways, better and worse, of doing this. You may want to look up terms like:

  1. Splines and generalised additive models.
  2. The kernel trick (how to make a model where $K\rightarrow \infty$).
  3. Feature engineering (of the manual variety, e.g. adding an $x^2$ column to your data).
  4. Deep learning, representation learning

As you might guess from such a varied bag of techniques, this is a large area. It goes without saying really but care has to be taken to avoid overfitting.

This paper Representation Learning: A Review and New Perspectives deals with some of the issues around what makes a particular set of features 'good', from a deep learning perspective.

Improve this answer
$\endgroup$
-2
$\begingroup$

I guess you are looking for features which extract out new features. A feature which best represents the dataset. If that is the case then we call such method "feature extraction".

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.