Dealing with irrelevant features in dataset (Homework)

Question

I have a specific question pertaining to one of my machine learning homeworks.

Basically, we are required to build a model that takes a 5000*10000 dataset X (5000 examples each with 10000 features), and predict Z, which is a 5000*2 Matrix. The dataset Z is synthetically generated from X, however, it only depends on two of those features from X. That is, only 2 of the 10000 features are relevant. Z has only 3 possible classes, [0,0], [1,0] and [1,1], and is not balanced. All of the features of X are sampled equally from a normal distribution, the only distinguishing feature is that Z is calculated from just 2 of them.

I thought that using PCA might be a good idea, and tried this. However, I had two problems. Firstly calling np.linalg.eig on [email protected] or X.T@X takes an unreasonable amount of time. And secondly, when I used np.linalg.eigh, it was slightly less computationally prohibitive, but many of the eigenvalues were actually very large. This makes sense to me as well since nothing actually distinguishes X except their relationship to Z.

I then built a simple neural network, linear -> tanh -> linear -> sigmoid. This descended fine, however it just learn to mirror the underlying distribution of the dataset. That is, it didn't learn which parameters of X were relevant and instead learnt to predict such that the overall error based on it (seemingly) randomly guessing was minimized.

Can anyone suggest some techniques that might help me to solve my problem?

Answer 1

This could be many things.

1. Since you want to calssify the data into three classes, I would use one-hot-encoding, rather than the binary enumeration, because you kind of introduce the information to the model that [1,0] and [1,1] are "closer" than [0,0] and [1,1] for example. Unless this is actually the case, better encode the classes like [1,0,0], [0,1,0] and [0,0,1].

2. That the eigenvalues in your PCA are large can have two reasons: a) you need to divide XX^T by the number of data points, as the PCA is computed on an approximation of the covariance matrix, which in turn is based on an expectation value b) not sure if you did, but make sure to center your data first

3. I think you discovered this yourself already, but if the PCA is useful on your problem depends actually on your data distribution and how you computet the Z. It sounds a bit as if there was no covariance between the features, so there are no primary components to discover. And in case you picked the features to compute Z randomly, you have no guarantee that the primary components tell you anything about Z. Maybe do a PCA on the data and append Z to X. This might help with dimensionality reduction or identifying relevent features.

4. Since you have many features and use saturating activation functions, it is important that you normalize the data before passing it to the model and that you initialize the weights properly, otherwise the neurons will saturate and the gradient will be almost zero. Also training might be very slow anyway, as with all the "useless" features you introduce a lot of noise.

I hope this helps. If not, feel free to post some more information on your training data and the neural network you wanted to use.