I have a specific question pertaining to one of my machine learning homeworks.
Basically, we are required to build a model that takes a 5000*10000 dataset X (5000 examples each with 10000 features), and predict Z, which is a 5000*2 Matrix. The dataset Z is synthetically generated from X, however, it only depends on two of those features from X. That is, only 2 of the 10000 features are relevant. Z has only 3 possible classes, [0,0], [1,0] and [1,1], and is not balanced. All of the features of X are sampled equally from a normal distribution, the only distinguishing feature is that Z is calculated from just 2 of them.
I thought that using PCA might be a good idea, and tried this. However, I had two problems. Firstly calling np.linalg.eig on X@X.T or X.T@X takes an unreasonable amount of time. And secondly, when I used np.linalg.eigh, it was slightly less computationally prohibitive, but many of the eigenvalues were actually very large. This makes sense to me as well since nothing actually distinguishes X except their relationship to Z.
I then built a simple neural network, linear -> tanh -> linear -> sigmoid. This descended fine, however it just learn to mirror the underlying distribution of the dataset. That is, it didn't learn which parameters of X were relevant and instead learnt to predict such that the overall error based on it (seemingly) randomly guessing was minimized.
Can anyone suggest some techniques that might help me to solve my problem?