Classification problem with no context in numerical features

Question

I have an extremely abstract and numeric data with equally abstract objective.

I have around 3000 rows of train data (df_train), where I have a binary target variable target (0 or 1), where I have 50 numerical (float) features num1, num2, ..., num50 ranging from 0 and 1, and 50 integer features int1, int2, ..., int50 being either -1, 0, or 1. This adds up to my data having 3000 rows and 101 columns.

My test data(df_test) has the same format as the train data excluding the target variable, having 500 rows. My objective is to classify the target variable based on other features in the test data.

Given that there are a lot of features, my instinct was to do a dimensionality reduction, and since the goal is to classify rather than cluster, I thought PCA would be more appropriate compared to other manifold methods such as t-SNE.

I have a couple questions regarding the designing of the solution:

1. Naturally, as the number of PCA components get closer to the number of features, it explains more variability. What is the good threshold to explain the data yet still reduce the dimension?

2. After fitting the PCA by scikit, how can the result play into actually classifying the target variable in my test data?

3. Is there a more appropriate dimensionality reduction technique, or furthermore is it actually necessary to do a dimensionality reduction?

Any insights are greatly appreciated.

Answer 1

1. There is no way to decide for a good threshold without more information about the data. For example, if the independent variables are highly correlated, you probably want to reduce the dimension. However, they might all be rich in information and only weakly correlated. I would not listen to people telling you that the number of predictors should be the square root of your number of observations. A good rule of thumb is: Whatever gives the best out-of-sample predictions is the best model.
2. I am not familiar with PCA in scikit. That being said, the basic idea is that you apply the same normalization steps (remember that PCA assumes that the variables are distributed with mean 0 and variances 1) using the test data coefficients and then define the principle components as linear combination of your original (scaled) variables. When you've transformed both your train and your test data set, you can use any model you like on these new data sets.
3. Using PCA for dimensionality reduction can improve prediction results at the cost of interpretability. In the example of a linear model, column num1 having a coefficient of 5.7 is interpreted easily. However, what does it mean if principle component 1 has a coefficient of 0.32? Again, in the example of a linear model, there are the Akaike and Bayesian information criteria (AIC and BIC) for dimensionality reduction. They keep the original variables as they are, only removing others. See https://en.wikipedia.org/wiki/Akaike_information_criterion.