I am building a Binary Classifier. There is no Real World Scenario Problem Statement, We have just given only the data set and some guidelines.

  • Number of features : 2040
  • All features are in decimal format.
  • Range of all features is also not very large (around -8.0 to 8.0), they are standardized.
  • There are only 2 outputs 0/1.
  • The Data Points are only 400. We need to create a binary classifier with evaluation metrics as f1_score and train/test split ration should be 80/20.

Also , while data exploration:

  • I found out the the classes are divided equally (i.e. 200 data points approximately for each class 0 and 1)
  • I have also imputed null values with the mean of columns (only 30 null values were their)
  • I am splitting the train, test data in 80/20 ratio using stratify on target column.

I have explored below things so far :

  • I applied PCA for dimensionality reduction to get 95% of variance (feature shape got reduced from 2040 to 258) from the dataset and then tried different classification algorithms like Random Forest, SVM, Naive Bayes, LGBM, Logisctic Regression and KNN. But I am unable to get f1_score more than 60%.
  • I also tried SelectKBest feature selection (from k=30 to 250 ) for same classification algorithms mentioned in above point, but was unable to get f1_score more than 70%.
  • I also tried an ANN with 3 layers but f1_score was around 58%.

I think the problem is with the less number of datapoints we have. I am also considering applying CNN as the dimension and data type of features are apt for this but the only problem is , it is in 1 dimension. Can someone please help me , how should I approach this problem.

  • 1
    $\begingroup$ You want an F1 score of at least 95%, but what makes you think that the two classes can actually be classified with such accuracy? For all we know the data is generated almost at random and the data does not contain any useful information to make a prediction. $\endgroup$
    – Oxbowerce
    Nov 18 '21 at 14:22
  • $\begingroup$ That is a valid point, But Can we identify if I am missing something or doing something wrong which can increase the metrics. I just want to know If my approach is correct or not. $\endgroup$
    – vaibhav
    Nov 18 '21 at 14:29

A few thoughts:

  • Always check for overfitting in experiments where the training data is small, especially with a high number of features.
  • If possible try more advance feature selection methods like genetic methods. It's very computationally expensive because it needs to train/evaluate with many subsets of features, but it usually gives better results than top K selection.
  • For some algorithms you might need to fine tune the parameters, it can have a big impact on performance.
  • As mentioned by Oxbowerce, don't expect any particular performance unless you have information that it's possible, e.g. you know that another system reached it.

Not the answer you're looking for? Browse other questions tagged or ask your own question.