What should I check if model accuracy is no better than baseline level(random guess)

Question

I have a data with only 8 columns:

• id
• created_time
• employee_id
• rank position
• hourly price
• num_work_completed
• work_category
• hired

Hired is the target variable with 1 representing hired and 0 representing not hired, and it is imbalanced with 5.7% hired(1) which makes the baseline accuracy 94.3% I am trying to build model that predict whether a employee will be hired. After I finished the EDA, feature engineering(dealing with NAs, encoding categorical variables, normalizing numeric variables), I used 80/20 as the splitting rule and built random forest with rank_position,hourly_price, num_work_completed, work_category_dummy

clf=RandomForestClassifier(n_estimators=100,class_weight=balanced)
clf.fit(X_train,y_train)
y_pred=clf.predict(X_test)

However the model's accuracy(Test accuracy) turned out to be 93% while the baseline is 94.3%.
The training accuracy is 99%. Compared to test accuracy 94.3%, I don't think there's a over-fitting problem The logistic regression also has the same problem. Based on correlation blot, most independent variables have pretty weak relationship with target variable smaller than +/- 0.3. what should I do next to improve my model accuracy? I tried parameter tuning but it doesn't help a lot.

Answer 1

This is a common problem with rare events modelling, and your options are relatively limited (as far as I am aware, at least). It may well be the case that the features you're using are not very informative with respect to predicting these outcomes.

The major issue is that your predictors, in the context of this model, are not very informative. The model tries to balance false positives and false negatives, but with so few true positives any mistakenly-predicted positive outcome will have a large effect on your classification accuracy.

It seems likely in this case that your predictors do not offer enough information to predict outcomes well. You may have reached the ceiling of what this model can do. This could be an artifact of the rarity of the "hired" outcome in your data set, or it could simply be that the relationship between these predictors and the outcome is weak.

There are a few options, involving use of different techniques (like a Firth regression, designed for rare events modelling). But using different predictors may be the best option, if it is possible to do so. Not every event can be modelled well with some arbitrary set of features, and it may be that you've found one of those.

Answer 2

As a complementary to Upper_Case's answer, you could just shuffle your labels randomly and do your training with these wrong labels again. If the results still don't change(it should get worse), it means that your input may actually not be very informative.