I'm new to machine learning so I'll summarize my problem with two examples without getting technical (because I can't).
The dog vs. cat classification example is solvable, in the sense that a human can tell you whether it's a dog or a cat with certainty. Many machine learning algorithms are able to replicate the human performance and identify dogs or cats with near certainty.
For my problem, there is no certainty, only a slight-better-than-random prediction. I am trying to predict whether a person who was recently released from incarceration will commit a crime within the next year. Let's assume the actual chances of re-offending are about 50/50. If I could use machine learning to make a modestly better than a random prediction, that would be a huge win for me. More specifically, if 50/50 is a random guess, then if I could achieve a
60% success rate, that would be considered wildly successful.
I know this task is possible since I have used a dataset (with around 50 features and 100,000 observations) to make a "man-made" linear regression that achieves around 52% out of the sample.
I have tried SKLearn's logistic regression and XGBoost but their performance has been lower than my
man-made attempt. I'm assuming that is because these algorithms aren't meant to deal with a prediction of an event that is mostly random.
Given that I am dealing with the prediction of an event that is mostly random and I am only looking to achieve slightly better than random predictions, is there a machine learning algorithm/strategy you could recommend to best tackle this problem?