I'm new to machine learning so I'll summarize my problem with two examples without getting technical (because I can't).
The dog vs. cat classification example is solvable, in the sense that a human can tell you whether its a dog or a cat with certainty. Many machine learning algorithms are able to replicate the human performance and identify dog or cat with near certainty.
For my problem, there is no certainty, only a slight-better-than-random prediction. I am trying to predict whether a person who was recently released from incarceration will commit a crime within the next year. Lets assume the actual chances of re-offending are about 50/50. If I could use machine learning to make a modestly-better-than-random prediction, that would be a huge win for me. More specifically, if 50/50 is a random guess, then if I could achieve 55% to 60% success rate, that would be considered wildly successful.
I know this task is possible since I have used a dataset (with around 50 features and 100,000 observations) to make a "man-made" linear regression which achieves around 52% out of sample.
I have tried SKLearn's logistic regression and XGBoost but their performance has been lower than my "man made" attempt. I'm assuming that is because these algorithms aren't meant to deal with a prediction of an event that is mostly random.
Given that I am dealing with the prediction of an event which is mostly random and I am only looking to achieve slightly-better-than-random predictions, is there a machine learning algorithm / strategy you could recommend to best tackle this problem?