As the comments suggest, it's not always helpful to think of points outside the whiskers as "outliers". What you are seeing in these boxplots is a strong positive skew. And yes, a strongly skewed target is typically harder to predict than a less-skewed target.
In traditional survival analysis, it's common to model this skewed data using a parametric probability distribution that naturally produces positive and positive-skewed data, e.g. the Weibull distribution.
Otherwise, you can also try a Box-Cox or inverse hyperbolic sine (IHS) transformation on the survival time to reduce the skew.
For visualizing strongly-skewed data, you can either apply one of the above skew-reducing transformations, or use an "adjusted" boxplot, which have an R implementation.