Classification training using probabilites and not raw classes (factors)

Question

I have a problem where instead of having classes, i.e. a vector of 0s and 1s, I have the probability of an observation belonging to a class.

A vector with 0.1, 0.95, 0.2, 0.3, etc.

The obvious approach is using regression and it works relatively well, but I'm interested in an approach that trains on these probabilities that an observation belongs to a class and classifies them.

A Multinomial Classification approach has also been tried. The problem with this approach is that it doesn't take into consideration the order of the classes (factors) which causes in some cases that the highest predicted class for an observation is at one end (let's say 0.2) and the second one is at the other end (0.8). Further, the more classes there are the less scalable this approach it becomes. Conversely, if there are too few classes gaps in predictions start to appear.

So my question is: Is there a classification algorithm that supports training probabilities instead of just factors (0s and 1s)? Alternatively, is there another approach that is not regression nor classification that can solve this problem?

Answer 1

Beta Regression

You could use beta-regression. I have no practical experience with this type of regression. However, it might be the right method for your task. As far as I understand, the link function is chosen so to restrict $\hat{y} \in [0,1]$.

Here is an R implementation, where the docs say:

Fit beta regression models for rates and proportions via maximum likelihood using a parametrization with mean (depending through a link function on the covariates) and precision parameter (called phi).

Example:

library("betareg")
data("GasolineYield", package = "betareg")
summary(GasolineYield$yield)

Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
0.0280  0.1165  0.1780  0.1966  0.2705  0.4570 

br = betareg(yield ~ batch + temp, data = GasolineYield)
preds = predict(br, newdata=GasolineYield)
summary(preds)

Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
0.04571 0.10309 0.16364 0.19655 0.26429 0.50792 

Regression Models For Ordinal Data

Ordinal Logistic Regression could be used for this problem since classes are ordered and multinomial classification does not take the order of classes into consideration. In practice, this algorithm doesn't scale to many classes or many observations because its computationally expensive.

Here is an example of fitting a cumulative link model (CLM) such as the proportional odds model to data using the ordinal package in R.

require("ordinal")
fm1 <- clm(rating ~ contact + temp, data=wine)
summary(fm1)

formula: rating ~ contact + temp
data:    wine

link  threshold nobs logLik AIC    niter max.grad cond.H 
logit flexible  72   -86.49 184.98 6(0)  4.01e-12 2.7e+01

Coefficients:
       Estimate Std. Error z value Pr(>|z|)    
contactyes   1.5278     0.4766   3.205  0.00135 ** 
tempwarm     2.5031     0.5287   4.735 2.19e-06 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Threshold coefficients:
    Estimate Std. Error z value
1|2  -1.3444     0.5171  -2.600
2|3   1.2508     0.4379   2.857
3|4   3.4669     0.5978   5.800
4|5   5.0064     0.7309   6.850

Regression with a Logistic Link Function

As suggested by Ben Reiniger in the comments of the question, another alternative is simply to use a Logistic Link function in a regression model.

An example would be using xgboost with reg:logistic as the objective function. However, many libraries may not support this behavior as they need the target be either one or zero.

Answer 2

What you are describing is just the cross entry loss (also known as relative entropy or kullback-leibler divergence). If you have target probabilities that are one-hot you get the NLL form of it that is most commonly seen, however it is actually a loss that tries to match probability distributions. Simple solution to your problem would be a linear layer followed by softmax and then for example torch.nn.KLDivLoss or the equivalent in your favorite framework.

Answer 3

Never tried this, but I think it will work: Let's say you are using a neural network. Continue to use your normal y values (1, 0, etc), but make sure to hold in memory the respective probabilities for each classification (they all should be greater than 1/number of classes). Then, once the algorithm has calculated the loss for the sample, multiply the loss by the probability for that sample, making sure to do this before any backpropagation. I think you would have to do this at a fairly low level. Tensorflow should be able to get you there, though I'm less sure if something like Keras would.

Answer 4

What exactly is the problem you are trying to solve? Are you trying to map the probabilities to classes? If so, why not just assign the observation to the class with the highest probability?

If you really want to use a classification method for some reason, have you considered boosted trees?

There are a few advantages to this approach:

• No need to normalize inputs
• No need to calibrate probabilities prior to inputting them into the model
• This method is robust to correlation between input features

There are open source libraries available depending on your language of choice. I know it sounds simple, but based on your question, this seems like the simplest solution that can quickly meet your needs.

Answer 5

I believe if you could tell me what are your actual number of classes for classification that would have helped,any way lets suppose you have n classes one way to easily increase your accuracy is to simply use an optimized rounder function here i.e a function that takes your predicted probabilities as input and you have your target classes with there respective probability vectors to compute your accuracy now try to find the optimal coefficient value that partitions the [0,1] range in such a way that it maximizes your accuracy(or any metric that you are optimizing, partitions should be as many as there are classes you can use DP to solve this problem and get your coefficinets the best ones) after this you could use this same approach on predictions from other various models and then stack them and it should give you much better results.Another possible approach is to run many ml algorithms such as tree based regressors or ensemble based regressors or a mix of them and then predict probability of the class and using the results from various other models combine the results to form a new training set(keep the probabilities as it is) and then train a meta learner on top of it and for me this gave me much much better results this is actually the true principle behind using ensemble or stacking try to use models that vary in accuracy as well as stability and i think you should get much better results.