3
$\begingroup$

I have a problem where instead of having classes, i.e. a vector of 0s and 1s, I have the probability of an observation belonging to a class.

A vector with 0.1, 0.95, 0.2, 0.3, etc.

The obvious approach is using regression and it works relatively well, but I'm interested in an approach that trains on these probabilities that an observation belongs to a class and classifies them.

A Multinomial Classification approach has also been tried. The problem with this approach is that it doesn't take into consideration the order of the classes (factors) which causes in some cases that the highest predicted class for an observation is at one end (let's say 0.2) and the second one is at the other end (0.8). Further, the more classes there are the less scalable this approach it becomes. Conversely, if there are too few classes gaps in predictions start to appear.

So my question is: Is there a classification algorithm that supports training probabilities instead of just factors (0s and 1s)? Alternatively, is there another approach that is not regression nor classification that can solve this problem?

$\endgroup$
  • $\begingroup$ Just an idea: you could build another dataset which simulates the distribution, i.e. if an instance has probs [0.3,0.2,0.5] of belonging to classes [A,B,C] you create say 100 instances: 30 labellied with A, 20 with B and 50 with C. $\endgroup$ – Erwan Aug 11 at 0:54
  • $\begingroup$ can you treat the probs as classes? $\endgroup$ – Peter Aug 11 at 11:24
  • $\begingroup$ I forgot to mention that I already tried a multiclass approach. However, it is hard to translate predictions back to probabilities and training probabilities as classes ignore the order of the factors. In practice it means that oftentimes the winning class is located at one side (for instance at 0.2) and the second highest probability is located at the opposite side (at 0.8). $\endgroup$ – wacax Aug 12 at 13:01
  • 1
    $\begingroup$ Why not regression? You have continuous outputs. You can use a logistic link function to get the right probability range, and from there you can use basically any model you like. Logistic regression, tree ensembles, neural nets with a final sigmoid activation, should all work without having 0/1 labels, though implementations may be lacking. E.g., with xgboost: datascience.stackexchange.com/a/57067/55122 , but not sklearn LogisticRegression: stackoverflow.com/q/47663569/10495893 $\endgroup$ – Ben Reiniger Aug 16 at 18:19
  • $\begingroup$ Probability can be anything between $0$ & $1$. When the number of classes tends to be infinite the classification problem tends to be the regression problem. I am not sure about the validity of your question. Can you explain further? $\endgroup$ – Mr.Sigma. Aug 17 at 6:35
2
+50
$\begingroup$

Beta Regression

You could use beta-regression. I have no practical experience with this type of regression. However, it might be the right method for your task. As far as I understand, the link function is chosen so to restrict $\hat{y} \in [0,1]$.

Here is an R implementation, where the docs say:

Fit beta regression models for rates and proportions via maximum likelihood using a parametrization with mean (depending through a link function on the covariates) and precision parameter (called phi).

Example:

library("betareg")
data("GasolineYield", package = "betareg")
summary(GasolineYield$yield)

Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
0.0280  0.1165  0.1780  0.1966  0.2705  0.4570 

br = betareg(yield ~ batch + temp, data = GasolineYield)
preds = predict(br, newdata=GasolineYield)
summary(preds)

Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
0.04571 0.10309 0.16364 0.19655 0.26429 0.50792 

Regression Models For Ordinal Data

Ordinal Logistic Regression could be used for this problem since classes are ordered and multinomial classification does not take the order of classes into consideration. In practice, this algorithm doesn't scale to many classes or many observations because its computationally expensive.

Here is an example of fitting a cumulative link model (CLM) such as the proportional odds model to data using the ordinal package in R.

require("ordinal")
fm1 <- clm(rating ~ contact + temp, data=wine)
summary(fm1)

formula: rating ~ contact + temp
data:    wine

link  threshold nobs logLik AIC    niter max.grad cond.H 
logit flexible  72   -86.49 184.98 6(0)  4.01e-12 2.7e+01

Coefficients:
       Estimate Std. Error z value Pr(>|z|)    
contactyes   1.5278     0.4766   3.205  0.00135 ** 
tempwarm     2.5031     0.5287   4.735 2.19e-06 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Threshold coefficients:
    Estimate Std. Error z value
1|2  -1.3444     0.5171  -2.600
2|3   1.2508     0.4379   2.857
3|4   3.4669     0.5978   5.800
4|5   5.0064     0.7309   6.850

Regression with a Logistic Link Function

As suggested by Ben Reiniger in the comments of the question, another alternative is simply to use a Logistic Link function in a regression model.

An example would be using xgboost with reg:logistic as the objective function. However, many libraries may not support this behavior as they need the target be either one or zero.

$\endgroup$
  • $\begingroup$ And, see the first paragraph of the vignette for an argument in favor of this over my comment's suggestion. $\endgroup$ – Ben Reiniger Aug 16 at 21:10
  • $\begingroup$ (I was about to expand my comment to an answer, when it got included here, which is also fine. Just wanted to add the following one thought.) To get around library support, you could transform to log-odds (now the range is all reals) and fit using any of your favorite methods. This will be different in the objective function, but may be useful in some cases. $\endgroup$ – Ben Reiniger Aug 21 at 18:17
1
$\begingroup$

What you are describing is just the cross entry loss (also known as relative entropy or kullback-leibler divergence). If you have target probabilities that are one-hot you get the NLL form of it that is most commonly seen, however it is actually a loss that tries to match probability distributions. Simple solution to your problem would be a linear layer followed by softmax and then for example torch.nn.KLDivLoss or the equivalent in your favorite framework.

$\endgroup$
0
$\begingroup$

Never tried this, but I think it will work: Let's say you are using a neural network. Continue to use your normal y values (1, 0, etc), but make sure to hold in memory the respective probabilities for each classification (they all should be greater than 1/number of classes). Then, once the algorithm has calculated the loss for the sample, multiply the loss by the probability for that sample, making sure to do this before any backpropagation. I think you would have to do this at a fairly low level. Tensorflow should be able to get you there, though I'm less sure if something like Keras would.

$\endgroup$
0
$\begingroup$

What exactly is the problem you are trying to solve? Are you trying to map the probabilities to classes? If so, why not just assign the observation to the class with the highest probability?

If you really want to use a classification method for some reason, have you considered boosted trees?

There are a few advantages to this approach:

  • No need to normalize inputs
  • No need to calibrate probabilities prior to inputting them into the model
  • This method is robust to correlation between input features

There are open source libraries available depending on your language of choice. I know it sounds simple, but based on your question, this seems like the simplest solution that can quickly meet your needs.

$\endgroup$
  • 1
    $\begingroup$ I guess his $y$ are probs and he wants to model this in a proper way (meaning restricted to the interval $[0,1]$). $\endgroup$ – Peter Aug 16 at 21:51
0
$\begingroup$

I believe if you could tell me what are your actual number of classes for classification that would have helped,any way lets suppose you have n classes one way to easily increase your accuracy is to simply use an optimized rounder function here i.e a function that takes your predicted probabilities as input and you have your target classes with there respective probability vectors to compute your accuracy now try to find the optimal coefficient value that partitions the [0,1] range in such a way that it maximizes your accuracy(or any metric that you are optimizing, partitions should be as many as there are classes you can use DP to solve this problem and get your coefficinets the best ones) after this you could use this same approach on predictions from other various models and then stack them and it should give you much better results.Another possible approach is to run many ml algorithms such as tree based regressors or ensemble based regressors or a mix of them and then predict probability of the class and using the results from various other models combine the results to form a new training set(keep the probabilities as it is) and then train a meta learner on top of it and for me this gave me much much better results this is actually the true principle behind using ensemble or stacking try to use models that vary in accuracy as well as stability and i think you should get much better results.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.