Assuming I am having the following data:

| f1  | f2  | f3  | f4  | output|
| 0.1 | 0.7 | 0.4 | 0.3 |  1    |
| 0.4 | 0.1 | 0.2 | 0.5 |  0    |
| 0.3 | 0.7 | 0.4 | 0.7 |  0    |
| 0.1 | 0.2 | 0.4 | 0.7 |  1    |

And I am training the data with logistic regression or another classifier. Is there a way to say 'I want an example that outputs 1'? Or is there a way to 'formulate' the 'borders' of the labels?

  • $\begingroup$ Can you specify your use case for this question? That might lead to a more useful answer $\endgroup$
    – wabbit
    May 17 '16 at 17:57

As with anything, it depends. :) Your question will very much depend on the model.

This question is reminiscent of the broader debate about discriminative versus generative models. Usually, we work with discriminative models. That is, you model $\text{P}(Y=1\,|\,X_1, X_2, \dots)$, where $Y$ are your classes and $X_i$ your observations.

There is another class of models: generative models, like Restricted Boltzmann Machines, which model joint distributions, $\text{P}(Y,X_1, X_2, \dots)$. They can be used for classification because you can easily turn a joint probability into a conditional probability. These models usually are broader in scope, and usually do not perform as well, which is why they aren't used as much. But they do exist.

Assuming that you are talking about discriminative models, like decision trees or the logistic regression, then what you want is how to invert this conditional probability, so that you have a distribution along $X_i$; that is, $\text{P}(X_1=x_1, X_2=x_2, \dots \,|\, Y)$.

From Bayes's theorem, $\text{P}(Y=1\,|\,X_1=x_1, X_2=x_2, \dots)=\text{P}(X_1=x_1, X_2=x_2, \dots \,|\, Y=1)\frac{\text{P}(Y=1)}{\text{P}(X_1=x_1, X_2=x_2, \dots)}.$

So, yes, you can. Where is the catch?

$X$ and $Y$ have to be discrete. It is very easy to create a probability density function from categorical variables by just getting conditional probabilities out of your model by just trying all combinations of $X$. And, once you have a probability density function, then you can just use a sampling technique to generate values.

It is less clear to me how you could make this work for continuous values, unless you are willing to use an approximation by discretizing your observations. I think an exact solution will depend very much on the discriminative model at hand. You cannot do this alone by getting conditional probabilities out of the model.

But if sampling is important for you, then you should just use a generative model.


Let s1 to s4 be the signs (+1 or -1) of the regression coefficients of f1 to f4.

Then the predicted value for a data item with columns s1*Inf to s4*Inf will be (as near as precision allows it) 1. You go to the extreme of the parameter space in the right direction (as told by the coefficients signs).

Inverting the signs and doing the same should give you a data item that is predicted to be 0.

Example, code in R

Create sample data set.

d2 = data.frame(f1=runif(10), f2 = runif(10), f3 = runif(10),

Fit model:

m2 = glm(output~f1+f2+f3, data=d2, family="binomial")

Look at model parameters:

## (Intercept)          f1          f2          f3 
##   5.258817   -4.848013  -15.370838    4.890984 

This means as f1 and f2 increase, the fitted probability of zero increases because their signs are negative. As f3 increases, the fitted probability of one increases because it has a positive sign. So....

predict(m2, newdata=data.frame(f1=-Inf, f2=-Inf, f3=Inf), type="response")

should be one, which it is, and:

predict(m2, newdata=data.frame(f1=Inf, f2=Inf, f3=-Inf), type="response")

should be zero, which it is, within tolerance.

Zeroes and ones in logistic regression responses are like plus or minus infinity in a linear regression, so if you think back to a simple X-Y fit, you get an infinite response to a linear regression when you go to +Inf on the X-axis if the slope is positive or -Inf on the X-axis if the slope is negative. Transform that to a logistic regression and then you get to +1 or 0 at the appropriate signed infinities of the covariates.

  • $\begingroup$ Thanks for the reply, but I am not sure if I understand ur answer. Can u make it more clear? $\endgroup$ May 16 '16 at 11:03
  • $\begingroup$ @MpizosDimitris Basically what I think he is saying is that, after estimating the coefficients of the logistic function, you could potentially manipulate it to get the inverse function. I think he is right, but I am not sure it would help to build a probability density function which is what you need to then sample observations. $\endgroup$ Jun 15 '16 at 15:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.