Linear regression on probabilistic data

I have a dataset which is probabilistic in nature. For instance, if a record has d features, then values of $x_1$, ..., $x_d$ are such that their sum adds to 1. In other words, for each $x_i$ data record (row)

$$P(x_{i,1}) + P(x_{i,2}) + ...+ P(x_{i,d}) = 1$$

I want to apply Linear Regression Model to predict a variable $y$.

My questions are:

1. can Linear Regression work for such datasets? What are other models I can use?
2. Should I drop one feature (some $x_{i,d}$) from each data record as I can easily calculate the value of dropped feature by using 1 - summing prob(remainingfeatures).
3. As I use Linear Regression all independent variables ($x_{i,0}$, $x_{i,d}$)'s are < 1 (because of the probability), the dependent variable is not. Should I use $log(y_i)$ instead (because weights $\mathbf{w}$ for $\hat{y}_i = \mathbf{w} \cdot \mathbf{x}_i$ calculated would be very large)?

Ideally, I am applying gradient descent for a convex optimization problem. It can be that $x_0...x_d$ belong to some unknown distribution and in my case, I know that distribution. But if there is an alternate model them pls suggest.

• I'd use a generative model: $P(X,Y) \equiv P(Y|X) P(X)$ where $P(Y|X)$ is Gaussian and $P(X)$ is Dirichlet. Then you can find the MLE, MAP, or Bayesian posterior over the regression coefficient.
– Emre
Jun 26 '17 at 20:51

The issue with linear regression in your case is not whether it can be used - it can nearly always be used to build a predictive regression model - it is whether the resulting model is useful, usually whether it is accurate enough for the intended use.

The accuracy and utility of your model does not depend only on the nature of $\mathbf{x}$. Instead it depends on the true nature of the $\mathbf{x} \rightarrow y$ relationship. Your measurements and model are an approximation of that relationship. Linear models are fast and stable to compute, but can be limited if the true relationship being approximated is non-linear.

Here are some basic thoughts/feedback on your questions:

1. Can Linear Regression work for such datasets?

Yes it can work. Will it be good enough? You will know after testing.

What are other models I can use?

Almost any regression model class could be applied to this problem. For a quick comparison, to see if a non-linear model will make more accurate predictions for you, then you could try an easy-to-apply model from an existing library, such as XGBoost (which has a stand-alone command-line version).

1. Should I drop one feature (some $x_{i,d}$) from each data record as I can easily calculate the value of dropped feature by using 1 - summing prob(remainingfeatures).

For a linear model, your intuition is correct that one feature is redundant, as it is itself a linear combination of the other features, so there is no impact to removing it. You could remove any single column. You should not need to though, and some model classes might work better if left in. As always, experiment if you are not sure, and use hold-out test data to verify your ideas (or even better a separate cross-validation set before final test stage)

1. As I use Linear Regression all independent variables ($x_{i,0}$, $x_{i,d}$)'s are < 1 (because of the probability), the dependent variable is not. Should I use $log(y_i)$ instead (because weights $W$ for $\hat{y}_i = W\mathbf{x}_i$ calculated would be very large)?

For linear regression this scaling is not very important. However, there are a few caveats:

• If you measure success by an error metric (such as mean square error), then to compare different ideas fairly, you should report the test error in the same way - e.g. you should scale back the $log(y)$-based predictions and calculate error on the same $(y - \hat{y})^2$ each time.

• For some model classes, they could work better if the target variable is within a certain range. You should check documentation for this.

• How useful the mapping is depends on the true nature of the relationship between $\mathbf{x}$ and $y$. Specifically for a linear model, you have completely changed the line you are trying to fit - it could as a result be a lot better or lot worse, but that is not dependent just on $\mathbf{x}$. So again you will have to give it a try and test the results.