Weighted linear regression with a DNN (in Keras)

Question

I have a fairly small dataset of 225 points. I have a target (labelled numeric), a feature (normalised numeric) and a quality index with set of normalised weights that describe the likelihood that the observation will contribute to a linear relationship between target and feature. Good data for weighted linear regression.

The dataset is currently at only 225 points but it's growing so I'd like to explore DNNs to improve on the performance of the weighted linear approach. Can Keras do this?

What I am curious about is the possibility that the weight optimization process in the DNN training could itself eliminate points with a low quality index, i.e. those points that do not contribute to a linear fit and result in a high loss function return.

But I am very new to deep learning so my understanding could be wrong.

Answer 1

This is a suggestion. I don't know if this will work. Maybe you could adapt a feature selection problem to a data point selection problem. For a standard linear regression, The LASSO technique can be used to do both the regression and the feature selection. The idea is to penalize the use of too much features by adding a term to the loss.

Where beta are the coefficients to be learned of your linear regression.

---

With only one feature x_i and with your quality index q_i (they should sum to 1) your loss can be written as :

I see 2 ways of doing this :

• Minimize the loss above. beta_1 and beta_0 are the only coefficients that can be learned. The process will not eliminate the points with low quality but they will contribute just a little to the loss.

• Consider the q_i as learnable parameters. You don't use the quality index that you already have. And then you can add a penalization term on the q_i. Those who end up to 0 will not contribute to the loss. The loss will look like this :

  

I think there are many details to solve. Like adding a constraint to the q_i like 0 < q_i < 1. Maybe this can not be solved easily. I suggest you to use Pytorch. This will be way much easier to implement.