3
$\begingroup$

I have a fairly small dataset of 225 points. I have a target (labelled numeric), a feature (normalised numeric) and a quality index with set of normalised weights that describe the likelihood that the observation will contribute to a linear relationship between target and feature. Good data for weighted linear regression.

The dataset is currently at only 225 points but it's growing so I'd like to explore DNNs to improve on the performance of the weighted linear approach. Can Keras do this?

What I am curious about is the possibility that the weight optimization process in the DNN training could itself eliminate points with a low quality index, i.e. those points that do not contribute to a linear fit and result in a high loss function return.

But I am very new to deep learning so my understanding could be wrong.

$\endgroup$
  • $\begingroup$ You mention that you want to use a CNN network. So your feature is in fact an image ? $\endgroup$ – Adrien D Jul 6 '18 at 8:03
  • $\begingroup$ Sorry, typo. this problem is DNN $\endgroup$ – Patrice Carbonneau Jul 7 '18 at 17:55
1
$\begingroup$

This is a suggestion. I don't know if this will work. Maybe you could adapt a feature selection problem to a data point selection problem. For a standard linear regression, The LASSO technique can be used to do both the regression and the feature selection. The idea is to penalize the use of too much features by adding a term to the loss.

enter image description here

Where beta are the coefficients to be learned of your linear regression.


With only one feature x_i and with your quality index q_i (they should sum to 1) your loss can be written as :

enter image description here

I see 2 ways of doing this :

  • Minimize the loss above. beta_1 and beta_0 are the only coefficients that can be learned. The process will not eliminate the points with low quality but they will contribute just a little to the loss.

  • Consider the q_i as learnable parameters. You don't use the quality index that you already have. And then you can add a penalization term on the q_i. Those who end up to 0 will not contribute to the loss. The loss will look like this :

    enter image description here

I think there are many details to solve. Like adding a constraint to the q_i like 0 < q_i < 1. Maybe this can not be solved easily. I suggest you to use Pytorch. This will be way much easier to implement.

|improve this answer|||||
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.