I am trying to fit a curve on the data( in the attached imageenter image description here). I see that there is a lot of variance in the response variable for each explanatory variable value. I am not sure how to model this. I tried a polynomial curve(GAM) but at best it is explaining only around 10% of the data. I understand that this indicates that I need to add more variables to the model. However, I need to avoid going down that path.

I have tried transforming the explanatory variable but haven't found much luck. I know there is very limited things we can do with such distribution but just throwing it out to all the brilliant minds out there if some one has had to do something similar or if someone has some ideas I could try.

I have attached an image of the distribution as I am not sure how to attach a data file.

Let me know if I haven't provided enough information in my request and I will try to add more details.

Thanks, Suhail

  • $\begingroup$ your graph can a bit misleading. E.g. is this for one variable? If yes then you re plotting the one-one relation of a variable against your target? Is you data time driven, meaning you have time in your data? Please post a bit of your data. You do not need to upload your datafile, just paste a few rows with all variables and target perhaps using print(df.head(5)). $\endgroup$ – TwinPenguins Jul 17 '19 at 4:43
  • $\begingroup$ Yes, This is a one to one relation. Let's say the explanatory variable is time period. Say, the time it took me to make a sale from the date the product was launched. The response variable is the sale amount. The challenge is that I have a lot of variability in the sale amount for the same amount of time it took me to make the sale. Here are a few records : Explanatory Response 19 429926 12 28069 13 65461 12 233119 15 68520 16 222355 17 216042 Also, as is evident from the graph, I have added transparency on the data points to show where most of the data lies. $\endgroup$ – Suhail Wali Jul 18 '19 at 1:02
  • $\begingroup$ It goes down to the question you are trying to answer. Are you going to precit foe example how many you may sell tomorrow? In these case, why doesn't aggregate your sale counts (for example mean) on daily basis instead of looking at each data point? In some business use cases it is enough even to look at weekly or monthly basis. Then in aggregate view you will be have a much less noisier data points! $\endgroup$ – TwinPenguins Jul 18 '19 at 5:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.