1
$\begingroup$

The features are of capital gain and capital loss, but very small amounts of people have one or the other. As far as I can see it may not signify any great difference, though I'm not sure how I would go about even normalising or whatnot these features:

     age        wgt       sex   gain        loss         hpw        
0   0.379310    0.119825    1   2174           0         -0.078664          
1   0.568966    0.131898    1   0              0         -2.327679          
2   0.362069    0.407596    1   0              0         -0.078664

The gain can skyrocket to 100,000 and that is definitely of significance considering I'm predicting whether someone will make over $50k, though they occur at such a small rate, close to 3-5% that I'm not sure how I could go about implementing them. I doubt imputing data would be a good idea considering just about everyone is on 0 for both results.

Also to note is I have 15 columns total (many more with get_dummies()).

$\endgroup$
4
  • $\begingroup$ It isn't clear what you are asking.. $\endgroup$
    – Mark.F
    Aug 16 '19 at 8:46
  • $\begingroup$ @Mark.F asking how I should approach this data, which if used correctly, could be used to tell the machine whether or not something is 100% possibility, though also considering it only occurs in 3-5% of rows, perhaps it is useless? $\endgroup$
    – bemzoo
    Aug 16 '19 at 8:50
  • $\begingroup$ This question could definitely be improved: Start with your goal, then list the data in birds eye. Next your idea's, then the question, finally add detailed data or functions. $\endgroup$ Aug 16 '19 at 10:36
  • $\begingroup$ @S van Balen my goal is max accuracy. Not sure what you mean by birds eye (sorry I'm new to programming), my idea was to come here and ask because I have a lack of ideas, my only idea at the moment is to drop the columns, which is fine but my goal is max accuracy. Question is stated clearly I would have thought, what do I do with these 2 columns. Would you like all the data? I'm not looking for a completed answer, just to push me in the right direction. What functions would you want? All I've provided is a dataset, didn't realise datascience was so focused on the coding aspect. $\endgroup$
    – bemzoo
    Aug 18 '19 at 22:27
1
$\begingroup$

A tree-based model, especially one that allows missing data in the input, seems a good choice: the tree(s) can split large capital gains out (presumably separating a nearly pure node from the rest).

Also consider something more basic: just hard-code an exception for those with large gains, and model for the rest.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.