0
$\begingroup$

I have a data with the following columns

    col1         col2       col3        col4        label
    7669.533073 7669.533073 7669.695497 7669.922593 1
    7669.533043 7669.533072 7669.695487 7669.922596 0

the mean across all the 50 columns are similar and also the min and maximum.

I am trying to build a classifier and the best model(random forest) is giving me a recall of .55 (doesn't seem so good), could there be anything I am missing in this?

I have thought about normalising the data but there seems to be no need as all columns have a similar mean and std.

Is there any statistics technique I could apply to the data to help get an improved result.

Note the data is from a simulated crypto price and I am trying to predict the price movement (up or down)

$\endgroup$
0
$\begingroup$

Your datapoints are too close to each other and hence it is really tough for any ML model to learn this inputs as it doesn't know how to differentiate almost same data to 1 and 0 label. That's why the result is random and you are getting around half accuracy.

| improve this answer | |
$\endgroup$
0
$\begingroup$

If the data values are this close together, it's possible the slight differences in values could be due to, or at least masked by, measurement error. If this is the case, you won't be able to model the data accurately, as measurement error is typically random, not related to any label that is attached. Also curious about the high precision of the data, with 10 significant digits. Decimal side is down to the millionths column, even with data values being in the thousands.

| improve this answer | |
$\endgroup$
  • $\begingroup$ The data is from a crypto simulation $\endgroup$ – funkyFunk Jun 24 at 13:48
  • $\begingroup$ Ok, not sure if this case is special, but usually 10 significant digits will be too many. With 10 digits, the last few would usually be more noise than anything. $\endgroup$ – Donald S Jun 24 at 14:30
  • $\begingroup$ However, if you think the last few digits are real, you could subtract the minimum from the value, separately for each column, to get data that is more useful for modelling. You will need to do a lot of data exploration to clean up this data to see any signal that is present. Something else to consider, if any model you use does any rounding internally, the differences would be removed. This is another reason to do the subtraction, or something similar. $\endgroup$ – Donald S Jun 24 at 14:30
  • $\begingroup$ To address the 10 digits issue, would normalising the data be a good idea? And what kind of exploration would you suggest? Ive tried checking Descriptive stats $\endgroup$ – funkyFunk Jun 24 at 15:30
  • $\begingroup$ Normalizing would be okay, but i would still do the subtraction first, before doing anything else with the data. Not sure how the common normalizing functions would handle this many significant digits, if there is any rounding there also. For data exploration, I usually like to look at histograms to start with. df.hist(bins=20, figsize=(20,15)) $\endgroup$ – Donald S Jun 24 at 15:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.