3
$\begingroup$

I have a financial dataset, where I'm trying to predict company types, based on the amount dollars, what time of day, and whether they buy or sell (currency pairs). It looks like this:

The features I use to predict:

X.head():

Dollars   | Hours      | Buy | Sell
-0.761916   0.364838     1     0
-0.924413   0.377558     1     0
-0.573336   0.397836     0     1
-0.561639   0.399144     0     1
-1.164036   0.423715     1     0

The features I want to predict could look like this:

y.head()
Bank  Tech  Fund  Holding  Defence  Financial Services  Pharma 
1     0     0     0        0        0                   0   
1     0     0     0        0        0                   0   
1     0     0     0        0        0                   0   
1     0     0     0        0        0                   0   
1     0     0     0        0        0                   0   

Agriculture  Commodities  Energi  Pension  
0            0            0       0  
0            0            0       0  
0            0            0       0  
0            0            0       0  
0            0            0       0  

In this snippet, the first five companies are banks.

Using a training/test ratio of 0.25, I get an accuracy of 0.99, which seems too good to be true:

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25)

rand_forest = RandomForestClassifier(max_depth = None, random_state = 0)

rand_forest.fit(X,y)

predictions = rand_forest.predict(X_test)

The result of the classification_report:

         precision    recall  f1-score   support

      0       0.98      0.95      0.97      5074
      1       0.98      0.91      0.94      2292
      2       0.98      0.82      0.89       572
      3       0.99      0.83      0.90       235
      4       0.98      0.91      0.94       261
      5       0.99      0.81      0.89       411
      6       0.98      0.83      0.90       239
      7       1.00      0.70      0.82       144
      8       1.00      0.81      0.89       384
      9       0.99      0.81      0.89       200
     10       1.00      0.81      0.90       232

avg / total   0.98      0.90      0.94     10044

Adjusting the max_depth parameter of the classifier changes this number significantly though, but I'm still reading up on what the consequences of that parameter actually are.

It is worth mentioning that there is only 50,000 entries in this dataset, across 11 different companytypes, which might be too little?

Using a simpler DecisionTreeClassifier yiels an accuracy of about 50%.

UPDATE:

I used the entire dataset for training, not the actual training set. Switching these two outs gives an accuracy of 54%, which sounds much better (or more realistic anyways).

$\endgroup$
1
  • $\begingroup$ 50,000 - important distinction :) $\endgroup$
    – Khaine775
    Commented Oct 25, 2017 at 13:29

1 Answer 1

4
$\begingroup$

rand_forest.fit(X,y)

Why are you using the whole data set for training? You are using the test set for training then evaluate the performance on it again?

In your code, I didn't see you actually used the training set you created.

$\endgroup$
1
  • 1
    $\begingroup$ Ooooh, that's right and a horrible typo by me. Of course that gives close to 100%, because I'm using the entire dataset for training. When I use the actual training set, I get 54%. Thanks for the pointer! $\endgroup$
    – Khaine775
    Commented Oct 25, 2017 at 13:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.