Random Forest Classifier gives very high accuracy on test set - overfitting?

Question

I have a financial dataset, where I'm trying to predict company types, based on the amount dollars, what time of day, and whether they buy or sell (currency pairs). It looks like this:

The features I use to predict:

X.head():

Dollars   | Hours      | Buy | Sell
-0.761916   0.364838     1     0
-0.924413   0.377558     1     0
-0.573336   0.397836     0     1
-0.561639   0.399144     0     1
-1.164036   0.423715     1     0

The features I want to predict could look like this:

y.head()
Bank  Tech  Fund  Holding  Defence  Financial Services  Pharma 
1     0     0     0        0        0                   0   
1     0     0     0        0        0                   0   
1     0     0     0        0        0                   0   
1     0     0     0        0        0                   0   
1     0     0     0        0        0                   0   

Agriculture  Commodities  Energi  Pension  
0            0            0       0  
0            0            0       0  
0            0            0       0  
0            0            0       0  
0            0            0       0  

In this snippet, the first five companies are banks.

Using a training/test ratio of 0.25, I get an accuracy of 0.99, which seems too good to be true:

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25)

rand_forest = RandomForestClassifier(max_depth = None, random_state = 0)

rand_forest.fit(X,y)

predictions = rand_forest.predict(X_test)

The result of the classification_report:

         precision    recall  f1-score   support

      0       0.98      0.95      0.97      5074
      1       0.98      0.91      0.94      2292
      2       0.98      0.82      0.89       572
      3       0.99      0.83      0.90       235
      4       0.98      0.91      0.94       261
      5       0.99      0.81      0.89       411
      6       0.98      0.83      0.90       239
      7       1.00      0.70      0.82       144
      8       1.00      0.81      0.89       384
      9       0.99      0.81      0.89       200
     10       1.00      0.81      0.90       232

avg / total   0.98      0.90      0.94     10044

Adjusting the max_depth parameter of the classifier changes this number significantly though, but I'm still reading up on what the consequences of that parameter actually are.

It is worth mentioning that there is only 50,000 entries in this dataset, across 11 different companytypes, which might be too little?

Using a simpler DecisionTreeClassifier yiels an accuracy of about 50%.

UPDATE:

I used the entire dataset for training, not the actual training set. Switching these two outs gives an accuracy of 54%, which sounds much better (or more realistic anyways).

Answer 1

rand_forest.fit(X,y)

Why are you using the whole data set for training? You are using the test set for training then evaluate the performance on it again?

In your code, I didn't see you actually used the training set you created.