0
$\begingroup$

I have a trading system where the model receives 9 time-series and predict :

A - strong down
B - week down
C - neutral 
D - week up
E - strong up

(these classes are generated from an histogram to have a balanced training dataset ... the histogram is separated in 20% parts of examples centered in 0... accuracy of 20% is the baseline)

For each class I activate a different parametrized trading mechanism...

My model is giving acceptable results. Here is the resulting confusion matrix for val data (28.99% acc):

[[32 20  3  6  8]
 [35 19  9  7 16]
 [30  9  6 14 24]
 [21 14  9 18 29]
 [ 9 14  3 14 45]]

My question starts here :

I.e. If the model predicts B-"week down" but in reality is A-"strong down" is a miss, but in reality, it will make money...

So in this confusion matrix we can see that it happens 20 times (cell[0,1]) ... also if it is B but the model says A it will make money in 35 trades (cell[1,0])...

And also the same for the UP cases..

All together (from the confusion matrix) : 32+20+35+19 + 18+29+14+45 = 212 winning trades 21+14+9+14 + 6+8+7+16 = 95 losing trades

Assuming negative trades cancel in equal (in reality will not be equal..) positive trades, the total is = 117 winning trades.

What I want is to create a loss function based on categorical_crossentropy but somehow consider:

  • pred A real B - half miss
  • pred B real A - half miss
  • pred D real E - half miss
  • pred E real D - half miss

Do not penalize too much this cases. I think this will increase a bit the total number of positive trades. It will guide the learning a litle bit better (maybe not for accuracy but for a better loss that generates a better confusion matrix for profit )...

I have created a custom loss function that reduces 3% the loss for these cases:

def my_loss(y_pred, y_true):

    y_pre_indexes = K.argmax(y_pred, axis=1) 
    y_test_indexes= K.argmax(y_true, axis=1)

    TN = K.tf.logical_or( K.tf.logical_or (K.tf.logical_and(K.equal(y_pre_indexes,0),K.equal(y_test_indexes,1)),
                          K.tf.logical_and(K.equal(y_pre_indexes,1),K.equal(y_test_indexes,0))) 
                        ,
                          K.tf.logical_or (K.tf.logical_and(K.equal(y_pre_indexes,3),K.equal(y_test_indexes,4)),
                          K.tf.logical_and(K.equal(y_pre_indexes,4),K.equal(y_test_indexes,3))))

    pos_neg = K.cast(TN, K.floatx()) *(-0.03) + 1 

    return K.categorical_crossentropy(y_pred, y_true)*pos_neg 

(in the code the classes are : 0-A 1-B 3-D 4-E. 2-C is predicting neutral - ignore..)

but fixing to a fixed number of 3% to reduce loss for these cases seems a little bit hard coded.... Something better inside the categorical_crossentropy math philosophy should be better.

Any suggestions?

Thanks in advance!

$\endgroup$
1
$\begingroup$

I worked on a similar problem where classes were "ordered", so mis-classifications of class B as C was better than mis-classification as D.

In that project, "weighted quadratic kappa" loss function worked well.

Details :

https://en.wikipedia.org/wiki/Cohen%27s_kappa#Weighted_kappa

Example of Keras implementation :

https://github.com/benhamner/Metrics/blob/master/Python/ml_metrics/quadratic_weighted_kappa.py

| improve this answer | |
$\endgroup$
  • $\begingroup$ thanks I will use this loss function to test is the confusion matrix become better for my case study ... $\endgroup$ – rjpg Jan 31 '19 at 16:10
  • $\begingroup$ One difference for me is that "proximity" in my case only applies between class (A and B) or (D and E). e.g. (B and C) should not consider "proximity"... But I will test it anyway. I'm also implementing now a new custom loss the will do the average of loss for my special cases (I will build a new y_test inside the loss function that alter the correct output for the special cases and when they occur I average - only this cases - the test with the real y_test and the new one. This will reduce loss accordingly with the probabilities of the prediction, only when the special cases appear) $\endgroup$ – rjpg Jan 31 '19 at 16:22
  • $\begingroup$ You can provide custom function of providing weight given two classes. With that you can instruct loss function to assign different weights for B & C (as compared to A & B / D & E) $\endgroup$ – Shamit Verma Feb 1 '19 at 4:08
  • $\begingroup$ is there any implementation of that function in keras ? $\endgroup$ – rjpg Feb 1 '19 at 18:48
0
$\begingroup$

I think I have a solution.

In special cases I do the average of two crossentropy loss, one assuming wrong and other assuming right ("half correct") taking in consideration the probabilities of the output softmax of the model. Again ONLY IN THE "SPECIAL CASES" :-) .

It is in keras TF so we have to do all things in "declarative vectorial programming" (no cycles or if's ) :

def my_loss(y_pred, y_true):

    y_pre_indexes = K.argmax(y_pred, axis=1) 
    y_test_indexes= K.argmax(y_true, axis=1)

    #identify special cases
    #True special cases - False all others 
    TN = K.tf.logical_or( K.tf.logical_or (K.tf.logical_and(K.equal(y_pre_indexes,0),K.equal(y_test_indexes,1)),
                          K.tf.logical_and(K.equal(y_pre_indexes,1),K.equal(y_test_indexes,0))) 
                        ,
                          K.tf.logical_or (K.tf.logical_and(K.equal(y_pre_indexes,3),K.equal(y_test_indexes,4)),
                          K.tf.logical_and(K.equal(y_pre_indexes,4),K.equal(y_test_indexes,3))))


    #1.00 special cases - 0.00 all athoers (float)
    TN_float = K.cast(TN, K.floatx())

    #1 special cases - 0 all athoers 
    TN_int=K.tf.cast(TN_float,K.tf.int64)

    #0 special cases - 1 all others
    TN_int_inverse=(TN_int*(-1))+1

    # make a tensor with all 0's but the predictions of special cases (in indexes)
    classes_only_TN=TN_int*y_pre_indexes

    # make 0 the special cases in y_test indexes 
    real_classes_remove_TN=TN_int_inverse*y_test_indexes

    # sum the two and we get a simulated y_test with the special cases correct and all other normal
    simulated_index=classes_only_TN+real_classes_remove_TN

    #make this simulated y_test one_hot to feed crossentropy
    simulated_one_hot=(K.tf.one_hot(simulated_index,5))

    #get the result of crossentropy assuming special cases correct (the simulated y_test)
    crosentr_result_sim=K.categorical_crossentropy(y_pred, simulated_one_hot)

    #get the result of crossentropy assuming special cases wrong (the real y_test)
    crosentr_result_real=K.categorical_crossentropy(y_pred, y_true)

    # make the float 1.00 that identify special cases be 0.5, all othes continue 0.0
    TN_float_half=TN_float/2

    #The result of testing with the simulated y_test will be devided bt 2 only on special cases 
    # and 0 for all others
    crosentr_result_sim_only_half_TN=crosentr_result_sim*TN_float_half

    # Have a tensor with (0.5 for special cases and 1 for all others)
    TN_float_inverse=((TN_float*(-1))+2)/2

    # devide the result of special cases by 2 only on special cases on the real y_test crosentr_result
    # maintain all others
    crosentr_result_real_half_TN=crosentr_result_real*TN_float_inverse

    #sum the two above: on simulated y_test we have all 0's except special cases wue have half of the value
    #                   on real y_test we have all the real values of loss except special case where we have half
    final_result_loss=crosentr_result_real_half_TN+crosentr_result_sim_only_half_TN
    #sum the two and we have average(half correct answer) only for special cases 
    # in result we lower the error on special cases ... and global

    return final_result_loss

[EDIT]

I have tested this loss with some tries and in global it does not improve the winning positions in the confusion matrix (on the final validation set - after test set)...

Then I also try to change the target values (on train and test) like this :

def convert_probs(arg):
    test= arg
    for i in range(len(test)):
        if test[i][0] == 1:
            test[i] = [0.8, 0.2, 0, 0, 0] 
        elif test[i][1] == 1:
            test[i]=[0.20,0.80,0,0,0]
        elif test[i][3] == 1:
            test[i]=[0,0,0,0.8,0.2]
        elif test[i][4] == 1:
            test[i]=[0,0,0,0.2,0.8]
    return test

and, with this transformation of the target distribution, use the kl_divergence loss function in the model but again it does not improve in relation to have normal target probabilities (with 1's on the right class) and using categorical cross entropy.

So I will stick to the normal modeling of the problem and not force the model to go to the right "quadrants" of the confusion matrix.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.