In theory, Deep Learning NN can predict a class with very few observations. My problem, I have a class that happens less than 4% of the time. Feeding the network data with distribution intact (96 one class and 4% the other class) results in the network predicting mostly the most common class. Using under sampling result in a model that is able to predict 90% of the less common class, but still have a high False Positive. I know that using F Score (with beta >= 2) in the training phase of the model will allow me to improve significantly the model performance. My question, how or where do I set my own performance metric in Tensor Flow? Any suggestions are appreciated. By the way, I did tune all the parameters (learning rate, momentum, etc..) performance increases but still not at the level I want. My network has 20 inputs 5 hidden layers with 60 unites My batch is 100 (tried up to 500), and I tried epoch from 100 to 1,200

The code is in Mathematica.

 netDrop1 = 
     NetGraph[<| "i1" -> BatchNormalizationLayer[], 
    "l1" -> CatenateLayer[], "l12" -> DropoutLayer[0.2], "l2" -> 60, 
    "l21" -> DropoutLayer[0.5], "l3" -> Tanh, "l22" -> 60, 
    "l221" -> DropoutLayer[0.5], "l33" -> Tanh , "l23" -> 60, 
    "l234" -> BatchNormalizationLayer[0.5] , "l34" -> LogisticSigmoid, 
    "l24" -> 60, "l2435" -> DropoutLayer[0.5], "l35" -> Ramp, 
    "l45" -> 60, "l435" -> DropoutLayer[0.5], "l36" -> Tanh, 
    "l55" -> 60, "l37" -> Tanh, "l4" -> 2, 
    "l5" ->  SoftmaxLayer[]|>, {NetPort["Input1"] -> 
     "i1" -> "l1" -> 
      "l12" -> 
       "l2" -> "l21" -> 
         "l3" -> "l22" -> 
           "l221" -> 
            "l33" -> 
             "l23" -> 
              "l234" -> 
               "l34" -> 
                "l24" -> 
                 "l2435" -> 
                  "l35" -> 
                   "l45" -> 
                    "l435" -> 
                    "l36" -> 
                    "l55" ->  "l37" -> "l4" -> "l5", {{NetPort[
        "MaritalStatus"], NetPort["Gender"], NetPort["Input"]} -> 
       "l1"}}, "Input" -> 11, "Input1" -> 2, 
       "MaritalStatus" -> NetEncoder[{"Class", marital, "UnitVector"}],
       "Gender" -> NetEncoder[{"Class", {"M", "F"}, "UnitVector"}],
       "Output" -> NetDecoder[{"Class", {"no", "yes"}}]]

From reading publication about NN, I have decided to create a Dropout with = 0.2 at the input level, and at all subsequent levels to have Dropout at 0.5. I have used both Tangent and ReLu (but found little differences in performance between them) As you notice, a simple FeedForward Network enter image description here I do under sampling, the under sampling with great success: the model is able to recall most of the yes, but still predict many negative samples as positive.

  <|{"yes", "no"} -> 972, {"no", "no"} -> 1082, {"yes", "yes"} -> 
  81, {"no", "yes"} -> 3|>

By the way, I have used these same data,and used RandomForest with a utility function with great results http://femvestor.blogspot.com/search?q=geico

  • $\begingroup$ By the way, Used Random Forest with a utility function, and C5.0 with a cost function with great success. $\endgroup$ – user34018 Jan 28 '17 at 1:24
  • $\begingroup$ Before focusing on performance, I would suggest you should focus on balancing your dataset. You have a very unbalanced distribution, changing the metrics or tuning hyperparameters wouldn't help until unless you pre-process your data $\endgroup$ – Nain Jan 28 '17 at 8:12
  • $\begingroup$ I did balance the dataset. Still, I get very good recall for the rarest class, and very good precision for the most common class. $\endgroup$ – user34018 Jan 28 '17 at 21:02
  • $\begingroup$ Paste your code $\endgroup$ – Nain Jan 29 '17 at 8:54
  • 1
    $\begingroup$ what is your loss function? are you using cross entropy? $\endgroup$ – Escachator Dec 29 '17 at 9:54

To Improve the performance of the network, I added a Batch Normalization layer, and other variables that I had to remove when using Random Forest. One of these variables is a categorical variable with 51 levels. I made sure to add an embedding layer that will represent the values by a vector of size 10. As the original dataset had 15 numerical variables and 9 categorical variables, I have decided to also perform dimension reduction using auto-encoding.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.