Deep Learning: Feed Forward for Unbalanced Classes Using Tensor Flow

In theory, Deep Learning NN can predict a class with very few observations. My problem, I have a class that happens less than 4% of the time. Feeding the network data with distribution intact (96 one class and 4% the other class) results in the network predicting mostly the most common class. Using under sampling result in a model that is able to predict 90% of the less common class, but still have a high False Positive. I know that using F Score (with beta >= 2) in the training phase of the model will allow me to improve significantly the model performance. My question, how or where do I set my own performance metric in Tensor Flow? Any suggestions are appreciated. By the way, I did tune all the parameters (learning rate, momentum, etc..) performance increases but still not at the level I want. My network has 20 inputs 5 hidden layers with 60 unites My batch is 100 (tried up to 500), and I tried epoch from 100 to 1,200

The code is in Mathematica.

 netDrop1 =
NetGraph[<| "i1" -> BatchNormalizationLayer[],
"l1" -> CatenateLayer[], "l12" -> DropoutLayer[0.2], "l2" -> 60,
"l21" -> DropoutLayer[0.5], "l3" -> Tanh, "l22" -> 60,
"l221" -> DropoutLayer[0.5], "l33" -> Tanh , "l23" -> 60,
"l234" -> BatchNormalizationLayer[0.5] , "l34" -> LogisticSigmoid,
"l24" -> 60, "l2435" -> DropoutLayer[0.5], "l35" -> Ramp,
"l45" -> 60, "l435" -> DropoutLayer[0.5], "l36" -> Tanh,
"l55" -> 60, "l37" -> Tanh, "l4" -> 2,
"l5" ->  SoftmaxLayer[]|>, {NetPort["Input1"] ->
"i1" -> "l1" ->
"l12" ->
"l2" -> "l21" ->
"l3" -> "l22" ->
"l221" ->
"l33" ->
"l23" ->
"l234" ->
"l34" ->
"l24" ->
"l2435" ->
"l35" ->
"l45" ->
"l435" ->
"l36" ->
"l55" ->  "l37" -> "l4" -> "l5", {{NetPort[
"MaritalStatus"], NetPort["Gender"], NetPort["Input"]} ->
"l1"}}, "Input" -> 11, "Input1" -> 2,
"MaritalStatus" -> NetEncoder[{"Class", marital, "UnitVector"}],
"Gender" -> NetEncoder[{"Class", {"M", "F"}, "UnitVector"}],
"Output" -> NetDecoder[{"Class", {"no", "yes"}}]]


From reading publication about NN, I have decided to create a Dropout with = 0.2 at the input level, and at all subsequent levels to have Dropout at 0.5. I have used both Tangent and ReLu (but found little differences in performance between them) As you notice, a simple FeedForward Network I do under sampling, the under sampling with great success: the model is able to recall most of the yes, but still predict many negative samples as positive.

  <|{"yes", "no"} -> 972, {"no", "no"} -> 1082, {"yes", "yes"} ->
81, {"no", "yes"} -> 3|>


By the way, I have used these same data,and used RandomForest with a utility function with great results http://femvestor.blogspot.com/search?q=geico

• By the way, Used Random Forest with a utility function, and C5.0 with a cost function with great success. – user34018 Jan 28 '17 at 1:24
• Before focusing on performance, I would suggest you should focus on balancing your dataset. You have a very unbalanced distribution, changing the metrics or tuning hyperparameters wouldn't help until unless you pre-process your data – Nain Jan 28 '17 at 8:12
• I did balance the dataset. Still, I get very good recall for the rarest class, and very good precision for the most common class. – user34018 Jan 28 '17 at 21:02
• Paste your code – Nain Jan 29 '17 at 8:54
• what is your loss function? are you using cross entropy? – Escachator Dec 29 '17 at 9:54