I have 2 classes of data A and C. I want to create a NN with 3 output nodes, classifying data as either A, B, or C. Classifying an A or C correctly should have zero cost (very good). Classifying an A as a C or vice versa should have high cost (very bad). However, for samples that are "blurry", I'd like to classify them as B, which would have relatively low cost (better than a misclassification, but still not ideal). I am new to tensorflow, and am not comfortable with the different ways to define cost functions. Can anyone point me in the right direction on this one? Thanks in advance
You are talking about having different missclassification costs.
Classification consists of outputing a continuous score which is then thresholded to decide the discrete classification. The threshold (ex. probability>0.5 leads to C) is chosen to satisfy your apetite on sensitivty/specificity.
The training algorithm only cares about your continuous score. This means that you can make it easy for yourself by doing simple binary classification (typically with
softmax_cross_entropy_with_logits-loss) and create your vague class B after training is done by thresholding say [0,0.4) is A, [0.4,0.6] is B and (0.6,1] is C.
To achieve exactly what you're saying, penalizing certain errors more, then you want to use the weighted_cross_entropy_with_logits. This throws out the interpretation of your outputted score as a probability, and you will need to work more to get a reasonable threshold afterwards so this is not recommended as a first step as it will probably add more complexity than results.
I also recommend reading the wikipedia article on roc analysis