Finite difference is a way to calculate the derivative of a function according to its value. It is based on Taylor's theorem.
You can have a good idea here and may be have look in this tensorflow API.
Let's suppose J is your cost function. The first question is to defined the definition domain of your J in which apply finite difference discretization, I should have taken your different layers (but I have never tried this).
The second question is linked to the fact that your error is "backpropagated" so when constructing your definition domain you should first think of the direction of your domain (from First layer to last or from last to first).
The you have to customize your gradient to use it in your TensorFlow implementation. So I suggest you to see here and to see in StackOverflow if other people has already ask for what you want and if you can find more elaborate answers.
I hope it will help you