I'm a beginner in the field and I wrote and trained my first Convolutional Neural Network from scratch. I would like to know what types of graphics and parameters a data scientist uses to plot to explain a neural network in the best possibile and mathematical way. As far as I know I use to plot loss and accuracy graphics using matplotlib.
There are different ways to explain your findings but associating the target with the inputs provided is tough and only a few architectures like Decision Trees do it well.
If you are trying to answer the question "why is my neural network missing out on a few samples from my validation set?" then its going to be a tough but possible road ahead. There are many good resources to interpretable machine learning
If you are trying to compare your model's (CNN) performance to other architectures on the same data set, then looking at Receiver Operator Characteristics is a good point to start.
If you're having an unbalanced data set, then the F1 Score is a good thing to have look at. Being a classification problem (which is my assumption from your question), you can indicate the performance by a variety of derived quantities.
You can plot the Confusion matrix of your model to show the number of true positive, true negative, false positive, false negative.
Code for Confusion matrix
from sklearn.metrics import confusion_matrix
For more details on confusion matrix refer to confusion matrix
The confusion matrix can be plotted using seaborn heatmap.
Code to plot Confusion matrix
import seaborn as sns
import matplotlib.pyplot as plt
for more details on heatmap refer to Seaborn Heatmap