To understand what is a cost function, you have to understand supervised machine learning.
In this setting, we are given n training samples $x_1,\ldots,x_n$, with $x_i \in \mathbb{R}^{f}$, for all training samples so that $f$ is the dimension of the feature space.
For each training sample $x_i$, we are given a label $y_i \in \mathbb{R}^{g}$.
Now the task of supervised machine learning is to find a function $\phi$,
such that $\phi(x_i) = y_i$, for all training samples $x_i$.
To this end, a parametrized function is choosen (e.g. SVM, Neural Network, Random Forest,..), so that we consider a function $\phi_{w}$, where $w$ represents some paramters of the function.
Then, we seek for the parameters $w$, such that $\phi_w$ maps the input samples to the labels as best as possible.
That means we are trying to calculate:
$\min_{w} \sum_{i=1}^{n} ||\phi_w(x_i)-y_{i}||,$
where $$|| \cdot ||$$ is a function measuring the difference between model prediction and desired outcome. It is also called cost function or loss function (I guess its called the loss as it measures the loss in accuracy when using the model's prediction instead of the true outcome).
In principle, any plausible loss function could be used, which delivers a function $\phi_w$ that has minimal loss according the chosen loss function (if the minimzation problem can be solved optimally).
Now depending on knowledge on your input data and the labels, you should select the loss function which is most approriate for the machine learning task.
A few examples,
If $y \in \mathbb{R}$, using the MSE loss can be considered as a least quares problem:
$\min_{w} \sum_{i=1}^{n} (\phi_w(x_i)-y_{i})^2,$ as we assume to have an overdetermined system of equations ($n > f$). For least squares problems, the theory is very well-developed, e.g. it is a maximum-likelihood estimator in case of gaussian noise.
If your noise is not guassian distributed, you should use robust regression, e.g. using the Huber loss, which gives less weights to outliers.
If you want to regress a function in $[0,1]$, so $y_i \in [0,1]$ for all $i$, applying a least square fit does not make sense as it would result in a function that might have values in $\mathbb{R} \setminus [0,1]$, which might not (when certain assumptions are satisfied) make any sense if you want to regress probabilities for example. Therefore, a different loss function should be used.
For the Logistic regression, the assumption is that y is Bernoulli distributed. In fact, the linear regression is fitted to the logit [$\mathrm{logit}(p) := log(\frac{p}{1-p})$], so we could compute $\min_{w} \sum_{i=1}^{n} (\mathrm{logit}(y_{i}) - \phi_{w}(x_i))^2$. This can be used to show that again, the maximum likelihood (under the assumption that all $y_i$ are independently Bernoulli distributed) is achieved, if the cross-entropy loss $\sum_{i=1}^{n} [−𝑦_{i}log(\phi_{w}(x_i))−(1−𝑦_{i})log(1−\phi_{w}(x_i))]$ is minimized.
The hinge loss $\sum_{i=1}^{n}\max(0,1-y_{i}\phi_{w}(x_{i}))$ is used to train a SVM in the case of a binary classification. Here the setting is $y_{i} \in \{-1,1\}$. For the SVM it can be shown that minimizing the loss is equivalent to maximizing the margin between the seperating hyperplane and the two classes.
