Yes there is a cost (score/utility) function and your intuition is correct. In vanilla PG we optimize the expected return $J$ of a trajectory $\tau$ under policy $\pi$ parametrized by $\theta$:
$$\nabla_{\theta} J\left(\tau\right)
\approx \frac{1}{N} \sum_{i=1}^{N} \nabla_{\theta} \log \pi_{\theta}\left(\tau_{i}\right) r\left(\tau_{i}\right)$$
(You can find lots of information here: https://spinningup.openai.com/en/latest/algorithms/vpg.html)
The vanilla PG is very closely related to maximum likelihood:
$$\nabla_{\theta} J_{\mathrm{ML}}(\theta) \approx \frac{1}{N} \sum_{i=1}^{N} \nabla_{\theta} \log \pi_{\theta}\left(\tau_{i}\right)$$
The two gradients are almost the same except the reward multiplication. This reward is the learning signal. Thinking in terms of backpropagation your gradients are being multiplied by the reward signal. You can think of the vanilla PG as a cost-sensitive classification - with the broad sense.
I will give you a very simple example with Neural Networks in order to demonstrate you the result of the PG learning mechanism. Assume a very simple architecture:
( input --> CNN --> CNN --> fully connected (fc) layer --> out1: V(input), out2: $\pi(a|input)$ )
which can be trained asynchronously or synchronously. The output has two heads one for the expected reward and one for the policy.
Let's assume a task in which an agent has to learn to select between two rewarding targets (orange r=10, green r=1). The targets are presented either both of them or one at a time randomly upon completion of an episode.
Assume now that the agent is fully trained. Taking the fc representations from various episodes and running tsne (clustering) we get the following picture:

This 2D representation tells us that the network has a "clear understanding" when there is only the green target, only the orange target and both of them together. We could colored differently the cluster to get a sense of the expected reward or the action preference (as the fc encodes spatial information, reward information and action preference).
Policy Gradients are mapping states to action distributions (and/or reward predictions). This means that the learned function (the Neural Network) should have the appropriate representations to do that mapping. And this is essentially what is learned by the network: a decision boundary (given input state). Please note that this is a very simplistic example to get an insight of what the network has learned so you can easily draw parallels to the classification case.