How should I interpret this? If a lower loss means more accurate predictions of value, naively I would have expected the agent to take more high-reward actions.
A lower loss means more accurate predictions of value for the current policy (technically it is more complicated for Q-learning off-policy estimates, but the covergence will still be limited by experience reachable in the current policy). Unfortunately a loss metric in RL cannot capture how good that policy is.
So what it means is that your policy has settled into a pattern where values can be estimated well by the neural network that you are using for Q. For some reason it is not finding improvements to that policy - typically it should be doing that before the loss metric drops, as each improvement in value estimates should reveal better possible actions, and once those start being taken by a new policy, the value estimates become out of date, and the loss increases again.
Could this be a sign of the agent not having explored enough, of being stuck in a local minimum?
Exploration could be an issue. The "local minimum" in that case is probably not an issue with the neural network, but that small variations in policy are all worse than the current policy. As you are learning off-policy, then increasing the exploration rate may help find the better states, at the expense of slower overall learning. Also, methods that explore more widely than randomly on each action could be better - e.g. action selection methods that consistently pick unexplored state/action pairs such as Upper Confidence Bound.
Also a possibility is that the structure of your network generalises well under the current policy, but is not able to cover better policies. In that case, whenever the exploration suggests a better policy, the network will also increase estimates of unrelated action choices - so it would try them, notice they are better, then back off as the new values also cause unwanted policy changes in other situations.
If you know a better policy than the one that is being found, then you could plot a learning curve with the policy fixed, see if the network can learn it. However, usually you will not know this, so you may be stuck with trying some variations of neural network architecture or other hyperparameters.
There are other methods than DQN (e.g. A3C, DDPG), as well as many add-ons and adjustments to DQN that you could try (e.g. eligibility traces, double learning).
