Suppose I have data in the form of Query/Document Pairs, along with corresponding relevance scores (or class labels). Is there a way to use topic modeling to devise a model so that later given a query and a document, we can predict its relevance score?
Certainly. A part of topic modeling output is typically some vector for each document describing it as a distribution over the topics in your model. You can use these as continuous-valued predictors for a classifier trained with the labels you have. I've done this before, but, depending on the domain characteristics of your problem, it may not be the best approach, but it is certainly a testable one!
Possible, yes, but why do you want to do this?
Topic modelling is a method of unsupervised learning with all it weaknesses.
With the type of data you sketch you can used methods of supervised learning that will be much more stable, less prone to overfitting, and more reliable in the long run.