6
$\begingroup$

I'm just starting to investigate machine learning concepts, so I'm sorry if this question is very naive, but I'm hoping that it will be an easy one to answer!

I have a document matching algorithm that individually calculates a match for each field (0-1, with 0 = no match, 1 = 100% match), and applies a separate weight to each field match to be used in calculating an overall weighted average relevance "score".

E.g., given a document of 3 fields (d1-3) and an input query against each of the fields (q1-3), field matches are calculated for each pair (m1-3) and then weights (w1-3) are applied using a weighted average for a final relevance score: s = sum(mi x wi)/sum(wi).

For this contrived example, perhaps we can simply say that a document is considered relevant if the score is above 0.5. I.e., there is either a "relevant" (0.5-1.0) or "not relevant" (0-0.5) outcome. But I don't want every field to have equal weight in determining the outcome.

So, my question is simply: What type of machine learning technique is "best" used to calculate the appropriate weights (w1-n), based on past, known results? Is this even an appropriate use of machine learning?

And secondly, if instead of a simple outcome of relevant and non-relevant, I actually want to rank the documents by relevancy, can this also be achieved using a machine learning technique?

$\endgroup$
  • $\begingroup$ Some questions: 1) Are your documents and queries all the same length, so you have a vector of field "score" the same length for many examples? 2) What data do you have in "past, known results" - do you have a clear relevant/not-relevant label for multiple documents and queries? If the answer is yes to both questions, then this looks a lot like a logistic regression problem (although that would involve changing your averaging routine, it isn't necessary of you don't want). There are other possibilities depending on your answers though. $\endgroup$ – Neil Slater Jun 7 '16 at 7:51
  • $\begingroup$ I'm not entirely sure what you're asking in question 1... My "documents" are a collection of fields, with each having one of several data types (free text, dates, categorical, etc. The documents can easily be represented as a row in a spreadsheet/CSV. The queries against these fields depend on the data type (search words/phrases against text fields, date/location "proximity", categorical set union, etc). Each of these queries results in a score, from 0-1, which indicate the strength of the match for that field. $\endgroup$ – SilentMiles Jun 7 '16 at 23:20
  • $\begingroup$ For your second question, yes, I have a collection of past documents and an "outcome" for each. The outcome can be one of several states, but I could reduce this down to just two (i.e. "relevant" and "not relevant"). The outcome is for the whole document, not the individual queries. They are a new way to predict results for a document, so that's why I'm searching for a way to train the appropriate weights that will result in the known outcomes. $\endgroup$ – SilentMiles Jun 7 '16 at 23:24
  • $\begingroup$ And again, a binary or categorical outcome would be useful, but my ideal is for the outcome to be a relevance "score", e.g. a value from 0-1 indicating how relevant the document is. $\endgroup$ – SilentMiles Jun 8 '16 at 2:13
  • $\begingroup$ For question 1, the important part of my question is whether the number of fields is fixed and same for all documents and queries (I understand each field can have a different data type and is converted to numeric type by your scoring system that combines document with query). Your comment implies that is the case, but you have not directly said so. $\endgroup$ – Neil Slater Jun 8 '16 at 6:41
6
$\begingroup$

Yes it is definitely possible to calculate optimised weightings provided you have some training examples where you know the document fields, the query, and either the outcome (relevant/not-relevant) or the desired score.

I think your training feature set should be the query score in range [0.0,1.0] for each field of each example. The training label should be either relevance 0 or 1 for each example, or the relevance score that the example has.

If you have a target score for each example

You want to determine the weights $W_i$ to use for each field $i$. Your calculated relevance score would be $\hat{y} = \sum_{i=1}^{N_{fields}} W_i * X_i$ where the caret signifies this is the estimate from your function and $N_{fields}$ is the number of field. Note I am ignoring your original idea of dividing by the sum of all $W_i$, because it makes things more complex. You can either add that term or force the sum to be equal to 1.0 if you wish (I am not going to show you how though, as this answer would get too long, and it probably won't help you much)

With a target score and training data, the simplest approach is to find the weights which cause the lowest error when used with the training data. This is a very common goal in supervised learning. You need a loss function. Having a target scalar value means you can use difference from target and a very common loss function for this kind of regression problem is the mean squared error:

$$E = \frac{1}{N_{examples}} \sum_{j=1}^{N_{examples}} (\hat{y}_j - y_j)^2$$

Where $\hat{y}_j$ is your calculated relevance score for example $j$ and $y_j$ is your training label for the same example.

There are a few different ways to solve for lowest $E$ with this loss function, and it is one of the simplest to solve. If you express your weights as a vector $W$ length $N_{fields}$ your example features as a matrix $X$ size $N_{examples} \times N_{fields}$ and the labels as a vector $Y$ length $N_{examples}$ then you can get an exact solution to minimise loss using the linear least squares equation

$$W = (X^TX)^{-1}X^TY$$

There are other approaches that work too - gradient descent or other function optimisers. You can look these up and see which you would prefer to use for your problem. Most programming languages will have a library with this already implemented.

Note that you will likely get scores greater than 1.0 or less than 0.0 from some document/query pairs.

You will have to use a adjust the technique if you want to divide by total of all weights or want sum of all weights equal to 1 in your scoring system.

If you have a relevance 0 or 1 for each example

You have a classification problem, relevant or not are your two classes. This can still be made to work, but you will want to change how you calculate your weighted score and use logistic regression.

Your weighted score under logistic regression would be:

$$\hat{y} = \frac{1}{1 + e^{-(b + \sum_{i=1}^{N_{fields}} W_i * X_i)}}$$

Where $b$ is a bias term. This looks complicated, but really it is just the same as before but mapped by a sigmoid function to better represent class probabilities - the result is always between 0 and 1.

You can look up solvers for logistic regression, and most stats or ML libraries will have functions ready to use.

Caveats

You have made a starting assumption that a simple combined relevance score will lead to a useful end result for your users performing search. This has led to simple linear models looking like a good solution. However, this may not be the case in practice, and you may need to re-visit that assumption.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.