I'd like to convert a set ducoments to term weight matrix(features) by tf-idf. Then calculate the similarity of two document by their features.
similarity is compute by
result = matrix*matrix.T > 0.9 detail here, and group them by the result(loop result, if
result[1,0] >0.9, then index 1 is similar with index 2)
Now I have a lot of resource to group.
For example, I have some books with different names(more complex in fact) I can roughly group these books name by similarity, like below:
group 1: 1.The Three Body Problem vol 1 2.[Chinese]The Three Body Problem no 1 3.The Three Body Problem 2 4.The Three Body Problem vol 3[Japanese] 5.Problem of Three Body vol 3 6.(xx)The Three Body Problem 2 7.The Three Body Problem 1[English] group 2: 1.Another book 1 ....
xxx 2 and
xxx vol3 is needless when I want to find
xxx vol 1 ,
So have to do
step2: tokenize each book name again, use some patterns/rules to extract the book number to distinguish them.
Is there any way to add some term(such as Arabic numerals:0-1, English number:one- twenty) with high dissimilarity weight, to make step1 result
group 1: 1.The Three Body Problem vol 1 2.[Chinese]The Three Body Problem no 1 7.The Three Body Problem 1[English] group 2: 3.The Three Body Problem 2 6.(xx)The Three Body Problem 2 group 3: 4.The Three Body Problem vol 3[Japanese] 5.Problem of Three Body vol 3 group 4: another book 1 [xx]another book vol.1
If there are some titles like below:
1. There are 2 man vol.1 2. (xx)There man 2 boy 2
I need add a lot detection(number position or something else), that's why I want a way to add a extra weight to somewhere(to avoid step 2, the redundant tokennizing and custom extraction rule).
I think the similarity weight may work like:
tfidf weight two title, plus each number weight, then calculate the similarity matrix.
But now I am using tfidf matrix power to get similarity matrix,
I don't know how to add a extra weight to tfidf weight result, the weight meaning is different between tfidf weight and what mentioned extra weight.
I want to know where and how to add the proper extra weight, how calculate it value?