2
$\begingroup$

I'd like to convert a set ducoments to term weight matrix(features) by tf-idf. Then calculate the similarity of two document by their features.

similarity is compute by result = matrix*matrix.T > 0.9 detail here, and group them by the result(loop result, if result[1,0] >0.9, then index 1 is similar with index 2)

Now I have a lot of resource to group.

For example, I have some books with different names(more complex in fact) I can roughly group these books name by similarity, like below:

step1:

group 1:
    1.The Three Body Problem vol 1
    2.[Chinese]The Three Body Problem no 1
    3.The Three Body Problem 2
    4.The Three Body Problem vol 3[Japanese]
    5.Problem of Three Body vol 3
    6.(xx)The Three Body Problem 2
    7.The Three Body Problem 1[English]
group 2:
    1.Another book 1
    ....

But xxx 2 and xxx vol3 is needless when I want to find xxx vol 1 , So have to do

step2: tokenize each book name again, use some patterns/rules to extract the book number to distinguish them.

Is there any way to add some term(such as Arabic numerals:0-1, English number:one- twenty) with high dissimilarity weight, to make step1 result

group 1:
    1.The Three Body Problem vol 1
    2.[Chinese]The Three Body Problem no 1
    7.The Three Body Problem 1[English]
group 2:
    3.The Three Body Problem 2
    6.(xx)The Three Body Problem 2
group 3:
    4.The Three Body Problem vol 3[Japanese]
    5.Problem of Three Body vol 3
group 4:
    another book 1
    [xx]another book vol.1

update

If there are some titles like below:

1. There are 2 man vol.1
2. (xx)There man 2 boy 2

I need add a lot detection(number position or something else), that's why I want a way to add a extra weight to somewhere(to avoid step 2, the redundant tokennizing and custom extraction rule).

I think the similarity weight may work like:

tfidf weight two title, plus each number weight, then calculate the similarity matrix.
But now I am using tfidf matrix power to get similarity matrix,
I don't know how to add a extra weight to tfidf weight result, the weight meaning is different between tfidf weight and what mentioned extra weight.

I want to know where and how to add the proper extra weight, how calculate it value?

$\endgroup$
1
$\begingroup$

When you have a similarity measure fixed already, you can use correlation clustering to turn your similarity matrix into groups as you wanted. Here's a good resource about correlation clustering http://www.francescobonchi.com/CCtuto_kdd14.pdf

For the second step to attempt to make sure volume 1 and volume 2 are not the same, I would add a number of rules that try to find the difference. Look for numerals, if they don't match they have 0 similarity. Play around a bit with this. After you have your new similarity matrix you can use correlation clustering which should seperate volumes 1 and 2 into different groups.

$\endgroup$
4
  • $\begingroup$ I have already group similar book together.But what I need is add some function to make it more correctly. xxxxx 1 and xxxxx vol.one can be accept as similar , but xxxxx 1 with xxxxx 2 or xxxxx vol.II not.That's the problem. $\endgroup$
    – Mithril
    Mar 13 '16 at 2:49
  • $\begingroup$ As I said in the question, My step now is 1. group similar book(what you mention correlation clustering, this can not solve step 2) 2. tokenize book name again, extract the book number , regroup them. I need a better step 2, or merge step 1,2 together. $\endgroup$
    – Mithril
    Mar 13 '16 at 2:53
  • $\begingroup$ I added a portion to describe how I would tacke it, which is basically merging the two steps $\endgroup$ Mar 17 '16 at 7:28
  • $\begingroup$ That is what I try to avoid.You must extract the number from book title(as I do now), rather than add a weight to number at step 1. Extract number have a lot work to do , such as when title like there are 2 man vol.1.I wouldn't fair such problem when using a similarity weight. $\endgroup$
    – Mithril
    Mar 17 '16 at 7:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.