1
$\begingroup$

I want to assign a certain category to a group of keywords. So i.e. people can upload images or videos, when they do this they can set keywords for this. These keywords are free to type so words can be spelled in different ways. The amount of keywords are 95% between 0 and 20 words.

I want to create categories from these. So that I can assign a combination of keywords to a category.

The categories and the amount of categories are undefined.

From what I've researched is that this is probably a Topic modeling or clustering issue. Although with topic modeling most examples I see are based on long texts instead of a couple of keywords.

What would be a good approach to handle this?

I thought about first some simple fuzzywuzzy to find different spellings of the same words.

Create a big word list from this. Then each keyword will be matched against the list and rewritten if it matches one and added if there isn't a good match.

Then I would need to create groups and here I don't know which algorithms I should use.

I was thinking maybe do k-means clustering and then see on which k I get the best results and then assign a category to it manually by looking at which keywords are in it.

So it would be nice to just let the algo figure the amount and categories out, but it can be relaxed that I will set them before.

Does anyone have a better suggestion or are there just complete algorithms for this already available?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

Does anyone have a better suggestion or are there just complete algorithms for this already available?

Apparently you want to do information retrieval (IR) but without the information part: normally an important part of the IR process is the set of documents that the user is searching for (for example text, images video). This is important because knowing the documents which correspond to a query gives semantic information about the keywords.

Now, if you only have the keywords available then there's no semantic to help with the clustering. So you are right that the only thing you can use is the spelling. i assume that by " simple fuzzywuzzy" you mean "fuzzy matching", i.e. using string similarity measures. I can think of two options:

  • You can compare each keyword with each other with some string similarity measure such as Levenshtein edit distance, Jaro, or characters n-gram based Jaccard, cosine etc.
  • You can represent each keyword as a vector based on the char n-grams in the word, then you can apply clustering with k-means for example.

Topic modeling techniques wouldn't work because you don't have large text documents.

Improve this answer
$\endgroup$
6
  • $\begingroup$ Thanks, however doesn't this just solve the first part of the issue, that keywords can be spelled differently? Not the second part on matching a list of keywords to a category? $\endgroup$
    – user3605780
    Commented Jun 14, 2019 at 13:18
  • $\begingroup$ Do you have the documents corresponding to the keywords queries? If yes there might be a way, if not I don't see any way to assign a semantic category to a group of keywords. $\endgroup$
    – Erwan
    Commented Jun 14, 2019 at 13:20
  • $\begingroup$ If you mean with documents the actual image/video which correspond to the keywords then no. Basically the keywords are the semantic categories assigned to the document. However the amount of categories is too large so I want to reduce this. So let's say there are 200 distinct keywords/categories which can be assigned. And by analysing i.e. how often they are used together create let's say 10 super-categories. $\endgroup$
    – user3605780
    Commented Jun 14, 2019 at 13:48
  • $\begingroup$ The problem is that the categories should be about the documents, not the keywords which describe the documents. Do you have instances in which keywords are associated with documents ids for instance? Otherwise it's going to be difficult to assign a semantically meaningful category, the best you could do would be to group the keywords into category1, category2... category10. $\endgroup$
    – Erwan
    Commented Jun 14, 2019 at 13:53
  • $\begingroup$ Yes I can get some unique identifier for the documents but I don't see how this would help. However just creating 10 buckets where each keyword belongs to one already helps. Then after we can just choose the category which has the most matches. $\endgroup$
    – user3605780
    Commented Jun 14, 2019 at 14:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.