3
$\begingroup$

I need to rank human skills depending on their similarity to the input skill. So if I enter "Dutch language", I want the list like this:

0.97 Dutch
0.86 Dutch lessons
0.55 Frisian
0.50 Flemish
0.27 German language

I have a database of around 4500 human skills (ranging from "programming in C" to "baking almond cake") with 600 manually categorized. I can already find corresponding article on BabelNet and pull domain, categories and related terms.

Example skill with the data from BabelNet:

name:"photography"
categories:
  0:"Photography"
  1:"French_inventions"
  2:"Optics"
  3:"1822_introductions"
manualCategory:"art & music"
domains:
   ART_ARCHITECTURE_AND_ARCHAEOLOGY:1
compounds:
  0:"digital_photography"
  1:"landscape_photography"
  2:"photographic_developing"
  3:"motion_photography"
  4:"nature_photography"
  ...
  48:"photographic_plates"
otherForms:
  0:"still_photography"
  1:"photo"
  2:"photos"
  3:"photographed"
  4:"photographers"
  ...
  20:"Photographer"

Can you suggest me the approach or at least steer in the right direction?

Improve this question
$\endgroup$
6
  • $\begingroup$ Great question. Do you have a corpus of text tagged with these skills, like how questions here are, or the option of crowd sourcing one? $\endgroup$
    – Emre
    Commented Apr 20, 2016 at 17:26
  • $\begingroup$ No, I don't have a text associated with the skills, these are the skills entered by users in their profiles. Though with the input skill comes a motivation for learning it with the length of at least 140 symbols. Not sure what do you mean by crowd sourcing it. $\endgroup$
    – Slaventsiy
    Commented Apr 21, 2016 at 8:35
  • $\begingroup$ I mean let users enter text, such as a bio, that can be associated with the tags like how questions here are associated with tags. Now what do you mean by "with the input skill comes a motivation for learning it with the length of at least 140 symbols"? $\endgroup$
    – Emre
    Commented Apr 21, 2016 at 15:38
  • $\begingroup$ On the platform users are looking for a teacher for a skill they would like to learn. They enter the skill they want to learn and the motivation, why they want to learn this skill. Then no, it is not an option, at least for now. $\endgroup$
    – Slaventsiy
    Commented Apr 22, 2016 at 10:30
  • $\begingroup$ Then you have the search log, and ratings of classes/teachers (which?), right? $\endgroup$
    – Emre
    Commented Apr 22, 2016 at 15:52

1 Answer 1

Reset to default
2
$\begingroup$

Pretty late but I'm surprised this wasn't answered more. "Cosine similarity" is a great technique to try, though simply letting users search with a hard string and then ranking by popularity isn't so bad (e.g. "dutch" brings up everything with "dutch" in it, though I would discard mid-word matches, so "ball" wouldn't return "football", but would return "ball room dancing").

I'd say in any approach a main issue will be deduplicating previous (non-standardized) skills input by users that weren't quite standardized. You could also try replacing the candidate skills with versions that have different synonyms substituted at search time, e.g. "soccer coaching" might be stored also as "football coaching" if most of your content is from Europeans.

Sometimes extreme accuracy might not be the best goal, though... You may want to encourage users to explore new skills that they never knew existed! Not sure what your needs are...

Whatever you settle on, it might be worth building a semi-hand-crafted test set of queries and relevant results so that you can see if the performance is terrible (Google precision and recall in the context of search results).

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.