4
$\begingroup$

I want to build a recommender system that suggests the next step in your career.

About the dataset. About 50'000 Users with following informations:

  • Skills (tags, string value)
  • every job they did (embedded job titles)
  • every school they went

Now the recommender system should tell you what the next step in your career can be, and if you need a another skill or degree for that step.

How should you build such a system? (Supervised, Unsupervised (Recommender System,...), ...)

$\endgroup$
  • $\begingroup$ Hi.Please can you confirm how you solved this? I am interested in the solution. Thanks. $\endgroup$ – Z Z Dec 20 '18 at 9:15
1
$\begingroup$

I have tried something similar once. I used an approach which you would not expect, but which gave some surprisingly good results.

I used the NMT (Neural Machine Translation) model in Tensorflow. There are some examples online for Vietnamese->English translation. I changed it to translate from "old job title" to "new job title", and trained it on my dataset. This is easy to do and you don't need to modify any code, except maybe tweak the model a little (e.g. length of document is now shorter than in the machine translation example).

The drawback of this approach is that only the most recent job title is taken into account when using it for inference. However you could try concatenating all previous job titles and other data (schools etc) as the input text, to produce a single job title as output, and this way you could make sure you are making use of their entire career path.

I would suggest tweaking aspects of the architecture (number of layers, number of previous jobs used, off-the-shelf word2vec vs domain specific trained word2vec etc) until you get the best performance against whatever evaluation metric you are using.

The other approach you could take would be to take the doc2vec of each title, and train some kind of RNN/LSTM to predict the next vec given all previous vecs. Then you need a postprocessing stage on the output to convert the output vec to a text. Unfortunately I don't know of a shortcut here to avoid building your model architecture from scratch, so this approach is more work.

Of course there are other ways of solving the problem but since no-one else has answered yet I thought I'd give my suggestion.

| improve this answer | |
$\endgroup$
0
$\begingroup$

It is a good idea to first shed some light on how the system's performance/accuracy will be gauged.

You said that the output of your recommender system is "what the next step in your career can be", but if the system tells me I should be a baker, how will you measure how good of a recommendation that is? Will it be based on how well it predicted my next career move? Will the system allow me to give it feedback on its recommendation, and gauge itself accordingly?

Once that is clarified, then you're in the game. You have some sort of goodness of fit function g and you are now looking for a function f such that g(f(x), y) is maximized (or minimized, depending on your definition), where x is the data and y is the recommendation.

Then two questions remain: How do you encode the explanatory data x and target variable y? Again, it depends a lot on what the goal is, but given how unnormalized the data you have seems to be, I would indeed convert everything into a text mining problem: You'll need to map your data to a set of tokens (english words, or any other symbols), filter out patterns that you know carry little information (punctuation, numbers, etc.), and transform the bag of words to a vector (for instance, with word2vec, mentioned by @Tom). You might want to also bucket y into categories, or on the contrary, expand it to a word vector as well, so that similar jobs can be compared.

Getting a model that predicts/estimates y based on x is then the easy part. You'll find plenty of off the shelf learners to try out (e.g. in sklearn (python) or tensorflow).

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.