I have a data-set that contains thousands of employee data, including their role, department (Applications Developer, IT Support, Network Management etc.), and using one-hot encoding all of the hardware or software they have been given.

I want to use this data so that when I input a new user with the data - role, department etc. their account will be compared to the model accounts and a prediction of what software or hardware they will need will be output.

What machine learning algorithm should I use for this? I was thinking an unsupervised approach would be appropriate but I am new to machine learning and data science so I could be going about this completely the wrong way.

Thanks for your help :)


1 Answer 1


Unsupervised Approach

Unsupervised learning can be a good starting point. You can do a clustering (k-means/hierarchical) of existing user-base to find patterns in it (example: cluster 1 contains employees with Windows laptop with 4 GB RAM and photoshop pre-installed , cluster 2 is Mac with 16 GB memory). Then for a new employee, predict the nearest matching cluster and use the pre-dominant setup of that cluster.

The problem is there may be lot of lot of variation in the same cluster, especially on softwares installed.

Supervised Approach

Other alternative is to convert it to a supervised learning problem with hand-rolled categories as outcome (example: 'laptops with lot of RAM and graphics software', 'laptops for development') and then predict that category. You will need upfront work and domain knowledge to create the categories. But it will reduce the predicted variable space by a large margin.

But How to Create Categories?

Maths can help us a little bit here. Let us assume that you have an N dimensional one-hot encoded vector representing the hardware and software setup. Let us make the assumption that there are a handful of latent K categories of the setup encoded by a large dimensional vector of size N. To go about finding those, you can take a low dimensional projection of the encoding vector. This is probably a mathematical way of finding the categories instead of what you'd do manually on the output of clustering/unsupervised approach.

Radical Alternative: Recommendations If you have enough data, you can pose it as 'recommending right setup to a new employee' problem. Take any standard movie recommendation kind of setting with users and movie ratings. Replace users -> employees and movies with a unique hardware + software setup. Use standard recommendation algorithm applied to binary data.

All these approaches are actually variant of low dimensional projection of the encoding vector + similarity calculation with existing users.

Hope this helps.

  • $\begingroup$ This helps a lot actually thank you. What would you constitute as being "enough data" for the recommendations alternative. I was looking into this briefly but it seems like it could be a good way to go. $\endgroup$
    – Top Lit
    Aug 7, 2019 at 7:23
  • $\begingroup$ @TopLit Hard to give a general recommendation. But there are standard datasets like movielens grouplens.org/datasets/movielens/100k with ~1k user and movies. So that could be a baseline. Another thumb rule could be do you have atleast few hundred users which could be kept in validation set? Then recommendations approach could be used. $\endgroup$
    – hssay
    Aug 7, 2019 at 8:12
  • $\begingroup$ This is brilliant help thank you. $\endgroup$
    – Top Lit
    Aug 7, 2019 at 8:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.