14
$\begingroup$

I have a dataset of users purchasing products from a website.

The attributes I have are user id, region(state) of the user, the categories id of product, keywords id of product, keywords id of website, and sales amount spent of the product.

The goal is to use the information of a product and website to identity who the users are, such as "male young gamer" or "stay at home mom".

I attached a sample picture as below:

enter image description here

There are all together 1940 unique categories and 13845 unique keywords for products. For the website, there are 13063 unique keywords. The whole dataset is huge as that's the daily logging data.

I am thinking of clustering, as those are unsupervised, but those id are ordered number having no numeric meaning. Then I don't know how to apply the algorithm. I am also thinking of classification. If I add a column of class based on the sales amount of product purchased. I think clustering is more preferred. I don't know what algorithm I should use for this case as the dimensions of the keywords id could be more than 10000 (each product could have many keywords, so does website). I need to use Spark for this project.

Can anyone help me out with some ideas or suggestions?

Thank you so much!

|improve this question
$\endgroup$
  • 1
    $\begingroup$ Can you provide more information? what is "categories id of product A" and is "searching keywords id of product A" of the same length for all entries? "the dimensions of the searching keywords id could be more than 10000" why? what are they? How many samples do you have? all questions can be answered if you post a few sample of your data here. Then I could probably suggest you something. $\endgroup$ – Kasra Manshaei May 21 '15 at 13:17
  • $\begingroup$ Are product A and product B two products that the user bought? The wording seems to suggest that products A and B are different for each user, since the keywords can vary. Is this so? And last comment, do you want to classify or cluster? Those are quite different techniques :) $\endgroup$ – logc May 21 '15 at 14:28
  • $\begingroup$ Thank you @kasramsh so much for your replies. I updated the description and also attached a sample data. Hope to get some suggestions from you! $\endgroup$ – sylvia May 21 '15 at 18:09
  • $\begingroup$ @logc yes, product( i said product A earlier) and website( i said product B earlier) are different from each user. Each product has a few keywords and each website has a few keywords too. Either clustering or classification is fine, as long as I can make an user profile, such as "male young gamer"; "stay at home mom". I think clustering is more preferable . Thank you!! $\endgroup$ – sylvia May 21 '15 at 18:09
  • $\begingroup$ @sylvia - I have similar problem to solve. I had posted it as a separate question. Could you give some suggestions on how you solved it? datascience.stackexchange.com/questions/12930/… My other doubt is for K means, did you group the records by customer? Meaning did each row represented a transaction or it represented aggregated purchases of that customer till date. $\endgroup$ – Neil Jul 28 '16 at 2:45
11
$\begingroup$

Right now, I only have time for a very brief answer, but I'll try to expand on it later on.

What you want to do is a clustering, since you want to discover some labels for your data. (As opposed to a classification, where you would have labels for at least some of the data and you would like to label the rest).

In order to perform a clustering on your users, you need to have them as some kind of points in an abstract space. Then you will measure distances between points, and say that points that are "near" are "similar", and label them according to their place in that space.

You need to transform your data into something that looks like a user profile, i.e.: a user ID, followed by a vector of numbers that represent the features of this user. In your case, each feature could be a "category of website" or a "category of product", and the number could be the amount of dollars spent in that feature. Or a feature could be a combination of web and product, of course.

As an example, let us imagine the user profile with just three features:

  • dollars spent in "techy" webs,
  • dollars spent on "fashion" products,
  • and dollars spent on "aggressive" video games on "family-oriented" webs (who knows).

In order to build those profiles, you need to map the "categories" and "keywords" that you have, which are too plentiful, into the features you think are relevant. Look into topic modeling or semantic similarity to do so. Once that map is built, it will state that all dollars spent on webs with keywords "gadget", "electronics", "programming", and X others, should all be aggregated into our first feature; and so on.

Do not be afraid of "imposing" the features! You will need to refine them and maybe completely change them once you have clustered the users.

Once you have user profiles, proceed to cluster them using k-means or whatever else you think is interesting. Whatever technique you use, you will be interested in getting the "representative" point for each cluster. This is usually the geometric "center" of the points in that cluster.

Plot those "representative" points, and also plot how they compare to other clusters. Using a radar chart is very useful here. Wherever there is a salient feature (something in the representative that is very marked, and is also very prominent in its comparison to other clusters) is a good candidate to help you label the cluster with some catchy phrase ("nerds", "fashionistas", "aggressive moms" ...).

Remember that a clustering problem is an open problem, so there is no "right" solution! And I think my answer is quite long already; check also about normalization of the profiles and filtering outliers.

|improve this answer
$\endgroup$
  • $\begingroup$ Thank you so so much! It's very helpful. I will start from the mapping. I really appreciate it! $\endgroup$ – sylvia May 21 '15 at 20:19
  • $\begingroup$ Happy to help. :) $\endgroup$ – logc May 21 '15 at 21:44
  • $\begingroup$ Hi @logc, I applied LDA for selecting the features. I considered each user_id as a "document" and the keywords are the "words" in the "document", then by applying LDA I got a few topics of keywords. However, i don't know why most of my topics are consist of the same keywords. Does that mean LDA is not the right method for my case or there are some mistakes? Thank you so much! $\endgroup$ – sylvia May 28 '15 at 0:33
  • $\begingroup$ @sylvia : I would suggest that you turn that question into a new question on this site. Otherwise, we might end up writing a ton of comments, and that is not the best format for Q&A. :) $\endgroup$ – logc May 28 '15 at 10:54
  • $\begingroup$ Thanks for the suggestion. Here is the link I posted if you have time to take a look datascience.stackexchange.com/questions/5941/… Thanks! $\endgroup$ – sylvia May 28 '15 at 22:10
-2
$\begingroup$

For orientation and exploration, I can recommend WeKa, which is a very nice toolkit for machine learning. It does take a certain input format (.ARFF) so you might need to look into that as well.

As for the keyword dilemma, I recommend performing some feature selection in order to eliminate redundant or non-indicative keywords.

|improve this answer
$\endgroup$
  • $\begingroup$ Thank you @Lennart Kloppenburg for your reply. How to perform feature selection if the attribute (keword_id) are ordered number? I updated a sample data above. Could you please take a look and give me some suggestions? Thank you! $\endgroup$ – sylvia May 21 '15 at 18:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.