1
$\begingroup$

Hi guys I'm very new to data science, I have intermediate background on programming and have used Pentaho Data Integration tool once for DB migration & data cleansing.

Let's say I have this kind of data:

item_details, timestamp
Wooden chairs, 01-07-2017
Plastic chairs, 02-07-2017
Stainless table, 11-07-2017
Decorated window, 12-07-2017

and so on

I want to know based on monthly time frame what are the top trending items in that month. Let's say in January the top 3 item is: 1. Table 2. Chairs 3. Window In February : 1. Door 2. Chair 3. Cupboard .. And so on

How can I achive this and using what kind of tool? (preferably free or open source tools, can be GUI based or script library, having a visualization or dashboard is a plus)

Thanks for the help. Sorry for noob questions

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ Any relational database will do this. Just parse the "timestamp" (actually a date). GROUP BY month, then ORDER BY COUNT. $\endgroup$
    – Emre
    Feb 27, 2017 at 18:44
  • $\begingroup$ The problem is item_details contains phrases, not just words. If I ORDER BY COUNT I think the result would consider "wooden chair" & "plastic chair" as different object, instead of considered as same "chair". I was thinking to use this approach before asking here. But that's why I'm asking this question here, to know how datascientist would approach this goal(and learn to do it as efficiently as possible) , instead of just doing manual db queries. Thanks anyway :) $\endgroup$
    – Raizerde
    Feb 28, 2017 at 5:32
  • $\begingroup$ You can extract infer the broader categories from the item_details. At the simplest level, you can tokenize the item_details string to see if your category is mentioned. This appears to work for your example. In more complicated cases, you can build a classifier to label (create derived columns) your items. I have no experience with Pentaho. $\endgroup$
    – Emre
    Feb 28, 2017 at 6:29

1 Answer 1

Reset to default
0
$\begingroup$

If you are new to data science and data munging, this could be kind of a tricky task, but a good one to get your feet wet. Many programming languages have the capability to do this (R, Python, Matlab, etc.). I use R primarily, so I'll give you a brief heuristic for how I'd approach the task in R. Perhaps looking into these steps will get you started.

  1. Install R
  2. Install some packages that will help you along ('tm' for text mining,'dplyr' for cleaning/organizing your data, and perhaps also 'lubridate' for working with dates/times)
  3. Read in your data from your source, be it a text file, spreadsheet, or some database (if a database, you'll have to conquer connecting R to said database too)
  4. You want to do a word frequency analysis for each month. How you accomplish this will depend on how the data is organized, which I do not know, but it would involve first rounding all your dates to month (using lubridate's 'floor_date()' function is one way), then parsing the text for each month into a corpus that can be analyzed (using package tm).
  5. Finally, for each month I would make a table counting words, sorting by frequency. That would give you, for each month, the top 'trending' words. To discount words like 'the' and 'a', I might also use some of the tools in the 'tm' package to clean things up.
  6. Note that in #5 I said 'words', not 'terms'. If you want to account for terms consisting of > 1 words, you'll have to 'tokenize' them, but that's beyond the scope of this very brief intro.

As with many data science tasks, there are many ways to attack this; the above is but one of many possibilities.

Hope that helps.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks for taking your time to write a nice comprehensive answer :) It really help me to get started, and at least to know what common terms and tools, so I can look around for more learning resources. I'll wait for another answer tho, before deciding to choose the best answer. $\endgroup$
    – Raizerde
    Feb 28, 2017 at 6:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.