2
$\begingroup$

I'm working on a problem where in I have some data sets about some power generating units. Each of these units have been activated to run in the past and while activation, some units went into some issues. I now have all these data and I would like to come up with some sort of Ranking for these generating units. The criteria for ranking would be pretty simple to start with. They are:

  1. Maximum number of times a particular generating unit was activated
  2. How many times did the generating unit ran into problems during activation

Later on I would expand on this ranking algorithm by adding more criteria. I will be using Apache Spark MLIB library and I can already see that there are quite a few algorithms already in place.

http://spark.apache.org/docs/latest/mllib-guide.html

I'm just not sure which algorithm would fit my purpose. Any suggestions?

| improve this question | | | | |
$\endgroup$
  • $\begingroup$ It sounds like you are not trying to predict anything but rather characterize the performance. True? $\endgroup$ – Pete Apr 13 '16 at 14:28
  • $\begingroup$ Yes, I just want to use the dense data and come up with a ranking order for all the power generating units. $\endgroup$ – sparkr Apr 13 '16 at 17:39
  • $\begingroup$ Do you want an exact count of number of activations etc. or approximate counts? If approximate then there are interesting algorithms if exact then it's just a group by operation for which you can look up the algo in spark $\endgroup$ – wabbit Apr 19 '16 at 14:13
2
$\begingroup$

You can use a clustering algorithm such as k-means to divide the generators into groups. You never know what kind of groups you'll get until you try it. Try and assess the character of each group of generators as you increase the number of clusters. At some point you should find a meaningful division of generators. The inputs to your k-means algorithm will be the criteria you mentioned in your post: the number of times it was activated, the number of activation problems, and so forth. When you are finished, the group a generator belongs to is its ranking. This method will not generate a ranking of 1-1000 if you have 1000 generators. Rather it will give you, for example with k=3: a group of 243 outstanding generators, 320 average generators, and 446 terrible generators.

| improve this answer | | | | |
$\endgroup$
  • $\begingroup$ Cool! That seems to be a good starting point. I will explore more on it! Thanks for the suggestion. $\endgroup$ – sparkr Apr 15 '16 at 5:05
0
$\begingroup$

With a few exceptions, you can pretty much use any machine learning algorithm for your model. The beauty of most machine learning packages is that the interface for each model is mostly the same (although the tuning parameters will differ), and it takes just a few lines of code to try out each model. There is no reason you should artificially constrain yourself to trying certain models.

Some exceptions to this rule are algorithms that might only work for classification or only work for regression. It sounds like you're trying to predict a continuous target variable that you'll then use for ranking. If that's the case, then you won't be able to use an algorithm called Naive Bayes because it can only output probabilities. In other rare cases like Deep Learning models, the run time can be very long (hours or days) and in those cases you wouldn't want to use an algorithm like that unless you had a good reason to do so (e.g., face recognition in images). You should be able to use nearly every algorithm in MLlib though: gradient boosting, random forests, etc.

| improve this answer | | | | |
$\endgroup$
  • $\begingroup$ Thanks for the post. Could you point me to some examples? $\endgroup$ – sparkr Apr 12 '16 at 14:24
  • $\begingroup$ Sure. Are you looking for MLlib examples specifically, or any type of machine learning library? What's your preferred language and what is the file size of your data? $\endgroup$ – Ryan Zotti Apr 12 '16 at 14:29
  • $\begingroup$ Yes, I'm only looking for MLib examples and I use Scala. I want to do this on a historical data set that would be say a hundred of MB in size to start with. $\endgroup$ – sparkr Apr 12 '16 at 14:31
  • $\begingroup$ Unfortunately I haven't personally used MLlib in Scala, only python, so maybe another poster can show you some of their own code (I would just end up pulling something from Google). Also I think you're trying to say MLlib instead of Mlib. There is supposed to be an extra L in there so that when fully expanded it shows: Machine Learning (ML) library (lib): MLlib $\endgroup$ – Ryan Zotti Apr 12 '16 at 14:38
  • $\begingroup$ I'm referring to the Spark-MLib library. Sorry for not being explicit! $\endgroup$ – sparkr Apr 12 '16 at 14:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.