In this wiki page there is a function corr() that calculates the Pearson coefficient of correlation, but my question is that: is there any function in Hive that enables to calculate the Kendall coefficient of correlation of a pair of a numeric columns in the group?
$\begingroup$
$\endgroup$
4
-
$\begingroup$ I have never seen such a thing in Hive. $\endgroup$– Sean OwenDec 8, 2014 at 14:23
-
$\begingroup$ That answer does not solve the problem :) $\endgroup$– MarcinDec 10, 2014 at 9:57
-
$\begingroup$ Is there a way of implementing a function on my own on Hive or maybe should I use different tool - maybe should I check Apache Spark? $\endgroup$– MarcinDec 10, 2014 at 10:13
-
$\begingroup$ Heh, yeah that's why I only commented. What I mean is that, having sniffed around Hive a long time, I've never heard of this, so it's unlikely to exist, but I don't know for sure that this is the answer. Spark does not have it either. $\endgroup$– Sean OwenDec 10, 2014 at 12:11
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
5
In Hive itself? Unfortunately, the answer is simply no -- as the language definition manual shows, that statistic is simply not built in. In addition to the language manual, you can get more information on statistics in development in Hive here and here.
Having said that, there are plenty of ways to calculate Kendall's W on data that's in Hive.
You could write out the data to a file or query it into R or a statistical package such as SAS, Stat, MATLAB, Excel, etc then run your calculation and, if necessary, write your results back to Hive.
In R, for instance, you could do something like this:
install.packages("RODBC")
require(RODBC)
db <- odbcConnect("Hive_DB")
hql <- "select * from table A"
data <- sqlQuery(db , hql)
kenw <- cor(x = data$a, y = data$b, method="kendall")
sqlSave(db, kenw, tablename = "new_table_of_kendall_coef")
or (if using Linux or Unix) then you could use RHive without needing to use an ODBC name.
Another way to go about it would be to take the functions that do exist in Hive (which you linked to) and calculate Kendall's coefficient yourself with a custom function. As to how to specifically implement that, well you'd probably want to post on Cross Validated (stats.stackexchange.com).
-
$\begingroup$ Importing 10^10 rows of data into R just to calculate kendall coefficient is just simply imposible and not smart. That's why I wrote a question here. I am aware on how to specify implementation of kendall coefficient, just simply check the code of
corfunction in R :) I think the best idea would be to implement user defined function to calculate that kendall coefficient. Do you maybe know where could I upload that function later on for public use? By the way, have you used RHive :) do you recommend any good materials for start? $\endgroup$– MarcinFeb 28, 2015 at 16:03 -
$\begingroup$ @MarcinKosinski Instead of importing 10^10 rows, why not just sample it? As for UDF's, if you have one or develop one with other people you can put it into a JIRA ticket for addition to Hive via the links in my solution, or become a committer and commit code to the project (or give your code to a committer): apache.org/dev/committers.html $\endgroup$– Hack-RFeb 28, 2015 at 20:12
-
$\begingroup$ What's the bias of coefficient calculated on 1000 rows-long sample when population has 10^10 observation? $\endgroup$– MarcinMar 1, 2015 at 17:08
-
$\begingroup$ @MarcinKosinski I'm not sure, but I pull samples of data that are anywhere from 10,000 - 30,000 rows from SQL Server and Hive on a daily basis. Sampling data in R from Hive and other databases has been a standard approach in my team at 2 different companies where I've worked. I expected that even if you had a Kendall Coefficient function in Hive it would be impractical to run it on all 10^10 rows without sampling, at least if it's something you plan to do more than once. That would be very time- and resource-intensive without buying you much. $\endgroup$– Hack-RMar 2, 2015 at 2:39
-
1$\begingroup$ I've found a spare.cor functions for sparse matrixes in R that calculates correlation for any dimension for sparse matrix. @Hack-R think about this bias. By the way sampling coefficients should have also something-like bootstrap confidence intervals: see this: stats.stackexchange.com/questions/126176/… And the sparse.cor function for sparse matrixes is here: stackoverflow.com/questions/5888287/… $\endgroup$– MarcinJun 29, 2015 at 9:44