# Measure of correlation for term frequency

I'm trying to write a framework to compare a set of labels such as (for a sample of 5 yes/no answers to a question) [0, 1, 1, 1, 0] to a series of features to determine correlation. For numerical non-sparse features, like "number of words" or "average word length", I know I can use a variance-covariance matrix and get a sense for whether or not "number of words" or "average word length" is an informative feature for a model to answer the question.

I'd like to be able to do the same thing for term frequency (let's say using CountVectorizer in scikit-learn), but the resultant covariance matrix will be rather large and will only indicate whether or not that particular term is an informative feature. How do I get some kind of "collapsed" or "aggregate" measure of correlation? Is this even possible?

• I'm not sure I understand your question entirely, but are you looking for something like TF-IDF (Term Frequency - Inverse Document Frequency)? – daniel3412 Oct 9 '15 at 12:54
• You look at frequency in the corpus and discount. Log of count in corpus is a common discount. – paparazzo Jan 8 '16 at 21:44

If you did a simple linear regression for each of the 5 outputs, your r-squared would be a good measure of the "aggregate correlation." If you wanted to compare to the other results, it would be helpful to do the same for each of your single features as well.

From what you describe, it seems that something like Pointwise Mutual Information could satisfy your requirements. It's often used in text mining and opinion analysis to analyse the degree of correlation between specific terms and classes (e.g. a term in a product review and the "positive reviews" class).

if you want a correlation for term frequency, assuming you have a frequency vector representation for each row, then you can compute the similarity between any two rows by computing cosine similarity.

Here is how one could do it in R:

data_set <- c("the big bad dog",
"the small cat and the orange cat",
"the big big dog")
words <- strsplit(data_set, split = " ") #tokenize sentences
vec <- unique(unlist(words)) #vector representation of sentences

m <- matrix(nrow = length(data_set), ncol = length(vec))

for (i in 1:length(words)) { #iterate the index of tokenized sentences
tokens <- words[[i]]
vec_rep <- as.integer(sapply(vec, function(w){sum( w == tokens)})) #create binary word-feature vector
m[i,] <- vec_rep #update matrix
}

df <- data.frame(m, row.names = NULL)
names(df) <- vec
df

##  the big bad dog small cat and orange
##1   1   1   1   1     0   0   0      0
##2   2   0   0   0     1   2   1      1
##3   1   2   0   1     0   0   0      0

cosineSimilarity <- function(df, row1, row2){
x <- as.numeric(df[row1,])
y <- as.numeric(df[row2,])
(x %*% y) / (sqrt(x%*%x * y%*%y))
}

cosineSimilarity(df,1,1) #1.00
cosineSimilarity(df,1,2) #0.30
cosineSimilarity(df,1,3) #0.82


If you have labelled training data, you can create a prototype vector representation for each label. Then during classification of unlabelled data you just compute cosine similarity between target text and each prototype, and then assign the label to prototype that maximizes similarity score.