I've been trying to create a similarity matrix in Pandas from with a matrix multiplication operation on a document-term count matrix with 2264 rows and 20475 columns.

The calculation completes in IPython but inspection shows the results all come back as NaN.

I've also tried doing the same job in numpy, tried converting the original matrix to_sparse and even re-casting the values as integers, but still no joy.

Can anyone suggest the best approach to tackle the problem?

EDIT: Here's my code thus far:

path = "../../reuters.db"
%pylab inline
import pandas as pd
import numpy as np
import pandas.io.sql as psql
import sqlite3 as lite
con = lite.connect(path)
with con:
    sql = "SELECT * FROM Frequency"
    df = psql.frame_query(sql, con)
    print df.shape
df = df.rename(columns={"term":"term_id", "count":"count_id"})
pivoted = df.pivot('docid', 'term_id', 'count_id')
similarity_matrix = pivoted.dot(pivoted.T)
  • $\begingroup$ Your code please. $\endgroup$ – Emre Jul 25 '14 at 17:52
  • $\begingroup$ Have you checked the contents of df? $\endgroup$ – damienfrancois Aug 1 '14 at 12:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.