I've been trying to create a similarity matrix in pandas from with a matrix multiplication operation on a document-term count matrix with 2264 rows and 20475 columns.
The calculation completes in IPython but inspection shows the results all come back as NaN
.
I've also tried doing the same job in numpy, tried converting the original matrix to_sparse
and even re-casting the values as integers, but still no joy.
Can anyone suggest the best approach to tackle the problem?
EDIT: Here's my code thus far:
path = "../../reuters.db"
%pylab inline
import pandas as pd
import numpy as np
import pandas.io.sql as psql
import sqlite3 as lite
con = lite.connect(path)
with con:
sql = "SELECT * FROM Frequency"
df = psql.frame_query(sql, con)
print df.shape
df = df.rename(columns={"term":"term_id", "count":"count_id"})
pivoted = df.pivot('docid', 'term_id', 'count_id')
pivoted.to_sparse()
similarity_matrix = pivoted.dot(pivoted.T)
df
? $\endgroup$ – damienfrancois Aug 1 '14 at 12:17