2
$\begingroup$

I've been trying to create a similarity matrix in pandas from with a matrix multiplication operation on a document-term count matrix with 2264 rows and 20475 columns.
The calculation completes in IPython but inspection shows the results all come back as NaN.
I've also tried doing the same job in numpy, tried converting the original matrix to_sparse and even re-casting the values as integers, but still no joy.
Can anyone suggest the best approach to tackle the problem?

EDIT: Here's my code thus far:

path = "../../reuters.db"
%pylab inline
import pandas as pd
import numpy as np
import pandas.io.sql as psql
import sqlite3 as lite
con = lite.connect(path)
with con:
    sql = "SELECT * FROM Frequency"
    df = psql.frame_query(sql, con)
    print df.shape
df = df.rename(columns={"term":"term_id", "count":"count_id"})
pivoted = df.pivot('docid', 'term_id', 'count_id')
pivoted.to_sparse()
similarity_matrix = pivoted.dot(pivoted.T)
$\endgroup$
2
  • $\begingroup$ Your code please. $\endgroup$
    – Emre
    Jul 25, 2014 at 17:52
  • 1
    $\begingroup$ Have you checked the contents of df? $\endgroup$ Aug 1, 2014 at 12:17

0

Browse other questions tagged or ask your own question.