I am trying to calculate a cosine similarity using Python in order to find similar users basing on ratings they have given to movies. As it can be expected there are a lot of NaN values. I am using movie dataset from Kaggle.

When I use np.dot() on two nd.arrays the outcome is:'nan'. I have checked with np.nansum() that there are some other than 'nan' values.

I do not want to change all 'nan' values to '0' as it would mean that users have given 0 rating to the movies which lead to 'false' similarity between users.

Please, give me some advice regarding how to proceed with this problem.

Thanks in advance.

  • $\begingroup$ Check if the arrays have the same size when you put them in the function. $\endgroup$ – Juan Esteban de la Calle Apr 27 '19 at 14:14
  • $\begingroup$ Can you provide an example matrix? A smaller version of your actual data. $\endgroup$ – n1k31t4 Apr 27 '19 at 16:01
  • $\begingroup$ If you need so more control, implement the dot multiplication function on your own in Python. $\endgroup$ – Shubham Panchal Apr 27 '19 at 16:19

I think it's rarely meaningful to consider cosine similarity on sparse data like this, not just because of sparsity (because it's only defined for dense data), but because it's not obvious the cosine similarity is meaningful. For example a user that rates 10 movies all 5s has perfect similarity with a user that rates those 10 all as 1. Magnitude doesn't matter in cosine similarity, but it matters in your domain.

It's much more likely that it's meaningful on some dense embedding of users and items, such as what you get from ALS.

To answer the question, you either need to impute the missing ratings (don't assume 0, but a mean value or similar), or ignore dimensions that aren't defined in both.


Have you couple of types na's handling. Its sample of code:

def na_handling(df, name_of_strategy):

        #list of stategies -> mean, mode, 0, spefic_value, next_row, previous_row

        if name_of_strategy=="previous_row":
            df.fillna(method="backfill", inplace=True)
            return df
        elif name_of_strategy=="next_row":
            df.fillna(method="ffill", inplace=True)
            return df
        elif name_of_strategy=="0":
            df.fillna(0, inplace=True)
            return df

        elif name_of_strategy=="mean":
            df.fillna(df.mean(), inplace=True)
            return df
        elif name_of_strategy=="mode":
            df.fillna(df.mode(), inplace=True)
            return df
            print("Wrong specified strategy")

vec1 = na_handling(old_vec, "next_row")
def cosine_sim(df1, df2):

    df1na = df1.isna()
    df1clean = df1[~df1na]
    df2clean = df2[~df1na]

    df2na = df2clean.isna()
    df1clean = df1clean[~df2na]
    df2clean = df2clean[~df2na]

    # Compute cosine similarity
    distance = cosine(df1clean, df2clean)
    sim = 1 - distance

    return sim
  • 1
    $\begingroup$ This doesn't explain what you are trying to do: consider dimensions only where both are non-NaN, which is reasonable. I think you swapped distance and similarity though. $\endgroup$ – Sean Owen Mar 22 '20 at 14:48
  • $\begingroup$ Please provide more context on your answer. the user does not actually ask for a coding solution for cosing similarity calculation. it is not clear as for what you are suggesting with your solution. $\endgroup$ – Wajdi yesterday

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.