New answers tagged cosine-distance
0
I think it's rarely meaningful to consider cosine similarity on sparse data like this, not just because of sparsity (because it's only defined for dense data), but because it's not obvious the cosine similarity is meaningful. For example a user that rates 10 movies all 5s has perfect similarity with a user that rates those 10 all as 1. Magnitude doesn't ...
0
Have you couple of types na's handling. Its sample of code:
def na_handling(df, name_of_strategy):
#list of stategies -> mean, mode, 0, spefic_value, next_row, previous_row
if name_of_strategy=="previous_row":
df.fillna(method="backfill", inplace=True)
return df
elif name_of_strategy=="next_row":
...
0
def cosine_sim(df1, df2):
df1na = df1.isna()
df1clean = df1[~df1na]
df2clean = df2[~df1na]
df2na = df2clean.isna()
df1clean = df1clean[~df2na]
df2clean = df2clean[~df2na]
# Compute cosine similarity
distance = cosine(df1clean, df2clean)
sim = 1 - distance
return sim
2
There are libraries that are specialized in exactly that task, for instance FAISS by Facebook AI Research:
Faiss is a library for efficient similarity search and clustering of dense vectors. It contains algorithms that search in sets of vectors of any size, up to ones that possibly do not fit in RAM. It also contains supporting code for evaluation and ...
0
Instead of doing a cosine similarity in the first place, I would like you to have a look at some of the similarity measures which exist for categorical data such as Eskin, IOF, OF, Lin, Lin1, Goodall1, 2, 3, 4, etc.
Since you are working with python, my suggestion to you would be to import the library named Categorical_similarity_measures and construct the ...
1
I would suggest also to give a shot to gensim. It's pretty quick compare to other self-written 'top_n retrieval by cosine-similarity' functions.
Save your embeddings in a .txt or .csv file and then load it using the command 'load_word2vecformat'. Once you load the model you can use the function 'similar_by_word' or 'similar_by_vector' to retrive the n ...
1
@lsbister,
You could create a pandas dataframe and use a dask function/lambda function to parellize the computation of one vs all at the same time.
If you use dask, you can create partitions and map the response back.
In case you use pandas, you can use the apply function and parallelize the computations to a certain extent.
0
I'm not really sure what you're asking, but in general, you need to fit an Estimator to data so it can learn what it has to do, then you transform data with it. fit_transform just does fit and then transform. Here you fit the transformer to Name_clean, and then apply it to both in turn. That's pretty normal.
Top 50 recent answers are included
