python (65.1k questions)

javascript (44.2k questions)

reactjs (22.7k questions)

java (20.8k questions)

c# (17.4k questions)

html (16.3k questions)

r (13.7k questions)

android (12.9k questions)

Questions - cosine-similarity

How to count word similarity between two pandas dataframe

Here's my first dataframe df1 Id Text 1 Asoy Geboy Ngebut 2 Asoy kita Geboy 3 Bersatu kita Teguh Here's my second dataframe df2 Id Text 1 Bersatu Kita 2 Asoy Geboy Jalanan Simila...
test-img

Nabih Bawazir

python

pandas

dataframe

text

cosine-similarity

Votes: 0

Answers: 1

Latest Answer

IIUC, you need to compute a set intersection: l1 = [set(x.split()) for x in df1['Text'].str.lower()] l2 = [set(x.split()) for x in df2['Text'].str.lower()] pd.DataFrame([[len(s1&s2)/len(s1) for s...
test-img

mozway

Efficiently compute cosine similarity

I have a bank of about 100k strings and when I get a new string, I want to match it to the most similar string. My thoughts were to use tf-idf (makes sense as keywords are quite important), then match...
test-img

user112633

python

pandas

tf-idf

cosine-similarity

Votes: 0

Answers: 1

Latest Answer

Consider using vectorized computations rather than looping over DataFrame rows (which is very slow and should be avoided). I'm not sure how the arrays are represented in the dataframe, so make sure yo...
test-img

Yoav Tulpan

How calculate clusters coherence/quality?

I did embeddings with fasttext and I have clusters thanks to KMeans. I would like to calculate similarities inside each cluster to check if the sentences inside are well clustered. I want to keep sent...
test-img

codelifevcd

machine-learning

data-science

cluster-analysis

k-means

cosine-similarity

Votes: 0

Answers: 1

Latest Answer

Maybe something like this... # clustering words into similar groups: import numpy as np from sklearn.cluster import AffinityPropagation import distance words = 'XYZ,LDPELDKSL,DFKLKSLFD,ABC,DLFKF...
test-img

ASH

Determining cosine similarity for large datasets

I am currently using a dataset of over 2.5 million images, of which I use the image itself as a comparison to eachother, for use in a content-based recommendation engine. I use the following code to c...
test-img

pippinpaddleopsicopolis

python

machine-learning

deep-learning

recommendation-engine

cosine-similarity

Votes: 0

Answers: 2

Latest Answer

You have 2,500,000 entries. So resulting matrix has 6.25e+12 entries. You need to ask yourself what do you plan to do with this data, compute only what you need, and then the storage will follow. Comp...
test-img

lejlot

Posts

Questions

Blogs

Jobs