I am looking for the way to get the similarity between two item names using integer encoding or one-hot encoding.

For example, "lane connector" vs. "a truck crane".
I have 100,000 item names consisting of 2~3 words as above.
also, items have its size(36mm, 12M, 2400*1200...) and unit(ea, m2, m3, hr...)

I wanna make (item name, size, unit) as a vector. To do this, I need to change texts to numbers using some way. All I found is only word2vec things, but my case has no context corpus. So I don't think it is possible to learn some context from my data.

Example Image of dataset

  • $\begingroup$ Will be good if you provide few samples with complete feature list, possibly will help me in my better understanding. $\endgroup$ – vipin bansal Jun 20 '19 at 5:11
  • $\begingroup$ Why word2vec or Glove wouldn't be useful for your item names? Also before that you could think of using TF-IDF to construct your vectors, it is very simple and I think it will be better than integer or one-hot encoding. $\endgroup$ – TwinPenguins Jun 20 '19 at 5:17
  • $\begingroup$ @vipinbansal added the image :) $\endgroup$ – Ken Kim Jun 20 '19 at 5:36
  • $\begingroup$ @TwinPenguins I don't think that I can use those method because of the data $\endgroup$ – Ken Kim Jun 20 '19 at 5:38

I'm not sure, if it's possible with this data set. Word2Vec is used to generate word embedding, which works on the principle of "words association" in a sentence.

So I dont think you can apply Word2Vec on this dataset which looks like doesn't have any association, except on some places where you can match(perform clustering) some parameters like:

  1. Units
  2. Size/dimension of the item-name

Interested to know some solution for such types of problems.


Okay, so what I understand is you just have a list of words and want to get word vectors for those. You are correct that you cannot train a word2vec model as it requires a corpus. But what you can do is use a pre-trained model (word2vec or glove). I suggest you use word2vec as gensim has a pretty simple implementation. You can download Google’s pre-trained model here. And then you can use the following code to get word_embed for a given word_list.

import gensim
model = gensim.models.KeyedVectors.load_word2vec_format('GoogleNews-vectors-negative300.bin', binary=True)
vocab = model.vocab.keys()
word_embed = {}
for word in word_list:
    if word in vocab:

Also, you'll have to apply some pre-processing to your word list so that you can get maximum matches from the pre-trained embeddings (like removing the etc.) And if a word is still not found in the pre-trained embeddings you can either initialize it randomly or take an average of the embeddings.

  • $\begingroup$ Thank you for your solution :) However, my dataset is made of Korean. i just translated it to post here. What I wanna do right now is making a kind of corpus that has mapping structure. It is consist of like {Base item : mapped item1, item2 ...}. I am just thinking of this to solve this problem. $\endgroup$ – Ken Kim Jun 20 '19 at 7:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.