1
$\begingroup$

sentence = 'अच्छा होगा अगर इसमें और गहने ना हों'

Which method will work for this task? Is any pretrained model available to convert this text to vectors? Please help by giving the code.

$\endgroup$
0
$\begingroup$

Hindi text is in Unicode format and can be read as follows in Python:

separators = [u"।", u",", u"."]
text = open("indiantext.txt").read()
#This converts the encoded text to an internal unicode object, where
# all characters are properly recognized as an entity:
text = text.decode("utf-8")

#this breaks the text on the white spaces, yielding a list of words:
words = text.split()

counter = 1

output = ""
for word in words:
    #if the last char is a separator, and is joined to the word:
    if word[-1] in separators and len(word) > 1:
        #word up to the second to last char:
        output += word[:-1] + u"(%d) " % counter
        counter += 1
        #last char
        output += word[-1] +  u"(%d) " % counter
    else:
        output += word + u"(%d) " % counter
    counter += 1

print

It is not clear from the question as to what exactly you mean by pretrained model for converting into vector but assuming that you are asking about the language model for hindi, ULMFiT is something you must refer to

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.