sentence = 'अच्छा होगा अगर इसमें और गहने ना हों'
Which method will work for this task? Is any pretrained model available to convert this text to vectors? Please help by giving the code.
Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.Sign up to join this community
Hindi text is in Unicode format and can be read as follows in Python:
separators = [u"।", u",", u"."] text = open("indiantext.txt").read() #This converts the encoded text to an internal unicode object, where # all characters are properly recognized as an entity: text = text.decode("utf-8") #this breaks the text on the white spaces, yielding a list of words: words = text.split() counter = 1 output = "" for word in words: #if the last char is a separator, and is joined to the word: if word[-1] in separators and len(word) > 1: #word up to the second to last char: output += word[:-1] + u"(%d) " % counter counter += 1 #last char output += word[-1] + u"(%d) " % counter else: output += word + u"(%d) " % counter counter += 1 print
It is not clear from the question as to what exactly you mean by pretrained model for converting into vector but assuming that you are asking about the language model for hindi, ULMFiT is something you must refer to