1
$\begingroup$

Ok, I have this data with 3 columns, unique id, raw text, and review text. My task is play with the dataset and find meaningful insights from it. Raw text is in plain English but review text is in another language. I have no idea how to proceed with the dataset. Even after I clean the data from the raw text, what should I do with review one because it is in another language. Which text analysis should I do and how can I implement it on the dataset?

dataset

$\endgroup$
1
$\begingroup$

The language that are used on review text just happened to be my native language. I can confirm that the review_text at least from what you showed above is a direct translation of the raw text (although I would say the translation is not perfect).

Maybe you can consider making unsupervised model and probably compare between these two, see how much they match (theoretically since they are the same text they should have many overlaps).

$\endgroup$
0
$\begingroup$

My first thought seeing this sample of data was parallel corpus, and Yohanes just confirmed that the text columns are translations from each other.

The main thing to do with this kind of data is to train a machine translation model :)

$\endgroup$
0
$\begingroup$

As mentioned in the answers, you can try an unsupervised approach to compare the two texts. To provide some more detail on that, you can use some existing word embeddings to generate word embeddings for the two texts. From a quick google search of the words in review_text it looks like it is Indonesian language. But nonetheless you can use FastText to generate word embeddings for it, it supports more than 150 languages.

The word embeddings would be a high dimensional vector, you can use some dimensionality reduction method like PCA or t-SNE and try visualizing these sentences to find how they're structured, if there is an overlap or maybe some other useful insight. You can also try tagging Part of Speech tags for both the languages and see how the structure of the sentence differs based on these tags. You will be able to find a lot of interesting patterns once you try visualizing the word embeddings.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.