I have a dataset having text documents as each entry , the documents are of various ethical languages which are not specified. How should be my strategy if I want to summarise the content of the dataset.

For eg. Doc1 : "Ruby is a dog lover.Ruby loves her dogs so much. Ruby has a pet whose name is a dog"

Now summarising it is easy as "Ruby loves dog.Ruby has a dog"

But How should be the approach , when the dataset also has documents in french, chinese, hindi etc...

Some of the problems are: there are cases of code switch, and I don't know what are the languages in the dataset before hand.

What I am guessing at this is to may be change all the dataset to english language and then do the tasks? In case if converting to english is a good choice then which library would be better?


There are two basic approaches to summarization: abstractive and extractive.

Extractive summarization is simpler and consists of selecting informative pieces/sentences from text. It is commonly performed using algorithms like TextRank (a variation of PageRank). An example implementation can be found in gensim. There are other approaches, for example check out sumy library.

Abstractive summarization on the other hand consists of summarizing documents with shorter sentences which do not have to come from the input text (an abstract may paraphrase some fragments of text). It is commonly performed using Deep Learning Seq2Seq methods.

Which approach is better for start? Extractive wins by far. First of all, you don't have to train any model, as these methods rely on information found solely in summarized text. Second, deep learning models, even if they exist and can be easily found *, are trained on specific texts, so they might not generalize well across application domains.

But How should be the approach , when the dataset also has documents in french, chinese, hindi etc...

If you have good tokenizers and stemmers for these languages you should be fine with TextRank, as it only relies on word counts (I'm not sure about Chinese though). BTW language detection is by far easier than summarization - just count number of stopwords of different languages that document contains.

*They're not, for example Google published a paper sometime ago on the topic and didn't even disclose model, since the training dataset was proprietary. I was trying to find something for abstractive summarization sometime ago but I couldn't find anything that had good pretraining and worked out of the box.

| improve this answer | |
  • $\begingroup$ Thanks for your comment, I am looking forward for abstractive summarisation.I am thinking of using word vecs generated from a multilingual corpora and using them to train my network, may be.. $\endgroup$ – dodo Jun 3 '18 at 12:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.