Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
I have f.ex.: two news-articles that report the same event. However, these two text are similar BUT not the same. I would like to combine these two texts creating one text that contains only the most "relevant" information.
I was thinking to check the texts paragraph wise on its "information value" and then only combine the paragraphs that are the most "relevant".
Any suggestions on an approach how to do this? Any research papers about this topic?
Your description could corresponds to the NLP task of summarization. This is an active field of research: https://scholar.google.com/scholar?q=text+summarization
A much simpler option is to only extract sentences from both: in this case the goal is not to produce a text which reads like a story, just an enumeration of sentences.
Even in this case you will have to define how to measure "information value", this is not easy afaik.
Maybe this is not what you are looking for (depends on how similar the texts are), but you could probably try to approach the problem via "string distance". This will not necessarily detect semantic similarity, but similar ngrams or sequences of words.
You could compare each para of the texts and keep only one of them if they are "similar" or keep both of them if they are "not similar" according to a predefined criterion. This will NOT give you a perfect final text, but a summary of (hopefully distinct) content.
I played arround with some code in R a while ago.
# 0) Load packages:
library(dplyr)
library(tidytext)
library(fuzzyjoin)
library(tokenizers)
library(stringdist)
library(pdftools)
library(parallel)
# Global parameter settings
# Length of word-sequences to be considered
# Ngram too long (danger of missing equal sequences)
# Ngram too small (will find many matches)
ngramlength = 7
# Regulation of string comparison (higher = less accurate)
maxd = 12
# Method of string comparison (see ?amatch) / Options: "osa", "lv", "dl", "hamming", "lcs", "qgram", "cosine", "jaccard", "jw", "soundex"
matchmethod = "osa"
###########################################################################
# 1) Read content:
# Read text
# Note: One sentecne in both examples below is very similar (text2, last sentence)
# IN TEXT 1:
# The EU ETS was launched in 2005 and is the first - and still by far the largest - international
# system for trading greenhouse gas emission allowances covering over three-quarters of the allowances
# traded on the international carbon market.
# IN TEXT2:
# The EU Eemissions Trading System has been launched in 2005 and is the first international
# system covering over three-quarters of the allowances traded on the carbon market.
text1=as.character(
"European Union Emissions Trading System (EU ETS) is the cornerstone of the European Union's policy to tackle climate change
and its key tool for cost-effective reduction of emissions of carbon dioxide (CO2) and other greenhouse gases (GHG) in the power,
aviation and industrial sectors. The EU ETS was launched in 2005 and is the first - and still by far the largest - international
system for trading greenhouse gas emission allowances covering over three-quarters of the allowances traded on the international carbon market.
The EU ETS operates in the 31 countries of the European Economic Area (EEA). It limits emissions from nearly 11,000 power plants and manufacturing
installations as well as slightly over 500 aircraft operators flying between EEA's airports (Report from the Commission to the
European Parliament and to the Council, Report on the functioning of the European carbon market, 23 November 2017 (COM(2017) 693 final, p. 7)."
)
text2=as.character(
"The primary intended audience of this package is scholars and professionals in fields where the impact of news on society is a prime factor,
such as journalism, political communication and public relations (Baum and Groeling 2008; Boczkowski and De Santos 2007; Ragas 2014).
To what extent the content of certain sources is homogeneous or diverse has implications for central theories of media effects, such as
agenda-setting and the spiral of silence (Bennett and Iyengar 2008; Blumler and Kavanagh 1999). Identifying patterns in how news travels
from the initial source to the eventual audience is important for understanding who the most influential gatekeepers are (Shoemaker and Vos 2009).
Furthermore, the document similarity data enables one to study news values (Galtung and Ruge 1965) by analyzing what elements of news
predict their diffusion rate and patterns. The EU Eemissions Trading System has been launched in 2005 and is the first international
system covering over three-quarters of the allowances traded on the carbon market."
)
mytext1=data_frame(text1)
mytext2=data_frame(text2)
###########################################################################
# 2) Generate ngrams:
ttext = unnest_tokens(mytext1, ngram, text1, token = "ngrams", n = ngramlength)
torig = unnest_tokens(mytext2, ngram, text2, token = "ngrams", n = ngramlength)
# Drop every second row (we might not need all ngrams)#toDelete <- seq(0, length(dat), 2)
# ATTENTION: ngrams should not be too long!
ttext <- ttext[-seq(0, length(ttext$ngram), 2), ]
torig <- torig[-seq(0, length(torig$ngram), 2), ]
###########################################################################
# 3) Compare each ngram in ttext to each in torig
# 3.1) Use the "stringdist" package
# https://cran.r-project.org/web/packages/stringdist/stringdist.pdf
# With "generous" distance allowed, similarities arround 151-153 are detected
amatch(ttext$ngram,torig$ngram,maxDist=10, method = matchmethod)
# Let's store results in a way that allows interpretation
results1 = cbind(ttext$ngram, torig$ngram[amatch(ttext$ngram,torig$ngram,maxDist=maxd, method = matchmethod)])
# Remove "nas"
results1 = results1[complete.cases(results1), ]
results1
# 3.2) "Join" similar ngrams (uses stringdist)
# THIS METHOD YIELDS SAME RESULT AS METHOD ABOVE
results2 = stringdist_join(ttext, torig,
by = "ngram",
mode = "left",
ignore_case = T,
method = matchmethod,
max_dist = maxd,
distance_col = "dist"
) %>%
group_by(ngram.x) %>%
top_n(1, -dist)
results2
How do you identify which words are relevant? If you already have a set of relevant words, you can use TFIDF for it. TFIDF considers keywords as features and checks explicitly for them and assigns a score based on its mathematical formula. You can understand further about it in this link: http://www.tfidf.com/ .
If you are looking to extract relevant keywords or extract a summary of text, you can use gensim's summarizer.
To get summary:
from gensim.summarization.summarizer import summarize
text = '''Rice Pudding - Poem by Alan Alexander Milne
... What is the matter with Mary Jane?
... She's crying with all her might and main,
... And she won't eat her dinner - rice pudding again -
... What is the matter with Mary Jane?
... What is the matter with Mary Jane?
... I've promised her dolls and a daisy-chain,
... And a book about animals - all in vain -
... What is the matter with Mary Jane?
... What is the matter with Mary Jane?
... She's perfectly well, and she hasn't a pain;
... But, look at her, now she's beginning again! -
... What is the matter with Mary Jane?
... What is the matter with Mary Jane?
... I've promised her sweets and a ride in the train,
... And I've begged her to stop for a bit and explain -
... What is the matter with Mary Jane?
... What is the matter with Mary Jane?
... She's perfectly well and she hasn't a pain,
... And it's lovely rice pudding for dinner again!
... What is the matter with Mary Jane?'''
print(summarize(text))
And she won't eat her dinner - rice pudding again - I've promised her dolls and a daisy-chain, I've promised her sweets and a ride in the train, And it's lovely rice pudding for dinner again!
To extract keywords:
from gensim.summarization import keywords
text = '''Challenges in natural language processing frequently involve
... speech recognition, natural language understanding, natural language
... generation (frequently from formal, machine-readable logical forms),
... connecting language and machine perception, dialog systems, or some
... combination thereof.'''
keywords(text).split('\n')
[u'natural language', u'machine', u'frequently']
You could also you use a combination of gensim and TFIDF, by extracting a summary/keywords using gensim and comparing them using TFIDF.
The Gensim NLP library is pretty strong and has a similarity API.
Easy to install and test.
https://radimrehurek.com/gensim/tut3.html#similarity-interface
Good luck!
