I am creating a summarizer in R of news articles. Hence, I am building an algorithm to get total score of each sentence based on sum of frequency of lemma in the sentences of the article, annotated using cleannlp annotator with the corenlp as the backend (and some other stuff NER, topic modelling)
While it is possible to get sentences scores, I currently do not know how to extract the sentences split that were used to create the token.
e.g. pseudocode below
y1= "hello, world hello again, you" y2="hello, world hello again, me" y=c(y1,y2) library(cleanNLP) library(coreNLP) library(rJava) cnlp_init_corenlp("en", anno_level = 2, lib_location = "~user/R/R-3.5.3/library/coreNLP/extdata/stanford-corenlp-full-2018-10-05", mem = "9g", verbose = FALSE) anno=cnlp_annotate(unlist(y), backend="corenlp") [some more code to produce the scores as below] doc id | sentence id | score 1 | 1 | 2+1=3 1 | 2 | 2+1+1=4 2 | 1 | 2+1=3 2 | 2 | 2+1+1=4
In this case, how do I get the second sentence of each document based on the sentences split done for the tokens? I am hessitant of using other sentence splitters e.g. from tm packages, as it may yield inconsistent splitting with the one done in the annotation.
Thank you in advance.