1
$\begingroup$

I am creating a summarizer in R of news articles. Hence, I am building an algorithm to get total score of each sentence based on sum of frequency of lemma in the sentences of the article, annotated using cleannlp annotator with the corenlp as the backend (and some other stuff NER, topic modelling)

While it is possible to get sentences scores, I currently do not know how to extract the sentences split that were used to create the token.

e.g. pseudocode below

y1= "hello, world
    hello again, you"

y2="hello, world
    hello again, me"

y=c(y1,y2)

library(cleanNLP)
library(coreNLP)
library(rJava)

cnlp_init_corenlp("en", anno_level = 2, lib_location = "~user/R/R-3.5.3/library/coreNLP/extdata/stanford-corenlp-full-2018-10-05",
mem = "9g", verbose = FALSE)

anno=cnlp_annotate(unlist(y), backend="corenlp")

[some more code to produce the scores as below]

 doc id  | sentence id   |   score
    1    |     1         |   2+1=3
    1    |     2         |   2+1+1=4
    2    |     1         |   2+1=3
    2    |     2         |   2+1+1=4

In this case, how do I get the second sentence of each document based on the sentences split done for the tokens? I am hessitant of using other sentence splitters e.g. from tm packages, as it may yield inconsistent splitting with the one done in the annotation.

Thank you in advance.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.