I am trying to analyze auto claims narrative documents which contain description about the accident usually free text written by claims executives. Is there a nlp technique I could use to identify cause of loss like : drunk and driving , negligence , bad weather , etc ?
I have used TF-IDF technique to rank order words per claim but it does really help to identify the exact cause of loss easily even if I concentrate on top 20% words. I am aware of word embedding but really not sure if they could help .
Also tried LDA topic modeling in the past but that gives topics for entire corpus rather than claim level.
Below is the code I used to calculate tf-idf scores for the unigrams.
library(dplyr)
library(tidytext)
library(janeaustenr)
library(tidyr) # for separate
library(qdapDictionaries)
#generate unigrams
loss_narrative_by_file_1 <- loss_narrative_by_file %>%
unnest_tokens(unigram, whole_text_proxy, token = "ngrams", n = 1)
# removing punctuation and numbers ( removing punct / digits faster on unigrams than whole text)
loss_narrative_by_file_1$unigram <- gsub("[[:punct:]]|[[:digit:]]|^http:\\/\\/.*|^https:\\/\\/.*","",loss_narrative_by_file_1$unigram)
#retaining non blank unigrams
loss_narrative_by_file_1 <- loss_narrative_by_file_1[unigram!="",]
# retain meaningful words only
loss_narrative_by_file_1 <- loss_narrative_by_file_1[unigram %in% GradyAugmented]
# remving stopwords
unigrams_filtered <- loss_narrative_by_file_1 %>%
filter(!unigram %in% stop_words$word)
#check coutns
unigrams_filtered %>%
count(unigram,sort=TRUE)
# after cleansing
unigram n
<chr> <int>
1 claim 56247
2 loss 55068
3 policy 50394
4 insured 47512
5 date 36879
6 coverage 32167
7 plaintiff 31358
8 vehicle 26156
9 umbrella 25055
10 company 24479
# tf- idf ###############################################
# concat claim & filename
unigrams_filtered$loss_file <- paste(unigrams_filtered$loss_Loss,unigrams_filtered$filename_Trimmed, sep = "_")
unigram_tf_idf <- unigrams_filtered %>%
count(loss_file, unigram) %>%
bind_tf_idf(unigram, loss_file, n) %>%
arrange(desc(tf_idf))
### validating results of tf-idf shows top 30% words with highest tf-idf score tend to contain the words associated with cause of loss. How can I narrow down further ?enter code here