1
$\begingroup$

Lets consider some sentences below:
"Datascience exchange is a wonderful platform to get answers to datascience related queries and it helps to learn various concepts too"
"Can company1 buy company2? What will be their total turnover then?"
"Coronavirus was originated in china. After that it is spreading all over the world. To prevent it everyone has to take care of cleanliness and prefer vegetarians."

In all above sentences you can see there are multiple questions or utternaces. Sometimes separated by and sometimes by question mark and sometimes by just a dot.
A rule based separation of these sentences fail in many cases. I want to split these sentences in individual intents.

One of the approach I am guessing is by using attention mechanism on different parts of sentences. I cant use gensim etc sentence embeddings as I dont have clear sentence boundaries here.
Can someone suggests if attention approach will work? If yes, any similar code if they can point to, that would be helpful as I haven't coded this before.

If any other better approach can solve this problem then please suggest.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

For the sentences you provided the nltk sentence tokeniser would work just fine. 

from nltk.tokenize import sent_tokenize

sentences = ['Datascience exchange is a wonderful platform to get answers to datascience related queries and it helps to learn various concepts too','Can company1 buy company2? What will be their total turnover then?','Coronavirus was originated in china. After that it is spreading all over the world. To prevent it everyone has to take care of cleanliness and prefer vegetarians.']

for sent in sentences:
    print(sent_tokenize(sent))

out:
['Datascience exchange is a wonderful platform to get answers to datascience related queries and it helps to learn various concepts too']
['Can company1 buy company2?', 'What will be their total turnover then?']
['Coronavirus was originated in china.', 'After that it is spreading all over the world.', 'To prevent it everyone has to take care of cleanliness and prefer vegetarians.']

It is true that these rule based tokeniser do suffer from several issues (see: https://github.com/nltk/nltk/issues/494 ). If you are specifically working on a sequential model for sentence slitting then an LSTM with attention mechanism would surely be a good choice, you can find many repositories on GitHub which implement the code of the original paper 'attention is all you need' (among others: https://github.com/kaushalshetty/Positional-Encoding )

Improve this answer
$\endgroup$
2
  • $\begingroup$ thanks for reply. But it failed for 1st sentence itself. 1st sentence is actually 2 sentences separated by "and". For the PositionalEncoder link you shared, can you please elaborate how that is related to this problem? $\endgroup$
    – Sandeep Bhutani
    Commented Mar 10, 2020 at 8:21
  • $\begingroup$ The code is just a replica of 'attention is all you need', the paper that introduced the attention mechanism. I linked it since you were asking for some code references. About the first sentence, it is not composed by two sentences, but by two subordinates. If you want to split different sentences and also sentences into subordinates what you have to look into are dependencies. If you elaborate a bit better on the outcome you would like to get I could check if what you want is doable with nltk or SpaCy. $\endgroup$
    – Edoardo Guerriero
    Commented Mar 10, 2020 at 11:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.