I have a set of text examples I need to learn as class A, and they are of varying lengths, say 10 sentences to 1 sentence long. I have to parse a document to find those strings of text that match one of the example texts as an example of class A.

What is the right/best/common way to go about handling the length of text variation of the examples vs how much text you use to compare to in a document? That is, do I just first go through the document 10 sentences at a time (windowing) and running a prediction on each group of 10 (since some training examples were 10 sentences long), and then do it again for 9 sentence long groups, etc., down to 1?

Or is there something already handles arbitrary text length?

  • 1
    $\begingroup$ Welcome to DataScienceSE. Depends what is the target application but in general it's not desirable to have 10 possible answers, one for each possible length. So in general I'd say that the system has to manage the varying length. I think a common way is to separate the sentences: for every sentence find the most similar 'class A' document. $\endgroup$
    – Erwan
    Dec 9, 2021 at 23:52
  • $\begingroup$ Hmmm. Basically, that is sort of what I've decided to do/try. Rather than having groups of sentences of varying length, I've just broken all of them into single sentences. Then I just train on the examples as single sentences, then go through a doc one sentence at a time to check the prediction on each individual sentence. $\endgroup$
    – superqd
    Dec 11, 2021 at 5:09


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.