0
$\begingroup$

I work with texts where there is a dialogue between two people (a client and a call center employee, the beginning and end of each person’s phrase is not defined). My goal is to classify texts in which a call center employee names words from my list. If the texts are manually marked up, can such a classification problem be solved? Are there any tricks to solve this type of problem?

Sample data: "hello hello my name is Sam Chin I'm calling for pizza delivery Okay now check your order wait a minute Sam"

$\endgroup$
1
$\begingroup$

There's a lot of ways to do this, one approach is to use token-based matching. You can use this to easily find any "tokens" in the text, like names, places, or just plain words.

Methodology

I'd recommend using Rule-based Entity Recognition in spaCy. You'll define the "rules" of what the entity looks like, here's the example from the docs where we define the following patterns to find:

  • An entity type of Organization and the word Apple
  • An entity type of Location and the words san and francisco

Here's that in code (live example):

from spacy.lang.en import English
from spacy.pipeline import EntityRuler

nlp = English()
ruler = EntityRuler(nlp)

# These are the rules you define, look at the docs to see what your options are.
# You don't have to use the "label", you can just look for a "pattern" if you want.
patterns = [{"label": "ORG", "pattern": "Apple"},
            {"label": "GPE", "pattern": [{"LOWER": "san"}, {"LOWER": "francisco"}]}]

ruler.add_patterns(patterns)
nlp.add_pipe(ruler)

# The text you're searching through to find your patterns
doc = nlp("Apple is opening its first big office in San Francisco.")

# This prints out the matches
print([(ent.text, ent.label_) for ent in doc.ents])

The output of this code is: [('Apple', 'ORG'), ('San Francisco', 'GPE')]

Usage

Thankfully spaCy has some fantastic online tools for helping you write your patterns, I highly recommend you check these links out.

  1. Install spaCy
  2. Evaluate if you should use rules or a model (I suggest rules but I could be wrong), and if you should use token matcher or phrase matcher
  3. Read the documentation to determine how to write your patterns
  4. Test your patterns using the Live Rule-based Matcher Explorer
  5. Use the code snippet above or the code samples in the docs to see how to use your new patterns
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.