approach to classify text with natural language processing methods

Question

I have a problem with regards to text classification/categorization. The task is bugging me for days already and as I am pretty new to AI and the field of natural language processing (NLP) I am just overwhelmed by the content online and available tools/libraries (e.g. NLTK, Keras, spaCy, etc.). It would be awesome if you could give me some guidance or clues on how you would approach the problem.

Issue: basicially I try to set up a tool for classifying text. I already have an extensive labeled dataset to work with. The input will always be a list of some sort (think of an Excel file with 500 rows). Each row contains a single word or a combination of words, i.e. no sentences.

A simplified example of my labeled dataset - input on the left, classification on the right:

"dog" -> "animal"
"dog owner" -> "person"
"dog owner house" -> "building"
"owner" -> "person"
"dog food" -> "food"
"food court" -> "building"

My existing labeled dataset has around 2,000 of these classifications with in total 50 unique categories. How can I set up an algorithm that scans the input for example for the word "dog" - if it is only "dog" then it is the category "animal", if it is "dog" and "owner" it is the category "person", if it is "dog", "owner" and "house" it is the category "building" and so on.

If I set up a ton of if-else-statements as a decision tree it is just cumbersome and intransparent. Is there a way with NLP to solve such an issue?

Thank you very much in advance! Looking very much forward to your ideas and please let me know if I have to be more specific in any way.

Best regards, pythoneer

Answer 1

This problem seems like a multi-class multi-label problem. The questioner seems to be comfortable in building a detailed ontology. These lead the author to propose the following approach. Please note that a detailed explanation of this can be found in an article here.

---

Steps to solve the problem:

1. Build a taxonomy file as a csv file as given below. Please note, the column headings should be identical to whats given below.
2. Put all your content in another csv file that looks like below. Please note, the column headings should be identical to whats given below.
3. In the following python code, please enter the path to content in the path to df and path to taxonomy in the path to df_tx. These steps are present near the comment import data for mapping. Add another path value for the output towards the end of the code.

---

Run the python code below. Please note that this code runs fine on Python 2.7 in Windows 10 machine. Please iron out any technical issues yourself as the author may not be of much help for such issues.

#Invoke Libraries
import pandas as pd
import numpy as np
import re

#import data for mapping
df = pd.read_csv("path to content csv");
df_tx = pd.read_csv("path to taxonomy csv");

#Build functions
#function that identifies taxonomy words ending with (*) and treats it as a wild character
def asterix_handler(asterixw, lookupw):
    mtch = "F"
    for word in asterixw:
        for lword in lookupw:
            if(word[-1:]=="*"):
                if(bool(re.search("^"+ word[:-1],lword))==True):
                    mtch = "T"
                    break
    return(mtch)

#function that removes all punctuations. helps in creation of set of words
def remov_punct(withpunct):
    punctuations = '''!()-[]{};:'"\,<>./?@#$%^&*_~'''
    without_punct = ""
    char = 'nan'
    for char in withpunct:
        if char not in punctuations:
            without_punct = without_punct + char
    return(without_punct)

#function to remove just the quotes(""). This is for the taxonomy
def remov_quote(withquote):
    quote = '"'
    without_quote = ""
    char = 'nan'
    for char in withquote:
        if char not in quote:
            without_quote = without_quote + char
    return(without_quote) 

#split each document by sentences and append one below the other for sentence level categorization and sentiment mapping
sentence_data = pd.DataFrame(columns=['slno','text'])
for d in range(len(df)):    
    doc = (df.iloc[d,1].split('.'))
    for s in ((doc)):        
        temp = {'slno': [df['slno'][d]], 'text': [s]}
        sentence_data =  pd.concat([sentence_data,pd.DataFrame(temp)])
        temp = ""

#drop empty text rows and export data
sentence_data['text'].replace('',np.nan,inplace=True);      
sentence_data.dropna(subset=['text'], inplace=True);  

data = sentence_data
cat2list = list(set(df_tx['Category2']))
data['Category'] = 0
mapped_data = pd.DataFrame(columns = ['slno','text','Category']);
temp=pd.DataFrame()

for k in range(len(data)):        
    comment = remov_punct(data.iloc[k,1])
    data_words = [str(x.strip()).lower() for x in str(comment).split()]
    data_words = filter(None, data_words)
    output = []

    for l in range(len(df_tx)):
        key_flag = False
        and_flag = False
        not_flag = False
        if (str(df_tx['Keywords'][l])!='nan'):
            kw_clean = (remov_quote(df_tx['Keywords'][l]))
        if (str(df_tx['AndWords'][l])!='nan'):
            aw_clean = (remov_quote(df_tx['AndWords'][l]))
        else:
            aw_clean = df_tx['AndWords'][l]
        if (str(df_tx['NotWords'][l])!='nan'):
            nw_clean = remov_quote(df_tx['NotWords'][l])
        else:
            nw_clean = df_tx['NotWords'][l]
        Key_words = 'nan'
        and_words = 'nan'
        and_words2 = 'nan'
        not_words = 'nan'
        not_words2 = 'nan'

        if(str(kw_clean)!='nan'):
            key_words = [str(x.strip()).lower() for x in kw_clean.split(',')]
            key_words2 = set(w.lower() for w in key_words)

        if(str(aw_clean)!='nan'):
            and_words = [str(x.strip()).lower() for x in aw_clean.split(',')]
            and_words2 = set(w.lower() for w in and_words)

        if(str(nw_clean)!= 'nan'):
            not_words = [str(x.strip()).lower() for x in nw_clean.split(',')]
            not_words2 = set(w.lower() for w in not_words)

        if(str(kw_clean) == 'nan'):
            key_flag = False        
        else:
            if set(data_words) & key_words2:
                key_flag = True
            elif(bool(re.search('"',df_tx['Keywords'][l]))==True and quote_handler(key_words, comment) == 'T'):
                key_flag = True            
            elif(asterix_handler(key_words2, data_words)=='T'):                
                    key_flag = True   

        if(str(aw_clean)=='nan'):
            and_flag = True
        else:
            if set(data_words) & and_words2:
                and_flag = True
            elif(bool(re.search('"',df_tx['AndWords'][l]))==True and quote_handler(and_words, comment) == 'T'):
                and_flag = True            
            elif(asterix_handler(and_words2, data_words)=='T'):
                and_flag = True

        if(str(nw_clean) == 'nan'):
            not_flag = False
        else:
            if set(data_words) & not_words2:
                not_flag = True
            elif(bool(re.search('"',df_tx['NotWords'][l]))==True and quote_handler(not_words, comment) == 'T'):
                not_flag = True            
            elif(asterix_handler(not_words2, data_words)=='T'):
                not_flag = True

        if(key_flag == True and and_flag == True and not_flag == False):
            output.append(str(df_tx['Category2'][l]))            
            temp = {'slno': [data.iloc[k,0]], 'text': [data.iloc[k,1].strip()], 'Category': [df_tx['Category2'][l]]}
            mapped_data = pd.concat([mapped_data,pd.DataFrame(temp)], sort = False)

#output mapped data
mapped_data = mapped_data[['slno', 'text', 'Category']]   

mapped_data.to_csv("Path here/mapped_data.csv",index = False)               

Final output looks like this:

Answer 2

You are trying to map hyponyms (words for specific examples of a thing, e.g. dog) to hypernyms (word’s for a general class of things, e.g. animal). This has probably already been done for you for most of your terms in Wordnet, so that’s the place to start if you want to quickly create a solution e.g. for commercial purposes.

If this is something you want/need to create your own solution for, here are three suggestions. You could pick one to use as a baseline and try to improve on it:

• Because of the way the English language works, you could probably get a long way by discarding everything except the final word within reach training example. You can then take the GloVe embedding of that word and feed it to a small feed forward neural net.
• You could take a generative pertained language model and feed it a dummy sentence fragment including the phrase you want to classify (e.g. “dog house” —> “a dog house is a type of”), and then train on some embedding of the predicted next word.
• You could use the [hashing trick]( to embed all your example phrases and then train a linear model from scratch. Never underestimate a linear model!

Good luck!

Answer 3

The first thing I would try to do is to simply use one-hot encoding to represent you're features. One-hot encoding consists in representing sentences (sequences of words in your case) as a sparse vector, if you're familiar with python sklearn has this function already implemented, it's called DictVectorizer

# the length is the total amount of different words in all your words sequences
"dog owner" --> [0, 0, 0, 0, 1, 0, 0, 1, ....]

I would then train several models (random forest, naive bayes, multi layer perceptron, support vector machine, etc.) to check what model works best. Usually with sparse features, svm works well, but it does mean that also in your case they would provide the highest results, that's why the only way to know is to train several models.

A more advanced technique (not in terms of coding though) would be using embedding vectors and deep learning. You could use pre-trained vectors, GloVe vectors are a standard choice, as an input for a Convolutional Neural Network (this architecture usually works pretty well with short texts and in classification tasks, and it's fast to train).

As a final consideration, from what I see on your example it seems that the label is related only to the last word of the sequence. If this is consistent across all dataset, another trick that might avoid completely deep learning could be to still use embedding vectors, but only to compute some semantic similarity scores, like cosine similarity. If the labels are semantically different enough, it might be possible to predict the label only by computing the similarity between the final word of each sequence and each label to then select the label with the highest score.