Use text similarity (cosine) instead of machine learning to classify companies into industries

Question

I am building an industry classifier. I.e. classifying companies into industries based on a company's description. Each company can only have one industry.

I took 2000 companies and assigned them into 15 industries. Then I trained a few models based on the company's description. All of them performed extremely bad.

So I wanted to get more data (indicated by the learning curves), i.e. more companies. But getting the classification right is difficult. Thus, I decided to try a simple similarity calculation.

For each industry I select words that fit best to this industry. Then I calculate the similarity between each industry and a new company and select the one that fits the best.

For the three industries I tested it worked great. But I am unsure if this will makes sense down the road. I think that the similarity approach is great because:

• I can clearly define (from a business perspective) what is what
• It scales nicely

Can anybody tell me if this approach makes sense? All other classification system are build using some model, so I am unsure.

EDIT: I want to build an industry classifier that can classifier "all" existing companies. By all I mean companies that fit the NACE industries (NACE is the industry standard classification system used in the European Union)

The 2000 companies that I used are arbitrary. They do not represent all the industries from NACE. The 15 industries I ended up with are mostly based on NACE. However, I grouped some of them together. Also I made up some industries based on the most frequent keywords across certain companies (sometimes that is very obvious. Example: "dating companies" like Tinder).

The company descriptions come directly from the company (mostly parsed from their website).

I have tried to cluster the companies but I didn't find any combination of clusters that make sense based on NACE.

Yes, I have filtered stopwords (and applied other normalization such as only keeping text characters and stemming).

Regarding objectivity (thanks to @skiddles): I think this is actually another pro-argument for similarity. If I extract all the keywords that are used to describe the industries in the NACE specification I end up (it seems) with at least one recognized standard.

So, I believe, my question should be: I have 20 industries. Each industry is described with 100 distinct words. I get a new company with a description. The words of this company description can be found in the descriptions for the industries. Should I classify the companies into industries based on similarity or use a classification model?

Answer 1

This is an interesting approach. For me though, it raises a few questions that might impact the effectiveness of the approach:

1. Were all the descriptions created by the same organization, or performed for the same purpose? e.g. Descriptions are taken from SEC filings.
2. Have you tried any unsupervised approaches to grouping these companies based on their descriptions?
3. How did you come up with 15 industries, not 14, not 16?
4. Have you applied stop words and removed very infrequent words?

Not being critical, but how confident are you that someone else would apply the same label to each of the 2000 instances you labeled? Could that be one reason why the models are performing poorly?

Ultimately, to answer your question, there is no reason why domain knowledge cannot be used to inform your key words, but it seems like you will struggle when you get to industries that are "similar but different." The good results that you have received using this method may just be indicative that you created both the labels, and the key words for the industries, which may well be what you want to achieve.

From a defense perspective, your classifier would only represent your opinion. In the long run, you may be better off giving up a bit of performance to gain objectivity. Company classification is difficult because of vertical and horizontal integration, and the practice of using holding companies to manage diversified businesses. If you can take your opinion out of the process, it may be "better". If someone disagrees with your model, they are just disagreeing with you. If they disagree with a more traditional ML model, they are disagreeing with the dispassionate similarity of the descriptions.

HTH

Answer 2

I will leave the bigger questions for wiser folks, but on the data science side, your on the right track for the reasons you state.

Cosine Similarity is a good solution for classifying sparse information sets.

Words are a sparse information set to start with.

This appears to be labeling summary data. So a single, or few, terms against summary data is sparse on sparse.

Done this more than a few times, it will be hard for you to find a better model than Cosine Simularity here.