1
$\begingroup$

I have 3 different datasets with company information, in all of them I have company name, but is not perfect: For example:

  • Dataset A: Company name: Facebook
  • Dataset B: Company name: Facebook, Inc
  • Dataset C: Company name: facebook

Some other signals like company url exists, but in terms of name matching wondering if text similarity is a good approach for this grouping problem.

$\endgroup$
2
$\begingroup$

It seems like you want to want to first get the datasets together using as shared key which in this case is the company name. That does not sound like a modelling problem but a data wrangling problem.

Assuming that each datasets are in csv, then get them to pandas dataframe:

df_a = pd.read_csv('data_set_a.csv')

Then assuming you have the datasets in df_a, df_b, df_c and each have a column 'name':

df_a.name = df_a.name.str.lower()
df_a.name = [i.split()[0] for i in df_a.name]

Now you will have a dataset where all three are 'facebook'. Then the three dataframes can be merged in to one:

df = pd.merge(df_a, df_b, left_on='name', right_on='name')
df = pd.merge(df, df_c, left_on='name', right_on='name')

And now you have the three different datasets merged, and ready for analysis.

$\endgroup$
2
$\begingroup$

You can consider string similarity based on edit distance. eg: Levenshtein distance

However, you also need to compute this with respect to prefixes of the strings, and not only the full strings. (eg: to compare "Facebook" with "Facebook Inc."). Also normalize the capitalization of the strings before comparison (make them all lowercase, etc.), and maybe remove punctuation as well.

Experiment with a few of the above, see if the results make sense, and fine tune the method accordingly. Also using other metadata will help. URL and context/topic words/entities occurring around the company name will be associated with the company, and can help find the right groupings. This will be more involved though, so you may want to attempt this later on, if the simpler approaches do not suffice.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.