I have been working on a problem but unable to make substantial progress so wanting some insights/advice.
I have a large set of tables (CSV files) having only column names. (Column values are not present.)
For example:
Users.csv --> ID, FirstName, LastName, City, Address, CardID
Usr.csv --> Residential, Name, State, UserID
I want to cluster files (tables) which represent similar entities. In the example above, both represent a person entity and hence should be clustered together.
Current Approaches: I tried coming up with a similarity score b/w two tables which is based on the overlapping columns b/w the two tables and similarity b/w their table names
sim_score = alpha * (column similarity) + beta * (table name similarity)
where
column similarity is a fraction of possible columns matching to a minimum
number of columns from both table.
possible columns match is computed by generating features of each column like jaccard_3gram,
tf-idf score etc. and thresholding the similarity to arrive at similarity score.
table name similarity is computed by computing the dice coeff. of the two tables names
Once sim_score is computed for all table pairs, then creating a weighted undirected graph and running a clustering algorithm (tried Markov, Girvan) to get the required clusters.
This approach is not working well when there are semantically related column names b/w two tables. I have been trying approaches around this thought process.
Is there an alternative to how this problem can be looked at? Any standard approaches in ML/NLP to tackle this. Any leads are helpful. Thanks