0
$\begingroup$

I have been working on a problem but unable to make substantial progress so wanting some insights/advice.

I have a large set of tables (CSV files) having only column names. (Column values are not present.)

For example:

Users.csv --> ID, FirstName, LastName, City, Address, CardID
Usr.csv --> Residential, Name, State, UserID

I want to cluster files (tables) which represent similar entities. In the example above, both represent a person entity and hence should be clustered together.

Current Approaches: I tried coming up with a similarity score b/w two tables which is based on the overlapping columns b/w the two tables and similarity b/w their table names

sim_score = alpha * (column similarity) + beta * (table name similarity)
where 
column similarity is a fraction of possible columns matching to a minimum 
number of columns from both table.
possible columns match is computed by generating features of each column like jaccard_3gram, 
tf-idf score etc. and thresholding the similarity to arrive at similarity score.

table name similarity is computed by computing the dice coeff. of the two tables names

Once sim_score is computed for all table pairs, then creating a weighted undirected graph and running a clustering algorithm (tried Markov, Girvan) to get the required clusters.

This approach is not working well when there are semantically related column names b/w two tables. I have been trying approaches around this thought process.

Is there an alternative to how this problem can be looked at? Any standard approaches in ML/NLP to tackle this. Any leads are helpful. Thanks

$\endgroup$

1 Answer 1

1
$\begingroup$

It looks like Schema's are ideas with variable descriptions. One schema can relate three-entities and another, related schema could involve the same entitities and an additional entity. Your clustering method should accommodate data with such a characteristic. Representing your schema's as graphs is one way to achieve this. Each schema node is defined by a set of attributes. The relationship between a pair of schemas is determined by a similarity metric such as the Jaccard similarity metric. You could then use a graph clustering algorithm to cluster the graphs.

Another way to think of this is to represent the schema as a document with the union of the terms in the schema being the vocabulary and then use document-clustering or topic analysis to find related schema's.

$\endgroup$
1
  • $\begingroup$ As it’s currently written, your answer is unclear. Please edit to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers in the help center. $\endgroup$
    – Community Bot
    Jan 31, 2022 at 18:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.