0
$\begingroup$

I'm trying to find similar data columns (text and numerical) in other tables by using classification in machine learning. What are some good features that can represent a column for my feature space ?

$\endgroup$
  • 2
    $\begingroup$ Can you elaborate your question with an example? $\endgroup$ – Ankit Seth Feb 26 '18 at 8:19
  • $\begingroup$ What language, data structure are you using? $\endgroup$ – user2505650 Feb 26 '18 at 12:40
  • $\begingroup$ Let say I have column C which have plant length values from a particular plant and column D which have values for the price of Milk from a certain supermarket. How do I recognize similar data columns that share similar column characteristics ? What features would I use ? I'm using machine learning with Python $\endgroup$ – DPascal Feb 26 '18 at 14:20
1
$\begingroup$

If I'm not mistaken, you are looking for some metrics to investigate if two columns of values represent similar information. Right?

When the order of numbers is fixed for both columns (such that the ith elements of both columns relate to the same record) and hence both columns have the same length, then you can use a wide range of techniques to measure the similarity/distance between the two vectors such as cosine similarity, Pearson correlation coefficient, mutual information, etc.

If the order of numbers in the columns are different or they basically refer to different concepts, then the above techniques are meaningless. In this case, you can look at these columns as two populations of samples. In this case, you may compare their statistical distributions using their distribution parameters (eg mean and std for a normal distribution) or statistical tests (eg ANOVA). In some cases, the number of items (population size) can be different.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.