I have a large reference data set and everyday I get new data points. I would like to know how can I test if those new data points are similar to the reference set?
Thanks in advance.
Maria
2
-
$\begingroup$ There are ways to make comparisons, but it will depend on the type of data you have. Can you add some information describing the data? For example: images, numbers, sentences, sound? $\endgroup$– n1k31t4Commented Jul 20, 2019 at 9:51
-
$\begingroup$ yes sure, is numeric data... :) $\endgroup$– MariaKKCommented Jul 20, 2019 at 10:04
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
2
If you have many features, likely meaning many columns in your table of data, then you could try clustering. Something as simple as k-Nearest Neighbours could work nicely.
You would first fit a model using your available data, then look at the resulting clusters. Each cluster will represent combinations of your features.
Next you can put new data points into the model and it will tell you to which cluster it would best fit.
This would actually be a fully fledged predictive model! It is unsupervised learning, because you are not using and pre-existing labels (for each cluster in this case).
If you are working with Python, the Sci-kit Learn documentation is a good place to start. There are many other clustering algorithms, which might work better for you, depending on your type of data (if it is spatial, DBSCAN is good, for example).
-
$\begingroup$ I think it's a good idea but OP would have to use the resulting similarity score against the clusters in order to determine how similar a fresh instance is compared to the training set. I would imagine taking the similarity with the closest cluster as a measure of how close the instance is to the training data instances. Otherwise obtaining the closest cluster doesn't say how close it is. $\endgroup$– ErwanCommented Jul 21, 2019 at 0:49
-
$\begingroup$ Thank you very much! That could work for my data :) I was wondering if exists some kind of index that with some p value will tell me that the new dataset is similar to the reference set? Both new and old data have the same distribution. $\endgroup$– MariaKKCommented Jul 21, 2019 at 6:22