Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
So I have a database of web-scraped cars, and I want to find the similarity between cars based on the km driven (e.g 69000), the model year (e.g 2012), and the trim of the car which will be one of three categories: base, mid, top.
What similarity measure can I use that will give me a decently accurate similarity ratio
I suggest you use a weighted per-attribute similarity, for instance let a and b a pair of tuples representing the attributes of some car A and another car B. For example:
a = (69000, 2012, base)
b = (71000, 2013, base)
For the km driven I would use Skm = 1 / (1 + |km(a) - km(b)|), so:
Skm = 1 / (1 + |69000 - 70000|) = 1 / 1001 = 0.0009
For the case of the year the same function could be use, then:
Syear = 1 / (1 + |2012 - 2013|) = 1 / 2 = 0.5
Last, for the trim of the car I would use a function that returns 0 if the categories were different 1 otherwise.
Strim = (base == base) = 1
Finally,
S = Wkm * Skm + Wyear * Syear + Wtrim * Strim
Where Wkm + Wyear + Wtrim = 1.0 and I would adjust the values according to what you think is decent accurate similarity ratio. Setting the values to be Wkm = 0.6, Wyear = 0.2, Wtrim = 0.2 the similarity S(a, b) = 0.30054. This similarity has the nice property that tuples with the same attributes yield a value of 1.
A good practice is to normalize the values of each attribute so you don't assign low values to any attribute for instance you could divide the columns of the km and year attributes for the maximum value, therefore for the pair of tuples in the example you will have the following transformed tuples:
a' = (0.985, 0.99, base)
b' = (1, 1, base)
Now the similarity using the same weights is S(a, b) = 0.9. Alternatively you could use a similarity matrix for the trim of the car, this is base could be more similar to mid thant to top. For example:
cat | base | mid | top |
-------------------------
base | 1 | 0.5 | 0.2 |
-------------------------
mid | 0.5 | 1 | 0.7 |
-------------------------
top | 0.2 | 0.7 | 1 |
The values of the matrix have to be adjusted to fit your needs. For a more in depth comparison between similarity functions between categorical attributes (trim) see this for a comparison of different similarity measures for continuous attributes (year, km) see this.
Again to see which similarity is the best will depend on the data and on your perception, I would select a random set of pairs of cars and manually assign a similarity. Then split this set into a train set and test set. Use the train set to adjust the parameters of the different possible similarities, in the case of my proposal Wkm, Wyear and Wtrim. Then once your satisfied evaluate on the test set.
