# Efficiently finding most similar sports from large dataset

As part of a past interview task, I’m working with a sports streaming dataset that looks like this:

pd.DataFrame({'away_contestant_country': {0: 'Japan', 1: 'Canada'},
'competition_name': {0: 'NPB', 1: 'FIBA AmeriCup (W)'},
'customer_country': {0: 'Japan', 1: 'Canada'},
'device_category': {0: 'Web', 1: 'Unknown'},
'home_contestant_country': {0: 'Japan', 1: 'Brazil'},
'live_or_on_demand': {0: 'Live', 1: 'Live'},
'match_date': {0: '2021-06-11T08:45:00.000Z', 1: '2021-06-13T19:10:00.000Z'},
'match_title': {0: 'Lions v Dragons', 1: 'Brazil vs. Canada'},
'sport_name': {0: 'Baseball', 1: 'Basketball'},
'stream_end_time': {0: '2021-06-11T12:02:18.000Z',
1: '2021-06-13T20:14:05.000Z'},
'stream_start_time': {0: '2021-06-11T09:41:37.000Z',
1: '2021-06-13T20:13:36.000Z'},
'tournament_end_date': {0: '2021-11-29T00:00:00.000Z',
1: '2021-06-19T00:00:00.000Z'},
'tournament_start_date': {0: '2021-03-02T00:00:00.000Z',
1: '2021-06-11T00:00:00.000Z'},
'venue_country': {0: 'Japan', 1: 'Puerto Rico'},
'viewer_id': {0: 5049, 1: 9062}})


where every instance is a user streaming an event. After extracting some features such as stream length, the data is mixed between numerical and categorical types. Our dataset has ~37k rows and 15 features.

My objective is:

1. To find the N most similar and dissimilar sports from the data.

2. To build a reusable function that can take a sport name as input and return the top N most similar sports.

I have tried using cosine similarity and Gower similarity to compute the similarity between every instance and averaging the scores for each sport, but I was told in my submission feedback that comparing each stream to every other stream would become incredibly computationally expensive as the dataset size increased. While I agree, I cannot figure out an appropriate way to find similarity between sports without some kind of similarity matrix.

I’ve also tried taking an ‘average’ vector of the input sport, which would have the median numerical feature values and the mode categorical feature values, and using this to compute similarity between all other instances. For example, making an average Rugby stream instance and then comparing it to all other sports streaming instances. While much faster, this does not provide totally satisfactory results.

This can’t be a classification problem because the desired function is simply given a sport name as input. I’ve considered clustering algorithms but I’m unsure how to proceed with the generated clusters to find similar sports.

Is this task feasible for large datasets or is there an easier/more efficient way to achieve my objectives?

## 1 Answer

If I understood your problem correctly, I would suggest the following:

1. Clean your data first
2. Encode categorical features
3. Run dimensionality reduction algorithm (such as PCA)

From this point you can start analysis. You could use k-means clustering as a baseline given that the number of clusters would be less than the number of sports. Or you can simply use Euclidean distance.

• Thanks for the suggestion. However, I’m uncertain how I could use these clusters to find the top N most similar sports when given a sport name as input. Nov 10, 2021 at 8:07
• 1. Encode the new data points 2. Predict the cluster for each data point 3. Get the existing points from that cluster 4. Calculate distance metric from new data point to each of the cluster points 5. Sort using the distance as key 6. Look through the array and fill the similar sports set 7. As soon as lenghth of the array is N - stop and return result. Nov 10, 2021 at 11:04
• This would work if there was more information given for the new input data; however, in this situation the end function is only taking sport name as input (e.g ‘Golf’) and we are supposed to return the most similar sports to Golf based on our dataset. I cannot predict what cluster the word ‘Golf’ will belong to because there are no additional values given for the other features. Nov 10, 2021 at 11:31
• Straightforward approach. For each cluster compute the occurrence of each sport, drop sports with zero occurrence. For each sport assign cluster based on occurrence. When the input comes, search the corresponding cluster for N different sports ordered by the occurrence. Nov 17, 2021 at 13:12