Efficiently finding most similar sports from large dataset

Question

As part of a past interview task, I’m working with a sports streaming dataset that looks like this:

pd.DataFrame({'away_contestant_country': {0: 'Japan', 1: 'Canada'},
 'competition_name': {0: 'NPB', 1: 'FIBA AmeriCup (W)'},
 'customer_country': {0: 'Japan', 1: 'Canada'},
 'device_category': {0: 'Web', 1: 'Unknown'},
 'home_contestant_country': {0: 'Japan', 1: 'Brazil'},
 'live_or_on_demand': {0: 'Live', 1: 'Live'},
 'match_date': {0: '2021-06-11T08:45:00.000Z', 1: '2021-06-13T19:10:00.000Z'},
 'match_title': {0: 'Lions v Dragons', 1: 'Brazil vs. Canada'},
 'sport_name': {0: 'Baseball', 1: 'Basketball'},
 'stream_end_time': {0: '2021-06-11T12:02:18.000Z',
  1: '2021-06-13T20:14:05.000Z'},
 'stream_start_time': {0: '2021-06-11T09:41:37.000Z',
  1: '2021-06-13T20:13:36.000Z'},
 'tournament_end_date': {0: '2021-11-29T00:00:00.000Z',
  1: '2021-06-19T00:00:00.000Z'},
 'tournament_start_date': {0: '2021-03-02T00:00:00.000Z',
  1: '2021-06-11T00:00:00.000Z'},
 'venue_country': {0: 'Japan', 1: 'Puerto Rico'},
 'viewer_id': {0: 5049, 1: 9062}})

where every instance is a user streaming an event. After extracting some features such as stream length, the data is mixed between numerical and categorical types. Our dataset has ~37k rows and 15 features.

My objective is:

1. To find the N most similar and dissimilar sports from the data.

2. To build a reusable function that can take a sport name as input and return the top N most similar sports.

I have tried using cosine similarity and Gower similarity to compute the similarity between every instance and averaging the scores for each sport, but I was told in my submission feedback that comparing each stream to every other stream would become incredibly computationally expensive as the dataset size increased. While I agree, I cannot figure out an appropriate way to find similarity between sports without some kind of similarity matrix.

I’ve also tried taking an ‘average’ vector of the input sport, which would have the median numerical feature values and the mode categorical feature values, and using this to compute similarity between all other instances. For example, making an average Rugby stream instance and then comparing it to all other sports streaming instances. While much faster, this does not provide totally satisfactory results.

This can’t be a classification problem because the desired function is simply given a sport name as input. I’ve considered clustering algorithms but I’m unsure how to proceed with the generated clusters to find similar sports.

Is this task feasible for large datasets or is there an easier/more efficient way to achieve my objectives?

Answer 1

If I understood your problem correctly, I would suggest the following:

1. Clean your data first
2. Encode categorical features
3. Run dimensionality reduction algorithm (such as PCA)

From this point you can start analysis. You could use k-means clustering as a baseline given that the number of clusters would be less than the number of sports. Or you can simply use Euclidean distance.