Let’s say I am modelling data on flights and delays. I have 14 airlines (carrier variable) in the data below. I would like to know if I can create fewer categories or clusters of this categorical data, so instead of having 14 airlines maybe end up with 3 or 4 clusters of airlines.

# Data takes ~30 seconds to load
flights <- as.data.table(read.csv(url("https://raw.githubusercontent.com/wiki/arunsrinivasan/flights/NYCflights14/flights14.csv")))

km <- kmeans(flights[,.(dep_delay,carrier)], centers = 4, nstart = 1)

Is there any methodology that could be used? I tried k-means but it seems it is not the most appropriate.

  • 1
    $\begingroup$ Recategorising the categorical data as numerical wouldn't work as the 14 carriers cannot be split (e.g. observations from American Airlines should all go to the same category) $\endgroup$ – user3507584 Oct 17 '17 at 14:59
  • $\begingroup$ Why not use a clustering algorithm with a modified metric? You can regard the 14 carriers as 14 objects. Then you just have to define distance between the objects. For instance, you could use the median distance between datapoints in the objects. $\endgroup$ – Dave Kielpinski Oct 17 '17 at 16:44

Without looking at the data, can't you just use your Domain Knowledge to create meaningful groups/attributes for the airlines data frame? Such as "low-cost carrier/premium", "foreign (non-USA) airline/US-Airline" , "long-distance carrier/domestic only", "founded in decade" , or whatever? With only 14 airlines you can assign appropriate values yourself, manually, without relying on some cryptic algorithm.

Besides, is your dataset identical to the dataset in the R package "nycflights13"?

If so, you could easily load it with library(nycflights13), after installing the package of course.

  • $\begingroup$ The flight data is a MWE. I need a solution based on the data, not on my expert judgement. $\endgroup$ – user3507584 Oct 19 '17 at 10:13
  • $\begingroup$ Well, in your kmeans() call you have set the number of clusters to 4 - thus, also relying on your expert judgement, implicitly, I think. $\endgroup$ – knb Oct 19 '17 at 12:44
  • $\begingroup$ I again say this is just a MWE. Ideally you would use your kmeans with 2,3,4,...n number of clusters and test which one works better. $\endgroup$ – user3507584 Oct 19 '17 at 12:47

With just 14 airlines, the clustering algorithm of choice is of course hierarchical clustering. But you will need to develop a good similarity measure.

Why don't you use e.g. airline alliances instead?

  • $\begingroup$ The airline is a MWE. I have a completely different data, just with the same problem and the idea is to use a clustering technique without using previous knowledge (e.g. existing airline alliances). $\endgroup$ – user3507584 Oct 23 '17 at 14:36
  • $\begingroup$ Try HAC nevertheless. $\endgroup$ – Has QUIT--Anony-Mousse Oct 23 '17 at 18:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.