Airline Fares - What analysis should be used to detect competitive price-setting behavior and price correlations?

Question

I want to investigate price-setting behavior of airlines -- specifically how airlines react to competitors pricing.

As I would say my knowledge about more complex analysis is quite limited I've done mostly all basic methods to gather a overall view of the data. This includes simple graphs which already help to identify similar patterns. I am also using SAS Enterprise 9.4.

However I am looking for a more number based approach.

Data Set

The (self) collected data set I am using contain around ~54.000 fares. All fares were collected within a 60 day time window, on a daily basis (every night at 00:00).

Hence, every fare within that time window occurs $n$ times subject to the availability of the fare as well as the departure date of the flight, when it is passed by the collection date of the fare. (You can't collect a fare for a flight when the departure date of the flight is in the past)

The unformatted that looks basically like this: (fake data)

+--------------------+-----------+--------------------+--------------------------+---------------+
| requestDate        | price| tripStartDeparture | tripDestinationDeparture | flightCarrier |
+--------------------+-----------+--------------------+--------------------------+---------------+
| 14APR2015:00:00:00 | 725.32    | 16APR2015:10:50:02 | 23APR2015:21:55:04       | XA            |
+--------------------+-----------+--------------------+--------------------------+---------------+
| 14APR2015:00:00:00 | 966.32    | 16APR2015:13:20:02 | 23APR2015:19:00:04       | XY            |
+--------------------+-----------+--------------------+--------------------------+---------------+
| 14APR2015:00:00:00 | 915.32    | 16APR2015:13:20:02 | 23APR2015:21:55:04       | XH            |
+--------------------+-----------+--------------------+--------------------------+---------------+

"DaysBeforeDeparture" is calculated via $I=s-c$ where

• I & interval (days before departure)
• s & date of the fare (flight departure)
• c & date of which the fare was collected

Here is a example of grouped data set by I (DaysBeforeDep.) (fake data!):

+-----------------+------------------+------------------+------------------+------------------+
| DaysBefDeparture | AVG_of_sale | MIN_of_sale | MAX_of_sale | operatingCarrier |
+-----------------+------------------+------------------+------------------+------------------+
| 0               | 880.68           | 477.99           | 2,245.23         | DL           |
+-----------------+------------------+------------------+------------------+------------------+
| 0               | 904.89           | 477.99           | 2,534.55         | DL           |
+-----------------+------------------+------------------+------------------+------------------+
| 0               | 1,044.39         | 920.99           | 2,119.09         | LH               |
+-----------------+------------------+------------------+------------------+------------------+

What I came up with so far

Looking at the line graphs I can already estimate that several lines will have a high correlation factor. Hence, I tried to use correlation analysis first on the grouped data. But is that the correct way? Basically I try now to make correlations on the averages rather then on the individual prices? Is there an other way?

I am unsure which regression model fits here, as the prices do not move in any linear form and appear non-linear. Would I need to fit a model to each of price developments of an airline

PS: This is a long text-wall. If I need to clarify anything let me know. I am new to this sub.

Anyone a clue? :-)

Answer 1

Word of warning from a former airline Revenue Management analyst: you might be barking up the wrong tree with this approach. Apologies for the wall of text that follows, but this data is a lot more complex and noisy than might appear at first glance, so wanted to provide a short description of how it's generated; forewarned is forearmed.

Airline fares have two components to them: all the actual fares (complete with fare rules and what have you) that an airline has available for a certain route, most of which are published the Airline Tariff Publishing Company (a few special-use ones are not, but those are the exception rather than the rule) and the actual inventory management performed by the airline on a day-to-day basis.

Fares can be submitted to ATPCO four times a day, at set intervals, and when airlines do so, it will usually consist of a mixture of additions, deletions, and modifications of existing fares. When an airline initiates a pricing action (assuming their competitors aren't trying to make their own moves here), they usually have to wait until the next update to see if their competitors follow/respond. The converse goes when a competitor initiates a pricing action, as the airline has to wait until the next update before they can respond.

Now, this is all well and good with respect to fares, but the problem is that, because this is all getting published in ATPCO, fares are the next best thing to public information... all your competitors get to see what you've got in your arsenal, so attempts to obfuscate are not unheard of, such as publishing fares that will never actually be assigned any inventory, listing all the fares as day-of-departure, etc.

In many ways, the secret sauce comes down to the actual inventory allocation, i.e. how many seats on each flight will you be willing to sell for a given fare, and this information is not publicly available. You can get some glimpses by scraping web info, but the potential combinations of departure time/date and fare rules are quite numerous and may quickly escalate beyond your ability to easily keep track of.

Typically an airline will only be willing to sell a handful of seats for a very low fare and the people who snag those have to book quite far in advance lest the fare rules lock them out, or other travelers simply beat them to the punch. The airline will be willing to sell a few more seats for a higher fare, and so on and so forth. They will be quite happy to sell all of the seats for the highest fare they've got published, but this is not usually feasible.

What you're seeing with fares getting higher the closer you get to the day of departure is simply the natural process of having the cheap seats get booked farther out, while the remaining inventory gradually gets more expensive. Of course, there are some caveats here. The RM process is actively managed and human intervention is quite common as the RM team generally strives to meet its revenue goals and maximize revenue on each flight. As such, flights that fill up quickly may be "tightened up" by closing out low fares. Flights that are booking slowly may be "loosened up" by allocating more seats to lower fares.

There is a constant interplay and competition between airlines in this area, but you are not very likely to capture the actual dynamics just from scraping fares. Don't get me wrong, we had such tools at our disposal, and, despite their limitations, they were quite valuable, but they were just one data source that fed into the decision-making process. You'd need access to the hundreds, if not thousands of operational decisions made by RM teams on a daily basis, as well as state-of-the-world information as they see it at the time. If you cannot find an airline partner to work with in order to get this data, you might need to consider alternate data sources.

I'd recommend looking into getting access to O&D fare data from the Official Airline Guide (or one of their competitors) and try to use that for your analysis. It's sample-based (about 10% of all tickets sold) and aggregated at a higher level than would be ideal so careful route selection is imperative (I'd recommend something with plenty of airlines, flying non-stop multiple times a day, with large aircraft), but you may be able to get a better picture of what was actually sold (average fare) and how much of it was sold (load factor), vs. merely what is available for sale at a given point in time. Using that information you might be in better position to at least explore the outcomes of the airlines' pricing strategy, and make your inferences from there.

Answer 2

In addition to exploratory data analysis (EDA), both descriptive and visual, I would try to use time series analysis as a more comprehensive and sophisticated analysis. Specifically, I would perform time series regression analysis. Time series analysis is a huge research and practice domain, so, if you're not familiar with the fundamentals, I suggest starting with the above-linked Wikipedia article, gradually searching for more specific topics and reading corresponding articles, papers and books.

Since time series analysis is a very popular approach, it is supported by most open source and closed source commercial data science and statistical environments (software), such as R, Python, SAS, SPSS and many others. If you want to use R for this, check my answers on general time series analysis and on time series classification and clustering. I hope that this is helpful.

Answer 3

I majored in Economics, and I immediately agreed with the first response. Hotel revenue management and airline fares don't have one of the "given" needs to apply even common-knowledge economics models... they do not have price transparency between competitors. Rather than bore you with the explanation, I think this is a great way to "get" why you should change your topic unless you (seriously) want to spend a few months studying the economics of these two industries: Priceline and hotel tonight. Both airlines and hotels are on popular sites that operate in a way where the customer agrees to a price (usually there are "clues" to what you're sort of gambling on with hotels), then pays, and then finds out what they bought. It's not to milk the consumer, it's that companies rely on opaque pricing to be competitive in the market. Again, it sounds like I don't know what I'm talking about without the econ models, but that's because they deviate SO much from common-knowledge supply and demand, it would be wasteful. Just think about the existence and success of priceline and hotel tonight. I think it's important to always remember what your data represents. The comments above are probably excellent ideas for other projects. But remember what your data is about! It's so important in data science! Those will not work due to the subject matter of your data. I am trying to be helpful.