How to treat time based ticket prices for train/test split

Question

I have a dataset of airfare price tickets that were scraped throughout a 6 month period where each observation represents a particular price for a specific flight on a specific date that it was scraped. In other words, a specific unique flight may appear multiple times in the dataset if it's scraped multiple times on different days. For example,

Scrape Date: 11/13/19, Days To Trip: 42, Flight: DL1345 , Departure: 12/25, Time: 5:00PM, Price: 290
Scrape Date: 11/22/19, Days To Trip: 33, Flight: DL1345 , Departure: 12/25, Time: 5:00PM, Price: 330
Scrape Date: 12/01/19, Days To Trip: 24, Flight: DL1345 , Departure: 12/25, Time: 5:00PM, Price: 349

I know that with time-series data such as stock prices, you want to split your training/testing data so that the data in testing is in the future and comes after the data in training. However, I don't believe the dataset I have would warrant a split like this and I can instead randomly shuffle the data for train/test split but I am not 100% sure on the right call. Should I split the data based on time or can I randomly sample since the price of the tickets don't depend on each other?

Answer 1

Indeed airfare data is different than many other price data. In To Buy or Not to Buy: Mining Airfare Data to Minimize Ticket Purchase Price the authors put it as following:

Computational finance is concerned with predicting prices and making buying decisions in markets for stock, options, and commodities. Prices in such markets are not determined by a hidden algorithm, as in the product pricing case, but rather by supply and demand as determined by the actions of a large number of buyers and sellers. Thus, for example, stock prices tend to move in small incremental steps rather than in the large, tiered jumps observed in the airline data.

(tho the reality is not that black and white as airfares also depend on seat availability and stock prices are also indirectly driven by algorithms but generally they have a point)

These algorithm-driven price changes do not so much rely on historical price development which is why I would give your approach a try. In line with that there are multiple papers in which airfare prices have not been treated as timeseries (besides the one already linked also see Predicting Airfare Prices - tho in this paper they did consider an attribute related to the most recent previous price of the ticket!).

Nevertheless, there are many date-related aspects your might want to consider. "Days to trip" is one which is in your data already. Others, easily available from your data, could be

• "weekday of departure",
• "departure during holiday season or not" and
• "weekday of price request".

And then of course there are tons of other variables which might be important, e.g. "number of stopovers", "overnight flight or not", "number of free baggage" (and obviously the cabin class or maybe even booking class) etc. (also see Airfare Prices Prediction Using Machine Learning Techniques)