0
$\begingroup$

I have a dataset of airfare price tickets that were scraped throughout a 6 month period where each observation represents a particular price for a specific flight on a specific date that it was scraped. In other words, a specific unique flight may appear multiple times in the dataset if it's scraped multiple times on different days. For example,

Scrape Date: 11/13/19, Days To Trip: 42, Flight: DL1345 , Departure: 12/25, Time: 5:00PM, Price: 290
Scrape Date: 11/22/19, Days To Trip: 33, Flight: DL1345 , Departure: 12/25, Time: 5:00PM, Price: 330
Scrape Date: 12/01/19, Days To Trip: 24, Flight: DL1345 , Departure: 12/25, Time: 5:00PM, Price: 349

I know that with time-series data such as stock prices, you want to split your training/testing data so that the data in testing is in the future and comes after the data in training. However, I don't believe the dataset I have would warrant a split like this and I can instead randomly shuffle the data for train/test split but I am not 100% sure on the right call. Should I split the data based on time or can I randomly sample since the price of the tickets don't depend on each other?

$\endgroup$
  • $\begingroup$ 'I don't believe the dataset I have would warrant a split like', can you explain why you think this? $\endgroup$ – Geert Immerzeel Dec 10 '19 at 6:55
  • $\begingroup$ I don't see how the tickets scraped for example in May could affect ticket prices in September. In addition, if you split the data based on time, testing data may include all the data from seasonal months (when more travel may occur) and the holidays during that time may be more abundant than the holidays in the training data or vice versa. In addition, one of the features in the dataset is "days to trip" so whether the trip is in August or in April, the notion of time is in relation to days away from the trip; the scrape date is not used. $\endgroup$ – fastlanes Dec 10 '19 at 7:07
0
$\begingroup$

Indeed airfare data is different than many other price data. In To Buy or Not to Buy: Mining Airfare Data to Minimize Ticket Purchase Price the authors put it as following:

Computational finance is concerned with predicting prices and making buying decisions in markets for stock, options, and commodities. Prices in such markets are not determined by a hidden algorithm, as in the product pricing case, but rather by supply and demand as determined by the actions of a large number of buyers and sellers. Thus, for example, stock prices tend to move in small incremental steps rather than in the large, tiered jumps observed in the airline data.

(tho the reality is not that black and white as airfares also depend on seat availability and stock prices are also indirectly driven by algorithms but generally they have a point)

These algorithm-driven price changes do not so much rely on historical price development which is why I would give your approach a try. In line with that there are multiple papers in which airfare prices have not been treated as timeseries (besides the one already linked also see Predicting Airfare Prices - tho in this paper they did consider an attribute related to the most recent previous price of the ticket!).

Nevertheless, there are many date-related aspects your might want to consider. "Days to trip" is one which is in your data already. Others, easily available from your data, could be

  • "weekday of departure",
  • "departure during holiday season or not" and
  • "weekday of price request".

And then of course there are tons of other variables which might be important, e.g. "number of stopovers", "overnight flight or not", "number of free baggage" (and obviously the cabin class or maybe even booking class) etc. (also see Airfare Prices Prediction Using Machine Learning Techniques)

| improve this answer | |
$\endgroup$
  • $\begingroup$ Thank you very much! This is indeed helpful and I also have many of those features you mentioned already engineered in my modeling so I appreciate the insight $\endgroup$ – fastlanes Dec 10 '19 at 21:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.