1
$\begingroup$

I am trying to transform data for use in regression, most likely the Ridge or Lasso technique implemented in sklearn.linear_model.

My training data contains time stamps , which I believe may have predictive power. The time stamps reflect the time that a user placed an order for pizza. Here is an example:

Edit: Including labels in field elapsed_time, which is in seconds.

import pandas as pd
import sklearn.linear_model as linear_model

delivery_data = {
    'order_time' : ['2018-09-12 21:43:08', '2018-09-13 06:33:04', '2018-09-13 09:12:18'],
    'price' : [34.54, 8.63, 21.24],
    'miles' : [6, 3, 7],
    'home_type' : ['apartment', 'house', 'apartment'],
    'elapsed_time' : [2023, 1610, 1918]
}

df = pd.DataFrame(delivery_data)
df['order_time'] = pd.to_datetime(df['order_time'])

The resulting DataFrame looks like this:

           order_time  price  miles  home_type  elapsed_time
0 2018-09-12 21:43:08  34.54      6  apartment          2023
1 2018-09-13 06:33:04   8.63      3      house          1610
2 2018-09-13 09:12:18  21.24      7  apartment          1918

I am trying to predict the time to deliver pizza (elapsed_time) given timestamp, quantitative, and categorical data.

I suspect that time of day is predictive but that date is less predictive.

So far, I am considering extracting only the hour from the time stamp. In this example, order_time would become [21, 6, 9]. My first concern is that 23:59 has an hour of 23 and 00:01 has an hour of 0. The two values are far apart, even though the order times are two minutes apart.

Is there a better way to transform this datetime data?

Does it make a difference that the dataset contains other quantitative data (price, miles_from_store) and categorical data (home_type)?

$\endgroup$
  • $\begingroup$ Question: do you have labels for any of your data (actual delivery times)? $\endgroup$ – Adrian Keister Sep 13 '18 at 19:29
  • $\begingroup$ @AdrianKeister, I have edited the question to include labels. The target column is elapsed_time, which reports time in seconds. $\endgroup$ – Jacob Quisenberry Sep 13 '18 at 20:15
  • $\begingroup$ Excellent! Obviously you're going to want more data than this, but a toy data set is a great way to spin up the machinery you're going to use on the bigger dataset. I would recommend a pipeline for doing all the preprocessing tasks. It really speeds things up, and is a very straightforward syntax (use the make_pipeline function from sklearn.pipeline. $\endgroup$ – Adrian Keister Sep 13 '18 at 20:22
  • $\begingroup$ @AdrianKeister the part I still do not understand is how should I manipulate the order_time data? Does my idea to extract the hour only have any merit? $\endgroup$ – Jacob Quisenberry Sep 13 '18 at 20:26
  • $\begingroup$ The problem is, as you've already hinted at in your question, you could get unexpected results when there are midnight deliveries. Using the approach I've hinted at in my answer and comments, you'll avoid that because these built-in Python libraries are quite capable of doing that arithmetic correctly. $\endgroup$ – Adrian Keister Sep 13 '18 at 20:28
2
$\begingroup$

The datetime type has arithmetic operations available. If you have two datetime types, you can find the delta between them - the result will be a datetime.timedelta class. The other quantitative data is easily incorporated into a linear regression model, either lasso or ridge. Pretty much all those scikit-learn models can use a vector of features.

$\endgroup$
  • $\begingroup$ I have edited my question to show that the training data shows elapsed time in seconds, which is what I am trying to predict. Since I do not have to calculate elapsed time from end minus start, is there any need to use datetime.timedelta. Is there a transformation I should perform on the order_time data? $\endgroup$ – Jacob Quisenberry Sep 13 '18 at 20:21
  • $\begingroup$ It's six of one, a half-dozen of the other, most likely. You can either convert the timedelta to seconds, or you can convert the datetime.datetime's to epoch time in seconds and do the subtraction later. $\endgroup$ – Adrian Keister Sep 13 '18 at 20:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.