2
$\begingroup$

I am trying to transform data for use in regression, most likely the Ridge or Lasso technique implemented in sklearn.linear_model.

My training data contains time stamps , which I believe may have predictive power. The time stamps reflect the time that a user placed an order for pizza. Here is an example:

Edit: Including labels in field elapsed_time, which is in seconds.

import pandas as pd
import sklearn.linear_model as linear_model

delivery_data = {
    'order_time' : ['2018-09-12 21:43:08', '2018-09-13 06:33:04', '2018-09-13 09:12:18'],
    'price' : [34.54, 8.63, 21.24],
    'miles' : [6, 3, 7],
    'home_type' : ['apartment', 'house', 'apartment'],
    'elapsed_time' : [2023, 1610, 1918]
}

df = pd.DataFrame(delivery_data)
df['order_time'] = pd.to_datetime(df['order_time'])

The resulting DataFrame looks like this:

           order_time  price  miles  home_type  elapsed_time
0 2018-09-12 21:43:08  34.54      6  apartment          2023
1 2018-09-13 06:33:04   8.63      3      house          1610
2 2018-09-13 09:12:18  21.24      7  apartment          1918

I am trying to predict the time to deliver pizza (elapsed_time) given timestamp, quantitative, and categorical data.

I suspect that time of day is predictive but that date is less predictive.

So far, I am considering extracting only the hour from the time stamp. In this example, order_time would become [21, 6, 9]. My first concern is that 23:59 has an hour of 23 and 00:01 has an hour of 0. The two values are far apart, even though the order times are two minutes apart.

Is there a better way to transform this datetime data?

Does it make a difference that the dataset contains other quantitative data (price, miles_from_store) and categorical data (home_type)?

$\endgroup$
6
  • $\begingroup$ Question: do you have labels for any of your data (actual delivery times)? $\endgroup$ Commented Sep 13, 2018 at 19:29
  • $\begingroup$ @AdrianKeister, I have edited the question to include labels. The target column is elapsed_time, which reports time in seconds. $\endgroup$ Commented Sep 13, 2018 at 20:15
  • $\begingroup$ Excellent! Obviously you're going to want more data than this, but a toy data set is a great way to spin up the machinery you're going to use on the bigger dataset. I would recommend a pipeline for doing all the preprocessing tasks. It really speeds things up, and is a very straightforward syntax (use the make_pipeline function from sklearn.pipeline. $\endgroup$ Commented Sep 13, 2018 at 20:22
  • $\begingroup$ @AdrianKeister the part I still do not understand is how should I manipulate the order_time data? Does my idea to extract the hour only have any merit? $\endgroup$ Commented Sep 13, 2018 at 20:26
  • $\begingroup$ The problem is, as you've already hinted at in your question, you could get unexpected results when there are midnight deliveries. Using the approach I've hinted at in my answer and comments, you'll avoid that because these built-in Python libraries are quite capable of doing that arithmetic correctly. $\endgroup$ Commented Sep 13, 2018 at 20:28

1 Answer 1

2
$\begingroup$

The datetime type has arithmetic operations available. If you have two datetime types, you can find the delta between them - the result will be a datetime.timedelta class. The other quantitative data is easily incorporated into a linear regression model, either lasso or ridge. Pretty much all those scikit-learn models can use a vector of features.

$\endgroup$
2
  • $\begingroup$ I have edited my question to show that the training data shows elapsed time in seconds, which is what I am trying to predict. Since I do not have to calculate elapsed time from end minus start, is there any need to use datetime.timedelta. Is there a transformation I should perform on the order_time data? $\endgroup$ Commented Sep 13, 2018 at 20:21
  • $\begingroup$ It's six of one, a half-dozen of the other, most likely. You can either convert the timedelta to seconds, or you can convert the datetime.datetime's to epoch time in seconds and do the subtraction later. $\endgroup$ Commented Sep 13, 2018 at 20:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.