0
$\begingroup$

I'm trying to build a classification model that predicts the price of New York taxi trips (year 2018). Datasource page

Since the original file is very large (112 234 626 rows), I constructed the smaller sample file (1 000 000 rows) to make the first exploratory analysis. By using this sample file I constructed Heatmap chart to display the correlations between dataset features. I found out that the model's dependent variable (fare_amount) is only correlated to trip_distance variable. The other independant variables didn't have any considerable correlation.

  1. Does this mean that the dataset I use is not appropriate for the classification task?
  2. Or I haven't solved the task correctly?
  3. How many independent variables should correlate to the dependent variable to count it "good enough" for further analysis?
  4. Which method should I prefer for feature selection and dimensionality detection? Should it be Heatmap chart, PCA or something else?

My code in GitHub

$\endgroup$
1
  • 1
    $\begingroup$ I don't understand what you mean by "good enough for further analysis". Fares are fundamentally a function of trip_distance, so that correlation is not surprising. What are the other features, and why do you think they would be associated with fare_amount? If trip_distance ends up predicting fare_amount precisely, then I would think your model is a success-- you can reliably predict an outcome from other data. Are you working towards some other application? $\endgroup$
    – Upper_Case
    Jun 4 '19 at 20:19
2
$\begingroup$

Think about the data generating process: For taxi fares, THE relevant thing is distance/time of the trip. This is by definition the case for most taxi services (Uber might be different). Maybe time of the day plays a role as well, e.g. when fares during day/night are different.

In a simple case you might not even need a statistic solution, viz. if the problem is deterministic, meaning fare is a simple linear function of distance.

You can use (for example) a linear regression to predict fare. In this case you could have a model:

fare = b0 + b1 * dist + error.

In this model, there is nothing wrong with having only one predictor (this is a univariate linear regression).

BTW: If you predict a continuous variable, you do regression. If you predict classes (like yes/no) you do classification.

If you have many variables with only weak correlation to y, you can also use regulation (by L1 norm), in linear regression this is called Lasso, to „shrink“ features which are irrelevant.

P.S. Plot price (y) against distance (x). You will likely find close alignment of both.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.