2
$\begingroup$

I have a dataset with one of the (important) features being the geographic distances from NYC. Of course, some of the values are missing.... The goal is predicting whether people with certain attributes (proximity being one of them, and the typical age, sex, education, etc. being the others) will engage in an activity in NYC (e.g., visiting MOMA, taking a Broadway show, moving to the city altogether, enrolling in an area school, things like that).

My basic question is - missing values aside - is it correct to just take distances into account "as is" or should they somehow be divided between "driving/train distances" and "flying distances" (essentially, converting them into "the number of hours it takes someone to get to NYC by the most efficient means")?

If we take Los Angeles and Richmond, VA as examples - the distance to NYC from LA is about 10 times that of Richmond; flight times are only 4 times longer, but flight time from LA and drive time from Richmond are approximately the same. So what's the right way to think about that?

And once the right approach to distances is determined, how does one go about imputing distances for missing values?

Improve this question
$\endgroup$
6
  • $\begingroup$ Do you also have the distances between the other cities, or is it just NYC? What's the question you are asking? distance or time? $\endgroup$
    – Matthieu Brucher
    Dec 19, 2018 at 18:04
  • $\begingroup$ The important conceptual value here is "proximity" to NYC - the "closer" one is to NYC, the more likely it is (or so I'm told...) that some action will be taken. So if I'm to predict the action, I have to decide how to define this "proximity": by time or distance? And then, of course, how to impute it when it's missing. $\endgroup$
    – Jack Fleeting
    Dec 19, 2018 at 18:09
  • $\begingroup$ Well, proximity doesn't make sense, you need to get some better measure. Either time or distance. Then for the missing data, you are missing another dimension to know how to reconstruct distances for these points. Do you have distances between other points or not? All these bits of information should be in your question. $\endgroup$
    – Matthieu Brucher
    Dec 19, 2018 at 18:50
  • $\begingroup$ I know that proximity is meaningless, that's why I'm trying to replace it with either distance or time; I just don't know which is "better". As far as the distances between the other points are concerned - yes, I have them all (as in - I can calculate the distance between LA and Richmond); but why is that important? $\endgroup$
    – Jack Fleeting
    Dec 19, 2018 at 19:27
  • $\begingroup$ To compute the missing proximities, once you have decided, you can use the distance from A to missing C as min(AB+BC) over all B (traditional approximation of geodesic distances). In what context will these proximities be used? For instance if it's travelling, then time maybe better than if it's for shipping where distance maybe more relavant? $\endgroup$
    – Matthieu Brucher
    Dec 19, 2018 at 19:29

2 Answers 2

Reset to default
0
$\begingroup$

I would say that for your goal, the time to NYC is better than the distance. Indeed, whether I'm 100km or 10km, if it takes me an hour to go to the city center, it's the same burden to me.

So I would advise you to use the time as a metric rather than the distance itself due to how logistics work.

Then for your missing measurements, the best is to use compute an approximate geodesic distance/time. This also makes sense because for a user to go from A to C but if there is no such path, he would need to first spend time to go to B and then from B to C.

So for the missing entries A->C, you can fill them with min(AB+BC+epsilon), where B is the set of all available cities (I don't think more than 1 hop is something people would consider, so you don't need the full distance matrix -> Floyd Warshall algorithm) and espilon may be a time to go from one train station/airport to another.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ @ Matthieu Brucher - thanks a lot! Unfortunately, looks like I can not upvote your answer, only mark it solved... $\endgroup$
    – Jack Fleeting
    Dec 19, 2018 at 20:21
1
$\begingroup$

I don't think time alone is what you are after, although it would certainly be useful.

It sounds to me like you would also want to consider the cost of transport.

Yes the drive and flight may take the same amount of time, but you could be looking at orders of magnitude difference in the total cost. If you are only looking at extremely wealthy people this may not matter, but otherwise this could have as big an impact as time alone.

My advice would be to use the existing distance column to approximate the minimum cost of transportation, similar to how you would approximate the minimum time of transportation. I would then use both time and cost as inputs to my model.

Improve this answer
$\endgroup$
2
  • $\begingroup$ "use the existing distance column to approximate the minimum cost of transportation" - good concept, but my guess (haven't tried yet) is that unlike time, "cost" is going to be really hard to estimate, let alone calculate accurately: between trains, plains, buses and automobiles (and seasonal variations), I'll have to get a PhD in transportation studies:) $\endgroup$
    – Jack Fleeting
    Dec 20, 2018 at 12:13
  • $\begingroup$ You could certainly get within an order of magnitude without much more effort than it would take to figure out the time. It doesn’t need to be perfect to be potentially helpful $\endgroup$
    – kbrose
    Dec 20, 2018 at 14:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.