# Dealing with a dataset where a subset of points live in a higher dimensional space

I have a classification problem where I am dealing with economic data in a high dimensional (~400) space which includes dates, addresses, salaries, and a number of other variables. Most of the variables are made up of discrete categories and so these end up being encoded into many dimensions of binary variables.

There is a subset of the data where observations are made in two separate locations instead of just one. As a result, many of the dimensions are repeated for this subset. Most of the repeated data results in extra dimensions of binary class variables which would just be zero for the compliment set (i.e. only one location) of observations. However, one dimension is a salary where there are real values for the 2-location subset but Nans for the remaining data. To illustrate:

Obs | pay_1 | loc_1_A | loc_1_B| ... | job_1_A | job_1_B | ... | pay_2 | loc_2_A | ...
======================================================================================
1   | 50k   |   1     |   0    |     |   1     |   0     |     |   40k |   1     |
2   | 40k   |   0     |   1    |     |   1     |   0     |     |   45k |   1     |
3   | 70k   |   1     |   0    |     |   0     |   1     |     |  NaN  |   0     |
4   | 50k   |   1     |   0    |     |   1     !   0     |     |  NaN  |   0     |


### Question:

How to deal with data that has this systematically disjoint set of dimensions in which it lives?

I believe for many of the variables in the higher-dimensional subset it is no problem, since they will simply equal zero for any of the data points not in this subset. But since it is a classification problem, it seems pertinent for the sake of numerical stability to standardise the salary data ($pay_2$) to have values similar to a standard gaussian. Thus, I could not set the NaNs to zero for the lower-dimensional observations as zero would be a meaningful value.

I can think of a couple of approaches, one being to have a kind of non-linear "switch" variable that turns the higher dimensions on and off:

$$y = A * pay_1 + B * \text{loc_1_A} + ... + S * ( M * pay_2 + N * \text{loc_2_A} + ... )$$

though I don't see how I can implement this in a lot of standard methods.

Another option could be to just copy the $pay_1$ values into the $pay_2$ column for those rows where there is no existing $pay_2$ value. This would seem to me like it is imposing a lot of structure onto the data that is difficult to justify. Maybe if I add another binary "switch" variable on top of this, it can separate the classes in my data space?

I know this is probably a very problem/domain-specific issue but I would like to know if there are any standard treatments for data which has this kind of structure, as I'm sure it is a pretty common scenario. Any suggestions of ways to think about this or similar problems in other fields to look at?

Why don’t you just put the data with pay_2 40k and 45k into two additional rows?

Additionally, you could add an indicator variable loc_1_2 and set it to 1 or 2 depending on the location.

In general, dummy encoding might be dangerous. Instead you could convert your addresses or locations into physical coordinates (latitude and longitude) or just distance from, let’s say, centre of the city or center of the country (depending on your data).

Edit

Ok, I see now that putting the data with pay_2 into additional rows will not work.

But since it is a classification problem, it seems pertinent for the sake of numerical stability to standardise the salary data (pay2) to have values similar to a standard gaussian.

For your information: not all classification algorithms need standardised data, e.g. you may want to consider random forest or GBMs. If you stick to SVMs, regularised logistic regression or similar then I agree, it’d make sense to standardise.

Thus, I could not set the NaNs to zero for the lower-dimensional observations as zero would be a meaningful value.

The standard approach you asked about, would depend on the properties of your classifier. I’m aware of the following options:

1. If you classifier can handle NaNs (NAs), e.g. R version of xgboost, then you can just use the data as you show in your table above.
2. If you have to standardise the data, then you can use mean-imputation. The average and standard deviation that are used for the standardisation should be estimated on non-NaN data, of course, (pay_2 column excl NaNs). In general, in order to prevent data leakage you should not use the data from holdout data subset (or “test”, “validation” or whatever you call it) for calculating the mean and standard deviation.
3. If your classifier doesn’t require standardisation, you can put some numerical sentinel values instead of NaNs, e.g. -1, 0, or -9999.

If you pick option 2 or 3, you’ll need to add a binary predictor indicating whether pay2 column was modified or not (NaN or not).

Re dummy encoding: You haven’t asked about it explicitly and it’s difficult to suggest anything concrete without knowing the cardinality of your nominal variables, so I just put here the link to my answer about impact coding

• It's not a matter of either location 1 or location 2. Some data points have 2 locations and therefore have two sets of certain variables. Other data points only have one location, pay, etc. What makes dummy encoding dangerous? My primary concern is what to do with the "pay" values, as it's not clear to me what a suitable replacement for NaN is in this case. – kiliantics Dec 10 '17 at 20:55
• I see. Then I’d just put 0 or -1. Depending on your classification approach it might work, e.g. for trees based approaches. BTW I don’t understand your comment re „meaningful value“. Re dummy encoding - your feature dimension might increases dramatically in case of high cardinality, which makes it harder / slower to find a solution. Check the site re alternatives on „encoding of categorical predictors“ or similar. – aivanov Dec 10 '17 at 21:12
• I understand the problem of dimensionality of encoding but I don't see another way without tossing out relevant information. In my case, I am looking at the entire USA, so replacing location with distances from the centre of USA is not ideal. I made location clusters and added them as encoded classes. – kiliantics Dec 11 '17 at 3:28
• The "meaningful value" is for the salary data. Some data points have one salary, some have two. Therefore, the column for second salary has NaN values (along with other columns associated with second location, which just have zeroes since they are encoded classes). The NaNs are not "missing information" however - when there is only one salary, there is different information to when there are two salaries. I don't know if there is a standard procedure for treating this kind of data. That was my original question. It would be helpful if you could try understanding the question before answering. – kiliantics Dec 11 '17 at 3:32