Sklearn Random Forest Prediction Correlation Issue

I'm having issues with fitting a Random Forest model to a completely new dataset. I'm trying to predict tenancy lengths for current tenants. I have a dataset with tenancy information since 2008, with both tenants that have finished their contract and others which are still in their properties.

Steps undertaken so far:

1) Remove the tenancies that are still on going. This is to actually get target values for "Tenancy Length".

2) Find a suitable model to train and test on the finished tenancies. I'm happy with a Random Forest, with MAE of ~35 on train and ~70 on test.

3) Create a new dataframe in python that only has the previously excluded information (tenancies that have not yet finished) in order to apply the .predict method on these. I've also combined the previous 'train' and 'test' into a single train, as there is no further need to split that data up.

df = finished tenancies; currentDf = ongoing tenancies

#Select the features and target variable from finished tenancies data
X, y = df.iloc[:, 5:].values, df.iloc[:, 4].values
clf = RandomForestRegressor() #with relevant params
clf.fit(X, y)
yPred= clf.predict(X)


This results in a 'new' training set consisting of all the finished tenancies.

4) I move on and predict my current tenancies:

X2 = currentDF.iloc[:, 5:].values
y2Pred = clf.predict(X2)


While I cannot create a residual plot for this (as I have no previously known information regarding targets), I've noticed that 'Tenancy Length' strongly correlates with 'Tenancy Start', which was not the case in my trained model.

Does anyone have any idea what's going on? Why would something like this be happening? I can't for the life of me figure out if I'm doing something wrong, yet I doubt it's just coincidence for some reason.

• Have you tried looking at which features have the highest feature importance in the fitted model? The occurrence of these features might be different among current tenants vs past tenants in important ways, but it's hard to know without knowing what features the model is using. Perhaps there is an important feature which correlates to tenancy start among the current tenants, but not among the past tenants. Perhaps tenancy start is more important when another feature is present, and that feature is present far more frequently in current tenants. – Tim Goodman Oct 17 '16 at 10:27
• Also consider that there's a sort of survivorship bias in how you split your data. People who moved in recently would only appear in your training set if they moved out shortly after they moved in. – Tim Goodman Oct 17 '16 at 10:59
• Hi Tim, thank you very much for your comments. In regards to the survivorship problem, I've been down that road and I'm aware that my model will be restricted to only a 8 year period (2008-2016), so it will not be capable of handling longer tenancies. I've not been able to figure out any other way on how to handle this issue, but since most tenancies are short-term anyway (<5y), it doesn't seem to be a major problem. In regards to relationships between feature importance, that's a really good suggestion, will be back in a bit with an update regarding that. Thank you. – Silviu Tofan Oct 17 '16 at 11:06
• You're welcome, Silviu. Regarding survivorship, I'm not so much concerned about the 8+ year tenants as I am the other end of things: People who only became tenants recently. Among people who moved in during the last year, the only ones included in your training data will be the ones who failed to last for a full year as tenants - and these might not be representative of the typical tenant. You may want to look into survival analysis if you haven't already (e.g., Cox regression. There are also some methods using random forests.) – Tim Goodman Oct 17 '16 at 12:05
• @TimGoodman So I've looked at the features and discarded some more based on the observations you made. However, I'm still getting the same kind of results, even if I remove the start date from my prediction imgur.com/a/fGqUa. Unfortunately I'm limited to the information I can share regarding the variables themselves. Is there anyway I can get around this issue by subsampling my initial training set (i.e. make sure i have equal % of people lasting 1, 2, 3 etc. years)? In regards to survival analysis, I've only applied it as a descriptive procedure, will try to do regression based on it – Silviu Tofan Oct 17 '16 at 12:32