I am aware there are time series cross validation methods however I don't think there is one yet which solves my task. I want to know if my idea seems reasonable or if there are better solutions out there.
Lets say I have a one time event each year for each observation in my data. Each observation experiences this one time event every year.
I also have 365 daily time series data for each observation.
- That is, say each observation is a person who releases their progress report (randomly throughout the year)
- They mostly just copy and paste their progress report from year to year - making small changes
- I can track the individuals work performance over the calendar year after they release their progress report.
I want to build a classifier based on performance ("good", "bad", "neutral") and then predict worker performance.
For exampe:
---- Say we are in 2007
---- I collect the reports for all workers from Jan 2005 - Dec 2005
---- Run a model to process the text and put into machine readable format for 2005 (Doc2Vec or some other word embedding model) - therefore each document is represented in some high dimensional space (each worker can submit their document at any stage throughout 2005 which is why I choose dec 2005 to construct this part of the model).
---- I calculate the "net" performance - such as some productivity score from Jan 2006 - Dec 2006 - therefore each worker will have a single score for their performance in Dec 2006.
---- I build my classification model in Jan 2007 to classify which workers are going to be "good", "bad" or "neutral" throughout 2007.
---- So I use the reports from 2005 to build the document embeddings from a Doc2Vec model. I then feed these document vectors to a machine learning classifier on the training sample of 2006 (i.e. the model gets to see the dec 2006 scores to train on). The out of sample testing occurs in Jan 2007 where the model is not able to see the end result (dec 2007).
Problem:
Say I train the model on 5000 reports in 2005, put these 5000 reports into a ML classifier in 2006 to learn from the true performance score in that year, finally I test in 2007 on the out of sample data. Since these document vectors are in a high dimensional space the model will simply just classify the documents in 2007 as it saw in the results in 2006 - therefore the results are biased....
To over come this issue...
Say I randomly sample 4000 reports to build the document vectors on in 2005 , I pass these 4000 documents to the ML classifier as training data in 2006 to obtain my prediction model based on these workers 2006 scores. Then in Jan 2007 I pass the remaining 1000 observations documents in 2006 (not 2005) as my testing data to the ML classifier - i.e. reports and scores the model has never seen before.
I thought that this method will not bias the results, have no look ahead bias, but I am only able to "forecast" a random sample of the workers - which isn't a problem.
I just want to know your opinions on this method. I do not want any biases at any stage of the process.
