1
$\begingroup$

I am attempting to use XGBoost in R to train a model that predicts a fixed number of target variables using all data from previous dates, as well as the two categorical variables (Cat1 and Cat2) for the current date as predictors. The original data is in this format:

╔═════════╦═════════╦══════════╦══════╦══════╦══════╦══════╗
║ Target1 ║ Target2 ║ Date     ║ Cat1 ║ Cat2 ║ Var1 ║ Var2 ║
╠═════════╬═════════╬══════════╬══════╬══════╬══════╬══════╣
║ 1       ║ 2       ║ 01/01/20 ║ A    ║ B    ║ 3    ║ 4    ║
║ 5       ║ 6       ║ 02/01/20 ║ C    ║ D    ║ 7    ║ 8    ║
║ 8       ║ 7       ║ 03/01/20 ║ A    ║ D    ║ 6    ║ 5    ║
║ 4       ║ 3       ║ 04/01/20 ║ C    ║ B    ║ 2    ║ 1    ║
║         ║         ║          ║      ║      ║      ║      ║
╚═════════╩═════════╩══════════╩══════╩══════╩══════╩══════╝

And I conceptualise the training data looking like this for each row of the data frame, where the Train_DataFrame column contains all data from previous dates:

╔═════════╦═════════╦══════╦══════╦═════════════════╦═════════╦══════╦══════╦══════╦══════╗
║ Target1 ║ Target2 ║ Cat1 ║ Cat2 ║ Train_DataFrame ║         ║      ║      ║      ║      ║
╠═════════╬═════════╬══════╬══════╬═════════════════╬═════════╬══════╬══════╬══════╬══════╣
║       4 ║       3 ║ C    ║ B    ║ Target1         ║ Target2 ║ Cat1 ║ Cat2 ║ Var1 ║ Var2 ║
║         ║         ║      ║      ║ 1               ║ 2       ║ A    ║ B    ║ 3    ║ 4    ║
║         ║         ║      ║      ║ 5               ║ 6       ║ C    ║ D    ║ 7    ║ 8    ║
║         ║         ║      ║      ║ 8               ║ 7       ║ A    ║ D    ║ 6    ║ 5    ║
║         ║         ║      ║      ║                 ║         ║      ║      ║      ║      ║
╚═════════╩═════════╩══════╩══════╩═════════════════╩═════════╩══════╩══════╩══════╩══════╝

First question; is it possible to pass an entire data frame as a variable to a model?

If so, then how can this be done in a memory efficient manner, so as to avoid data duplication? I know I could have the Train_DataFrame column contain lists of each "past" data, however, this would lead to data duplication and inefficient memory usage. Is there a way to have this column contain a sub-setting function to pass to the original data frame, for example?

Or is there a better approach to the problem that I am potentially missing?

Thanks in advance.

$\endgroup$
2
$\begingroup$

First question; is it possible to pass an entire data frame as a variable to a model?

No, each feature must be a single value. In other words you could provide the data frame as a vector containing all the values, assuming the size is fixed: each column would correspond to a specific cell in the original data frame.

But I think an even better option in your case is to look into methods which take (chronological) sequences into account. Conditional Random Fields might be a good option, assuming you need to predict the target variables for the whole sequence?

| improve this answer | |
$\endgroup$
  • $\begingroup$ Sorry for the delayed reply, been spending a lot more time on the data engineering side of things so haven't had a look at my modelling project for a while. Thanks for the pointer to Conditional Random Fields, I'll certainly have a look into that. $\endgroup$ – Michael Pyle Jul 24 at 12:33
0
$\begingroup$

Lets break down your questions into sub-parts. Q1: do you mean the predictor needs to predict 2 target values? (Target 1 and target2)

Answer: the model can always predict only ONE target value. This target value can be either categorial or numerical.

Q2. One column ( train_dataframe) contains all data from previous dates. Why?

Ans: we don’t need to do this. We can simply use the entire data from previous dates in separate columns too.

Q3 : passing an entire dataframe as 1 variable? Ans: not required. Pass it as it is.

Q4: list of each past data is duplicate? Ans: that is ok. Your question is a bit unclear. Is the data imported as it is or you are using an external engine for it? Like mysql/ mongodb? Then you can put a query to get only unique values.

Q5: Is there a better approach? Ans: it looks like you have a big dataset. Try using mongodb. Load your data into mongodb and import it to R. R has a wonderful feature of showing the values of all variables in its right side window. It will be helpful to check it.

I hope this helps!

| improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks for the reply and sorry for not getting back to you sooner. Q1) So I'd have to train two separate models. Fine. Q2-4) I think I maybe haven't explained myself clearly enough. What I mean is that for say predicting a the two variables on yesterdays date I want to use the entire dataset for the previous dates as predictor for these variables as well as the two variables (Cat1 and Cat2) for the current date. The issue with memory comes in that if I pass the entire dataset plus yesterdays data as the variable for todays prediction, this will be duplicated data if it's not... (c) $\endgroup$ – Michael Pyle Jul 24 at 12:33
  • $\begingroup$ a pointer to the master dataset held in memory, if that makes sense? Q5) I make use of MongoDB and R already for this project actually! $\endgroup$ – Michael Pyle Jul 24 at 12:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.