I have an unbalanced tensorflow windowed dataset with labels (over 90% negative examples) which I am trying to balance by filtering. I am having trouble filtering as my windowed dataset with labels does not fall under the cases I have come across in my searches, or the tensorflow documentation.
I'm working on a model to predict a binary classification based on time series data. I start with a time series dataframe with a number of columns (price, volume, etc.) where each row is one minute.
Currently I am still stuck on filtering the different labels. My next step after filtering would be to get the size of both filtered datasets, find the smaller size (n), and then concatenate the smaller dataset with (n) elements from the bigger dataset, after shuffling the bigger dataset. This way I would have a balanced dataset with an equal number of 1 and 0 labels. If you have a better Idea I would be happy to hear it.
EXPLAINING MY CODE: DFrame is a pandas dataframe with columns such as price, volume, etc., and each row is a different minute, with the first row being the earliest/oldest time period. The last column of DFrame is the classifier 0 or 1.
I then create a tensorflow dataset from slices with the first input being all the DFrame columns except for the label which is in the last column, and the second input (the label) being the last column which is the classifier.
I then use the window function to create windows of size (hindsight) which is currently 512, meaning (If i'm not mistaken) it takes the previous 511 minutes as well as the current minute and uses this as a rolling window to associate with the label of the current minute. So my understanding is that x is then an array of 512 arrays, from the row of the current minute to the row of 511 minutes ago, and the y is the label of the current minute. So x is an array of 512 arrays (rows for each minute, from the dataframe), and y is just one integer, 1 or 0.
Ideally I would like to be able to apply the same balancing logic to a multiclass classification problem, where I essentially add additional labels for additional price movement ranges.
The error comes from the filter. The model seems to run without that, and even trains my keras model. as explained I would actually want to add more code after the filter once I get it working to balance the dataset but I need to filter it first.
tensor= tf.data.Dataset.from_tensor_slices((tf.constant(DFrame[DFrame.columns.values[:-1]].values), tf.constant(DFrame[DFrame.columns.values[-1]].values)))
tensor = tensor.window(hindsight,1,1,True)
tensor = tensor.shuffle(1000)
tensor = tensor.filter(lambda x,y: tf.equal(y, 0))
tensor = tensor.flat_map(lambda x,y:tf.data.Dataset.zip((x.batch(hindsight), y.batch(1))))
tensor = tensor.batch(Batch_size).prefetch(1)
TypeError: Failed to convert object of type <class 'tensorflow.python.data.ops.dataset_ops._VariantDataset'> to Tensor. Contents: <_VariantDataset shapes: (), types: tf.int64>. Consider casting elements to a supported type.
