1
$\begingroup$

I am playing the timeGAN model, using the example code from ydata-synthetic repo. To train the model, we used

synth.train(stock_data, train_steps=50000)

to generate data, we used

synth_data = synth.sample(len(stock_data))

One way to evaluate the synthetic data is to measure discriminative score. Ideally, high-quality synthetic data should be similar to real data. So if we build a post-hoc real/fake data classifier, the classifier should have difficulty to tell data is real or generated, thus accuracy ~ 0.5 and discriminative score ~ 0.

I have two questions:

  1. Does synth_data have the same time order as stock_data? For example, synth_data[123] somehow is synthesized from stock_data[123]? I doubted it since synth.sample(xxx) where xxx could be any number > real data size.
  2. Does anyone (using the timeGAN on the stock data) reproduce the discriminative score ~ 0.1 from the paper Time-series Generative Adversarial Networks? I trained a model with 50000 iterations myself and confirmed all the results shown in the example code, except discriminative score. I used GRU or LSTM to build real data/fake data classifier, which can show high accuracy (> 0.9) such that the score > 0.4.
$\endgroup$

1 Answer 1

2
$\begingroup$

For my master thesis AI, I've studied the TimeGAN model in-depth, including the architecture and the used metrics.

1: Not necessarily. Vanilla TimeGAN uses (variations of) RNNs. These nets share the weights over all different timesteps. Hence, the generation is just a learned abstraction of the time-series data. synth_data[123] does not necessarily have to be based on stock_data[123]. However, if all 123rd timesteps of your data have some specific correlation, your model might pick up on this.

2: Apologies for inserting my own opinion, but you should not put too much focus on trying to reproduce the discriminative scores. It is possible to reproduce the discriminative scores, I did so for my master thesis. However, please note that the results of the discriminative scores are VERY unstable. For the stocks data, there is an (unspecified kind of) instability of $.02$ on a metric score of $.1$ for TimeGAN. That's a lot, no matter how you turn it.

Also, note that the discriminative score is not a well-established metric in GAN research. Preferably, a metric agrees with human judgement, is consistent, and is easy to calculate. The discriminative score is non of these. You train a non-optimized neural network for 2000 epochs, with a random learning rate, with a hidden neural network size half that of the time-series features (though you have to check their non-official implementation for these details). If you train with a different learning rate, a different network size, longer (or shorter) epochs, you could get a discriminative score of $0.00$ or $0.5$ on every single dataset. You can get any model to score $0.00$ by simply putting the learning rate or number of epochs on $0$.

$\endgroup$
6
  • $\begingroup$ Robin, according to your reply, what else is a better metric to judge your synthetic data is "good"? I found predictive score is stable but less useful. Did you see the same behavior? $\endgroup$
    – TripleH
    Commented May 14, 2023 at 22:02
  • $\begingroup$ Robin, may I have your thesis title or link? Thanks. $\endgroup$
    – TripleH
    Commented May 14, 2023 at 22:17
  • $\begingroup$ The predictive score is indeed also not the best metric. It too easily converges to the optimum. Metrics for TimeSeries synthetic data are very underdeveloped, unfortunately. You can adapt the metrics from "How Faithful is your Synthetic Data?" to time-series data (section 5.3). However, those also have their own problems. There is a lot of additional research still to be done in the space of time-series synthetic data metrics. $\endgroup$ Commented May 15, 2023 at 7:04
  • $\begingroup$ I'm still working on my master thesis so i cannot share it yet, unfortunately. $\endgroup$ Commented May 15, 2023 at 7:05
  • $\begingroup$ Really appreciate for your suggestion. $\endgroup$
    – TripleH
    Commented May 16, 2023 at 3:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.