Your task of completing sketches is interesting, and it can be tackled using sequence-to-sequence models such as RNNs or transformers like GPT. However, as you noted, your input data is in stroke-3 format, which is not text or a sequence of numbers. Therefore, you will need to preprocess your data and convert it into a sequence of numerical vectors that can be fed into a pre-trained model like BERT or GPT.
One possible approach for preprocessing the stroke-3 data is to convert it into a sequence of relative coordinates. You can start by computing the absolute coordinates of each stroke point by cumulatively summing the x and y coordinates of the strokes. Then, you can convert the absolute coordinates into relative coordinates by subtracting the previous absolute coordinate from the current absolute coordinate. This will give you a sequence of relative coordinates that represent the strokes.
Next, you can represent each stroke point as a numerical vector. One common representation is to use a three-dimensional vector (dx, dy, p), where dx and dy are the relative coordinates of the stroke point, and p is the pen state (1 if the pen is down, 0 if the pen is up). This will give you a sequence of numerical vectors that can be fed into a pre-trained model.
After preprocessing your data, you can fine-tune a pre-trained model like GPT or BERT on your sketch completion task. You can use the pre-trained model's sequence-to-sequence capabilities to generate the missing strokes for an incomplete sketch. During fine-tuning, you can use a loss function that penalizes the model for generating strokes that are too different from the original sketch data.
Overall, the key steps to approach this problem are:
Preprocess your stroke-3 data by converting it into a sequence of relative coordinates and representing each stroke point as a numerical vector.
Fine-tune a pre-trained sequence-to-sequence model like GPT or BERT on your sketch completion task.
During fine-tuning, use a loss function that penalizes the model for generating strokes that are too different from the original sketch data.