Fine-tune GPT on sketch data (stroke-3)

Question

These past days I have started a personal project where I would like to build a model that, given an uncompleted sketch, it can finish it. I was planning on using some pretrained models that are available in HuggingFace and fine-tune them with my sketch data for my task. The sketch data I have is in stoke-3 format, like the following example:
[
[10, 20, 1],
[20, 30, 1],
[30, 40, 1],
[40, 50, 0],
[50, 60, 1],
[60, 70, 0]
]
The first value of each triple is the X-coordinate, the second value the Y-coordinate and the last value is a binary value indicating whether the pen is down (1) or up (0). I was wondering if you guys could give me some instruction/tips about how should I approach this problem? How should I prepare/preprocess the data so I can fit it into the pre-trained models like BERT, GPT, etc. Since it's stroke-3 data and not text or a sequence of numbers, I don't really know how should I treat/process the data.

Thanks a lot! :)

Answer 1

I suggest that you input the sketch data as text. There is no problem encoding these as just text. Also, have the model generate text as well. With that in mind, you should use a model that is meant for text generation, like GPT-2 or any other Transformer decoder model (unlike BERT).

Answer 2

Your task of completing sketches is interesting, and it can be tackled using sequence-to-sequence models such as RNNs or transformers like GPT. However, as you noted, your input data is in stroke-3 format, which is not text or a sequence of numbers. Therefore, you will need to preprocess your data and convert it into a sequence of numerical vectors that can be fed into a pre-trained model like BERT or GPT.

One possible approach for preprocessing the stroke-3 data is to convert it into a sequence of relative coordinates. You can start by computing the absolute coordinates of each stroke point by cumulatively summing the x and y coordinates of the strokes. Then, you can convert the absolute coordinates into relative coordinates by subtracting the previous absolute coordinate from the current absolute coordinate. This will give you a sequence of relative coordinates that represent the strokes.

Next, you can represent each stroke point as a numerical vector. One common representation is to use a three-dimensional vector (dx, dy, p), where dx and dy are the relative coordinates of the stroke point, and p is the pen state (1 if the pen is down, 0 if the pen is up). This will give you a sequence of numerical vectors that can be fed into a pre-trained model.

After preprocessing your data, you can fine-tune a pre-trained model like GPT or BERT on your sketch completion task. You can use the pre-trained model's sequence-to-sequence capabilities to generate the missing strokes for an incomplete sketch. During fine-tuning, you can use a loss function that penalizes the model for generating strokes that are too different from the original sketch data.

Overall, the key steps to approach this problem are:

• Preprocess your stroke-3 data by converting it into a sequence of relative coordinates and representing each stroke point as a numerical vector.

• Fine-tune a pre-trained sequence-to-sequence model like GPT or BERT on your sketch completion task.

• During fine-tuning, use a loss function that penalizes the model for generating strokes that are too different from the original sketch data.