1
$\begingroup$

I'll build next-word generation using Tensorflow to predict address mapping. But, I saw many tutorial, next-word generation use long-text narration for its training dataset. But, I have dataset structure like this below (in dataframe). I'm confusing to slicing it into the predictors and label. this data below assumed as a dataframe:

input                                        output/label
250 Hartford Avenue, Bellingham, MA, 2019    {'address': 'Hartford Avenue', 'city': 'Bellingham', 'state': 'MA','zip': '2019'}  
700 Oak Street, Brockton, MA, 2301           {'address': 'Oak Street', 'city': 'Brockton', 'state': 'MA', 'zip': '2301'}

if the dataset is narration, next-word generation usually has slicing data like this:

Hartford Aven u
artford Avenu e

But, if I have my own dataset that has to slicing the label from different column (the dataframe), how it must be?

$\endgroup$
2
  • $\begingroup$ I don't understand the question: what do you mean by "I'm confusing to slicing it into the x and y" and "it's if the dataset is narration", and " have my own dataset that has to slicing the label from different column" ? $\endgroup$
    – Erwan
    Aug 5, 2022 at 7:52
  • $\begingroup$ I have edited the caption, hope it's better to understand. $\endgroup$
    – Mico S
    Aug 5, 2022 at 9:42

1 Answer 1

1
$\begingroup$

The model is currently predicting the next single character. The output of the model should be the next token.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.