I want to use T5 to do sentiment analysis on IMDB dataset. My dataset is of the following format:

# train data
f = open("train.csv", "r")
lines = f.readlines()
lines = [line.strip().split(",") for line in lines]
lines = [[line[0], line[1], ",".join(line[2:])] for line in lines] 
train = pd.DataFrame(lines[1:])
train = train.drop(train.columns[0], axis=1) # drop first column
print("\ntrain set size:", train.shape)
print("\nNumber of positives: ", train[1].astype(int).sum())
train = train.rename(columns={1: 'sentiment', 2: 'review'})
imdb_reviews = train["review"]
sentiments = train["sentiment"]
sentiments = [int(v) for v in sentiments]
sentiments = sentiments["sentiment"].tolist()

# test data
f = open("test.csv", "r")
lines = f.readlines()
lines = [line.strip().split(",") for line in lines]
lines = [[line[0], ",".join(line[1:])] for line in lines] 
test = pd.DataFrame(lines[1:])
id_test = test[0]
print("\ntest set:", test.shape)
test = pd.DataFrame(test[1])
print("Number of test sentences: {:,}\n".format(test.shape[0]))
test = test.rename(columns={1:'review'})

I found this code, but I could not understand how to adapt it to my own data format. I would appreciate it if you could inform me how to do it. The train set includes 25000 observations, of which 10% should be used as validation set.

  • $\begingroup$ Hey boy! It seems that you're not familiar with hugging face and its tokenisation pipelines. First, take a look at here which elaborates tokenisation. Based on your code, you're simply splitting the words based on white space, space more specifically. It seems it demands a more advance approach as the link you've provided is doing so in its middle lines. $\endgroup$ Commented Apr 3, 2023 at 16:49

2 Answers 2


That code is a combination of raw Python and the Pandas library. It might be more useful to use the Hugging Face library which is designed for this task.

Something like:

# Load imbd dataset
from datasets import load_dataset
imdb = load_dataset("imdb")

# Load T5 specific tokenizer
from transformers import T5Tokenizer
tokenizer = T5Tokenizer.from_pretrained('t5-small', model_max_length=512)

# Apply T5 specific tokenizer to the imdb dataset
def preprocess_function(examples):
    return tokenizer(examples["text"], truncation=True)

tokenized_imdb = imdb.map(preprocess_function, batched=True)

Why not directly use the Github code made available by HuggingFace? Preparing your dataset for fine tuning T5 is as simple as putting your train, dev and test sentences and labels into separate text files. Fine tuning is as simple as running one command in the terminal, see their README.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.