0
$\begingroup$

I am working on a chatbot that helps students.
So, I wanted to make use of bert model which has better performance on mathematics, which lead to me to math-bert, but the paper on it said that it was trained only on mathematical corpus, which means it wont have great performance on general sentences (example in image), so is there a method to combine sentence-bert and math-bert?
[![enter image description here][1]][1]
Or, the only way is to train bert model from scratch using corpus used for sentence-bert and math-bert.
$\endgroup$

3 Answers 3

0
$\begingroup$

Another solution is to directly use GPT for maths.

Currently GPT-3 is one of the best models to answer maths problems, but it requires a poweful hardware.

https://openai.com/blog/grade-school-math/

Nevertheless, you can have good results using GPT-2:

https://github.com/openai/grade-school-math

https://github.com/openai/grade-school-math/blob/master/grade_school_math/train.py

I recommend starting with the smallest model (124M parameters). It should be enough if it is just a chatbot to solve classic maths problems.

https://huggingface.co/gpt2

$\endgroup$
0
$\begingroup$

Yes, it is possible to combine sentence-BERT and math-BERT models to improve the performance of a chatbot that helps students. There are a few different approaches that you could take to do this:

Fine-tuning: One approach would be to fine-tune a BERT model on a combined dataset that includes both mathematical and non-mathematical sentences. This would allow the model to learn both types of language and improve its performance on a wide range of tasks. You could use the sentence-BERT and math-BERT models as pre-trained models and then fine-tune them on your own dataset using the transformers library in Python.

Ensemble learning: Another approach would be to use ensemble learning, which involves training multiple models and combining their predictions to make a final prediction. You could train a sentence-BERT model and a math-BERT model separately, and then combine their predictions using a simple voting or averaging technique.

Multi-task learning: Another option would be to use multi-task learning, which involves training a single model to perform multiple tasks simultaneously. You could design a multi-task learning model that is able to classify mathematical and non-mathematical sentences and use it to improve the performance of your chatbot.

$\endgroup$
0
$\begingroup$

I am also working on building a model to evaluate students' essays and I am trying to combine word embeddings from bert, bertweet and roberta. I am able to combine the cls tokens from these models. My question is on what encoding and attention masks, I will be fitting the final model as I have three sets of word encodings and attention masks. This is the model structure:- enter image description here

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.