$\begingroup$
I am working on a chatbot that helps students.
So, I wanted to make use of bert model which has better performance on mathematics, which lead to me to math-bert, but the paper on it said that it was trained only on mathematical corpus, which means it wont have great performance on general sentences (example in image), so is there a method to combine sentence-bert and math-bert?
[![enter image description here][1]][1]
Or, the only way is to train bert model from scratch using corpus used for sentence-bert and math-bert.
$\endgroup$
I am working on a chatbot that helps students.
So, I wanted to make use of bert model which has better performance on mathematics, which lead to me to math-bert, but the paper on it said that it was trained only on mathematical corpus, which means it wont have great performance on general sentences (example in image), so is there a method to combine sentence-bert and math-bert?
[![enter image description here][1]][1]
Or, the only way is to train bert model from scratch using corpus used for sentence-bert and math-bert.
Add a comment
|
3 Answers
$\begingroup$
$\endgroup$
Another solution is to directly use GPT for maths.
Currently GPT-3 is one of the best models to answer maths problems, but it requires a poweful hardware.
https://openai.com/blog/grade-school-math/
Nevertheless, you can have good results using GPT-2:
https://github.com/openai/grade-school-math
https://github.com/openai/grade-school-math/blob/master/grade_school_math/train.py
I recommend starting with the smallest model (124M parameters). It should be enough if it is just a chatbot to solve classic maths problems.
answered Dec 26, 2022 at 9:21
$\begingroup$
$\endgroup$
Yes, it is possible to combine sentence-BERT and math-BERT models to improve the performance of a chatbot that helps students. There are a few different approaches that you could take to do this:
Fine-tuning: One approach would be to fine-tune a BERT model on a combined dataset that includes both mathematical and non-mathematical sentences. This would allow the model to learn both types of language and improve its performance on a wide range of tasks. You could use the sentence-BERT and math-BERT models as pre-trained models and then fine-tune them on your own dataset using the transformers library in Python.
Ensemble learning: Another approach would be to use ensemble learning, which involves training multiple models and combining their predictions to make a final prediction. You could train a sentence-BERT model and a math-BERT model separately, and then combine their predictions using a simple voting or averaging technique.
Multi-task learning: Another option would be to use multi-task learning, which involves training a single model to perform multiple tasks simultaneously. You could design a multi-task learning model that is able to classify mathematical and non-mathematical sentences and use it to improve the performance of your chatbot.
$\begingroup$
$\endgroup$
I am also working on building a model to evaluate students' essays and I am trying to combine word embeddings from bert, bertweet and roberta. I am able to combine the cls tokens from these models. My question is on what encoding and attention masks, I will be fitting the final model as I have three sets of word encodings and attention masks. This is the model structure:- 