3
$\begingroup$

I am doing research for Google NLP AutoML, What methodologies they have used, techniques, models, feature selection, hyper parameter optimization, etc.

I could not find any paper on how google built their NLP AutoML.

Can anyone guide me on that? how to find google's research on that field for academic research?

Any paper you may have will help.

Thanks

$\endgroup$

2 Answers 2

1
$\begingroup$

I don't think you will find details of this resource hanging around - it is trade secret. My bet is Google first trains a huge language model as base, and let users fine-tune it with their own data, just like any usual transfer learning.

As for how they train the base model, the closest thing I find is the Pathways Language Model PaLM, developed around 2021, which is described as 'a 540-billion parameter, densely activated, Transformer language model', trained on '6144 TPU v4 chips'. Though Google may be keeping more dark magics, PaLM is already the state of the art.

$\endgroup$
-1
$\begingroup$

As you know, Auto ML can have a general meaning. To know better the meaning, some paper could help you to catch some idea about auto ml (that can be applied in NLP too). This paper could be helpful.

The success of machine learning in a broad range of applications has led to an ever-growing demand for machine learning systems that can be used off the shelf by non-experts. To be effective in practice, such systems need to automatically choose a good algorithm and feature preprocessing steps for a new dataset at hand, and also set their respective hyperparameters. Recent work has started to tackle this automated machine learning (AutoML) problem with the help of efficient Bayesian optimization methods.

$\endgroup$
1
  • $\begingroup$ Thanks, I have read many general AutoML papers, but did not get anything specifically for NLP AutoML $\endgroup$
    – asmgx
    Commented Sep 15, 2019 at 22:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.