TLDR: How to use ML to automatically assign best possible weights so my algorithm knows how much to value each feature to make a decision based on training data?
I am using Python to create a product title shortening algorithm that does something like the following:
INPUT: 3.5mm Jack Wireless Bluetooth-compatible 5.0 Receiver Adapter Wireless Aux Receiver Adapter For Headphone PC Music MP3 OUTPUT (60 chars max): 3.5mm Wireless Bluetooth 5.0 Adapter for Headphone
Except that the titles I am working with are in Portuguese. The program does this through a simple NLP algorithm that is based on grammatical segmentation of the title, meaning it essentially splits the title into noun chunks and then, if appropriate, intelligently merges these chunks together to produce new chunks.
After the title has been segmented, I then run it through a prioritization algorithm that is incomplete and, as of now, simply takes segment length and position into account to output the final shortened title, meaning longer segments that come first take priority. It might be relevant to emphasize that for 75% of the cases the non-ML algorithm that I have at the moment already picks the best choice but it doesn't use any of the many potentially relevant features I describe below and seems to me to still have a lot of room to improve.
I am extending this prioritization algorithm by adding more features to consider such as number of nouns/adjectives/connecting words/known words/unknown words, word frequency in large corpus of text such as Wikipedia, word frequency in all of the titles, word frequency in the product descriptions, etc. I also extracted every possible adjacent word combination to form millions of N-grams then kept only those that repeated 5 or more times to form a list of thousands of terms that range from 2-7 words in length that should, in theory, be more descriptive in nature due to their repetitiveness.
Now, these are WAY too many parameters for me to be able to decide how much each should be valued. This seems to me to be a perfect application for ML!
MY EXPERIENCE WITH ML:
How would I go about doing this? I have created a Neural Network from scratch many years ago following Andrew Ng's famous course but that is the extent of my experience with ML. I have also taken Kaggle's short introductory courses, also a while ago.
What is the best algorithm/library for this particular problem? Ideally I would like something extremely powerful, even if it's not completely trivial to set up, that I could run in Python.
How many training samples would be enough?
Any other tips?
Thank you all very much!!!