4
$\begingroup$

I am learning machine learning and I'm trying to implement a solution for a real problem: predict from a human sentence what programming function he/she is trying to do.

I have a series of programming functions related with a series of descriptions (there can be n > 0 descriptions for each unique function).

I created a neural network and a bag of words model trying to convert a human sentence "we get the data from the database" to a programming function. So far it works with very easy examples but not with my real data.

Something like this works:

"description"     | programming function
lala lolo lulu ka | function1
lala lolo lulu ke | function1
lala lolo lulu ko | function1
lala lele lili ka | function2
lala lele lili ki | function2
lala lele lili ko | function2

Every word in the description is converted to an neuron-input (with value of 1 if present and 0 if not present) and every possible function is converted to a neuron-output.

I'm using pyBrain with back propagation and an error threshold of 0.005. The neural network has three layers, and the middle one has lenght: number of possible words + number of possible programming functions (this is kind of arbitrary).

I know full text search or auto-complete is probably a better alternative for this task, but I'm just experimenting with machine learning and I'd like this to work if possible. In my real data I have 1000 descriptions related with ~500 functions.

So my question is:

  • Is bag of words + neural networks a good approach for solving this?
  • Maybe Word2vec is a better option?
  • If neither is good, is there any known machine learning approach that could work with something like that?
$\endgroup$
4
$\begingroup$

Yes this problem is extremely well suited for machine learning. However, I think you should be careful as to which algorithms you tend to use.

A machine learning algorithm should be structured as follows: feature extraction and then your model. These are two things that should be done separately.

Feature Extraction

This is the bag of words, n_grams and word2vec. These are all good choices for text examples. I think bag of words is a good choice in your case. However, if this generates a sparse matrix then maybe n_grams can be better. You can test all 3 methods.

The Model

Theoretically, the more parameters in your model the more data you need to train it sufficiently otherwise you will retain a large amount of bias. This means a high error rate. Neural networks tend to have a very high number of parameters. Thus they require a lot of data to be trained.

But, you have 1000 instances!!! Yes. But, you also have 500 classes. So imagine you have a very young child and you want him to be able to correctly classify 500 different types of images. Then you can't just show the kid 2 different examples of each class for him to truly understand what each class really means.

As a very general rule of thumb, the number of instances you need to train a model increases exponentially with number of classes. So you will need a MASSIVE amount of data to properly train a neural network model.

I would suggest a less intensive model. Moreover, looking at your example, it seems that the classes should be linearly separable. So you can use something really simple, linear regression, logistic regression, naive bayes or knn. These methods would do MUCH MUCH better than a neural network.

My Suggestion

I would start with bag of words and then use knn. This should be a good starting point.

A neural network is 0% recommended for the amount of data you have.

|improve this answer|||||
$\endgroup$
  • 1
    $\begingroup$ Thanks a lot! I did a short example and it's working better than my NN approach. And I learned about knn in the process :) $\endgroup$ – hhaamm Mar 17 '17 at 22:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.