Add new attribute to the input of nerual network

Question

I have few students'(38) sequential data. For example, one student sequential data is like:

A
B
B
A
C
D
E
.
.

Each student has different length of sequences and their used letter was within this 5 letter. So, to predict the next letter/letters I used feed-forward neural network and recurrent-neural network(LSTM). I got reasonable accuracy with these models.

Now, I want to add a new attribute of the students to the input. Suppose, the students are from four different departments. I added the one hot vector of a particular department to the corresponding student.

I used Leave one out as a cross-validation. But for this new input, the testing accuracy is always 100% which is not possible.

My question is, Are there any methods for adding a new attribute to the sequential data?

Answer 1

When adding an attribute, increase the number of neurons in the input layer, then train the network again.

Having 100% accuracy may indicate that the model suffers over-fitting. To solve the problem:

1- make sure that the model doesn't see the test set at training time

2- If you have sufficient number of records, I would recommend going for K-folds validations rather than leave-one out cross-validation (for example 70% training 30% testing)

3- if the number of records is insufficient, (such as in your case: 38 records at best won't allow you go deep), either

• try to get more more data
• or simplify your model (maybe just 1-2 layers with 1-3 neurons!)
• or try machine learning methods other than neural networks. plz see (https://www.cs.sfu.ca/~jpei/publications/Sequence%20Classification.pdf).

Answer 2

My question is, Are there any methods for adding a new attribute to the sequential data?

The method that you are looking for is exactly called Transfer Learning in the Machine Learning framework.

In other words, it refers to the case where you train your model based on a situation but you want to generalize to broader or different cases, like the ones where extra attributes are available.

Make sure that you read this helpful post too.