How do I go from number data to string data (NN)?

Question

I am quite confident with using number data with a neural network, but I want to use string data. My question is where do I begin? Obviously you cant times weights and strings together (x * w) because they are of different data types, so what should I do to the string data to turn it into numbers? If you're interested, here is my dataset. It's by no means the final version but here it is just to give you an example. It's based on spelling:

1. enormous..........| Difficulty: 2
2. rhythm..............| Difficulty: 3
3. hamster.............| Difficulty: 2
4. walk...................| Difficulty: 1
5. accommodate...| Difficulty: 3
6. blue....................| Difficulty: 1
7. projector.............| Difficulty: 1
8. regression..........| Difficulty: 2
9. go.......................| Difficulty: 1
10. playwright........| Difficulty: 3
11. weird.................| Difficulty: 3
12. conscience.......| Difficulty: 2

...so after training, when I will input a word, the network should return either 1, 2 or 3, depending on how hard the network thinks the word is to spell. To sum up, my question is: I am used to using number data with my projects instead of strings, so what steps should I take when creating a network based around strings for the first time? I use numpy only

Answer 1

There are various ways to handle string inputs to neural networks, but since you are trying to predict spelling difficulty, I suggests representing your words as a sequence of characters. This will preserve information about the particular spelling of the words including the order of the letters.

To represent a sequence of characters you can one-hot encode each character, so each word will be represented as a sequence of one-hot length 26 vectors.

To handle this kind of input I suggest either a 1D Convolutional Neural Network or some flavor of a Recurrent Neural Network.

If you choose a 1D CNN you will have to feed it fixed sized inputs. To do this choose a max word length, k, and either cut off or pad with zeros each input word to fit into a k*26 input matrix.

Here is an example of a CNN architecture you might use in Keras:

model = Sequential()
model.add(Convolution1D(nb_filter=32, filter_length=5, activation='relu', input_shape=(k, 26))
model.add(MaxPooling1D())
model.add(Convolution1D(nb_filter=32, filter_length=5, activation='relu')
model.add(MaxPooling1D())
model.add(Dropout(0.25))
model.add(Flatten())
model.add(Dense(64, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(3, activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

Answer 2

so what steps should I take when creating a network based around strings for the first time?

Neural networks work with numerical data. They also work best with relatively small floating point numbers, centred around zero. You can be less strict about that part, but you will often see the approach in neural networks of calculating the mean and standard deviation from the training data, for each feature, then converting all the features by doing x = (x - means) / stds (you want to store these scaling factors you used along with the network data, because you will want to re-used the same values when you use the network to make predictions later).

So what do you do if the input data is not already in this form? You prepare it in your code, just before using it to train or predict. It is a very common structure to see in machine learning scripts:

raw_features, raw_labels = load_from_disk( some_data_source )
all_features = convert_features( raw_features )
all_labels = convert_labels( raw_labels )
train_X, test_X, train_y, test_y = split_data( all_features, all_labels )
model = build_model( .... various model params ....)
model.fit( train_X, train_y )
test_predictions = model.predict( test_X )
report_accuracy( test_predictions, test_Y )

The above is rough pseudo code, so typically all the functions above have different names, or are multiple lines that do the same thing that you might not bother to encapsulate into a re-usable method if you are writing a quick script. The part I have shown that splits the features might be built in to the training function, and it is also common that the training process can use the test data to help monitor progress.

If the loading and conversion takes a long time, you might do it in a separate script and save the resulting NumPy array in a separate file to load it quicker next time.

So the part you are concerned about is how you might build a convert_features section of your code from the starting strings. The answer is to use whatever values you can extract from your strings that might be relevant. The string length might be a simple start i.e. len( text ) - but you can also look into any other measure you can figure out (e.g. number of vowels, which uncommon bi-grams are in the word). Deciding which features to try and testing between them is feature engineering, and this often involves some creativity. The important thing is that the features must all be numeric. For a neural network, you should also try and make them relatively small and/or convert them to have mean 0, standard deviation 1 before going to next stage.

When you use the network to make predictions later, you have to repeat most of the pipeline:

model = load_model( model_file_or_identifier )
raw_features, raw_labels = fetch_data( some_data_source )
X = convert_features( raw_features )
y = convert_labels( raw_labels )
predictions = model.predict( X )
report_predictions( X, y )