1
$\begingroup$

I've seen other similar questions and followed their solutions, to little improvement. I'm making a model to identify the gender of names. As training data I'm using a list of baby names found here: https://www.ssa.gov/oact/babynames/limits.html. I extracted this data to a new data frame, keeping only one instance of those names occurring more than once, and sorted randomly.

Each name string in a column was converted to a numeric array of lenght max_len and normalized by the function:

def text_to_numeric(column, max_len):
    word_characters = []
    for word in column:
        word_characters.append([c for c in word])

    letters_kept = 25
    tokenizer = tf.keras.preprocessing.text.Tokenizer(num_words=letters_kept, oov_token='<UNK>')
    tokenizer.fit_on_texts(word_characters)

    word_sequence = tokenizer.texts_to_sequences(word_characters)
    words_pre = tf.keras.preprocessing.sequence.pad_sequences(word_sequence, maxlen=max_len,padding="pre")
    words_pre = tf.keras.utils.normalize(input_data)

    return list(words_pre)

The expected output is an array of 2 element list where [1,0] means “Male” and [0,1] means “Female”. The model, where data_file contains processed names and labels, looks like this:

input_length, input_data, output_data = data_reader(data_file)
model = tf.keras.Sequential()
model.add(tf.keras.layers.Dense(100, input_dim=input_length, activation='relu'))
model.add(tf.keras.layers.Dense(100, activation='relu'))
model.add(tf.keras.layers.Dense(2, activation='softmax'))

model.compile(loss='binary_crossentropy', optimizer="adam", metrics=['accuracy'])

model.fit(input_data, output_data, epochs=30, verbose=1, validation_split=0.1)

No matter what, I always get an accuracy of around 75%. I don't know how to choose the model parameters, but I’ve tried with many combinations and the accuracy changes little. So far I've tried: normalizing input, balancing the input dataset so there are the same number of men and women, changing the optimizer, defining an optimizer and change the learning rate, changing layer number, nodes per layer and activation function, increasing number of epochs.

All of this with no significant change in the model's accuracy. Am I missing something or doing something completely wrong? Is this accuracy as good as it gets?

$\endgroup$
1
$\begingroup$

When you can't improve the model any more, improve the data.

  • are there any unisex names in your dataframe? Those will obviously hurt your accuracy. Either toss them out for now or do multi label binary classification.

  • Visualize your data: are the labels clearly separated? Which are the top losses?

  • I just saw that the data has the popularity of a name. I think it would make sense to try adding this popularity as a weight of each datapoint when training the model. Intuitively, you want your model to understand what does it mean for a name to be feminine. And "Anna" (score=2604) is much more representative of this meaning than "Jewel" (score=5), don't you think?

PS: I just saw that "Joe" is included in the list of feminine names.

$\endgroup$
  • $\begingroup$ Thanks, good advices. What are labels "clearly separated" and "top losses"? How could I add the pupularity as a weight? $\endgroup$ – Chegon Jun 5 '20 at 15:40
  • 1
    $\begingroup$ Regarding separation of the labels: visualize your datapoints in a 2 dimensional chart with a color for F and a color for M. To be able to do this, you can perform a dimensionality reduction, for example using PCA. If the two colored groups overlap too much then that means your labels are not really separable $\endgroup$ – Guillermo Mosse Jun 5 '20 at 15:50
  • 1
    $\begingroup$ About top losses: you can get the datapoints from your test set for which the classifier most confidently predicts that they are the incorrect label. For example, maybe "Michel" is a masculine name but your classifier predicts with that is 0.89 feminine. Grab the ones with higher probability. Then make sense of why the model fails on those. $\endgroup$ – Guillermo Mosse Jun 5 '20 at 15:54
  • 1
    $\begingroup$ About the popularity thing: consider using the sample_weight parameter when building your model $\endgroup$ – Guillermo Mosse Jun 5 '20 at 15:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.