1
$\begingroup$

I have a dataset which includes one column with a URL and another column with value 0 or 1 indicating if it is a phishing link.

enter image description here

I want to process this dataset using LSTM. Do I need to first convert the string column into some other data type?

The model I'm hoping to use would be something like below:

# Model building using the Sequential API
model3 = Sequential()
model3.add(Dense(10, activation='relu',
          kernel_initializer='uniform',input_dim=x3.shape[1]))

# Add a LSTM layer with 128 internal units.
model3.add(layers.LSTM(128))

# Add a Dense layer with 10 units.
model3.add(layers.Dense(10))
$\endgroup$
2
  • 2
    $\begingroup$ Yes; tokenisation and all needs to be done to convert the words to be represented as id's (ints) $\endgroup$
    – Aditya
    Mar 6 '20 at 22:01
  • $\begingroup$ @Aditya doesn't tokenization (at least in my experience with nltk) turn a sentence into an array of words? What tokenizer would I use to turn a URL into a int? $\endgroup$ Mar 6 '20 at 22:19
1
+50
$\begingroup$

Do I need to first convert the string column into some other data type?

Yes, it's very important. Neural Networks don't take raw words and/or letters as inputs. Textual information must be processed numerically in order to be fed into the model.

I thought about it, and came up with three things you could do:

  1. Use one-hot encoding, as it was suggested. I don't prefer this option: one-hot encoding generates very sparse matrices, meaning very little variation in almost everything dimension at each step. Moreover, the number of websites is absurdly high, it can be unmanageable through one-hot encoding. Additionally, it's not robust: your model have to be tested on unseen data, that by definition couldn't be operationalized with that technique. Also, some websites might be more or less similar to others, they play more or less similar roles for your model; but one-hot encodind fails at representing their differences: they will all look equally different if you compute any distance measure between them.

  2. Assuming you are working with sequences, my main suggestion is to represent web USLs with embedding vectors. Just as words can be translated into embedding vectors in NLP (it's the case of word2vec or glove), you can apply the same technique to represent the "relative meaning" of websites. In this way you can: a) keep the number of dimensions at bay, and b) have a non-sparse matrix that your models need. You would end up with a web2vec model, that fits perfectly with RNNs. Moreover, the meaning of new, unseen websites could be learned effortlessly by the model. This can be done with libraries such as gensim, or using Keras Embedding() layers.

  3. A more extreme, time consuming option is to use character embeddings. This is going to be useful only if you thing the name of the website itself contains the relevant information for your tasks. I'm not sure if it's the case. Character-level embeddings produce very sofisticated models, but they are more computationally expensive than the others. And I'm not sure they would be useful in your case.

If I had to choose, I'd go for option 2.

Finally, let me close with a hint: do not put Dense() layers before LSTM() layers. First, Recurrent layers process sequential data, then their output is sent to dense layers that execute the prediction. The output layer should have either one node + Sigmoid activation + Binary Crossentropy loss, or two nodes + Softmax activation + Categorical Crossentropy loss. As you prefer.

$\endgroup$
2
  • $\begingroup$ Thanks, I found my computer doesn't have enough RAM for option #1. If I understand right Keras Embedding() layers allow me to input text directly by including the conversion process as part of the model? $\endgroup$ Mar 9 '20 at 0:15
  • 1
    $\begingroup$ Keras Embedding() layers receives numbers. I suggest you to convert each URL into a unique numerical index, and feed this sequence of indexes into the layer. It will "learn" to represent embedding vectors. You can do manually with list comprehensions, or with sklearn's LabelEncoder that does the job for you. $\endgroup$
    – Leevo
    Mar 9 '20 at 8:13
1
$\begingroup$

You can use one-hot encoding to encode your domains as char arrays. Then your training samples should have dimension (samples, longest domain length, all chars used). Here is a code sample:

import pandas as pd
import numpy as np
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, LSTM

# make some fake samples
urls = ['https://datascience.stackexchange.com/', 
        'https://github.com/', 
        'www.google.com/']

labels = [1,0,1]
df = pd.DataFrame(zip(urls, labels), 
                  columns=['domain', 'label'])

# Make it all to a long string
concat_domains = '\n'.join(df['domain']).lower()

# Find all unique characters by using set()
chars = sorted(list(set(concat_domains)))
num_chars = len(chars)

# Build translation dictionaries, 'a' -> 0, 0 -> 'a'
char2idx = dict((c, i) for i, c in enumerate(chars))
idx2char = dict((i, c) for i, c in enumerate(chars))

# Use longest name length as our sequence window
max_sequence_length = max([len(name) for name in df['domain']])

# build dataset with domains as one-hot encoded chars
X = np.zeros((df.shape[0], max_sequence_length, num_chars), dtype=np.bool)
y = df['label'].values

for i, sequence in enumerate(df['domain']):
    for j, char in enumerate(sequence):
        X[i, j, char2idx[char]] = 1


# build a model with input dim: (length of longest domain name, number of unique chars found)
model = Sequential()
model.add(LSTM(64, input_shape=(max_sequence_length, num_chars)))
model.add(Dense(units=1, activation='sigmoid'))

model.compile(loss='binary_crossentropy', optimizer='adam')

# train for 10 epochs
model.fit(X, y, epochs=10)
$\endgroup$
4
  • $\begingroup$ Thanks, I'm getting KeyError: 'W' on the line X[i, j, char2idx[char]] = 1. Any idea what caused this? imgur.com/a/Ap0Klfh $\endgroup$ Mar 8 '20 at 23:19
  • $\begingroup$ I was able to fix the error by removing .lower() on concat_domains however the resulting one-hot encoding requires more RAM than my computer has. $\endgroup$ Mar 9 '20 at 0:09
  • 1
    $\begingroup$ You need something that preprocesses batches on the fly to avoid keeping it all in RAM. In Keras that is done by making a custom generator which is fit with fit_generator $\endgroup$ Mar 9 '20 at 6:29
  • $\begingroup$ But other things that can reduce dimensionality and RAM is. 1. Converting everything to lowercase. 2. Manually setting max_sequence_length and clipping all domain names at that length. $\endgroup$ Mar 9 '20 at 7:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.