1
$\begingroup$

I am trying to do a project using NLP. My goal is to process Cyber Threat Intelligence articles like this to extract information such as actor’s name, malwares and tools used…

To do that I want to use NER. However, there isn’t training data available on the web. So I was wondering if I should process manually 10-20 articles to make my training data or if I could do something like taking only interesting lines such as “Rancor conducted at least two rounds of attacks intending to install Derusbi or KHRat malware on victim systems” in multiples articles and replacing the group name by another actor. This way I could deduplicate my training data by the number of known actors. But doing that, only the actor name is changing. So, the context is always the same.

I am wondering what’s the best way to train my model considering the quantity of training data available?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

I would start by training some very strong Named Entity classifier on available datasets for NER. One is the Annotated Corpus for Named Entity Recognition available on Kaggle.

Additionally, you can find a good list of datasets here. I know they have nothing to do with cybersecurity, but I think it's important to incorporate very different sources in a big, final dataset, in order to make a model that is good at generalizing on texts it has never seen before.

Another source of data for NER tasks is the annotated corpora available from nltk library, such as the free part of the Penn Treebank dataset, and Brown corpus.

Please beware that different datasets might use different categories for classification (i.e. the set of Named Entities can be different from dataset to dataset). Make sure you make all your data compatible to your classifier before training

After that, I suggest you to go with seq2seq models. Every state-of-the-art RNN is some form of seq2seq. Once you trained a classifier, you could try to annotate few articles manually, and check the performance of your model on those. It's time consuming, but I personally like these "qualitative" checks, I think they can tell you a lot.

Improve this answer
$\endgroup$
3
  • $\begingroup$ A stronger option would be to use transformers, but they are very difficult to train. Huggingface's transformers library has some pretrained models, but I haven't tried it yet. $\endgroup$
    – Leevo
    Commented Dec 22, 2019 at 11:28
  • $\begingroup$ What's the advantage of using seq2seq over CRF model in my case ? The seq2seq model seems to be more complex and maybe overkill for what i want to do. But I suppose that the training remain the same, so that's definitely a good answer. $\endgroup$
    – Kn0wledge
    Commented Dec 22, 2019 at 11:52
  • $\begingroup$ Yes, the training set would be the same for any ML model. I suggested seq2seq because that's the fanciest, state-of-the-art RNN, but if you find CRF good for your needs then you can go for it! $\endgroup$
    – Leevo
    Commented Dec 22, 2019 at 12:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.