Skip to main content
Completely overhauled answer in response to comments.
Source Link
fswings
  • 388
  • 2
  • 7

Consider what are the inputs toBased on your model. You refer to pcap filescomments and sample provided, containing ~50 sequences.

1st Interpretation

As I understand fromwoud summary/rephrase your question, you want to have 1 label per pcap file (representing one type of Internet activity). The number of sequences therefore doesn't matter. as:

How to create a Machine Learning model to classify Network Traffic?

What's needed is that whole pcap file contents is presented to the model with a single labelThe samples you have provided, sosuggest in effect I would extract out all 50 sequencesgeneral at least two possible approaches, combine themone similar to (string concatenation)Text Classification and store them in a single cellthe other like any generic supervised learning example.

2nd Interpretation

If each sequence independently demonstratesThe samples show as a typefunction of Internet activitytime, then each sequence can occupythe various packets to and from a row withdestination, which may not be the label isame length or duration (i.e. all 50 rows have the same labelnumber of captured packets could vary). I don't think you mean this given my experience with wireshark Both approaches require an input (typically referred to as features) and an output (typically called a target).

Other Considerations

In principle this approach will enable youThe first step is to set uporganise the data such that one file/entry/capture containing multiple packets maps to a single classification model. In practice it'll be much harderlabel. For exampleNetwork traffic captured using wireshark, Wireshark can provide both an absolute and relative time sequence. You haveor tcpdump or similar tools use line breaks to consider whether how quickly thedenote different packets arrive and change is related. These can be difficult to process for most ML frameworks so in the internet activityfirst instance these can be removed or replaced with something else. So the datasetAn example output may require substantial cleaning and modification to reflect what you want.look the following:

InputOutput
1 0.000 192.168.0.5 192.168.0.2 HTTP 250 Get /HTTP/1.1 ....Social Media
1 0.000 192.168.0.5 192.168.0.2 HTTP 250 Get /HTTP/1.1 ....Streaming Video

The key challengeOnce data is in this format you can probably follow a Text Classification type approach 1. You need to be careful how the sequencesthese "words" are combined and presentedtranslated to the ML model itself. An alternative strategy is if you always have around 50 sequences Most ML models can't read, isso need to place each sequencetext to translated to numbers a process of vectorisation or tokenisation. You need to be careful of using some methods in Natural Language Processing (NLP) as they use existing models to map words to numbers. In your case, HTTP, TCP etc are not typically used in natural languages and may have no mapping 2.

I believe this answers your question.

However, as a columnSubject Matter Expert (SME) in network traffic, you may identify aspects in the data that you feel are important in classifying it. For example:

  • Does the actual source and destination IP influence the classification? If not, you might to replace it with perhaps a direction or replace the IP addresses with the text "src" and "dest" instead.
  • Does the sequence number matter?
  • Does how long something took to respond a factor?
  • If a HTTP Code is received, does that a have strong influence on the outcome?

In all casesIf so, howyou could add them as extra columns e.g.

TrafficHTTP CodeOutput
1 0.000 192.168.0.5 192.168.0.2 HTTP 250 Get /HTTP/1.1 ....200Social Media
1 0.000 192.168.0.5 192.168.0.2 HTTP 250 Get /HTTP/1.1 ....400Streaming Video

The idea of manipulating the text is presentedinput data to help the model will be the challenging aspect.is called feature engineering.


 

Edit: FormattingBased on my experience, using raw network traffic is likely to cause most ML models problems, without some modification or feature extraction.

Consider what are the inputs to your model. You refer to pcap files, containing ~50 sequences.

1st Interpretation

As I understand from your question, you want to have 1 label per pcap file (representing one type of Internet activity). The number of sequences therefore doesn't matter.

What's needed is that whole pcap file contents is presented to the model with a single label, so in effect I would extract out all 50 sequences, combine them (string concatenation) and store them in a single cell.

2nd Interpretation

If each sequence independently demonstrates a type of Internet activity, then each sequence can occupy a row with the label i.e. all 50 rows have the same label. I don't think you mean this given my experience with wireshark.

Other Considerations

In principle this approach will enable you to set up a classification model. In practice it'll be much harder. For example, Wireshark can provide both an absolute and relative time sequence. You have to consider whether how quickly the packets arrive and change is related to the internet activity. So the dataset may require substantial cleaning and modification to reflect what you want.

The key challenge is how the sequences are combined and presented to the model. An alternative strategy is if you always have around 50 sequences, is to place each sequence in a column.

In all cases, how the text is presented to the model will be the challenging aspect.


 

Edit: Formatting.

Based on your comments and sample provided, I woud summary/rephrase your question as:

How to create a Machine Learning model to classify Network Traffic?

The samples you have provided, suggest in general at least two possible approaches, one similar to Text Classification and the other like any generic supervised learning example.

The samples show as a function of time, the various packets to and from a destination, which may not be the same length or duration (i.e. the number of captured packets could vary). Both approaches require an input (typically referred to as features) and an output (typically called a target).

The first step is to organise the data such that one file/entry/capture containing multiple packets maps to a single classification label. Network traffic captured using wireshark, or tcpdump or similar tools use line breaks to denote different packets. These can be difficult to process for most ML frameworks so in the first instance these can be removed or replaced with something else. An example output may look the following:

InputOutput
1 0.000 192.168.0.5 192.168.0.2 HTTP 250 Get /HTTP/1.1 ....Social Media
1 0.000 192.168.0.5 192.168.0.2 HTTP 250 Get /HTTP/1.1 ....Streaming Video

Once data is in this format you can probably follow a Text Classification type approach 1. You need to be careful how these "words" are translated to the ML model itself. Most ML models can't read, so need to text to translated to numbers a process of vectorisation or tokenisation. You need to be careful of using some methods in Natural Language Processing (NLP) as they use existing models to map words to numbers. In your case, HTTP, TCP etc are not typically used in natural languages and may have no mapping 2.

I believe this answers your question.

However, as a Subject Matter Expert (SME) in network traffic, you may identify aspects in the data that you feel are important in classifying it. For example:

  • Does the actual source and destination IP influence the classification? If not, you might to replace it with perhaps a direction or replace the IP addresses with the text "src" and "dest" instead.
  • Does the sequence number matter?
  • Does how long something took to respond a factor?
  • If a HTTP Code is received, does that a have strong influence on the outcome?

If so, you could add them as extra columns e.g.

TrafficHTTP CodeOutput
1 0.000 192.168.0.5 192.168.0.2 HTTP 250 Get /HTTP/1.1 ....200Social Media
1 0.000 192.168.0.5 192.168.0.2 HTTP 250 Get /HTTP/1.1 ....400Streaming Video

The idea of manipulating the input data to help the model is called feature engineering.

Based on my experience, using raw network traffic is likely to cause most ML models problems, without some modification or feature extraction.

Formatting.
Source Link
fswings
  • 388
  • 2
  • 7

Consider what are the inputs to your model. You refer to pcap files, containing ~50 sequences.

1st Interpretation

1st Interpretation As As I understand from your question, you want to have 1 label per pcap file (representing one type of Internet activity). The number of sequences therefore doesn't matter.

What's needed is that whole pcap file contents is presented to the model with a single label, so in effect I would extract out all 50 sequences, combine them (string concatenation) and store them in a single cell.

2nd Interpretation

2nd Interpretation If If each sequence independently demonstrates a type of Internet activity, then each sequence can occupy a row with the label i.e. all 50 rows have the same label. I don't think you mean this given my experience with wireshark.

Other Considerations

Other Considerations In In principle this approach will enable you to set up a classification model. In practice it'll be much harder. For example, Wireshark can provide both an absolute and relative time sequence. You have to consider whether how quickly the packets arrive and change is related to the internet activity. So the dataset may require substantial cleaning and modification to reflect what you want.

The key challenge is how the sequences are combined and presented to the model. An alternative strategy is if you always have around 50 sequences, is to place each sequence in a column.

In all cases, how the text is presented to the model will be the challenging aspect.


Edit: Formatting.

Consider what are the inputs to your model. You refer to pcap files, containing ~50 sequences.

1st Interpretation As I understand from your question, you want to have 1 label per pcap file (representing one type of Internet activity). The number of sequences therefore doesn't matter.

What's needed is that whole pcap file contents is presented to the model with a single label, so in effect I would extract out all 50 sequences, combine them (string concatenation) and store them in a single cell.

2nd Interpretation If each sequence independently demonstrates a type of Internet activity, then each sequence can occupy a row with the label i.e. all 50 rows have the same label. I don't think you mean this given my experience with wireshark.

Other Considerations In principle this approach will enable you to set up a classification model. In practice it'll be much harder. For example, Wireshark can provide both an absolute and relative time sequence. You have to consider whether how quickly the packets arrive and change is related to the internet activity. So the dataset may require substantial cleaning and modification to reflect what you want.

The key challenge is how the sequences are combined and presented to the model. An alternative strategy is if you always have around 50 sequences, is to place each sequence in a column.

In all cases, how the text is presented to the model will be the challenging aspect.

Consider what are the inputs to your model. You refer to pcap files, containing ~50 sequences.

1st Interpretation

As I understand from your question, you want to have 1 label per pcap file (representing one type of Internet activity). The number of sequences therefore doesn't matter.

What's needed is that whole pcap file contents is presented to the model with a single label, so in effect I would extract out all 50 sequences, combine them (string concatenation) and store them in a single cell.

2nd Interpretation

If each sequence independently demonstrates a type of Internet activity, then each sequence can occupy a row with the label i.e. all 50 rows have the same label. I don't think you mean this given my experience with wireshark.

Other Considerations

In principle this approach will enable you to set up a classification model. In practice it'll be much harder. For example, Wireshark can provide both an absolute and relative time sequence. You have to consider whether how quickly the packets arrive and change is related to the internet activity. So the dataset may require substantial cleaning and modification to reflect what you want.

The key challenge is how the sequences are combined and presented to the model. An alternative strategy is if you always have around 50 sequences, is to place each sequence in a column.

In all cases, how the text is presented to the model will be the challenging aspect.


Edit: Formatting.

Source Link
fswings
  • 388
  • 2
  • 7

Consider what are the inputs to your model. You refer to pcap files, containing ~50 sequences.

1st Interpretation As I understand from your question, you want to have 1 label per pcap file (representing one type of Internet activity). The number of sequences therefore doesn't matter.

What's needed is that whole pcap file contents is presented to the model with a single label, so in effect I would extract out all 50 sequences, combine them (string concatenation) and store them in a single cell.

2nd Interpretation If each sequence independently demonstrates a type of Internet activity, then each sequence can occupy a row with the label i.e. all 50 rows have the same label. I don't think you mean this given my experience with wireshark.

Other Considerations In principle this approach will enable you to set up a classification model. In practice it'll be much harder. For example, Wireshark can provide both an absolute and relative time sequence. You have to consider whether how quickly the packets arrive and change is related to the internet activity. So the dataset may require substantial cleaning and modification to reflect what you want.

The key challenge is how the sequences are combined and presented to the model. An alternative strategy is if you always have around 50 sequences, is to place each sequence in a column.

In all cases, how the text is presented to the model will be the challenging aspect.