# how to avoid tokenizing w/ sklearn feature extraction

I'm trying to analyze some machine log files and the column I'm looking at can have values like 'Part.C1.11.Reading Status'. I want to treat the complete string as one token and I don't want it to be split into 'Part', 'C1', '11' and 'Reading# and 'Status'.

I've got the vague feeling that the token_pattern is the parameter I need to adjust so I tried to specify the beginning and the end of a string like so:

from sklearn.feature_extraction.text import CountVectorizer
cvo = CountVectorizer(token_pattern='^$',lowercase=False) OriginCV = cvo.fit_transform(log['Message_Origin']).toarray()  However, the last line throws an error: ValueError: empty vocabulary; perhaps the documents only contain stop words I've also tried to explictly include dot and space in the token_pattern like so: cvo = CountVectorizer(lowercase=False, token_pattern=r"(?u)\b\w\w+\b|\.|\s")  Throws no errors but does not do the trick (no change except for an additional token '.') Not changing the default token_pattern does split the string at the spaces and colons though. I found this solution, which however modifies the string by removing e.g. the colons. Any other idea how to solve this? Thanks, Mark • did you read the documentation? scikit-learn.org/stable/modules/generated/… – Louic Jul 2 '19 at 13:10 • Yes, I did but couldn't find a hint as how to avoid tokenization at punctuation. It says "punctuation is completely ignored and always treated as a token separator" but I guess it is my lack of regex expertise to modify it correctly. – MarkH Jul 2 '19 at 13:38 • Meanwhile I tried token_pattern='.*' which get's me very close but it generates one additional token '' which does not exist. – MarkH Jul 2 '19 at 13:46 ## 4 Answers The default for token_pattern is (?u)\b\w\w+\b where \w\w+ translates to [a-zA-Z0-9_][a-zA-Z0-9_]+ (which can be written as [a-zA-Z0-9_]{2,}). This matches 2 or more alphanumeric characters (as defined between the square brackets). The \b matches word boundaries: anything that is not an alphanumeric character, next to something alphanumeric. This includes spaces and punctuation, so it also includes the dot and causes the separation. The (?u) activates unicode matching, but it is not so important for your question. If we also want to match the dot as part of the words, we can copy the default regexp, replace \w with its expanded version and add the dot to it: (?u)\b[a-zA-Z0-9_.]{2,}\b, as can be seen on this website. To add the word "Status" as well you could list it as an option explicitly: (?u)\b[a-zA-Z0-9_.]{2,}(=? Status)?\b. Alternatively, you can pass a function to the tokenizer parameter to define a more complex algorithm for tokenization. Edit: If you want to avoid tokenization completely (as your own answer states), the CountVectorizer, which is a token counter may not be the correct pre-processing step to choose: it will simply make everything a single token and return the count of 1. (Or maybe I misunderstood your question) • Thanks for the detailed answer. I was aware of the default token_pattern but couldn't e.g. figure out the meaning of (?u). You're right w/ regards to the usage of the vectorizer but I wanted to keep it generic since there are other columns in the log files where need the functionality and would like to recycle the code. Maybe not a great idea but if run time allows... – MarkH Jul 3 '19 at 9:01 You can change the tokenizer instead of the token pattern. The token pattern is used by the tokenizer. You can set as tokenizer any function that returns a list of string. For example, str.split tokenizer=str.split  From the official documentation, tokenizer callable, default=None Override the string tokenization step while preserving the preprocessing and n-grams generation steps. Only applies if analyzer == 'word'. If all your 'tokens' are characterized by beginning with Part and ending with Status and you can have anything in between, try token_pattern='^Part[.]*Status$' to specify the regex for any string that starts with Part and ends with Status, or try to adapt it to your need.

Side remark, if you do a Vectorizer, you would most likely need multiple tokens in each observation, because else it doesn't make much sense to Vectorize. I'm not sure whether you have multiple tokens per observation but if you have only one token per observation that would be some sort of OneHotEncoding, so in that case maybe look into the sklearn OneHotEcoder or pandas.get_dummies().

• Thanks, Joos! Unfortunately there are lot's of other tokens but they have no punctuation and create no problems so I did not mention them. With regards to observations: I am slicing my log files into overlapping windows and I am summarizing all observations within a given window. – MarkH Jul 2 '19 at 13:57
• How about doing a pre-treatment on your log-file where you replace all those 'complicated' tokens into a concatenated string without punctuation and then apply the CountVectorizer? – Joos Korstanje Jul 2 '19 at 14:04

I figured it out :-) Indeed it was my lack of regex expertise. What is happening when using

token_pattern='.*'


is that every combination of strings independent of the total length is considered a token. This includes 0 repetitions so I get one token for the message and one token for nothing (''). I modified it to

token_pattern='.+'


which excludes 0 repetitions.

• Won't that leave you with a single token (the entire sentence)? Is that what you wanted? – Louic Jul 2 '19 at 14:30