I'm trying to analyze some machine log files and the column I'm looking at can have values like 'Part.C1.11.Reading Status'. I want to treat the complete string as one token and I don't want it to be split into 'Part', 'C1', '11' and 'Reading# and 'Status'.
I've got the vague feeling that the token_pattern is the parameter I need to adjust so I tried to specify the beginning and the end of a string like so:
from sklearn.feature_extraction.text import CountVectorizer cvo = CountVectorizer(token_pattern='^$',lowercase=False) OriginCV = cvo.fit_transform(log['Message_Origin']).toarray()
However, the last line throws an error: ValueError: empty vocabulary; perhaps the documents only contain stop words
I've also tried to explictly include dot and space in the token_pattern like so:
cvo = CountVectorizer(lowercase=False, token_pattern=r"(?u)\b\w\w+\b|\.|\s")
Throws no errors but does not do the trick (no change except for an additional token '.')
Not changing the default token_pattern does split the string at the spaces and colons though. I found this solution, which however modifies the string by removing e.g. the colons. Any other idea how to solve this?