I'm trying to perform classification on a large dataset with mixed numerical and categorical features. The dataset is all semi-structured text, so everything is a String. Does anyone know of a library I could use to automatically identify numeric features (e.g. "width (in.)", which corresponds to the width in inches)? This is important because numeric and categorical features are encoded/handled differently in my analytics pipeline. I think this is basically a named entity recognition problem wherein the entity is a numeric variable or feature that exists in the data set as a String/text.
There are many good libraries for identifying number-like values, but identifying corresponding fieldnames is trickier and likely very problem-specific.
A purely data driven approach might look for co-occurrences with numbers, for example:
if [*number-like* capacity] or [capacity: *number-like*] occurs in > x% of
the instances where "capacity" occurs, then "capacity" can be guessed to be a label of number-like things.
Even the nature of these relationships (eg, whether it should be before, after, capitalised) could be learnt from existing known labels.
A practical approach, might aim to classify these strings as
unsure or even more classes, and then do human review of the latter. (There are some very tricky cases, for example, "capacity" is numerical, but "capacity type" would not be.)
As a starting list you can use:
Note that they are in the singular. For your domain considering finding other such lists.
If a string is a full match with one of those labels, you can consider it a
Your next concern is tolerant, fuzzy matching. You could treat '.' as a wildcard (so "in*" matches "inch"), or find the actual abbreviations of units like "inch". These you can label as
unsure and then review. Likewise if the word is simply contained in the string, eg "arch length" contains "length".
I think once you have done this you can make some refinements and add some special cases. Without seeing even a sample of your data, it is difficult to say more. If most of your strings are numerical units, then it may be easier to identify categorical units instead.