I'm trying to perform classification on a large dataset with mixed numerical and categorical features. The dataset is all semi-structured text, so everything is a String. Does anyone know of a library I could use to automatically identify numeric features (e.g. "width (in.)", which corresponds to the width in inches)? This is important because numeric and categorical features are encoded/handled differently in my analytics pipeline. I think this is basically a named entity recognition problem wherein the entity is a numeric variable or feature that exists in the data set as a String/text.

  • $\begingroup$ I would do that with regex. Could you post some sample data? You can find numeric and non numeric patterns and look ahead and look behind with regex. For the second time please post some data. OK so then you are already able to extract the terms in the < > ? $\endgroup$ – paparazzo Oct 23 '15 at 19:11
  • 1
    $\begingroup$ Depending on what your data looks like, you might be able to simply use the regular expression r'^\d+\s+?(\S+\.)$' (see rubular.com/r/ZS7vZaR3XD, for an interactive example). Can you post some example rows from your data? $\endgroup$ – Kyle. Oct 23 '15 at 19:25
  • $\begingroup$ it really is just a line of text with terms separated by a "<" symbol. Each term is a String and corresponds to a feature. The problem is that the number of terms per observation (line of text) is about 2K terms (Strings). Some of those are numeric features, e.g. "width (in.)", "Height (in.)", "Capacity" etc, while other terms (Strings) correspond to categorical feature, e.g. "connection type"..etc. If the number of terms per observation is small/manageable, I would just do this manually, but with 2K I think I should use some kind of NER for recognizing numeric features like measurements. $\endgroup$ – Kai Oct 23 '15 at 19:44
  • $\begingroup$ I should add that this is needed to avoid numeric features being treated as categorical by something like DictVectorizer() from sickit-learn which would make the data extremely wide and is basically a wrong way of encoding features. $\endgroup$ – Kai Oct 23 '15 at 19:48
  • $\begingroup$ I appreciate your help and caring enough to ask again. I can't post any data. However, if you read my comment above to Kyle, you'll know exactly what I'm asking. Or simply answer this: given a line of text consisting of terms separated by a known delimiter ('<'), you can get the individual terms with Split('<'). Then the question is HOW do you find out which of these text terms refer to a numeric feature, e.g. "width" and which refer to a categorical feature, e.g. "connection type". That's it. $\endgroup$ – Kai Oct 23 '15 at 20:02

There are many good libraries for identifying number-like values, but identifying corresponding fieldnames is trickier and likely very problem-specific.

A purely data driven approach might look for co-occurrences with numbers, for example:

if [*number-like* capacity] or [capacity: *number-like*] occurs in > x% of 

the instances where "capacity" occurs, then "capacity" can be guessed to be a label of number-like things.

Even the nature of these relationships (eg, whether it should be before, after, capitalised) could be learnt from existing known labels.

This can be done with Google BigQuery using the public tri-gram sample data. But I assume that's out of scope and impractical here.

A practical approach, might aim to classify these strings as numerical_units, categorical_units or unsure or even more classes, and then do human review of the latter. (There are some very tricky cases, for example, "capacity" is numerical, but "capacity type" would not be.)

As a starting list you can use:
Note that they are in the singular. For your domain considering finding other such lists.

If a string is a full match with one of those labels, you can consider it a numerical_unit. Your next concern is tolerant, fuzzy matching. You could treat '.' as a wildcard (so "in*" matches "inch"), or find the actual abbreviations of units like "inch". These you can label as unsure and then review. Likewise if the word is simply contained in the string, eg "arch length" contains "length".

I think once you have done this you can make some refinements and add some special cases. Without seeing even a sample of your data, it is difficult to say more. If most of your strings are numerical units, then it may be easier to identify categorical units instead.

| improve this answer | |
  • $\begingroup$ Ok, thanks for the details. Let me ask the question differently; machine learning is being applied to wide datasets (# of features is large, thousands or more). There MUST be standard tools, libraries etc that people use in order to distinguish between numeric and categorical features .. I doubt people are going through the data manually or even with regular expressions designating the type of each feature... this is an important step before applying your classifier, there's no escaping it, hence I think there must be existing tools to do this. IF not, what do Machine Learning Gurus do? $\endgroup$ – Kai Oct 25 '15 at 23:50
  • $\begingroup$ Yes, there are good standard tools for it, but they are based on input of feature values (many per field), not the arbitrary feature name. If you do indeed have a list of such values (eg "4.2"...), then it's easy. If the values are mixed (eg "4.2 in") then it's also solvable. But in your examples you give strings that do not contain numbers but rather look like column names. In same cases these could be written as "length_inches" or "leninches" or even "field 32" - much less solvable. $\endgroup$ – Adam Bittlingmayer Oct 26 '15 at 11:41
  • $\begingroup$ OK, so it appears that this step of recognizing numeric vs categorical is highly application dependent; if the problem space guarantees meaningful col names ("width") then we can use some NER API, otherwise you have to create some kind of algorithm that looks at the feature values to recognize feature type... there is no universal solution. Can you point me t these "standard tools" for recognizing feature type from feature values? as in the example you give where the feature value is a String: "4.2 in." ? I am inclined to write this myself now, but I still like to see existing solutions. $\endgroup$ – Kai Oct 26 '15 at 14:05
  • $\begingroup$ Pandas will do it. I would preproc out the " in." though. $\endgroup$ – Adam Bittlingmayer Oct 26 '15 at 19:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.