Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
Confused as to when to use StringIndexer vs StringIndexer+OneHotEncoder.
The OneHotEncoder docs say
For string type input data, it is common to encode categorical features using StringIndexer first.
In what situations would I want to take the extra step of transforming StringIndex'ed output to one-hot encoded features?
it depends on the features you want to use (and their datatypes).
In the Docs it says:
One-hot encoding maps a categorical feature, represented as a label index, to a binary vector with at most a single one-value
This means that:
if your categorical feature is already "represented as a label index", you don't need to use StringIndexer first. Instead, you can directly apply one-hot encoding.
On the other hand:
if your categorical feature is, e.g. represented as string values, it becomes necessary to use StringIndexer first to convert the string values into label indices (numeric values).
In the example from the OneHotEncoder-Docs you can see that the DataFrame that is being created already has features of DoubleType and thus StringIndexer is not applied before one-hot encoding.
StringIndexer indexes your categorical variables into numbers, that require no specific order. Essentially, maps your strings to numbers, and keeps track of it as metadata attached to the DataFrame. You can also use StringIndexer to apply strings to columns that currently aren’t of string type; which once converted, then are indexed as strings.
I don’t see many people using StringIndexer, when indexing, but see OneHot as the primary form for categorical indexing.
