1
$\begingroup$

Confused as to when to use StringIndexer vs StringIndexer+OneHotEncoder.

The OneHotEncoder docs say

For string type input data, it is common to encode categorical features using StringIndexer first.

In what situations would I want to take the extra step of transforming StringIndex'ed output to one-hot encoded features?

$\endgroup$

2 Answers 2

1
$\begingroup$

it depends on the features you want to use (and their datatypes).

In the Docs it says:

One-hot encoding maps a categorical feature, represented as a label index, to a binary vector with at most a single one-value

This means that:

if your categorical feature is already "represented as a label index", you don't need to use StringIndexer first. Instead, you can directly apply one-hot encoding.

On the other hand:

if your categorical feature is, e.g. represented as string values, it becomes necessary to use StringIndexer first to convert the string values into label indices (numeric values).

In the example from the OneHotEncoder-Docs you can see that the DataFrame that is being created already has features of DoubleType and thus StringIndexer is not applied before one-hot encoding.

$\endgroup$
1
$\begingroup$

StringIndexer indexes your categorical variables into numbers, that require no specific order. Essentially, maps your strings to numbers, and keeps track of it as metadata attached to the DataFrame. You can also use StringIndexer to apply strings to columns that currently aren’t of string type; which once converted, then are indexed as strings.

I don’t see many people using StringIndexer, when indexing, but see OneHot as the primary form for categorical indexing.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.