Is One Hot Encoding Vectorized?

Question

Sorry for such a weird question - I do not even know if this makes sense, however I thought of this during my intro to Python course at uni and have wondered about it since.

So I have some experience with Python for data science, but have never taken a structured course. I am in an intro class, and on the first day we were asked to count how many times a certain name appeared in a list. Obviously the answer I came up with was to initialize a variable to zero then iterate through the list and add one to itself whenever the i'th name in the list was equal to the target name.

However, in Andrew Ng's deep learning courses, there is a heavy emphasis on vectorizing our calculations whenever possible. This got me wondering how I could vectorize this task.

What I came up with, in an ideal scenario, is to create a vector where each the target name is replaced with 1, and the other names are replaced with 0. Then I could just take the sum of the vector and I would have my answer. The problem with this is the only way I know how to create said vector is to iterate through the original list, defeating the purpose this way of solving the problem.

Anyways, while I understand that creating this vector is not exactly one hot encoding, is there any way to vectorize this process? Is one hot encoding done iteratively. If not, are there other examples where basic iterative tasks can be turned into tasks able to be processed in parallel?

Sorry if this is a dumb question or does not make any sense - just something I was curious about as a noob to base Python.

Answer 1

You can read the sklearn documentation here. If you click in source you can check the code.

If you want to do one hot encoding you can do

    >>> from sklearn.preprocessing import OneHotEncoder
    >>> enc = OneHotEncoder(handle_unknown='ignore')
    >>> X = [['Male', 1], ['Female', 3], ['Female', 2]]
    >>> enc.fit(X)
    OneHotEncoder(handle_unknown='ignore')
    >>> enc.categories_
    [array(['Female', 'Male'], dtype=object), array([1, 2, 3], dtype=object)]
    >>> enc.transform([['Female', 1], ['Male', 4]]).toarray()

You should only try to implement this if you are an experimented developer. A couple lines of code that implement OHE in a non optimal way is:

for col in columns_encoding:
    variables = df[col].unique()
    for v in variables:
        df[col + str(v)] = [1 if row == v else 0 for row in df[col].values]

where columns_encoding is a list of the columns you want to encode and df your dataframe

Answer 2

Apart from performance reasons, maintenance is far more important. Just use OneHotEncoder from sklearn library and don't think about parallelizing this task. One hot encoding is rather inexpensive preprocessor and you won't see much difference running whole pipeline. Fitting model is most time consuming so focus on that.

If you actually wonder how to vectorize this, you need to iterate over whole data set to get unique values and one-hot vector length. You can parallelize second pass of substituting categorical values for one-hot vectors and merge data set chunks.

Parallelization isn't always the best thing to do. There's always some additional cost of starting, managing and merging multiple threads. Unless you have a lot to do, you won't see much difference.

When we talk about vectorization we mostly mean some computation on numerical arrays, which can be easily separated. This is mostly done using numpy operations, which are internally vectorized (native code, outside python machine). It's more about how CPU handles array operations internally - processing multiple elements in a loop instead one at a time. You can read more about AVX on a wiki, though this is really low-level, hardware stuff ;)

tl; dr

• use methods from libs
• don't use loops, use numpy when possible,