0
$\begingroup$

So im having this paticular problem triying to do one hot encoding on multilabel data, the encoder is reading more classes than it should, and i dont know why.

let me show you:

Here's my data (17 classes)

#'Admisión_de_Aire','Antisurge','Compresión','Control','Electrónico','Eléctrico','Enfriamiento','Escape','Gas_Combustible','Lubricación','Mecánico','Normal','Proceso','Sellos','Surge','Vibración','Válvulas'

>>>in: y

>>> Out:
357      Normal
1776     Normal
1777     Normal
1778     Normal
11927    Normal
          ...  
67226    Normal
67227    Normal
67682    Normal
67704    Normal
67707    Normal
Name: CLASS_TARGET, Length: 39519, dtype: object

running the code

 >>>in:
 # Encode
    label_encoder = LabelEncoder()
    label_encoder.fit(y)
    num_classes = len(label_encoder)
    print('num classes:',num_classes)
    y.describe() #double check clasess

>>> Out:
    num classes: 49   #Reads 32 more classes?

    count      39519
    unique        17  #Describe finds 17 classes
    top       Normal
    freq       32266
    Name: CLASS_TARGET, dtype: object

With those 32 more classes it finds, i get this result, but i dont know why this happen any suggestion?.

     >>>in:
label_encoder.class_to_index
    
    >>> Out:
{'A': 0,
 'C': 1,
 'E': 2,
 'G': 3,
 'L': 4,
 'M': 5,
 'N': 6,
 'P': 7,
 'S': 8,
 'V': 9,
 '_': 10,
 'a': 11,
 'b': 12,
 'c': 13,
 'd': 14,
 'e': 15,
 'f': 16,
 'g': 17,
 'i': 18,
 'l': 19,
 'm': 20,
 'n': 21,
 'o': 22,
 'p': 23,
 'r': 24,
 's': 25,
 't': 26,
 'u': 27,
 'v': 28,
 'á': 29,
 'é': 30,
 'ó': 31,
 'Admisión_de_Aire': 0,
 'Antisurge': 1,
 'Compresión': 2,
 'Control': 3,
 'Electrónico': 4,
 'Eléctrico': 5,
 'Enfriamiento': 6,
 'Escape': 7,
 'Gas_Combustible': 8,
 'Lubricación': 9,
 'Mecánico': 10,
 'Normal': 11,
 'Proceso': 12,
 'Sellos': 13,
 'Surge': 14,
 'Vibración': 15,
 'Válvulas': 16}

this is how the encoder works

import itertools
from collections import Counter
from typing import List, Sequence, Tuple
class LabelEncoder(object):
    """Label encoder for tag labels."""
    def __init__(self, class_to_index={}):
        self.class_to_index = class_to_index
        self.index_to_class = {v: k for k, v in self.class_to_index.items()}
        self.classes = list(self.class_to_index.keys())

    def __len__(self):
        return len(self.class_to_index)

    def __str__(self):
        return f"<LabelEncoder(num_classes={len(self)})>"

    def fit(self, y):
        classes = np.unique(list(itertools.chain.from_iterable(y)))
        for i, class_ in enumerate(classes):
            self.class_to_index[class_] = i
        self.index_to_class = {v: k for k, v in self.class_to_index.items()}
        self.classes = list(self.class_to_index.keys())
        return self

    def encode(self, y: pd.Series) -> np.ndarray:
        """Encode a collection of labels using (multilabel) one-hot encoding.
        Args:
            y (pd.Series): Collection of labels as a pandas Series object.
        Returns:
            Labels as (multilabel) one-hot encodings
        """
        y_one_hot = np.zeros((len(y), len(self.class_to_index)), dtype=int)
        for i, item in enumerate(y):
            for class_ in item:
                y_one_hot[i][self.class_to_index[class_]] = 1
        return y_one_hot

    def decode(self, y):
        classes = []
        for i, item in enumerate(y):
            indices = np.where(item == 1)[0]
            classes.append([self.index_to_class[index] for index in indices])
        return classes

    def save(self, fp):
        with open(fp, 'w') as fp:
            contents = {'class_to_index': self.class_to_index}
            json.dump(contents, fp, indent=4, sort_keys=False)

    @classmethod
    def load(cls, fp):
        with open(fp, 'r') as fp:
            kwargs = json.load(fp=fp)
        return cls(**kwargs)
$\endgroup$

1 Answer 1

1
$\begingroup$

Your code processes symbols instead of words.

Fixes

# classes = np.unique(list(itertools.chain.from_iterable(y)))
classes = np.unique(y)

# for class_ in item:
#     y_one_hot[i][self.class_to_index[class_]] = 1
y_one_hot[i][self.class_to_index[item]] = 1

Also, take a look at sklearn.preprocessing.OneHotEncoder

from sklearn.preprocessing import OneHotEncoder

label_encoder = OneHotEncoder(sparse=False)
label_encoder.fit(y.to_frame())
label_encoder.transform(y.to_frame())
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.