There are some cases where LabelEncoder or DictVectorizor are useful, but these are quite limited in my opinion due to ordinality.
LabelEncoder can turn [dog,cat,dog,mouse,cat] into [1,2,1,3,2], but then the imposed ordinality means that the average of dog and mouse is cat. Still there are algorithms like decision trees and random forests that can work with ...
The standard k-means algorithm isn't directly applicable to categorical data, for various reasons. The sample space for categorical data is discrete, and doesn't have a natural origin. A Euclidean distance function on such a space isn't really meaningful. As someone put it, "The fact a snake possesses neither wheels nor legs allows us to say nothing about ...
While AN6U5 has given a very good answer, I wanted to add a few points for future reference. When considering One Hot Encoding(OHE) and Label Encoding, we must try and understand what model you are trying to build. Namely the two categories of model we will be considering are:
Tree Based Models: Gradient Boosted Decision Trees and Random Forests.
In my opinion, there are solutions to deal with categorical data in clustering. R comes with a specific distance for categorical data. This distance is called Gower (http://www.rdocumentation.org/packages/StatMatch/versions/1.2.0/topics/gower.dist) and it works pretty well.
(In addition to the excellent answer by Tim Goodman)
The choice of k-modes is definitely the way to go for stability of the clustering algorithm used.
The clustering algorithm is free to choose any distance metric / similarity score. Euclidean is the most popular. But any other metric can be used that scales according to the data distribution in each ...
This question seems really about representation, and not so much about clustering.
Categorical data is a problem for most algorithms in machine learning. Suppose, for example, you have some categorical variable called "color" that could take on the values red, blue, or yellow. If we simply encode these numerically as 1,2, and 3 respectively, our algorithm ...
This can be done using StringIndexer in PySpark and the reverse using IndexToString for reference please check this:
from pyspark.ml.feature import StringIndexer
df = sqlContext.createDataFrame(
[(0, "a"), (1, "b"), (2, "c"), (3, "a"), (4, "a"), (5, "c")],
indexer = StringIndexer(inputCol="category", outputCol="categoryIndex")
If your categorical columns are currently character/object you can use something like this to do each one:
char_cols = df.dtypes.pipe(lambda x: x[x == 'object']).index
for c in char_cols:
df[c] = pd.factorize(df[c])
If you need to be able to get back to the categories I'd create a dictionary to save the encoding; something like:
char_cols = df....
It is very good question; in fact this problem has been around for a while and I have not yet found the perfect solution. Yet more than happy to share my experience:
Avoid one-hot-encode as much as possible (contrary to what was suggested above). The reasoning is that it won't work. A model with one-hot-encode features only works when all those sublevels ...
For those who are interested, I've spent some time, finally figured out that the problem was the way one has to prepare the categorical encoding for the Entity Embedding suitable for a neural network architecture; unfortunately none of the examples provided in blogposts or Kaggle kernels were clear about this step!
Here is the link to the repository ...
If you really care about the number of dimensions, you still can try to apply a dimensionality reduction algorithm, such as PCA (Principal Component Analysis) or LDA (Linear Discriminant Analysis), after your one hot encoding.
But know that "56 features" isn't really large and it's highly common in the industry to have thousands, millions or even billions ...
Simply put because one level of your categorical feature (here location) become the reference group during dummy encoding for regression and is redundant. I am quoting form here "A categorical variable of K categories, or levels, usually enters a regression as a sequence of K-1 dummy variables. This amounts to a linear hypothesis on the level means."
Taking a stab:
I am trying to identify a clustering technique with a similarity measure that would work for categorical and numeric binary data.
Gower Distance is a useful distance metric when the data contains both continuous and categorical variables.
There are techniques in R kmodes clustering and kprototype that are designed for this type of ...
The main difference I can think of is that using one-hot encoding will mean that all your strings will be at the same (hamming) distance from each other, while using a scalar value means that distances between the resulting features will be meaningless (it may encode "red" as 1, "blue" as 2 and "green" as 3, but there is no reason why red is more similar to ...
You may want to use Factor analysis of mixed data.
It allows you to do dimension reduction on a complete data set.
A R implementation could be found in the FactoMineR package. But this function struggle when you have a high number of data/columns.
I am not aware of the existence of the equivalent in python.
One possible reason is that when you use one-hot-encoding for categorical data, you should set the intercept property in the function to be False:
model = LinearRegression(fit_intercept=False, normalize=True).fit(X_train, y_train)
This will avoid the dummy variable trap:
You could also ...
One advantage of get_dummies is that it can operate on values other than integers (so you don't need the LabelEncoder) and returns a DataFrame with the categories as column names. Also, you can conveniently drop one redundant category using drop_first=True.
One advantage of scikit-learn's OneHoteEncoder lies in the scikit-learn API. OHE gives you a ...
I found what I was looking for - it's called Theil's U, or the Uncertainty Coefficient.
I've used it in this Kaggle kernel, you can check it out for an example and code implementation in Python
EDIT: I also have a blogpost about it.
One way to handle this is to use 'supervised classification'. In this model, you manually classify a subset of the data and use it to train your algorithm. Then, you feed the remaining data into your software to classify it.
This is accomplished with NLTK for Python (nltk.org).
If you are simply looking for strings like "hardware" and "software", this is a ...
I would suggest you to use the idea of so-called 'fuzzy clustering', where you put each of your hours of the day value into several clusters at the same time. Details in paper: http://home.deib.polimi.it/matteucc/Clustering/tutorial_html/cmeans.html
The idea is trivial:
You decide how many clusters you want to have. For example, 4 (so you divide your day ...
I'm not aware of a foolproof way to do this. Here's one idea off the top of my head:
Treat values as categorical by default.
Check for various attributes of the data that would imply it is actually continuous. Weight these attributes based on how likely they are to correlate with continuous data. Here are some possible examples:
Values are integers: +.7
One-hot-encoded ZIP codes shouldn't present a problem with modern tools, where features can be much wider (millions, billions even), but if you really want you could aggregate area codes into regions, such as states. Of course, you should not use strings, but bit vectors. Two other dimensionality reduction options are MCA (PCA for categorical variables) and ...
This is an old question. I am surprised that I don't see anyone mentioned Mean Encoding (a.k.a Target Encoding). It is very popular in supervised learning problems. Besides, I have seen people use frequency or the cdf of the frequency (to avoid noise generated by heavy-tailed pdf), and they achieved pretty good results with lightGBM. However, i cannot really ...
First, let's create a mcve to play with:
import pandas as pd
import numpy as np
In : categorical_array = np.random.choice(['Var1','Var2','Var3'],
df = pd.DataFrame(categorical_array,
columns=map(lambda x:chr(97+x), range(categorical_array.shape)))
The encoding leads to a question of representation and the way that the algorithms cope with the representation.
Let's consider 3 methods of representing n categorial values of a feature:
A single feature with n numeric values.
one hot encoding (n Boolean features, exactly one of them must be on)
Log n Boolean features,representing the n values.
Note that ...
There are three types of missing data: Missing Completely At Random (MCAR), Missing At Random (MAR) and Missing Not At Random (MNAR).
Your case is the second, where according to wikipedia it:
occurs when the missingness is not random, but where missingness can be fully accounted for by variables where there is complete information
This means that the ...
You have partly answered this question yourself ("because converting to integers implies that there is an ordering between features").
I will just clarify the terminology a bit more.
Categorical data: information has categories, but no natural ordering defined between them (gender, name of user's cat)
Ordinal data: information has categories with natural ...
It depends on how you're one-hot encoding them. Many automated solutions for that will name all the converted booleans with a pattern so that a categorical variable called "letter" with values A-Z would end up like:
letter_A, letter_B, letter_C, letter_D,....
If after you've figured out feature importance you've got an array of feature and the associated ...
You could concatenate your train and test datasets, crete dummy variables and then separate them dataset.
Something like this:
train_objs_num = len(train)
dataset = pd.concat(objs=[train, test], axis=0)
dataset = pd.get_dummies(dataset)
train = copy.copy(dataset[:train_objs_num])
test = copy.copy(dataset[train_objs_num:])
There's three main approaches to solving this:
Building two models separately and then training an ensemble algorithm that receives the output of the two models as an input
Concating all the data into a single vector/tensor as a preprocessing step and then train a simple single input NN
The multiple input NN architecture you proposed
The ensemble approach ...