Questions tagged [categorical-data]

Categorical data can take on a limited (usually fixed) number of possible values called categories. Categorical values "label", they do not "measure". Nominal and dichotomous/binary scale types are categorical. Some people consider ordinal scale categorical too.

Filter by
Sorted by
Tagged with
1
vote
1answer
13 views

Using changes in categorical variables in prediction

I want to predict the length of new contracts and I have a lot of categorical variables e.g. Income bracket (5 levels). These may change over time. I receive an update for them every quarter as long ...
0
votes
1answer
15 views

What's the minimum percentage of categories should be present in the categorical variable for to ignore the variable entirely

For example, if i have a feature "colour_codes" that has close to 5000 distinct color codes inside it. And the number of samples/rows is 10 million. Then should I ignore the feature "...
1
vote
1answer
15 views

Over-sampling when predicting a contionuous variable

Lets say i am predicting house selling prices (continuous) and therefore have multiple independent variables (numerical and categorical). Is it common practice to balance the dataset when the ...
0
votes
1answer
18 views

Separating numerical and categorical features in a binary classification problem

I have a dataset with employee data with around 9500 rows, and have to predict if the target is 0 or 1. Some of my features are the department of an employee, gender, salary, review_score(numerical),...
0
votes
0answers
10 views

Anomaly Detection with totally Categorical features

I am working on an anomaly detection project that aims to discover which merchant is the point of compromise. The data contains no numerical value and it looks like this; date account merchant fraud ...
0
votes
1answer
21 views

Analysis with categorical variable

My dataset consists of a numeric variable (called "N4") and several categorical variables that affect the numeric variable. For example there is a categorical variable called "die" ...
2
votes
2answers
121 views

Categorical to One hot encoding - Big data [closed]

I have a sales dataset which consists of binary label as output - "Business win" and "Business loss" of our products. We have a set of 1st level customers (lets call that group as ...
0
votes
0answers
12 views

Training a ML model with a table for each observation

I have several csv's which are inputs for a row of outputs. A sample input dataset can look something like this: whose output would be as follows: The task is to train a model by reading each csv as ...
0
votes
0answers
13 views

Group related items by their description and tag each group. [Pen, Eraser] : Stationary

So have a list of data similar to the table below. It will be captured by a chatbot so I expect natural language but in the form of a structured command: Add {Qty} {item description} to {location} ID ...
0
votes
0answers
12 views

Is it possible to classify type of a product from textual tags of those product (no image data)

I'm new to ML, but I'm looking to classify/categorise apparel products based on text data that comes with those products. Type of products range from Shirts to Earrings. I have approximately 60k ...
0
votes
2answers
29 views

Activation and Loss Function not chosen correctly when use Neural Network

I have three classes for my text dataset before. These are my classes: 0 = Cat 1 = Not Both 2 = Dog Then I use this code: ...
0
votes
0answers
21 views

Dealing with discrete variables as continuous for K-means clustering (or not)

It is well established that k-means works best, and is designed for, continuous variables. I am considering a clustering problem where I have data like this: total spend / $ number of items in basket ...
0
votes
0answers
5 views

TensorFlow Lite for Microcontrollers: Predicting with missing values and categorical variables

I work on a project where sensor data have categorical variables and missing values. Preprocessing sensor data with, for example, tfp.sts.impute_missing_values and ...
2
votes
0answers
24 views

(R) can I convert a categorical variable into a numeric equivalent in linear regression to predict a continuous variable?

Specifically, I have an item code as one of the independent variables that can have several hundred possible values results in underfitting when predicting the projected availability of that item. I'd ...
0
votes
1answer
22 views

How to make model not too dependent on one variable?

Let's suppose I have a generic model: Variable A | Variable B | Variable C | Variable D Variable Dis a categorical variable. ( ...
1
vote
1answer
40 views

How to make predictions on unseen data with different cardinality using xgboost

I am training an XGBoost regression model on a feature set $X$ that includes a feature $x_k$ with high cardinality (~100). First, I am using one-hot-encoding to convert $x_k$ and then split the set ...
1
vote
0answers
11 views

How to get dummy variables from "first name"

I intend to predict the age of customers using some features. There are some categorical features that I need to convert to dummy variables before the modelling stage. Since the datasets are so big (...
1
vote
0answers
18 views

Clustering algorithm for time series data with categorical dtypes

I have a large dataset with around 200 features, consisting mostly of timeseries and categorical data, with some continuous. The dataset is extracted from/by a postal service. Small example: Random (...
0
votes
0answers
13 views

Add noise to categorical data in tf / Keras?

Just being curious about adding noise to categorical variables. I suspect that adding gaussian noise over a one hot encoded variable wouldn't be enough. I was thinking about adding noise after ...
3
votes
4answers
537 views

How can I deal with tiny categories?

I'm playing around with UCI Bank Marketing Dataset. So, there is a categorical variable named default which tells us if client "has credit in default". ...
1
vote
1answer
18 views

How to test whether data is clustered wrt. subcategories?

I have a dataset of about 2000 entries, containing two numerical values, one categorical and one sub-categorical label for each entry. The data is from chemistry lab data, but for the purpose of this ...
-1
votes
3answers
64 views

Dealing with Extra Categories in Test Set

Suppose I have a data set which consists a dependent variable y and independent variables X. Suppose that there is a specific ...
0
votes
0answers
17 views

Chi - Square test for Validating Sampled Data

I have a large dataset (stored in a dataframe) that needs to be sampled, so I have performed sampling on it (sampled data also stored in a dataframe) and now wish to check if the sample data is ...
0
votes
0answers
10 views

How to determine correlation between ports and ticket prices?

I have titanic dataset, from which I have extracted ticket fares and embarkment ports. I am trying to find out if there is a correlation between embarkment port and ticket fare, so I constructed ...
0
votes
0answers
17 views

How to handle both the categorical and ordinal features in a single data sets?

I was practicing Lasso regression with the SPARCS hospital dataset. There are two kinds of features in the dataset: Categorical features like location of the hospital, demographics of patients, etc. ...
0
votes
0answers
7 views

How to predict categorial variable with another variable which is quantitative if present and qualitative if missing?

Here is my 2 step biological problem : Step 1 : I track single cells through time in order to detect parameter A At the end of this step, whether a single cell presented the parameter A and I record ...
1
vote
1answer
30 views

Encoding "histogram bins"

I am currently working on a regression problem where I have one variable (x) of the data in the form of "histogram bins". I.e. I could have value ranges ...
0
votes
0answers
12 views

Conceptual question - is it correct to use categorical variables such as day, month, year as a fixed sequence input in LSTM?

I am working on a problem where I have to try to predict the dependent variable (continuous) every hour based on hourly temperature (the single continuous variable in predictor space), along with 4 ...
0
votes
0answers
12 views

Sequence to Sequence learning applied to list of numbers

I am looking to apply ML methods to genetic data. My goal is to predict which rare (generally de novo) mutations a person has based on what non-rare (generally inherited) mutations. I have worked on ...
0
votes
0answers
13 views

Categorical variables: create a risk class or include in the model?

I think this is a very basic question so sorry for the wordy format. I am trying to get my head around it. I am thinking about predicting earthquake damage to property in the US using a GLM algorithm. ...
0
votes
1answer
18 views

Encoding concept for categorical data - pick one for all the columns or different for different kinds in the same df

[Beginner here] If dataset contains - both ordinal, nonordinal (few categories) & nonordinal (multiple categories > 30). Is one supposed to pick one to encapsulate of all such situations or ...
0
votes
0answers
8 views

Categories for binning weather / meteorological data

For EDA purposes I would like to bin continuous weather variables (temperature, rainfall, etc.) into well defined intervals such as the ones you hear in your daily weather report such as: For ...
0
votes
0answers
28 views

How to consider the change in categorical variable in multiple linear regression?

I am building a multiple linear regression model to predict the mileage of tires and one of the independent variables is the wheel position. It is categorical and I could encode it to run the model ...
1
vote
0answers
15 views

Why is a column detected as text and not as categorical when opening excel file?

I have the following values in an excel sheet Aguascalientes Baja California Baja California Sur Campeche Chiapas Chihuahua Ciudad de México Coahuila Colima Durango Estado de México Guanajuato ...
1
vote
0answers
31 views

feature selection for categorical variables

I have been working on this issue for quite a while and going nowhere. If I have categorical features in my dataset and some of them have high dimensions, if I OHE them, I get a dataset with high ...
0
votes
1answer
197 views

Separate numerical and categorical variables

I have a dataset (42000, 10) which contains 7 categorical features and 3 numerical. I would like to separate both the numerical and categorical features into 2 different data frames i.e I would like 2 ...
0
votes
3answers
64 views

Transforming Categorical to Numerical variable

I have a categorical variable with 4 levels ('8 c', '6 c','NAN','Others') and I want to convert it to numerical form. an Obvious way is to simply remove the 'c' part from the first two categories and ...
0
votes
0answers
14 views

What is the recommended embedding for a categorical variable with more than 40000 thousand categories?

I have a feature called Planning_id with more than 40000 categories. What is the recommended embedding size? I read that: embedding_dimension = # categories * 0.25 is a good rule of thumb, but I still ...
0
votes
0answers
15 views

How to treat a column that contains a list of categorical, high cardinality values for a classification problem?

The list cannot be exploded into several columns because this will result in very high dimensionality. I would like to know the following: How to treat this kind of column in a dataframe? Can I keep ...
0
votes
2answers
142 views

logistic regression or density estimation for binary dependent variable and binary (or categorical) features [closed]

I have a binary dependent variable $t$ and categorical features. We can even simplify to binary features since I can one-hot encode the categorical variables. In practice the one-hot encoding induces ...
0
votes
0answers
77 views

How to train a GAN to generate categorical variable

I am trying to train a simple GAN to generate a categorical variable size, which takes discrete values between 1-100. I am looking for some tips or directions on ...
0
votes
0answers
29 views

How to run PCA when data contains some categorial features?

Assume that we have a dataset with various features, and some of the features are categorial. And PCA dosn't work good on categorical features. How should I handle such datasets using PCA, what is ...
2
votes
1answer
56 views

Two questions about one-hot encoding: drop first? and features with thousands of categories [closed]

I have two questions about one-hot feature encoding: (1) Is it considered a "best practice" to drop the first (or at least one) one-hot encoded feature when one-hot encoding, like you would ...
1
vote
0answers
30 views

When doing feature selection, are features like "year", "month" considered as ordinal features or should I convert them to strings?

I am working on a hotel reservation dataset that has both categorical and numerical (continuous and discrete) features (26 columns, 30244 rows). Target is also categorical and it says if the user &...
1
vote
1answer
82 views

Handling nominal category features in decision tree

I have been reading some stackoverflow questions on how to handle nominal features for decision tree (sklearn implementation). One of the answer states that : Using a OneHotEncoder is the only current ...
1
vote
1answer
32 views

spark ml StringIndexer vs OneHotEncoder, when to use which?

Confused as to when to use StringIndexer vs StringIndexer+OneHotEncoder. The OneHotEncoder docs say For string type input data, it is common to encode categorical features using StringIndexer first. ...
3
votes
2answers
59 views

How to identify if there is a relationship between 5 categorical independent variables to a binary dependent variable?

My dataset has 5 independent variables, each with a value of either Large, Medium or None and a binary dependent variable. The dataset has 67 rows with a split of 17:50. I would like to identify if ...
1
vote
0answers
29 views

An Unsupervised learning method suitable for large categorical data sets

I want to detect anomalies in the bank data set in an unsupervised learning method. However, in the bank data set, all columns except time and amount were categorical data, and about half of them had ...
0
votes
1answer
25 views

Handling categorical data with more over 100 unique classes

I am working with a pure categorical data set. And some classes have more than 100 unique values. I could not find any appropriate encoding possibility. So I created a SQL table, where each value got ...
1
vote
2answers
174 views

How do I assign specific values to categorical variables

I have a Pandas data frame with columns within a survey with the following categorical values - "Increased, Decreased, Neutral". My question is how can I assign specific numerical values to ...

1
2 3 4 5
8