Questions tagged [categorical-data]

Categorical data can take on a limited (usually fixed) number of possible values called categories. Categorical values "label", they do not "measure". Nominal and dichotomous/binary scale types are categorical. Some people consider ordinal scale categorical too.

Filter by
Sorted by
Tagged with
1 vote
0 answers
17 views

What methods I could use to analyze the contingency table?

I am data science beginner, and I have a question about methods that I could use to analyze the following data. It is a simple case, I am trying to check the influence of cohabitation before marriage ...
user avatar
  • 11
0 votes
0 answers
9 views

How to handle tags/lists with CatBoost?

I have database like this: Id, type, category1, category2, tags 1, ‘cosmetics’, 123, 456, [446, 354] 2, ‘electronics’, 234, 213, [55, 978, 12] … And I want to predict some value with ...
user avatar
0 votes
0 answers
10 views

Constructing pandas DF with model trained categorical variable

I've trained a lightGBM model on a dataset X where X has a categorical (in the pandas sense) variable. This model trains fine and when I predict using it all looks good - I can even change the value ...
user avatar
0 votes
0 answers
21 views

Library for Phi correlation coefficient in python?

I want to calculate correlation b/w categorical features in my data. I reviewed the literature and found phi coefficient can be used for this purpose. I found one library called phik in python enter ...
user avatar
  • 27
0 votes
0 answers
8 views

improve the project of binary classification with categorical features

Here is a interview task of training a binary classification model. I will not show the detail of training data as the confidential issue. The variable $X$ has ten features and all the features are ...
user avatar
0 votes
0 answers
10 views

How to calculate distance for symmetric binary and nomianl variables?

In the existing function dist(), the only method for nominal variable is 'binary', and it's for asymmetric binary. However, I ...
user avatar
0 votes
0 answers
9 views

How do GANs learn category distributions

I'm currently getting more into the topic of GANs and Generating Models. I've understood how the Generator and Discriminator work together in optimization to generate synthetic samples. Now I'm ...
user avatar
  • 1
1 vote
0 answers
43 views

Encode each comma separated value in Pandas

I have a dataset ...
user avatar
  • 19
0 votes
1 answer
24 views

Categorical data preprocessing for training a algorithm

I have a training dataset where values of "Output" col is dependent on three columns (which are categorical [No ordering]). ...
user avatar
  • 19
1 vote
0 answers
14 views

Model a classification problem with multiple categorical varialbes as input features only. Diff Model performance

I'm having an input data with 100k rows, 8 input features, I'm trying to predict y (binary 1/0). But all the X are categorical variables(strictly nominal variables, not ordinal). Some with 8 levels, ...
user avatar
  • 11
0 votes
0 answers
6 views

What kind of hypothesis testing in Python can be used to validate that 4 job titles are significantly different based on their skillset?

I have 4 job titles, for each of which I scraped hundreds of job descriptions and classified them by if they contain words related to a predefined list of skills. For each job description, I now have ...
user avatar
0 votes
0 answers
4 views

Whether effect of one categorical variable on a continuous variable depends on levels of another categorical variable

In the dataset I need to analyse, I need to look at whether the effect of people's profession (3 categories) on their scores on a test (I have already tested for this effect and found one) differs ...
user avatar
2 votes
1 answer
28 views

How do I get the mean values that are greater than .5 for my model?

I am trying to build a classification model. One of the variables called specialty has 200 values. Based on a previous post I saw, I decided I wanted to include the values that have the highest mean. ...
user avatar
0 votes
0 answers
42 views

Value Error: Shapes (None,128,128) and (None, 4) are incompatible

I am trying to perform CNN on my dataset. I came across the below error ValueError: Shapes (None, 128, 128) and (None, 4) are incompatible The shape of my xTrain ...
user avatar
0 votes
1 answer
75 views

XGBOOST with target column has categorical data and features also has categorical data

I have a huge dataset with the categorical columns in features and also my target variable is categorical. All the values are not ordinal so I think it is best to use one hot encoding. But I have one ...
user avatar
1 vote
0 answers
21 views

Handling date and time fields for classification task

I'm working on a classification task(The dataset is 400,000 rows and 30 columns) and one of my features was date-time. I've extracted the month, day of the week, and hour from the dataset (year is a ...
user avatar
1 vote
1 answer
146 views

How to tell CatBoost which feature is categorical?

I am excited to learn that CatBoost can handle categorical features by itself. One of my features, Department ID, is categorical. However, it looks like numeric, since the values are like 1001, 1002, ....
user avatar
2 votes
0 answers
95 views

Can one use PCA to reduce the dimensionality of One-Hot-Encoded data?

I read a couple times that PCA was used as a method to reduce dimensionality for one-hot-encoded data. However, there were also some comments that using PCA is not a good idea since one-hot-encoded ...
user avatar
0 votes
2 answers
61 views

Custom Encoding for Categorical Features - sklearn

Just wanted to check if there are any obvious flaws with a custom encoding idea I have - for categorical features used with RandomForestClassifer or any tree-based classifier. As all of you would know ...
user avatar
  • 101
0 votes
1 answer
29 views

Collapsing categorical data into more than 3 categories

I have a bunch of categorical, part of speech data that I want to collapse into fewer categories. np.where() won't do because I want to have 6 categories at the end: noun, verb, adjective, adverb, ...
user avatar
0 votes
0 answers
123 views

How to group multiple categories of a categorical variable before feeding the data to a machine learning algorithm?

I have a labelled dataset to which I wish to fit a classification model (say, a Decision Tree). One of the categorical variables (say STATE) in the data has a lot ...
user avatar
0 votes
1 answer
25 views

What's the minimum percentage of categories should be present in the categorical variable for to ignore the variable entirely

For example, if i have a feature "colour_codes" that has close to 5000 distinct color codes inside it. And the number of samples/rows is 10 million. Then should I ignore the feature "...
user avatar
1 vote
1 answer
19 views

Over-sampling when predicting a contionuous variable

Lets say i am predicting house selling prices (continuous) and therefore have multiple independent variables (numerical and categorical). Is it common practice to balance the dataset when the ...
user avatar
  • 11
0 votes
1 answer
45 views

Separating numerical and categorical features in a binary classification problem

I have a dataset with employee data with around 9500 rows, and have to predict if the target is 0 or 1. Some of my features are the department of an employee, gender, salary, review_score(numerical),...
user avatar
  • 101
0 votes
0 answers
23 views

Anomaly Detection with totally Categorical features

I am working on an anomaly detection project that aims to discover which merchant is the point of compromise. The data contains no numerical value and it looks like this; date account merchant fraud ...
user avatar
0 votes
1 answer
40 views

Analysis with categorical variable

My dataset consists of a numeric variable (called "N4") and several categorical variables that affect the numeric variable. For example there is a categorical variable called "die" ...
user avatar
  • 459
2 votes
2 answers
228 views

Categorical to One hot encoding - Big data [closed]

I have a sales dataset which consists of binary label as output - "Business win" and "Business loss" of our products. We have a set of 1st level customers (lets call that group as ...
user avatar
  • 2,215
0 votes
0 answers
15 views

Training a ML model with a table for each observation

I have several csv's which are inputs for a row of outputs. A sample input dataset can look something like this: whose output would be as follows: The task is to train a model by reading each csv as ...
user avatar
0 votes
0 answers
16 views

Group related items by their description and tag each group. [Pen, Eraser] : Stationary

So have a list of data similar to the table below. It will be captured by a chatbot so I expect natural language but in the form of a structured command: Add {Qty} {item description} to {location} ID ...
user avatar
  • 101
0 votes
0 answers
14 views

Is it possible to classify type of a product from textual tags of those product (no image data)

I'm new to ML, but I'm looking to classify/categorise apparel products based on text data that comes with those products. Type of products range from Shirts to Earrings. I have approximately 60k ...
user avatar
0 votes
2 answers
47 views

Activation and Loss Function not chosen correctly when use Neural Network

I have three classes for my text dataset before. These are my classes: 0 = Cat 1 = Not Both 2 = Dog Then I use this code: ...
user avatar
  • 13
0 votes
0 answers
31 views

Dealing with discrete variables as continuous for K-means clustering (or not)

It is well established that k-means works best, and is designed for, continuous variables. I am considering a clustering problem where I have data like this: total spend / $ number of items in basket ...
user avatar
0 votes
0 answers
11 views

TensorFlow Lite for Microcontrollers: Predicting with missing values and categorical variables

I work on a project where sensor data have categorical variables and missing values. Preprocessing sensor data with, for example, tfp.sts.impute_missing_values and ...
user avatar
  • 263
2 votes
0 answers
26 views

(R) can I convert a categorical variable into a numeric equivalent in linear regression to predict a continuous variable?

Specifically, I have an item code as one of the independent variables that can have several hundred possible values results in underfitting when predicting the projected availability of that item. I'd ...
user avatar
  • 21
0 votes
1 answer
25 views

How to make model not too dependent on one variable?

Let's suppose I have a generic model: Variable A | Variable B | Variable C | Variable D Variable Dis a categorical variable. ( ...
user avatar
1 vote
1 answer
199 views

How to make predictions on unseen data with different cardinality using xgboost

I am training an XGBoost regression model on a feature set $X$ that includes a feature $x_k$ with high cardinality (~100). First, I am using one-hot-encoding to convert $x_k$ and then split the set ...
user avatar
1 vote
1 answer
120 views

How to get dummy variables from "first name"

I intend to predict the age of customers using some features. There are some categorical features that I need to convert to dummy variables before the modelling stage. Since the datasets are so big (...
user avatar
1 vote
1 answer
62 views

Clustering algorithm for time series data with categorical dtypes

I have a large dataset with around 200 features, consisting mostly of timeseries and categorical data, with some continuous. The dataset is extracted from/by a postal service. Small example: Random (...
user avatar
  • 11
0 votes
0 answers
35 views

Add noise to categorical data in tf / Keras?

Just being curious about adding noise to categorical variables. I suspect that adding gaussian noise over a one hot encoded variable wouldn't be enough. I was thinking about adding noise after ...
user avatar
  • 2,145
3 votes
4 answers
540 views

How can I deal with tiny categories?

I'm playing around with UCI Bank Marketing Dataset. So, there is a categorical variable named default which tells us if client "has credit in default". ...
user avatar
1 vote
1 answer
20 views

How to test whether data is clustered wrt. subcategories?

I have a dataset of about 2000 entries, containing two numerical values, one categorical and one sub-categorical label for each entry. The data is from chemistry lab data, but for the purpose of this ...
user avatar
-1 votes
3 answers
325 views

Dealing with Extra Categories in Test Set

Suppose I have a data set which consists a dependent variable y and independent variables X. Suppose that there is a specific ...
user avatar
  • 99
0 votes
0 answers
20 views

Chi - Square test for Validating Sampled Data

I have a large dataset (stored in a dataframe) that needs to be sampled, so I have performed sampling on it (sampled data also stored in a dataframe) and now wish to check if the sample data is ...
user avatar
0 votes
0 answers
10 views

How to determine correlation between ports and ticket prices?

I have titanic dataset, from which I have extracted ticket fares and embarkment ports. I am trying to find out if there is a correlation between embarkment port and ticket fare, so I constructed ...
user avatar
  • 101
0 votes
0 answers
24 views

How to handle both the categorical and ordinal features in a single data sets?

I was practicing Lasso regression with the SPARCS hospital dataset. There are two kinds of features in the dataset: Categorical features like location of the hospital, demographics of patients, etc. ...
user avatar
0 votes
0 answers
7 views

How to predict categorial variable with another variable which is quantitative if present and qualitative if missing?

Here is my 2 step biological problem : Step 1 : I track single cells through time in order to detect parameter A At the end of this step, whether a single cell presented the parameter A and I record ...
user avatar
1 vote
2 answers
60 views

Encoding "histogram bins"

I am currently working on a regression problem where I have one variable (x) of the data in the form of "histogram bins". I.e. I could have value ranges ...
user avatar
0 votes
0 answers
15 views

Conceptual question - is it correct to use categorical variables such as day, month, year as a fixed sequence input in LSTM?

I am working on a problem where I have to try to predict the dependent variable (continuous) every hour based on hourly temperature (the single continuous variable in predictor space), along with 4 ...
user avatar
0 votes
0 answers
12 views

Sequence to Sequence learning applied to list of numbers

I am looking to apply ML methods to genetic data. My goal is to predict which rare (generally de novo) mutations a person has based on what non-rare (generally inherited) mutations. I have worked on ...
user avatar
  • 101
0 votes
0 answers
13 views

Categorical variables: create a risk class or include in the model?

I think this is a very basic question so sorry for the wordy format. I am trying to get my head around it. I am thinking about predicting earthquake damage to property in the US using a GLM algorithm. ...
user avatar
  • 101

1
2 3 4 5
8