Questions tagged [categorical-data]

Categorical data can take on a limited (usually fixed) number of possible values called categories. Categorical values "label", they do not "measure". Nominal and dichotomous/binary scale types are categorical. Some people consider ordinal scale categorical too.

Filter by
Sorted by
Tagged with
1
vote
1answer
16 views

How to test whether data is clustered wrt. subcategories?

I have a dataset of about 2000 entries, containing two numerical values, one categorical and one sub-categorical label for each entry. The data is from chemistry lab data, but for the purpose of this ...
0
votes
3answers
56 views

Dealing with Extra Categories in Test Set

Suppose I have a data set which consists a dependent variable y and independent variables X. Suppose that there is a specific ...
0
votes
0answers
17 views

Chi - Square test for Validating Sampled Data

I have a large dataset (stored in a dataframe) that needs to be sampled, so I have performed sampling on it (sampled data also stored in a dataframe) and now wish to check if the sample data is ...
0
votes
0answers
9 views

How to determine correlation between ports and ticket prices?

I have titanic dataset, from which I have extracted ticket fares and embarkment ports. I am trying to find out if there is a correlation between embarkment port and ticket fare, so I constructed ...
0
votes
0answers
16 views

How to handle both the categorical and ordinal features in a single data sets?

I was practicing Lasso regression with the SPARCS hospital dataset. There are two kinds of features in the dataset: Categorical features like location of the hospital, demographics of patients, etc. ...
0
votes
0answers
6 views

How to predict categorial variable with another variable which is quantitative if present and qualitative if missing?

Here is my 2 step biological problem : Step 1 : I track single cells through time in order to detect parameter A At the end of this step, whether a single cell presented the parameter A and I record ...
1
vote
1answer
26 views

Encoding "histogram bins"

I am currently working on a regression problem where I have one variable (x) of the data in the form of "histogram bins". I.e. I could have value ranges ...
0
votes
0answers
9 views

Conceptual question - is it correct to use categorical variables such as day, month, year as a fixed sequence input in LSTM?

I am working on a problem where I have to try to predict the dependent variable (continuous) every hour based on hourly temperature (the single continuous variable in predictor space), along with 4 ...
0
votes
0answers
12 views

Sequence to Sequence learning applied to list of numbers

I am looking to apply ML methods to genetic data. My goal is to predict which rare (generally de novo) mutations a person has based on what non-rare (generally inherited) mutations. I have worked on ...
0
votes
0answers
13 views

Categorical variables: create a risk class or include in the model?

I think this is a very basic question so sorry for the wordy format. I am trying to get my head around it. I am thinking about predicting earthquake damage to property in the US using a GLM algorithm. ...
0
votes
1answer
17 views

Encoding concept for categorical data - pick one for all the columns or different for different kinds in the same df

[Beginner here] If dataset contains - both ordinal, nonordinal (few categories) & nonordinal (multiple categories > 30). Is one supposed to pick one to encapsulate of all such situations or ...
0
votes
0answers
7 views

Categories for binning weather / meteorological data

For EDA purposes I would like to bin continuous weather variables (temperature, rainfall, etc.) into well defined intervals such as the ones you hear in your daily weather report such as: For ...
0
votes
0answers
27 views

How to consider the change in categorical variable in multiple linear regression?

I am building a multiple linear regression model to predict the mileage of tires and one of the independent variables is the wheel position. It is categorical and I could encode it to run the model ...
1
vote
0answers
12 views

Why is a column detected as text and not as categorical when opening excel file?

I have the following values in an excel sheet Aguascalientes Baja California Baja California Sur Campeche Chiapas Chihuahua Ciudad de México Coahuila Colima Durango Estado de México Guanajuato ...
1
vote
0answers
28 views

feature selection for categorical variables

I have been working on this issue for quite a while and going nowhere. If I have categorical features in my dataset and some of them have high dimensions, if I OHE them, I get a dataset with high ...
0
votes
1answer
91 views

Separate numerical and categorical variables

I have a dataset (42000, 10) which contains 7 categorical features and 3 numerical. I would like to separate both the numerical and categorical features into 2 different data frames i.e I would like 2 ...
0
votes
3answers
53 views

Transforming Categorical to Numerical variable

I have a categorical variable with 4 levels ('8 c', '6 c','NAN','Others') and I want to convert it to numerical form. an Obvious way is to simply remove the 'c' part from the first two categories and ...
0
votes
0answers
13 views

What is the recommended embedding for a categorical variable with more than 40000 thousand categories?

I have a feature called Planning_id with more than 40000 categories. What is the recommended embedding size? I read that: embedding_dimension = # categories * 0.25 is a good rule of thumb, but I still ...
0
votes
0answers
13 views

How to treat a column that contains a list of categorical, high cardinality values for a classification problem?

The list cannot be exploded into several columns because this will result in very high dimensionality. I would like to know the following: How to treat this kind of column in a dataframe? Can I keep ...
1
vote
3answers
99 views

logistic regression or density estimation for binary dependent variable and binary (or categorical) features [closed]

I have a binary dependent variable $t$ and categorical features. We can even simplify to binary features since I can one-hot encode the categorical variables. In practice the one-hot encoding induces ...
0
votes
0answers
48 views

How to train a GAN to generate categorical variable

I am trying to train a simple GAN to generate a categorical variable size, which takes discrete values between 1-100. I am looking for some tips or directions on ...
0
votes
0answers
28 views

How to run PCA when data contains some categorial features?

Assume that we have a dataset with various features, and some of the features are categorial. And PCA dosn't work good on categorical features. How should I handle such datasets using PCA, what is ...
2
votes
1answer
42 views

Two questions about one-hot encoding: drop first? and features with thousands of categories [closed]

I have two questions about one-hot feature encoding: (1) Is it considered a "best practice" to drop the first (or at least one) one-hot encoded feature when one-hot encoding, like you would ...
1
vote
0answers
24 views

When doing feature selection, are features like "year", "month" considered as ordinal features or should I convert them to strings?

I am working on a hotel reservation dataset that has both categorical and numerical (continuous and discrete) features (26 columns, 30244 rows). Target is also categorical and it says if the user &...
1
vote
1answer
51 views

Handling nominal category features in decision tree

I have been reading some stackoverflow questions on how to handle nominal features for decision tree (sklearn implementation). One of the answer states that : Using a OneHotEncoder is the only current ...
1
vote
0answers
11 views

spark ml StringIndexer vs OneHotEncoder, when to use which?

Confused as to when to use StringIndexer vs StringIndexer+OneHotEncoder. The OneHotEncoder docs say For string type input data, it is common to encode categorical features using StringIndexer first. ...
3
votes
1answer
44 views

How to identify if there is a relationship between 5 categorical independent variables to a binary dependent variable?

My dataset has 5 independent variables, each with a value of either Large, Medium or None and a binary dependent variable. The dataset has 67 rows with a split of 17:50. I would like to identify if ...
1
vote
0answers
25 views

An Unsupervised learning method suitable for large categorical data sets

I want to detect anomalies in the bank data set in an unsupervised learning method. However, in the bank data set, all columns except time and amount were categorical data, and about half of them had ...
0
votes
1answer
21 views

Handling categorical data with more over 100 unique classes

I am working with a pure categorical data set. And some classes have more than 100 unique values. I could not find any appropriate encoding possibility. So I created a SQL table, where each value got ...
1
vote
2answers
57 views

How do I assign specific values to categorical variables

I have a Pandas data frame with columns within a survey with the following categorical values - "Increased, Decreased, Neutral". My question is how can I assign specific numerical values to ...
0
votes
0answers
31 views

Encoding Categorical Data

I am trying to predict a car price from a given dataset. When I am trying to encode the categorical data I am getting an error. I am attaching an image with the error below. Can someone help me know ...
1
vote
1answer
34 views

How to build multiple variable regression having a mix of numerical & categorical features?

There is a need to estimate Annual Average Daily Traffic Volume (AADT). We have bunch of data about vehicles' speeds during several years. It is noticed that AADT depends on the average number of such ...
1
vote
1answer
72 views

one hot encoding target variable in tree and non tree (knn) methods

I am learning about label encoders, one hot encoding etc applied to datasets for classification via KNN and XGBoost type trees. However, I am a bit confused as to whether the target variable should be ...
1
vote
2answers
34 views

Linear Regression with Category variables

I'm currently learning and exploring machine learning and understand the basics of linear regression based on two numerical variables, but now I wish to go a little further and need some guidance ...
1
vote
0answers
24 views

How to get correlation between the categories of two categorical variable?

I have a categorical variable with 2 categories ("Health") ('healthy', 'not_healthy') and another categorical variable ("country") with 5 categories ("english", "eua&...
1
vote
2answers
42 views

Different number of features in train vs test when using Label Encoding

This is not a duplicate of Different number of features in train vs test There are some categorical columns in my data, and the cardinality for each of them is large, so I chose to use LabelEncoding ...
0
votes
0answers
28 views

Equitable selection of users through ranking

I am looking to take a dataset largely derived of user input in categorical form, this sign up sheet asks for many data points such as age group, race, sign up date, as well as a few others. My goal ...
0
votes
0answers
16 views

Which are the features selection techniques depending on the combination on cat num columns in independent and dependent features?

I am very confused: For what I understood I should: Multicollinearity check with Pearson corr and possibly consider to drop multicolliner features Then? I am very confused feature selection should be ...
2
votes
1answer
120 views

Why do my target labels need to begin at 0 for sparse categorical cross entropy to work?

I'm following a guide here to implement image segmentation in Keras. One thing I'm confused about are these lines: ...
0
votes
0answers
18 views

How to convert the categorical features into one-hot encoded dense vectors before applying the min-max scaler of sklearn?

like the question says I am trying to figure out how to convert one-hot encoded vectors into dense vectors, because my data-set contains both continuous and categorical features, I used the ...
0
votes
1answer
84 views

Problem with binning

I am trying to change continuous data points to categorical by using binning. I know two techniques, i) equal width bins ii) bins with equal number of elements. My questions are: Which type of ...
1
vote
1answer
79 views

Calculating correlation for categorical variables

I am struggling to find out a suitable way to calculate correlation coefficient for categorical variables. Pearson's coefficient is not supported for categorical ...
2
votes
1answer
39 views

Using the curse of dimensionality for encoding non-ordered (nominal) categorical variables of high cardinality

When the dimension is high, all data are approximately at the same distance away from each other. This makes distance-based methods such as k-nearest neighbors less useful if the data are more or less ...
1
vote
1answer
68 views

Anomaly detection on sparse categorical data

I have a big dataset with a column "clientid" and a categorical column "choice". I want to find out what are the clients that have strange combinations of choices (less frequent ...
1
vote
0answers
21 views

Feature Selection Statistical Test for Nominal Response Vs Continuous Predictors? (in R)

I cannot find much information on this, none so far useful. I have a sparse data set with 17K+ columns of continuous gene expressions, an example of a typical column: (3.15, 0, 7.1294, 0, 0, 0, 2300.2,...
0
votes
1answer
41 views

Dealing with unseen data/categories in machine learning models for stream data

I want to build a machine learning model (xgb and lgbm) that has to handle streaming data on a weekly basis. The models are trained on a bi-weekly basis. The data includes order information and I want ...
1
vote
1answer
62 views

Find correlation between grades from two raters [duplicate]

The question is whether we can find a correlation between two sets of grades (categorical data). Let’s say we have a dog competition and there are 1000 dogs participating. There are two rounds of ...
1
vote
0answers
11 views

multi-observed features [closed]

I am working on a ML model where individual features may have a highly variable number of observed values. The model will predict a continuous variable so I am planning to use a Regressor. More ...
2
votes
0answers
42 views

How to get the maximum likelihood estimate of the categorical distribution parameters using Lagrange optimization?

Let's say our data is discrete-valued and belongs to one of $K$ classes. The underlying probability distribution is assumed to be a categorical/multinoulli distribution given as $p(\textbf{x}) = \...
7
votes
5answers
647 views

How do I encode the categorical columns if there are more than 15 unique values?

I'm trying to use this data to make a data analysis report using regression. Since regression only allows for numerical types, I then need to encode the categorical data. However, most of these have ...

1
2 3 4 5
7