20
There are at least two general considerations to make:
Domain-related
If an attribute potentially has predictive power in your domain and more specifically for your task your models might benefit from a direct encoding. For example: if being trans is correlated with different psychological disorders then I'd include a direct feature for this. This way it ...
8
It is quite an interesting question. I guess that you can call it "dealing with non-binary gender roles in a binary language" or something like this.
In the past I did once something similar. I created 3 features:
sex at birth [male,female]
sex identification [male,female]
Attracted sexually to [male,female].
All these features are binary and you can ...
5
No need to drop from the analysis, for sure. Your analyses should be capable of classifying by domain, even if you just assign them to a third (or fourth or ...) category. You'll be basically comparing Female:Not Female, Male:Not Male, etc.; keeping them in the dataset means you have a better result when you're comparing those domains.
The decision you ...
3
What you are trying to do is text normalization, which is a part of NLP pipeline.
To successfully preprocess your data, please try this below:
Install ekphrasis
pip install ekphrasis
Apply seg & spell correction:
from ekphrasis.classes.spellcorrect import SpellCorrector
from ekphrasis.classes.segmenter import Segmenter
sp = SpellCorrector(corpus="...
3
First I would determine the number of people who fall into male and female and if the number left over will not end up being statistically significant then they would be best to be discarded. After that if the "other" group is big enough maybe split it up, but once again consider if the split groups are big enough for statistical significance otherwise I ...
3
If it is an open mental health data set, then those using it would benefit from filtering into as many categories as possibly relevant as the end user may need to specify between the given subsets.
In the end, data sets are easy to modify into narrowing categories or maintaining the same categories.
If the end user wants to combine those data categories, ...
3
Some considerations here:
How has the data been collected?
If it's self-reporting, it's quite likely that most trans people will simply have replied with "male", "female", or other equivalent terms that give no indication of trans status. If it's reported by others, it's quite likely that the reporter will often not know that the person is trans.
If most ...
3
I think there si some confusion with the quantile transformer : https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.QuantileTransformer.html#sklearn.preprocessing.QuantileTransformer it actually scale the values to a 0-1 range. the goal is to get a uniform distribution. It serves a different purpose than minmax scaler imho. Notably it ...
2
This transformation is called min-max-scaling and also often referred to as standardization.
Scikit learn provides the MinMaxScaler() for this (see here). Here is an example adapted from "Introduction to Machine Learning with Python" by Mueller and Guido:
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
...
2
I've been sitting on this idea for decades, never sharing it with anyone. I don't expect it to be accepted by anybody. But here we go anyhow:
When looking at the question of gender, I came to the conclusion that 8 bits were needed to define gender properly, including groups (which of course can be of both genders) and uncertainties. The bits are:
NBM (...
2
That generally depends on what you are trying to achieve.
What people report as their gender is basically the output of a black box function with a lot of input variables. As any endocrinologist can tell you, it's not as simple as "high testosteron" vs "high estrogen" but more on the order of a hundred different hormones involved, most of which have ...
1
First thing first, you should remove all the space from the columns, this would create problems when you have written enough code and one mistake in spacing would stop the program from running.
So since you're columns are in the Syn dataframe, maybe use this to remove the spaces and fill the spaces between words with '_' :
columns = Syn.columns.tolist()
...
1
Gender Analysis is a pretty common trend in data science, especially when it comes to mental health. But breaking it down into categories can be difficult.
I would break it down in to two columns, minimum.
One that is designated as either 'Assigned Male at Birth (AMAB)' or 'Assigned Female at Birth (AFAM)'. This is necessary from a medical standpoint as ...
1
As such we don't have much ways to handle under-fitting issue.
Generally I follow following ways to handle it:
Add more features to training data. That includes deriving new features front the existing one.
Make a complex model.
Increase iterations.
Adding new training data only, I believe will not solve this problem.
1
Well, it always depends, for example, on what model you might be training (i.e. some are robust to multicollinearity). I am pretty sure you are aware, but to have it said as a rule of thumb it is always helpful if you know what you are looking for, rather than hoping naively one function or method would give all the answers.
Said that, there are good ...
1
The normalize function works by calculating z-score for the given data. Now z-score is given by:
$$z_{i} = \frac{x_{i} - mean(X)}{s.d.(X)}$$
where $X$ is your original vector/matrix.
Now, when you give a vector as input it first calculates the mean and standard deviation for the row/column. Then the output you ask for is calculated the above formula. So, if ...
1
Whether you want to do data transformation, really depends on the algorithm that you are using. Tree based algorithms (Decision Tree, Random Forest, Gradient Boosting algorithms), are scale invariant and thus will not benefit from the transformation. While for K-Nearest Neighbors, you probably would want to scale your features, otherwise features with larger ...
1
Firstly, the way that you decide to transform your variables should be dependent on the purpose that you are using them for.
Generally, I would not recommend doing what you stated.
However, something that is commonly done to deal with the problem of variables not being in s a 'similar range' is normalization. To normalize you just subtract the mean from ...
1
I took a look at that link. It's pretty informative. As you already know, The cut function is used to specifically define the bin edges. There is no guarantee about the distribution of items in each bin. In fact, you can define bins in such a way that no items are included in a bin or nearly all items are in a single bin. The qcut function is slightly ...
Only top voted, non community-wiki answers of a minimum length are eligible
