4
So when we generate labels via machine learning models like clustering above, is it a recommended approach? Only if you can really make highly distinct 2 clusters/groups. This will be highy unlikely, especially for complicated and high dimensional datasets. One of the reasons is that clustering algorithms are just weaker than the supervised algorithms. If ...
3
No, deterministic probability is when you know for certain. If a person does not have a diagnosis, then he doesn't have the disease/condition. Doctors are not supposed to give a probability but we as human beings always like to know the likelihood.
For example, person A who is 27 years old who has the coronavirus is highly unlikely to die of the virus but ...
3
What you are trying to do is text normalization, which is a part of NLP pipeline.
To successfully preprocess your data, please try this below:
Install ekphrasis
pip install ekphrasis
Apply seg & spell correction:
from ekphrasis.classes.spellcorrect import SpellCorrector
from ekphrasis.classes.segmenter import Segmenter
sp = SpellCorrector(corpus="...
3
Yes, you are right that when number of observations are very large, k fold cross validation (CV) are less useful. Let's look at why this is so:
1) Very high number of observations imply high training time for model and validation. Already the number of observations is large for the model to be trained and validated and now we are demanding it to be done k ...
2
Beeing X in the future and beeing X in specific time in the future is just a subset of the first one.
So what one really needs to do is just determine the probabilities (or parameters that give us these probabilities)
P(X|t>30)
Where you can model t, also as your feature. So just fit a model on this data, where you have mutliclassification of:
dead ...
2
Calibration, agreement between observed and predicted risk, is more important in prognostic settings, because we would like to predict future risk of the target population, and the intercept (disease prevalence) is very important
Discrimination, separating people with disease from without disease, is more important in diagnostic settings, because we want to ...
2
Discrimination is the separation of the classes while calibration gives us scores based on risk of the population.
For example, there are 100 people that we’d like to predict a disease for and we know that only 3 out of 100 people have this disease. We get their probabilities from our model. Due to good predictability power, our model predicts probabilities ...
2
1) Manual labeling--- this is not as bad as it sounds. Especially when you apply transfer learning, and for most of the datasets you have a lot of pre-trained models. There are products for that, but also inline python libraries
2) Rule based--- not advisible, since your model will just focus on these if-else rules itself. It would be the the best if these ...
1
Based on the information available above I think the solution below should work
# Number of unique orders per seller
a = pd.pivot_table(df, index = ['seller_id'], values = ['order_id'],
aggfunc = {'order_id' : pd.Series.nunique})
# Products that are being ordered the most per seller
# Products that are being ordered the most in general
b ...
1
Yes you definitely can.
Here's an example: Using Convolutional Neural Networks to Classify Hate-Speech
The authors used classic embeddings concatenated with a vector of size 28 representing the presence or not (in a tweet) of each letter of the alphabet (26) plus any digit (1) plus any other symbol (1).
So basically for a tweet like 'I love NLP!' the ...
1
A decent simple approach is:
Label some data, either manually, or by applying rules to get some unreliable labels with something like https://github.com/snorkel-team/snorkel
Build a model.
Assess which outputs are least certain with something like https://github.com/modAL-python/modAL
Label those manually.
Repeat.
1
You created the labels using the data. If you are able to label them with the data, then why do you need a machine learning model? It simply becomes a rule based classifier. What you would like to do, is to find a function that fits your data points.
For example, if you run a decision tree classifier, then it's going to find perfect splits based on your ...
1
But I read that during model building process, features that were used to create the labels will have to be excluded because they might result in perfect separation of classes and model might fail?
No, just because an expert used them does not mean that that feature only hast to be helpfull or not. If that was the truth than you could write a couple of if ...
1
What you are doing is right. You can build a ML model using this. In your case your input and output are correlated. Consider the salary of an employee wrt his experience. These both are related and sometimes used to derive salary based on experience.
What you might have read is if one feature is used to derive other feature and dont use both features, ...
1
1) I have two classes (Admitted & Not-admitted) 2) Around 25 input variables 3) Run a logistic regression (Statsmodel logit or Scikit-learn?) Do we always have to predict the outcome class to know the risk factors that lead to admission/hospitalization? 5) Then identify the significant risk factors based on p-value.
Not necessary, you can just perform ...
1
I took a look at that link. It's pretty informative. As you already know, The cut function is used to specifically define the bin edges. There is no guarantee about the distribution of items in each bin. In fact, you can define bins in such a way that no items are included in a bin or nearly all items are in a single bin. The qcut function is slightly ...
Only top voted, non community-wiki answers of a minimum length are eligible
