One way to estimate the level of confidence we have about an ANN prediction is to use dropout perturbations. The idea was proposed in this paper: Dropout as a Bayesian Approximation. Representing Model Uncertainty in Deep Learning. The core idea is to use dropout as a perturbation method, and check how predictions change with varying levels of dropout. Once ...
What you do in business is program algorithms to solve biz problems.
Most all common problems have algorithms and processes already. So what you would do is adopt and apply it to your particular situation.
Data science being the buzzword du jour, and very poorly understood by almost everyone as to what it is and how to use it, is more a fancy word ...
It is indeed extremely rare for someone to develop a novel algorithm to solve their problem. In my experience it is more important to understand the business domain, how to normalize the data and choose what loss function should be minimized.
But it is very valuable to have experience with various kinds of algorithms so that you can pick the right tool for ...
In industry its usually variations (but important ones) of the ground ideas.
Look at this boosting timeline:
(Ada)Boosting Formally by two profesors in 2003
xgboost by DLMC Distributed Machine Learning Community in 2014
lightgbm by Microsoft in 2017
catboost by yandex in 2017
+- all the variations in between that did not caught up
Building on "basic" idea ...
Calculation of standard deviation on the fly is possible (turn to our brothers at math.stackexchange):
It is easier to keep track of the variance and only take the square root to calculate the stdev when you really need it.
And the mean is even easier,...
I am no data scientist, only an aspiring one for two years, moving from my background in software engineering and mathematics. So I took some courses, had some interviews, read a lot on the subject online. My opinion:
New algorithms are developed in research centers and at universities. And even then, most algorithms used in companies are already developed, ...
I don't think machine learning is the right tool. It's difficult (at least for me) to formulate this as a learning problem.
You could consider genetic algorithms as an alternative approach. Based on your description of how you do the matching by hand, it seems like the following assumptions hold:
You're looking for an assignment of payments to loans such ...
This Paper by Chih-Wei Hsu et All is a good starting point for kernel selection. On page 3 they suggest using RBF kernel and fine tune it.
Andrew Ng also provide a very high level thumb rule in his video on SVM - If number of observations are larger than features use Gaussian kernel. Use linear otherwise.
This seems to be a pretty common scenario in digital marketing, and a few companies have published their approach to lookalike modeling.
Here are a few links:
Lookalike at LinkedIn
Lookalike at Pinterest
Lookalike at Yahoo
Another lookalike from Yahoo
Academic paper on lookalike (not sure where the authors work)
Google's lookalike patent (this one is a lot ...
If you have past data you could create a machine learning binary classification model to predict if any of the customers look like them.
This should give you for each new customer if it has a similar behaviour than the past one and whether or not send a promotion.
This is indeed the expected behavior.
(N.B. You should not balance test sets, since they are supposed to inform you about performance on unseen, original-distribution data.
In a binary classification, should the test dataset be balanced?
A quick example may be clearer than being general. Let's fix a ...
Just stick to F1.
Yes NRI is cools but there are edgecases and its not a magic bullet. Read this paper
IF you really want to implement it you can embedd r programs in python.
Since there is already implementation in r what you could do is something like this. Just change it to use NRI.
A small remark to the often suggested mean/median imputation.
Applying this method would assume that your analysis is only dependent on the first moment of your variable´s distribution.
Just imagine you would impute all values of your variable with mean/median. The mean/median probably would have very low bias. But the variance would go (close to) zero. ...
As far as I can tell, you use a "normal" Logit in one approach and a Logit with L1 penalty in the other case (penalty': ['l1']), which is called "Lasso". In Statsmodels L1 penalty would be implemented like stated in the docs.
Lasso and "normal" Logit are two different approaches. In the first case, (some) parameters are shrunken and can be set to zero, ...
scikit learn itself has some good ready to use packages for imputation. details below
MICE is not available in scikit learn as far as i know. Please check statsmodel for MICE
So how many classifiers did it improve, only 1 than dont add it.
Formally there are a couple of statistical tests.
Cochran's Q test
Is a generalisation of the McNemars test for comparing Machine Learning models.
or read this formal paper where they discuss it.
A trick I have seen on Kaggle.
Step 1: replace NAN with the mean or the median. The mean, if the data is normally distributed, otherwise the median.
In my case, I have NANs in Age.
Step 2: Add a new column "NAN_Age." 1 for NAN, 0 otherwise. If there's a pattern in NAN, you help the algorithm catch it. A nice bonus is that this strategy doesn't care if it'...
To decide which strategy is appropriate, it is important to investigate the mechanism that led to the missing values to find out whether the missing data is missing completely at random (MCAR), missing at random (MAR), or missing not at random (MNAR).
MCAR means that there is no relationship between the missingness of the data and any of the values.
Start with linear kernel and see if your data is linearly seperable or not. Performing that is simpler than looking for early indications.
Linear kernels are suggested when the number of features is larger than the number of observations in the dataset (otherwise RBF will be a better choice).
However, once you conclude that you have non-linear data, you ...
It sounds like you are interested in multivariate time series forecasting. Below are some references to an example that sounds similar -- using multiple input variables to forecast air quality.
Kaggle competition- see Discussion tab for some nice examples and, well, discussion.
I think the most highly-referenced source for these terms is Horizontal and Vertical Ensemble with Deep Representation for Classification (Xe, Xu, Chuang 2013). That would be the best place to get a technical answer to your first question. For purposes of searching, you could also look into "Stacking" as a synonym for Vertical ensembling. I will provide some ...
Giving a single label in a model is not the thing i have heard ever before for labeled data, if you give like that also internally it is binary model only for eg : take log regression model example we have a loss function for all model and for log regression we have : J(w)=∑i=1_to_m y(i)logP(y=your label)+(1−y(i))logP(y= not your label)?
you are not showing ...
Solutions to over fitting:
Simplify model by reducing the number of attributes in the training data or by constraining the model(regularization).
Gather more training data
Reduce the noise in the data (Fix data errors, remove outliers)
Solutions to under fitting:
Reducing the constraints in the model.
To make good predictions your ...
Complexity, just look at this simple deplyoment from fast ai. you get a web app in a couple of simple steps. Where as for restapi you need to know your way around a lot more libraries. For example the cool fastapi (flask+more capabilities). Example app
Online Learning there is no possibility to update (not trivial atleast) your model in the pickeled file. ...
It appears (if I understand you correctly) that you want to model the causal effect of some confounders $X$ on some outcome $y$ by using Logit. If this is correct, and if you know the outcome $y$, you should be fine with using just Logit, because in this case (and if you are able to contol for all relevant confounders $X$), you can identify the marginal ...
A lot of questions here. Here are some thoughts.
should you be happy about a 1 point increase in AUC? Yes. An effect can be genuine, but small. A 1 point advantage is still an improvement. But do I trust that outcome? Not sure.
you need some more data. Your sample size is not large. Furthermore, cross-validation is a wonderful thing, but you have been ...
It sounds like a great opportunity for feature engineering. You're contemplating this in your last paragraph so you're on the right track, but I'll elaborate on a possible solution here.
You could use the features that you know will exist in the test set to construct synthetic features based on the context information. For example, you could predict the ...
Xgboost does the feature selection for you. If you want to report how valuable certain features are for prediction, print the feature importances. Those will, however, just tell you "feature $x_1$is very important for predicting the outcome, feature $x_2$ is nearly useless for predicting the outcome, etc.".
To get risk factors with p-values, you need a ...
Though most of the Libraries will through an error for this, even if we manage(let's assume) to create the features based on any logic e.g. Fill NAN, it will not work
ML model just creates a pattern based on data. If it has created a pattern using some set of features and at the moment of prediction those features are unavailable it will definitely impact ...
I would suggest you to:
Balance your dataset (since you are not doing an anomaly detection task, that should be fine).
Measure Precision/Recall/F1 Score and argue for them depending on your problem.
Perform K-Fold Cross Validation to compare models
Balancing your dataset
You said you have a 33:67 ratio. Why keep it like that an not just make it 50:...
F1 is just based on the confusion matrix(and taking into account class imbalance), hence different models should only focus on predicting the confusion matrix correctly and if they dont they are wrong not sensitive.
F1Score is a metric to evaluate predictors performance using the formula
F1 = 2 * (precision * recall) / (precision + recall)
recall = TP/(...
Yes it's correct. You can do it in two ways:
Classical regression approach: you feed sequence [ A B C D ] to predict [ E ], or [ E F G ] in case of multistep prediction.
Seq2seq approach: you feed sequence [ A B C D ] to predict sequence [ B C D E ] - i.e. the same input sequence but shifted forward.
Both approaches can work. If you are working with series ...
There is no one size fits all solution.
AutoML is cool, but you wont get tailored and best-possible solutions using it.
Reason being is the fact that DS has an "art" component to it. Sure theoretically you can put everything in an huge optimization framework and find the optimum params but realistically it will take for ever. Maybe with quantum computers ...
1) Feature Selection should be done by AutoML on the other hand preprocessing is normally done by the user in order to make sense fo the data.
2) AutoML takes care of the hyper-parametrization.
3) The disadvantage that I mostly find is that is extremely computationally expensive. And from what I have seen in Kaggle most of the winning solutions use manual ...
I do not agree that Bootstrapping is generally superior to using a separate test data set for model assessment.
First of all, it is important here to differentiate between model selection and assessment. In "The Elements of Statistical Learning" (1) the authors put it as following:
Model selection: estimating the performance of different models in
Feature selection is a combinatorial optimization problem. And genetic algorithms is an optimization technique.
So there really isn't anything special, you just need to formulate your problem as an optimization one, and understand how do genetic algorithms optimize. There are enough tutorials on this.
Whether it's better or worse you already know the ...
1) Whether is there any other consideration for determining the number of predictors in logistic regression model?
The right number of predictors depends on your data and on your theories about data, and on that only. All these rules of thumb seem completely arbitrary to me, and they lack any scientific ground.
2) How do people calculate min sample size ...
I am just gonna copy from Oreilly's book Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow, 2nd Edition:
With bagging, some instances may be sampled several times for any given predictor, while others may not be sampled at all. By default a BaggingClassifier samples m training instances with replacement (bootstrap=True), where m is the size ...
Thats not how it works, there is not juice to be extracted if the data is missing from the test set.
You will have features in train that might be discriminative, but missing in test. Then when you try to predict there wont be any inputs that you can map to the outputs.
This beeing said I should mention that NaiveBayesclassifier will not use missing ...
Common use cases include:
Transactions volume prediction
Next transaction date
This is usually tackled with anomaly detection.
It requires information on the two transaction parties and using machine learning to figure out when a transaction is out of the norm and flagging as a potential case of fraud.
Transactions volume ...
While I'm not sure what you mean when you say "adjust for confounders", I suppose your question is about model choice (or variable/feature selection).
Here are some thoughts on this problem:
Clearly define what you want to achieve: If you want to achieve a good prediction (so you are not up to causal modeling), choose a suitable metric to measure model ...
Confounder (lurking variable) is a variable that influences both the dependent variable and independent variable. While you are right that feature interactions are "missing" in logistic regression I am not sure how can "adjusting for confounders help"
What can definitely help is including these interactions in log.regression formula IF there are any ...
You had a post where we discussed causality, but with ML models assumption is that data represents your problem entirely and has all the information in it. In other words every pattern that you pickup in your train data you can expect it to behave pretty similar in production, hence with this assumption what you want is to "evaluate the ...
I've never practiced this package myself, but I've read a few analyses based on SHAP, so here's what I can say:
A day_2_balance of 532 contributes to increase the predicted output. In this area, such a value of day_2_balance would let to higher predictions.
The axis scale represents the predicted output value scale. The actually predicted value is in bold ...
The subsets of features that you select might vary between algorithms and also between parameters of the same algorithm.
You can try, for the same algorithm try different hyperparameters, the feature importance might change.
There is no guarantee that you get the same subset of features from any feature selection algorithm. There might be multiple subsets that all perform approximately equally. For instance, perhaps feature C is highly correlated with feature I. So, there is no reason to be surprised by what you are observing. There is no notion of "too low".
Problem in this competition is VERY similiar. Instead of me copy-pasting some interesting ideas just check out winner write-ups in the discussions.
Add count of values of features as new features.
This is information that LightGBM can’t see
Check unique value counts for all features of train and test set.
Model stacking almost always works ...