Questions tagged [probability]
Probability is the branch of mathematics concerning numerical descriptions of how likely an event is to occur or how likely it is that a proposition is true.
295
questions
2
votes
2
answers
195
views
Modeling the influence of events order on probability
The case is to model if the sequence of events influences the probability of binary target variable. We have for example five different events which occur in time (event: A,B,C,D,E). They can occur in ...
CommunityBot
- 1
0
votes
1
answer
201
views
Compare cross validation values of Bernoulli NB and Multinomial NB
I'm testing the Multinomial NB and Bernoulli NB on my dataset and I'm using the cross validation score to better understand which of the two algorithms work better. This is the first classifier:
...
CommunityBot
- 1
2
votes
2
answers
608
views
Odds vs Likelihood
Odds is the chance of an event occurring against the event not occurring.
Likelihood is the probability of a set of parameters being supported by the data in hand.
In logistic regression, we use log ...
CommunityBot
- 1
2
votes
1
answer
286
views
Threshold tuning with one-vs-rest for multi classification python
I’m currently using a One vs Rest Random forest algorithm for multi class classification problem using Python, and I want to find the optimal threshold for each class, How can I do this with OVR (One-...
CommunityBot
- 1
9
votes
4
answers
3k
views
Loss Function for Probability Regression
I am trying to predict a probability with a neural network, but having trouble figuring out which loss function is best. Cross entropy was my first thought, but other resources always talk about it in ...
CommunityBot
- 1
0
votes
1
answer
46
views
Why Maximum Likelihood Estimation for normal distribution?
Since we can compute the mean and the standard deviation of a set of random variables, why do we use Maximum Likelihood Estimation to estimate these parameters?
- 191
0
votes
0
answers
31
views
Looking for suggestions on how to distinguish between two different distributions in my data when no labels are available
My data consists of medical claims from thousands of providers across the country. The data contains how much they were paid for each service they performed and how many units of each service they ...
- 271
0
votes
0
answers
11
views
Understanding the calculations of bayesian classification rule given features with known distribution
I have some difficulties using the Bayes rule and the calculations used for estimation of a given class in a classification task if we know the distribution (and its parameters) of the feature vector:
...
- 131
1
vote
2
answers
105
views
Intuition behind using non-hypercubic kernels in density estimation
Suppose that we perform density estimation in m-dimensional space: we estimate the value $p(a)$ for some point $a$ given observations $\{x_1, \dots, x_n \}$.
It is known that if region $A \subset \...
CommunityBot
- 1
1
vote
1
answer
90
views
Trying to extrapolate info from a partial data set - statistical inference
I am wondering if my logic is OK here or not.
98% of a group without a device has an event occur
2% of group with device has an event occur
Since we know that correlation isn't causation I can't say ...
- 63
1
vote
1
answer
109
views
Probabilistic Machine Learning model to match spatial data
I have spatial data from multiple sources. This data consists of ID, lat, long, and time.
My goal is that given a new lat-long, the model needs to return (preferably with a probability) the data ...
CommunityBot
- 1
1
vote
2
answers
213
views
Use distribution probability as a feature in ML model
I built an LSMT model to predict sick cows. I also have risk factors like cow size and height (static risk factor) that I want to combine into the ML model. I found that size is geometrically ...
CommunityBot
- 1
0
votes
1
answer
525
views
derivation for expected value for variance
Hi Im taking a course about probability distribution in datascience and below is derivation of the expected value for the variance
Variance = expected value of the squared difference from mean for ...
CommunityBot
- 1
3
votes
1
answer
62
views
Marginalization of joint distribution
I am trying to understand how you marginalise a joint distribution.
In my case I have a fair coin, $P(C) = \frac12$ and a fair dice $P(D) = \frac16$. I am told I win a prize if I flip the coin and ...
CommunityBot
- 1
0
votes
0
answers
7
views
Does the central limit theorem and confidence intervals apply to any distribution, including memoryless distributions?
Is it possible to assign a 1 or a 0 to values in any distribution, be it a memoryless power law or pareto distribution to match some criteria such as if a value is under 10 it gets a 1, otherwise it ...
- 1
4
votes
3
answers
232
views
Notation for features (general notation for continuous and discrete random variables)
I'm looking for the right notation for features from different types.
Let us say that my samples as $m$ features that can be modeled with $X_1,...,X_m$. The features Don't share the same distribution (...
CommunityBot
- 1
1
vote
3
answers
2k
views
Non-mutually exclusive classification sum of probabilities
So I have the following problem: I realized (while writing my master thesis) that I am still not sure/have vague descriptions of some of the machine learning principles. I already asked one question ...
CommunityBot
- 1
0
votes
0
answers
26
views
Common cross-validation code: why does it work?
The following Python code is common practice when creating a folds column for multi-label stratified k-fold cross-validation:
...
- 101
1
vote
1
answer
61
views
memory error- python N-th order Markovian transition matrix from a given sequence
Ok. What is wrong with you code!
I am trying to calculate transition probabilities for each leg.
The code works for small array but for the actual dataset I got memory error. I have 64 g version ...
CommunityBot
- 1
0
votes
1
answer
210
views
correcting conditional and marginal distribution in transfer learning
I understand that in case of transfer learning, we can have the target and the source data having different domain distributions. In such cases, authors in many papers suggest bringing the marginal ...
CommunityBot
- 1
1
vote
3
answers
248
views
How to predict when an appointment will be scheduled?
I have a dataset of tens of thousands of appointments. Appointments have a created date and scheduled date. Something like this:
...
CommunityBot
- 1
0
votes
1
answer
15
views
Do models of social systems suffer from prediction drift?
Background
I've created a binary classification model that predicts the probability of fraud for a given sample.
The choice of threshold allows me to set how many frauds are captured in the training ...
1
vote
1
answer
191
views
Update of mean and variance of weights
I'm trying to understand the Bayes by Backprop algorithm from the paper Weight Uncertainty in Neural Networks, the idea is to make a NN in which each weight has it's own probability distribution. I ...
CommunityBot
- 1
1
vote
1
answer
202
views
Probability of the occurrence of an event over time
I need to answer the following question:
What is the probability that the event 1 will occur at some point in the time for a new sample? At which time point is this more likely to occur?
1: event ...
CommunityBot
- 1
1
vote
1
answer
175
views
Probability of Gaussian Naive Bayes
How would I go about attaching a probability to the prediction outputted by a Gaussian Naive Bayes model ?
I'm asking because the predict_proba function U can use with sklearn's Gaussian Naive Bayes ...
CommunityBot
- 1
0
votes
0
answers
11
views
Problem formulation of future timeframe prediction based on current time
I have a problem where I want to predict "when is the next action happening" based on the time.
Example problem: Imagine you have a dataset of transactions per user, your goal is to predict ...
- 1
0
votes
0
answers
15
views
Getting probability to finish in the last 3 ones after each game week
I'am working on a dataframe with five differents features :
team
game_week
season
cum_points : pre game cumulative number of points
final_position
Those datas are covering 10 seasons of Premier ...
1
vote
1
answer
87
views
How to compare Poisson Point Process, ARIMA and LSTM?
I am trying to compare three forecasting techniques:
A stationary stochastic Poisson-GEV: where the rate of occurrence of the events is given by a Poisson process and, it's intensity is given by a ...
CommunityBot
- 1
0
votes
0
answers
14
views
Framing a probabilistic time dependent problem
I need help framing the following problem:
I have a dataset where I know for each day, at customer level, that someone with device X bought device Y.
Example:
At day 1
50 people with device X bought ...
- 101
0
votes
0
answers
20
views
Probability of a Maximum in a Time Series Given Past Data
I'm trying to predict the peak power usage of an EV charging station. I would like figure out probability bounds given the peak power throughout the month.
Imagine that our EV time series consists of ...
- 1
4
votes
1
answer
148
views
Relation between an underlying function and the underlying probability distribition function of data
I heard and read a lot of times the following statements and got a lot of confusion over time.
Statement 1: The goal of machine learning is to get a function from
the given data
Statement 2: The goal ...
CommunityBot
- 1
0
votes
0
answers
14
views
Kernel Density Estimation and performance evaluation
I am doing a data science project about Kernel Density Estimation, specifically about finding the best bandwidth and kernel function to use.
I need to use data that I don't know the actual underlying ...
1
vote
1
answer
247
views
Log odds vs Log probability
Log-odds has a linear relationship with the independent variables, which is why log-odds equals a linear equation.
What about log of probability? How is it related to the independent variables? Is ...
CommunityBot
- 1
1
vote
1
answer
27
views
probability distribution
Just wanted to know if the value we get by passing, say, random.normal(shape=(3,2)) in the Tensorflow, etc, are normally distributed or if they are randomly chosen ...
- 19.8k
2
votes
1
answer
131
views
Store's unseen items sales forecasting
I am working on sales forecasting problem.I am able to provide data about which items got sold and not sold to the algorithm.How to provide algorithm information about items that are not present in ...
CommunityBot
- 1
1
vote
1
answer
298
views
Very low probability in naive Bayes classifier 1
I have some training data (TRAIN) and some test data (TEST). Each row of each table contains an observed class (X) and some columns of binary (Y). I'm using a Python script that is intended to predict ...
CommunityBot
- 1
3
votes
2
answers
333
views
Analysis of probability distribution of each features and Machine Learning
While I know that probability distributions are for hypothesis testing, confidence level constructions, etc. They definitely have many roles in statistical analysis.
However, it is not obvious to me ...
CommunityBot
- 1
0
votes
1
answer
67
views
Precision vs probability
Say I have a model which predicts a class $C_i$ from an input $X$, with a probability of 0.95 i.e $P(C_i| X)=0.95$. That would mean that if we do this over and over, then 95/100 times we would be ...
- 164
0
votes
2
answers
35
views
Predicting probability of reaching a milestone -- How much data should I use from production universe to train/test model?
If I am predicting probability of a business to reach (x) milestone (classification 1), but the only data I have is live production data, how much of the production data should I use to train the ...
0
votes
0
answers
13
views
Evaluating models which classify on rolling time intervals
TLDR: I am trying to predict the probability of an incident occurring within a specific time interval. I have data from multiple years, and I know the exact time of year that incidents occur. I have ...
- 1
0
votes
1
answer
33
views
Get dependant probabilities in multiclassification
After training my CatBoostClassifier model I call get_proba function which returns me list of probabilities. The problem starts from an another point... I transfer that data into dataframe then to ...
0
votes
0
answers
7
views
What does it mean to "condition' a net's output?
Graves talks about conditioning the predictions of a net based on inputs. What does that mean, and how is it done?
- 101
0
votes
1
answer
15
views
How to demonstrate two variables are orthogonal with respect to the output in a 3-D Python dataset?
I have a Python dataset with 300 samples and 3 columns: 2 independent integer variables X,Y and the dependent continuous variable ...
- 4,454
1
vote
1
answer
33
views
whats the difference between these two value function definisions?
I've seen in literature two different yet similar approaches when writing the value function in an MDP:
$V_\pi(s)=\sum\limits_{a\in A}\pi(a|s)\sum\limits_{s'\in S}\sum\limits_{r\in R} Pr[s',r|s,a][r+\...
- 818
-1
votes
1
answer
72
views
How is Probability Used in Data Science? [closed]
This is my first Question so apologies if I do not stick to the standards.
What I want to understand is how is all of the following topics:
Probability
Different Probability Distributions.
Baye's ...
- 11
1
vote
0
answers
36
views
Calculationg perplexity (in natural language processing) manually
I am trying to understand Perplexity within Natural Language Processing as a metric more fully. And I am doing so by creating manual examples to understand all the component parts. Is the following ...
- 125
0
votes
0
answers
11
views
Is there any formula for finding the smallest no. of chapters needed to be learnt for an exam/test, based on the number of questions they can ask?
I understand that this is a highly unconventional and specific question, so bear with me. Also, this is my first time using the site, so be a little lenient with the downvotes.
I want to know if there ...
0
votes
1
answer
33
views
Good models for predicting whether a customer would make a purchase given details like age, gender, ethnicity, salary, etc?
I have around 30,000 data points and for those data points I have some numerical fields like customer_age, ...
- 1
0
votes
0
answers
43
views
Practical example of difference between p(y|x) and p(x|y)
I've been reading about the difference between generative models and discriminative models. I know that for generative models we learn the joint probability p(x,y) or just p(x|y) and p(y). For a new ...
- 93
0
votes
0
answers
26
views
Worse performance on positive class - probability prediction with lightgbm
I would like to predict probabilities in a binary class setting. I want to use the probabilities directly to make decisions, rather than using the exact class label. E.g. I want to vary some features ...
- 1