Questions tagged [probability]

Probability is the branch of mathematics concerning numerical descriptions of how likely an event is to occur or how likely it is that a proposition is true.

Filter by
Sorted by
Tagged with
0 votes
1 answer
12 views

Do models of social systems suffer from prediction drift?

Background I've created a binary classification model that predicts the probability of fraud for a given sample. The choice of threshold allows me to set how many frauds are captured in the training ...
  • 523
0 votes
0 answers
9 views

Problem formulation of future timeframe prediction based on current time

I have a problem where I want to predict "when is the next action happening" based on the time. Example problem: Imagine you have a dataset of transactions per user, your goal is to predict ...
0 votes
0 answers
15 views

Getting probability to finish in the last 3 ones after each game week

I'am working on a dataframe with five differents features : team game_week season cum_points : pre game cumulative number of points final_position Those datas are covering 10 seasons of Premier ...
0 votes
0 answers
14 views

Framing a probabilistic time dependent problem

I need help framing the following problem: I have a dataset where I know for each day, at customer level, that someone with device X bought device Y. Example: At day 1 50 people with device X bought ...
0 votes
0 answers
18 views

Probability of a Maximum in a Time Series Given Past Data

I'm trying to predict the peak power usage of an EV charging station. I would like figure out probability bounds given the peak power throughout the month. Imagine that our EV time series consists of ...
1 vote
1 answer
85 views

Trying to extrapolate info from a partial data set - statistical inference

I am wondering if my logic is OK here or not. 98% of a group without a device has an event occur 2% of groups with devices have an event occur. Since we know that correlation isn't causation, I can't ...
  • 63
0 votes
0 answers
11 views

Kernel Density Estimation and performance evaluation

I am doing a data science project about Kernel Density Estimation, specifically about finding the best bandwidth and kernel function to use. I need to use data that I don't know the actual underlying ...
1 vote
1 answer
24 views

probability distribution

Just wanted to know if the value we get by passing, say, random.normal(shape=(3,2)) in the Tensorflow, etc, are normally distributed or if they are randomly chosen ...
0 votes
1 answer
29 views

Precision vs probability

Say I have a model which predicts a class $C_i$ from an input $X$, with a probability of 0.95 i.e $P(C_i| X)=0.95$. That would mean that if we do this over and over, then 95/100 times we would be ...
0 votes
2 answers
33 views

Predicting probability of reaching a milestone -- How much data should I use from production universe to train/test model?

If I am predicting probability of a business to reach (x) milestone (classification 1), but the only data I have is live production data, how much of the production data should I use to train the ...
0 votes
0 answers
12 views

Evaluating models which classify on rolling time intervals

TLDR: I am trying to predict the probability of an incident occurring within a specific time interval. I have data from multiple years, and I know the exact time of year that incidents occur. I have ...
0 votes
1 answer
26 views

Get dependant probabilities in multiclassification

After training my CatBoostClassifier model I call get_proba function which returns me list of probabilities. The problem starts from an another point... I transfer that data into dataframe then to ...
0 votes
0 answers
7 views

What does it mean to "condition' a net's output?

Graves talks about conditioning the predictions of a net based on inputs. What does that mean, and how is it done?
0 votes
1 answer
12 views

How to demonstrate two variables are orthogonal with respect to the output in a 3-D Python dataset?

I have a Python dataset with 300 samples and 3 columns: 2 independent integer variables X,Y and the dependent continuous variable ...
1 vote
0 answers
34 views

Calculationg perplexity (in natural language processing) manually

I am trying to understand Perplexity within Natural Language Processing as a metric more fully. And I am doing so by creating manual examples to understand all the component parts. Is the following ...
  • 125
0 votes
0 answers
11 views

Is there any formula for finding the smallest no. of chapters needed to be learnt for an exam/test, based on the number of questions they can ask?

I understand that this is a highly unconventional and specific question, so bear with me. Also, this is my first time using the site, so be a little lenient with the downvotes. I want to know if there ...
1 vote
1 answer
27 views

whats the difference between these two value function definisions?

I've seen in literature two different yet similar approaches when writing the value function in an MDP: $V_\pi(s)=\sum\limits_{a\in A}\pi(a|s)\sum\limits_{s'\in S}\sum\limits_{r\in R} Pr[s',r|s,a][r+\...
  • 137
0 votes
1 answer
29 views

Good models for predicting whether a customer would make a purchase given details like age, gender, ethnicity, salary, etc?

I have around 30,000 data points and for those data points I have some numerical fields like customer_age, ...
  • 1
0 votes
0 answers
42 views

Practical example of difference between p(y|x) and p(x|y)

I've been reading about the difference between generative models and discriminative models. I know that for generative models we learn the joint probability p(x,y) or just p(x|y) and p(y). For a new ...
  • 93
-1 votes
1 answer
66 views

How is Probability Used in Data Science? [closed]

This is my first Question so apologies if I do not stick to the standards. What I want to understand is how is all of the following topics: Probability Different Probability Distributions. Baye's ...
0 votes
0 answers
22 views

Worse performance on positive class - probability prediction with lightgbm

I would like to predict probabilities in a binary class setting. I want to use the probabilities directly to make decisions, rather than using the exact class label. E.g. I want to vary some features ...
  • 1
0 votes
0 answers
12 views

Role of Expectation and (1-y) in GAN Equation

I am reading about the GAN equation and I was wondering why Expectation was used twice? As of my (lack of) understanding, Expectation is a generalization of the mean because there is an infinite ...
1 vote
2 answers
86 views

Mathematically rigorous NLP

I'm looking for resources (books/articles/whatever) that provide mathematical formalization of NLP and statistical language theory. By that I mean clear exposition of the subject in terms of ...
1 vote
2 answers
45 views

Strategy to choose maximum value from an unknown array of n numbers

Suppose you have an array of n normally distributed numbers whose values are initially unknown(and the probability parameters are unknown too). You must choose one number and you want it to have ...
0 votes
1 answer
46 views

Compute the pdf from pandas kde

I have data (features/targets in machine learning terminology), e.g. X1(t), X2(t), ... XN(t) and dependent variable y(t). I can use pandas to plot the kde's of the independent variables (X1(t),...). I ...
  • 91
0 votes
1 answer
49 views

How can I obtain the mean of a Poisson distribution given the first improbable point of the distribution?

I generated a Poisson distribution with mean equal to 3 and 10000 samples by using np.random.poisson(3,10000). The plot is the following: from this plot I see that ...
0 votes
0 answers
7 views

Part random sample, part force sample with multiple event=0

Question: Is this approach sound? Goal: Model probability of event = 1 Problem: Time series with multiple event = 0 per id, only one event=1 Approach: Get all records where event = 1 (not random ...
  • 1
1 vote
1 answer
32 views

Probability for Nth Place in Race from Bradley-Terry Model Inputs and Outputs

I have created a motorcycle race prediction model that is given pairs of racers and outputs the probability of each rider beating the other in each pairwise comparison. That info is then processed ...
0 votes
0 answers
15 views

Role of segment length in NLP inside-outside algorithm

I encountered an issue with the algorithm used for finding the probability of a string in syntactic parsing in NLP, using the inside-outside algorithm. Here is a section from Christopher Manning and ...
0 votes
1 answer
22 views

Bayesian state description in Reinforcement Learning

What's the best approach to feed a bayesian description of an observed state to a Reinforcement Learning agent? Brief context: I have an agent situated in an environment, which it perceives through a ...
0 votes
0 answers
101 views

Marginal Probability Distribution of Feature space - meaning

I'm reading some literature on Transfer Learning in NLP, and this is one of the definitions that I came across in Pan & Yang (2010) Here is another definition from Sebastian Ruder which is a ...
0 votes
0 answers
47 views

How does factoring probability distributions help?

I'm trying to make sense of machine learning and am reading Deep Learning by Goodfellow, Bengio and Courville. In section 3.14 they show an example of factoring a distribution then say "These ...
  • 1
0 votes
0 answers
9 views

Understanding error in bayesian inference

Let us say we have: Data $X$ Parameter that we are trying to estimate is $\Theta$ The Bayesian estimation method is to Assume a prior on $\Theta$ Sample $x$ from $X$ Use Bayes theorem. Compute the ...
2 votes
1 answer
228 views

Threshold tuning with one-vs-rest for multi classification python

I’m currently using a One vs Rest Random forest algorithm for multi class classification problem using Python, and I want to find the optimal threshold for each class, How can I do this with OVR (One-...
  • 21
0 votes
0 answers
12 views

approximating probability mass function from a large data

I am learning elementary probability; especially I am interested in learning how to find probability mass functions and density functions from data. I think I perfectly understand the theory: For ...
  • 101
1 vote
0 answers
16 views

Inject external prior distribution to my dataset

Input: External Information - distribution between the feature_i & binary_target Internal Dataset - tabular data. ...
  • 21
2 votes
1 answer
191 views

Correct way of calculating probability

I have some data which shows how many orders were made by a certain customer group that bought a certain product type: And the same format but showing how many refunds were made: I am trying to ...
0 votes
0 answers
124 views

XGBClassifier's predictions are not probabilities with objective='binary:logistic'

I am using a XGBoost's XGBClassifier, a binary 0-1 target, and I am trying to define a custom metric function. It supposedly receives an array of predictions and a DMatrix with the training set ...
0 votes
0 answers
36 views

Wave Function in Python

How to apply Wave Function to a Data Set in Python to derive frequency distribution and probability amplitude?
0 votes
0 answers
18 views

Text similarity for badly written text

Consider the following scenario: Suppose two lists of words $L_{1}$ and $L_{2}$ are given. $L_{1}$ contains just bad-written phrases (like 'age' instead of '4ge' or 'blwe' instead of 'blue' etc.). On ...
0 votes
0 answers
32 views

Machine Learning for conditional density estimation

Suppose I have a set of examples $X = (x_1,x_2,..,x_n)$ with continuous numeric targets $Y = (y_1,y_2,..,y_n)$. While it is standard to use regression models to make point predictions of $y_i$ as $f(...
  • 123
0 votes
1 answer
67 views

Is there any way to artificially create a probability calibration for data coming from another model?

I have predictions, which come from a survival model, this model gives me very low probabilities, and I am not sure if they fulfill the real probability of the phenomenon. For example, I calculate $P\...
1 vote
0 answers
26 views

Best metric to evaluate model probabilities

i'm trying to create ML model for binary classification problem with balanced dataset and i care mostly about probabilities. I was trying to search web and i find only advices to use AUC or logloss ...
1 vote
1 answer
68 views

Incorrect example of applying Bayes theorem

I have been reading the book "The Data Science Design Manual" (by Steven S. Skiena) and I came across an example that explained how the Bayes theorem can be applied that confused me and made ...
  • 111
0 votes
0 answers
8 views

Aggregated probability based on multiple predictions on independent samples using the same classifier

i have a understanding question regarding the interpretation of a aggregation of a machine learning classifier. Lets assume i have trained a binary classifier and it was validated with a accuracy of ...
  • 41
0 votes
0 answers
22 views

how to deal with features in pairwaise comparison models?

I am working on a dataset of ATP (Association of Tennis Professionals - men only) tennis games over several years. I want to predict the outcome of tennis so one way to do that is using a Bradley-...
0 votes
0 answers
13 views

the mean and standard deviation aren't the same as those of the input data i provided after sampling

I have a log-normal mean and a standard deviation. after i converted them to the underlying normal distribution's parameters mu and sigma, I sampled from the log-normal distribution however when i ...
0 votes
0 answers
12 views

Maximum entropy optimization for approximating image quality score distribution - as in Google's "Neural Image Assessment" paper

I am asking this question after a thorough research on the internet and having read every single detail of "NIMA: Neural Image Assessment" by Hossein Talebi and Peyman Milanfar. Before ...
1 vote
1 answer
198 views

Log odds vs Log probability

Log-odds has a linear relationship with the independent variables, which is why log-odds equals a linear equation. What about log of probability? How is it related to the independent variables? Is ...
  • 273
0 votes
0 answers
19 views

Understanding of odds

Odds is the probability of an event occurring against the event not occurring. Suppose I play 10 games & I win them all. So my odds of winning are obviously 100%. According to the formula, odds of ...
  • 273

1
2 3 4 5 6