Questions tagged [probability]

Probability is the branch of mathematics concerning numerical descriptions of how likely an event is to occur or how likely it is that a proposition is true.

Filter by
Sorted by
Tagged with
0 votes
0 answers
36 views

Adaptive stopping algorithm find bound that holds with probability

The technique sets aside a validation set Sval, which is used to monitor the improvement of the training process. Let $h_1,h_2,h_3,...$ be a sequence of models obtained after $1,2,3,...$ epochs of ...
  • 101
1 vote
1 answer
33 views

Strategy to choose maximum value from an unknown array of n numbers

Suppose you have an array of n normally distributed numbers whose values are initially unknown(and the probability parameters are unknown too). You must choose one number and you want it to have ...
0 votes
1 answer
14 views

Compute the pdf from pandas kde

I have data (features/targets in machine learning terminology), e.g. X1(t), X2(t), ... XN(t) and dependent variable y(t). I can use pandas to plot the kde's of the independent variables (X1(t),...). I ...
  • 71
0 votes
1 answer
33 views

How can I obtain the mean of a Poisson distribution given the first improbable point of the distribution?

I generated a Poisson distribution with mean equal to 3 and 10000 samples by using np.random.poisson(3,10000). The plot is the following: from this plot I see that ...
0 votes
0 answers
7 views

Part random sample, part force sample with multiple event=0

Question: Is this approach sound? Goal: Model probability of event = 1 Problem: Time series with multiple event = 0 per id, only one event=1 Approach: Get all records where event = 1 (not random ...
  • 1
1 vote
1 answer
22 views

Probability for Nth Place in Race from Bradley-Terry Model Inputs and Outputs

I have created a motorcycle race prediction model that is given pairs of racers and outputs the probability of each rider beating the other in each pairwise comparison. That info is then processed ...
0 votes
0 answers
10 views

Role of segment length in NLP inside-outside algorithm

I encountered an issue with the algorithm used for finding the probability of a string in syntactic parsing in NLP, using the inside-outside algorithm. Here is a section from Christopher Manning and ...
0 votes
1 answer
19 views

Bayesian state description in Reinforcement Learning

What's the best approach to feed a bayesian description of an observed state to a Reinforcement Learning agent? Brief context: I have an agent situated in an environment, which it perceives through a ...
0 votes
0 answers
40 views

Marginal Probability Distribution of Feature space - meaning

I'm reading some literature on Transfer Learning in NLP, and this is one of the definitions that I came across in Pan & Yang (2010) Here is another definition from Sebastian Ruder which is a ...
0 votes
0 answers
39 views

How does factoring probability distributions help?

I'm trying to make sense of machine learning and am reading Deep Learning by Goodfellow, Bengio and Courville. In section 3.14 they show an example of factoring a distribution then say "These ...
  • 1
0 votes
0 answers
7 views

Understanding error in bayesian inference

Let us say we have: Data $X$ Parameter that we are trying to estimate is $\Theta$ The Bayesian estimation method is to Assume a prior on $\Theta$ Sample $x$ from $X$ Use Bayes theorem. Compute the ...
2 votes
1 answer
60 views

Threshold tuning with one-vs-rest for multi classification python

I’m currently using a One vs Rest Random forest algorithm for multi class classification problem using Python, and I want to find the optimal threshold for each class, How can I do this with OVR (One-...
  • 21
0 votes
0 answers
12 views

approximating probability mass function from a large data

I am learning elementary probability; especially I am interested in learning how to find probability mass functions and density functions from data. I think I perfectly understand the theory: For ...
1 vote
0 answers
15 views

Inject external prior distribution to my dataset

Input: External Information - distribution between the feature_i & binary_target Internal Dataset - tabular data. ...
  • 21
2 votes
1 answer
182 views

Correct way of calculating probability

I have some data which shows how many orders were made by a certain customer group that bought a certain product type: And the same format but showing how many refunds were made: I am trying to ...
0 votes
0 answers
51 views

XGBClassifier's predictions are not probabilities with objective='binary:logistic'

I am using a XGBoost's XGBClassifier, a binary 0-1 target, and I am trying to define a custom metric function. It supposedly receives an array of predictions and a DMatrix with the training set ...
0 votes
0 answers
18 views

Wave Function in Python

How to apply Wave Function to a Data Set in Python to derive frequency distribution and probability amplitude?
0 votes
0 answers
17 views

Text similarity for badly written text

Consider the following scenario: Suppose two lists of words $L_{1}$ and $L_{2}$ are given. $L_{1}$ contains just bad-written phrases (like 'age' instead of '4ge' or 'blwe' instead of 'blue' etc.). On ...
0 votes
0 answers
24 views

Machine Learning for conditional density estimation

Suppose I have a set of examples $X = (x_1,x_2,..,x_n)$ with continuous numeric targets $Y = (y_1,y_2,..,y_n)$. While it is standard to use regression models to make point predictions of $y_i$ as $f(...
  • 119
0 votes
1 answer
45 views

Is there any way to artificially create a probability calibration for data coming from another model?

I have predictions, which come from a survival model, this model gives me very low probabilities, and I am not sure if they fulfill the real probability of the phenomenon. For example, I calculate $P\...
1 vote
0 answers
24 views

Best metric to evaluate model probabilities

i'm trying to create ML model for binary classification problem with balanced dataset and i care mostly about probabilities. I was trying to search web and i find only advices to use AUC or logloss ...
1 vote
1 answer
58 views

Incorrect example of applying Bayes theorem

I have been reading the book "The Data Science Design Manual" (by Steven S. Skiena) and I came across an example that explained how the Bayes theorem can be applied that confused me and made ...
  • 111
0 votes
0 answers
5 views

Aggregated probability based on multiple predictions on independent samples using the same classifier

i have a understanding question regarding the interpretation of a aggregation of a machine learning classifier. Lets assume i have trained a binary classifier and it was validated with a accuracy of ...
  • 41
0 votes
0 answers
21 views

how to deal with features in pairwaise comparison models?

I am working on a dataset of ATP (Association of Tennis Professionals - men only) tennis games over several years. I want to predict the outcome of tennis so one way to do that is using a Bradley-...
0 votes
0 answers
9 views

the mean and standard deviation aren't the same as those of the input data i provided after sampling

I have a log-normal mean and a standard deviation. after i converted them to the underlying normal distribution's parameters mu and sigma, I sampled from the log-normal distribution however when i ...
0 votes
0 answers
10 views

Maximum entropy optimization for approximating image quality score distribution - as in Google's "Neural Image Assessment" paper

I am asking this question after a thorough research on the internet and having read every single detail of "NIMA: Neural Image Assessment" by Hossein Talebi and Peyman Milanfar. Before ...
1 vote
1 answer
98 views

Log odds vs Log probability

Log-odds has a linear relationship with the independent variables, which is why log-odds equals a linear equation. What about log of probability? How is it related to the independent variables? Is ...
  • 263
0 votes
0 answers
18 views

Understanding of odds

Odds is the probability of an event occurring against the event not occurring. Suppose I play 10 games & I win them all. So my odds of winning are obviously 100%. According to the formula, odds of ...
  • 263
0 votes
0 answers
50 views

Is this a successful implementation of KL Divergence from scratch, and how can I graph all distributions?

I'm attempting to implement a KL Divergence on two imaginary dice from scratch and I'm not sure if it is correct. By 'how do I graph all distributions' I'm wondering if the KL Divergence just a number ...
  • 157
1 vote
2 answers
80 views

Odds vs Likelihood

Odds is the chance of an event occurring against the event not occurring. Likelihood is the probability of a set of parameters being supported by the data in hand. In logistic regression, we use log ...
  • 263
1 vote
0 answers
21 views

How are the weights defined in a (linear-chain) Conditional Random Field?

Edit: i saw that i mixed up i (in the graph) and t (in the formula), in the following i equivalent to t I am trying to understand the theory behind linear chain Conditional Random Fields. I have now ...
  • 11
1 vote
1 answer
35 views

How to test likelihood hypothesis on dataset?

How to test the following hypothesis? The larger the fare the more likely the customer is to be travailing alone. Using the data below, how would one be able to test the hypothesis? ...
1 vote
1 answer
23 views

memory error- python N-th order Markovian transition matrix from a given sequence

Ok. What is wrong with you code! I am trying to calculate transition probabilities for each leg. The code works for small array but for the actual dataset I got memory error. I have 64 g version ...
1 vote
1 answer
81 views

Probability of the occurrence of an event over time

I need to answer the following question: What is the probability that the event 1 will occur at some point in the time for a new sample? At which time point is this more likely to occur? 1: event ...
  • 11
1 vote
0 answers
19 views

Predictions using calibrated classifer

I find myself asking alot of calibration related questions recently - but i cannot find adequate material on it! I am training a binary classifier to predict default. This probability will be used in ...
  • 486
0 votes
0 answers
51 views

How can I interpret my rho risk values when performing probabilistic time series forecasting?

I am currently exploring different probabilistic time series forecasting models for car sales data and have planned to evaluate the probabilistic forecasts with the metrics rho-risk as described on ...
0 votes
0 answers
15 views

Policy from an ensemble tree model?

I'm trying to output a policy (pdf on a fixed number of finite labels) from a Random Forest model. What is the best way to achieve this? I'm thinking that I can just normalize the votes from all the ...
2 votes
1 answer
82 views

Generating a random numpy ndarray of 0 and 1 with a specific range of 1 values

I want to generate a random numpy ndarray of 0 and 1. I want that the number of occurrences of 1 in every specific rowto range between 2 and 5. I tried:...
  • 51
1 vote
1 answer
20 views

What thought processes do people use for generating priors on a variable's probability distribution?

Example Consider a block of text with a variety of sentence types within it, of which there are 7. Within a text these sentences will be more or less likely to appear, dependent on where in the text ...
  • 133
0 votes
1 answer
72 views

Imbalanced data: understanding example from Bishop PRML book?

I'm trying to understand the 3-step procedure to compensate for the effects of imbalanced data described in Section 1.5.4 - pg 45 of Bishop's PRML book. Please refer to the following excerpt from the ...
  • 3
0 votes
0 answers
18 views

How come the cost function changed in linear regression in andrew Ng stanford course in two different videos?

I'm a little bit confused, here is the cost function Andrew gave at the first place to minimize but when he derived using probability, we minimize the same one but in a different sign here h(theta) ...
1 vote
1 answer
59 views

Probability Distribution on the Test Set

How can one interpret the probability distibution of the predictions for the target of the test set? For example if we wanted to interpret the plot below, can we say it is overfitted? Here x axis ...
  • 11
0 votes
0 answers
13 views

Why use Bayesian network instead of joint probability distribution table?

Is there a reason, except the number of parameters why one prefers to use a bayesian network instead of just a joint probability distribution table ?
0 votes
0 answers
21 views

clustering - purchase probability - machine learning

I'm looking for a machine learning scenario: Data: time, when a customer enters the store weight, how much the customer bought Can I use machine learning to train an algorithm, which will give me a ...
  • 1
0 votes
2 answers
66 views

Probability vs Odds

I know what Probability & Odds ratio is, but I want to know under what circumstances each is used. When is Odds preferred over Probability?
  • 263
3 votes
2 answers
144 views

Analysis of probability distribution of each features and Machine Learning

While I know that probability distributions are for hypothesis testing, confidence level constructions, etc. They definitely have many roles in statistical analysis. However, it is not obvious to me ...
  • 359
0 votes
0 answers
11 views

Estimating the Probability of Sequence of Events Based on the Historical Data

I am working on the project where I need to estimate the delivery time(PICKUP to DROPOFF) based on the delivery route data by individual deliverers. So, I might need to figure out: How many packages ...
1 vote
0 answers
24 views

Shannon Information Content related to Uncertainty?

I'm a data scientist student currently writing my master thesis which resolves around the Cross Entropy (CE) Loss Function for neural networks. From my understanding, the CE is based on the Entropy, ...
  • 11
1 vote
0 answers
29 views

Train a parametrized model to sample from a known target distribution

I wonder if there is a way to train a parametrized model to sample from a known distribution such as Gaussian. We usually don't need a model to sample from a known distribution (if we know the CDF for ...
0 votes
0 answers
10 views

Approximation of a confidence scores from a neural network with a final softmax layer: Softmax vs other normalization methods

Say that there is a neural network for classification and the 2nd to last layer are 3 nodes, and the final layer is a softmax layer. During training the softmax layer is needed, but for inference it ...

1
2 3 4 5 6