I have a set of data that I'm trying to generate a z-score with. I know I need standard deviation as part of my calculations. I am using the formula of: $\sigma = \sqrt{p * n * (1-p)}$
My data is binary - the value can either go up or down. However, through historical data, I know that it is more likely to go up than down.
So, which value for $ p $ should I use above? Do you use the binary value of $0.50$ or should I use the historical value of my data going up?
I agree with some aspects of Skiddles' answer, but not all.
Assume your data set contains n observations. Based on your question, I see three possibilities:
If you're interested in the number of successes (1s) in n trials, then your standard deviation should be sqrt(np(1-p)), which is the standard deviation for a binomial distribution.
If you're interested in a specific ORDER of successes and failures, then you should be using sqrt(p(1-p)), which is the standard deviation for a Bernoulli distribution.
If you're interested in estimating how the sample proportion of successes will change if you get additional samples, then you should sqrt(p(1-p))/n), which is the standard deviation of the sampling distribution of the sample proportion.
As to what value of p to use, if you don't know, then it's typical to use the most conservative value, 0.5, and conduct a hypothesis test to determine if the actual proportion is different from the hypothesized proportion of 0.5. That would be a Z test.
EDIT based on comment:
The question was "is the historical value of p the right value to use for calculating the standard deviation"?
Answer: No, I think you're confusing two separate but related concepts. You'll often see binomial distribution-type questions saying things like "based on past studies, it is assumed that the probability of success is 0.4" or something like that. However, this is not a case of "past studies", as far as I can see. It looks like you have only one study, in which case you don't really know the population proportion (p), you only have the sample proportion (p hat). You should use 0.5 for p, to calculate the standard deviation.
In your example, you should calculate the probability that the value goes up, and use that as your $p$.
That said, I think the calculation should be $\sigma = \sqrt{p(1-p)}$ otherwise, your standard deviation is inflated by a factor tied to $n$. By using probabilities, you are essentially taking $n$ into account because: $$p = {Number\ of\ "Up"\ instances \over n}$$ Where $n$ is the total number of observations.
Given the nature of the data, that is, it is binary, we would expect the $\sigma$ to be between 0 and 1 and not particularly sensitive to the number of observations.
That said, if we look at a few calculations, the value of $n$ in your formula significantly impacts the $\sigma$.
Assume for simplicity that $p$ is 0.5, that is an equal chance of going up or down. If you have three different values of $n$ you get three vastly different values for $\sigma$.
Example 1: $n$ = 1 (Or not included) $$\sigma = \sqrt{0.5 * (1-0.5)},\ \therefore \sigma = 0.5$$
Example 2: $n$ = 100 $$\sigma = \sqrt{0.5 * 100 * (1-0.5)},\ \therefore \sigma = 5$$
Example 3: $n$ = 1000 $$\sigma = \sqrt{0.5 * 1000 * (1-0.5)},\ \therefore \sigma = 15.8$$
Of the three, only example 1 really makes any sense intuitively. The mean of all instances would be 0.5 and you would expect the observed values to be 0.5 away from the mean.
