1
$\begingroup$

The problem I face is of a language barrier. I have a set of samples consisting of the following data:

Sample number    Organism Count
1                5
2                8
3                5
4                7
5                13
...              ...

I need to find the number of samples required to make an accurate estimate of the organism within a square meter of ground. This is for a school assignment. I don't need an answer to that. What I need to do is find the standard deviations between the set like follows:

(sample 1) = Standard deviation
(sample 1, sample 2) = standard deviation
(sample 1, sample 2, sample 3) = standard deviation

In a rolling way like that. The only information I can find about rolling standard deviation is with a specific window of samples. What I need is to grow from the first one. What is this called? I'm writing a program in python to calculate and display it in a graph. I want to know the name so I can prevent reinventing the wheel in a likely less performant way (if there is a name to this specific question). It is likely that the libraries I'm working with already provide a method to calculate what I'm asking.

So the problem is: I need to find the standard deviation of a 'growing set' and I want to know if that has a specific name, so I can look for it in the documentation of the python library I'm using.

Feel free to change the name of the question to make it better fit, since I didn't know the name I couldn't state it in the title.

$\endgroup$
1
$\begingroup$

If you are using Python, you can use pandas. It comes with an expanding standard deviation function.

import pandas as pd
import numpy as np

# Generate some random data
df = pd.DataFrame(np.random.randn(100))

# Calculate expanding standard deviation
exp_std = pd.expanding_std(df, min_periods=2)

# Print results
print exp_std
$\endgroup$
1
$\begingroup$

The word you might be looking for is "rolling standard deviation" or "running standard deviation". At small scale it might be simplest to just re-compute the standard deviation of each subset. But you can compute the standard deviation of a bunch of numbers without storing all of the numbers, and there are versions that are highly numerically stable too. See for instance https://www.johndcook.com/blog/standard_deviation/ if you're interested in the details or implemetning it yourself, but, you should probably use a library method if possible. Java's Commons Math certainly does this.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.