Expanding Standard deviation

Question

The problem I face is of a language barrier. I have a set of samples consisting of the following data:

Sample number    Organism Count
1                5
2                8
3                5
4                7
5                13
...              ...

I need to find the number of samples required to make an accurate estimate of the organism within a square meter of ground. This is for a school assignment. I don't need an answer to that. What I need to do is find the standard deviations between the set like follows:

(sample 1) = Standard deviation
(sample 1, sample 2) = standard deviation
(sample 1, sample 2, sample 3) = standard deviation

In a rolling way like that. The only information I can find about rolling standard deviation is with a specific window of samples. What I need is to grow from the first one. What is this called? I'm writing a program in python to calculate and display it in a graph. I want to know the name so I can prevent reinventing the wheel in a likely less performant way (if there is a name to this specific question). It is likely that the libraries I'm working with already provide a method to calculate what I'm asking.

So the problem is: I need to find the standard deviation of a 'growing set' and I want to know if that has a specific name, so I can look for it in the documentation of the python library I'm using.

Feel free to change the name of the question to make it better fit, since I didn't know the name I couldn't state it in the title.

Answer 1

If you are using Python, you can use pandas. It comes with an expanding standard deviation function.

import pandas as pd
import numpy as np

# Generate some random data
df = pd.DataFrame(np.random.randn(100))

# Calculate expanding standard deviation
exp_std = pd.expanding_std(df, min_periods=2)

# Print results
print exp_std

Answer 2

The word you might be looking for is "rolling standard deviation" or "running standard deviation". At small scale it might be simplest to just re-compute the standard deviation of each subset. But you can compute the standard deviation of a bunch of numbers without storing all of the numbers, and there are versions that are highly numerically stable too. See for instance https://www.johndcook.com/blog/standard_deviation/ if you're interested in the details or implemetning it yourself, but, you should probably use a library method if possible. Java's Commons Math certainly does this.