An "outlier" is an observation that is so unexpected that we suspect it wasn't valid -- corrupted by noise or something. But what is unexpected? An observation that is highly improbable? But then how do we know what's probable?
Unless you're able or willing to make some assumptions about the distribution that generated these numbers, you can't really declare things outliers. For example, quartiles don't help, since it will only help you find the x% largest or smallest values. But every data set has these -- indeed every data set has a min and max, and being the min or max doesn't mean being an outlier.
In practice, I sense there's an assumption lurking here, that the numbers are probably normally distributed about some mean with roughly some standard deviation. If you know what the mean and/or stdev is supposed to be, you can use it directly to decide how unlikely an observation is and discard it as an outlier if it exceeds a threshold you choose.
You can take the mean and stdev of this sample as a surrogate for that, and for a large enough sample, the sample mean and stdev could be close enough to the real population mean and stdev to work. Here, with such small sets, the outliers influence stats like the mean so much that they throw off attempts to evaluate them in terms of the stats they influence. It's kind of circular.