"400 of anything is..." Topic

… always a statistically significant sample.

I'm sure this isn't techncally correct, and my survey-level statistics class in college was a long time ago. However, I seem to remember the professor saying that 400 of anything (dice rolls, rutabagas, whatever) would, unless you were dealing with astronomically large numbers, always give you a pretty statistically significant sample to use for whatever you were trying to figure out.

Any statisticians out there?

30 is infinity in the world of normal distribution

I lived in the world of small sample statistics, where we were happy to get a dozen data points…

As important as the sample size is the sampling methodology. You pick 400 residents of Waco and try to extrapolate to the nation you would probably not get valid results.

The movie claimed "300" was enough.

In medieval Japan, seven was enough.

In Texas or Canada, one is enough…

A sample size of 187 will provide you with at least 95% power for any statistical tests you carry out (the probability of a Type II error). As already stated anything over 30 will remove any statistically significant random chance. Any level of confidence or significance is significantly tighter with larger samples. 400 (187+ either way)should cover you for absolutely any two-tailed significance tests you want to carry out.

I can say this … if rolling a d120 … would only give a distribution of 3 1/3 rolls per side … I cant imagine that would be statistically significant … However … for a D6 that would be 66 2/3 rolls per side … which should be significant … I suppose some stats whiz out there could tell us a formula of statistically significant with regard to the degreees of freedom (ie for a die … the number of faces)

… the average size of a Civil War regiment, circa 1863.

The only caveat I'd make is that if the distribution of your data isn't close to normal, or if you're dealing with sub-populations with very different standard deviations, no such simple rule of thumb will necessarily apply. Apart from those truly pathological instances, yes 400 is darn safe.

Of course that presumes your sample is somewhat representative or random -- otherwise see Waco Joe's concern above.

JJW-T.C.H.
(Who passed a Ph.D. level general exam in stats, but doesn't claim that means too much)

"Canada, one is enough"

We just take the US numbers and divide by ten. Saves a fortune at stats Canada.

If the sample is truely random and representative of the population then 400 is safe. So asking the members of this forum would probably satisfy the first criterion (some members being more random than others) but not the second.

Polling companies, such as NOP, use 1,000 made up of a statistically representative sample (51% women etc). My Stats lecturer said this was considered to be 'safe'

Though how 51% is a minority I don't understand.

But probably not sensitive enough if you really need to determine if the difference between 49.8999% and 49.9000% is significant.

Oh my.

There is a certain amount of unintentional misinformation in the postings to date, even ignoring the tongue-in-cheek responses.

Void Trekker,

The term "statistically significant sample" has no well-defined meaning in statistics. So without meaning any offense to you, I will take some liberties in answering what I think you question may have been. Namely, if we assume a reasonably representative sample taken from some larger population of similar items, how confident can we be in making judgments about a typical member of the larger population if the sample size is 400?

As John wrote, you are probably "safe". The Last Hussar's answer (1,000) will make you even "safer".

This is what's going on: if we have an average uncertainty of some amount, which I'll call "s", in a single observation, then the average of a sample size of 400 will have an average uncertainty of s/sqrt(400), or 5% of the value of s. (sqrt = square root.) Not bad, considering ROPlayer's sample of size 30 gives us an average uncertainty of s/sqrt(30) or 18.3%. If we increase the sample size in the other direction, going to a sample size of 1,000 will result in an average uncertainty of 3.2%. So, in this case we get a decrease in the uncertainty factor of 40% at the cost of taking 2.5 times as much data, which may or may not be worth it to us, depending on the situation.

You should have noticed that the key determinant in my answer is 1/sqrt(sample size).

Notes (no statistician ever writes anything without a lot of notes and be warned, my use of English stopped above):

(1) I have deliberately kept this in English, so most readers could follow along. When I write about the average uncertainty of a single observation, I mean the standard deviation (or square root of the second central moment) of the population. When I talk about the average of the sample, I am referring to the sample mean, so s/sqrt(sample size)is the standard error of the sample mean. Why do I do this? Because the mean is a good estimate of what a typical member of the population looks like, based on the sample.

(2) The reference to Type II error in one of the posts is not relevant since you have not posed any hypothesis testing. Even so, it is a misstatement. Except in limited cases related to the Neyman-Pearson Theorem which occur rarely in practice, we can never determine the power of a statistical test. We usually seek to maximize it by taking as large a sample as is economically feasible. If the boss wants to have a test of a certain power against a specified simple alternative hypothesis, then we can use that to set a minimum sample size. Often, when the boss finds out how much it will cost, the boss settles for less power.

(3) Whether we can distinguish between 49.8999 and 49.9000 is more a function of the standard deviation s. In other words, it depends. If we are measuring with a laser interferometer, no problem! A sample of size 1 or 2 will do it. But if we are measuring with a yardstick marked off in quarter-inch increments (or a meter stick marked off in centimeters), then no way! (I'm mixing metaphors here. I switched wballard's %s to length to make the point easy to see.)

(4) I've ignored a few complicating factors. If you have a finite population, then there is something called the finite population correction factor that intrudes itself. Assuming 400 is small relative to the population size, this itself is safe. If not, then actually you should be safer. Consider if you sample 400 out of 500 items. You are going to be pretty certain even without any statistics. Also, jjwhite raises the issue of non-normality. This will not usually be a concern since in wargaming we deal with well-behaved distributions, so the Central Limit Theorem is applicable. All you need is that "s" be finite, which is not hard to achieve even if the dice are loaded. (That's a joke, but it is still true.) jjwhite's concern about subpopulations is covered by my assumption you were talking about a "reasonably representative sample . . . from . . . similar items." Dividing and conquering doesn't work in sampling theory. (That's another joke.)

Rick

Florida Tory

My comment about sensitivity comes from analyzing random number generators for use in some modeling activities. 400 samples did not even come close to determining if the generator was random 'enough'. We had to get to about 100,000 pairs before the standard deviations got small enough to call it. Which was the point of my comment, how sensitive do you need to be.

And what happens when you are dealing with relatively rare occurrences such as rare diseases. If you have a sample with no cases it gets pretty hard to say anything about it.

Sample size does depend on what you're looking at.

There has been a recent spat re polling methods here in the UK. YouGov sends out email invitations to those signed up, who then take the poll online. This obviously means they can poll more people, as no one has to ring each respondant up, so there is no 'running our of time'. Obviously they hold information about you to get a correct cross section. I get far fewer polls than my wife, as I guess as a 45-55 old woman there are reletively fewer of them than me- male 35-45.

The traditional polsters say that their method is inaccurate, the arguement seems to revolve round internet users are unrepresentative. YouGov say it's just the older companies don't want to admit they are overcharging, and they have an awful lot tied up in call centres etc.

Last Hussar

Having worked in survey research and used internet methods such as YouGov there are concerns depending upon what you are expecting to get and how you report on it.

With the 'classic' phone interview there -may- be a lot of precalling information about the group of people you are attempting to survey, usually based on geography (at least in US). You find out how many phone numbers are assigned to an area and randomly select phones to call allowing calculation of a first stage sampling probability.

This is the major difference when working with internet surveys, especially when using a third party provide the email requests. Generally you do not have -any- idea how many valid email addresses reach people in the area of interest. This is complicated by people with multiple email addresses.
Many of the survey email providers have some sort of incentive plan to encourage people to participate in the surveys. It quite likely that some of the participants provide multiple email addresses and/or personal profiles to increase the frequency of invitations.

There is also the issue that not everyone has regular access to the internet and an email address, but that is becoming more of secondary concern as time goes on.

Incentives encourage a greater percentage of poorer people.

No incentives encourage a greater proportion of pensioners (or people who have nothing better to do).