"At what point is stats variation significant" Topic

Does anyone know what variation from the norm is considered statistically significant, especially in random numbers? For instance a Guru says he can predict coin tosses- he should get about 50% right. At what % does it need to be looked at because it isn't guess work- 60 right from a 100? 70? Is the % deviation a standard, or does it change with sides of dice- ie if we say he needs to predict 50% better than standard on d10 (ie guessing 15 in every 100) does that hold true for d6 (getting 24 per hundred vs the guess of 16/100), and coin toss of 75 per 100?

link

It depends on the number of successes out of trials given the expected success rate.

You can use the BINOMDIST function in MS Excel to work out the probability.

Statistical significance depends on standard deviation, sample size, variability and other factors. It can be mathematically determined (see Mike's link) but there is no constant or universal "this many" because of the above.

link

Being able to guess a coin toss half the time is *AWESOME* if you know which half of the time.

And you could make a lot of money being right 51% of the time betting on coin flips, if you flip enough.

The problems are that if you sat down and guessed at coin flips for a *long* time, there would be some point at which you looked back and said, "Hey, I just got 750 out of my last 1000, I'm finally getting it!" But it would be random. Random numbers *will* contain patterns.

Suppose, for the moment, that if you go to "Vegas" with $1,000 USD and play Black Jack until you've doubled your money or lost it all, you'll double your money 40% of the time, regardless of the "system" you use. A thousand people do this, 400 double their money, the other six hundred had a good time and forget about the trip.

But one of the four hundred writes a book about his system. He sells a thousand copies to people who use the book. Four hundred win, twenty of them write him a letter about how his system works. He puts those on the back cover and sells ten thousand books (the four hundred are telling their friends, too). Six thousand people lose the money and think, "I knew it wouldn't work," and forget about the whole thing. The four thousand produce the people he puts on his commercials…

Repeat with stocks and real estate.

I once won a game rolling four sixes in a row at the critical moment. I was losing before that happened, I certainly would have lost if it didn't happen. Did I "really lose" that game?

So the question is bigger than you think.

Andrew

Andew,
Excellent post and spoken like a 'true' frequentist.

Jon

As may be inferred from Andrew's post, the significance of statistical variation depends on what economists, for example, would define as the weights or costs or rewards (pick a term that appeals to you), associated with each possible outcome in the model, game, experiment, etc. Thus, variation is significant at the point at which it hurts you to deviate from expectations. Generally, a pure probability model can only define the expected path and the probable range of variation. Your objective function is needed to complete the puzzle, i.e. tell me when it hurts!

At what % does it need to be looked at because it isn't guess work</q?

Don't forget statistically significant doesn't mean it isn't guess work … just that it is unlikely to be guess work.

3 sigma, a common test for significance for example, just means that there is just a 3 in 1,000 chance it was sheer guess work. Unlikely, but far from impossible.

My understanding of this is amateur, and the question you ask is a little sophisticated. I understand the "I just want to know *this*" feeling, but its a little more complicated than that. Its as if someone asked "I just want to know how many figures I need to play wargames."

You might roam around the internet and look for terms like "Significance Test," "Degrees of Freedom," and "Critical T Value."

link

link
link

Be sure and read the "Pitfalls" section in the wikipedia article, the first link.

Andrew

I'm going to try to answer the question you actually asked.

The probability of guessing 60 correct of 100 coin tosses is about 0.0285 from the binomial distribution. In Excel you could write =binomdist(40,100,0.5,true) to return the probability that you are wrong no more than 40 times out of 100. This is "statistically significant" at the conventional type I error rate of 0.05 (i.e. we would only expect to see this deviation from the expected success rate of 50% in about 1 in 20 times by chance). Getting 70 of 100 is much more unlikely (0.00039). The probability is a function of the rate of success (or number of dice faces, cards in the deck, etc) and number of trials that are successful out of number of attempts. It's not unusual to get 6 of 10 right (will happen 38% of the time by chance) but 60 of 100 is much more unlikely (about 3% of the time by chance). Unless of course you try it a whole bunch of times (about 40 times actually) and you'd expect to do it about once by chance alone.

But to answer your last question, the % that would need to be looked at changes with both probability of success and number of trials-

75 of 100 coin tosses right P = 0.00000028
15 of 100 d10 right P = 0.073
24 of 100 d6 right P = 0.036
48 of 200 d6 right P = 0.005

Dave

Thanks (I think!) for the answers- the ones that didn't confuse me I knew mostly.

Allow me to clarify- I know how to do the maths – I know the chance of rolling all 6s on 'd' dice is 1 in 6^n, for instance. I know how to work out the probabilities of having a 'good' round in 'Deal or no deal' at any particular point (and can predict fairly closely the bankers offer). I know that sooner or later you are going to throw 10 heads in a row.

What I am interested in is when do we accept that something isn't 'chance'.

Think of it as the 'James Randi challenge'- how many dice throws from a set of 100 do I have to predict to win his $1m? 17 would be the expected. If I regularly predict 32% on every set of 100. Or say not even predict- I say that no matter how you throw the dice I can force it to come up 6 more than it should by using brain power. The reason for the 68% failure is all the other forces in play competing with my ESP.

I remember from my stats lectures 20 years ago the 95% confidence level, but never really understood how it worked.

Andrew the wiki you posted was probably the best, but either I'm not understanding it, or it isn't what I'm looking for!

Thanks

I always come on these threads long after they have gone cold, so chances are nobody cares anymore. But I'll put in my two cents anyway!

LH, it sounds like you're asking the "when is it a heap?" question. Don't ever ask that of a statistician! And you certainly will not get a convincing answer from anyone, even economists and philosophers, because there isn't one.

Consider again your Randi challenge. What if your guesses improve as you go along. Does this mean you have detected the subtle biases in a not so fair die? Does it mean your ESP is warming up? Or does it simply mean that you are just plain lucky? If your guesses get worse, does that mean anything? How about if they are consistent throughout the experiment? Can you even say anything about your guessing patterns without posing another statistical conundrum?

The same question comes up over and over again in a slightly different form when dicussing the existence (or not) of hot and cold streaks in baseball. Sure you can explain it all as random variation. Or you can latch on to any number of supposedly independent variables that contribute to the observed patterns. None ever really prove causation or even correlation.

Sorry, this wasn't really two cents. More like a wooden nickle.

pigbear- thanks for turning up, you are welcome to warm this up again.

The 'Hot Streak' thing is interesting. I had a book about stats abuse called 200% of nothing. In it the author argued there is no such thing as a 'hot streak' etc- players do not get better, but its just those hits happen to come one after another- like heads being tossed. I disagreed with this section of the book, as it treated people as machines, ignoring the psychology- a player in form will feel better, more confident, and physicaly respond to that- a man on a losing streak will try 'too hard'

What is out of the ordinary is important in gambling that relies on pure chance. Take Roulette.

On the 'normal' part of a table the house will break even, and nothing more. This is because the bets pay out 'true' odds- ie Red/Black evens, thirds 2:1 etc to 35:1 on individual numbers (plus of course your stake back) The house advantage is the Green 0. Although you can bet on it the odd for the table is still only 35:1, but there are now 37 numbers requiring odds of 36:1 (this is right, not 37:1- if you cover every number the pay out of 36:1 would get ypou all your losses, and no more- the stake covers the bet).

For all other purposes the Green 0 loses- its not in a third, column, red black odd or even, so though red pays 1:1, your odds of winning are 18:37

Now obviously if the ball landed on 0 more than one in every 37 spins, the casino profits go up. A bias wheel can lose the casino its licence. At what point do the regulators have to act? when 11 out of 370 land '0'? 15? 20?