Statistics

[Big Lin]

This is a post from an interesting (if old) blog I found on the net.

marylaine.com/myword/numbers.html

November 25, 2001


DOWN FROM THE COUNT
According to the latest official figures, 43% of all statistics are totally worthless.
Ashleigh Brilliant

I got a call yesterday from a machine. It was conducting a poll. Did I approve of George Bush? Yes or no. Was I pro-life? Yes or no. Like I said, it was a machine. I couldn't argue with it; I couldn't say "yes and no," or "sort of," or "not exactly;" I couldn't explain any of the nuances or caveats behind a yes or no answer. Nor could anybody else. This means that that poll cannot possibly give a truthful representation of public opinion, but it will be treated as if it is, because we have a mysterious faith in numbers.

We trust them even though we have no idea what they really mean. When we read or hear about poll numbers, we virtually never know who was asked, or what they were asked. We don't know if they were asked yes or no questions, or if they were allowed to express gradations of opinion, from agree strongly to disagree strongly. We don't know if they had the option of saying "don't know" or "no opinion." That's why, when four polls come out the week before an election with wildly conflicting results, we have no idea why, or which poll to believe.

And yet we believe in numbers. When we argue, we use them as a trump card -- we're not only on the side of the angels, we're on the side of 73% of the American public according to the latest opinion poll.

Maybe we believe because most of us aren't good at math. We don't know the difference between averages and means and medians, and we can't make sense of the difference between millions and billions and trillions, because we never SEE things in those quantities. For all practical purposes, there IS no difference to us; anything over a million is just a whole bunch. That means that we won't recognize preposterous numbers when we see them. If somebody says, "Four hundred million Americans fly on business every year," many of us lack the basic numerical knowledge to say, "Huh? The entire current American population is only 280 million."

We also rarely ask where numbers came from. It's as if we thought they were lying around on the ground like rocks. But even if they were, somebody has to do the counting, and nobody goes to that trouble without a reason. They may need objective data, or they may need data that proves a particular pre-ordained point or serves a particular purpose. Reporters, government officials, insurance agencies and others all have their own reasons for needing to know, for example, exactly how many people died in the World Trade Center attacks.

Gathering that data may seem simple; all you have to do is count, right? But somebody has to decide WHAT to count. Names from lists of missing persons? Death certificates? Identified bodies or luggage or clothing?

Any method of counting is going to take time, but the fact is that everybody wanted a number immediately. So public officials did what most of us do when asked for statistics: they made a guess, maybe 5000 to 6000 people. That guess then took on a life of its own. Reporters and public officials locked onto that original figure and kept on using it, in speech after speech, story after story. The more we heard that figure, the more we believed it.

Meanwhile, New York City's public employees kept on identifying bodies, issuing death certificates, eliminating duplications in the missing persons lists, all the time coming closer to something like a real number, Whatever the real figure turns out to be -- they think now it's fewer than 3,600 -- may not matter, may never even register, because reporters do not bother to check what they think they already know. Or if they do check, they may simply check the database of previous stories, find the old number, and requote it. Don't you think that might explain why Rudolph Giuliani was so reluctant to name a figure early on?

Even apparently objective data can vary wildly depending on who was asked. Are violent crimes going down? The Uniform Crime Reports and the National Crime Victimization Survey, both considered authoritative sources, have different answers. The first one gets its numbers from police departments, the second from asking Americans if they've been victimized by crime. How many children were killed by guns in a given year? The numbers are different depending on definitions: how is the term "children" defined, and how is the term "killed by guns" defined -- homicides? accidents? suicides? All of the above?

Let's suppose, though, that there's a statistic we trust, a hard and fast, government-guaranteed number like, for instance, the number of abortions performed in a given year, or the number of AIDS cases reported. That doesn't mean the number remains that way. We may become part of the problem as we pass the number on to other people.

For one thing, we don't have a good memory for numbers. We can tack on or lose a zero without even noticing. We might mix the number up with another number, perhaps remembering AIDS cases as AIDS deaths. We almost certainly will forget where we got the number: A right-to-life group or NOW? The Centers for Disease Control or ACT UP or Rush Limbaugh? This means that, in all innocence, we may pass our errors on, in articles, news reports, web sites, listservs, bulletin boards, or e-mail. Our mistakes may become part of the historical record. [Did I ever tell you about how Google cited me as a source on how many patents had been issued for mousetraps? Scary, since the number I gave was entirely mistaken.]

The only thing we have to protect ourselves, both from our own bad numbers and other people's, is skepticism. We need to ask where the numbers came from and how they were gathered. We need to ask if they make sense in terms of what we know already: if somebody reports that 16,000 American girls die of anorexia every year, we should be saying, "Wait a minute, do that many American girls die from ANYTHING in a given year, let alone just anorexia?"

In fact, it wouldn't hurt to have some armor in the form of a few anchoring numbers: How many Americans are there? How many of us are married? How many divorced? How many are teenagers? How many of us are baby boomers? Will there really only be two working people paying for our social security in 20 years? How many of us die in plane crashes each year? How many die in auto accidents? (Peter Greenberg, the "travel detective," says, "More people die every year in the bathtub than die in airplane crashes; you can stay off planes to protect yourself, but if you get in the tub, you're on your own.") If you don't want to be suckered by numbers, one of the handiest books (and web sites) you could have around is Statistical Abstract of the United States.

That, and the certain knowledge that God didn't make the numbers. We did, and the last I heard, we're not infallible.

[firedancer]

Interesting post Lin. It wasn't until I had to work in a job that involve conveying lots of medical statistics in a way that could be understood correctly in context and which minimised misinterpretation by the media and the public that I realised just how statistically naive most of us are.

One of the things that I had tirelessly to explain to the public and non-specialist media was how to interpret risk/benefit ratios and the the importance of using absolute risk/benefit alongside relative risk/benefit.

Another bugaboo, for example, was how much the public misunderstands the play of chance.

One of the best reads for a lay person wanting to understand stats and their interpretation that I know is Gerd Gigerenzer's book Reckoning with Risk. It's a real eye-opener on how statistics can be perfectly correct (numerically) yet totally misleading because of the way they are presented.

[june]

97% of statistics are made up on the spot

[Deleted]

Jul 26, 2011 21:29:54 GMT @june said:

97% of statistics are made up on the spot