The Medical Test

[Gomez]

Hi all. First I need to come clean. This isn't a puzzle. Well, not really. It's kinda, sorta, maybe a puzzle, but not the kind of puzzle we normally do round here.

Here's the situation. I wrote a little article about using puzzles as energising exercises to help break up long corporate training sessions. The article focussed on puzzles with counter-intuitive solutions. One of the puzzles I quoted was this:

There's a disease that affects 1 in 10,000 people and kills everyone it infects. There's a test for the disease that is 99% accurate. You take it and it comes back positive. Should you be worried?

My reasoning was 'No. You shouldn't.' This is because the disease only affects 1 in 10,000 people. The test has an error rate of 1%. 1% of 10,000 is 100, only one of which will be ill. Therefore, even if you get a positive result, the odds you are actually sick are 1 in 100.

Anyway, I send this off to the editor and he comes back with this criticism:

"Overall I like the article, but actually, the answer is yes, you should certainly be concerned. This is because the error rate of the test is irrelevant to the frequency of the disease. In other words, although only 1 in 10,000 people can have the disease, the test is 99 percent certain that you are one of those people. Therefore, it's more contingent on how many people took the test and how many came back positive, and it involves slightly more complex statistics. In other words, if 10,000 people took the test and you are the 122nd person to receive a positive test result, there's an 81 percent chance that you're infected, as 99 percent of 100 out of those 122 people definitely have the disease."

Now, I've read this about fifty times today and I just can't make head or tail of it. I'm not saying I'm definitely right, because my grasp of statistics isn't so hot, so I've not queried this with him. I was just wondering, does his criticism make sense? Which answer is correct?

Thanks in advance.

P.S. - By way of revenge I took the riddle out and replaced it with the Monty Hall problem

[Quailman]

I don't think it matters what the infection rate is (1 in 10,000). If you are tested and it comes back positive, it was nice knowing you. If it affects 1 in 100,000 and the test is positive, you're equally screwed.

BTW, I think tests of this sort are usually expressed in terms of the false positive rate.

[Zag]

_{(Edited to correct for my misreading of the problem. I had thought he had said that 1/100,000 has the disease, not 1/10,000.)}

No, you only have a 1% chance of having the disease, which is a LOT worse than the general population, but still doesn't mean that it's time to tell your boss to shove it.

Think of it this way (which, btw, is usually the case with tests that have false positives). In the general population, only 1/10,000 people have the fatal disease, but 1% of the people (100/10,000) have an unusual but harmless protein which presents a false positive on the test. If you show up positive on the test, you could be the 1/10,000 who has the disease, but it is far more likely that you are among the 1/100 who has the harmless protein.

(This includes a slight round-off error, because I'm not accounting for the one person in a million who has the harmless protein but also has the disease.)

[bgg1996]

1:9999
1:99


1,000,000
100 have
1 have, negative
99 have, positive

999,900 don't have
9,999 don't have, positive
989,901 don't have, negative

99:9999 have/don't have
99/10098
1/102 chance you have the disease
_________________
The one member below 18

[bgg1996]

Anyway, if the disease always kills you, I'd still be worried with that percentage.
_________________
The one member below 18

[Jack_Ian]

Baye's Theorem is what you need here.

The problem is in your wording though.
99% accurate can mean that you have a 1% chance of survival, however the more likely scenario is that 1 in 100 tests is either a false positive or a false negative, which then increases your chances of survival to 99.02% which is not as good as the 99.99% that the general population have, but still pretty good odds.

Here's my reasoning:

1,000,000 people tested
100 should die from disease
10,000 will get a false reading from the test

There will be 99 correct positive readings
There will be 1 false negative

There will be 9,999 false positives

Chances of getting a positive reading is (99 + 9999)/1000000 = 0.010098
Chances you have the disease given that you have a positive reading = 99/(99 + 9999) = 0.009803921568627

[Quailman]

So the assumption is that the rate of false positives = the rate of false negatives? I do not believe that is ever the case IRL.

[Zahariel]

Jack_Ian and bgg1996 have nailed it. This is one of the fundamental problems with tests of this nature, especially in the medical field. Simply knowing the "error rate" of the test is completely useless, especially if you have reason to believe that the false-positive rate is the same as the false-negative rate. In fact, a test with this property is itself essentially useless for diagnosing even somewhat rare conditions, regardless of the knowledge of this error rate, for exactly the reason you have described. False positives on a large (negative) population will overwhelm false negatives or even true positives on the smaller (positive) population. A real medical test will accept false-negative rates of as much as 50% or more, in return for a false-positive rate of exactly zero. If you get "negative" from your test but you have reason to suspect it's false, you can usually just do the test again.

This is a real-life problem; many of the early HIV tests had a very small but non-zero false-positive rate, which then completely swamped any actual positive results. Thus causing a HUGE amount of problems in the early 1990s for quite a large number of patients who were incorrectly informed that they were HIV-positive, because of the busted test.

[Icarus*]

I always wondered when you here surveys like: 1 out of 100, etc etc have the disease.

Why do people always assume that of the first 100 people you grab, only 1 of them is infected? Isn't it possible that all of the 100 that you sample are infected, and the 10,000 people that you didn't survey are clean?

It's like that joke, 1 out of every 3 men is gay. Look at the guy to your left, now look at the guy to your right. If niether one of them are gay, then does that mean you are gay? Of course not.

If the test is 99% accurate, and you came up positive - you gonna die.

[Jack_Ian]

[Zag]

[Zag]

Rereading, I realize that the disease affects 1/10,000, not 1/100,000 as I thought (both in that note and my one above). The general principle still stands, though. It's just that you don't get to wildly unlikely until you try to claim that more than 5 of your 100 positives has the disease.

[Ferris]

A good book which makes sense of exactly this kind of counter-intuitive situation is Risk - the science and politics of fear by Dan Gardner.

[jesternl]

Isn't that the difference betrween incidence and prevalence?