Diagnostic Tests

In many fields people try to predict group membership. One example includes making diagnostic predictions (e.g., does this person have a disease or diagnosis?). Another example is predicting who will attempt a terrorist act. Malcolm Gladwell, in the New Yorker, has an excellent article on this topic. One difficulty that he and Meehl and Rosen (1955) point out is the base rate problem.

For example, we could ask: What is the probability that people have panic disorder given our new measure of panic disorder identifying them as having panic disorder? Let’s start with two extreme examples. If no one in our sample has panic disorder (the probability is zero; p=0.0), then there are no people to find with panic disorder and we are wasting people’s time (anyone we identify with panic disorder actually does not have it, so we are wrong when we identify them as having it. If everyone in our sample has panic disorder (the probability is one; p=1.0) then everyone has panic disorder and our test is great at identifying anyone with panic disorder (we have high accuracy because we can’t go wrong), but we do terribly when we miss people. In both of those examples, our test could be a great test or an awful test, but it does not have great utility because of the extremeness of the samples.

A similar problem exists when we try to identify people who want to conduct a terrorist act. There are very few of these people in the population (so probability is close to zero). Thus, we incorrectly identify many people who aren’t terrorists in the hope of finding those few people whom are.

The mathematics of these probabilities were figured out by the Scottish logician Thomas Bayes. Bayes' theorem requires three estimations:

1. The base-rate for panic disorder (we’ll continue with the same panic disorder example) i.e. what proportion of the population taking the test have panic disorder;
2. The accuracy rate of the test, i.e., the probability that people with real panic disorder will be identified as having panic disorder by our test;
3. The misidentification rate of the test, i.e., the probability that people without panic disorder will be misidentified by our test as having panic disorder.
Figure 1. General Formulas for Diagnostics
If we were to use our new test for panic disorder on the general population (which probably has a base rate of 3 to 10% of people having panic disorder) we will do much worse at identifying panic disorder than if we test people in out specialty panic clinic (where everyone walking through the door thinks they have panic disorder. Thus, the population at this clinic probably has a much higher rate of panic disorder). The above figures illustrate this concept by using a hypothetical test with a sensitivity (identifying people who have the disorder) of .85 and a specificity (identifying people who do not have the disorder) of .89 (these numbers would indicate that we have a pretty good test).

By keeping the same test, which has the same sensitivity and specificity of finding panic disorder and changing the sample of people we test, we find greatly different results. Figure 2 shows the sample of people coming into our specialty panic clinic. 96% of these people actually have panic disorder. Figure 3 shows the general population with a prevalence of 5% (5% of the people in this sample have panic disorder). In the general population sample, we incorrectly identify 10 people as having panic disorder, when indeed they do not (although, we do correctly identify 4 out of the 5 people with panic disorder as having panic disorder). In the panic clinic sample, we miss identifying 14 people who actually have panic disorder. Thus, the same test has very different results in different populations.
Figure 2. 96% Prevalence Rate
Figure 3. 5% Prevalence Rate

Risk-Seeking and Framing of the Question

In the August 4, 2006 issue of Science, research discuss a study describing how framing a question can effect how people decide to act. In this study, researchers told participants they would receive a sum of money and then the researchers repeatedly posed them one of two choices. Either the volunteers were told they could keep a chunk of money or gamble, or informed they could lose some fraction or gamble. Those told they could keep money or gamble were generally leerier of risk. On the other hand, volunteers informed they could lose money or gamble often were more risk-seeking.

Daniel Kahneman won the Nobel Prize for his work on this topic. Here is a quote from him on this topic:
“You have two people, both of whom get their quarterly returns on their stock portfolios. One of them learns his wealth has gone from $1 million to $1.2 million, and the other one learns his wealth has gone down from $4 million to $3.5 million. I can ask you two questions. I can ask you who is happier. There is no question the first one is happier than the second. Then I can ask you who is better off financially. The second one is better off. Bernoulli's analysis was in terms of who is better off financially--basically in terms of wealth. But when people think of the outcomes of their decisions, they think much more short term than that. They think in terms of gains and losses. ...It turns out it affects their decision-making in very major ways. If you think in terms of major losses, because losses loom much larger than gains--that's a very well-established finding--you tend to be very risk-averse. When you think in terms of wealth, you tend to be much less risk-averse. I'll give you an example: Suppose someone offered you a gamble on the toss of a coin. If you guess right, you win $15,000; if you guess wrong, you lose $10,000. Practically no one wants it. Then I ask people to think of their wealth, and now think of two states of the world. In one you own [your current assets] minus $10,000 and in the other you own [your current assets] plus $15,000. Which state of the world do you like better? Everybody likes the second one. So when you think in terms of wealth--the final state--you tend to be much closer to risk-neutral than when you think of gains and losses. That's the fundamental way prospect theory departs from utility theory.”

Training in Latent Growth Curve Modeling

There is a great summer institute sponsored by the American Psychological Association at the University of Virginia. Here is the description from the website:

"This ATI is designed to highlight recent methodological advances in the analysis of longitudinal psychological data using structural equation modeling (SEM). This training is designed for post-graduate level researchers and advanced graduate students and covers a range of topics, including fundamental measurement problems, dealing with incomplete data, and new techniques for dynamic analyses.

Course materials will include basic readings on the fundamental theoretical issues in contemporary longitudinal data analysis. These materials will also include all computer scripts (e.g., AMOS, LISREL, Mplus, Mx) that are used in the practical applications. Participants are encouraged to bring along their own data and research problems, and notebook computers equipped with a SEM software program. Up to 30 faculty members and advanced graduate students will be selected to take part in this ATI. Time will be set aside daily for individual meetings with members of the faculty."

More Six Word Stories

I wrote an earlier post on a six word story contest held by Wired magazine in honor of Ernest Hemingway’s six word story (“For sale: baby shoes never worn”).

A six word story is right up my alley for my writing capabilities. Thus, I have written a few more:


Missed opportunity. I forgot the legos.


Hot pretzels, don’t serve well cold.


That’s why I use a blender.


This is the dishwasher Jesus fixed.


I have had a few people submit their six word stories. Please continue submitting them and let me know whether I can publish them in a later entry.

Are Positive Events More Likely?

Many studies have found an effect in which people predict that positive items are more likely to occur to them than negative items. A classic study by Rosenhan and Messick (1966) provided people with drawings of people with smiling faces or frowning faces. In one condition, people had a set of pictures that were made up of 70 percent smiling faces (thus, 30 percent were frowning), while the other condition had 70 percent frowning faces. The participants were asked to predict whether a smiling or frowning face would be shown next. People in the 70 percent smiling faces condition were very accurate in their predictions in that they predicted 68 percent to be smiling. People in the 70 percent frowning condition were much less accurate in that they predicted a frowning face only 57 percent of the time (much less than the actual 70 percent). Thus, it seems that the people in this study were biased to predict positive events as being more likely (or negative events as being less likely). That is, they may have thought that they were more likely to have a smiling face shown to them than a frowning face.

Other studies have found similar effects in predicting life events. Weinstein (1980) asked college students to predict how likely positive and negative events will occur in their lifetime as compared to their fellow classmates. These students predicted that they would be 42 more likely to have a good starting salary after graduation and 38 percent less likely to have a heart attack. Thus, the students in this sample thought they were more likely to have better outcomes and less likely to suffer bad outcomes than their fellow students.

"For sale: baby shoes, never worn"

Wired.com had a short story contest commemorating what Ernest Hemingway called his best work, a six word short story (“For sale: baby shoes, never worn”). Many authors submitted their best stories. Some of my favorites are:

“Nevertheless, he tried a third time.” James P. Blaylock

“Starlet sex scandal. Giant squid involved.” Margaret Atwood

“Thought I was right. I wasn't.” Graeme Gibson

“Please, this is everything, I swear.” Orson Scott Card

“He read his obituary with confusion.” Steven Meretzky

Here is my attempt:
Six words. Creative, he was not.