Regression to the Mean
Key Concept of Regression to the Mean
Regression to the mean is a statistical phenomenon where if a variable is extreme on its first measurement, it will tend to be closer to the average on its second measurement, and vice versa. This concept, often misunderstood or unrecognized, highlights how any extreme performance, whether good or bad, is likely to be followed by performance that is less extreme.
Misinterpretations and Practical Examples
Flight Training Misinterpretation: A flight instructor observed that cadets' performances worsened after praise and improved after criticism. However, this was a classic instance of regression to the mean, misunderstanding random performance fluctuations as effects of the instructors' actions.
Demonstration with Coin Tosses: To explain this concept to flight instructors, Kahneman conducted an exercise involving coin tosses aimed towards a target, displaying that performance fluctuation was natural rather than influenced by external praise or criticism.
Sports and Media: Common misconceptions include the "Sports Illustrated jinx," where people believe that athletes featured on the magazine cover perform poorly the next season due to overconfidence or pressure. In reality, this is another example of regression to the mean, where an unusually great performance is likely followed by a more average one.
Statistical Insights into Everyday Observations
Lucky and Unlucky Days: In sports, like golf tournaments, a player doing exceptionally well on one day is likely benefitted from some luck, which is not guaranteed to continue, leading to more average performance subsequently.
Everyday Misinterpretations: Everyday observations of regression are often causally misattributed to luck or specific actions, though they simply reflect statistical inevitability due to less-than-perfect correlations between performance on two occasions.
Challenges in Understanding Regression
Causal Bias: People have a strong preference for causal explanations and struggle with accepting statistical explanations like regression to the mean that do not involve clear causes.
Difficult Conceptualization: Both intuitive (System 1) and analytical (System 2) thinking find regression to the mean a challenging concept, often requiring significant effort and education to understand, and often still misunderstood in practical applications, including scientific research.
Practical Implications and Usage
Treatment and Control Groups: In experimental settings, particularly medical and psychological tests, understanding regression is crucial. Without it, improvements in a target group might falsely be attributed to the treatment, whereas they might just be natural regressions to the mean.
Business and Forecasting: In fields like sales forecasting, a naive prediction would add a flat percentage increase to all sectors or units. An understanding of regression to the mean encourages adjusting expectations based on prior performances, predicting lesser growth for previously high-performing units and greater growth for underperforming ones.
This concept, once grasped, reveals its presence across various fields from sports and media to scientific research and financial forecasting, illustrating a ubiquitous but often unnoticed influence on judgments and decision-making processes.