Reappraisal of 1985 Carlson study finds support for astrology

*See the update* Support for astrology from the 1985 Carlson double-blind experiment.

After nearly a quarter of a century, a small group of dedicated astrologers who participated in a scientific study that reached a devastating verdict against their craft may have something to feel good about. The study, published in 1985 in the journal of scientific record, Nature (vol. 318), launched its author, Shawn Carlson, who at the time of the study was an undergraduate student at the University of California at Berkley, to instant celebrity among the community of scientific skeptics. A new assessment, however, “Appraisal of Shawn Carlson’s Renowned Astrology Tests” in Journal of Scientific Exploration (vol. 23:2) by Suitbert Ertel, professor of psychology at Göttingen University, has found serious flaws in the study's analysis. In an surprising turnaround, Ertel finds that, when correctly analyzed, according to the method that Carlson initially states but then changes, the study's data actually provides support for astrology. It now appears that the reputation of the Carlson study as a definitive test against astrology is unjustified.

Suitbert Ertel is known for his analysis and criticism of statistical research on both sides of the science/astrology controversy. He is also known for his 1988 discovery, and later replications, of planetary eminence effects, which had been predicted by the late astrology researcher Michel Gauquelin. These eminence effects strongly support the traditional astrological properties of the tested planets and have presented an irrefutable conundrum for astrology skeptics.

In the introduction to his study, “A double-blind test of astrology,” Carlson states his intention to design “an experiment that would meet the tight specifications of both the scientific and astrological communities.” The main test in the study challenged the 28 astrologer participants to match the birth charts of 116 volunteer students with personality profiles from the California Psychological Inventory (CPI), a standard personality questionnaire.

In Carlson’s assessment, the astrologers could not match birth charts to profiles any better than chance and therefore failed in their task. He concluded, “We are now in a position to argue a surprisingly strong case against natal astrology as practiced by reputable astrologers.” In Ertel's reassessment of the study's data, however, the astrologers were able to perform the matches significantly better than chance, even though they did not perform as well as they had predicted.

Because Carlson’s study was published in a highly regarded scientific journal, it is exceptional among astrology research papers. It has stood at the pinnacle of scientific recognition and easily ranks as the most frequently cited study of its kind (500+ Google links). A major contributor to the study’s credibility was the participation of qualified astrologers, all members of the National Council for Geocosmic Research (NCGR), an organization that was active in astrology research.

When Carlson’s article appeared in Nature, it immediately drew fire from critics. Some of the astrologer participants protested that Carlson ignored their suggestions, contrary to his stated intention. Carlson had refused to supply the gender identities of the CPI profiles, a necessary consideration because the CPI makes crucial distinctions between male and female responses. The eminent psychologist Hans Eysenck, late author of the book Astrology: Science or Superstition, argued that the CPI explicitly states that it should be interpreted only by trained and experienced users, and the astrologers lacked the necessary training and experience. Other critics questioned whether the CPI and astrology evaluate personality in the same ways, and whether there was enough common ground for astrologers to make valid matches. Many critics have noted that the student subjects failed to recognize both their CPI profiles and their natal chart interpretations, yet the CPI was given a pass while the astrology results were declared a failure. Over time the controversy subsided with neither side being dissuaded.

Eventually, certain aspects of the Carlson study drew Ertel’s scrutiny. Normally, articles in Nature, or any scientific journal, are peer reviewed before publication. The peer review process subjects scientific beliefs and claims of fact to critical analysis by qualified experts. Yet, even though the Carlson study makes claims of scientific fact, doubts had been raised by others as to whether it had been adequately peer reviewed. Nature had published the article in the Commentary section, and this seemed to characterize it as editorial content, where peer review might be less rigorously applied. Moreover, despite the outcry over the Carlson study voiced elsewhere, Nature had never published any responses to the study and no thorough reanalysis had ever been done, and this, Ertel believed, was cause for concern.

Ertel, in his peer-reviewed reappraisal, finds the Carlson study to be flawed in test design, test power, effect size, and sample size. The design of the study violates the demands of fairness, Ertel says, and even Carlson’s own stated protocol. Instead of presenting the astrologer participants with pair choices, which is the normal format for such tests, and the format followed in an earlier well-known astrological study by Vernon Clark (1961), Carlson presented a three-choice format, an unusual method that consists of one genuine object and two selected at random. This three-choice format, Ertel notes, is less powerful than a two-choice format. Furthermore, Carlson’s random selections of the comparison objects (students of relatively the same age) produced avoidable similarities between the objects, which reduced discrimination and further elevated the three-choice problem. In fairness, he says, dissimilar objects should have been used throughout.

Ertel is also critical of Carlson’s "piecemeal" analysis of the sampled data, in which only sub-samples are examined instead of the total effects. The accepted analysis of a three-choice format, as Ertel cites from a standard textbook, is to calculate the proportion of combined first and second choices against the third choice. Carlson initially states his intention to do this but then disregards this method for no given reason. Re-analysis of the published data by using the standard procedure shows that the astrologers correctly matched CPI profiles to natal charts better than would be expected by chance with marginal significance (p = .054). This positive result, Ertel found, was replicable with even better results (p = .04) for the astrologers’ ten-point rating of profiles fit to birth charts, a test that Carlson had requested of the astrologers but the significance of which had eluded him in the end.

In Ertel’s assessment, the proper evaluation of the test data, according to research methods that are commonly accepted to be the most fair and precise, gives two significant test results in favor of the astrologer participants, although there are numerous flaws in the study that cast doubt on any reliable conclusion. “The results are regarded as insufficient to deem astrology as empirically verified,” Ertel warns, “but they are sufficient to regard Carlson’s negative verdict on astrology as untenable.”

*Update* Support for astrology from the 1985 Carlson double-blind experiment
Famous Test of Astrology is Seriously Flawed
A Comprehensive Review of the Carlson Astrology Experiments