They say you can prove anything with statistics, but I’m not so sure. An article in Sunday’s New York Times presents a recent ACS study that purports to challenge the “conventional wisdom” that the Roberts Court is “exceptionally supportive of free speech” by comparing it to previous Courts. The study’s methodology is simple: It compares the rates at which each of the four most recent Courts has ruled in favor of the would-be speaker, as opposed to the government, in free speech cases. For example, the Warren Court ruled in favor of the party asserting a free speech right in 69% of cases; the cumulative rate for the Warren, Burger, and Rehnquist Courts is 54%. By contrast, the Roberts Court has ruled in favor of free speech in just ten of its 29 argued free speech cases thus far—only 34.5%!
The study asserts that the difference is statistically significant (as does this more detailed report on the study), and so it might tell us something interesting if the study were comparing comparable things. But it is not; the study’s comparisons are invalid. It can provide no hint of how the Roberts Court would have dealt with the Warren Court’s cases, or vice versa; nor does it show which Court was more pro-speech for its day, relative to the contemporary median American. The problem is that the assertion that the numbers show a “statistically significant” shortfall in support for free speech by the Roberts Court relies on an implicit assumption that the Roberts Court and the other Courts are deciding comparable cases–either the exact same cases under the same conditions (as in standardized testing), or else comparable distributions of cases drawn from the same pool. Yet that assumption is obviously false.
Supreme Courts do not decide a randomly selected distribution of free speech cases, nor do different Courts decide the same or analogous cases or even pick and decide their cases under similar circumstances. The differences are tremendous, and I see no way of controlling for them (though this study didn’t try). The primary factors are the Court’s ability to pick and choose which cases it will hear, the effect of pre-existing precedents on the cases that come before a given Court, and the effect of the lower courts on the cases that end up being decided by the Supreme Court.
For starters, the Court picks its own docket and exercises great discretion in doing so. That alone means that the cases addressed by different Courts will not be comparable. The Court is much more likely to take a case when it disagrees with how the lower court handled it, or where it agrees with that court’s decision but disagrees with the earlier, conflicting decision of a different lower court (a “circuit split“). But the Court can take any case it likes so long as the lower court loser asks it to do so, and with around 100 denied cert petitions for every petition granted, the Court has massive discretion to shape its docket. A Court can decide to take an “easy” case, in which there is no circuit split and it agrees with the lower court, just to affirm the lower court—and this will make it appear more pro-speech according to the study’s methodology. According to the study, all three of the Roberts Court’s pro-free speech decisions outside of campaign finance and commercial speech were like this: highly publicized decisions on sexy issues (dog-fighting videos, the Westboro Baptist Church, violent video games) that the study characterizes as easy, since the Court decided them 8-1 or 7-2, affirmed the lower court, and acted in the absence of a circuit split. (Stevens, the dog-fighting case, is not a substantial deviation from standard Court practice because the lower court had invalidated a federal statute; the Court is much more likely to grant review in such instances.)
A related point is that a “minimalist” Court will decide less in any given case and so will require several decisions to expand free speech as far as a more active Court might have done in one fell swoop; in that case the minimalist Court will appear more speech-friendly in terms of statistics, even though the active Court actually was the more vigorously pro-speech.
Second, each Court operates against the background of precedent established by previous Courts. When the Court expands rights by striking down state action, as the Warren Court did across a broad range of free speech scenarios, this limits the government in the future. Once the Supreme Court says that public schools cannot require students to recite the Pledge of Allegiance, governments are unlikely to impose that requirement or a similar one in the future (unless they think a new Court might overturn the precedent), so it will not be litigated again; even if they do and it is, the lower courts will almost certainly adhere to the Supreme Court’s initial decision and invalidate the requirement, leaving the Supreme Court with no reason to take the case. The same goes for any issue directly controlled by a Supreme Court precedent. So because the freedom of speech has expanded over the past century with very little contraction, the previous Courts have greatly limited the questions still likely to reach the Roberts Court.
To a substantial extent this means that the “easier” questions get answered first, by earlier Courts. If the Court has been expanding the freedom of speech, then novel free speech cases will tend to be increasingly far from the clear core of the First Amendment protection. This will also happen because, as free speech is broadened, would-be speakers with increasingly far-out positions will start to think they might have a chance of winning, and so will find it worthwhile to press their claims in court (e.g., Pleasant Grove v. Summum; Milavetz). For these reasons, even a modern Court with a commitment to free speech as strong as or stronger than the Warren Court’s will get to make far fewer “easy” pro-speech decisions and will be faced with more hard cases, meaning that its anti-speech decisions will tend to make up a larger proportion of its speech decisions overall. It’s unsurprising, then, that 7 of the 10 cases in which the Roberts Court has issued pro-speech rulings have been in areas—campaign finance and commercial speech—in which previous Courts had afforded notably limited First Amendment protections.
Third, the ideology of the lower courts (especially the federal Courts of Appeals and state high courts) relative to that of the Supreme Court will determine to a substantial extent whether the Supreme Court tends to rule on the “liberal” or “conservative” side of any given issue. In keeping with its stated certiorari practice, the Supreme Court generally grants cert to correct a perceived error by a lower court: it usually reverses the decision it reviews, and even where it affirms it is generally acting to resolve a circuit split, and thus is correcting some other lower court. It is very rare for the Supreme Court to take a case where it is in agreement with every lower court that has addressed the issue it raises (though I’ve just mentioned three such cases).
That the Court so often reverses the Ninth Circuit in favor of the government in criminal cases does not show that the Court is “pro-government” in the abstract (whatever that would mean), or relative to the Warren Court or to the median American. It only suggests that the Court is more pro-government than the Ninth Circuit; if the Ninth Circuit is generally quite liberal, then a Court that always rules for the government when reversing that court’s criminal decisions is presumably more conservative than the Ninth Circuit but may or may not be more conservative than the median American, or than prior Courts. The statistics simply can’t help us decide.
A final point is that society changes, too. Whether or not the change was caused in part by the Warren Court’s expansive free speech decisions, modern America is more pro-free speech than the America of the 1950s. So the government, being fairly representative, is less likely to make obvious infringements on the freedom of speech, giving the courts fewer opportunities for easy pro-speech decisions.
All this means that if we want to draw valid comparisons between different Courts we’ll have to renounce the patina of objectivity provided by the numbers and do a qualitative analysis: we’ll have to actually read the cases, and the comparisons–like those made by Erwin Chemerinsky in this paper on the Roberts Court’s reputation as a champion of free speech–will be speculative at best rather than rigorous or objective.
My hostility is not to the use of statistics in legal scholarship, or even in evaluating the relative ideologies of various judges based on their votes in a certain class of case (free speech, employment discrimination, etc.) rather than the opinions by which they explain those votes–that method simplifies, but it may be valuable nonetheless. Because cases are assigned randomly to judges of a given district court or Court of Appeals, valid comparisons could be made among the judges of such a court in their handling of a certain class of case over a certain span of time. Intra-jurisdictional comparisons (for instance, between Ninth Circuit and First Circuit judges) may be plausible, too, if there is a legitimate way to control for differences in the cases those courts hear or if there is no reason to believe that relevant differences exist. But quantitative comparisons of Supreme Courts at different times, like the ones described in the article, are apt to be meaningless.