Statistical Support for the Sixth Amendment

 Posted on July 02, 2007 in Uncategorized

The criminal defense world is abuzz over this analysis (PDF), which estimated that juries in a recent National Center for State Courts (NCSC) study reached "accurate" verdicts only 87% of the time. In the 290 cases analyzed, the author, Bruce D. Spencer of Northwestern University, estimated that 10% of convictions were wrongful, and that the chance of an innocent person being convicted was 25%.

Spencer makes it clear: the NCSC study "should not be regarded as a representative sample with respect to jury accuracy." In fact, the NCSC study doesn't say anything about cases outside the study: "the statistical inferences do not extend outside the cases in the NCSC study." I know that it's fun to do so, but the author deprecates the use of his analysis to draw conclusions about juries overall. My comments on Spencer's estimates apply only to the 290 cases included in the NCSC study.

Spencer isn't a lawyer, so he's looking at the system from an outsider's frame of reference. What he's trying to establish is the probability that a factually innocent person was convicted, and that a factually guilt person was acquitted:

The interpretation of the latent variable U requires care: does it refer to correct status under the omniscient viewpoint, or under the procedural viewpoint, or something else? Although we want U to refer to the omniscient view of the true state, the data analysis uses statistical patterns of agreement observed in real cases to estimate the parameters in the models.

(my emphasis). (For the difference between factual guilt and legal guilt, see my posts, Factual Guilt vs. Legal Guilt and Right v. Wrong.)Spencer describes a factually innocent person being convicted as "type I error" and a factually guilty person being acquitted as "type II error." I'm not interested in type II error - this is the system functioning as it was deliberately designed to do; I don't consider it error at all. Within the framework of the system, when the jury acquits it's never wrong because the accused is, by definition, legally innocent.

Type II error is of interest to those who have faith that the system works. I don't care about type II error, except to note that it is less common than type I error - that is, that it is less probable that a factually "guilty" person will be acquitted than that a factually "innocent" person will be convicted. This would appear to be a systemic violation of the "benign... old... general... maxim... that has been long and generally approved" that it is better that 10, or 100, or 1,000 guilty men escape than that one innocent suffer. (a must-read: Alexander Volokh's n Guilty Men, 146 University of Pennsylvania Law Review 173 (1997)).

The only data Spencer uses from the NCSC study are: juries' verdicts, judges' opinions, juries' opinions on the strength of the evidence, and judges' opinions on the strength of the evidence. Spencer first estimates jury error by comparing juries' verdicts with judges' opinions. "Simple estimates of jury accuracy can be developed form the judge-jury agreement rate," he says, "the judge's verdict [opinion] is not taken as the gold standard." He makes the assumption that judges aren't always wrong - that sometimes, when the jury acquits and the judge would convict, it's a type II error rather than a judging error. So some cases in which the jury acquitted but the judge would have convicted are counted as type II errors. Spencer explicitly assumes that the judge is at least as accurate as the jury. I wouldn't be comfortable making this assumption, but I'm a lawyer who actually deals with judges every day and not a statistician who might, perhaps, hold them to a higher standard than they deserve. Spencer does give alternate estimates for the probability of a correct jury decision if the judge is more or less accurate than the jury.

If you accept Spencer's statistical analysis and assumptions, it's notable how much more often judges would commit type I error - convicting the factually innocent - than juries.

Even if you don't accept Spencer's analysis and assumptions, it's notable how much more often judges claim that they would have convicted than juries actually convict. In 47 of 290 cases, the judges opined that the accused was guilty while the juries acquitted. In 15 of 290 cases, the juries convicted while the judges opined that the accused was innocent. So in those cases in which there was disagreement between judge and jury - in other words, in which it would have made a difference whether the judge or jury heard the case - more than 75% of the defendants had cause to be glad they had jury trials. The judges would have convicted more often than the juries whether the evidence (as rated by judges or by juries) was strong, medium, or weak.

Clearly, if juries get it wrong 13% of the time overall the system is not perfect. But we all know the system isn't perfect, and that in itself doesn't mean it isn't working. If, however, 10% of convictions are wrongful, and if a factually innocent person has a 25% chance of being convicted, the system isn't working. But this would be an indictment of the system, and not of juries, because whatever can be said of an accused's chances with the jury, it is clear that they are worse with a judge.

Technorati Tags: jury trial, statistics