With the ongoing problems with our public school system, it is natural for people to look to find some objective criteria to judge which schools, states, programs, etc. are working and which aren't. Given the importance of this area, obviously there is a lot of attention focused on the problem. Unfortunately, almost all of the data that people try to use to perform this task are woefully inadequate and/or misused. That those who use this data are by and large unaware of the problem only compounds it. Two of the most egregious examples, to my mind, are drop-out rates and SAT/ACT scores.
While drop-out rates are important in some respects, it is important to keep in mind the accounting principle of comparability. When comparing the financial statements of two different companies, it is vitally important for any type of meaningful analysis that they are following the same rules in recognizing income, expenses, etc (see. Enron et. al). It is just as important when comparing drop-out rates between states, and thus in effect makes any such valid comparison impossible. The reason is that differing states can have wildly differing stances on social promotion, widely different demographics, varied levels of difficulty in curricula, require passing grades on a certain standardized tests (versus not doing so), etc. etc. Drop-out rates can be a valuable tool for evaluating a particular school, or school district, but it's usefulness does not extend much farther in scale. It's also important to keep in mind that drop-out rates generally say something about how well a school or school system does with the children at the bottom, but says little about it's utility for the average or 'gifted' student .
SAT scores are just as bad. First, they have the converse problem to drop-out rates: they only measure the students who are planning to go (or at least wish to go) to college. The fact that the percentage of students who fit this criteria varies from school to school, from school district to school district, from county to county, and from state to state, should thus put you on your guard that composite SAT scores are highly misleading. Yet every year, state rankings are announced with great fanfare across the country, generally with only token or no acknowlegement of the limitations. This is one of the few pieces I have seen to highlight these limitations, though it is notable that it is in response to a politician trying to make hay out of the raw numbers.
With regard to murder rates, you would think that it would be pretty easy to come up with a murder rate that would compare quite well between different states and countries. It seems that you would be wrong. From this Reason article comes the startling news:
[i]"The murder rates of the U.S. and U.K. are also affected by differences in the way each counts homicides. The FBI asks police to list every homicide as murder, even if the case isn’t subsequently prosecuted or proceeds on a lesser charge, making the U.S. numbers as high as possible. By contrast, the English police "massage down" the homicide statistics, tracking each case through the courts and removing it if it is reduced to a lesser charge or determined to be an accident or self-defense, making the English numbers as low as possible. [/i]
In other words, if someone kills another in self defense, that death would be counted as a 'murder' in the US, but not in the UK. Also, if someone drives recklessly and kills another in a car accident and eventually pleads to vehicular manslaughter that would be counted as a murder in the US, but not in the UK. There are clearly any number of similar scenarios which would result in something counted as a murder here but not there. This goes a VERY long way to explaining how the UK can have such a higher violent crime rate, and yet have a lower murder rate. The disconnect had puzzled me for years, and, as far as I have been able to determine, the Reason article is the only place that this difference of methods has been articulated, and even here it was somewhat buried in the middle, and not given nearly the attention it deserves.
We finally get to the economy and where the idea for this post originated. When looking over what John Quiggin wrote (see below), he was relying in part on country-wide productivity measures.