January 13, 2004
The American Heritage Index of Economic Freedom
Every year, a US think tank called the American Heritage Foundation publishes something called the "Index of Economic Freedom". The Index claims to measure the economic freedom of the economies of almost every country in the world. Thus, it should give you an idea of whether (say) a Bulgarian in Bulgaria has more economic freedom than a Mexican in Mexico.
The Index rates almost every country in the world on ten categories, including things like "Trade", "Regulation", and "Monetary Policy". It gives each category a numerical rating between 1 (best) and 5 (worst). Then it averages the ten and comes up with a composite rank between 1 and 5. This lets it rank the world's countries from best (Hong Kong) to worst (North Korea). Finally, based on the numerical score, it states whether they are "free", "mostly free", "mostly unfree", or "unfree".
The Index just came out with its 2004 edition, and Romania didn't do so well. Its overall score was 3.7, it came in 129th out of 180 or so countries, and its rating was "mostly unfree".
This has caused a certain amount of consternation here in Romania.
Is the Index an accurate reflection of how things are here? Well, it says that it is. "The internationally acclaimed Index of Economic Freedom remains one of the world?s most reliable and authoritative guides to economic growth: an essential resource for anyone who wants to understand why some countries prosper while others still lag behind." Or so says the website.
Unfortunately, at least with regard to Romania, the Index is pretty worthless.
I think I'd have to do a longer post to explain why in detail, but here are a few reasons.
My biggest problem with this index is that it's so damn crude. It uses a lot of really arbitrary figures, most of which seem to have been chosen for how easy they are to find rather than how relevant they are to economic freedom or economic growth.
Here's one example: "Monetary Policy". (You might want to open up Romania's entry in the Index and follow along.)
MONETARY POLICY
Score: 5.0
From 1993 to 2002, Romania?s weighted average annual rate of inflation was 29.08 percent.
"Monetary Policy", according to the Heritage Foundation, consists of just one thing: average inflation over the last ten years. I checked the entries for several other countries and, yep, this is how they do it for everyone.
Since Romania had a couple of bouts of heavy inflation -- on in the mid-'90s, another around 2000 -- it's cursed with a bad score until at least 2010 or so.
This is particularly stupid, because Romania has had pretty decent macroeconomic and monetary policy for the last two or three years. Inflation is down, interest rates are falling slowly, the leu is pretty stable. And this has been the case for a while.
But it doesn't matter how good monetary policy is. It can be good, and stay good, for the next five years; but the score won't come down from 5.0.
That's just dumb; and it doesn't say anything meaningful about Romania in 2004.
Here's another example:
REGULATION
Score: 4.0
The Financial Times reports, based on a U.S. Agency for International Development study, that ?it takes anything from 49 to 102 days to register a new company: 83 pages of forms have to be completed, weighing half a kilo?. Small to medium-sized enterprises have between 11 and 23 inspections a year?. A business start-up needs between 23 and 29 authorisations and approvals.?
I know the study that they're talking about. It's the IRIS study, done in May 2000. You can find it here (.pdf file -- click on "English version" and then on "Red Tape"). It was a pretty good study, but it's nearly four years old, and much has changed since then. The fact that these turkeys are using it as a basis for their study is... suggestive.
And then there's this:
GOVERNMENT INTERVENTION IN THE ECONOMY
Score: 2.5
The World Bank reports that the government consumed 6.3 percent of GDP in 2001, down from the 12.5 percent reported in the 2003 Index.
Pause right there. Is it just me, or does that seem a really suspect figure? How did government consumption drop by 50% in just one year?
GDP grew only by about 4%; goverment expenditures went down by only about 2%. So... is it possible that there might be an error here?
Based on the decrease in the level of government consumption, Romania?s government intervention score is 1.5 point better this year.
Apparently the Heritage Foundation doesn't give it a second thought.
I could go on. There are ten categories, and I'd say at least half of them are suspect, and two or three are really completely worthless. But, really, the whole thing is just crap.
This wouldn't bother me if it was labelled honestly. If it said, "Here's a very rough, crude measurement culled from easily assembled secondary sources. It applies the same methodologies to the United States, China and Congo, but hey -- we had to cover 180 countries, and this was quick and simple" -- if it said that, I guess I'd be OK with it.
But it says it's "one of the world?s most reliable and authoritative guides to economic growth"; an "essential resource for anyone who wants to understand why some countries prosper while others still lag behind".
And, you know, that's just not so. The more I stare at this thing, the sloppier and lazier it looks. Puffing it up as "authoritative" and "essential" is just adding insult to injury.
Unfortunately, the Romanians appear to have taken it pretty seriously; it got major articles in all the respectable daily newspapers and was mentioned on national TV. More's the pity.
Posted by douglas at January 13, 2004 05:22 PM | TrackBackWelll, this is the Heritage Foundation-- a right-wing neo-con think-tank-- we're talking about. Their whole reason for existence is to push the right-wingers' agenda.
Given that, why would you expect them to have unbiased data?
Posted by: Diane E at January 13, 2004 08:16 PMI just read the methodology chapters. Oh, my aching head.
Diane E, the raw data look to be fine. It's the way the categories are set up and interpreted that makes my head spin.
Chapter 3 is basically a defense of taking the scores from ten categories and *averaging* them, even after their *own* statistical tests show that one of the scores is useless (fiscal burden), and that the heaviest component (informal market) should have *three* times the weight of the lightest (government intervention).
Chapter 5 goes into how the score per category is calculated. OMG. Let's look at factor 1, trade policy. There, you get a number between 1 and 5 based on your tariff rate. That's fair enough. But let's see how the categories are set up:
1 (Very low) Weighted average tariff rate less than or equal to 4 percent.
2 (Low) Weighted average tariff rate greater than 4 percent but less than or equal to 9 percent.
3 (Moderate) Weighted average tariff rate greater than 9 percent but less than or equal to 14 percent.
4 (High) Weighted average tariff rate greater than 14 percent but less than or equal to 19 percent.
5 (Very high) Weighted average tariff rate greater than 19 percent.
It's a *linear* scale, based as far as I can tell on no historical trends or norms. The world's nations in the rip-roaring, uber-capitalist days of the 1920s would all be marked 5, straight down the line. Nor does it appear to be based on current international trade regulations.
Looking at the other numerically-based scales, some avoid even the pretense of linearity: the income tax scale has 10 percent increments up to a score of two per half-point, but then shifts to 5 percent increments after that. Government consumption ranks go on an *exponential* scale, with wider ranges for higher percentages, thus making the difference between a country that spends 6% of its GDP on the government and one that spends 4% of its GDP on the government as important as the difference between a country that spends 11% versus one that spends 39%. And the ranks for scoring for inflation rates look cherry-picked. What's so special about a six percent inflation rank that breaking it over a ten year period makes you go from low inflation to moderate?
Then, in the more subjective categories, there are procedures that automatically bias for higher scores: "A country need only meet a majority of the conditions for a particular score to receive that score," for the regulation grading scale. Which means, to get a 5, all you need is for the indexers to say, "the government impedes the formation of new business and corruption is widespread. That's two out of three; sorry, Charlie."
To my eye, the only saving grace this index might have is if it has been applied consistently in the past ten years. But what I have read doesn't fill me with confidence even that far.
C.
Posted by: Carlos at January 13, 2004 11:31 PMHi,
Doug, interesting as always. But which annual report do you like better? Not, that is, for results -- but for methodology? There are dozens of agencies and NGOs that rank the world's cultures: Amnesty International, Human Rights Watch, Freedom House ... some of which attempt to put a number to the rank, like a Michelin Guide to hotels and resturants.
"UK: 4 Stars (of 5) Good courts, competitive press, responsive political process --somewhat alarming decline in personal security in recent years attributable either to underfunded police forces or confiscation of firearms..."Probably my own biases showing, but it seems to me the "right wing" publications are both (a) more likely to attempt a consistant "scorecard" sort of approach to the rankings, while "left wing" annual pubs tend toward the subjective and contemporary. Freedom House, for instance, shows the past ten years scores for Romania (1=good, 6-worst, "F"ree to "N"ot-free) ranging from 4,4,PF to 2,2,F between 1993 and 2003. Human Rights Watch, AFAICT, has completely omitted discussion of Romania in some years' reviews. (To Romania's credit, I suppose.)
To be REALLY sloppy, I suppose one could take the annual reviews of ten NGOs; create a new scale of 10=best and 1=worst; place the NGO's rankings somewhere (subjectively) all on the new scale
( a "no mention" from HWR weighs the same as either a "1" or "2" from Freedom House" at a "9" on our scale) then do a geometric mean of those ten (somewhat arbitrarily assigned) values ... A stupid and complicated methodology, true, but useful to disguise the highly subjective nature of the initial data selection.
Or, we could just admit to ourselves that most of these evaluations are not-quite-as-useful as a weather report.
First, a big shout out to Carlos for going to the library and looking that up. Thank you; and I'd love to hear more.
As to Pouncer's question... I haven't looked at a lot of these in detail, so I can't really say. About the only other one I'm somewhat familiar with is Transparency International's CorruptionPerception Index. (Scroll down to see the index, and check out the explanatory notes at the bottom.)
I have a lot of respect for that one. Several reasons. One, they tell you right up front what it is -- a /perception/ index, like those surveys that rank colleges, not an objective measurement. Two, they try to use a lot of in-country on-site primary sources. And three, they give you key methodological information right in the index -- score range, standard deviation, number of surveys used, and a link to an explanation of their methodology at the bottom of it all. (The American Heritage index doesn't give its methodology online, BTW, which is why I asked Carlos to look at it in his local library.)
The interesting result of this is that I give TI's "subjective" index more weight than I do the Heritage collection of "objective" statistics; they're more likely to tell me something useful.
Doug M.