19 April 2013
Last updated at 19:53 ET
It's 4 January 2010, the Marriott Hotel in Atlanta. At the
annual meeting of the American Economic Association, Professor Carmen
Reinhart and the former chief economist of the International Monetary
Fund, Ken Rogoff, are presenting a research paper called Growth in a
Time of Debt.
At a time of economic crisis, their finding resonates - economic growth slows dramatically when the size of a country's debt rises above 90% of Gross Domestic Product, the overall size of the economy.
Word about this paper spread. Policymakers wanted to know more.
And so did student Thomas Herndon. His professors at the University of Massachusetts Amherst had set his graduate class an assignment - pick an economics paper and see if you can replicate the results. It's a good exercise for aspiring researchers.
Thomas chose Growth in a Time of Debt. It was getting a lot of attention, but intuitively, he says, he was dubious about its findings.
Some key figures tackling the global recession found this paper a useful addition to the debate at the heart of which is this key question: is it best to let debt increase in the hope of stimulating economic growth to get out of the slump, or is it better to cut spending and raise taxes aggressively to get public debt under control?
EU commissioner Olli Rehn and influential US Republican politician Paul Ryan have both quoted a 90% debt-to-GDP limit to support their austerity strategies.
But while US politicians were arguing over whether to inject more stimulus into the economy, the euro was creaking under the strain of forced austerity, and a new coalition government in the UK was promising to raise taxes and cut spending to get the economy under control - Thomas Herndon's homework assignment wasn't going well.
No matter how he tried, he just couldn't replicate Reinhart and Rogoff's results.
We are grateful to Herndon et al. for the careful attention to our original Growth in a Time of Debt AER paper and for pointing out an important correction to Figure 2 of that paper. It is sobering that such an error slipped into one of our papers despite our best efforts to be consistently careful. We will redouble our efforts to avoid such errors in the future. We do not, however, believe this regrettable slip affects in any significant way the central message of the paper or that in our subsequent work.
His professors were also sure he must be doing something wrong.
"I remember I had a meeting with my professor, Michael Ash, where he basically said, 'Come on, Tom, this isn't too hard - you just gotta go sort this out.'"
So Herndon checked his work, and checked again.
By the end of the semester, when he still hadn't cracked the puzzle, his supervisors realised something was up.
"We had this puzzle that we were unable to replicate the results that Reinhart-Rogoff published," Prof Ash, says. "And that really got under our skin. That was really a mystery for us."
So Ash and his colleague Prof Robert Pollin encouraged Herndon to continue the project and to write to the Harvard professors. After some correspondence, Reinhart and Rogoff provided Thomas with the actual working spreadsheet they'd used to obtain their results.
"Everyone says seeing is believing, but I almost didn't believe my eyes," he says.
Thomas called his girlfriend over to check his eyes weren't deceiving him.
Australia, Austria, Belgium, Canada and Denmark were missing.
Oops.
Herndon and his professors found other issues with Growth in a Time of Debt, which had an even bigger impact on the famous result. The first was the fact that for some countries, some data was missing altogether.
Reinhart and Rogoff say that they were assembling the data series bit by bit, and at the time they presented the paper for the American Economic Association conference, good quality data on post-war Canada, Australia and New Zealand simply weren't available. Nevertheless, the omission made a substantial difference.
Thomas and his supervisors also didn't like the way that Reinhart and Rogoff averaged their data. They say one bad year for a small country like New Zealand, was blown out of proportion because it was given the same weight as, for example, the UK's nearly 20 years with high public debt.
"I think that's a mistaken way to examine these data."
There's no black and white here, because there are also downsides to the obvious alternatives. But still, it's controversial and it, too, made a big difference.
All these results were published by Thomas Herndon and his professors on 15 April, as a draft working paper. They find that high levels of debt are still correlated with lower growth - but the most spectacular results from the Reinhart and Rogoff paper disappear. High debt is correlated with somewhat lower growth, but the relationship is much gentler and there are lots of exceptions to the rule.
Reinhart and Rogoff weren't available to be interviewed, but they did provide the BBC with a statement.
In it, they said: "We are grateful to Herndon et al. for the careful attention to our original Growth in a Time of Debt AER paper and for pointing out an important correction to Figure 2 of that paper. It is sobering that such an error slipped into one of our papers despite our best efforts to be consistently careful. We will redouble our efforts to avoid such errors in the future. We do not, however, believe this regrettable slip affects in any significant way the central message of the paper or that in our subsequent work."
Accidents do happen, and science progresses through the identification of previous mistakes. But was this a particularly expensive mistake?
"I don't think jobs were destroyed because of this but it provides an intellectual rationalisation for things that affect how people think about the world," says Daniel Hamermesh, professor of economics at Royal Holloway, University of London.
"And how people think about the world, especially politicians, eventually affects how the world works."
Discovering a spreadsheet error was never going to end the debate over austerity - and nor should it, according to Megan McArdle, special correspondent for Newsweek and The Daily Beast.
"There is other research showing that you can have these slowdowns when you get to high levels of debt," she says. "We have a very vivid [example] in Greece."
Thomas Herndon 's view is that austerity policies are counter-productive. But right now he's delighted that the first academic paper he's ever published has made such a splash.
"I feel really honoured to have made a contribution to the policy discussion," he says.
This
week, economists have been astonished to find that a famous academic
paper often used to make the case for austerity cuts contains major
errors. Another surprise is that the mistakes, by two eminent Harvard
professors, were spotted by a student doing his homework.
At a time of economic crisis, their finding resonates - economic growth slows dramatically when the size of a country's debt rises above 90% of Gross Domestic Product, the overall size of the economy.
Word about this paper spread. Policymakers wanted to know more.
And so did student Thomas Herndon. His professors at the University of Massachusetts Amherst had set his graduate class an assignment - pick an economics paper and see if you can replicate the results. It's a good exercise for aspiring researchers.
Thomas chose Growth in a Time of Debt. It was getting a lot of attention, but intuitively, he says, he was dubious about its findings.
Some key figures tackling the global recession found this paper a useful addition to the debate at the heart of which is this key question: is it best to let debt increase in the hope of stimulating economic growth to get out of the slump, or is it better to cut spending and raise taxes aggressively to get public debt under control?
EU commissioner Olli Rehn and influential US Republican politician Paul Ryan have both quoted a 90% debt-to-GDP limit to support their austerity strategies.
But while US politicians were arguing over whether to inject more stimulus into the economy, the euro was creaking under the strain of forced austerity, and a new coalition government in the UK was promising to raise taxes and cut spending to get the economy under control - Thomas Herndon's homework assignment wasn't going well.
No matter how he tried, he just couldn't replicate Reinhart and Rogoff's results.
Reinhart and Rogoff reply...
We are grateful to Herndon et al. for the careful attention to our original Growth in a Time of Debt AER paper and for pointing out an important correction to Figure 2 of that paper. It is sobering that such an error slipped into one of our papers despite our best efforts to be consistently careful. We will redouble our efforts to avoid such errors in the future. We do not, however, believe this regrettable slip affects in any significant way the central message of the paper or that in our subsequent work.
"My heart sank," he says. "I
thought I had likely made a gross error. Because I'm a student the odds
were I'd made the mistake, not the well-known Harvard professors."
"I remember I had a meeting with my professor, Michael Ash, where he basically said, 'Come on, Tom, this isn't too hard - you just gotta go sort this out.'"
So Herndon checked his work, and checked again.
By the end of the semester, when he still hadn't cracked the puzzle, his supervisors realised something was up.
"We had this puzzle that we were unable to replicate the results that Reinhart-Rogoff published," Prof Ash, says. "And that really got under our skin. That was really a mystery for us."
So Ash and his colleague Prof Robert Pollin encouraged Herndon to continue the project and to write to the Harvard professors. After some correspondence, Reinhart and Rogoff provided Thomas with the actual working spreadsheet they'd used to obtain their results.
"Everyone says seeing is believing, but I almost didn't believe my eyes," he says.
Thomas called his girlfriend over to check his eyes weren't deceiving him.
“Start Quote
New Zealand's single year, 1951, at -8% growth is held up with the same weight as Britain's nearly 20 years in the high public debt category at 2.5% growth”
Prof Michael Ash
But no, he was correct - he'd
spotted a basic error in the spreadsheet. The Harvard professors had
accidentally only included 15 of the 20 countries under analysis in
their key calculation (of average GDP growth in countries with high
public debt).
Oops.
Herndon and his professors found other issues with Growth in a Time of Debt, which had an even bigger impact on the famous result. The first was the fact that for some countries, some data was missing altogether.
Reinhart and Rogoff say that they were assembling the data series bit by bit, and at the time they presented the paper for the American Economic Association conference, good quality data on post-war Canada, Australia and New Zealand simply weren't available. Nevertheless, the omission made a substantial difference.
Thomas and his supervisors also didn't like the way that Reinhart and Rogoff averaged their data. They say one bad year for a small country like New Zealand, was blown out of proportion because it was given the same weight as, for example, the UK's nearly 20 years with high public debt.
"New Zealand's single year, 1951,
at -8% growth is held up with the same weight as Britain's nearly 20
years in the high public debt category at 2.5% growth," Michael Ash
says.
There's no black and white here, because there are also downsides to the obvious alternatives. But still, it's controversial and it, too, made a big difference.
All these results were published by Thomas Herndon and his professors on 15 April, as a draft working paper. They find that high levels of debt are still correlated with lower growth - but the most spectacular results from the Reinhart and Rogoff paper disappear. High debt is correlated with somewhat lower growth, but the relationship is much gentler and there are lots of exceptions to the rule.
Greece is an example of a country with high debt that has suffered a slump
In it, they said: "We are grateful to Herndon et al. for the careful attention to our original Growth in a Time of Debt AER paper and for pointing out an important correction to Figure 2 of that paper. It is sobering that such an error slipped into one of our papers despite our best efforts to be consistently careful. We will redouble our efforts to avoid such errors in the future. We do not, however, believe this regrettable slip affects in any significant way the central message of the paper or that in our subsequent work."
Accidents do happen, and science progresses through the identification of previous mistakes. But was this a particularly expensive mistake?
"I don't think jobs were destroyed because of this but it provides an intellectual rationalisation for things that affect how people think about the world," says Daniel Hamermesh, professor of economics at Royal Holloway, University of London.
"And how people think about the world, especially politicians, eventually affects how the world works."
Discovering a spreadsheet error was never going to end the debate over austerity - and nor should it, according to Megan McArdle, special correspondent for Newsweek and The Daily Beast.
"There is other research showing that you can have these slowdowns when you get to high levels of debt," she says. "We have a very vivid [example] in Greece."
Thomas Herndon 's view is that austerity policies are counter-productive. But right now he's delighted that the first academic paper he's ever published has made such a splash.
"I feel really honoured to have made a contribution to the policy discussion," he says.
