Richard Tol, guest blogger
Over the last 20 years, I have developed (and later co-developed) an integrated assessment model of climate change called FUND. Model code and documentation used to be available to anyone on request. It can now be freely downloaded. Reproducibility and transparency are cornerstones of scientific inquiry.
Some modelers prefer to keep their code private. There are a number of reasons for this. One reason is the potential for abuse. Someone may borrow your model, do something inappropriate or silly with it, and use the results to embarrass you.
I have borrowed other people’s models. I have found bugs in their codes, or what seemed to be bugs. I always discuss this with the modeler in question. If there really was an error – more often it is a misunderstanding on the part of the outsider – I left it to the modeler to correct this and whatever results that were affected.
Colleagues have treated me, my code and my data in the same way. I have had to publish two errata.
Frank Ackerman borrowed our model. He discovered that in one equation there may be a division by zero. He contacted my co-developer, David Anthoff, on 16 December 2010. In the ensuing email exchange, David introduced Mr Ackerman to our standard diagnostic procedures and explained to him that our results are not affected. David added bespoke diagnostic tests on Mr Ackerman’s request and again found immaterial effects.
Much to our surprise, Mr Ackerman claimed in a blog post on 15 March 2011 that there is an error in FUND. He recently repeated this claim (and again). In an email dated 17 March 2011, Mr Ackerman admits that he used different methods than we do to work around the division by zero; and that if he uses these methods, there is a substantial error in the estimates of the social cost of carbon.
In a comment to yet another one of his postings, Mr Ackerman admits that he “made a one-line correction to the code”. A single change to its code can affect the integrity of a model. Mr Ackerman is wrong to present the changed code as work by David Anthoff and me. In his email of 17 March 2011 mentions more extensive changes to the code. Unfortunately, I cannot release that email without violating Mr Ackerman’s legal right to privacy. I have asked Mr Ackerman for permission to make his email public.
In sum, the computational error identified by Mr Ackerman is his own. He falsely puts the blame on us.
I am reminded of Phil Jones’ “Why should I give information to you when all you want to do is find something wrong with it?” although in this case it is more like “Why should I give my code to you when all you want to do is introduce a bug and blame it on us?” Nonetheless, FUND will remain in the public domain.
Dr Richard S.J. Tol MEA is a Research Professor at the Economic and Social Research Institute, Dublin, the Professor of the Economics of Climate Change at the Vrije Universiteit Amsterdam, and an Adjunct Professor, Department of Economics, Trinity College, Dublin. Ranked among the top 200 economists in the world, his work focuses on the economics of energy, environment and climate.
When I discovered, in late 2010, that the FUND model (version 3.5 and earlier) risks dividing by zero in the core of its calculations of agricultural impacts, I wrote to David Anthoff (the co-developer of the model), pointed out the potential error, and asked if he had considered alternative approaches to that equation.
His first response was to say that it would be fixed in FUND version 3.6, then under development.
I then pointed out to him that simple fixes to that equation caused large changes in the results of the model. He said that he prefers the following technique for dealing with the algebra error: run the model as is, and then inspect the results to see if they are driven by a handful of outliers. He offered his conclusion that the results used by the US government’s calculation of the SCC were not affected by outliers, and that the results were not materially affected by removing all the runs that come closest to division by zero.
That’s one way to deal with a known error in a model: run it as is, then manually inspect all the results (we’re talking about 10,000 iterations of a Monte Carlo analysis) to see if the error made any difference.
To spell out what I did instead, I need to briefly explain the error. In FUND version 3.5 and earlier, one of the equations contains fractions with the denominator (10.24 – 6.4T), where T is a Monte Carlo variable, drawn anew from a probability distribution for each iteration. (In fact, it is drawn once for each of 16 regions for each iteration.) If T = 1.6, the denominator is zero. And it turns out that 1.6 is fairly close to the mean of the probability distribution for T, so it’s reasonably likely to draw values quite close to 1.6. This makes the denominator close to zero, and the fraction becomes enormous as a result.
To eliminate this risk – here’s what Richard Tol refers to as the “bug” that I introduced – I tried either of two approaches: using a constant, average value for T for each region, preventing it from taking on values close to 1.6; or alternatively using a constant global average value for the denominator as a whole, clearly preventing division by zero. Either of these methods produces quite a large change in the results of FUND.
So take your choice. You can accept Tol and Anthoff’s assurance that running the FUND model with an algebra error that risks division by zero is okay in practice, because they have manually inspected the results and found it doesn’t matter. Or you can accept my approach, which finds that either of two obvious, simple, one-line fixes to the code to avoid the division by zero error produces a large change in the result. I would expect the developers of FUND to be interested in doing their own research to figure out why these two approaches differ so greatly.
Regarding my supposed refusal to have my communications on this subject publicized, I have no idea what Tol is talking about. In my March 17 e-mail to him, I said
“I would be happy to have your objection and my response posted on the blog site that ran my piece; please let me know if you want to pursue this.”
He declined that offer in an e-mail to me on March 18. This blog post is the first I’ve heard of his desire to publicize our exchange. Sure, go ahead.
Rather than reading months-old e-mails, those who want to understand what I did with the FUND model might find it easier to look at my paper on the subject: it’s at http://sei-us.org/publications/id/375.