Richard Tol, academic freedom and the ESRI.

The ESRI-Richard Tol brouhaha has been strangely ignored by many academic blogs. I guess its because we are all on holidays…

To summarize; in May 2012 Richard, who had left the ESRI under rather strained circumstances earlier this year uploaded a working paper on the welfare-work wedge, whether and under what circumstances it is “better” to be on the welfare than to work. A part of the findings were that for some cohorts there was a significant wedge.

The paper was, according to the details on IDEAS (along with SSRN one of the larger repositories of working papers) uploaded in May.

It appears to me, but maybe I misunderstood, from the various statements that staff in the ESRI may be able to upload papers without any formal checking, but then again they are seasoned academics and researchers who will not generally issue working papers that are manifestly flawed as that will rebound on their reputation.

In any case, the newspapers picked up the findings, and it was widely reported. Here we must note that it was generally reported as “ESRI report” finding something. It was not. To describe it as such is to misunderstand the nature of the research process and to disregard the clear statement on each ESRI working paper that it is not official and should not be treated as such. Why the newspapers and other media choose to so mislead is a source of concern for another day.

In any case, yesterday the ESRI took what it described as the unprecedented step of withdrawing the (unofficial) working paper, declaring it to be in effect shoddy and error prone. It is now no longer on the ESRI working paper site, but the ever reliable Michael Taft has an extensive summary of it here and the entire paper can be downloaded here. Why it felt the need to create a precedent where none existed is unclear. This is worrying for those of us concerned with academic freedom. The ESRI press release states the ostensible reason for withdrawal as

“it has emerged that the underlying analysis requires major revision and that the paper’s estimates overstate the numbers of people who would be better off on the dole than in work.”

This seems to me to fly in the face of how research works. The ESRI working paper series is like any other. Papers start as ideas, then go to drafts then to working papers and then after informal and formal critiquing at seminars and conferences may be submitted for publication where extensive peer review suggests major revisions and then perhaps eventual publication. If the ESRI are now to require all working papers to be error free then they should set up a journal. As a journal editor the only time I would countenance the withdrawal of a paper from any series or publication is if it becomes clear that there is plagiarism, fraud or unethical behavior. None are alleged here. What is clear is that the public debate spooked the ESRI.

When the ESRI next publish a working paper on any area of controversy (and as a social science research institute that is going to happen on a regular basis) we now can no longer be sure that the publication is the thoughts of the author pure and unsullied, but is instead the outcome of internal editing. Nat O’Connor notes that the whole point of working papers is to gain comments from a wide range of people prior to further work ongoing. Editing, as we now seem to see will be the norm at the ESRI, implies control, and control is manifested in a power relationship. The role of an editor or a research institute director, in my view, is not and should not be to ensure that material which might confuse the public is not published. The role should be to ensure that the abstract and press release make clear the main finding, the strengths and weaknesses of the paper, and its role as merely one stage in academic research.

A more reasoned response from the ESRI might have been to note the unofficial, working paper nature, stress that it was part of an ongoing process, and to suggest that (as is always the case in social science) we cannot be sure and must weigh countervailing evidence. What has happened now is frank censorship – and that is not a good outcome. Nor is it feasible in the wired world.

I am neither a labour or tax economist, and so have no clue about the academic quality of the work. It seems methodologically sound, but the data are weak, and that is acknowledged in the paper. Do we wait for perfect data to investigate social phenomena, or press on, carefully, with poor data and the inevitable extra hedging of our conclusions that that will bring? I do know that Tol is a most prolific researcher who well understands the nature of the research process and who is imho unlikely to have put out knowingly flawed work. Although best known as an economist, he has published in many areas, including tax policy. The ESRI director, Frances Ruane, is also a seasoned, well regarded researcher . When the paper, as it almost certainly will, appears in a peer reviewed academic journal with the main thrust of its findings unaltered as seems to be the case, the ESRI will stand as having for publicity or politic reasons withdrawn a working paper. That mars its reputation and the reputation of Irish academia.

17 thoughts on “Richard Tol, academic freedom and the ESRI.

  1. Pingback: Ninth Level Ireland » Blog Archive » Richard Tol, academic freedom and the ESRI

  2. cormac

    Interesting post, and many thanks for the plug.
    I’m not sure I agree that the story is an issue of academic freedom. Given that ESRI publications are quite influential amongst politicians and civil servants, I would expect a high standard and was surprised to learn the paper in question is still in the revision stage as regards publication in an academic journal. I gather this is the norm, but that’s what surprises me – it’s a rather strange norm!
    I would have thought papers from an important, influential body like the ESRI on sensitive economic topics would be of a standard well above your average academic paper, not below it. If it’s just an exploratory paper, then it received quite a lot of media attention, even before it was pulled. If there are any issues of methodology it’s very unfortunate that it should receive so much attention…

    1. brianmlucey Post author

      Its the norm in every area. Working papers are the lifeblood. In the social sciences the turnaround times are so lengthy the only way to get material out for discussion is WP format

  3. Jerry

    There is not such thing as a paper that settles a question, whether published in a journal or in some official publication. Having worked at an institution and at universities, I can say that official institutions can be quite sensitive about subject matter and findings. It is not surprising. Regrettable but not surprising. ESRI is no worse than some U.S. institutions.

  4. Pingback: The Irish Economy » Blog Archive » Richard Tol and the ESRI

  5. bossbutteringbee

    All credit to you BL for addressing ‘Tol-dole-gate’ amid silence elsewhere. Thank you too for the hotlink to actually read the WP.
    The central finding of the paper was that costs associated with conventional work patterns are now so high in Ireland that they pose a serious demotivation for many, given the alternatives. The extensive evasive action and attempted distortion of the working paper’s findings by many talking heads today when asked to engage with this issue is typical of the times we live in. And yes, the issue of academic freedom most definitely arises, in the face of propaganda, a barrage of circumstantial noise and distortion from vested interests.

    Tol as a (occasional) truthteller about Irish economic issues challenges the sustainability of a number of things that have been done in Ireland in the name of public good but which were ill thought out and were implemented in a vacuum of strategic thought. In this case, just one of the root issues is commuting cost. This however is linked to parking charges, fuel and road tax pricing, uncoordinated and embryonic public transport, traffic calming policies and one of the most non-nuclear settlement patterns in the EU. And most of these latter issues reflect a major Irish problem with over-politicised state agences of patchy competence who struggle to deliver basic infrastructure in a cost effective manner.

    Perhaps the unwelcome truth that Tol has drawn attention to, is that an increasing number of Irish people, faced with the choice between low paying salaried employment options with no tenure, or the uncertainties and sacrifices of self-employment/entrepreneurship, have simply decided to kick back and hibernate. As the state sector exacts increasing rents on use of public infrastructure and services for economic activity, to try and maintain an unreformed and unaffordable welfare state, more and more workers look at the cost/benefit aspects and decline to work.

    I predict that extensive restructuring will come during the next five years, accompanied by deficit elimination, workshare schemes, substantial pay and pension reductions for public sector workers, reduced transfer payments. Much further ahead – a decade or two perhaps – we’ll finally see the retreat of the dead hand of the state from areas such as enterprise training and support, scientific and social research, utilities, higher education, air and sea links, housing and the many other areas in which it has no real business, and from which it has driven out entrepreneurs and replaced them with rent-seekers. When this has progressed to the extent that the costs of earning an honest living fall to core EU levels, Ireland may once again experience real-economy growth, in place of the sterile tax-shifting brokerage that seems to be the sole remaining value proposition. And perhaps then we will see fewer fear-driven responses when an outsider like Tol underlines home truths about how out of shape our economy has become.

    1. Pangur Ban

      @bossbutteringbee states:
      “The central finding of the paper was that costs associated with conventional work patterns are now so high in Ireland that they pose a serious demotivation for many, given the alternatives”
      With respect, that is not the central finding. The data on which the findings are based are from the Household Budget Survey in 2004-5 – but the costs identified in the paper did not, in fact, pose a serious demotivation for many, as unemployment in Ireland at the time was less than 5%.
      Putting criticisms of the methodology employed to one side, we don’t know what the costs are now – welfare payments, wages and (some) costs have fallen since then – and we can certainly draw no conclusions as to their motivational effects.
      Unemployment rose dramatically in the early months of 2009 – at a time when there were no significant movements in welfare, wages or costs. It does not seem unreasonable to conclude that there were other factors driving this.

      1. brianmlucey Post author

        ok. My main point of this blogpost is NOT the issues in the paper. I am not a tax/labour economist so wont comment. There are issues on academic freedom that I am most concered about and I would appreciate if we can think on that here?

      2. bossbutteringbee

        @Pangur Ban
        “With respect that is not the central finding”
        I would agree that it is not a robust finding given the age of the data on which its based. But in the years elapsed since 2005, costs associated with the popular forms of commuting have soared. Whatever about the relativities of welfare incomes vs nett salaried incomes today vs 2005, the dominating cost category of working for many – commuting cost and (for some) parking cost – is much higher today.

        Tol’s observations (may we agree to call them that) would be very familiar to low paid commuters to metropolis cities all over the world – Shanghai, Chengdu, Mumbai, London, San Fran – static or deflating earnings and sharply rising commuting costs destroy work incentive.

        It is unfortunate that ESRI has responded as it has done. If fair minded people gave the organisation the benefit of the doubt concerning Tol’s previous charges of political capture, any remaining perception that the organisation ( as distinct from its PI’s) is independent will drown in a sea of cynicism. In the private sector this would raise questions about leadership.

  6. Elpenor Dignam (@Elpenor_Dignam)

    Irrespective of Tol’s seniority Brian I would have expected any working paper to have undergone an in house peer review prior to release into the public domain? Secondly to the best of my knowledge most academic journals will only accept original, previously unpublished work for copyright reasons.

    1. brianmlucey Post author

      Eleanor. Working papers don’t count as publication for journal submissions. The internal processes of the institute clearly need a house clean

  7. Conor

    From an academic point of view, I would have some serious concerns relating to the econometrics underlying the analysis. The say they use a heckman two stage but dont anywhere identify the fact that you need an exclusion criteria variable that effects the first stage and uncorrelated with the regression in the second stage. They use a logit model and OLS and dont do any testing about whether the logit or probit model is more suitable (what is the distribution of the error structure, normal or logistic?) They use a tobit model and dont test for non-normality and no where across any of the models do they treat heteroskedasiticity and ensure their estimates are robust to this consideration. (Maybe they did this and are not telling us…) .

    1. brianmlucey Post author

      none of which means that the paper should be withdrawn as it was. IF there are major issues then that will rebound to their reputation. The point of a WP is to attract these sort of comments.

    2. Pangur Ban

      This is the kind of discussion that should be happening in relation to the paper – any chance you could explain these points in English for us non-economists?!

  8. Conor

    For sure, agreed that Working Papers are there to be evaluated in this way. However, there seems to be a view, outlined by Richard Toll himself that the data is not great but the methodology is fine. I was just trying to evaluate this factor on its merits.

  9. cormac

    This is a very interesting discussion and a great blog. As a scientist who knows little of the world of academic economics, I’m still a little puzzled, not just at the withdrawal, but at the way this ‘working paper ‘ business works. Professor Tol himself has said somewhere that the paper in question is undergoing substantial revision at the bequest of a journal referee. Yet as a working paper at the ESRI, the original might have been influential in the formation of future policy.
    In this particular case, it seems the the paper is saying something that either the Institute or possibly their political masters don’t want to hear; but I’m surprised that working papers don’t go through some sort of review process, if only to protect them from this sort of censorhip….

  10. Richard Tol (@RichardTol)

    The ESRI is both a think tank and a research institute. Reports that are deemed relevant to the public debate are released to the press and public. Early versions of academic work are put on the web and announced through dedicated lists such as IDEAS/RePEc.

    That is the theory. In practice, there is one journalist who keeps track of the ESRI’s academic output and she occasionally writes a piece — small articles somewhere deep in the Sunday Times. She faithfully represents both the research and its early stage.

    These pieces are widely ignored. Last Sunday’s, however, was picked by a journalist of the Independent. He tried to talk to me, but I was on holiday. So, a derivative piece was written. The working paper that was a proof of concept became a research report with a policy conclusion.

    The ESRI should have explained that the paper was a working paper, trying to establish a method to answer an important question.

    Instead, they withdrew the paper because of “methodological flaws” (that remain hidden from me) and claimed that the paper was revised (which it is not).

    Journalists then start asking questions about relationships and political pressure.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s