Here are some thoughts from Maggie Leggett in the Centre for Public Engagement about the recent report on the REF Impact Pilot:Public engagement questions arising from the REF Impact Pilot reportIn November 2010, Hefce issued a report of the findings from the Impact Pilot exercise, conducted with 27 institutions (including the University of Bristol) in 2009/10. The full report is available here -
http://www.hefce.ac.uk/news/hefce/2010/refpilot.htmIn general, from a public engagement perspective this report is positive. There is a recommendation specifically that public engagement should be counted as impact, and looking through the case studies submitted obviously many institutions submitted PE case studies. Features of strong public engagement case studies include:
i. There was a clear link between the department’s research and the engagement activity.
ii. Evidence was provided about dissemination (such as audience or visitor numbers) as well as a clear explanation of the significance or benefits to the audiences.
iii. The activity went beyond ‘business as usual’ engagement (such as public lectures), and attracted widespread interest or involved a programme of activity which was often innovative and was sustainable or created ‘legacy’ resources.
iv. In some cases there were additional, clearly identified benefits to ‘intermediaries’ (such as cultural institutions or curators), local communities or to the economy.
There are, however, some outstanding issues:
1. Evidencing the impact of PE activity. There is an assertion that ‘just activity is not enough’, e.g. ‘Submissions should...make a case for the
benefits arising from the public engagement activity. This must go beyond showing how the research was disseminated.’ This seems contradictory with statements elsewhere in the report, where for example media activity with an estimate of audience number is applauded as an end in itself. Assuming activity is not enough, we need clarity on the evidence of the effect of public engagement that is required. It needs to be accepted that in some cases, evidencing the effects of public engagement would require extensive further research to understand what, if any, change in behaviour or knowledge had been prompted by involvement in a public engagement activity. Earlier iterations with HEFCE suggested they had understood this challenge, and would not require that level of evidence. There is mention of satisfaction surveys in the report, and (whatever my personal views are on the value of such things) we need to know if that sort of thing is sufficient evidence, or if they are looking for something more involved (and if so what)
2. Media activity. In the physics feedback there is the following statement which I simply don’t understand: “Care must be taken not to simply reward the efforts of outreach activities themselves as that does not give a clear indication of the quality, and hence societal impact, of the underlying research. Thus, once the issue of ‘sexy’ or media-friendly research topics is factored out it is quite appropriate to assume that greater media interest in a research programme means just that: a greater impact of the research.” I would say media activity is a process, not an impact – but the last sentence of the quote seems to suggest otherwise.
3. Further guidance. There is an assertion from the earth sciences panel that “the panel felt that further guidance on assessing the impact of public engagement would be helpful”. I would suggest we recommend that this guidance is produced in consultation with the HE community, through the National Coordinating Centre for Public Engagement (which HEFCE funds and of course which we host in partnership with UWE)
4. Cross over with environment section. There is a comment suggesting that departments should have ‘active, operationalised public engagement activity’ – to show continued engagement with issues rather than one off activity. Whilst I think this is laudable, there may be some inevitable cross over with the environment section
5. What does 4* look like? Finally, I don’t find the report helpful in showing what 4* public engagement case study would look like. I think prompting them to either produce very clear criteria, or mock up examples would be helpful
Some of the pilot case studies have now been published, see
http://www.hefce.ac.uk/research/ref/impact/. There are public engagement case studies in English and in Physics, and although they are not graded, they were highly rated by the panels and are published as exemplars. It seems to me there is little 'evidence' of behaviour change - panels have been satisfied with media reach/visitor numbers/satisfaction surveys etc. I would argue that's a good thing, as depending on the project it's almost impossible to get that evidence without running a second research project. The prevalence of working with the media is striking - possibly because it is easy to make the reach sound impressive.
Three members of the Impact Working group have attended meetings with Hefce and Pilot panel members. They have raised some of the above issues. Although no specific answers have been forthcoming, there was a sense that in theory engagement ought to be shown to have some kind of transforming effect to count as highly rated impact. On the other hand it is recognized that finding evidence of this is difficult.
Download this document from the
CPE Impact site, or leave your comments below:
