Impact and evaluation of public engagement

Some outcomes from the workshop on impacts and evaluation.

• There's a lot to learn from reviewing evaluations already done (meta-evaluations). Unfortunately, too often evaluations are done as a check-marking and this needs to be avoided. It's important to take up the lessons learned; RCUK is doing a bit of this by reviewing evaluations.
• Categorising types of impacts and benefits is a critical step in evaluating public engagement.
• There are then stories and matrices that can be developed based on the type of impact being explored.
• Experience shows that quantitative indicators can be hard to find, difficult (and expensive) to measure; qualitative indicators have value. One of the greatest challenges with them, though, is making qualitative evaluations systematic; how to make generalisations and themes emerge from storytelling and case studies etc.

Top tip: Be clear on why you're doing evaluation (while recognising that motivations may change over time)

• How to select measures which reinforce or triangulate other measures or data - need to be creative, but this is worth thinking through.
• Important drivers for impact evaluation are RCUK and other funding councils.
• Need to find ways to ensure it fits your own purposes too.
• Need to be creative in finding ways to record.
• An important starting point for making use of evaluations is a desire to improve your practice. Ultimately, evaluations (at least formative ones, not after activity is over) should inform your practice.
• Evaluations must be simple and not a difficult add-on; efficiencies should be a cornerstone. Participatory methodologies are worth exploring.
• Self-reflection is also a valid tool in evaluation; how did doing something change you and your thinking on an issue, a methodology, etc.? This might be as valuable as any impact evaluation directed to recipients of your activity. For example, if your practice or insight changes as a result of doing something.
• Seeing and quantifying impacts is extremely difficult/unlikely/not realistic with most short-term projects and research. Too much time needs to pass, too many attributing other factors.
• Choosing the right time to evaluate is also key; to get valid feedback (i.e. choosing to gather responses immediately following an event, vs a week or a month or more later).
• Making meaning of results can be challenging; support in understanding the process of drawing conclusions would be helpful.
• Despite usually messy and less than rigorous methods, evaluations that seek trends or generalities are still important. A basic sense of costs vs. benefits of doing something (vs. not doing it) can be really important.
• Evaluating inputs and processes can be really helpful, and more realistic sometimes than looking for long-term impact.
• There's lots of expertise in evaluation out there, just as there are in public engagement. Researchers can't be expected to become experts in this area and so need to have help to connect to appropriate resources and people. This doesn't have to be expensive ongoing evaluation support, just resources, possibly a workshop, etc.
• Language used in evaluations is important - can isolate others.

Steven Hill, Research Council UK Strategy Unit

Dr Steven Hill from Research Councils UK presents perspectives on public engagement as part of the conference plenary, looking specifically at the impact agenda and the role of public engagement.

Excellence, impact and engagement

View more documents from cpebristol.

Some main points below.

According to RCUK, impact means 'demonstrable contribution that excellent research makes to society and the economy'.

Researchers are expected to:

• disseminate their findings widely
• engage with users of research/public
• ensure exploitation of research findings
• equip their students with engagement skills

Engagement matters because:

• research is responsive to societal needs
• benefits of research are shared through talking to the public about it
• it maximises the impact of research - leading to the development of an accessible responsive research base. The public can see the difference that research makes which might inspire the researchers of tomorrow.

The RCUK wants to find out what people think about science and research, and to reach them by engaging them in contemporary research. It also aims to encourage young researchers to engage with the public using different tools to keep people informed.

An example of investment supporting researchers public engagement includes the Beacons for public engagement - an initiative charged with culture change within higher education institutions. RCUK also supports a number of schemes that help researchers engage directly with the public (e.g. NSEW awards, centrally-managed scheme).

Other ways in which RCUK promotes public engagement include:

• supporting the placement of researchers in schools, sowing the seeds of contemporary science in classrooms
• helping early career researchers to communicate their work widely - see the Vitae programme website for further information
• running poster competitions for early career researchers - emphasising impact of their research on society.

Dr Hill also described a forthcoming publication looking at the benefits of public engagement from the perspectives of the researchers. Their conclusions to date are that these benefits can best be described through a narrative/case study approach (do you agree?).